Skip to content

Deploy by committing, and gate every deployment on the fuzz suite #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sAchin-680 merged 1 commit into from
Sep 20, 2026
Merged

Deploy by committing, and gate every deployment on the fuzz suite #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
274 changes: 274 additions & 0 deletions .github/workflows/cd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,274 @@
# Continuous deployment.
#
# The gate is the point. Everything here is downstream of the same correctness
# checks a pull request runs — including a fuzz campaign over the deterministic
# simulator — so a commit that breaks consensus under simulated faults never
# produces an image, let alone reaches a cluster.
#
# That ordering is deliberate rather than conventional. A consensus bug does not
# announce itself: it is a stale read served once under a partition that healed,
# or a committed entry lost in an election nobody noticed. Finding it in staging
# requires the same unlucky interleaving that hid it in testing, so "deploy to
# staging and watch" is not a safety net here in the way it is for a web service.
# The simulator searches those interleavings deliberately, and it is the only
# thing in this pipeline that can.
#
# Deployment itself is a commit. Nothing here runs kubectl or helm against a
# cluster: it writes an image tag into a values file, and ArgoCD does the rest.
# That keeps the deployed version knowable from the repository and revertible by
# reverting a commit.

name: CD

on:
push:
branches: [main]
workflow_dispatch:
inputs:
promote:
description: Promote the current staging image to production
type: boolean
default: false

concurrency:
# One deployment at a time, and never cancel one in flight. A cancelled run
# that has pushed an image but not yet written the tag leaves the repository
# describing a version that is not deployed.
group: cd-${{ github.ref }}
cancel-in-progress: false

permissions:
contents: read

env:
REGISTRY: ghcr.io
IMAGE: ${{ github.repository_owner }}/raftkv

jobs:
# ---------------------------------------------------------------------------
# The gate
# ---------------------------------------------------------------------------

verify:
name: Correctness gate
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true

- name: Static analysis and determinism guards
run: |
make vet
make determinism

- name: Unit tests
run: go test ./... -race -count=1 -timeout=15m

# A test suite that cannot fail is indistinguishable from one that passes.
# Each mutation is a plausible mistake with a named test that must catch
# it; a survivor means that test is decorative.
- name: Mutation gate
run: make mutation

# The campaign the rest of this pipeline is gated on. Every seed is a
# distinct schedule of message delivery, loss, reordering and partition,
# and any failing one fails the build and is replayable from its integer.
- name: Simulation fuzz gate
run: make fuzz FUZZ_SEEDS=2000

# ---------------------------------------------------------------------------
# Build
# ---------------------------------------------------------------------------

build:
name: Build and push
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: read
packages: write
outputs:
tag: ${{ steps.meta.outputs.tag }}
digest: ${{ steps.push.outputs.digest }}
steps:
- uses: actions/checkout@v4

- name: Work out the tag
id: meta
# The commit SHA, not a version and not "latest". A tag that can be
# moved makes the deployed code unknowable from the repository; a tag
# that is a commit makes "what is running in production" a question with
# an exact answer.
run: echo "tag=sha-${GITHUB_SHA::12}" >> "$GITHUB_OUTPUT"

- uses: docker/setup-buildx-action@v3

- uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build and push
id: push
uses: docker/build-push-action@v6
with:
context: .
target: runtime
push: true
tags: |
${{ env.REGISTRY }}/${{ env.IMAGE }}:${{ steps.meta.outputs.tag }}
build-args: |
VERSION=${{ steps.meta.outputs.tag }}
COMMIT=${{ github.sha }}
# The cache is scoped to the branch so a pull request cannot poison
# what main builds from.
cache-from: type=gha,scope=${{ github.ref_name }}
cache-to: type=gha,mode=max,scope=${{ github.ref_name }}
provenance: true
sbom: true

# ---------------------------------------------------------------------------
# Staging — automatic
# ---------------------------------------------------------------------------

staging:
name: Deploy to staging
needs: build
if: github.event_name == 'push'
runs-on: ubuntu-latest
timeout-minutes: 10
environment:
name: staging
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}

# Deployment is a commit. ArgoCD is watching this file and will sync
# within its poll interval; nothing here talks to a cluster.
- name: Point staging at the new image
run: |
set -euo pipefail
file=deploy/helm/raftkv/values-staging.yaml
tag='${{ needs.build.outputs.tag }}'

# Anchored to the tag line inside the image block rather than any
# line containing "tag:", so an unrelated key is never rewritten.
python3 - "$file" "$tag" <<'PY'
import re, sys
path, tag = sys.argv[1], sys.argv[2]
text = open(path).read()
new, n = re.subn(r'(?m)^( tag: ).*$', r'\g<1>"%s"' % tag, text, count=1)
if n != 1:
sys.exit(f"expected exactly one image tag line in {path}, replaced {n}")
open(path, "w").write(new)
PY

if git diff --quiet; then
echo "staging already points at $tag"
exit 0
fi

git config user.name "${{ github.actor }}"
git config user.email "${{ github.actor }}@users.noreply.github.com"
git add "$file"
# [skip ci] so the tag commit does not start another pipeline, which
# would build an identical image and write an identical tag forever.
git commit -m "Deploy ${tag} to staging [skip ci]"
git push

- name: Summary
run: |
{
echo "### Staging"
echo
echo "Image \`${{ needs.build.outputs.tag }}\` written to \`values-staging.yaml\`."
echo "ArgoCD syncs it from git; nothing in this pipeline touched a cluster."
} >> "$GITHUB_STEP_SUMMARY"

# ---------------------------------------------------------------------------
# Production — behind an approval
# ---------------------------------------------------------------------------

production:
name: Deploy to production
# Deliberately depends on nothing. Promotion reads the tag staging is
# already running out of git and copies it across, so it needs no build
# output — and a `needs: [staging]` would be worse than redundant: staging
# does not run on a workflow_dispatch, and a skipped dependency skips its
# dependents, so the promote path could never fire.
if: github.event_name == 'workflow_dispatch' && inputs.promote
runs-on: ubuntu-latest
timeout-minutes: 10
# The approval lives here rather than on an ArgoCD sync button, and the
# distinction matters. A human clicking Sync means git no longer describes
# production — the commit is merged, the cluster is not running it, and
# reverting the commit would not roll production back. Gating the commit
# keeps git true, so a rollback stays a revert.
#
# Configure required reviewers on the "production" environment in the
# repository settings; without them this gate is decorative.
environment:
name: production
permissions:
contents: write
steps:
- uses: actions/checkout@v4

# Production is promoted from whatever staging is running, not built
# again. Rebuilding could produce a different image from the same source —
# a moved base image, a different toolchain patch — so the artifact that
# was exercised in staging is the one that ships.
- name: Promote the image staging is running
run: |
set -euo pipefail
staging=deploy/helm/raftkv/values-staging.yaml
production=deploy/helm/raftkv/values-production.yaml

# Reading the tag and writing it are one script, so the value cannot
# be mangled by a round trip through the shell.
tag=$(python3 - "$staging" "$production" <<'PY'
import re, sys
staging, production = sys.argv[1], sys.argv[2]

found = re.search(r'(?m)^ tag: *"?([^"\n]+)"?$', open(staging).read())
if not found:
sys.exit(f"no image tag found in {staging}")
tag = found.group(1).strip()

text = open(production).read()
new, n = re.subn(r'(?m)^( tag: ).*$', lambda m: m.group(1) + f'"{tag}"', text, count=1)
if n != 1:
sys.exit(f"expected exactly one image tag line in {production}, replaced {n}")
open(production, "w").write(new)
print(tag)
PY
)
echo "promoting $tag"

if git diff --quiet; then
echo "production already runs $tag"
exit 0
fi

git config user.name "${{ github.actor }}"
git config user.email "${{ github.actor }}@users.noreply.github.com"
git add "$production"
git commit -m "Promote ${tag} to production [skip ci]"
git push

{
echo "### Production"
echo
echo "Promoted \`${tag}\`, the image staging was running."
echo "Rolling back is \`git revert\` of this commit."
} >> "$GITHUB_STEP_SUMMARY"
35 changes: 34 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,38 @@ Commit index before and after the handover: **344,703**. The cluster never
noticed the thing managing it had changed.
[deploy/argocd/](deploy/argocd/) has the rest.

### Staging and production, gated on the fuzz suite

Two Applications read the same chart with different value files — staging runs
three nodes, production five — so staging stays a rehearsal rather than a
different system.

The pipeline never touches a cluster. It builds an image tagged with the commit
SHA, writes that tag into a values file, and commits; ArgoCD does the rest. That
keeps the deployed version knowable from the repository and revertible by
reverting a commit.

```
verify ──> build ──> staging ──> production
│ │
fuzz campaign, mutation gate, manual approval
race suite, determinism guards
```

**Everything is downstream of the correctness gate**, and that ordering is
deliberate rather than conventional. A consensus bug does not announce itself —
it is a stale read served once under a partition that healed, or a committed
entry lost in an election nobody noticed. Catching it in staging needs the same
unlucky interleaving that hid it in testing, so "deploy and watch" is not the
safety net here that it is for a web service. The simulator searches those
interleavings on purpose, and it is the only thing in the pipeline that can.

Production's approval gates the **commit**, not the sync. A human clicking Sync
in ArgoCD would mean git no longer describes production: the commit is merged,
the cluster is not running it, and reverting would not roll back. Gating the
commit keeps git true, so a rollback stays a revert.
[.github/workflows/cd.yml](.github/workflows/cd.yml) is the pipeline.

## Measured

Core-only numbers — no disk, no network, no goroutines. They say how much
Expand Down Expand Up @@ -522,7 +554,8 @@ Built in milestones, each with an explicit exit criterion rather than a vibe.
| done | Rolling upgrade under load, measured against a live cluster |
| done | Prometheus rules, Alertmanager, Grafana dashboard, alerts fired on purpose |
| done | GitOps with ArgoCD, drift broken on purpose and corrected in two seconds |
| next | Continuous deployment gated on the fuzz suite, and Terraform |
| done | Staging and production environments, deployment gated on the fuzz suite |
| next | The fuzz write-up, and Terraform |

## Documentation

Expand Down
34 changes: 29 additions & 5 deletions deploy/argocd/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,28 @@
# GitOps

ArgoCD follows `deploy/helm/raftkv` on `main`. Nothing in the cluster is applied by hand; the desired state is a commit.
ArgoCD follows `deploy/helm/raftkv` on `main`, as two Applications — staging and production — reading the same chart with different value files. Nothing in the cluster is applied by hand; the desired state is a commit.

```bash
make argocd-up # install ArgoCD, hand the release over, create the Application
make argocd-demo # break the cluster by hand, watch it corrected
make argocd-up # install ArgoCD, create both Applications
make argocd-status # what each environment is running
make argocd-demo # break staging by hand, watch it corrected
make argocd-open # the UI, with the admin password
make argocd-down
```

## Two environments, one chart

| | staging | production |
| --- | --- | --- |
| Nodes | 3 | 5 |
| Namespace | `raftkv-staging` | `raftkv-production` |
| Log level | debug | info |
| Image tag | written by the pipeline | promoted from staging, behind an approval |

Same chart deliberately. If staging rendered different manifests it would stop being a rehearsal, and the class of bug it exists to catch — the one that only appears on a real cluster — is exactly the class it would stop covering. What differs is the values, and the values are in git.

Three nodes in staging is the one real difference, and a deliberate one: three tolerates a single failure, which exercises every path that matters — election, catch-up, a rolling upgrade holding quorum — at 60% of the cost. What it does not exercise is the two-failure case, which is why production runs five.

## It was broken on purpose

"Self-healing" is a checkbox in a YAML file until something has drifted and been corrected. The failure modes are quiet: a project that forbids the resource, a sync policy that detects drift but never acts, an Application stuck `Progressing` because a StatefulSet never goes Ready. None of that is visible from the manifest.
Expand Down Expand Up @@ -52,6 +66,16 @@ An Application with `selfHeal` is an agent holding write access to the cluster.

`image.tag: "0.1.0"`, not `latest`. A floating tag means the deployed version is whatever was last pushed to the registry — unknowable from the repository, and unrevertible by reverting a commit. Those two properties are the entire reason for this arrangement, and a floating tag quietly removes both.

## What this does not do yet
## Why production also syncs automatically

The obvious arrangement would be to leave production un-automated, so a human clicks Sync. This does the opposite, and the reason is worth stating.

A human clicking Sync means git no longer describes production: the commit is merged, the cluster is not running it, and the difference exists only in somebody's intent. Reverting the commit would then not roll production back — which removes the single property that makes any of this worth doing.

So the approval gates the **commit** that changes the production image tag, not the sync. It lives on the `production` GitHub Environment in [the pipeline](../../.github/workflows/cd.yml). Once the tag is in git, git is true again, and a rollback is a revert.

## Deleting these takes an order

The `resources-finalizer` needs the `AppProject` to still exist in order to work out what it is allowed to delete. Handing both to a single `kubectl delete -f` takes the project first often enough that the Application wedges: the finalizer can never complete, the delete blocks until it times out, and the workloads it should have removed are orphaned in the cluster. Recovering means patching the finalizer off by hand.

Sync is automated from `main` with no staging gate, so a merge reaches this cluster directly. The continuous-deployment work adds the part that matters for a real system: an image built and tagged by commit SHA, staging synced automatically, and production behind a manual approval — all of it gated on the fuzz suite, so a commit that breaks consensus under simulated faults never reaches a cluster at all.
`argocd.sh down` deletes the Applications first, then the project, and clears the finalizer if one is stuck. This was found the direct way.
Loading
Loading