[sAchin-680]

Staging and production as two ArgoCD Applications reading the same chart with different value files, and a pipeline that never touches a cluster.

Deployment is a commit

The pipeline builds an image tagged with the commit SHA, writes that tag into a values file, and commits. ArgoCD does the rest. Nothing here runs kubectl or helm against a cluster, which is what keeps the deployed version knowable from the repository and revertible by reverting a commit.

verify  ──>  build  ──>  staging  ──>  production
  │                                        │
  fuzz campaign, mutation gate,            manual approval
  race suite, determinism guards

Everything is downstream of the correctness gate, and that ordering is deliberate rather than conventional. A consensus bug does not announce itself — it is a stale read served once under a partition that healed, or a committed entry lost in an election nobody noticed. Catching it in staging requires the same unlucky interleaving that hid it in testing, so "deploy and watch" is not the safety net here that it is for a web service. The simulator searches those interleavings on purpose and is the only thing in this pipeline that can.

Two environments, one chart

	staging	production
Nodes	3	5
Image tag	written by the pipeline	promoted from staging, behind an approval

Same chart deliberately: different manifests would stop staging being a rehearsal, and the bugs it exists to catch are exactly the ones that only appear on a real cluster. Three nodes still exercise election, catch-up and a rolling upgrade holding quorum; what they do not exercise is the two-failure case, which is why production runs five.

Production's approval gates the commit, not the sync — the opposite of the obvious arrangement. A human clicking Sync in ArgoCD means git no longer describes production: the commit is merged, the cluster is not running it, and reverting would not roll back, which removes the single property that makes this worth doing. Production is also promoted rather than rebuilt, because rebuilding from the same source can produce a different image (a moved base layer, a different toolchain patch), so the artifact exercised in staging is the one that ships.

Verified against this branch before merging

Both Applications were pointed at feat/cd-pipeline on a freshly rebuilt cluster:

raftkv-staging     Synced     Healthy   3 pods
raftkv-production  Synced     Healthy   5 pods

staging:     node 3 leader, term 2, 3/3 caught up
production:  node 2 leader, term 1, 5/5 caught up

drift: scaled staging below quorum by hand
  replica count back to 3 after 1s
  PodDisruptionBudget restored after 2s

The tag-rewriting logic that staging and production deployment depend on was tested locally end to end before being trusted in CI.

Three bugs found by running it rather than reading it

• clusterResourceWhitelist: [] with CreateNamespace=true fails every sync with resource :Namespace is not permitted in project raftkv, before a single workload is created. Namespace is now the one cluster-scoped exception, which is still far narrower than the default project.
• The resources-finalizer needs its AppProject to still exist. Handing both to one kubectl delete -f wedges the Application: the finalizer never completes, the delete blocks until timeout, and the workloads it should have removed are orphaned. argocd.sh down now deletes in dependency order and clears a stuck finalizer.
• needs: [staging] on the production job made the promote path unreachable, because staging does not run on a workflow_dispatch and a skipped dependency skips its dependents.

Note on the approval gate

The production GitHub Environment needs required reviewers configured in repository settings. Without them the gate is decorative — the job will run unattended.

Verification

make check passes: vet, determinism guards, -race suite, 22/22 mutations caught.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: c7678d72-67ee-4d1a-b3d8-0480c38fab08

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}