[sAchin-680]

Prometheus scraping every node individually, alert rules for the ways a consensus cluster actually fails, an Alertmanager routing them, and a Grafana dashboard provisioned from a file in this repository.

The alerts were fired on purpose

An alert rule that has never fired is an assertion, not a safeguard — the expression may not match the metric names, the threshold may be unreachable, the routing may silently drop it, and none of that surfaces until an incident. So make monitoring-demo removes quorum deliberately and checks the alert arrives at Alertmanager. Against the live five-node cluster:

17:27:45  cluster has 5 nodes, quorum is 3
17:27:45  scaling down to 2 — one short of a majority
17:28:15  alerts firing: RaftKVBelowQuorum RaftKVNoLeader
17:28:37  healing: scaling back to 5
17:28:51  alerts cleared; the cluster recovered on its own

Both critical alerts fired within 30s of quorum loss and cleared within 10s of recovery, without intervention. All eight rules load; the dashboard is provisioned and its panel queries return data.

There is a detail in that demo worth keeping. Scaling the StatefulSet from five to two does not remove anything from the Raft configuration — membership lives in the replicated log and moves only through joint consensus. The cluster still believed it had five voters, so the two survivors could not assemble a majority and correctly refused to serve. The alert measures a majority of the configuration, not a count of pods.

Why the rules are written against consensus state, not pod health

RaftKVNoLeader is expressed as "no node can name a leader" rather than as a count of running pods, because those are different questions: five pods can be Running and healthy while a partition leaves none of them able to reach a majority. Every pod fine, the cluster down — Kubernetes cannot observe that difference.

It waits 15s, not 0s. Every election has a leaderless instant, so a zero threshold pages on routine failover and trains people to ignore it; 15s is about ten election timeouts, which separates "failing to elect" from "in the middle of electing".

RaftKVBelowQuorum reads the voter count from the cluster rather than hardcoding five, so it survives membership changes. RaftKVNodeDown is a warning rather than a page, because one node down on five is survivable by construction — it matters only when a second follows, which the quorum alert covers. RaftKVScrapeFailing alerts on the monitoring, not the cluster: if the exporter cannot read a node's state the node may be perfectly healthy, and treating a broken exporter as a broken cluster is how a healthy node gets restarted mid-incident. RaftKVStuckInJointConfiguration catches a membership change that never completed, a state strictly more fragile than either configuration alone and one nothing else would notice.

Alertmanager inhibits RaftKVNodeDown under RaftKVBelowQuorum, and groups by alert name rather than by pod — during a rolling upgrade every pod goes down in turn, and per-pod grouping would send five notifications for one expected event.

Scraping and the dashboard

Pod discovery, not the Service. The client Service load-balances, so scraping it would return an arbitrary node per scrape with nothing to distinguish the series. Consensus metrics are only meaningful per node. The scrape interval is 5s because failover takes 1–2s; at a conventional 60s most elections would be invisible.

The dashboard is a reviewable JSON file mounted into Grafana, not a row in Grafana's database that disappears with the pod.

One fix found by looking at the dashboard

The leader was publishing its replication lag to itself. It tracks its own progress legitimately — its own log counts toward the majority that commits an entry — but a node's lag behind itself is zero by definition, and publishing it puts a line on the dashboard that reads as a perfectly healthy follower while not being a follower at all. That is the exact confusion the metric's own comment claims to avoid. Verified after the fix: the leader emits four peer series on a five-node cluster, not five.

Commands

make monitoring-up     # install alongside a running cluster
make monitoring-open   # Grafana :3000, Prometheus :9090, Alertmanager :9093
make monitoring-demo   # break quorum, watch the alert fire, heal, watch it clear
make monitoring-down

Verification

make check passes: vet, determinism guards, -race suite, 22/22 mutations caught. The stack was installed against the live kind cluster and the quorum demo run end to end.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a30774ac-e581-4cb3-9840-1d6616e3dbe0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}