Repository navigation
Labels
area:metricsMetrics, analytics, monitoringMetrics, analytics, monitoringarea:replicationReplication, cluster sync, peer connectionsReplication, cluster sync, peer connectionsenhancementNew feature or requestNew feature or request
Milestone
Description
Workstream W8 of #430 · observability & metrics pipeline
Summary
Today there are two half-pipes that don't meet: cluster_status (an ephemeral poll snapshot of health gauges) and hdb_analytics (durable time-series, but only byte counters). Neither gives an operator a divergence alarm — which is why #426 (silent ~39% loss) and #386 (344 lost attachments) went undetected with connected:true. Connect the pipes and add the missing signals.
Tier 1 — extend cluster_status (cheap, high diagnostic value, mostly already filed)
- Last connection error {message, code} per socket (cluster_status: include last connection error message and code #214) — store the error the close/error handlers already see. Biggest "why is it down" win.
- Resume-cursor + applied-vs-head lag per socket — surface
lastDurableSequenceId/persistedseqIdand a computed lag; fixlastReceivedVersionnot populating on the relay/datapath (it's a non-signal exactly when you need it — see rapid-reconnect stress: follower permanently loses replication backlog after restart churn (data loss; fast-skip ruled out) #426). - Removed-node truth + replace the hardcoded
is_enabled:truewith real membership (cluster_status on a removed node should clearly indicate removal and close sockets #217). - Shard count (Expose shard count via cluster status API #204).
- Blob in-flight depth + "cursor held by blob gap" flag (complements the already-shipped
blobReplicationFailures). - Audit/transaction-log backlog depth (pending between cursor and head). (Reframes the abandoned-NATS Add NATS message stream backlog counts to cluster_status #261 onto the live audit log.)
- Make the op cheap — drop the full
getSystemInfo(cluster_status: avoid full getSystemInfo call for performance #260).
Tier 2 — a real metrics pipeline (trends + alerts + divergence)
- Promote replication-health gauges into
hdb_analytics(it already ingests replication bytes — extend to lag, back-pressure %, blob-failure rate, connection state, cursor position). - Prometheus / OpenTelemetry export over that series (none exists today).
- Divergence detection — periodic cross-peer record/blob-count comparison emitting an alertable metric. This is the connective tissue with W2: it converts every "silent divergence" into a loud, actionable signal before data loss.
Tier 3 — log hygiene
- Rate-limit/dedup repeated log lines with run-counts (Rate-limit / deduplicate repeated identical log entries in replication #262) — defuses the replication: no-backoff subscription-setup retry storm on transient boot-time DNS failure ends in OOM #327/Replication: 5.0.31 wedge-reconcile spins at 100% CPU re-subscribing an unreachable peer (forceResubscribe leaks into persistent listeners) #349 storm floods and makes logs alert-parseable.
Retires / advances
- cluster_status: include last connection error message and code #214, cluster_status on a removed node should clearly indicate removal and close sockets #217, Expose shard count via cluster status API #204, cluster_status: avoid full getSystemInfo call for performance #260, Add NATS message stream backlog counts to cluster_status #261 (reframed), Rate-limit / deduplicate repeated identical log entries in replication #262
- Closes the loop W2 opens (divergence alarm)
Dependencies
W1 (registry/last-error surfacing), W2 (gap signal), W4 (per-origin lag). Tier 1 can largely proceed in Phase 1 as quick wins; Tier 2 is the strategic piece.
Effort / risk
M / low.
Acceptance criteria
- An operator can see per-subscription lag, last connection error, and cursor position.
- A diverging cluster raises an alert before data loss.
- Storm-condition logs are bounded and parseable.
🤖 Filed by Claude on behalf of Kris.
Metadata
Metadata
Assignees
Labels
area:metricsMetrics, analytics, monitoringMetrics, analytics, monitoringarea:replicationReplication, cluster sync, peer connectionsReplication, cluster sync, peer connectionsenhancementNew feature or requestNew feature or request
Type
Fields
Priority
P2
Activity
cluster_status.blobReplicationFailuresalways reports 0 for copy-mode blob failures (observability gap) #502kriszyp commentedon Jul 16, 2026
Three fold-ins from Chris Nelson's monitoring feedback (2026-07-15; reconciled in #532):
Also relevant to W8's per-peer labels: the two new replication-observability gap issues #589 (inbound-link awareness / half-open self-report) and #590 (repl cert expiry).
🤖 Filed by KrAIs on behalf of Kris.
cluster_status.blobReplicationFailuresalways reports 0 for copy-mode blob failures (observability gap) #502