Skip to content

apollo_dashboard: alert on committer block number lag and block commit latency - #14890

Open
itamar-starkware wants to merge 1 commit into
mainfrom
committer-latency-alerts
Open

apollo_dashboard: alert on committer block number lag and block commit latency#14890
itamar-starkware wants to merge 1 commit into
mainfrom
committer-latency-alerts

Conversation

@itamar-starkware

Copy link
Copy Markdown
Contributor

No description provided.

@reviewable-StarkWare

Copy link
Copy Markdown

This change is Reviewable

@cursor

cursor Bot commented Jul 28, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Dashboard and alert-definition changes only; no runtime committer or consensus logic is modified.

Overview
Adds committer health visibility and paging on two signals: how far the committer trails consensus, and how long single-block commits take.

The Committer Grafana row gains a per-namespace Committer Lag (blocks) timeseries (consensus_block_number - committer_offset, with pod filters stripped so cross-pod metrics compare correctly) and a Block Commit Latency histogram panel (p50/p95 from committer_block_commit_latency).

Two new block production delay alerts are wired into alert_definitions and exported in dev_grafana_alerts.json:

  • committer_block_number_lag (p2, 5m): fires when lag exceeds 9 blocks, only on non-observer namespaces with no sync-driven consensus progress in the last 5m.
  • committer_block_commit_latency_too_high (p4): fires when the 2m average commit latency exceeds ~9s, intended to catch slowdowns before lag builds.

Reviewed by Cursor Bugbot for commit 1ee2983. Bugbot is set up for automated code reviews on this repo. Configure here.

@itamar-starkware
itamar-starkware marked this pull request as ready for review July 28, 2026 11:29
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch from 2a70bcc to fe2372b Compare July 28, 2026 12:05
Comment thread crates/apollo_dashboard/src/panels/committer.rs
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from ca9c6d1 to 64da3ff Compare July 28, 2026 13:42
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch 2 times, most recently from e598d8c to 57d1f68 Compare July 28, 2026 14:21
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 64da3ff to 501db04 Compare July 28, 2026 14:21
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch from 57d1f68 to cee73da Compare July 29, 2026 10:28
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 501db04 to 40ce976 Compare July 29, 2026 10:28

@yoavGrs yoavGrs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yoavGrs made 1 comment.
Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on dan-starkware and itamar-starkware).


a discussion (no related file):
@dan-starkware Do we want a new alert where the gap between consensus and batcher is > 7?

@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 40ce976 to 64cff66 Compare July 29, 2026 13:11
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch from cee73da to 3999c5f Compare July 29, 2026 13:11
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 64cff66 to 881ce09 Compare July 30, 2026 10:38
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch from 3999c5f to 28b2098 Compare July 30, 2026 10:38
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 881ce09 to 6b3efac Compare July 30, 2026 11:07
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch from 28b2098 to 06b11a2 Compare July 30, 2026 11:07
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 6b3efac to fd44500 Compare August 2, 2026 11:49
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch 2 times, most recently from 5d951af to d636e27 Compare August 2, 2026 13:23

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d636e27. Configure here.

@itamar-starkware
itamar-starkware changed the base branch from committer-block-commit-latency-metric to graphite-base/14890 August 3, 2026 06:52
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch from d636e27 to 1ee2983 Compare August 3, 2026 06:53
@itamar-starkware
itamar-starkware changed the base branch from graphite-base/14890 to main August 3, 2026 06:53

@yoavGrs yoavGrs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yoavGrs made 1 comment.
Reviewable status: 0 of 5 files reviewed, 2 unresolved discussions (waiting on dan-starkware and itamar-starkware).


a discussion (no related file):
Please split this PR, one for each metric.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants