Skip to content

apollo_dashboard: alert on committer block number lag - #14936

Closed
itamar-starkware wants to merge 2 commits into
committer-latency-alertsfrom
committer-block-number-lag-alert
Closed

apollo_dashboard: alert on committer block number lag#14936
itamar-starkware wants to merge 2 commits into
committer-latency-alertsfrom
committer-block-number-lag-alert

Conversation

@itamar-starkware

Copy link
Copy Markdown
Contributor

No description provided.

@cursor

cursor Bot commented Aug 10, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Dashboard and alert configuration only; no changes to committer or consensus runtime behavior.

Overview
Adds committer block lag observability: a Grafana timeseries on the Committer row showing per-namespace consensus_block_number - committer_offset, with the pod label removed from the query so consensus and committer metrics can be compared.

Introduces a P2 alert committer_block_number_lag that fires when lag is > 9 blocks for 5 minutes, scoped per namespace and filtered to non-observer nodes with no sync-driven consensus decisions in the last 5m. The alert is registered in dashboard definitions and mirrored in dev_grafana.json / dev_grafana_alerts.json.

Reviewed by Cursor Bugbot for commit cc98834. Bugbot is set up for automated code reviews on this repo. Configure here.

@reviewable-StarkWare

Copy link
Copy Markdown

This change is Reviewable

itamar-starkware commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@itamar-starkware
itamar-starkware force-pushed the committer-block-number-lag-alert branch from eead3b1 to cc98834 Compare August 10, 2026 05:56
@itamar-starkware
itamar-starkware force-pushed the committer-latency-alerts branch from 29e359e to 835b2c6 Compare August 10, 2026 05:56

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit cc98834. Configure here.

}

/// 9 fires at the STORED_BLOCK_HASH_BUFFER gate (10): integer gauges + strict `>`.
const COMMITTER_LAG_ALERT_THRESHOLD_BLOCKS: f64 = 9.0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Off-by-one lag alert threshold

Medium Severity

COMMITTER_LAG_ALERT_THRESHOLD_BLOCKS is 9.0 with strict >, so the alert only fires at lag >= 10. The state-commitment gate uses (height + 1) - STORED_BLOCK_HASH_BUFFER, i.e. it already fails when lag reaches 9 (committer_offset <= height - 9). The comment assumes the gate is at 10, so paging starts one block after proposal building is already blocked.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit cc98834. Configure here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true? Are we blocked on 9?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itamar-starkware itamar-starkware left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itamar-starkware made 1 comment.
Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on yoavGrs).

}

/// 9 fires at the STORED_BLOCK_HASH_BUFFER gate (10): integer gauges + strict `>`.
const COMMITTER_LAG_ALERT_THRESHOLD_BLOCKS: f64 = 9.0;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true? Are we blocked on 9?

@itamar-starkware itamar-starkware left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itamar-starkware made 1 comment.
Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on yoavGrs).

}

/// 9 fires at the STORED_BLOCK_HASH_BUFFER gate (10): integer gauges + strict `>`.
const COMMITTER_LAG_ALERT_THRESHOLD_BLOCKS: f64 = 9.0;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants