Skip to content

apollo_committer: add per-block commit latency histogram metric - #14889

Open
itamar-starkware wants to merge 1 commit into
committer-long-block-alertfrom
committer-block-commit-latency-metric
Open

apollo_committer: add per-block commit latency histogram metric#14889
itamar-starkware wants to merge 1 commit into
committer-long-block-alertfrom
committer-block-commit-latency-metric

Conversation

@itamar-starkware

Copy link
Copy Markdown
Contributor

No description provided.

@reviewable-StarkWare

Copy link
Copy Markdown

This change is Reviewable

Copy link
Copy Markdown
Contributor Author

@cursor

cursor Bot commented Jul 28, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Observability-only change; no commit logic, storage, or API behavior is modified.

Overview
Adds a Prometheus histogram committer_block_commit_latency so operators can see per-block end-to-end commit latency (seconds), not only cumulative millisecond counters.

Each successful block path that already calls update_metrics now also records durations.block into that histogram. The metric description notes this applies to commits and reverts alike.

Registration happens in register_metrics alongside existing committer gauges/counters. A new integration test wires a local Prometheus recorder, runs commit_block, and asserts the histogram has count 1 and a positive sum.

Reviewed by Cursor Bugbot for commit 6b3efac. Bugbot is set up for automated code reviews on this repo. Configure here.

@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 8e0f49c to ca9c6d1 Compare July 28, 2026 11:22
@itamar-starkware
itamar-starkware force-pushed the committer-long-block-alert branch from c9a778b to 7213a58 Compare July 28, 2026 11:22
@itamar-starkware
itamar-starkware marked this pull request as ready for review July 28, 2026 11:27
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from ca9c6d1 to 64da3ff Compare July 28, 2026 13:42
@itamar-starkware
itamar-starkware force-pushed the committer-long-block-alert branch from 7213a58 to 298adf8 Compare July 28, 2026 13:42
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 64da3ff to 501db04 Compare July 28, 2026 14:21
@itamar-starkware
itamar-starkware force-pushed the committer-long-block-alert branch 2 times, most recently from 4dc351e to 8fc73a6 Compare July 29, 2026 10:28
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 501db04 to 40ce976 Compare July 29, 2026 10:28

@yoavGrs yoavGrs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yoavGrs made 1 comment.
Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on itamar-starkware).


crates/apollo_committer/src/committer.rs line 683 at r1 (raw file):

) {
    BLOCKS_COMMITTED.increment(1);
    TOTAL_BLOCK_DURATION.increment((durations.block * 1000.0) as u64);

I see we already have the TOTAL_BLOCK_DURATION metric.

@itamar-starkware
itamar-starkware force-pushed the committer-long-block-alert branch from 8fc73a6 to 9df2eae Compare July 29, 2026 13:11
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch 2 times, most recently from 64cff66 to 881ce09 Compare July 30, 2026 10:38
@itamar-starkware
itamar-starkware force-pushed the committer-long-block-alert branch from 9df2eae to 40afe90 Compare July 30, 2026 10:38
@itamar-starkware

Copy link
Copy Markdown
Contributor Author

crates/apollo_committer/src/committer.rs line 683 at r1 (raw file):

Previously, yoavGrs wrote…

I see we already have the TOTAL_BLOCK_DURATION metric.

You are right. The new metric gives you full distubition p50/p95/p99 and not only a mean over a window. I think it gives extra info, I also want to identify a peak easily.
I can remove the old metric if you want.

@itamar-starkware
itamar-starkware force-pushed the committer-long-block-alert branch from 40afe90 to b495b93 Compare July 30, 2026 11:07
@itamar-starkware
itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 881ce09 to 6b3efac Compare July 30, 2026 11:07

@yoavGrs yoavGrs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yoavGrs reviewed 5 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on itamar-starkware).


crates/apollo_committer/src/committer.rs line 683 at r1 (raw file):

Previously, itamar-starkware wrote…

You are right. The new metric gives you full distubition p50/p95/p99 and not only a mean over a window. I think it gives extra info, I also want to identify a peak easily.
I can remove the old metric if you want.

If you think it's redundant, please remove it in a following PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants