apollo_committer,apollo_committer_config: warn when block commit exceeds duration threshold

[itamar-starkware]

No description provided.

[cursor]

PR Summary

Low Risk
Observability and config only; commit logic and metrics are unchanged aside from log level selection.

Overview
Adds commit_duration_warn_threshold_millis to committer config (default 3000 ms), wired through deployment JSON and the config schema.

After each block commit, revert, or OS witness commit path, block timing stats are still emitted the same way, but total commit duration above the threshold is logged at warn! instead of debug!, with a note that slow commits can stall consensus. Under the threshold, behavior stays at debug!.

Tests use tracing-test to assert the warn path when the threshold is zero and no warn when it is very high.

^{Reviewed by Cursor Bugbot for commit b495b93. Bugbot is set up for automated code reviews on this repo. Configure here.}

[reviewable-StarkWare]

This change is 

[itamar-starkware]

• apollo_batcher: debug-log compressed byte size of committer-returned commitment infos #14891
• apollo_dashboard: alert on committer block number lag and block commit latency #14890
• apollo_committer: add per-block commit latency histogram metric #14889
• apollo_committer,apollo_committer_config: warn when block commit exceeds duration threshold #14888 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Artifacts upload workflows:

• Started at 2026-07-28T10:46:08Z
• Started at 2026-07-28T11:22:40Z
• Started at 2026-07-28T13:42:27Z
• Started at 2026-07-28T14:21:45Z
• Started at 2026-07-29T10:28:47Z
• Started at 2026-07-29T13:11:40Z
• Started at 2026-07-30T10:38:13Z
• Started at 2026-07-30T11:07:39Z

[itamar-starkware]

Stack overview — committer slowdown observability (PR 1 of 4)

Observability guardrail for the committer under the os_input rollout: makes a slow committer loud, measurable, and diagnosable — before it lags the N‑10 retrospective‑hash gate and risks a consensus freeze.

PR	Crates	Adds
#14888 (this)	apollo_committer, apollo_committer_config	Per‑block stats line escalates from debug! to warn! (same format) when commit exceeds block_commit_duration_warn_threshold_millis (default 3000 ms, configurable)
#14889	apollo_committer	Per‑block commit‑latency histogram metric (drives the latency alert + dashboard percentiles)
#14890	apollo_dashboard	2 alerts + 2 panels ("Block Commit Latency", "Committer Lag")
#14891	apollo_batcher	debug! of the compressed commitment‑infos (witness) byte size on the batcher receive‑side

Metric added

Metric	Type	Unit	Recorded
committer_block_commit_latency	histogram	seconds	Per block, commits and reverts

Alerts added

Alert	Fires when	Threshold	Severity	Pending
committer_block_commit_latency_too_high	5‑min avg commit latency exceeds	> 4.0 s (½ the 8.1 s round‑0 build deadline)	p4 (WorkingHours)	30 s
committer_block_number_lag	per‑namespace consensus_block_number − committer_offset exceeds, for live, non‑observer, non‑syncing nodes only	> 7 blocks (3 of headroom before the 10‑block gate)	p2 (Regular)	5 m

The lag alert computes lag per node (each committer vs its own consensus height) and gates out observer nodes (is_observer == 0) and nodes catching up via sync (increase(consensus_decisions_reached_by_sync[5m]) == 0), so it fires only when a live validator's committer is the actual bottleneck. Lag reuses existing gauges — only the latency alert depends on the new histogram.

Panels added (dashboard, committer row)

• Committer Lag (blocks) — consensus_block_number − committer_offset per namespace.
• Block Commit Latency — p50/p95 from the new histogram.

Also in the stack (logs)

• warn!/debug! per‑block stats (PR apollo_committer,apollo_committer_config: warn when block commit exceeds duration threshold #14888) — see threshold above.
• debug! compressed witness byte size, batcher‑side (PR apollo_batcher: debug-log compressed byte size of committer-returned commitment infos #14891).

[yoavGrs]

@yoavGrs reviewed 8 files and all commit messages, and made 6 comments.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on itamar-starkware).

---

crates/apollo_committer/src/committer.rs line 797 at r1 (raw file):

    // A slow committer eventually stalls consensus (proposer waits on the N-10 hash).
    let warn_threshold_millis = block_commit_duration_warn_threshold.as_secs_f64() * 1000.0;
    if durations.block * 1000.0 > warn_threshold_millis {

Works?

Suggestion:

    if Duration::from_secs_f64(durations.block) > block_commit_duration_warn_threshold {

---

crates/apollo_committer/src/committer.rs line 798 at r1 (raw file):

    let warn_threshold_millis = block_commit_duration_warn_threshold.as_secs_f64() * 1000.0;
    if durations.block * 1000.0 > warn_threshold_millis {
        warn!("{stats}, above the {warn_threshold_millis:.0} ms warn threshold");

It's not clear what metric is above the threshold.

Code quote:

warn!("{stats}, above the

---

crates/apollo_committer_config/src/config.rs line 39 at r1 (raw file):

        deserialize_with = "deserialize_milliseconds_to_duration",
        serialize_with = "serialize_duration_as_milliseconds"
    )]

Do we support such backward compatibility?

Code quote:

    // Deployed nodes load config with schema defaults ignored, so absent params must still
    // deserialize; `default` keeps configs that predate this field loadable.
    #[serde(
        default = "default_block_commit_duration_warn_threshold",
        deserialize_with = "deserialize_milliseconds_to_duration",
        serialize_with = "serialize_duration_as_milliseconds"
    )]

---

crates/apollo_committer_config/src/config.rs line 40 at r1 (raw file):

        serialize_with = "serialize_duration_as_milliseconds"
    )]
    pub block_commit_duration_warn_threshold_millis: Duration,

Can you shorten the field name?

Code quote:

block_commit_duration_warn_threshold_millis

---

crates/apollo_committer_config/src/config.rs line 45 at r1 (raw file):

fn default_block_commit_duration_warn_threshold() -> Duration {
    Duration::from_millis(3000)
}

Define a constant instead of a function.

Code quote:

fn default_block_commit_duration_warn_threshold() -> Duration {
    Duration::from_millis(3000)
}

---

crates/apollo_committer_config/src/config_test.rs line 3 at r1 (raw file):

use super::{default_block_commit_duration_warn_threshold, ApolloCommitterConfig};

/// A deployed node loads config with schema defaults ignored, so a preset that predates

Do we support such backward compatibility?

[itamar-starkware]

crates/apollo_committer/src/committer.rs line 797 at r1 (raw file):

Previously, yoavGrs wrote…

Works?

Yes, done

[itamar-starkware]

crates/apollo_committer/src/committer.rs line 798 at r1 (raw file):

Previously, yoavGrs wrote…

It's not clear what metric is above the threshold.

Added more words here.

[itamar-starkware]

crates/apollo_committer_config/src/config.rs line 39 at r1 (raw file):

Previously, yoavGrs wrote…

Do we support such backward compatibility?

Yes, it's the point here. We want to be able to support missing config.

[itamar-starkware]

crates/apollo_committer_config/src/config.rs line 40 at r1 (raw file):

Previously, yoavGrs wrote…

Can you shorten the field name?

Done

[itamar-starkware]

crates/apollo_committer_config/src/config.rs line 45 at r1 (raw file):

Previously, yoavGrs wrote…

Define a constant instead of a function.

Function as input is a constraint of serde default. Anyway I added a const inside.

[itamar-starkware]

crates/apollo_committer_config/src/config_test.rs line 3 at r1 (raw file):

Previously, yoavGrs wrote…

Do we support such backward compatibility?

Yes, it's the point here. We want to be able to support missing config.

[yoavGrs]

@yoavGrs reviewed 4 files and all commit messages, made 3 comments, and resolved 3 discussions.
Reviewable status: 7 of 8 files reviewed, 4 unresolved discussions (waiting on itamar-starkware).

---

crates/apollo_committer_config/src/config.rs line 39 at r1 (raw file):

Previously, itamar-starkware wrote…

Yes, it's the point here. We want to be able to support missing config.

The updated node will start with the updated config. When may the field be missed?

---

crates/apollo_committer/src/committer.rs line 800 at r2 (raw file):

        warn!("{stats}, block commit duration above the {threshold_millis} ms warn threshold");
    } else {
        debug!("{stats}, block commit duration within the {threshold_millis} ms warn threshold");

Consider removing it.

Code quote:

, block commit duration within the {threshold_millis} ms warn threshold

---

crates/apollo_committer_config/src/config.rs line 42 at r2 (raw file):

}

pub const DEFAULT_COMMIT_DURATION_WARN_THRESHOLD: Duration = Duration::from_millis(3000);

consts at the top of the file.

[itamar-starkware]

@itamar-starkware made 3 comments.
Reviewable status: 3 of 10 files reviewed, 4 unresolved discussions (waiting on yoavGrs).

---

crates/apollo_committer/src/committer.rs line 800 at r2 (raw file):

Previously, yoavGrs wrote…

Consider removing it.

Done.

---

crates/apollo_committer_config/src/config.rs line 39 at r1 (raw file):

Previously, yoavGrs wrote…

The updated node will start with the updated config. When may the field be missed?

It was a fallback. But I compared the behavior of other configs and I understand we don't support such fallbacks. We want node to panic if the configmap has missing field.
I changed some of the code in this PR accordingly.

---

crates/apollo_committer_config/src/config.rs line 42 at r2 (raw file):

Previously, yoavGrs wrote…

consts at the top of the file.

Done.

[yoavGrs]

@yoavGrs reviewed 7 files and all commit messages, made 1 comment, and resolved 4 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on itamar-starkware).

[gkaempfer]

Security scan complete — no issues detected.

---

Generated by Claude Code