Skip to content

ci: skip nightly benchmarks on an already-benchmarked HEAD - #10276

Closed
orizi wants to merge 1 commit into
mainfrom
claude/blissful-gates-9ju9w9
Closed

ci: skip nightly benchmarks on an already-benchmarked HEAD#10276
orizi wants to merge 1 commit into
mainfrom
claude/blissful-gates-9ju9w9

Conversation

@orizi

@orizi orizi commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two changes to the nightly benchmark job so the benchmark baseline actually advances past a degradation instead of re-alerting against a stale value forever:

  1. Gate the job on whether HEAD already has stored results. A new lightweight benchmark-gate job reads the last gh-pages benchmark-commit message and skips the whole benchmark job when it already records the current HEAD. github-action-benchmark compares each new result against the last entry of a different commit, so re-running an unchanged HEAD always skips its own stored entries and re-fails against the pre-regression baseline — night after night, no matter how many times the new value is stored.
  2. Fetch gh-pages in a dedicated step instead of implicitly inside the first store step. Previously, if the timing suite crashed, its store step (the only one with skip-fetch-gh-pages: false) was skipped, the local gh-pages branch was never created, and every other suite's store + the final push failed too — losing all of that night's results.

Type of change

  • Bug fix (fixes incorrect behavior)

Why is this change needed?

The nightly LS-flow heap benchmark regressed on 7/19 (growth_per_edit_bytes/ls-body-edit: 5533 → 9322). #10265 correctly made the store/push steps run despite the alert, so the 9322 result is stored on gh-pages (verified: it's there for both 7/29 and 7/30). But since no new commit landed on main after 7/28, every nightly re-benchmarks the same HEAD (8e2d714), and the store action's same-commit skip makes each of those runs compare 9322 against the 10-day-old 5533 baseline — producing an identical failure every night (runs 30410676963 and 30502247291) with no way for the alert to ever clear on its own.


What was the behavior or documentation before?

  • Nightly re-runs on an unchanged HEAD appended duplicate data points and re-fired the same stale degradation alert indefinitely.
  • A crash in the timing suite silently dropped that night's results for all suites (heap, LS-flow, ls_reexec), because their store steps depended on the timing store step having fetched the gh-pages branch.

What is the behavior or documentation after?

  • If the current HEAD already has stored benchmark results, the benchmark job is skipped entirely (also saving ~1h of runner time on idle nights). A degradation therefore alerts exactly once per offending commit; the next run with a genuinely new HEAD compares against the newest stored value.
  • Each store step reuses a gh-pages branch fetched up front, independent of every other suite's outcome; the coupling comments ("MUST run after Store timing benchmark results…") are gone because the coupling is gone.

Related issue or discussion (if any)

Follow-up to #10265.


Additional context

The gate parses the trailing 40-hex SHA of the last gh-pages commit subject (add <suite> benchmark result for <sha>, the format the store action itself writes). If the subject doesn't match that format, the gate falls back to running the benchmarks.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KDW1JHcCbDRgiGw8QBiDXE


Generated by Claude Code

The store action compares each new result against the last entry of a
*different* commit, so nightly re-runs on an unchanged HEAD keep failing
against a baseline from before the last real change (currently: two
nights in a row re-alerting 9322 vs the 10-day-old 5533, despite 9322
being stored on gh-pages both nights). Gate the benchmark job on whether
HEAD already has stored results.

Also fetch gh-pages in a dedicated step instead of inside the first
store step, so a crash of the timing suite no longer drops the results
of every other suite (their stores reuse the fetched branch and cannot
create it themselves).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDW1JHcCbDRgiGw8QBiDXE
@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
CI workflow-only changes to benchmark scheduling and gh-pages fetch order; no compiler or runtime behavior.

Overview
Fixes nightly benchmark noise and lost results by gating runs and decoupling gh-pages fetches.

A new benchmark-gate job reads the latest gh-pages commit subject, extracts the trailing 40-char SHA, and sets already-benchmarked. The main benchmark job runs only when that SHA differs from GITHUB_SHA, so unchanged main no longer re-runs ~1h of benches, appends duplicate points, or re-fails degradation alerts against a stale pre-regression baseline.

Inside benchmark, git fetch origin gh-pages:gh-pages runs once up front; every github-action-benchmark store step uses skip-fetch-gh-pages: true and no longer depends on the timing store having fetched the branch. If the timing suite crashes, heap/LS-flow/ls_reexec stores and the final push can still proceed.

Reviewed by Cursor Bugbot for commit bd46963. Bugbot is set up for automated code reviews on this repo. Configure here.

@reviewable-StarkWare

Copy link
Copy Markdown

This change is Reviewable

@orizi orizi closed this Aug 4, 2026
@orizi
orizi deleted the claude/blissful-gates-9ju9w9 branch August 4, 2026 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants