ci: skip nightly benchmarks on an already-benchmarked HEAD - #10276
Conversation
The store action compares each new result against the last entry of a *different* commit, so nightly re-runs on an unchanged HEAD keep failing against a baseline from before the last real change (currently: two nights in a row re-alerting 9322 vs the 10-day-old 5533, despite 9322 being stored on gh-pages both nights). Gate the benchmark job on whether HEAD already has stored results. Also fetch gh-pages in a dedicated step instead of inside the first store step, so a crash of the timing suite no longer drops the results of every other suite (their stores reuse the fetched branch and cannot create it themselves). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDW1JHcCbDRgiGw8QBiDXE
PR SummaryLow Risk Overview A new Inside Reviewed by Cursor Bugbot for commit bd46963. Bugbot is set up for automated code reviews on this repo. Configure here. |
Summary
Two changes to the nightly
benchmarkjob so the benchmark baseline actually advances past a degradation instead of re-alerting against a stale value forever:benchmark-gatejob reads the lastgh-pagesbenchmark-commit message and skips the whole benchmark job when it already records the current HEAD.github-action-benchmarkcompares each new result against the last entry of a different commit, so re-running an unchanged HEAD always skips its own stored entries and re-fails against the pre-regression baseline — night after night, no matter how many times the new value is stored.gh-pagesin a dedicated step instead of implicitly inside the first store step. Previously, if the timing suite crashed, its store step (the only one withskip-fetch-gh-pages: false) was skipped, the localgh-pagesbranch was never created, and every other suite's store + the final push failed too — losing all of that night's results.Type of change
Why is this change needed?
The nightly LS-flow heap benchmark regressed on 7/19 (
growth_per_edit_bytes/ls-body-edit: 5533 → 9322). #10265 correctly made the store/push steps run despite the alert, so the 9322 result is stored ongh-pages(verified: it's there for both 7/29 and 7/30). But since no new commit landed onmainafter 7/28, every nightly re-benchmarks the same HEAD (8e2d714), and the store action's same-commit skip makes each of those runs compare 9322 against the 10-day-old 5533 baseline — producing an identical failure every night (runs 30410676963 and 30502247291) with no way for the alert to ever clear on its own.What was the behavior or documentation before?
gh-pagesbranch.What is the behavior or documentation after?
gh-pagesbranch fetched up front, independent of every other suite's outcome; the coupling comments ("MUST run afterStore timing benchmark results…") are gone because the coupling is gone.Related issue or discussion (if any)
Follow-up to #10265.
Additional context
The gate parses the trailing 40-hex SHA of the last
gh-pagescommit subject (add <suite> benchmark result for <sha>, the format the store action itself writes). If the subject doesn't match that format, the gate falls back to running the benchmarks.🤖 Generated with Claude Code
https://claude.ai/code/session_01KDW1JHcCbDRgiGw8QBiDXE
Generated by Claude Code