Skip to content

improve output for changed benchmarks - #124

Merged
ev-br merged 6 commits into
benchx-dev:mainfrom
ev-br:not-comparable-benchmarks
Nov 20, 2026
Merged

ev-br merged 6 commits into
benchx-dev:mainfrom
ev-br:not-comparable-benchmarks

Conversation

@ev-br

@ev-br ev-br commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

As discussed #120 (comment): an edited benchmark is reported as not comparable and does not block the whole comparison:

  NOT COMPARABLE (1)
    f   wall-time  (benchmark-changed: its version differs between the sides)

  - --tables: the change table lists the edited function by default with mark x and the verdict not comparable (benchmark-changed), as asv compare does. Before, it had no row.

This PR is a draft since it needs a rebase on top of gh-118.rebased

@ev-br
ev-br marked this pull request as draft November 19, 2026 11:53
ev-br's Claude Code Agent and others added 2 commits November 19, 2026 15:48
A benchmark whose recorded version differs between the sides, or between a
side's rounds, and in nothing else, was edited. It was a failed `sides-agree`
invariant, listed under "failed invariants" and absent from the change table.
Make it a unit that is indeterminate with reason `benchmark-changed`: no pair
is used, the other benchmarks are compared as before, `bx compare` lists it
under NOT COMPARABLE, and the change table shows it with `x`, as asv does.
A version that differs together with another fact is still a failed
invariant.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Describe the `benchmark-changed` reason in comparator.md and
prototype-design.md, and say in benchx.md that an edited pyperf function is
listed as not comparable while the others are compared.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ev-br
ev-br force-pushed the not-comparable-benchmarks branch from f91b5e6 to 9ea9bf3 Compare November 19, 2026 15:59
@ev-br
ev-br marked this pull request as ready for review November 19, 2026 16:00
@ev-br ev-br changed the title WIP: improve output for changed benchmarks improve output for changed benchmarks Nov 19, 2026
@ev-br

ev-br commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

With gh-118 in, this is ready from my side. @MarcoGorelli this is in responce to your comment, #120 (comment)

@ev-br
ev-br requested a review from MarcoGorelli November 19, 2026 16:03
Comment thread docs/prototype.md Resolved
ev-br's Claude Code Agent and others added 2 commits November 20, 2026 10:45
…d by mistake

9ea9bf3 (DOC: an edited benchmark is not comparable...) also added docs/prototype.md, 229 lines that
match the README as it was at ef0ecbb ("benchx prototype"), two lines apart. `main` has never had a file of
that name: its documentation is docs/benchx.md, which this commit's real edits already changed. Nothing links
to the copy; the nav in zensical.toml does not list it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nd on the verdict

CI (macOS) printed NO CHANGE DETECTED for the two values, a hundred times apart in work, so the change table
had no row and `Before [...]` never appeared. The verdict rests on three timed rounds, and one spoiled
round is enough: with ratios of [98, 98, 10] the mean effect, 69, is within k x noise, 87. The test is
about what the tables call the sides, so it now asks for every row (--all) and no longer expects a `+` mark.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MarcoGorelli
MarcoGorelli previously approved these changes Nov 20, 2026

@MarcoGorelli MarcoGorelli left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, just a minor comment

Comment thread src/benchx/compare.py Outdated
"""The unit's reason; for `not-measured`, which side measured nothing, and how its attempts ended:
`not-measured: contender skipped`, or `baseline error 2, skipped 1` when the statuses differ."""
if unit["reason"] == "benchmark-changed":
return "benchmark-changed: its version differs between the sides"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it necessarily between sides, or could it also be between rounds? maybe "its version differs between the results" is more generic?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Changed the message, here and in one other place.

ev-br's Claude Code Agent and others added 2 commits November 20, 2026 12:36
The version can also differ between the rounds of one side, which the
comparator already treats the same way (and tests). The message said "between
the sides" and the comment "its two sides"; both now say "the results".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ev-br
ev-br merged commit 3c966cc into benchx-dev:main Nov 20, 2026
12 checks passed
@ev-br

ev-br commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Merging as approved, thank you @MarcoGorelli !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants