Repository navigation
improve output for changed benchmarks - #124
Merged
Merged
Conversation
ev-br
marked this pull request as draft
November 19, 2026 11:53
A benchmark whose recorded version differs between the sides, or between a side's rounds, and in nothing else, was edited. It was a failed `sides-agree` invariant, listed under "failed invariants" and absent from the change table. Make it a unit that is indeterminate with reason `benchmark-changed`: no pair is used, the other benchmarks are compared as before, `bx compare` lists it under NOT COMPARABLE, and the change table shows it with `x`, as asv does. A version that differs together with another fact is still a failed invariant. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Describe the `benchmark-changed` reason in comparator.md and prototype-design.md, and say in benchx.md that an edited pyperf function is listed as not comparable while the others are compared. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ev-br
force-pushed
the
not-comparable-benchmarks
branch
from
November 19, 2026 15:59
f91b5e6 to
9ea9bf3
Compare
ev-br
marked this pull request as ready for review
November 19, 2026 16:00
Collaborator
Author
|
With gh-118 in, this is ready from my side. @MarcoGorelli this is in responce to your comment, #120 (comment) |
…d by mistake 9ea9bf3 (DOC: an edited benchmark is not comparable...) also added docs/prototype.md, 229 lines that match the README as it was at ef0ecbb ("benchx prototype"), two lines apart. `main` has never had a file of that name: its documentation is docs/benchx.md, which this commit's real edits already changed. Nothing links to the copy; the nav in zensical.toml does not list it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nd on the verdict CI (macOS) printed NO CHANGE DETECTED for the two values, a hundred times apart in work, so the change table had no row and `Before [...]` never appeared. The verdict rests on three timed rounds, and one spoiled round is enough: with ratios of [98, 98, 10] the mean effect, 69, is within k x noise, 87. The test is about what the tables call the sides, so it now asks for every row (--all) and no longer expects a `+` mark. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MarcoGorelli
previously approved these changes
Nov 20, 2026
MarcoGorelli
left a comment
Collaborator
There was a problem hiding this comment.
looks good, just a minor comment
| """The unit's reason; for `not-measured`, which side measured nothing, and how its attempts ended: | ||
| `not-measured: contender skipped`, or `baseline error 2, skipped 1` when the statuses differ.""" | ||
| if unit["reason"] == "benchmark-changed": | ||
| return "benchmark-changed: its version differs between the sides" |
Collaborator
There was a problem hiding this comment.
is it necessarily between sides, or could it also be between rounds? maybe "its version differs between the results" is more generic?
Collaborator
Author
There was a problem hiding this comment.
Agreed. Changed the message, here and in one other place.
The version can also differ between the rounds of one side, which the comparator already treats the same way (and tests). The message said "between the sides" and the comment "its two sides"; both now say "the results". Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Merging as approved, thank you @MarcoGorelli ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As discussed #120 (comment): an edited benchmark is reported as
not comparableand does not block the whole comparison:This PR is a draft since it needs a rebase on top of gh-118.rebased