Skip to content

BUG: detect changes to benchmark functions - #120

Merged
ev-br merged 4 commits into
benchx-dev:mainfrom
ev-br:pyperf-function-version
Nov 19, 2026
Merged

ev-br merged 4 commits into
benchx-dev:mainfrom
ev-br:pyperf-function-version

Conversation

@ev-br

@ev-br ev-br commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Just stumbled into a footgun in benchx-dev/benchx_examples#18 : changed the benchmark source of a pure pyperf benchmark, and it happily placed results of the changed benchmark next to the previous results.

Thus, follow asv, which computes the hash of the benchmark function itself. This is not foolproof, but is better than nothing IMO.

For completeness, the last two commits add and remove the plan file.

ev-br's Claude Code Agent and others added 4 commits November 18, 2026 17:19
A benchmark function that lives in the subject's tree and is edited between two
revisions was compared with its old self without a word: its benchmark_tree is the
subject's own, which the schema leaves unchecked. asv guards this with a per-benchmark
`version`, and the asv adapter records it (`protocol.asv.benchmark_version`); pyperf had
nothing of the kind.

A pyperf function now has a version: its `version` attribute when it declares one, else
a SHA-256 of its source, as asv_runner hashes a plain function (trailing whitespace
stripped, dedented; comments and docstrings count; what it calls, its `params` and the
library under test do not). It is found when the module is listed
(`benchx_params.source_version`), carried by `run_case`, and recorded by a new
`case_protocol` in `protocol.pyperf.benchmark_version`. Protocol is what the sides of a
comparison must agree on, so two revisions whose function differs fail `sides-agree`; a
function without readable source records none. No schema change.

The mapping version stays `pyperf-to-benchx/v1`, by decision: results made before lack
the key.

The fixtures of test_compare_harnesses.py edited the function body between their two
commits; they now change a module constant instead, which is how a benchmark that
exercises code elsewhere looks. A benchmark function that is itself the subject needs a
declared `version`.

The design docs gain the version for both adapters: the schema's "not checked" passage
for in-tree benchmarks, the identity-policy paragraph and the pyperf adapter section.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ashing

`bx bench TARGET --adapter asv --suite DIR`, with DIR outside the subject's checkout,
failed with a traceback:

    RuntimeError: the error result is itself invalid: malformed at
    /provenance/benchmark_tree: None is not of type 'string'

A suite outside the checkout is identified by its file's blob id (runner-schema.md
§3.3: the file is the unit, not its directory). An asv suite is a directory, `blob_id`
cannot read it and gives None, and the result failed its own validation.

`session._found_benchmark`, where the order's benchmark identity is computed, now
raises a SessionError that says so and says to put the suite inside the checkout: the
CLI prints it as `bx bench: error: ...` and refuses before anything runs. The check is
in the session, not the adapter, because with a provider the target path is the build
directory and NumPy's asv suite, given by its absolute path, is inside the source
checkout: it is found there and still runs. A pyperf suite outside the tree is one file
and is unaffected.

Supporting a directory outside the checkout would need a decision the design has not made
(what identifies a directory), so it is refused for now.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…efusal

PYPERF_VERSION_PLAN.md: the problem, a check of the plan against the design docs, the
decisions (including the user's: the pyperf mapping version stays v1), a short
discussion of asv suites outside the checkout, the steps and the outcome.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ev-br
ev-br requested review from MaanasArora and MarcoGorelli and removed request for MaanasArora November 18, 2026 17:36
@MarcoGorelli

Copy link
Copy Markdown
Collaborator

thanks @ev-br !

if i edit one benchmark function, then does this mean that the whole benchmark comparison is refused, or just the comparison for that particular edited function?

@ev-br

ev-br commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

if i edit one benchmark function, then does this mean that the whole benchmark comparison is refused, or just the comparison for that particular edited function?

On this branch, it's the whole comparison.
Once gh-118 lands, it'll change into a per-function failure (which is how ASV does it).
The technical reason is that the per-unit sides-agree change (ba72add) is only in that PR, while this PR is off main to keep gh-118 from doing too many things at once.
Do you want me to fold this back into gh-118, or hold this PR until gh-118 lands, or something else?

@MarcoGorelli MarcoGorelli left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's probably to do in this order, thanks @ev-br

@ev-br
ev-br merged commit 61577a8 into benchx-dev:main Nov 19, 2026
16 of 17 checks passed
@ev-br

ev-br commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Merging as approved, thanks @MarcoGorelli ! I'll make a note to add a test for per-function refusals after gh-118.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants