Repository navigation
BUG: detect changes to benchmark functions - #120
Conversation
A benchmark function that lives in the subject's tree and is edited between two revisions was compared with its old self without a word: its benchmark_tree is the subject's own, which the schema leaves unchecked. asv guards this with a per-benchmark `version`, and the asv adapter records it (`protocol.asv.benchmark_version`); pyperf had nothing of the kind. A pyperf function now has a version: its `version` attribute when it declares one, else a SHA-256 of its source, as asv_runner hashes a plain function (trailing whitespace stripped, dedented; comments and docstrings count; what it calls, its `params` and the library under test do not). It is found when the module is listed (`benchx_params.source_version`), carried by `run_case`, and recorded by a new `case_protocol` in `protocol.pyperf.benchmark_version`. Protocol is what the sides of a comparison must agree on, so two revisions whose function differs fail `sides-agree`; a function without readable source records none. No schema change. The mapping version stays `pyperf-to-benchx/v1`, by decision: results made before lack the key. The fixtures of test_compare_harnesses.py edited the function body between their two commits; they now change a module constant instead, which is how a benchmark that exercises code elsewhere looks. A benchmark function that is itself the subject needs a declared `version`. The design docs gain the version for both adapters: the schema's "not checked" passage for in-tree benchmarks, the identity-policy paragraph and the pyperf adapter section. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ashing
`bx bench TARGET --adapter asv --suite DIR`, with DIR outside the subject's checkout,
failed with a traceback:
RuntimeError: the error result is itself invalid: malformed at
/provenance/benchmark_tree: None is not of type 'string'
A suite outside the checkout is identified by its file's blob id (runner-schema.md
§3.3: the file is the unit, not its directory). An asv suite is a directory, `blob_id`
cannot read it and gives None, and the result failed its own validation.
`session._found_benchmark`, where the order's benchmark identity is computed, now
raises a SessionError that says so and says to put the suite inside the checkout: the
CLI prints it as `bx bench: error: ...` and refuses before anything runs. The check is
in the session, not the adapter, because with a provider the target path is the build
directory and NumPy's asv suite, given by its absolute path, is inside the source
checkout: it is found there and still runs. A pyperf suite outside the tree is one file
and is unaffected.
Supporting a directory outside the checkout would need a decision the design has not made
(what identifies a directory), so it is refused for now.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…efusal PYPERF_VERSION_PLAN.md: the problem, a check of the plan against the design docs, the decisions (including the user's: the pyperf mapping version stays v1), a short discussion of asv suites outside the checkout, the steps and the outcome. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… suite refusal" This reverts commit 5b9c5f5.
|
thanks @ev-br ! if i edit one benchmark function, then does this mean that the whole benchmark comparison is refused, or just the comparison for that particular edited function? |
On this branch, it's the whole comparison. |
MarcoGorelli
left a comment
There was a problem hiding this comment.
it's probably to do in this order, thanks @ev-br
|
Merging as approved, thanks @MarcoGorelli ! I'll make a note to add a test for per-function refusals after gh-118. |
Just stumbled into a footgun in benchx-dev/benchx_examples#18 : changed the benchmark source of a pure
pyperfbenchmark, and it happily placed results of the changed benchmark next to the previous results.Thus, follow
asv, which computes the hash of the benchmark function itself. This is not foolproof, but is better than nothing IMO.For completeness, the last two commits add and remove the plan file.