Repository navigation
Open
Labels
enhancementNew feature or requestNew feature or request
Description
I tried this with SciPy, comparing HEAD and main without any performance-relevant changes.
After plumbing everything in with spin and pixi, running:
pixi run bench --compare main HEAD --case linalg.Bench.time_solve
yields output with multiple problems, shown here (full output in the foldout at the end:
differs: benchmark.revision.key
caveats: baseline: dirty-benchmark-tree, baseline: dirty-subject-tree, baseline: setup-per-call-unbatched, contender: dirty-benchmark-tree, contender: dirty-subject-tree, contender: setup-per-call-unbatched
...
NO CHANGE DETECTED (18)
linalg.Bench.time_solve wall-time effect -0.28% noise 1.57% rounds=3
linalg.Bench.time_solve wall-time effect -2.38% noise 3.54% rounds=3
...
INDETERMINATE (6)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
...
excluded (36)
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
Issues:
- Every case is indentified as
linalg.Bench.time_solve, so it's not clear which parametrization is reported on. - The dirty tree determination is too aggressive and too verbose. It should check for changes to files only, not for uncommitted files.
- It's unclear why there are
rounds=0results - probably those were skipped, but how does that differ from the "excluded" results below it? - The
excludedoutput is pure noise, I suggest leaving that out completely.
Details
48 result(s) -> /workspace/benchmarks/results/benchx/bench-20261117T105803-01560d96 (order sha256:2d7d43b68483)
/workspace/benchmarks/results/benchx/store.parquet: ingested 48, duplicate 0
Switched to branch 'benchx'
run bench-20261117T105803-01560d96 profile revisions baseline 976c3b526bdf contender e2f901ecec94 [local-only]
976c3b526bdf5c70720e3f770237e776b2111247: observed env: MKL_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1
e2f901ecec946749d5d13df403e9fdd711649ae7: observed env: MKL_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1
differs: benchmark.revision.key
caveats: baseline: dirty-benchmark-tree, baseline: dirty-subject-tree, baseline: setup-per-call-unbatched, contender: dirty-benchmark-tree, contender: dirty-subject-tree, contender: setup-per-call-unbatched
NO CHANGE DETECTED (18)
linalg.Bench.time_solve wall-time effect -0.28% noise 1.57% rounds=3
linalg.Bench.time_solve wall-time effect -2.38% noise 3.54% rounds=3
linalg.Bench.time_solve wall-time effect -0.31% noise 2.64% rounds=3
linalg.Bench.time_solve wall-time effect +0.85% noise 0.54% rounds=3
linalg.Bench.time_solve wall-time effect +3.44% noise 3.01% rounds=3
linalg.Bench.time_solve wall-time effect -0.40% noise 0.78% rounds=3
linalg.Bench.time_solve wall-time effect -0.51% noise 0.42% rounds=3
linalg.Bench.time_solve wall-time effect -3.07% noise 5.44% rounds=3
linalg.Bench.time_solve wall-time effect +4.54% noise 1.75% rounds=3
linalg.Bench.time_solve wall-time effect -0.38% noise 1.74% rounds=3
linalg.Bench.time_solve wall-time effect -1.99% noise 3.81% rounds=3
linalg.Bench.time_solve wall-time effect +0.59% noise 0.66% rounds=3
linalg.Bench.time_solve wall-time effect +1.82% noise 1.58% rounds=3
linalg.Bench.time_solve wall-time effect +1.53% noise 4.39% rounds=3
linalg.Bench.time_solve wall-time effect -0.41% noise 0.38% rounds=3
linalg.Bench.time_solve wall-time effect -0.13% noise 0.40% rounds=3
linalg.Bench.time_solve wall-time effect -0.39% noise 5.34% rounds=3
linalg.Bench.time_solve wall-time effect -0.76% noise 1.06% rounds=3
INDETERMINATE (6)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
excluded (36)
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
Results: /workspace/benchmarks/results/benchx/bench-20261117T105803-01560d96
Store: /workspace/benchmarks/results/benchx/store.parquet
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
ev-br commentedon Nov 18, 2026
Thanks Ralf!
Could you try #118? It adds the
--tablesoption to emit asv-style tables.Also, could you show how you drive
bx? I seem to be getting different outputs with benchx-dev/benchx_examples#18 (which runs a real numpy benchmark, builds numpy with spin, but the driver is plumbed out-of-tree).In particular,
The too-verbose output is probably best covered by #85, where most of the verbosity goes into a log file with controllable logging levels.
ev-br commentedon Nov 18, 2026
Also,
AFAIU, the main objection is that untracked files make the tree "dirty"? That's on purpose actually, because an untracked
bench_new.py,conftest.pyor.socan change what runs.The duplicate "dirty" warning is because numpy/scipy benchmarks are in-tree, and benchx checks both the subject and benchmarks (which can be out of tree as well).
rgommers commentedon Nov 18, 2026
Thanks @ev-br. With #118:
excludednoise are fixed.rounds=0thing isn't quite fixed. Those benchmarks are intentional skips fornumpyfunctionality; tables show "skipped", but the summary still says "insufficient rounds".That's not all that interesting. This is very poor UX. There are many reasons to have uncommitted files. E.g., a single
AGENTS.mdat the root of your repo that you can't commit will permanently mark everything as dirty. You should just drop this check, and only consider uncommitted changes to existing files.rgommers commentedon Nov 18, 2026
Or just leave that
dirtyinfo out completely - I really don't care about it, it's pure noise for pretty much every real-world local benchmark execution scenario I can think of.ev-br commentedon Nov 19, 2026
Thanks for checking it @rgommers ! Could you please check it once more, with the tip of gh-118? Also add
--tablesto where you plumb it tospinandpixi. In benchx-dev/benchx_examples#18 the addition isIt now emits hopefully clearer diagnostics:
In the end of the day, there's a limit on how clear the messaging can be without changing the benchmark itself: the original SciPy function emits a
NotImplementedErrorin thesetup, and the benchmark runner cannot tell an intentional skip from a failure.There's a clear need to control the level of verbosity of the output: here some output comes from
bx benchand some more frombx compare --run. This should again come together with logging, where logs get everything and the console output reacts to arguments like-q / -v. I'm working on this in a separate branch.ev-br commentedon Nov 19, 2026
Re what counts as "dirty":
I suppose in your example there's a reason you don't
.gitignorethe AGENTS.md.That said,
.soshadowing.pymodule, these are all real scenarios which must not pass silently.rgommers commentedon Nov 19, 2026
Done, and it looks better now, thanks. If you want to keep output verbosity and dirty tree handling separate, then it looks to me like that PR is ready.
There's also some measurement accuracy thing I noticed in the same tests, but I'll open a separate issue for that.
It's just an example, it could just as easily be .md plan files, random notes, or whatever - it's just quite common to have local files in-tree. It would not make sense to have a general .gitignore for
.md,.txt, etc. nor to open SciPy PRs to extend.gitignoreeach time one has such files.Even if it is - which is arguable and should be rare - it's a thing you may learn the hard way once, and then you know or at least quickly realize. While output pollution is continuous, and quite in-your-face.
A log message seems fine to me.
ev-br commentedon Nov 19, 2026
thanks Ralf!
Yes, that PR is large enough already to keep these two to a follow-up PR. gh-118 has thus landed, and I'll look into logging (which keeps smoking unrelated issues , the last of which is #125) and better defining what "dirty" is.
Will keep the issue open while that's in flight.