Skip to content

--compare output needs improvement #117

Open

Description

@rgommers

I tried this with SciPy, comparing HEAD and main without any performance-relevant changes.

After plumbing everything in with spin and pixi, running:

pixi run bench --compare main HEAD --case linalg.Bench.time_solve

yields output with multiple problems, shown here (full output in the foldout at the end:

  differs: benchmark.revision.key
  caveats: baseline: dirty-benchmark-tree, baseline: dirty-subject-tree, baseline: setup-per-call-unbatched, contender: dirty-benchmark-tree, contender: dirty-subject-tree, contender: setup-per-call-unbatched
...
NO CHANGE DETECTED (18)
  linalg.Bench.time_solve          wall-time  effect   -0.28%  noise   1.57%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -2.38%  noise   3.54%  rounds=3
...

INDETERMINATE (6)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
...

excluded (36)
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}

Issues:

  1. Every case is indentified as linalg.Bench.time_solve, so it's not clear which parametrization is reported on.
  2. The dirty tree determination is too aggressive and too verbose. It should check for changes to files only, not for uncommitted files.
  3. It's unclear why there are rounds=0 results - probably those were skipped, but how does that differ from the "excluded" results below it?
  4. The excluded output is pure noise, I suggest leaving that out completely.
Details
48 result(s) -> /workspace/benchmarks/results/benchx/bench-20261117T105803-01560d96  (order sha256:2d7d43b68483)
/workspace/benchmarks/results/benchx/store.parquet: ingested 48, duplicate 0
Switched to branch 'benchx'
run bench-20261117T105803-01560d96  profile revisions  baseline 976c3b526bdf  contender e2f901ecec94  [local-only]
  976c3b526bdf5c70720e3f770237e776b2111247: observed env: MKL_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1
  e2f901ecec946749d5d13df403e9fdd711649ae7: observed env: MKL_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1
  differs: benchmark.revision.key
  caveats: baseline: dirty-benchmark-tree, baseline: dirty-subject-tree, baseline: setup-per-call-unbatched, contender: dirty-benchmark-tree, contender: dirty-subject-tree, contender: setup-per-call-unbatched

NO CHANGE DETECTED (18)
  linalg.Bench.time_solve          wall-time  effect   -0.28%  noise   1.57%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -2.38%  noise   3.54%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.31%  noise   2.64%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +0.85%  noise   0.54%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +3.44%  noise   3.01%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.40%  noise   0.78%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.51%  noise   0.42%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -3.07%  noise   5.44%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +4.54%  noise   1.75%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.38%  noise   1.74%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -1.99%  noise   3.81%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +0.59%  noise   0.66%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +1.82%  noise   1.58%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +1.53%  noise   4.39%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.41%  noise   0.38%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.13%  noise   0.40%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.39%  noise   5.34%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.76%  noise   1.06%  rounds=3

INDETERMINATE (6)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)

excluded (36)
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
Results: /workspace/benchmarks/results/benchx/bench-20261117T105803-01560d96
Store: /workspace/benchmarks/results/benchx/store.parquet

Activity

  1. ev-br commented on Nov 18, 2026

    @ev-br
    Collaborator

    Thanks Ralf!

    Could you try #118? It adds the --tables option to emit asv-style tables.

    Also, could you show how you drive bx? I seem to be getting different outputs with benchx-dev/benchx_examples#18 (which runs a real numpy benchmark, builds numpy with spin, but the driver is plumbed out-of-tree).

    In particular,

    • for your point 1, I do get the parameter values,
    12 result(s) -> /home/ev-br/proj/quansight/benchx_user/examples/numpy-revisions/out/numpy-revisions  (order sha256:9accb515aad0)
    /home/ev-br/proj/quansight/benchx_user/examples/numpy-revisions/out/store.parquet: ingested 12, duplicate 0
    run numpy-revisions  profile revisions  baseline fdcebd8854a7  contender f72cb64e8b4f
      differs: benchmark.revision.key
      caveats: baseline: setup-per-call-unbatched, contender: setup-per-call-unbatched
    
    NO CHANGE DETECTED (6)
      bench_function_base.Mean.time_mean [size=1] wall-time  effect   +2.46%  noise   1.00%  rounds=5
      bench_function_base.Mean.time_mean [size=10] wall-time  effect   +0.14%  noise   0.44%  rounds=5
      bench_function_base.Mean.time_mean [size=100000] wall-time  effect   -0.32%  noise   0.78%  rounds=5
      bench_function_base.Mean.time_mean_axis [size=1] wall-time  effect   +0.14%  noise   0.89%  rounds=5
      bench_function_base.Mean.time_mean_axis [size=10] wall-time  effect   -0.43%  noise   0.56%  rounds=5
      bench_function_base.Mean.time_mean_axis [size=100000] wall-time  effect   -0.06%  noise   1.37%  rounds=5
    

    The too-verbose output is probably best covered by #85, where most of the verbosity goes into a log file with controllable logging levels.

  2. ev-br commented on Nov 18, 2026

    @ev-br
    Collaborator

    Also,

    1. The dirty tree determination is too aggressive and too verbose. It should check for changes to files only, not for uncommitted files.

    AFAIU, the main objection is that untracked files make the tree "dirty"? That's on purpose actually, because an untracked bench_new.py, conftest.py or .so can change what runs.

    The duplicate "dirty" warning is because numpy/scipy benchmarks are in-tree, and benchx checks both the subject and benchmarks (which can be out of tree as well).

  3. rgommers commented on Nov 18, 2026

    @rgommers
    CollaboratorAuthor

    Thanks @ev-br. With #118:

    • the missing parameters and the excluded noise are fixed.
    • The rounds=0 thing isn't quite fixed. Those benchmarks are intentional skips for numpy functionality; tables show "skipped", but the summary still says "insufficient rounds".

    AFAIU, the main objection is that untracked files make the tree "dirty"? That's on purpose actually, because an untracked bench_new.py, conftest.py or .so can change what runs.

    That's not all that interesting. This is very poor UX. There are many reasons to have uncommitted files. E.g., a single AGENTS.md at the root of your repo that you can't commit will permanently mark everything as dirty. You should just drop this check, and only consider uncommitted changes to existing files.

  4. rgommers commented on Nov 18, 2026

    @rgommers
    CollaboratorAuthor

    Or just leave that dirty info out completely - I really don't care about it, it's pure noise for pretty much every real-world local benchmark execution scenario I can think of.

  5. ev-br commented on Nov 19, 2026

    @ev-br
    Collaborator

    Thanks for checking it @rgommers ! Could you please check it once more, with the tip of gh-118? Also add --tables to where you plumb it to spin and pixi. In benchx-dev/benchx_examples#18 the addition is

    $ git diff
    diff --git a/examples/numpy-revisions/compare.sh b/examples/numpy-revisions/compare.sh
    index 0b7e643..c1456b2 100755
    --- a/examples/numpy-revisions/compare.sh
    +++ b/examples/numpy-revisions/compare.sh
    @@ -65,4 +65,4 @@ for round in 0 1 2 3 4; do
         measure "$contender" "$contender_build" "$round" $((2 * round + 1))
     done
     
    -bx compare --run "$run_key" --baseline "$baseline" --tables
    +bx compare --run "$run_key" --baseline "$baseline" --tables --all

    It now emits hopefully clearer diagnostics:

    INDETERMINATE (2)
      bench_function_base.Mean.time_mean [size=100000] wall-time  effect        ?  noise       ?  rounds=0  (not-measured: contender skipped)
      bench_function_base.Mean.time_mean_axis [size=100000] wall-time  effect        ?  noise       ?  rounds=0  (not-measured: contender skipped)
    ...
    
    excluded (10)
      skipped: harness.skipped: NotImplementedError()  (10 results)
    
    ...
    
    bench_function_base.Mean.time_mean_axis  skipped
      ======== ============
        size               
      -------- ------------
         1      7.59±0.2μs 
         10     7.85±0.1μs 
       100000    skipped   
      ======== ============
    
    | Change   | Before [6e1aca7e]   | After [dbf82318]   | Ratio   | Noise   | Verdict                      | Benchmark (Parameter)                           |
    |----------|---------------------|--------------------|---------|---------|------------------------------|-------------------------------------------------|
    | ?        | 106±1μs             | skipped            | n/a     | n/a     | indeterminate (not-measured) | bench_function_base.Mean.time_mean(100000)      |
    | ?        | 107±0.6μs           | skipped            | n/a     | n/a     | indeterminate (not-measured) | bench_function_base.Mean.time_mean_axis(100000) |
    

    In the end of the day, there's a limit on how clear the messaging can be without changing the benchmark itself: the original SciPy function emits a NotImplementedError in the setup, and the benchmark runner cannot tell an intentional skip from a failure.


    There's a clear need to control the level of verbosity of the output: here some output comes from bx bench and some more from bx compare --run. This should again come together with logging, where logs get everything and the console output reacts to arguments like -q / -v. I'm working on this in a separate branch.

  6. ev-br commented on Nov 19, 2026

    @ev-br
    Collaborator

    Re what counts as "dirty":

    AFAIU, the main objection is that untracked files make the tree "dirty"? That's on purpose actually, because an untracked bench_new.py, conftest.py or .so can change what runs.

    That's not all that interesting. This is very poor UX. There are many reasons to have uncommitted files. E.g., a single AGENTS.md at the root of your repo that you can't commit will permanently mark everything as dirty. You should just drop this check, and only consider uncommitted changes to existing files.

    I suppose in your example there's a reason you don't .gitignore the AGENTS.md.
    That said,

    • I believe the footgun of untracked files messing up what runs in unexpected ways is real. Autouse fixtures in an untracked conftest, benchmarks importing things from uncommitted files, an .so shadowing .py module, these are all real scenarios which must not pass silently.
    • It's probably fine to make the "dirty" criterion only look at tracked files, provided there's a log message to say that there are uncommitted files, Add logging #85 (comment)
  7. rgommers commented on Nov 19, 2026

    @rgommers
    CollaboratorAuthor

    Could you please check it once more, with the tip of gh-118?

    Done, and it looks better now, thanks. If you want to keep output verbosity and dirty tree handling separate, then it looks to me like that PR is ready.

    There's also some measurement accuracy thing I noticed in the same tests, but I'll open a separate issue for that.

    I suppose in your example there's a reason you don't .gitignore the AGENTS.md.

    It's just an example, it could just as easily be .md plan files, random notes, or whatever - it's just quite common to have local files in-tree. It would not make sense to have a general .gitignore for .md, .txt, etc. nor to open SciPy PRs to extend .gitignore each time one has such files.

    I believe the footgun of untracked files messing up what runs in unexpected ways is real.

    Even if it is - which is arguable and should be rare - it's a thing you may learn the hard way once, and then you know or at least quickly realize. While output pollution is continuous, and quite in-your-face.

    It's probably fine to make the "dirty" criterion only look at tracked files, provided there's a log message

    A log message seems fine to me.

  8. ev-br commented on Nov 19, 2026

    @ev-br
    Collaborator

    thanks Ralf!

    If you want to keep output verbosity and dirty tree handling separate, then it looks to me like that PR is ready.

    Yes, that PR is large enough already to keep these two to a follow-up PR. gh-118 has thus landed, and I'll look into logging (which keeps smoking unrelated issues , the last of which is #125) and better defining what "dirty" is.
    Will keep the issue open while that's in flight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions