Repository navigation
UI: add asv-stype table for comparisons - #118
Conversation
|
One other change: a failing benchmark does not fail the whole comparison: in benchx-dev/benchx_examples#18, With ba72add, the output is |
|
... and several commits to address other items from gh-117, plus a couple of drive-by cleanups:
|
asv is timed by pyperf and takes the same precision, but `bx bench --adapter asv` was refused when the file's `adapter.name` was `pyperf`, so one benchx.conf.json could not hold the timing for both kinds of suite. Harnesses now have a precision family: pyperf and asv (which subclasses it) are one, google-benchmark is another. `--adapter` may name any adapter of the file's block family. Across families it is still refused, with a message that says so. No schema change: `adapter.name` stays required, and only its description is corrected (it did not mention asv). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`bx compare --tables` prints, after the usual summary, what `spin bench --compare` (asv) prints: a table of each side's results, then a table of the changes with Ratio, Noise and the comparator's verdict. As with `asv continuous`, only the changed rows are listed unless `--all` is given. The layout and the number formatting follow asv's (`format_text_table`: every cell centered; `tabulate`'s github format; `human_time` and `human_float` for values). The `±` is benchx's own: the standard deviation of the per-round medians, and a side's value is their median. The per-side numbers are recomputed from the run's results for display, so the comparison document and its schema are unchanged. `--tables` is a text view: it is refused together with `--json`. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…spaces The grids sit two spaces in under the benchmark's title line. The title, the one-line value of a benchmark without parameters, and the change table are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… compared
`bx compare` returned no verdict at all when the matched facts of one benchmark
disagreed between the sides. The `sides-agree` check is about one unit, but any
disagreement made `compare()` return a document with no units, so a benchmark that
raised on one side (it reports no harness version or runtime) stopped `time_mean`
from being compared, though it ran fine on both sides. `equal-attempts` already
dropped only its unit, and the design says a failed invariant leaves "no verdicts for
the units it affects".
- `sides-agree` now drops only the unit that disagrees: it is recorded in
`failed_invariants` with its workload, and the other units are compared. The
invariants about the whole run (the sides, their tree ids, the alternation, the
benchmark file) still leave no unit compared.
- The matched facts are those of the attempts that succeeded. A failed attempt reports
none of them: it is excluded and listed, and its benchmark is `indeterminate`
(`insufficient-rounds`). A benchmark edited between the revisions, so that its
benchmark version differs on two successful sides, still fails `sides-agree`, for
itself only.
- A unit's `unmatched_differences` (the `differs:` line) also counts successful
attempts only, so the facts a failed attempt did not report are not differences.
No schema change. `comparator.md` and `prototype-design.md` say which invariants affect
the run and which one unit.
An example output:
```
bench_function_base.Mean.time_mean_axis error
======== =======
size
-------- -------
1 error
10 error
100000 error
======== =======
| Change | Before [6e1aca7e] | After [7b3b54e0] | Ratio | Noise | Verdict | Benchmark (Parameter) |
|----------|---------------------|--------------------|---------|---------|-------------------------------------|-------------------------------------------------|
| | 7.04±0.1μs | 7.01±0.1μs | 1.00 | 1.33% | no change detected | bench_function_base.Mean.time_mean(1) |
| | 7.21±0.1μs | 7.21±0.2μs | 1.00 | 1.06% | no change detected | bench_function_base.Mean.time_mean(10) |
| | 106±0.8μs | 106±0.3μs | 1.00 | 0.34% | no change detected | bench_function_base.Mean.time_mean(100000) |
| ? | 7.71±0.05μs | error | n/a | n/a | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(1) |
| ? | 7.91±0.1μs | error | n/a | n/a | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(10) |
| ? | 108±1μs | error | n/a | n/a | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(100000) |
```
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A benchmark that raised was shown as `error (harness.error)` and nothing more, though
the worker's exception line is stored in the result (`provenance.info.message`).
Under the table `bx bench` now has a section:
not measured (5)
KeyError: 'axis' (4 cases: bench_axes [n=1], bench_axes [n=10], bench_axes [n=100], …)
bench_boom: ValueError: array has no axis 3
one line per message, with up to three case names; skipped asv benchmarks show their
reason there too. The first line of a message, cut at 160 characters, is
`projection.said`, shared with `bx compare` (next commit).
FIX: the pyperf adapter took as the message the first unindented line of the worker's
stderr, so a warning printed before the traceback was shown instead of the exception.
It is now the line that ends the traceback (the last one when exceptions are chained),
not the manager's "failed with exit code" wrapper; without a traceback, the first line
said. Results stored before keep the message they have.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The `missing`, `failed invariants` and `excluded` listings printed the document's dicts: one line for each failed attempt (15 for one failing benchmark with three sizes over five rounds), with an ingest key and the category `harness.error` and nothing about what went wrong. - excluded: the reason of an exclusion ends with what the failed attempt said (`error: harness.error: ValueError: array has no axis 3`; unchanged when it said nothing), and results excluded for the same reason are one line with their number. Which results they are is in `--json`. - failed invariants: `invariant: detail`, naming the unit it affects, and units that failed alike are one line, `(K units: a, b, c, …)`. - missing: `not on contender: a, b, c, … (+N more)`, one line for each side. No schema change: the reason is free text. Naming the benchmark on an excluded line would need fields of the exclusion, a schema addition left for later. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…elpful A filter that named a parametrized benchmark but gave its arguments in the wrong order was refused with the first ten benchmark names, on one long line, with the filter shown as a Python string (`\.` as `\\.`). The refusal is now one fact per line: - the filter as typed; - when the filter, up to its first `\(`, matches the name of a parametrized benchmark, how its cases are searched: `name(p0, p1)`, each as its repr, in the order of `param_names`, with the parameters of that benchmark and one case; - the benchmarks similar to the filter (words of 3+ characters compared with the dotted parts of each name, difflib ratio >= 0.75, the names matching the most words first, 5 at most), else the first 5 available. Only the text of the refusal changes: not what a filter selects, no schema, no new option. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
For a parametrized case the generated driver had `if <index> is not None:` with the combination index written in, so `if 0 is not None:`. Python warns about `is not` with a literal: a SyntaxWarning on the stderr of every worker of every parametrized case (pyperf and asv). The warning also came before a failing case's traceback. The index is now bound to a name first (`index = 0`) and the name is tested. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`bx compare` labelled `Partition.time_argpartition` as `[array_type=..., dtype='int16', k=10]` and the per-commit table had the columns `array_type, dtype, k`, though the benchmark declares `dtype, array_type, k`: the order `--filter` searches `name(p0, p1, ...)` in, and asv shows. A filter copied from the output selected nothing. The result file had the order, but the store keeps a document in canonical form, with its keys sorted, and `bx compare --run`, `bx head --table` and the tables read the store. A JSON object has no order a schema promises, so the order is now data: the runner records it in `provenance.info.parameter_names` (free-form, no schema change) for a workload with two or more parameters, and `projection.declared_workload` puts the parameters back in it. `compare` (units, failed invariants, missing, and so the tables) and `bx bench` / `bx head --table` read through it. Units are listed with the first declared parameter changing slowest, as asv lists a product of params; before, by parameter name. Results made before have no list and show as they did. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
e652dc9 to
488125c
Compare
|
@MarcoGorelli as discussed earlier today, this is the base PR in my "stack". It mainly addresses the comments from gh-117 and tries to keep other changes to a minimum (cf the foldout in the OP) |
MarcoGorelli
left a comment
There was a problem hiding this comment.
cool, this should improve the user experience
couple of minor comments claude raised:
- both sides get the same tag. This is the one that matters. Comparing uncommitted changes against HEAD is probably the most common way people will use bx compare. In that case the change table, which is the main output of this feature, labels both columns Before [abcd1234] and After [abcd1234]. Falling back to the tree id, or adding "(dirty)" to the tag, would fix it.
- The schema change doesn't match the description. The PR description and the tables.py docstring both say the comparison document and its schema don't change. The PR still adds an attempts field to comparison-document/0.1.0, editing that version in place. Either the description is out of date or the change belongs in its own PR. Since the description explicitly asks Maanas about schema changes, it's worth pointing out.
…nge table Uncommitted changes against HEAD are one commit, so both columns of the change table were labelled `[abcd1234]`, and so were the per-side headings. A side with uncommitted changes now carries `(dirty)` in its tag. When both sides are the same commit and both are dirty, the tree id is added to each, so they still differ. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The docstring said the document and its schema are unchanged. The PR adds an optional `attempts` to each unit of `comparison-document/0.1.0` and the reason `not-measured`, in compare.py; say so, and point to comparator.md. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Thanks for the review Marco! The points that you've raised:
Great catch! Fixed now, comparing HEAD against an uncommitted tree adds
Ouch, yeah. The schema change is for #117 (comment), which improved the wording for the failure reasons. |
|
I'm going to press the green button given Marco's approval, and what I'm going to interpret as an approval of the effect of this PR, #117 (comment). The question from the foldout of OP still stands but can be done at leisure in a follow-up or several. Thank you @MarcoGorelli for the review! |
Addresses a part of gh-117; a driver is at benchx-dev/benchx_examples#18
Add
asv-style tables for the output ofbx --compare: with the new option,--table, it additionally emits:@MaanasArora two asks:
could please take a quick peek? You're working on comparisons, and while this is mainly about the output, I'd rather make sure I'm not messing with your work!
One larger item is that the comparison documents do not carry enough information to construct these tables from the store. This PR recomputes the values at display time, but maybe it's useful to add them to the documents themselves? That would require updating the schema though. Claude-generated discussion is under the fold below.
Details
Discussion: recomputing at display time, or putting the numbers in the comparison document
What the tables need that the comparison document does not carry: each side's value and
±, itsunit, its status when it has no value, and its commit. The document has
reference_estimatefor thebaseline only. (
effect,noiseand the verdict are in it, so the Noise and Verdict columns need norecomputation.) Two ways to get the rest.
A. Recompute at display time (chosen for now, decisions 1 and 6).
tables.summarize(doc, documents)reads the run's results from the store, rebuilds the projection from
doc["varying"], assigns eachresult to a side, groups by workload variant and quantity, and takes the median and sd of the per-round
medians.
version question arises, and the feature can be dropped or reworked without a migration. It is a
text view and stays one.
compare(): matching a result to a side byaxis[0], the workload-variant key,the sort order (reused through
compare._unit_orderand_workload). A change to the comparator'sgrouping has to be made twice, or the tables drift from the verdicts.
varyingstring (provenance.labels.NAME,coordinates.workload.parameters.NAME), which is lossy if a name ever contains a..--tablesreads the store in measure mode, andtables.rendercannot be given a document alone (for example one read from--jsonoutput or a CIartifact).
comparator's
reference_estimateuses only the rounds both sides completed. They agree when noround failed, and can differ in the third decimal when one did.
Ratiois After over Before of theprinted values,
effectthe mean of the per-round ratios, so they differ slightly too (documented).B. Put them in the document (the alternative; what I first recommended).
_unitalready holds eachside's per-round medians and the pairs, so it would add
side_estimates: {baseline, contender}, each{estimate, spread, rounds, status}, plus the quantity'sunit, andsides.<role>.revisionfor thecommit.
tables.render(doc)would then be a pure function of the document.by construction; the numbers are in
--json; the renderer needs no store, no projection and nosummarize, and is testable from a document alone. It keeps the comparator's contract(
comparator.md§5): documents are derived and recomputed on demand from the recorded inputs.runUnithasunevaluatedProperties: false, andsidesis closed too, so it is a schema change:optional properties in
schema.json, therun.jsonexample (the history unit is a separatedefinition),
docs/schemas/comparison-document.md, and tests intest_comparator.py.runUnitandsidesare closed, so a document with the new fields does not validateagainst 0.1.0: this cannot be an amendment of 0.1.0, it is a new version. The scheme the docs state
(
docs/design/runner-schema.md§7: semver in the directory path, minor versions only add optionalfields) is written for the runner's schemas and the result schema, not for the comparison document,
which has the same layout. Read that way, this is
0.2.0: a new directory with the schema andexamples, a new
COMPARISON_DOCUMENTversion string, and every consumer that checks the string seesa new value. Whether the comparison document follows that scheme is for the project to say.
kept, or versioned, even though
±is deliberately benchx's own and not asv's.Why A for now. The tables are a view, the numbers' definitions (decision 2) are still the
author's to change after seeing them used, and a view should not be what forces a schema version
decision. A costs a duplicated grouping and a read of the store; B costs a new schema version.
When to move to B. When a second consumer needs the per-side numbers (JSON, a CI gate, another
renderer); or when the tables and the verdicts are seen to disagree (a drift of the kind above); or when
the grouping in
compare()changes. Then the steps are: add the fields in_unitandsides; settle theversion question (a new minor version, if the comparison document follows §7); update
schema.json, the examples,docs/schemas/comparison-document.mdand
test_comparator.py; changetables.renderto take the document alone; deletesummarize,_sideand_projection; and drop thedocumentsargument incli._show. The golden tests of thetables stay as they are, since the output does not change.