Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/benchx.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,14 @@ resolved values are written into the order. `--quantity` takes a comma-separated
and one run collects them all. The workers run the `python` first on `PATH`, which must have
pyperf.

A function has a version, as an asv benchmark does: its `version` attribute, else a hash of
its source (as asv_runner hashes a plain function: comments and docstrings count; what it
calls, its `params` and the library under test do not). Each result records it in
`protocol.pyperf.benchmark_version`, and protocol is what the two sides of a comparison must
agree on, so two revisions whose function differs fail `sides-agree`. That suits a function
that exercises code elsewhere in the tree. When the function body is what changes, declare a
`version` (`bench_a.version = "1"`). A function without readable source has no version.

`bx bench MODULE.py [FUNCTION …]` is the shorthand for the quickest case, timing a few
functions: the module's git checkout is the target, its path the suite, and the
functions are exact case names. `bx run` is the layer underneath: it executes a finished
Expand Down Expand Up @@ -208,7 +216,9 @@ calibrates loops as usual. The flags are pyperf's
`--case-timeout` covers all its rounds (default 1800 s). The harness in
every result is pyperf. Results also record the installed project's version
(`provenance.info.asv.installed`) and asv's hash of the benchmark's code
(`protocol.asv.benchmark_version`), so editing a benchmark starts a new series.
(`protocol.asv.benchmark_version`), so editing a benchmark starts a new series. An asv
suite outside the subject's checkout is refused (an asv suite is a directory, and a suite
outside the checkout is identified by one file): put it inside the checkout.

## What it does

Expand Down
2 changes: 1 addition & 1 deletion docs/design/benchmark-result-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -506,7 +506,7 @@ Results carry facts; a **profile** is a comparator-time check that a set of resu

A comparison in which any side is not `clean` is **local-only** under the profiles that allow it (`single-run`, `variants`, `revisions`, `environments`): the comparator produces it, labels it so, names the sides that are not clean, and never promotes it into tracked history. The tree id is what makes each side internally consistent even though neither may have a commit. Under `revisions` this is the comparison of an uncommitted change against HEAD, or against another revision, that the one-off and Arrow local stories want (UC-03 §10). What `revisions` still requires is that every side can be identified: a side whose dirty state is `unknown` has no tree id (§4.1), so it cannot be told apart from the other side or checked for a change during the run, and it fails the invariant that each side has exactly one tree id, so its units get no verdict.

A benchmark is **recorded separately** from the subject when its suite lives outside the subject's checkout (`runner-schema.md` §3.3). Its `benchmark_tree` is then the suite file's own id and differs from the `subject_tree`; this includes a suite in another checkout of the same repository, which has the same source URI but its own file id. A benchmark inside the subject's checkout has `benchmark_tree` equal to `subject_tree` and the subject's own source and revision as `benchmark.source` and `benchmark.revision`; a result whose two trees are equal but whose `benchmark.source` differs from its `source` is malformed. Under `revisions` and `variants` a separately recorded benchmark must be one file on both sides and in every round (invariant `one-benchmark-tree`): the suite is part of the code identity, which UC-03 requires the sides to agree on and UC-02 holds controlled (UC-02 §5), and an edit or commit of the suite during the run is otherwise invisible, because the identity policy maps exact benchmark revisions to declared versions (§4.1) and `benchmark_dirty` only compares the file with its own HEAD. Benchmarks inside the subject's checkout differ between the sides exactly as the subject trees do under `revisions` and are not checked; under `variants` an edit of one changes the subject tree, which the sides must share (`sides-agree`). A result without a `benchmark_tree`, such as a thin result, is outside the rule.
A benchmark is **recorded separately** from the subject when its suite lives outside the subject's checkout (`runner-schema.md` §3.3). Its `benchmark_tree` is then the suite file's own id and differs from the `subject_tree`; this includes a suite in another checkout of the same repository, which has the same source URI but its own file id. A benchmark inside the subject's checkout has `benchmark_tree` equal to `subject_tree` and the subject's own source and revision as `benchmark.source` and `benchmark.revision`; a result whose two trees are equal but whose `benchmark.source` differs from its `source` is malformed. Under `revisions` and `variants` a separately recorded benchmark must be one file on both sides and in every round (invariant `one-benchmark-tree`): the suite is part of the code identity, which UC-03 requires the sides to agree on and UC-02 holds controlled (UC-02 §5), and an edit or commit of the suite during the run is otherwise invisible, because the identity policy maps exact benchmark revisions to declared versions (§4.1) and `benchmark_dirty` only compares the file with its own HEAD. Benchmarks inside the subject's checkout differ between the sides exactly as the subject trees do under `revisions` and are not checked, except through the version that a harness adapter records for each benchmark function in `protocol` (a hash of its source, or one the function declares: `protocol.asv.benchmark_version`, `protocol.pyperf.benchmark_version`), which the sides must agree on like the rest of protocol; under `variants` an edit of one changes the subject tree, which the sides must share (`sides-agree`). A result without a `benchmark_tree`, such as a thin result, is outside the rule.

Thin results (§4.1) may claim `single-run`, `variants`, and `revisions` within their own run; the other profiles require a full environment identity.

Expand Down
2 changes: 1 addition & 1 deletion docs/design/harness-adapter.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ pyperf's JSON (`-o`) holds `benchmarks[]`, each with `metadata` and `runs[]`; me
| `date` of the first run, the benchmark, or the file, most specific first | `started_at` fallback |
| `tags` | `provenance.labels` |

pyperf has no parameters of its own, so the implementation's adapter reads asv's convention from the benchmark function: the attributes `params` (one list of values per parameter, or a flat list) and `param_names`. A combination is a case named `function-INDEX`, the driving half calls the function with it as positional arguments, and the values go to `workload.parameters` as `repr()` strings with default addresses removed, as for ASV (§4.3, §6.7). A filter is searched in `function(p0, p1)` for such a function, once per combination, so the bare name selects all of them. `setup`, `teardown` and `skip_params` are ASV's and are not read.
pyperf has no parameters of its own, so the implementation's adapter reads asv's convention from the benchmark function: the attributes `params` (one list of values per parameter, or a flat list) and `param_names`. A combination is a case named `function-INDEX`, the driving half calls the function with it as positional arguments, and the values go to `workload.parameters` as `repr()` strings with default addresses removed, as for ASV (§4.3, §6.7). A filter is searched in `function(p0, p1)` for such a function, once per combination, so the bare name selects all of them. `setup`, `teardown` and `skip_params` are ASV's and are not read. A function also has a version, as in ASV: its `version` attribute, else a SHA-256 of its source as asv_runner computes it for a plain function (comments and docstrings count; helpers it calls and `params` do not). It is found when the module is listed and recorded in `protocol.pyperf.benchmark_version` (ASV's adapter records the same hash in `protocol.asv.benchmark_version`), so that protocol, which the sides of a comparison must agree on, tells an edited benchmark from the one it was. A function whose source cannot be read has no version. The mapping version is unchanged by this addition (a decision, not the rule of principle 7): results made before simply lack the key.

Attempt key: one per benchmark per invocation. The workers are pyperf's own replication and become observation groups, never attempts (§4.6); fewer runs with values than requested processes is `partial`. The file stores no summaries: pyperf's mean and standard deviation, and the median of `pyperf stats`, are computed when displayed, so the result carries observations only and the server derives estimates. If the driving half runs `pyperf check`, its output is an artifact and a reported instability is the warning `producer-unstable`, never a status. A failing benchmark can end the invocation without an output file, so the account step of §5 is what turns it into `error` results, with stderr as the artifact. With `--append`, the driving half translates only the benchmarks and runs its own invocation added, so older runs in the file are not claimed by this attempt.

Expand Down
10 changes: 7 additions & 3 deletions docs/design/prototype-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -456,9 +456,13 @@ apart, so adding an estimator never renames the policy:
- **Identity policy** `benchx/benchx-identity/v1` (`benchx/identity.py`):
the default policy (`schemas/identity-policy/0.1.0/examples/default.json`)
with the benchmark's revision ignored. A test checks it against the schema
and pins it to its name, so an edit without a new name fails. The implementation has no declared benchmark versions, and
with benchmarks in the subject's checkout the benchmark revision changes on
every commit, so matching it would split history at every point (schema §1).
and pins it to its name, so an edit without a new name fails. The implementation has no *declared* benchmark versions (a
project's own mapping, schema §4.1), and with benchmarks in the subject's checkout the
benchmark revision changes on every commit, so matching it would split history at every
point (schema §1). It does record a computed version of each benchmark function in
`protocol` (asv, pyperf: a hash of the function's source, or its `version` attribute),
which changes only when the function does and is matched like the rest of protocol: an
edited benchmark is not compared with its old self and starts a new series.
Being ignored, its change is flagged on each point where it happens. A thin
result's identity carries its implicit project.
- **Estimators** (`ESTIMATORS`): `median`, derived from observations, plus
Expand Down
20 changes: 20 additions & 0 deletions src/benchx/adapters/benchx_params.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,11 @@
and a value is recorded as the string `stable_repr` gives, as asv does.
"""

import hashlib
import inspect
import itertools
import re
import textwrap
from collections import Counter


Expand All @@ -26,6 +29,23 @@ def stable_repr(obj) -> str:
return text


def source_version(func):
"""The version of a benchmark function, as asv has it: an explicit `version` attribute (any value,
kept as a string), else a SHA-256 of its source text, which is what asv_runner hashes for a plain
function: the lines stripped of trailing whitespace, dedented. Comments and docstrings count. What the
function calls, its `params` and the library under test do not. None when the source cannot be read
(a built-in, code made by `exec`) and no version is declared."""
declared = getattr(func, "version", None)
if declared is not None:
return str(declared)
try:
lines, _ = inspect.getsourcelines(func)
except (OSError, TypeError):
return None
source = textwrap.dedent("\n".join(line.rstrip() for line in lines)).rstrip()
return hashlib.sha256(source.encode("utf-8")).hexdigest()


def normalize(params, param_names):
"""(values per parameter, parameter names, value strings per parameter).

Expand Down
23 changes: 22 additions & 1 deletion src/benchx/adapters/pyperf.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,13 @@
`setup`, `teardown` and `skip_params` are asv's, and pyperf's `bench_func` has no
place for them.

Version: a function has one, as in asv, so that an edited benchmark is not compared with its
old self. It is the function's `version` attribute when it has one, else a hash of its source
(`benchx_params.source_version`), found when the module is listed, and recorded in each
result's `protocol.pyperf.benchmark_version`. Protocol is what the sides must agree on, so two
revisions whose function differs fail `sides-agree`. A function without readable source has no
version.

The driver records the environment it actually had, before it imports the
user's module: a digest in every worker's run metadata and, from a worker,
the full environment as an artifact. The adapter reports what the workers had, not what the manager
Expand Down Expand Up @@ -166,6 +173,12 @@ def locate(target_path, suite, env, timeout=None) -> dict:
getattr(func, "params", []), getattr(func, "param_names", []))
except Exception as error:
info["error"] = f"{type(error).__name__}: {error}"
try:
version = params.source_version(func)
except Exception: # a callable with odd attributes: no version, still a benchmark
version = None
if version is not None:
info["version"] = version
found.append(info)
print(json.dumps(found))
"""
Expand Down Expand Up @@ -301,7 +314,8 @@ def run_case(runnable, case, flags, env, timeout, native_path, env_names=None) -
artifacts.append(("worker-environment", "application/json", worker_env))
return {"exit_status": status, "timed_out": status is None, "stdout": stdout or "", "stderr": stderr or "",
"native": native, "artifacts": artifacts,
"worker_env": captured, "passed_env_digest": env_digest(env)}
"worker_env": captured, "passed_env_digest": env_digest(env),
"benchmark_version": runnable.get("benchmarks", {}).get(function, {}).get("version")}


def _failure_message(stderr: str) -> str:
Expand Down Expand Up @@ -336,6 +350,13 @@ def repetition_levels(applied: dict, run: dict, case: str) -> list[dict]:
return levels


def case_protocol(applied: dict, run: dict, case: str) -> dict:
"""One case's protocol: the order's, and the version of the benchmark function when there is one
(`benchx_params.source_version`), so that sides that ran different code do not agree."""
version = run.get("benchmark_version")
return {**applied, "pyperf": {"benchmark_version": version}} if version else applied


def harness(info: dict) -> dict:
out = {"name": NAME}
if info.get("library_version"):
Expand Down
8 changes: 7 additions & 1 deletion src/benchx/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,13 @@ def _found_benchmark(harness, path: Path, suite: str, subject_source: str) -> di
snapshot.benchmark_identity), else None. A suite that is a built binary is the subject's own."""
if not adapters.get(harness.name).SUITE_IS_SOURCE:
return None
return snapshot.benchmark_identity(path / suite, subject_source)
found = snapshot.benchmark_identity(path / suite, subject_source)
if found is not None and not found.get("tree"):
# Outside the checkout a suite is identified by one file (its blob id); an asv suite is a directory.
raise SessionError(f"cannot record the suite {(path / suite).resolve()}: it is outside the subject's "
f"checkout ({subject_source}), where a suite is identified by one file, which a "
"directory is not; put it inside the checkout")
return found


def _benchmark(found: dict | None) -> dict | None:
Expand Down
Loading
Loading