Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ jobs:
working-directory: benchx
run: |
uv python install 3.12
uv sync --extra test --python 3.12
uv sync --extra parquet --extra test --python 3.12
echo "$PWD/.venv/bin" >> "$GITHUB_PATH"

- name: Install NumPy's build tools
Expand Down Expand Up @@ -208,7 +208,7 @@ jobs:
working-directory: benchx
run: |
uv python install 3.13
uv sync --extra test --python 3.13
uv sync --extra parquet --extra test --python 3.13
echo "$PWD/.venv/bin" >> "$GITHUB_PATH"

# cmake and a C++ compiler are preinstalled on the runner; CMake fetches
Expand Down Expand Up @@ -270,7 +270,7 @@ jobs:
working-directory: benchx
run: |
uv python install 3.13
uv sync --extra test --python 3.13
uv sync --extra parquet --extra test --python 3.13
echo "$PWD/.venv/bin" >> "$GITHUB_PATH"

- name: Show what is under test
Expand Down Expand Up @@ -325,7 +325,7 @@ jobs:
working-directory: benchx
run: |
uv python install 3.13
uv sync --extra test --python 3.13
uv sync --extra parquet --extra test --python 3.13
echo "$PWD/.venv/bin" >> "$GITHUB_PATH"

# The README's install lines. torch comes from PyTorch's CPU index: the default
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ jobs:
run: |
uv venv --python 3.13 "$RUNNER_TEMP/venv"
source "$RUNNER_TEMP/venv/bin/activate"
uv pip install "$(ls dist/*.whl)[test]"
uv pip install "$(ls dist/*.whl)[parquet,test]"
# Away from the checkout, so that nothing can import it instead of the installed package.
cd "$RUNNER_TEMP"
bx --help
Expand Down
48 changes: 45 additions & 3 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,40 @@ jobs:
run: uv python install ${{ matrix.python-version }}

- name: Install package and test dependencies
run: uv sync --extra test --python ${{ matrix.python-version }}
run: uv sync --extra parquet --extra test --python ${{ matrix.python-version }}

- name: Run tests
run: uv run --extra test --python ${{ matrix.python-version }} python -m pytest tests -ra
run: uv run --extra parquet --extra test --python ${{ matrix.python-version }} python -m pytest tests -ra

minimal:
name: minimal (no pyarrow) / py3.13
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: uv.lock

- name: Install Python 3.13
run: uv python install 3.13

- name: Install package without parquet extra
run: uv sync --extra test --python 3.13

- name: Verify pyarrow and jsonpointer are absent
run: |
uv run --extra test --python 3.13 python -c '
import importlib.util
for mod in ("pyarrow", "jsonpointer"):
assert importlib.util.find_spec(mod) is None, f"unexpected dependency installed: {mod}"
'

- name: Run tests without pyarrow
run: |
uv run --extra test --python 3.13 bx --help > /dev/null
uv run --extra test --python 3.13 python -m pytest tests -ra

# An editable install finds the schemas next to the source, so the matrix above cannot
# tell whether a wheel carries them. This installs the built wheel (not editable) into a
Expand All @@ -70,8 +100,20 @@ jobs:
run: |
uv venv --python 3.13 "$RUNNER_TEMP/venv"
source "$RUNNER_TEMP/venv/bin/activate"
uv pip install "$(ls dist/*.whl)[test]"
uv pip install "$(ls dist/*.whl)"
# Away from the checkout, so that nothing can import it instead of the installed package.
cd "$RUNNER_TEMP"
python -c '
import importlib.metadata
reqs = importlib.metadata.requires("benchx") or []
default_reqs = [r for r in reqs if "extra ==" not in r]
parquet_reqs = [r for r in reqs if "extra ==" in r and "parquet" in r]
assert any(r.startswith("pyarrow") for r in parquet_reqs), reqs
for unwanted in ("pyarrow", "jsonpointer", "format-nongpl", "fqdn", "idna", "isoduration", "rfc3987-syntax", "uri-template", "webcolors"):
assert not any(unwanted in r for r in default_reqs), f"unexpected default dep {unwanted!r} in {default_reqs}"
'
bx --help
uv pip install "$(ls "$GITHUB_WORKSPACE"/dist/*.whl)[test]"
python -m pytest --pyargs benchx -ra
uv pip install "$(ls "$GITHUB_WORKSPACE"/dist/*.whl)[parquet,test]"
python -m pytest --pyargs benchx -ra
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ allow all of them.

## BenchX usage and development

Install with `pip install benchx` ([PyPI](https://pypi.org/project/benchx/)).
Install with `pip install benchx` (or `pip install 'benchx[parquet]'` for the local Parquet store) ([PyPI](https://pypi.org/project/benchx/)).

We welcome contributions of all sorts: add a user story, tell us about your use case, try
`benchx` on your workflow/project, open an issue if you found a rough edge,
Expand Down
37 changes: 29 additions & 8 deletions docs/benchx.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ implement a missing feature.
Install from [PyPI](https://pypi.org/project/benchx/):

```console
$ pip install benchx
$ pip install benchx # minimal: run with --no-ingest, validate, and compare --results
$ pip install 'benchx[parquet]' # adds PyArrow for the local Parquet store and history queries
```


Expand Down Expand Up @@ -55,7 +56,7 @@ bx compare BASELINE CONTENDER --profile revisions|environments
--suite NAME [--filter REGEX] [--quantity wall-time]
--rounds R [-e NAME=VALUE ...] [--repetitions N] [--min-time S]
[--project P] [--label NAME=BASE,CONTENDER]
[--run-key KEY] [--out DIR] [--no-build] [--k 3] [--json | --tables [--all]]
[--run-key KEY] [--out DIR] [--no-build] [--no-ingest] [--k 3] [--json | --tables [--all]]

# measure mode, one target: alternate the two values of a benchmark parameter
bx compare TARGET --profile variants --adapter pyperf|asv --suite NAME --rounds R
Expand All @@ -73,6 +74,10 @@ bx ingest <dir|file> # result files and work orders → the
bx series [--workload … --quantity …] # list series
bx history <series> # one series' points (a query); dirty ones are marked
bx head [-n 10] [--project P] [--workload W] [--json|--table] # the latest results, newest first; dirty ones are marked
bx view [--out DIR] [--serve] [--host 127.0.0.1] [--port 8000] [--open]
[--store PATH | --results PATH… | --runs DIR]
[--project-file PATH] [--title TITLE] [--repo DIR] [--marts | --no-marts] [--stamp STR]
# build and serve the static benchmark dashboard (no --out: temp dir + serve)

bx target describe|prepare TARGET # a target's description (a provider's answer, or a build directory's sidecar)
bx validate <file>… [--kind KIND] # every problem in each document, by schema and rules
Expand Down Expand Up @@ -247,7 +252,8 @@ outside the checkout is identified by one file): put it inside the checkout.
and it refuses an order that names a deferred field (`environment_policy`,
a declared `build`, `components`, `schedule`) instead of ignoring it.
- **The store** is one Parquet file, by default `store.parquet` in the project root
(`paths.store` in `benchx.conf.json` moves it). It reads documents strictly, validates them, and applies the
(`paths.store` in `benchx.conf.json` moves it), and requires the optional `parquet` extra
(`pip install 'benchx[parquet]'`). It reads documents strictly, validates them, and applies the
schema §5.4 contract: idempotency and the rejection codes `malformed`,
`idempotency-conflict`, `unit-conflict`, `identity-violation`, and
`attempt-conflict`. Each result is one row: the columns
Expand All @@ -257,6 +263,12 @@ outside the checkout is identified by one file): put it inside the checkout.
query, under the identity policy in `benchx/identity.py`: the default of
`schemas/identity-policy/0.1.0` with the benchmark's revision ignored.
Every accepting ingest rewrites the file atomically.
- **Schema validation** checks documents against the schemas shipped inside the package, or custom
schemas pointed to by `$BENCHX_SCHEMAS`. The shipped schemas use only the `date-time` and `uri`
JSON Schema formats, checked via `rfc3339-validator` and `rfc3986-validator`. Custom schemas
supplied through `$BENCHX_SCHEMAS` that declare additional formats require the corresponding
`jsonschema` format validator package (or `jsonschema[format-nongpl]`); if a declared format has
no installed checker, validation fails explicitly rather than silently accepting unchecked values.
- **The session loop** (`benchx/session.py`, `bx compare` in measure mode) is
the workbench's part: it checks both targets (and, for `revisions`, that both
trees have a working-tree id, before running anything; it says which sides
Expand All @@ -278,8 +290,8 @@ outside the checkout is identified by one file): put it inside the checkout.
|effect| > *k* × noise). `environments` names its sides by the label
`--label` gives, and `variants` by the two values of one workload parameter
(`--vary`); under `variants` a failed attempt on either side fails its unit
(`side-failed`). `--results` compares result files through a throwaway
store: the ad hoc path. Condition measurements (`measurement.kind:
(`side-failed`). `--results` validates and compares JSON result files in memory
without PyArrow or a Parquet store: the ad hoc path. Condition measurements (`measurement.kind:
condition`) are not compared, and deterministic quantities are deferred.
- **The tables** (`--tables`, `benchx/tables.py`) print, after the usual summary, what
`asv continuous` prints, laid out and rounded as asv does: a table of each side (a value
Expand All @@ -290,12 +302,20 @@ outside the checkout is identified by one file): put it inside the checkout.
deviation (not asv's `±`); `Change` and `Verdict` are the comparator's verdict (`+`
regressed, `-` improved), `Noise` the one it compares the effect with (a change is more than *k*
times it), and `Ratio` is After over Before. A text view: not with `--json`.
- **Views** (`bx view`, `benchx/views/`) derive flat tables (`commits`, `points`, `changes`,
`latest`, `movers`) and static JSON view payloads (`landing`, `series`, `sparks`,
`history/<suite>`, `commits/<sha>`) in pure Python from the local Parquet store, JSON result
files (`--results`, no PyArrow required), or a CI `runs/<sha>/<node>/` directory (`--runs`,
with `run.parquet` and `results.parquet`). With no `--out`, `bx view` builds into a temporary
directory and serves the bundled static dashboard over local HTTP until Ctrl-C; with `--out DIR`,
it writes a deployable static site (and optional Parquet marts under `data/` when `pyarrow` is
installed, unless `--no-marts`).

## Run it

```console
$ uv venv && uv pip install -e '.[test]' # editable; a plain `uv pip install .` works too, the schemas ship in the package
$ .venv/bin/python -m pytest tests # 470 tests, fake Google Benchmark binary, no compiler
$ uv venv && uv pip install -e '.[parquet,test]' # editable; omit `parquet` for a minimal Arrow-free install
$ .venv/bin/python -m pytest tests # fake Google Benchmark binary, no compiler
```

`tests/test_e2e_numpy_revisions.py` is an end-to-end test that builds NumPy with
Expand All @@ -308,7 +328,8 @@ keeps the builds between runs, `BENCHX_E2E_REVISIONS` picks the two tags,
scenario, and the test's docstring has the rest.

A regular (non-editable) install ships the test suite too, so an installed copy can test itself:
`pip install 'benchx[test]'` followed by `python -m pytest --pyargs benchx`.
`pip install 'benchx[parquet,test]'` (or `pip install 'benchx[test]'` for the Arrow-free subset)
followed by `python -m pytest --pyargs benchx`.

The end-to-end examples that build real software are in
[benchx_examples](https://github.com/benchx-dev/benchx_examples), and the `e2e`
Expand Down
6 changes: 3 additions & 3 deletions docs/releasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ package, waits for approval, publishes to PyPI, and creates the GitHub release.

```console
$ python -m venv venv && source venv/bin/activate
$ pip install 'benchx[test]==0.1.0'
$ pip install 'benchx[parquet,test]==0.1.0'
$ bx --help
$ python -m pytest --pyargs benchx
```
Expand All @@ -42,8 +42,8 @@ package, waits for approval, publishes to PyPI, and creates the GitHub release.
After a new version reaches PyPI, conda-forge's bot opens a pull request there, which can take
several hours. A maintainer of the feedstock then:
- Compares the recipe's dependencies with `dependencies` in `pyproject.toml`, and updates the
pull request if they changed. `jsonschema[format-nongpl]` is called
`jsonschema-with-format-nongpl` on conda-forge.
pull request if they changed (`pyarrow` is optional via the `parquet` extra, and format
validation uses `rfc3339-validator` and `rfc3986-validator` directly).
- Merges the pull request once its CI passes.
- Tests the install, once the package is available:

Expand Down
9 changes: 5 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ license = "Apache-2.0"
license-files = ["LICENSE"]
requires-python = ">=3.11"
dependencies = [
"pyarrow>=19",
# format-nongpl makes `format: uri` and `format: date-time` actually checked.
"jsonschema[format-nongpl]>=4.18",
"jsonpointer>=2.0",
"jsonschema>=4.18",
# Format validators for the formats used by the shipped schemas (`date-time`, `uri`).
"rfc3339-validator",
"rfc3986-validator>0.1.0",
"referencing",
"rfc8785",
]
Expand All @@ -24,6 +24,7 @@ Documentation = "https://benchx-dev.github.io/benchx/benchx/"
Repository = "https://github.com/benchx-dev/benchx"

[project.optional-dependencies]
parquet = ["pyarrow>=19"]
test = ["pytest", "pyperf", "asv_runner"]

[project.scripts]
Expand Down
2 changes: 1 addition & 1 deletion src/benchx/bench.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from . import core, snapshot
from .errors import SessionError
from .projection import declared_workload, said, workload_label
from .store import warning_codes
from .results import warning_codes


def tree_of(module: Path, tree=None) -> Path:
Expand Down
Loading
Loading