Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,5 @@ Currently, the following examples are provided:
- `examples/matmul-params`: parametrized benchmarks to compare installed NumPy and PyTorch
- `examples/numpy-venvs`: the same benchmarks under two NumPy versions, each in its own virtual
environment, alternated round by round and compared.
- `examples/pandas-asv`: one benchmark from pandas' own asv suite, run unchanged with the asv
adapter under two pandas versions, each in its own virtual environment, and compared.
4 changes: 4 additions & 0 deletions examples/pandas-asv/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.venv-*/
out/
pandas/
__pycache__/
124 changes: 124 additions & 0 deletions examples/pandas-asv/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# pandas-asv: one pandas asv benchmark, two pandas versions

One benchmark from pandas' own [asv](https://asv.readthedocs.io) suite, run unchanged by
`bx bench PATH --adapter asv` under two pandas versions, each in its own virtual environment,
alternated round by round, then compared. The asv adapter reads asv's benchmark format as a
language (discovery, `params`, `setup`, `setup_cache`, asv's benchmark version hash) and times it
with pyperf: neither asv nor asv_runner is installed.

The suite is pandas' `asv_bench/` at one tag (`v3.1.1`), the same for both sides; only the
installed pandas differs. That is
[UC-04](https://github.com/benchx-dev/benchx/blob/main/docs/use-cases/UC-04-cross-env.md),
"fixed branch, comparing environments", driven the same way as `examples/numpy-venvs`.

The case is `strings.Methods.time_lower('str')`: one parameter combination of one benchmark.
`dtype="str"` is object dtype in pandas 2.x and the pyarrow-backed string dtype from pandas 3.0,
so the two sides are expected to differ.

## Run it

You need [`uv`](https://docs.astral.sh/uv/), git, and network access the first time. Then, from
this directory:

```console
$ ./run.sh
```

benchx itself runs only inside the two venvs, and the script installs it there: the
benchx of the active environment if there is one (its checkout if it is an editable install,
else the same version from PyPI), else the latest release from PyPI. To try a benchx checkout
without installing it anywhere, name it: `BENCHX_SRC=path/to/benchx ./run.sh`.

The tag, the two pandas versions, the case, the Python and the number of rounds are set at the
top of `run.sh`. Everything it creates is in `pandas/`, `.venv-a`, `.venv-b` and `out/`, all
git-ignored. A run takes under a minute after the venvs are cached.

## What it does

1. **Fetches the suite.** A shallow, sparse clone of pandas at `v3.1.1`, with only `asv_bench/`
checked out (a few MB). Nothing is built: the benchmarks import the pandas installed in each
venv, and each case runs from a scratch directory, so the unbuilt checkout cannot shadow it.
2. **Prepares two venvs**, `.venv-a` with pandas 2.3.3 and `.venv-b` with pandas 3.1.1, each
with pyarrow, pyperf and benchx.
3. **Runs the caller's loop.** For each of the 3 rounds, one call per side, in its venv:

```console
$ bx bench pandas/asv_bench --adapter asv "strings.Methods.time_lower('str')" \
--run-key "$RUN_KEY" --label env=pandas-2.3.3 --round 0 --slot 0 --out "out/$RUN_KEY"
```

`pandas/asv_bench` holds `asv.conf.json`, whose `benchmark_dir` is the suite; its checkout
is the target. The exact case name selects one parameter combination; `--filter REGEX`
(asv's `--bench`) would select by pattern instead. Modules the venv cannot import (no
matplotlib, scipy, ...) are listed on stderr and skipped. Each call prints a table:

```
case median mean ± sd min .. max n status
---------------------------------------- -------- ------------------- -------------------- -- -----------------------------------
strings.Methods.time_lower [dtype='str'] 890.2 us 937.6 us ± 122.9 us 835.2 us .. 1.202 ms 10 success [setup-per-call-unbatched]
```

4. **Compares.** `bx compare --run S --profile environments --label env --baseline pandas-2.3.3`:

```
run pandas-1793878687 profile environments baseline pandas-2.3.3 contender pandas-3.1.1
differs: coordinates.environment.system.deps.pandas, coordinates.environment.system.deps.sha256
caveats: baseline: setup-per-call-unbatched, contender: setup-per-call-unbatched

IMPROVED (1)
strings.Methods.time_lower wall-time effect -85.40% noise 1.76% rounds=3
```

(The numbers vary from run to run.) `effect` is the contender against the baseline, averaged
over the rounds' paired medians: `-85.40%` means pandas 3.1.1 takes about 15% of the time
pandas 2.3.3 does, so about 7x faster, well beyond the round-to-round `noise`. `differs`
names the environment facts that tell the sides apart: `deps.pandas` is the version the
collector printed, and `deps.sha256` is the hash of the collector's whole output, which
changes whenever any of its facts do. Here numpy and pyarrow are the same on both sides, so
pandas is the one difference.
5. **Lists the series.** One per environment, because the collected pandas version is part of
a series' identity. Every point carries the suite's revision (pandas `v3.1.1`), whichever
pandas it ran under: the target is the checkout the benchmarks come from, and what varies is
the environment, recorded by the collector.

## How it is configured

`benchx.conf.json` is found by walking up from the working directory, so `bx` reads it when
`run.sh` runs from here:

- `adapter`: the harness (`asv`) and its precision, in pyperf's terms: 2 processes (asv's
rounds) × 5 values (asv's repeat), 1 warmup, loops calibrated. Flags such as `--processes`,
`--values`, `--min-time` and `--loops` override them per call. What neither gives comes from
each benchmark's own `repeat`, `number`, `sample_time` and `warmup_time`.
- `collectors.deps`: `describe_env.py`, a **context collector**. benchx runs it once per call,
in the environment the call runs in, and it prints the pandas, numpy and pyarrow it imports.
That lands in each result's `environment.system.deps`, which is how the `environments`
comparison knows what differs between the sides.
- `paths`: the store and the results go in `out/`, which `run.sh` clears first, so `bx series`
shows this run only. `$BENCHX_HOME` would relocate the store over this setting, so `run.sh`
unsets it.

Each result also records the installed project's version (`provenance.info.asv.installed`) and
asv's hash of the benchmark's code (`protocol.asv.benchmark_version`), so editing the benchmark
starts a new series.

## Caveats

- **`setup-per-call-unbatched`.** `Methods` has a `setup`, so, as in asv_runner >= 0.3.0, every
timed call sees fresh state: one call per loop, setup re-run between calls outside the timer.
For a ~100 µs call that timing includes some timer overhead, and the result is flagged. A
benchmark whose calls leave the set-up state intact can declare `setup_scope = "sample"`.
- **Memory benchmarks** (`peakmem_*`, `mem_*`) and other non-`time_*` types are listed and
skipped.
- **Unstable boot time.** pyperf's `boot_time` metadata is part of `environment.system`, which
the `environments` profile uses to tell sides apart. On hosts where the kernel's boot time
drifts (seen on WSL2), calls with the same label can then become separate sides, and the
comparison fails the `two-sides` invariant. This is a benchx issue, not specific to asv.

## What is in this directory

| File | Role |
|---|---|
| `run.sh` | the driver: the suite, the venvs, the loop, the comparison |
| `benchx.conf.json` | the project file: the adapter and its precision, the `deps` collector, the paths |
| `describe_env.py` | the collector: prints the pandas, numpy and pyarrow this Python imports |
17 changes: 17 additions & 0 deletions examples/pandas-asv/benchx.conf.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"schema_version": "benchx/config/0.1.0",
"project": "pandas",
"adapter": {
"name": "asv",
"precision": {
"repetitions": { "mode": "fixed", "levels": [{ "unit": "process", "n": 2 }, { "unit": "value", "n": 5 }] },
"warmup": { "mode": "count", "n_warmup": 1 }
}
},
"collectors": {
"deps": {
"command": ["python", "{root}/describe_env.py"]
}
},
"paths": { "store": "out/store.parquet", "results": "out" }
}
16 changes: 16 additions & 0 deletions examples/pandas-asv/describe_env.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
"""The collector: print the pandas this Python imports, and the libraries that decide how
fast it is.

benchx.conf.json runs it once per call, in the environment the call runs in, so the two sides
can only differ here if their environments do. It prints the same text for the same
environment: no paths, no timestamps.
"""

import json

import numpy
import pandas
import pyarrow

print(json.dumps({"summary": {"pandas": pandas.__version__, "numpy": numpy.__version__,
"pyarrow": pyarrow.__version__}}, sort_keys=True))
100 changes: 100 additions & 0 deletions examples/pandas-asv/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
#!/usr/bin/env bash
# One benchmark from pandas' own asv suite, run unchanged by `bx bench PATH --adapter asv` (asv's
# benchmark format, timed by pyperf) under two pandas versions, each in its own venv, alternated
# round by round, then compared. See README.md for what each step shows.
#
# Needs `uv` and git. Both venvs get the benchx of the active environment: its checkout if it is
# an editable install, else the same version from PyPI (the latest if there is none). Set
# BENCHX_SRC to a benchx checkout (the directory with its pyproject.toml) to install that instead.
# Everything it creates is here: pandas/, .venv-a, .venv-b and out/ (git-ignored).
# benchx builds nothing, and neither asv nor asv_runner is installed.
set -euo pipefail
cd "$(dirname "$0")"

# Where the venvs get benchx from: BENCHX_SRC, else the checkout of an editable benchx in the
# active environment, else PyPI, at the active environment's version if it has one.
BENCHX_SRC=${BENCHX_SRC:-}
if [ -z "$BENCHX_SRC" ]; then
BENCHX_SRC=$(python - 2>/dev/null <<'EOF' || true
import importlib.metadata as md, json, urllib.parse
url = {}
for name in ("benchx", "benchx-prototype"): # older checkouts install benchx-prototype
try:
url = json.loads(md.distribution(name).read_text("direct_url.json") or "{}")
break
except md.PackageNotFoundError:
pass
if url.get("dir_info", {}).get("editable"):
print(urllib.parse.unquote(urllib.parse.urlparse(url["url"]).path))
EOF
)
fi
if [ -n "$BENCHX_SRC" ]; then
BENCHX_REQ=(-e "$BENCHX_SRC")
else
BENCHX_VERSION=$(python -c 'import importlib.metadata as m; print(m.version("benchx"))' 2>/dev/null || true)
BENCHX_REQ=("benchx${BENCHX_VERSION:+==$BENCHX_VERSION}")
fi

# The store is out/store.parquet (benchx.conf.json). $BENCHX_HOME would relocate it, and mix
# this run's series with whatever else is stored there.
unset BENCHX_HOME

# The suite is pandas' asv_bench at one tag, the same for both sides: only the installed pandas
# differs. The case is one parameter combination of one benchmark: `str` is object dtype in
# pandas 2.x, and the pyarrow-backed string dtype from pandas 3.0.
PANDAS_REF=v3.1.1
PANDAS_A=2.3.3
PANDAS_B=3.1.1
CASE="strings.Methods.time_lower('str')"
PYTHON_VERSION=3.12
ROUNDS=3 # the fewest that give a verdict
SIDES=("a $PANDAS_A" "b $PANDAS_B")
RUN_KEY=pandas-$(date +%s)
SLOT=0

echo "== 1. fetch pandas' asv suite at $PANDAS_REF (asv_bench/ only)"
if [ "$(git -C pandas describe --tags --exact-match 2>/dev/null)" != "$PANDAS_REF" ]; then
rm -rf pandas
git clone -q --depth 1 --branch "$PANDAS_REF" --filter=blob:none --sparse \
https://github.com/pandas-dev/pandas.git pandas 2>/dev/null
git -C pandas sparse-checkout set asv_bench
fi
echo " pandas/asv_bench at $(git -C pandas rev-parse --short HEAD)"

rm -rf out .venv-a .venv-b
mkdir -p out

echo "== 2. the user prepares both environments; benchx builds nothing (benchx: ${BENCHX_REQ[*]})"
for SIDE_VERSION in "${SIDES[@]}"; do
read -r SIDE VERSION <<<"$SIDE_VERSION"
uv venv -q --python "$PYTHON_VERSION" ".venv-$SIDE"
uv pip install -q --python ".venv-$SIDE/bin/python" "pandas==$VERSION" pyarrow pyperf "${BENCHX_REQ[@]}"
echo " .venv-$SIDE: pandas $VERSION"
done

echo "== 3. the caller's loop: alternate the sides, round by round, one slot per call"
for ROUND in $(seq 0 $((ROUNDS - 1))); do
for SIDE_VERSION in "${SIDES[@]}"; do
read -r SIDE VERSION <<<"$SIDE_VERSION"
# Activating a venv replaces the one activated before; no deactivate is needed.
source ".venv-$SIDE/bin/activate"
# Modules the venv cannot import (no matplotlib, scipy, ...) are listed on stderr and skipped.
bx bench pandas/asv_bench --adapter asv "$CASE" \
--run-key "$RUN_KEY" --label "env=pandas-$VERSION" \
--round "$ROUND" --slot "$SLOT" --out "out/$RUN_KEY" | tee "out/bench-$SLOT.txt"
echo " round $ROUND, slot $SLOT: side $SIDE"
SLOT=$((SLOT + 1))
done
done

echo "== 4. compare: the sides differ in the pandas the collector read"
bx compare --run "$RUN_KEY" --profile environments --label env \
--baseline "pandas-$PANDAS_A" | tee out/compare.txt
if grep -q 'failed invariants' out/compare.txt; then echo "run.sh: the comparison was refused" >&2; exit 1; fi
if ! grep -q 'differs:.*deps\.pandas' out/compare.txt; then # the differing fact is shown
echo "run.sh: the comparison does not name deps.pandas as differing" >&2; exit 1
fi

echo "== 5. the series: one per environment, each point at the suite's revision"
bx series
Loading