From 59b11ca00092a5b53a6cd4ffc1a8ccb96bf4f025 Mon Sep 17 00:00:00 2001 From: Matthew J Mucklo Date: Sun, 12 Jul 2026 19:11:26 -0700 Subject: [PATCH 1/3] Establish a benchmark baseline with regression comparison MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds tooling to record a tagged PhpBench baseline and diff runs against it: - composer bench:baseline — record/overwrite the local `baseline` tag - composer bench:compare — run and diff each subject vs the baseline Stabilizes the runner via phpbench.json (5 iterations, 2 warmup, 5% retry threshold), dropping rstdev from ~25% to <4%. Iteration/warmup counts were being overridden by @Iterations/@Warmup annotations in ParseBench.php; removed them so phpbench.json is the single source of truth. Local storage (.phpbench/) is git-ignored — wall-clock times are machine- and PHP-specific and not portable. benchmarks/BASELINE.md documents the workflow, reference figures, and host context. ROADMAP updated; CI-side baselining left as a follow-up. --- .gitignore | 3 +++ ROADMAP.md | 4 +++- benchmarks/BASELINE.md | 46 +++++++++++++++++++++++++++++++++++++++ benchmarks/ParseBench.php | 5 +++-- composer.json | 2 ++ phpbench.json | 5 +++-- 6 files changed, 60 insertions(+), 5 deletions(-) create mode 100644 benchmarks/BASELINE.md diff --git a/.gitignore b/.gitignore index 07a7323..4e11f6b 100644 --- a/.gitignore +++ b/.gitignore @@ -20,3 +20,6 @@ coverage.xml infection.log infection-summary.log .infection-tmp/ + +# phpbench local storage (machine-specific wall-clock times; not portable) +.phpbench/ diff --git a/ROADMAP.md b/ROADMAP.md index 9afadc3..4eaf0fd 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -92,7 +92,9 @@ Not tied to a specific release; picked up as time allows. **Performance:** - [x] PhpBench suite — `benchmarks/ParseBench.php` covers single ASCII, name-addr, UTF-8 local-part, IDN, obs-route, 10-address comma batch, 100-address `parseStream` batch, invalid inputs, and comment extraction. Run with `composer bench`. -- [ ] Profile the state machine under mailing-list-sized inputs. Likely hot path: `mb_substr` in the main loop — investigate byte iteration for pure-ASCII inputs. +- [x] Benchmark baseline + regression comparison — `composer bench:baseline` records a tagged reference (5 iterations, 5% retry threshold for stable numbers); `composer bench:compare` diffs a run against it. Reference figures and host context in `benchmarks/BASELINE.md`. Local storage (`.phpbench/`) is git-ignored since wall-clock times are machine-specific. +- [ ] Wire `bench:compare` into CI on the standard runner (record a runner-local baseline artifact) so regressions are flagged automatically. +- [ ] Profile the state machine under mailing-list-sized inputs. Likely hot path: `mb_substr` in the main loop — investigate byte iteration for pure-ASCII inputs (baseline in `benchmarks/BASELINE.md` shows ≈134 μs/addr at batch scale). **Community / documentation:** - [x] `CONTRIBUTING.md` — dev setup, all `composer` scripts, test-case guidance, code-style rules, RFC citation expectations. diff --git a/benchmarks/BASELINE.md b/benchmarks/BASELINE.md new file mode 100644 index 0000000..afa8bc6 --- /dev/null +++ b/benchmarks/BASELINE.md @@ -0,0 +1,46 @@ +# Benchmark Baseline + +Reference numbers for the PhpBench suite (`benchmarks/ParseBench.php`), used to +catch performance regressions across changes. **Absolute times are hardware- and +PHP-version-specific** — treat the shape and relative cost of the subjects as the +signal, not the raw microseconds, and always compare against a baseline recorded +on the *same* machine. + +## How to use + +```bash +composer bench # run the suite, print a report (no comparison) +composer bench:baseline # record/overwrite the local `baseline` tag +composer bench:compare # run the suite and diff each subject vs `baseline` +``` + +The baseline is stored in `.phpbench/` (git-ignored — machine-specific wall-clock +times are not portable). Re-record it on your machine before relying on +`bench:compare`. Runner settings (iterations, warmup, retry threshold) live in +`phpbench.json`; per-subject rev counts in the benchmark class. + +## Reference run + +Recorded values below are for context only — regenerate locally to compare. + +- **Host:** Intel Core i7-8550U @ 1.80GHz, Linux 6.18 (WSL2) +- **PHP:** 8.1.34 · **PhpBench:** 1.4.3 +- **Commit:** `1073d4c` (v3.3.2) +- **Config:** 1000 revs, 5 iterations, 2 warmup, 5% retry threshold + +| Subject | Mode (time/parse) | rstdev | +|---|---|---| +| `benchSimpleAsciiAddress` | 103.7 μs | ±1.4% | +| `benchSimpleAsciiAddressArrayApi` | 91.0 μs | ±2.5% | +| `benchNameAddr` | 137.3 μs | ±2.1% | +| `benchUtf8LocalPart` | 130.4 μs | ±2.0% | +| `benchIdnDomain` | 119.2 μs | ±2.3% | +| `benchObsRoute` | 108.7 μs | ±1.2% | +| `benchInvalidAddress` | 71.1 μs | ±1.7% | +| `benchCommentExtraction` | 152.5 μs | ±2.8% | +| `benchBatch10Comma` | 682.0 μs | ±1.2% (≈68 μs/addr) | +| `benchBatch100StreamCount` | 13,414.9 μs | ±3.5% (≈134 μs/addr) | + +The per-address cost at batch scale (`benchBatch100StreamCount`) is dominated by +the main-loop `mb_substr($emails, $i, 1, $encoding)` call — see the ROADMAP +performance item on a pure-ASCII byte-iteration fast path. diff --git a/benchmarks/ParseBench.php b/benchmarks/ParseBench.php index 4c7b0f3..08ef848 100644 --- a/benchmarks/ParseBench.php +++ b/benchmarks/ParseBench.php @@ -15,10 +15,11 @@ * Use these to catch regressions across releases; absolute throughput * depends on the host PHP version and hardware. * + * Iteration/warmup/retry counts are governed centrally by phpbench.json so the + * baseline stays consistent across all subjects. + * * @BeforeMethods({"setUp"}) * @Revs(1000) - * @Iterations(3) - * @Warmup(1) */ class ParseBench { diff --git a/composer.json b/composer.json index 58b7380..c4e96d3 100644 --- a/composer.json +++ b/composer.json @@ -63,6 +63,8 @@ "psalm": "psalm --memory-limit=512M", "infect": "XDEBUG_MODE=coverage infection --threads=max", "bench": "phpbench run --report=default", + "bench:baseline": "phpbench run --store --tag=baseline --progress=none", + "bench:compare": "phpbench run --ref=baseline --report=default", "ci": [ "@cs:check", "@stan", diff --git a/phpbench.json b/phpbench.json index 7980d04..f9e1f11 100644 --- a/phpbench.json +++ b/phpbench.json @@ -3,7 +3,8 @@ "runner.bootstrap": "autoload.php.dist", "runner.path": "benchmarks", "runner.file_pattern": "*Bench.php", - "runner.iterations": 3, + "runner.iterations": 5, "runner.revs": 1000, - "runner.warmup": 1 + "runner.warmup": 2, + "runner.retry_threshold": 5.0 } From 829d4417d2624d3181e2f71bf061812671677885 Mon Sep 17 00:00:00 2001 From: Matthew J Mucklo Date: Sun, 12 Jul 2026 22:40:25 -0700 Subject: [PATCH 2/3] CI: add non-blocking benchmark comparison job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records a baseline from the PR base's src/ using this PR's benchmark harness, then compares the head against it on the same runner — the only apples-to-apples comparison for machine-dependent wall-clock times. Runs on pull_request only. Non-blocking (continue-on-error) with a generous assertion: it flags only catastrophic regressions (>50% slower per subject) to tolerate shared-runner noise, and never blocks a merge. Surfaces the per-subject comparison table in the job log. --- .github/workflows/ci.yml | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f2cccd3..2a29f45 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -155,3 +155,39 @@ jobs: - name: Run Psalm run: bin/psalm --memory-limit=512M --output-format=github + + benchmarks: + name: Benchmarks (vs base) + runs-on: ubuntu-latest + # Non-blocking: shared runners are noisy, so this reports a comparison and + # only flags catastrophic regressions (>50% slower) — it never blocks a PR. + if: github.event_name == 'pull_request' + continue-on-error: true + + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 # need the base commit to benchmark against + + - name: Setup PHP + uses: shivammathur/setup-php@v2 + with: + php-version: '8.3' + extensions: mbstring, intl + coverage: none + + - name: Install dependencies + run: composer install --prefer-dist --no-progress + + # Record the baseline from the PR base's library code, using this PR's + # benchmark harness/config, on this one runner — the only apples-to-apples + # comparison for machine-dependent wall-clock times. + - name: Record baseline from base commit + run: | + git checkout ${{ github.event.pull_request.base.sha }} -- src/ + composer bench:baseline + git checkout HEAD -- src/ + + - name: Compare head against baseline + run: composer bench:compare -- --assert="mode(variant.time.avg) <= 1.5 * mode(baseline.time.avg)" From 089d4c355510205f9d8ea4150d9c957605a207b5 Mon Sep 17 00:00:00 2001 From: Matthew J Mucklo Date: Sun, 12 Jul 2026 22:40:48 -0700 Subject: [PATCH 3/3] docs: mark bench CI integration done in ROADMAP --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 4eaf0fd..3c907a8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -93,7 +93,7 @@ Not tied to a specific release; picked up as time allows. **Performance:** - [x] PhpBench suite — `benchmarks/ParseBench.php` covers single ASCII, name-addr, UTF-8 local-part, IDN, obs-route, 10-address comma batch, 100-address `parseStream` batch, invalid inputs, and comment extraction. Run with `composer bench`. - [x] Benchmark baseline + regression comparison — `composer bench:baseline` records a tagged reference (5 iterations, 5% retry threshold for stable numbers); `composer bench:compare` diffs a run against it. Reference figures and host context in `benchmarks/BASELINE.md`. Local storage (`.phpbench/`) is git-ignored since wall-clock times are machine-specific. -- [ ] Wire `bench:compare` into CI on the standard runner (record a runner-local baseline artifact) so regressions are flagged automatically. +- [x] Wire `bench:compare` into CI — a non-blocking `benchmarks` job records a baseline from the PR base's `src/` and compares the head against it on the same runner. Generous 50%-regression assertion (shared runners are noisy) and `continue-on-error`, so it reports without blocking. - [ ] Profile the state machine under mailing-list-sized inputs. Likely hot path: `mb_substr` in the main loop — investigate byte iteration for pure-ASCII inputs (baseline in `benchmarks/BASELINE.md` shows ≈134 μs/addr at batch scale). **Community / documentation:**