Establish a benchmark baseline with regression comparison

.github/workflows/ci.yml

@@ -155,3 +155,39 @@ jobs:
 
       - name: Run Psalm
         run: bin/psalm --memory-limit=512M --output-format=github
+
+  benchmarks:
+    name: Benchmarks (vs base)
+    runs-on: ubuntu-latest
+    # Non-blocking: shared runners are noisy, so this reports a comparison and
+    # only flags catastrophic regressions (>50% slower) — it never blocks a PR.
+    if: github.event_name == 'pull_request'
+    continue-on-error: true
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # need the base commit to benchmark against
+
+      - name: Setup PHP
+        uses: shivammathur/setup-php@v2
+        with:
+          php-version: '8.3'
+          extensions: mbstring, intl
+          coverage: none
+
+      - name: Install dependencies
+        run: composer install --prefer-dist --no-progress
+
+      # Record the baseline from the PR base's library code, using this PR's
+      # benchmark harness/config, on this one runner — the only apples-to-apples
+      # comparison for machine-dependent wall-clock times.
+      - name: Record baseline from base commit
+        run: |
+          git checkout ${{ github.event.pull_request.base.sha }} -- src/
+          composer bench:baseline
+          git checkout HEAD -- src/
+
+      - name: Compare head against baseline
+        run: composer bench:compare -- --assert="mode(variant.time.avg) <= 1.5 * mode(baseline.time.avg)"

.gitignore

@@ -20,3 +20,6 @@ coverage.xml
 infection.log
 infection-summary.log
 .infection-tmp/
+
+# phpbench local storage (machine-specific wall-clock times; not portable)
+.phpbench/

ROADMAP.md

@@ -92,7 +92,9 @@ Not tied to a specific release; picked up as time allows.
 
 **Performance:**
 - [x] PhpBench suite — `benchmarks/ParseBench.php` covers single ASCII, name-addr, UTF-8 local-part, IDN, obs-route, 10-address comma batch, 100-address `parseStream` batch, invalid inputs, and comment extraction. Run with `composer bench`.
-- [ ] Profile the state machine under mailing-list-sized inputs. Likely hot path: `mb_substr` in the main loop — investigate byte iteration for pure-ASCII inputs.
+- [x] Benchmark baseline + regression comparison — `composer bench:baseline` records a tagged reference (5 iterations, 5% retry threshold for stable numbers); `composer bench:compare` diffs a run against it. Reference figures and host context in `benchmarks/BASELINE.md`. Local storage (`.phpbench/`) is git-ignored since wall-clock times are machine-specific.
+- [x] Wire `bench:compare` into CI — a non-blocking `benchmarks` job records a baseline from the PR base's `src/` and compares the head against it on the same runner. Generous 50%-regression assertion (shared runners are noisy) and `continue-on-error`, so it reports without blocking.
+- [ ] Profile the state machine under mailing-list-sized inputs. Likely hot path: `mb_substr` in the main loop — investigate byte iteration for pure-ASCII inputs (baseline in `benchmarks/BASELINE.md` shows ≈134 μs/addr at batch scale).
 
 **Community / documentation:**
 - [x] `CONTRIBUTING.md` — dev setup, all `composer` scripts, test-case guidance, code-style rules, RFC citation expectations.

benchmarks/BASELINE.md

@@ -0,0 +1,46 @@
+# Benchmark Baseline
+
+Reference numbers for the PhpBench suite (`benchmarks/ParseBench.php`), used to
+catch performance regressions across changes. **Absolute times are hardware- and
+PHP-version-specific** — treat the shape and relative cost of the subjects as the
+signal, not the raw microseconds, and always compare against a baseline recorded
+on the *same* machine.
+
+## How to use
+
+```bash
+composer bench            # run the suite, print a report (no comparison)
+composer bench:baseline   # record/overwrite the local `baseline` tag
+composer bench:compare    # run the suite and diff each subject vs `baseline`
+```
+
+The baseline is stored in `.phpbench/` (git-ignored — machine-specific wall-clock
+times are not portable). Re-record it on your machine before relying on
+`bench:compare`. Runner settings (iterations, warmup, retry threshold) live in
+`phpbench.json`; per-subject rev counts in the benchmark class.
+
+## Reference run
+
+Recorded values below are for context only — regenerate locally to compare.
+
+- **Host:** Intel Core i7-8550U @ 1.80GHz, Linux 6.18 (WSL2)
+- **PHP:** 8.1.34 · **PhpBench:** 1.4.3
+- **Commit:** `1073d4c` (v3.3.2)
+- **Config:** 1000 revs, 5 iterations, 2 warmup, 5% retry threshold
+
+| Subject | Mode (time/parse) | rstdev |
+|---|---|---|
+| `benchSimpleAsciiAddress` | 103.7 μs | ±1.4% |
+| `benchSimpleAsciiAddressArrayApi` | 91.0 μs | ±2.5% |
+| `benchNameAddr` | 137.3 μs | ±2.1% |
+| `benchUtf8LocalPart` | 130.4 μs | ±2.0% |
+| `benchIdnDomain` | 119.2 μs | ±2.3% |
+| `benchObsRoute` | 108.7 μs | ±1.2% |
+| `benchInvalidAddress` | 71.1 μs | ±1.7% |
+| `benchCommentExtraction` | 152.5 μs | ±2.8% |
+| `benchBatch10Comma` | 682.0 μs | ±1.2% (≈68 μs/addr) |
+| `benchBatch100StreamCount` | 13,414.9 μs | ±3.5% (≈134 μs/addr) |
+
+The per-address cost at batch scale (`benchBatch100StreamCount`) is dominated by
+the main-loop `mb_substr($emails, $i, 1, $encoding)` call — see the ROADMAP
+performance item on a pure-ASCII byte-iteration fast path.

benchmarks/ParseBench.php

@@ -15,10 +15,11 @@
  * Use these to catch regressions across releases; absolute throughput
  * depends on the host PHP version and hardware.
  *
+ * Iteration/warmup/retry counts are governed centrally by phpbench.json so the
+ * baseline stays consistent across all subjects.
+ *
  * @BeforeMethods({"setUp"})
  * @Revs(1000)
- * @Iterations(3)
- * @Warmup(1)
  */
 class ParseBench
 {

composer.json

@@ -63,6 +63,8 @@
     "psalm": "psalm --memory-limit=512M",
     "infect": "XDEBUG_MODE=coverage infection --threads=max",
     "bench": "phpbench run --report=default",
+    "bench:baseline": "phpbench run --store --tag=baseline --progress=none",
+    "bench:compare": "phpbench run --ref=baseline --report=default",
     "ci": [
       "@cs:check",
       "@stan",

phpbench.json

@@ -3,7 +3,8 @@
     "runner.bootstrap": "autoload.php.dist",
     "runner.path": "benchmarks",
     "runner.file_pattern": "*Bench.php",
-    "runner.iterations": 3,
+    "runner.iterations": 5,
     "runner.revs": 1000,
-    "runner.warmup": 1
+    "runner.warmup": 2,
+    "runner.retry_threshold": 5.0
 }