docs(benchmarks): blockers accuracy scale — 100 PRs / 10 repos, brick 1.1.0 CLEAN

CHANGELOG.md

@@ -25,6 +25,10 @@ the version on nuget.org, which is why it trails `VERSION` between releases rath
   findings (F1–F3) come from reading required checks and approvals only from admin-only branch protection.
   Harness, snapshot and raw runs in `tools/blockers-accuracy/`. A re-run on a second snapshot after the fix
   (`github.blockers` 1.1.0, #874) found 109 of 109 blockers with no wrong claims; the old brick found 19.
+- `docs/benchmarks/blockers-accuracy-scale.md`: scale pass on 100 pull requests in 10 public repositories
+  (including `IanFrelinger/Ashlar`), brick-raw only, same rubric. Verdicts 100/100, blockers 211/211, no wrong
+  claims, $0 Anthropic spend. `score.py` strips #872 inline-code quotes on check names so brick reasons still
+  match truth keys.
 
 ### Added — token benchmark follow-up arms
 

docs/benchmarks/blockers-accuracy-scale.md

@@ -0,0 +1,95 @@
+# Accuracy scale pass: `github.blockers` 1.1.0 on ~100 PRs
+
+Companion to the [44-PR accuracy study](blockers-accuracy.md). Same rubric (`tools/blockers-accuracy/truth.py`
+and `score.py`), same brick (`github.blockers` **1.1.0**, post-[#874](https://github.com/IanFrelinger/Ashlar/pull/874),
+including the #872 quoting of GitHub-sourced names). Expanded sample and brick-raw grading only.
+
+**Result (CLEAN).**
+
+| Metric | Value |
+|---|---|
+| Pull requests | **100** |
+| Public repositories | **10** (was 5) |
+| Verdicts right | **100 / 100** |
+| Blockers found | **211 / 211** |
+| Overcalls / fabricated / unattributed | **0 / 0 / 0** |
+| Blocked PRs with every blocker named | **64 / 64** |
+| Anthropic spend | **$0** (brick-raw only; Sonnet skipped) |
+
+Read with the same limit as the 44-PR re-run: the brick and the rubric both read required checks and
+`reviewDecision` from the public GitHub sources, so agreement shows the brick applies those facts without
+error on a wider sample. It is not independent evidence that GitHub's merge box would list exactly these
+blockers.
+
+## Sample
+
+Selected 2026-10-25. Keeps the original 44 from five repos and adds 56 more across five further repos,
+including `IanFrelinger/Ashlar` and other .NET-heavy hosts. `selected_as` is a selection label only;
+grading uses the rubric on the capture snapshot.
+
+| Repository | PRs |
+|---|---|
+| IanFrelinger/Ashlar | 19 |
+| AvaloniaUI/Avalonia | 17 |
+| modelcontextprotocol/python-sdk | 11 |
+| dotnet/efcore | 10 |
+| microsoft/semantic-kernel | 9 |
+| cli/cli | 8 |
+| modelcontextprotocol/csharp-sdk | 8 |
+| dotnet/aspnetcore | 8 |
+| dotnet/runtime | 7 |
+| App-vNext/Polly | 3 |
+
+Selection mix (label counts): blocked-failing-ci 14, blocked-review 14, merged 12, conflict 10, behind 9,
+changes-requested 9, draft 8, ready-clean 8, closed 8, unstable 6, blocked-pending-ci 2.
+
+Snapshot truth on capture: 64 blocked, 15 ready, 13 merged, 8 closed. Blocker kinds: `check_pending` 100,
+`review_required` 40, `check_failing` 28, `behind` 14, `conflict` 12, `changes_requested` 9, `draft` 8.
+
+## Method
+
+- **Brick.** Built from this branch's `Ashlar.Mcp.Server.Host` (Release, net10.0) with `github.blockers`
+  1.1.0. Asked through `actions/pr-blockers/pr_blockers.py`, same path as CI capture.
+- **Snapshot.** `tools/blockers-accuracy/capture.py` → `fixtures-scale/`, 2026-10-25 21:57–22:08 UTC.
+  Token: the cloud agent's `gh` token (no admin rights on the foreign repos). Head SHA matched GraphQL and
+  REST on all 100.
+- **Grading.** Brick-raw only: map each blocker reason sentence to rubric keys with `score.py`
+  `BRICK_PATTERNS`, compare to `truth.py`. No Sonnet arm. Brick-raw already covers the full rubric; an LLM
+  answer arm would spend tokens without adding signal (on the 44-PR study, brick+Sonnet matched brick-raw
+  on every PR).
+- **Harness fix in this pass.** After #872, check and status names in brick reasons are one inline code
+  span (`` `name` ``). `score.py` now strips a surrounding backtick pair before comparing to truth keys.
+  Without that, every required-check claim scored as a fabricated name with backticks. Re-grading the
+  post-#874 44-PR snapshot still yields 109/109 after the strip.
+
+## Brick findings
+
+None on this sample. No new product bug to file. Verdict and blocker-list agreement was complete after the
+harness quote strip.
+
+F1–F3 from the 44-PR study remain fixed in 1.1.0 ([#874](https://github.com/IanFrelinger/Ashlar/pull/874)).
+
+## Spend
+
+Hard budget $6. Actual Anthropic spend: **$0**. The key was exported into the environment for a possible
+Sonnet arm, never sent (no `run.py` call), and unset afterward. Capture and scoring used only GitHub and
+local compute.
+
+## Reproduce
+
+```bash
+cd tools/blockers-accuracy
+python3 score.py results/2026-10-25-scale --fixtures fixtures-scale --brick-dir brick
+```
+
+Raw inputs: `tools/blockers-accuracy/fixtures-scale/` (truth, brick 1.1.0, REST). Scores:
+`results/2026-10-25-scale/score.json` and `per-pr.md`. Targets: `targets.json` (100 PRs).
+
+## Limits
+
+Same as the 44-PR study, plus:
+
+- Brick-raw only; model variance and native-arm recall are not re-measured on this sample.
+- Ten public repos with GitHub Actions (or equivalent check contexts). Private repos and other CI hosts are
+  not covered.
+- Snapshot is one moment (2026-10-25 ~22:00 UTC). Re-capture changes truth.

docs/benchmarks/blockers-accuracy.md

@@ -24,6 +24,10 @@ pull request URLs and the cause. All three share one root: the brick learns whic
 required only from branch protection, which needs admin rights, and GitHub publishes the same facts without
 them.
 
+**Scale pass.** The same 1.1.0 brick was graded brick-raw on a third snapshot of **100** pull requests in **10**
+public repositories (including `IanFrelinger/Ashlar` and other .NET-heavy hosts): **100/100** verdicts, **211/211**
+blockers, no wrong claims, **$0** Anthropic spend. See [blockers-accuracy-scale.md](blockers-accuracy-scale.md).
+
 **After the fix.** `github.blockers` 1.1.0 ([#874](https://github.com/IanFrelinger/Ashlar/pull/874)) reads those
 facts from the non-admin sources. It was re-run on a second snapshot of the same 44 pull requests, taken 2026-10-25
 about 18:00 UTC, with the old and new bricks asked back to back:

tools/blockers-accuracy/README.md

@@ -1,20 +1,22 @@
 # Blockers accuracy harness
 
 Grades answers to "what is blocking this PR?" against ground truth computed from GitHub's API at one recorded
-snapshot. Method, rubric and the published result:
-[`docs/benchmarks/blockers-accuracy.md`](../../docs/benchmarks/blockers-accuracy.md).
+snapshot. Method, rubric and the published results:
+[`docs/benchmarks/blockers-accuracy.md`](../../docs/benchmarks/blockers-accuracy.md) (44 PRs) and
+[`docs/benchmarks/blockers-accuracy-scale.md`](../../docs/benchmarks/blockers-accuracy-scale.md) (100 PRs, brick-raw).
 
 | File | What it is |
 |------|------------|
-| `targets.json` | The 44 pull requests and the category each was selected as |
-| `capture.py` | Records the snapshot: GraphQL truth facts, the REST responses, and the brick's answer through `Ashlar.Mcp.Server.Host` (needs a built host, so it runs in CI) |
+| `targets.json` | The pull requests under study and the category each was selected as (100 for the scale pass) |
+| `capture.py` | Records the snapshot: GraphQL truth facts, the REST responses, and the brick's answer through `Ashlar.Mcp.Server.Host` (needs a built host) |
 | `fixtures.py` | Recorded-response store both `run.py` and `score.py` read |
 | `truth.py` | The rubric as code: verdict and blocker keys from the snapshot, and the classification of a wrong claim |
 | `run.py` | Asks Sonnet in the brick and native arms from the recorded data; writes `runs.jsonl` and `meta.jsonl`; stops at `--budget-usd` |
-| `score.py` | Grades the brick payload and both arms' answers; writes `score.json` and `per-pr.md` |
-| `fixtures/truth/`, `fixtures/brick/`, `fixtures/github/` | The snapshot (captured 2026-10-25 16:28–16:34 UTC) |
+| `score.py` | Grades the brick payload and both arms' answers; writes `score.json` and `per-pr.md`. Strips #872 inline-code quotes on check names when mapping brick reasons |
+| `fixtures/truth/`, `fixtures/brick/`, `fixtures/github/` | The first snapshot (captured 2026-10-25 16:28–16:34 UTC) |
 | `fixtures-rerun/` | The second snapshot (2026-10-25, about 18:00 UTC), after the fix in #874: truth, `brick/` (1.1.0), `brick-old/` (1.0.0), REST responses |
-| `results/<date>/` | Raw runs and scores of a published run (`2026-10-25-rerun*` grade the second snapshot) |
+| `fixtures-scale/` | The scale snapshot (2026-10-25 21:57–22:08 UTC): 100 PRs, 10 repos, brick 1.1.0 |
+| `results/<date>/` | Raw runs and scores (`2026-10-25-rerun*` grade the second snapshot; `2026-10-25-scale` grades `fixtures-scale`) |
 
 Re-grading needs no key and no network: `python3 score.py results/2026-10-25`. `score.py --fixtures DIR --brick-dir NAME`
 grades another snapshot or brick build; `run.py --arms brick` runs one arm only; `capture.py --extra NAME=COMMAND`

tools/blockers-accuracy/fixtures-scale/brick/App-vNext_Polly_pull_3288.json

@@ -0,0 +1,59 @@
+{
+ "brickId": "github.blockers",
+ "brickVersion": "1.1.0",
+ "auditId": "7bb9b0fcac384535bc3a921e39eb23de",
+ "summary": "PR #3288 is BLOCKED by 9 blocker(s).",
+ "outputs": {
+  "verdict": "blocked",
+  "blockers": [
+   {
+    "reason": "The required check `CodeQL` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   },
+   {
+    "reason": "The required check `code-ql` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   },
+   {
+    "reason": "The required check `dependency-review` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   },
+   {
+    "reason": "The required check `lint` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   },
+   {
+    "reason": "The required check `macos-latest` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   },
+   {
+    "reason": "The required check `mutation-tests` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   },
+   {
+    "reason": "The required check `ubuntu-latest` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   },
+   {
+    "reason": "The required check `validate-packages` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   },
+   {
+    "reason": "The required check `windows-latest` has not reported on 1316d2576d4f.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   }
+  ],
+  "unknowns": [],
+  "notes": [
+   {
+    "text": "Read 2 check run(s) and 0 commit status(es) on the head commit 1316d2576d4f; none is failing or pending.",
+    "evidence": "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+   }
+  ],
+  "evidence": [
+   "https://github.com/App-vNext/Polly/pull/3288",
+   "https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069"
+  ],
+  "answer": "**What is blocking PR #3288?** PR #3288 is BLOCKED by 9 blocker(s).\n\nBlockers:\n- The required check `CodeQL` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n- The required check `code-ql` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n- The required check `dependency-review` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n- The required check `lint` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n- The required check `macos-latest` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n- The required check `mutation-tests` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n- The required check `ubuntu-latest` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n- The required check `validate-packages` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n- The required check `windows-latest` has not reported on 1316d2576d4f. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n\nNotes:\n- Read 2 check run(s) and 0 commit status(es) on the head commit 1316d2576d4f; none is failing or pending. ([evidence](https://github.com/App-vNext/Polly/commit/1316d2576d4f0f03b0c12e2e14ff7e521d318069))\n\nRead from https://github.com/App-vNext/Polly/pull/3288\n"
+ }
+}

tools/blockers-accuracy/fixtures-scale/brick/App-vNext_Polly_pull_4.json

@@ -0,0 +1,16 @@
+{
+ "brickId": "github.blockers",
+ "brickVersion": "1.1.0",
+ "auditId": "e34e6c04c4a34c2e9e8ac8199e597aae",
+ "summary": "PR #4 is already merged.",
+ "outputs": {
+  "verdict": "merged",
+  "blockers": [],
+  "unknowns": [],
+  "notes": [],
+  "evidence": [
+   "https://github.com/App-vNext/Polly/pull/4"
+  ],
+  "answer": "**What is blocking PR #4?** PR #4 is already merged.\n\nRead from https://github.com/App-vNext/Polly/pull/4\n"
+ }
+}

tools/blockers-accuracy/fixtures-scale/brick/App-vNext_Polly_pull_7.json

@@ -0,0 +1,16 @@
+{
+ "brickId": "github.blockers",
+ "brickVersion": "1.1.0",
+ "auditId": "4e5a9bdfbe894838b44572d2d3aafb3a",
+ "summary": "PR #7 is already merged.",
+ "outputs": {
+  "verdict": "merged",
+  "blockers": [],
+  "unknowns": [],
+  "notes": [],
+  "evidence": [
+   "https://github.com/App-vNext/Polly/pull/7"
+  ],
+  "answer": "**What is blocking PR #7?** PR #7 is already merged.\n\nRead from https://github.com/App-vNext/Polly/pull/7\n"
+ }
+}

tools/blockers-accuracy/fixtures-scale/brick/AvaloniaUI_Avalonia_pull_21713.json

@@ -0,0 +1,77 @@
+{
+ "brickId": "github.blockers",
+ "brickVersion": "1.1.0",
+ "auditId": "a4ff34f4460e43c3b82ca8a9cd73ecec",
+ "summary": "PR #21713 is BLOCKED by 7 blocker(s).",
+ "outputs": {
+  "verdict": "blocked",
+  "blockers": [
+   {
+    "reason": "The pull request is a draft.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/pull/21713"
+   },
+   {
+    "reason": "The branch is behind `main`, and the base branch requires branches to be up to date.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/pull/21713"
+   },
+   {
+    "reason": "GitHub reports that an approving review is required (review decision REVIEW_REQUIRED); the pull request has 0 approving review(s).",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/pull/21713"
+   },
+   {
+    "reason": "The required check `Avalonia - Integration Tests` has not reported on f1c968897432.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65"
+   },
+   {
+    "reason": "The required check `Linux` has not reported on f1c968897432.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65"
+   },
+   {
+    "reason": "The required check `Windows` has not reported on f1c968897432.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65"
+   },
+   {
+    "reason": "The required check `macOS` has not reported on f1c968897432.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65"
+   }
+  ],
+  "unknowns": [],
+  "notes": [
+   {
+    "text": "The check `AvaloniaUI.Avalonia` concluded failure. It is not required, so it does not block the merge.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/runs/85510935776"
+   },
+   {
+    "text": "The check `AvaloniaUI.Avalonia (macOS)` concluded failure. It is not required, so it does not block the merge.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/runs/85510933992"
+   },
+   {
+    "text": "The check `AvaloniaUI.Avalonia (Windows)` concluded failure. It is not required, so it does not block the merge.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/runs/85510867804"
+   },
+   {
+    "text": "The check `AvaloniaUI.Avalonia (Linux)` concluded failure. It is not required, so it does not block the merge.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/runs/85508955891"
+   },
+   {
+    "text": "The check `WIP` is still in progress. It is not required, so it does not block the merge.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/runs/85508132960"
+   },
+   {
+    "text": "The check `pull_request_label` is still in progress. It is not required, so it does not block the merge.",
+    "evidence": "https://github.com/AvaloniaUI/Avalonia/runs/85508132266"
+   }
+  ],
+  "evidence": [
+   "https://github.com/AvaloniaUI/Avalonia/pull/21713",
+   "https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65",
+   "https://github.com/AvaloniaUI/Avalonia/runs/85510935776",
+   "https://github.com/AvaloniaUI/Avalonia/runs/85510933992",
+   "https://github.com/AvaloniaUI/Avalonia/runs/85510867804",
+   "https://github.com/AvaloniaUI/Avalonia/runs/85508955891",
+   "https://github.com/AvaloniaUI/Avalonia/runs/85508132960",
+   "https://github.com/AvaloniaUI/Avalonia/runs/85508132266"
+  ],
+  "answer": "**What is blocking PR #21713?** PR #21713 is BLOCKED by 7 blocker(s).\n\nBlockers:\n- The pull request is a draft. ([evidence](https://github.com/AvaloniaUI/Avalonia/pull/21713))\n- The branch is behind `main`, and the base branch requires branches to be up to date. ([evidence](https://github.com/AvaloniaUI/Avalonia/pull/21713))\n- GitHub reports that an approving review is required (review decision REVIEW_REQUIRED); the pull request has 0 approving review(s). ([evidence](https://github.com/AvaloniaUI/Avalonia/pull/21713))\n- The required check `Avalonia - Integration Tests` has not reported on f1c968897432. ([evidence](https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65))\n- The required check `Linux` has not reported on f1c968897432. ([evidence](https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65))\n- The required check `Windows` has not reported on f1c968897432. ([evidence](https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65))\n- The required check `macOS` has not reported on f1c968897432. ([evidence](https://github.com/AvaloniaUI/Avalonia/commit/f1c968897432ae3651a093e197b44d629b445f65))\n\nNotes:\n- The check `AvaloniaUI.Avalonia` concluded failure. It is not required, so it does not block the merge. ([evidence](https://github.com/AvaloniaUI/Avalonia/runs/85510935776))\n- The check `AvaloniaUI.Avalonia (macOS)` concluded failure. It is not required, so it does not block the merge. ([evidence](https://github.com/AvaloniaUI/Avalonia/runs/85510933992))\n- The check `AvaloniaUI.Avalonia (Windows)` concluded failure. It is not required, so it does not block the merge. ([evidence](https://github.com/AvaloniaUI/Avalonia/runs/85510867804))\n- The check `AvaloniaUI.Avalonia (Linux)` concluded failure. It is not required, so it does not block the merge. ([evidence](https://github.com/AvaloniaUI/Avalonia/runs/85508955891))\n- The check `WIP` is still in progress. It is not required, so it does not block the merge. ([evidence](https://github.com/AvaloniaUI/Avalonia/runs/85508132960))\n- The check `pull_request_label` is still in progress. It is not required, so it does not block the merge. ([evidence](https://github.com/AvaloniaUI/Avalonia/runs/85508132266))\n\nRead from https://github.com/AvaloniaUI/Avalonia/pull/21713\n"
+ }
+}