feat(sleep): paired A/B evalkit with McNemar and bootstrap CIs

CHANGELOG.md

@@ -7,6 +7,11 @@ All notable changes to SkillOpt are documented here. This project adheres to
 ## [Unreleased]
 
 ### Added
+- **SkillOpt-Sleep paired A/B evalkit** (`python -m skillopt_sleep.evalkit`):
+  McNemar plus percentile-bootstrap CIs on a fixed task manifest, with
+  task-cluster multi-seed inference and seeded null calibration for exact-test
+  type-I error and bootstrap coverage. The nightly gate is unchanged
+  (thanks @bogdanbaciu21).
 - **SkillOpt-Sleep multi-skill fan-out and reviewed subset adoption**: each
   hinted skill is consolidated from its own pinned live baseline, staged as an
   independent proposal with per-skill gate evidence, and promoted only through

docs/reference/cli.md

@@ -123,8 +123,12 @@ skillopt-sleep <action> [options]
 python -m skillopt_sleep <action> [options]
 ```
 
-Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`, and
-`unschedule`. Common options include:
+Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`,
+`unschedule`, and `evalkit`. `evalkit` is also available as
+`python -m skillopt_sleep.evalkit` and compares two conditions on one fixed
+task manifest (McNemar + bootstrap CI). Exactly one of its `--b` comparison
+input or `--aa` identity-check flag is required. See `docs/sleep/evalkit.md`.
+Common options for the nightly actions include:
 
 | Argument | Description |
 |---|---|

docs/sleep/README.md

@@ -339,6 +339,21 @@ correctness signal; the validation gate still governs what ships.
 | `recall_k` | `0` | Associative recall — pull the K most-similar past tasks (from a persisted archive) into tonight's dream. |
 | `dream_factor` | `0` | Add N lightweight synthetic variants of each task. |
 
+### Paired A/B evalkit
+
+Reports and PRs that claim "B beats A" should go through the shared evalkit
+rather than quoting a single-run cell. One command pairs two conditions on one
+fixed task manifest, runs McNemar's test, and reports a bootstrap CI on the
+success-rate delta. The nightly gate is unchanged.
+
+```text
+python -m skillopt_sleep.evalkit --manifest tasks.json --a cond_a.json --b cond_b.json
+```
+
+See [`evalkit.md`](evalkit.md) for the id-set contract, task-cluster inference
+over positional seed repeats, A/A checks, and the point-estimate-only published
+RESULTS cell replay.
+
 ## Results
 
 > 📊 **More results & analysis — the gate-safety stress test, experience-replay

docs/sleep/evalkit.md

@@ -0,0 +1,77 @@
+# Paired A/B evalkit
+
+Sleep contributors have a shared instrument for "condition B beats condition A":
+
+```text
+python -m skillopt_sleep.evalkit --manifest tasks.json --a cond_a.json --b cond_b.json
+```
+
+The kit pairs outcomes by task id, runs McNemar's test on binary successes, and
+reports a percentile-bootstrap confidence interval on the success-rate delta.
+It does not change the nightly gate.
+
+## Inputs
+
+- `--manifest`: JSON list of task ids, or `{"ids": [...]}` / `{"tasks": [{"id": ...}]}`.
+  Every id must be a non-empty JSON string; ids are never coerced from numbers,
+  booleans, nulls, arrays, or objects.
+- `--a` / `--b`: JSON objects mapping those same ids to `0`/`1` (or an ordered
+  list of repeated-seed `0`/`1` values). A wrapper `{"outcomes": {...}}` is also
+  accepted. Direct keys that exactly match the manifest take precedence over
+  wrapper detection, so a task literally named `outcomes` remains unambiguous.
+- `--aa`: A/A identity smoke check (reuses `--a` as both conditions).
+  Exactly one of `--b` or `--aa` is required.
+- `--allow-graded`: permit non-binary scores. McNemar is omitted; bootstrap only.
+- `--boot`, `--seed`, `--alpha`, `--json`.
+
+The id sets of the manifest, A, and B must be identical. Cross-manifest
+comparisons and duplicate JSON object keys are refused. Seed lists must be
+non-empty. Scores must be JSON numbers (never booleans or numeric strings),
+finite, and in `[0, 1]`. `alpha` must be strictly between 0 and 1, and `--boot`
+must be between 1 and 1,000,000. The total bootstrap workload is capped at
+50,000,000 paired draws (`n_tasks * n_boot`). Exact McNemar evaluation is capped
+at 1,000,000 discordant pairs and accumulates its tail from a bounded-memory
+stream. JSON parsing is strict throughout the document, including metadata, and
+rejects `NaN`/`Infinity` rather than silently accepting non-standard constants.
+
+## Multi-seed
+
+When each task maps to a same-length list of seed repeats, the lists are
+positional: A and B must use the same seed ordering. The JSON report calls each
+position `seed_index`; it does not claim to verify the underlying RNG seed id.
+The kit:
+
+1. averages per task across seeds for the headline delta and bootstrap CI
+2. resamples whole tasks, preserving the task as the independent cluster
+3. omits McNemar rather than treating repeated seeds as independent samples
+4. publishes the per-position deltas plus their mean and sample sd as a
+   descriptive diagnostic only; the sd is not a confidence interval, standard
+   error, or other inferential uncertainty estimate
+
+That is the house answer to single-seed noise (see issue #108 and the
+single-seed warning in `RESULTS.md`).
+
+## RESULTS cell replay
+
+`tests/fixtures/evalkit/results_searchqa_nano_gated.json` replays the published
+SearchQA / GPT-5.4-nano / gated / cumulative nights=5 cell (baseline 0.560,
+after 0.679, Δ +11.9 on n=1400). Per-task pairs were not published, so the
+replay uses a documented maximum-concordance reconstruction: the first
+`round(n * rate)` tasks succeed in each condition. The harness recovers the
+published point delta only. It does not claim to recover the original microdata,
+and p-values or confidence intervals from the reconstructed pairs must not be
+cited as evidence for the published experiment.
+
+## A/A checks
+
+```text
+python -m skillopt_sleep.evalkit --manifest tests/fixtures/evalkit/aa_manifest.json \
+  --a tests/fixtures/evalkit/aa_outcomes.json --aa
+```
+
+The command above is an identity smoke check: identical conditions must report
+delta 0, McNemar p_exact = 1, and a CI that includes 0. The test suite separately
+runs seeded null simulations with genuine discordant pairs, bounds the empirical
+McNemar type-I-error rate on both sides of its nominal level, and checks paired
+bootstrap coverage. These are seeded null calibration checks. Comparing one
+array with itself is not presented as a statistical calibration.

mkdocs.yml

@@ -50,6 +50,7 @@ nav:
     - Deep Learning Analogy: guide/dl-analogy.md
   - SkillOpt-Sleep:
     - Overview: sleep/README.md
+    - Paired A/B Evalkit: sleep/evalkit.md
     - Multi-skill Staging: sleep/multi-skill-staging.md
     - OpenAI-compatible Endpoints: sleep/openai-compatible-endpoints.md
     - Results: sleep/RESULTS.md

skillopt_sleep/__main__.py

@@ -882,6 +882,19 @@ def main(argv=None) -> int:
     p_unsched = sub.add_parser("unschedule", help="remove the nightly cron entry")
     _add_common(p_unsched)
     p_unsched.add_argument("--all", action="store_true", help="remove all managed entries")
+    p_eval = sub.add_parser(
+        "evalkit",
+        help="paired A/B comparison (McNemar + bootstrap CI)",
+    )
+    p_eval.add_argument("--manifest", required=True)
+    p_eval.add_argument("--a", required=True)
+    p_eval.add_argument("--b", default=None, help="required unless --aa")
+    p_eval.add_argument("--aa", action="store_true", help="mutually exclusive with --b")
+    p_eval.add_argument("--alpha", type=float, default=0.05)
+    p_eval.add_argument("--boot", type=int, default=10000)
+    p_eval.add_argument("--seed", type=int, default=42)
+    p_eval.add_argument("--allow-graded", action="store_true")
+    p_eval.add_argument("--json", action="store_true")
 
     args = parser.parse_args(argv)
     if args.cmd == "run":
@@ -898,6 +911,19 @@ def main(argv=None) -> int:
         return cmd_schedule(args)
     if args.cmd == "unschedule":
         return cmd_unschedule(args)
+    if args.cmd == "evalkit":
+        from skillopt_sleep.evalkit import main as evalkit_main
+        argv = ["--manifest", args.manifest, "--a", args.a]
+        if args.b is not None:
+            argv.extend(["--b", args.b])
+        if args.aa:
+            argv.append("--aa")
+        argv.extend(["--alpha", str(args.alpha), "--boot", str(args.boot), "--seed", str(args.seed)])
+        if args.allow_graded:
+            argv.append("--allow-graded")
+        if args.json:
+            argv.append("--json")
+        return evalkit_main(argv)
     parser.print_help()
     return 2
 

skillopt_sleep/adapters/superpowers.py

@@ -717,7 +717,7 @@ def _run_scenario(
             if p.is_symlink():
                 raise ValueError(f"Refusing symlinked overlay path: {p}")
         skill_dir.mkdir(parents=True, exist_ok=True)
-        if not skill_dir.resolve().is_relative_to(workspace):
+        if not skill_dir.resolve().is_relative_to(workspace.resolve()):
             raise ValueError(f"Skill path {skill_dir} escapes workspace {workspace}")
         shutil.copy2(skill_overlay, skill_dest, follow_symlinks=False)