test(bench): verbose aksops output — the mock wasn't the confound

docs/superpowers/specs/2026-07-06-benchmark-findings.md

@@ -37,6 +37,20 @@ study finds **no case where it clearly wins** — code-edit shapes are neutral-t
 ops shape came out ~neutral once measured reliably. A win, if it exists, likely needs workflows with
 *much* larger per-episode work that a compiled action can wholesale replace (not four cheap commands).
 
+**The structural reason (the deepest finding): the token cost is not where JIT optimizes.** Every episode
+costs ~250–330k tokens, and that is dominated by **fixed agent overhead** — the system prompt, tool
+definitions, and the context re-read every turn via cache (~44k cache-creation just to *boot* a session,
+then cache-reads compounding across turns). The *task-specific* work — a code edit, four commands, even
+17KB of verbose command output (tested: `az`/`kubectl` mocks emitting a 120-row pod table + JSON + rollout
+logs still moved the result <3%) — is a small slice of the total. A skill can only shortcut that slice, so
+**JIT compilation is structurally limited at saving tokens for Claude-Code-shaped tasks.** To make the
+task-specific portion dominate, the per-episode work would have to be enormous (hundreds of KB of output,
+or very long tool sequences) — not typical.
+
+**Implication:** AgentJIT's real value is more likely **latency and determinism** (a compiled script runs
+in milliseconds and can't hallucinate or skip a step) than raw token savings — a dimension this token
+benchmark does not yet measure. A benchmark v2 should add wall-clock and reliability.
+
 **Lesson baked in:** always compare at iso-accuracy and check the *verified* sample size on both arms — a
 mean over a handful of lucky successes is not a result. (`aj bench --compare` prints an iso-accuracy WARN
 when success rates differ; trust it.)

internal/bench/aksops_fixture.go

@@ -81,11 +81,28 @@ func (f AKSOpsFixture) SeedSessions(logsDir string) error {
 // output shaped like the real CLI. The header comment states plainly that these
 // are the runbook's intended local tools, so the agent runs them rather than
 // second-guessing whether "fake" commands are worth executing.
+//
+// The output is intentionally SIZABLE (JSON blobs, a many-row pod table, rollout
+// progress) to mirror real az/kubectl: each command's output round-trips into the
+// model's context in the baseline arm, which is the per-command cost a compiled
+// runbook skill (run in one shot) might avoid. Near-silent mocks understate that.
 const mockAz = `#!/usr/bin/env bash
 # az — local test double for this ops runbook environment. Records to ops.log.
 echo "az $*" >> ops.log
 if [ "$1 $2" = "aks get-credentials" ]; then
   echo "Merged \"prod\" as current context in ~/.kube/config"
+  cat <<'JSON'
+{
+  "apiVersion": "v1",
+  "clusters": [
+    {"cluster": {"certificate-authority-data": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...", "server": "https://prod-dns-a1b2c3d4.hcp.eastus.azmk8s.io:443"}, "name": "prod"}
+  ],
+  "contexts": [{"context": {"cluster": "prod", "user": "clusterUser_rg_prod"}, "name": "prod"}],
+  "current-context": "prod",
+  "kind": "Config",
+  "users": [{"name": "clusterUser_rg_prod", "user": {"token": "eyJhbGciOiJSUzI1NiIsImtpZCI6..."}}]
+}
+JSON
 else
   echo "{\"status\": \"ok\"}"
 fi
@@ -96,10 +113,23 @@ const mockKubectl = `#!/usr/bin/env bash
 # kubectl — local test double for this ops runbook environment. Records to ops.log.
 echo "kubectl $*" >> ops.log
 case "$1" in
-  scale)   echo "deployment.apps/web scaled" ;;
-  rollout) echo "deployment \"web\" successfully rolled out" ;;
-  get)     printf 'NAME                READY   STATUS    RESTARTS   AGE\nweb-7d9f8c6b5-abcde 1/1     Running   0          5m\n' ;;
-  *)       echo "ok" ;;
+  scale)
+    echo "deployment.apps/web scaled" ;;
+  rollout)
+    echo "Waiting for deployment \"web\" rollout to finish: 0 of 5 updated replicas are available..."
+    echo "Waiting for deployment \"web\" rollout to finish: 1 of 5 updated replicas are available..."
+    echo "Waiting for deployment \"web\" rollout to finish: 2 of 5 updated replicas are available..."
+    echo "Waiting for deployment \"web\" rollout to finish: 3 of 5 updated replicas are available..."
+    echo "Waiting for deployment \"web\" rollout to finish: 4 of 5 updated replicas are available..."
+    echo "deployment \"web\" successfully rolled out" ;;
+  get)
+    echo "NAME                    READY   STATUS    RESTARTS   AGE     IP            NODE                                NOMINATED NODE   READINESS GATES"
+    for i in $(seq 1 120); do
+      printf 'web-7d9f8c6b5-%s   1/1     Running   0          %dm    10.244.%d.%d   aks-nodepool1-24680123-vmss0000%02d   <none>           <none>\n' \
+        "$(printf '%05x' $((RANDOM % 1000000)))" "$i" $((i % 3)) $((i + 10)) $((i % 6))
+    done ;;
+  *)
+    echo "ok" ;;
 esac
 exit 0
 `