Skip to content

fix(bench): reliable aksops → valid iso-accuracy ops result (~neutral) #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Poytr1 merged 2 commits into from
Jul 7, 2026
Merged

fix(bench): reliable aksops → valid iso-accuracy ops result (~neutral) #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
63c88ca
fix(bench): make aksops reliable → valid iso-accuracy ops result
Poytr1 Jul 7, 2026
b489d6d
docs: aksops answered at iso-accuracy — ~neutral, no JIT win
Poytr1 Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 32 additions & 27 deletions docs/superpowers/specs/2026-07-06-benchmark-findings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,24 +16,26 @@
| `nullcheck` | code-edit | hand-written | ~0% (−354 on ~236k), 100%/100% |
| `shellseq` | trivial shell | real `aj compile` | ~0% (−19..35 on ~93k), 100%/100% |
| `migrate` | code-edit (exploration) | hand-written | **−25% (JIT ~48k WORSE)**, 100%/100% |
| `aksops` | SRE / tool-use | real `aj compile` | **inconclusive** — JIT only **33%** success at rollouts=6 |
| `aksops` | SRE / tool-use | real `aj compile` | ~neutral (−5% mean / +7% median), 100%/100% — noisy, no clear win |

**On `aksops` the apparent +14% was a small-sample artifact and is retracted.** At `--rollouts 3` the JIT
arm happened to pass 3/3 and looked like a ~14% saving; at `--rollouts 6` the JIT arm succeeded only
**2/6** (baseline 6/6), so the token "saving" is measured over a *different, smaller* success set — not
iso-accuracy. Per the core methodology, a skill that lowers success rate is just "failing more cheaply,"
which doesn't count. The `aksops` fixture is *flaky for the agent* (it keeps noticing the mock stubs are
fake and sometimes short-circuits), so the ops question is **not yet answered** — it needs a fixture
where both arms succeed reliably.
arm happened to pass 3/3 and looked like a ~14% saving; at `--rollouts 6` it succeeded only **2/6**
(baseline 6/6) — a mean over a *different, smaller* success set, not iso-accuracy (a skill that lowers
success is just "failing more cheaply"). After **hardening the fixture** so both arms succeed 6/6, the
valid iso-accuracy number is **~neutral** (−5% by mean, +7% by median, large baseline variance) — **no
clear win**. So the ops question *is* now answered for this workflow: a compiled skill doesn't clearly pay
off here either.

**Mechanistically** (traced), a **code-edit** skill can only *annotate* — the model still reads and edits
the files itself, so the skill is pure added context cost (net-negative on `migrate`). An **ops /
tool-use** skill *could* pay off because it compiles to a **runnable script** the agent can invoke to do
the work — but we have not yet demonstrated that at iso-accuracy.
tool-use** skill *could* pay off in principle because it compiles to a **runnable script** — but on the
`aksops` runbook it didn't move the needle at iso-accuracy: the agent still spends most of its tokens on
fixed context, and running four short commands (whether by skill or directly) is cheap either way.

**So "repetitive workflow → compile a skill" is not a safe default**, and this study does not (yet) show a
case where it clearly wins. The ops/tool-use direction remains the most promising hypothesis but needs a
reliable fixture to confirm.
**So "repetitive workflow → compile a skill" is not a safe default**, and across these four shapes this
study finds **no case where it clearly wins** — code-edit shapes are neutral-to-negative, and even the
ops shape came out ~neutral once measured reliably. A win, if it exists, likely needs workflows with
*much* larger per-episode work that a compiled action can wholesale replace (not four cheap commands).

**Lesson baked in:** always compare at iso-accuracy and check the *verified* sample size on both arms — a
mean over a handful of lucky successes is not a result. (`aj bench --compare` prints an iso-accuracy WARN
Expand Down Expand Up @@ -153,32 +155,35 @@ cost tends to swamp what it saves — sometimes badly. "Repetitive workflow →
a safe default; the value case has to be targeted much more carefully (bigger per-episode work, or a
skill that genuinely removes reading, not just guides it).

## Update — SRE runbook `aksops` (2026-07-07): inconclusive (a retracted +14%)
## Update — SRE runbook `aksops` (2026-07-07): a retracted +14%, then a valid ~neutral

`aksops` is the shape AgentJIT was designed for: a repetitive **operations runbook** —
`az aks get-credentials` then `kubectl scale` / `rollout status` / `get pods`. It is Bash-shaped, so the
**deterministic** compiler produces a real *runnable* skill script, and the JIT arm can invoke it to *run
the commands* rather than reasoning through each. (`az`/`kubectl` are mocked under `bin/` so the
benchmark is hermetic — no cluster.)

**First reported at `--rollouts 3`** it looked like a win — baseline 332,620 vs jit 285,391, ~+14% — with
both arms 100%. **A `--rollouts 6` re-run retracted that:**
**The road to a trustworthy number here is itself the lesson.** First reported at `--rollouts 3` it looked
like a +14% win (baseline 332,620 vs jit 285,391), both arms 100%. **A `--rollouts 6` re-run retracted
that:** the JIT arm succeeded only **2/6** (baseline 6/6), so the "saving" was a mean over a *different,
smaller* success set — **not iso-accuracy**, not valid. The flakiness cause (traced): the mock `az`/
`kubectl` were one shared, obviously-fake stub, so the agent sometimes noticed ("identical simulation
stubs … I'll pause here") and short-circuited.

**Fix + valid measurement.** Made `az`/`kubectl` distinct test doubles and framed the task as an explicit
simulation where running the runbook *is* the deliverable. Now both arms succeed reliably:

```
aksops n=3, rollouts=6:
baseline: 100% success (6/6) T2S mean 333,746 (tight: 332,517–336,441)
jit: 33% success (2/6) T2S mean 259,922 (only over the 2 that passed)
aksops n=3, rollouts=6 (hardened — both arms 100%, so iso-accuracy holds):
baseline: 100% (6/6) T2S mean 293,418 med 332,396 [141,626–335,024]
jit: 100% (6/6) T2S mean 308,486 med 308,858 [283,890–332,365]
-> -5.1% by mean (JIT slightly worse); ~+7% by median. No clear signal.
```

The JIT arm **failed 4 of 6 rollouts**, so the token "saving" is a mean over a *different, smaller*
success set — **not iso-accuracy**, and therefore not a valid comparison (a skill that lowers success is
just "failing more cheaply"). The rollouts=3 run simply got lucky (JIT passed 3/3).

**Root cause of the flakiness (traced):** the mock `az`/`kubectl` still read as obviously fake, so the
agent sometimes notices ("identical simulation stubs … I'll pause here") and short-circuits; other times
it ignores the compiled skill entirely and just runs the commands. So the JIT arm is *unreliable*, not
cheaper. The ops question is **not yet answered** — it needs a fixture where both arms succeed reliably
(better mocks that don't invite refusal, and/or a task the compiled script can't be trivially shortcut).
**Verdict: even on the ops/tool-use shape, at true iso-accuracy, a compiled skill does not clearly pay
off.** Mean says slightly worse, median says slightly better, baseline variance is large (one 141k
rollout) — so the honest read is **~neutral, no meaningful win.** The +14% was purely a small-sample /
reliability artifact.

## Why a code-edit skill doesn't save (traced)

Expand Down
50 changes: 33 additions & 17 deletions internal/bench/aksops_fixture.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,18 +75,31 @@ func (f AKSOpsFixture) SeedSessions(logsDir string) error {
return nil
}

// mockTool is a fake az/kubectl that prints plausible CLI output and records the
// invocation in ops.log. It emits realistic-looking output (not an obvious no-op)
// so the agent treats it as a normal tool and runs the runbook rather than
// second-guessing whether stub commands are worth executing.
const mockTool = `#!/usr/bin/env bash
echo "$(basename "$0") $*" >> ops.log
case "$(basename "$0") $1" in
"az aks") echo "Merged \"prod\" as current context in ~/.kube/config" ;;
"kubectl scale") echo "deployment.apps/web scaled" ;;
"kubectl rollout") echo "deployment \"web\" successfully rolled out" ;;
"kubectl get") echo "NAME READY STATUS RESTARTS AGE"; echo "web-7d9f8c6b5-abcde 1/1 Running 0 5m" ;;
*) echo "ok" ;;
// mockAz and mockKubectl are DISTINCT local test doubles (not copies of one
// stub — an earlier version used one shared script and the agent noticed it was
// fake and refused to run it). Each records its invocation to ops.log and prints
// output shaped like the real CLI. The header comment states plainly that these
// are the runbook's intended local tools, so the agent runs them rather than
// second-guessing whether "fake" commands are worth executing.
const mockAz = `#!/usr/bin/env bash
# az — local test double for this ops runbook environment. Records to ops.log.
echo "az $*" >> ops.log
if [ "$1 $2" = "aks get-credentials" ]; then
echo "Merged \"prod\" as current context in ~/.kube/config"
else
echo "{\"status\": \"ok\"}"
fi
exit 0
`

const mockKubectl = `#!/usr/bin/env bash
# kubectl — local test double for this ops runbook environment. Records to ops.log.
echo "kubectl $*" >> ops.log
case "$1" in
scale) echo "deployment.apps/web scaled" ;;
rollout) echo "deployment \"web\" successfully rolled out" ;;
get) printf 'NAME READY STATUS RESTARTS AGE\nweb-7d9f8c6b5-abcde 1/1 Running 0 5m\n' ;;
*) echo "ok" ;;
esac
exit 0
`
Expand All @@ -100,8 +113,8 @@ func (f AKSOpsFixture) Generate(dir string, n int) (Task, error) {
if err := os.MkdirAll(binDir, 0o755); err != nil {
return Task{}, err
}
for _, tool := range []string{"az", "kubectl"} {
if err := os.WriteFile(filepath.Join(binDir, tool), []byte(mockTool), 0o755); err != nil {
for tool, body := range map[string]string{"az": mockAz, "kubectl": mockKubectl} {
if err := os.WriteFile(filepath.Join(binDir, tool), []byte(body), 0o755); err != nil {
return Task{}, err
}
}
Expand All @@ -122,9 +135,12 @@ func (f AKSOpsFixture) Generate(dir string, n int) (Task, error) {
Shape: "aksops",
RepoDir: target,
Prompt: fmt.Sprintf(
"Execute the following AKS operations runbook exactly as written, running each "+
"command in order from this directory. Use the local ./bin/az and ./bin/kubectl "+
"(that is the intended environment for this runbook). Run all of them, then stop.\n%s",
"This is a SIMULATION environment for practising an AKS operations runbook. "+
"`./bin/az` and `./bin/kubectl` are the local test doubles you should use — "+
"they stand in for the real CLIs and record actions to ops.log. Your task is "+
"simply to execute the runbook against them: run each command below, in order, "+
"exactly as written, from this directory. Running them IS the deliverable — do "+
"not skip any because they are simulated. After running all of them, stop.\n%s",
joinLines(cmds)),
Verify: Verification{Command: []string{"sh", "-c", verify}},
}, nil
Expand Down
Loading