Skip to content

docs: retract the aksops +14% (iso-accuracy violation)#16

Merged
Poytr1 merged 1 commit into
mainfrom
fix/findings-retract-aksops
Jul 7, 2026
Merged

docs: retract the aksops +14% (iso-accuracy violation)#16
Poytr1 merged 1 commit into
mainfrom
fix/findings-retract-aksops

Conversation

@Poytr1

@Poytr1 Poytr1 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Why

I reported aksops as a +14% JIT win from a --rollouts 3 run. A --rollouts 6 re-run shows that was a small-sample artifact and it is retracted.

aksops n=3, rollouts=6:
  baseline: 100% success (6/6)   T2S ~333,746
  jit:       33% success (2/6)   T2S ~259,922  (mean over only the 2 that passed)

The "saving" was a mean over a different, smaller success set — not iso-accuracy, so not a valid comparison. The rollouts=3 run got lucky (JIT 3/3). The aksops fixture is flaky for the agent (it notices the mock stubs are fake and sometimes short-circuits).

What this changes

  • No shape shows a valid JIT win: nullcheck ~0%, shellseq ~0%, migrate −25%, aksops inconclusive.
  • Ops/tool-use remains the most promising hypothesis but is unconfirmed — needs a fixture where both arms succeed reliably.
  • Adds the lesson: compare at iso-accuracy and check the verified sample size on both arms before believing a mean. (aj bench --compare already prints an iso-accuracy WARN; I bypassed it in a JSON post-process and misled myself.)

Docs-only; corrects a claim already on main (from #15).

🤖 Generated with Claude Code

A --rollouts 6 re-run showed the aksops JIT arm succeeds only 2/6
(baseline 6/6), so the earlier +14% (from a lucky 3/3 at rollouts 3) was a
mean over a different, smaller success set — not iso-accuracy, and not a
valid win. Rewrites the TL;DR and the aksops section: no shape shows a
valid JIT win; ops/tool-use remains the promising-but-unconfirmed
hypothesis, blocked on a fixture where both arms succeed reliably (the
mock stubs currently make the agent short-circuit).

Adds the lesson: compare at iso-accuracy and check the verified sample
size on BOTH arms before believing a mean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Poytr1 Poytr1 merged commit 9b78294 into main Jul 7, 2026
2 checks passed
@Poytr1 Poytr1 deleted the fix/findings-retract-aksops branch July 7, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant