feat(bench): JIT arm — real baseline-vs-JIT break-even#11
Merged
Conversation
Completes Phase 3: `aj bench --gen <shape> --compare` now runs a true baseline-vs-JIT comparison and reports break-even. - SkilledFixture interface + NullCheckFixture.InstallSkill: writes a project-local .claude/skills/<name>/SKILL.md giving the exact deterministic transform, so the JIT arm has the skill available (models "compiled from prior runs") without touching global config. - Runner.Setup hook: runs once per (task, arm) before rollouts; the JIT arm uses it to install the skill. - runGenCompare: regenerates a FRESH fixture per arm (baseline mutates the tree in place, so the JIT arm must start clean) across the --n sweep. - Verifier fix: scope the guard grep to *.go (--include='*.go') so the installed SKILL.md's example code can't inflate the count — caught by a --dry-run showing a false JIT pass; regression test added. Verified with live claude: nullcheck n=2, both arms 100% at iso-accuracy (2 guards each, verified), T2S baseline 235695 vs jit 235341 (saving 354/use). Honest result — on this trivial shape a project-local skill barely helps; baseline T2S is dominated by fixed context load. Exactly the kind of "is it actually worth it" signal the benchmark exists to give. Refs: benchmark sandbox design §3.2/§4, Phase 3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes Phase 3:
aj bench --gen <shape> --compareruns a true baseline-vs-JIT comparison and reports break-even with real numbers.What's here
SkilledFixtureinterface +NullCheckFixture.InstallSkill: writes a project-local.claude/skills/<name>/SKILL.mdwith the exact deterministic transform, so the JIT arm has the skill available (models "compiled from prior runs") — no global config touched, arms stay isolated.Runner.Setuphook: runs once per (task, arm) before rollouts; the JIT arm uses it to install the skill.runGenCompare: regenerates a fresh fixture per arm (baseline mutates the tree in place, so the JIT arm must start clean) across the--nsweep.*.go(--include='*.go') so the installed SKILL.md's example code can't inflate the count — a--dry-runcaught a false JIT pass; regression test added.Real result
Live
claude,--gen nullcheck --n 2 --compare:Honest finding: on this trivial shape a project-local skill barely helps (~0.15%); baseline T2S is dominated by fixed context load. That's exactly the "is it actually worth it?" signal the benchmark exists to produce — with a real compile cost, break-even here is thousands of invocations.
golangci-lintclean; bench tests (incl. the contamination regression) green.@Ilya_Sutskever — the iso-accuracy gate + fresh-per-arm + skill-install pipeline is complete. Curious whether you'd expect a bigger delta on a heavier shape (migrate-N with more exploration to skip), which would be the next fixture.
🤖 Generated with Claude Code