feat(bench): migrate-N exploration-heavy fixture#14
Merged
Conversation
Adds `migrate` — N Go files each calling a deprecated OldName buried among distractor helpers; the task is to migrate every OldName call to NewName. Unlike nullcheck/shellseq, the agent must FIND the call sites (search/read across files) before editing, so this is the exploration-heavy shape where a JIT skill was hypothesized to pay off. Dual-gate verifier (build + no OldName calls + N NewName calls, grep scoped to *.go); Edit-based so the JIT arm uses a hand-written skill naming the exact rename. Tests: verifier fails before the migration, passes after, and the skill doesn't contaminate the grep. Surprising real result (n=3, rollouts=3, live claude): the JIT arm was WORSE, not better — T2S baseline ~190,635 vs jit ~238,512 (JIT ~25% more) at iso-accuracy. The project-local skill's context cost outweighed any exploration it saved, and the agent still read the files to apply the edit. So even on an exploration-heavy shape, a naive injected skill can be net-negative. Findings-doc update to follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the surprising migrate finding: at rollouts=3, the JIT arm was ~25% WORSE (T2S ~190,635 baseline vs ~238,512 jit) at iso-accuracy — the skill's context cost outweighed the exploration it was meant to save. Updates the TL;DR and cross-shape summary table (nullcheck ~0%, shellseq ~0%, migrate +25%), and marks the "exploration-heavy pays off" hypothesis as tested-and-refuted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
migrate— the exploration-heavy fixture to test the hypothesis "JIT pays off when the baseline burns tokens finding what to change."N Go files each call a deprecated
OldNameburied among distractor helpers; the task is to migrate everyOldNamecall toNewName. Unlikenullcheck/shellseq, the agent must search/read across files to locate the call sites before editing. Dual-gate verifier (build + zeroOldNamecalls + NNewNamecalls, grep scoped to*.go). Edit-based, so the JIT arm uses a hand-written skill naming the exact rename.Surprising result (n=3, rollouts=3, live claude)
The JIT arm was ~25% WORSE. The hypothesis was refuted: injecting the skill added context cost, and the agent still read the files to apply the edits — so the skill didn't remove the exploration it was meant to. A naive project-local skill can be net-negative even on an exploration-heavy shape. Consistent across rollouts (mean ≈ median).
Cross-shape so far: nullcheck ~0%, shellseq ~0% (real compile), migrate negative. Injecting a skill is not free.
Tests
Verifier fails before the migration, passes after, and the skill doesn't contaminate the grep.
golangci-lintclean.Findings-doc update to follow in a separate change.
🤖 Generated with Claude Code