test(bench): verbose aksops output — the mock wasn't the confound#18
Merged
Conversation
@pchsu asked if the ~neutral ops result was a mock artifact (real az/kubectl emit large output that round-trips through the model per command; near-silent mocks understate what a compiled runbook skill could save). Made the mocks emit realistic sizable output (~17KB/runbook: az kubeconfig JSON, multi-line rollout progress, a 120-row `kubectl get pods -o wide` table). Result (rollouts=6, both arms 100% → iso-accuracy): still ~neutral — baseline mean 321,820 (med 330,304) vs jit mean 330,034 (med 329,740) -> -2.6% by mean, +0.2% by median. No JIT win. So the mock was NOT the main issue. Deeper finding: ~17KB (~4-5k tokens) of output is a rounding error against the ~330k baseline, which is dominated by fixed system-prompt/tool-def/cache-reread load, not task-specific work. A skill that shortcuts the task-specific portion can only ever save a small fraction — the tokens aren't where JIT optimizes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t the work Adds the deepest conclusion from the verbose-mock test: every episode is ~250-330k tokens dominated by fixed system-prompt/tool-def/cache-reread overhead; the task-specific work (edit, few commands, even 17KB of output) is a small slice, so JIT can only shortcut that slice and is structurally limited at saving tokens for Claude-Code-shaped tasks. Reframes AgentJIT's likely value as latency/determinism (not measured here) and recommends a benchmark v2 with wall-clock + reliability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Question tested
Was the ~neutral
aksopsresult an artifact of near-silent mocks? (Realaz/kubectlemit large output that round-trips through the model per command; a compiled runbook skill run in one shot might avoid that.)What changed
Made the mock
az/kubectlemit realistic sizable output (~17KB/runbook: az kubeconfig JSON, multi-line rollout progress, a 120-rowkubectl get pods -o widetable).Result (rollouts=6, both arms 100% → iso-accuracy)
The mock was not the confound. ~17KB (~4-5k tokens) is a rounding error against the ~330k baseline, which is dominated by fixed system-prompt / tool-def / cache-reread overhead — not task-specific work. A skill can only shortcut the small task-specific slice, so JIT can't win on tokens here. To make output matter it'd need to be massive (100s of KB), unrealistic for a runbook.
Reframes the study conclusion: JIT-compiling skills to save tokens is structurally limited for Claude-Code-shaped tasks; the value is more likely latency/determinism (not yet measured). Findings-doc update to follow.
🤖 Generated with Claude Code