Repository navigation
Record the first work-versus-instructions A/B result in the accurate-work-steps design - #116
Merged
Merged
Conversation
…work-steps design Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Start here: this records the first A/B test of
work_stepsagainst measured instructions in theaccurate-work-stepsdesign. It changes docs only.Across six real pairs and a no-change control on five grammars,
work_stepsgave the right direction but not a reliable size:The design now gives Motif a provisional band: at least 5% reads as better or worse, 2-5% as probably, and under 2% as no clear change. New tasks 2.5 and 2.6 add the memo and
distincthard cases and re-measure the two disagreements withcost_units.Review focus: whether the band is too confident, since seven pairs produce these results and they are not independent.
Limits: sampled word lists exclude the heavy tail (work up to 3M), cache and memory effects are not measured, and grammar edits are not covered. Raw data and tools stay outside the repository.
Base
c31ace80.pg.ps1 -Mode prelandpassed, with conformance and scoreboard skipped because no gated path changed.openspec validatepassed.🤖 Generated with Claude Code