Skip to content

Record the first work-versus-instructions A/B result in the accurate-work-steps design - #116

Merged
johnml1135 merged 1 commit into
mainfrom
spec/accurate-work-steps-ab
Nov 1, 2026
Merged

johnml1135 merged 1 commit into
mainfrom
spec/accurate-work-steps-ab

Conversation

@johnml1135

Copy link
Copy Markdown
Collaborator

Start here: this records the first A/B test of work_steps against measured instructions in the accurate-work-steps design. It changes docs only.

Across six real pairs and a no-change control on five grammars, work_steps gave the right direction but not a reliable size:

  • Control: work moved exactly 0 and instructions at most 0.28%. Cycles moved up to 17%, so they cannot serve as the signal.
  • Sign agreement with instructions: 11 of 11 at 5% or more, 14 of 15 at 2% or more, 16 of 18 at 1% or more.
  • That misses the provisional 95% bar at 1%. The two disagreements are listed, and their cause is not yet measured.
  • Steps stayed still on 8 of the 18 results at 1% or more, which supports retiring them.

The design now gives Motif a provisional band: at least 5% reads as better or worse, 2-5% as probably, and under 2% as no clear change. New tasks 2.5 and 2.6 add the memo and distinct hard cases and re-measure the two disagreements with cost_units.

Review focus: whether the band is too confident, since seven pairs produce these results and they are not independent.

Limits: sampled word lists exclude the heavy tail (work up to 3M), cache and memory effects are not measured, and grammar edits are not covered. Raw data and tools stay outside the repository.

Base c31ace80. pg.ps1 -Mode preland passed, with conformance and scoreboard skipped because no gated path changed. openspec validate passed.

🤖 Generated with Claude Code

…work-steps design

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnml1135
johnml1135 merged commit 43fb152 into main Nov 1, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant