Skip to content

Plan accurate work steps and stop reading the clock per work unit - #115

Merged
johnml1135 merged 1 commit into
mainfrom
spec/accurate-work-steps
Nov 1, 2026
Merged

johnml1135 merged 1 commit into
mainfrom
spec/accurate-work-steps

Conversation

@johnml1135

Copy link
Copy Markdown
Collaborator

Start here: openspec/changes/accurate-work-steps/design.md, then rust/crates/pg-fst/src/work.rs.

Adds the plan for making work_steps the one work unit and retiring steps, and lands the two pieces that are safe today. With a word deadline set, parsing no longer pays a clock read on every work unit: that cost falls from about 90 instructions per unit to 1.6-4.3, or from 3.5-4% of instructions to 0.06-0.19%.

The main question for review is whether a timeout may now fire late. WorkBudget::consume reads the clock once per 64 units, so a timeout can overshoot by at most 63 units. Timeouts are already machine-dependent and non-certifying. The cap check still runs on every reservation, and stopped() always reads the clock, so a poll-only loop still sees the deadline.

  • The deadline still acts: a_deadline_stops_reservations_within_one_poll_interval fails with the interval removed (checked by mutation).
  • Between polls: a_poll_observes_the_deadline_between_reservation_polls and the_cap_is_exact_between_clock_reads.
  • Work counts unchanged: identical work_steps with and without a deadline on Indonesian and Amharic (1,613,681 and 32,238,244).
  • Rename: SynthesisBudget::steps() returned the work total, so it is now work_steps(). No behaviour change.

Not done here: steps still exists everywhere, and no charge, cap, default, batch TSV column or ranking changes. Those are later task groups behind stated gates, including a recall gate before the attempt cap goes. The A/B test of whether a change in work_steps predicts a change in instructions has not run, so D5's thresholds are provisional.

Base 71245076, head 14eac325. pg.ps1 -Mode test: 2,244 passed, 66 skipped (before the last spec-only amend). -Mode preland: passed, with conformance skipped because no gated path changed. A forced -All run failed only that driver, because hc-conformance.exe for the pin is not built here. No parse comparison against a base build was run on real grammars.

Next: land the measurement tools (task group 2), then run the A/B set.


Evidence behind the plan

Measured on native Windows (i7-12700) over five real grammars, with PMC on context switch. Instructions repeat to 0.15-0.40% per word, core cycles to 5-17%. Instructions per step span 52x and per work step 2.2x. One per-grammar constant predicts a word's instructions to a median 3-14%.

Valgrind in WSL agrees with PMC at per-word log R² 0.995-0.999. Size-aware counters cut held-out median error from 18.6% to 9.0%, using bytes allocated and FST registers copied. Details and caveats are in design.md. Raw data contains real-language words and stays local.

Paths not taken
  • Cycles as the unit: not deterministic, and no unprivileged Windows counter.
  • A fitted per-class cost table: weights vary up to 18x across grammars and do no better than one per-grammar constant.
  • Moving Score::key to work now: a plain swap shrinks raw_paths about 30x and reopens a recorded mis-ranking, so it waits for a measured weight (D7).

🤖 Generated with Claude Code

Adds the accurate-work-steps OpenSpec change: instructions as ground truth, size-aware charges, a sign-agreement acceptance test, and retiring steps behind a recall gate.

WorkBudget::consume now reads the clock once per 64 units. On Indonesian and Amharic this took the deadline's cost from about 90 instructions per unit to 1.6-4.3, with work_steps unchanged. SynthesisBudget::steps() returned the work total, so it is now work_steps().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnml1135
johnml1135 merged commit c31ace8 into main Nov 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant