Skip to content

feat(core): execution metadata on the durable log + foldTurnRecord - #162

Open
lipowen wants to merge 3 commits into
junebuild:mainfrom
lipowen:feat/turn-usage-and-record
Open

feat(core): execution metadata on the durable log + foldTurnRecord#162
lipowen wants to merge 3 commits into
junebuild:mainfrom
lipowen:feat/turn-usage-and-record

Conversation

@lipowen

@lipowen lipowen commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What

Two small, stacked additions to the agent core — the shared foundation for the observability and eval lines:

1. Execution metadata on the durable log (usage + durationMs)

  • ModelDelta's terminal done gains usage?: ModelUsage{ inputTokens, outputTokens, raw? }, normalized across providers, the provider's own usage object preserved as raw (mirroring ModelFinish.raw). Any adapter can claim it; no claim = no field (spread, never assigned undefined, so the pre-existing delta shape stays byte-identical).
  • The engine persists the claim on the assistant Msg at commit time, together with a new durationMs (model-call wall-clock, measured before the commit so storage time never inflates it). Every executed tool Msg records its own durationMs; the synthetic results a cancelled batch commits carry none — nothing ran. On a crash-replay the cached step skips, so the persisted numbers honestly describe the run that actually paid.
  • The messages log is the one durable surface a fold can read back (step checkpoints are presence-checked, never enumerated), so this lands cost/latency accounting with no side table and no schema change (stores serialize Msg as JSON).
  • The anthropic adapter previously dropped the SDK's usage on the floor; finalMessage().usage now maps through usageFromAnthropic (exported beside finishFromStopReason; partial claims are dropped whole — no half-truth for a cost report to trust). replyStream accepts an optional third usage argument for scripted models.

2. foldTurnRecord(msgs, turnId) + AgentSession.record(turnId)

The durable log already IS a complete account of a turn — this fold assembles ONE turn's interleaved rows into a self-contained record: opening (user text or attributed proactive trigger), steps (model steps with usage/duration; tool steps with the input recovered from the requesting model step, so the record stands alone), terminal status by the live engine's own condition, final text, and an aggregate usage that exists only when every model step claimed it (a cost report can't silently undercount). Anything the log cannot distinguish (failed / cancelled / suspended — live-only outcomes) folds to "incomplete", never guessed.

Why

This record is the atom the observability and eval lines share:

  • a trace exporter folds on turn-terminal and ships one document (OTel GenAI-shaped, in a follow-up);
  • an eval dataset curates records from production and replays them against the frozen tool results — June's durable-by-construction turns make "prod trace → eval case" structurally free, which most agent frameworks have to assemble by hand;
  • feedback (a future recordFeedback(turnId, …)) anchors on the same record.

Neither change touches the server↔core contract shape (additive optional fields; the server constructs none of them), so no RUNTIME_API_VERSION bump.

Tests

  • adapter: usageFromAnthropic mapping (full claim + raw passthrough; partial/absent dropped whole), done-delta carries normalized usage, byte-identical shape when the transport omits it (existing tests unchanged).
  • engine: usage + durationMs land on assistant/tool Msgs; no-claim assistant rows carry no own usage property; a cancelled batch's synthetic results carry no durationMs.
  • fold: completed tool-loop turn (steps order, standalone tool input recovery, aggregate usage), incomplete turn after a mid-turn crash (no text, no aggregate), unknown turnId → undefined, proactive opening attribution.

Full suite: 1181 pass / 0 fail.

https://claude.ai/code/session_01CQ4hXUQHnkRPfPDSdQZsEx

The Model seam gains a normalized usage claim: ModelDelta's terminal
`done` carries `usage?: ModelUsage` ({ inputTokens, outputTokens, raw? }),
and the engine persists it on the assistant Msg at commit time together
with a new durationMs (model-call wall-clock, measured before the commit
so storage time never inflates it). Every executed tool Msg records its
own durationMs; the synthetic results a cancelled batch commits carry
none — nothing ran. On a crash-replay the cached step skips, so the
persisted numbers honestly describe the run that actually paid.

The messages log is the one durable surface a fold/export can read back
(step checkpoints are presence-checked, never enumerated), so putting
the metadata on the Msg makes per-turn cost/latency accounting possible
with no side table and no schema change (stores serialize Msg as JSON).

The anthropic adapter previously dropped the SDK's usage on the floor:
finalMessage().usage now maps through usageFromAnthropic (exported,
mirroring finishFromStopReason; partial claims are dropped whole, the
provider object rides along as raw). A transport that omits usage
yields the byte-identical delta shape as before — fake-client tests
and the adapter conformance suite pass unchanged. replyStream accepts
an optional third usage argument for scripted models.

Claude-Session: https://claude.ai/code/session_01CQ4hXUQHnkRPfPDSdQZsEx
The durable log already IS a complete account of a turn — opening,
every model reply, every tool call with its frozen result, and (since
the previous commit) usage and durations — but stored as interleaved
Msg rows. foldTurnRecord(msgs, turnId) assembles ONE turn's rows into
a self-contained record, exposed as AgentSession.record(turnId):

- opening: the user text, or a proactive trigger attributed to `by`
- steps: model steps (text, toolCalls, usage?, durationMs?) and tool
  steps with the input recovered from the requesting model step, so
  the record stands alone — a replay needs no join back
- status: "completed" iff the last model step has no tool calls (the
  live engine's own terminal condition); everything the log cannot
  distinguish (in-flight / failed / cancelled / suspended — live-only
  outcomes) is "incomplete", never guessed
- usage: aggregated only when EVERY model step claimed it, so a cost
  report can't silently undercount

This record is the atom the observability and eval lines share: a
trace exporter folds on turn-terminal and ships one document; an eval
dataset curates records from production and replays them against the
frozen tool results. Pure over Msg[], beside foldTranscript.

Claude-Session: https://claude.ai/code/session_01CQ4hXUQHnkRPfPDSdQZsEx
…ool duration, naming

Findings from two independent review passes, applied:

- ModelUsage semantic contract pinned: inputTokens is the TOTAL input
  the call consumed, cache reads/writes INCLUDED. Providers disagree
  (OpenAI/Gemini headline counts include cached tokens; Anthropic's
  input_tokens excludes its separately-reported cache fields), so the
  adapter now sums them back in and surfaces the split via normalized
  cachedInputTokens / cacheCreationInputTokens — cross-provider cost
  sums no longer silently undercount cache-heavy turns, and the split
  a cost report needs (cache reads ~10%, writes ~125%) no longer hides
  in raw.
- Sync local tools record no durationMs: they do no I/O, and workerd's
  request-frozen clock only advances on I/O — the measurement would
  read 0 on the edge. Recording nothing beats recording a lie; remote
  (async) tools keep their wall-clock.
- AgentSession.record(turnId) renamed turnRecord(turnId): record reads
  as a write verb (channels already use feedback.record for one); the
  noun getter aligns with transcript()/snapshot().
- Aggregate-usage honesty rule now pinned by a test: one model step
  without a claim drops the aggregate while per-step usage stays.
- Changesets bumped patch → minor (new public API surface, matching
  the google-drive precedent) and reworded to the corrected semantics.

Claude-Session: https://claude.ai/code/session_01CQ4hXUQHnkRPfPDSdQZsEx
@lipowen

lipowen commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Two independent review passes (correctness/durability + API design) ran against this branch; findings applied in bdac6fa:

  • ModelUsage semantic contract pinnedinputTokens = TOTAL input consumed, cache reads/writes included. Anthropic's input_tokens excludes its separately-reported cache fields (unlike OpenAI/Gemini headline counts), so the adapter sums them back in and surfaces the split via normalized cachedInputTokens/cacheCreationInputTokens. Cross-provider cost sums no longer undercount cache-heavy turns.
  • Sync local tools record no durationMs — no I/O means workerd's request-frozen clock would read 0; recording nothing beats recording a lie. Remote (async) tools keep their wall-clock.
  • AgentSession.record()turnRecord() — noun getter aligned with transcript()/snapshot(); record reads as a write verb and channels already use feedback.record for one.
  • Aggregate-usage honesty rule now pinned by a test (one unclaiming model step drops the aggregate, per-step stays).
  • Changesets bumped to minor (new public API, matching the google-drive precedent).

What the reviews confirmed correct and unchanged: exactly-once/replay determinism, byte-shape spread discipline, the completed condition's equivalence with foldEvents/result(), no RUNTIME_API_VERSION bump needed (server treats Msg as an opaque JSON blob). Full suite: 1183 pass / 0 fail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant