feat(core): execution metadata on the durable log + foldTurnRecord - #162
Open
lipowen wants to merge 3 commits into
Open
feat(core): execution metadata on the durable log + foldTurnRecord#162lipowen wants to merge 3 commits into
lipowen wants to merge 3 commits into
Conversation
The Model seam gains a normalized usage claim: ModelDelta's terminal
`done` carries `usage?: ModelUsage` ({ inputTokens, outputTokens, raw? }),
and the engine persists it on the assistant Msg at commit time together
with a new durationMs (model-call wall-clock, measured before the commit
so storage time never inflates it). Every executed tool Msg records its
own durationMs; the synthetic results a cancelled batch commits carry
none — nothing ran. On a crash-replay the cached step skips, so the
persisted numbers honestly describe the run that actually paid.
The messages log is the one durable surface a fold/export can read back
(step checkpoints are presence-checked, never enumerated), so putting
the metadata on the Msg makes per-turn cost/latency accounting possible
with no side table and no schema change (stores serialize Msg as JSON).
The anthropic adapter previously dropped the SDK's usage on the floor:
finalMessage().usage now maps through usageFromAnthropic (exported,
mirroring finishFromStopReason; partial claims are dropped whole, the
provider object rides along as raw). A transport that omits usage
yields the byte-identical delta shape as before — fake-client tests
and the adapter conformance suite pass unchanged. replyStream accepts
an optional third usage argument for scripted models.
Claude-Session: https://claude.ai/code/session_01CQ4hXUQHnkRPfPDSdQZsEx
The durable log already IS a complete account of a turn — opening, every model reply, every tool call with its frozen result, and (since the previous commit) usage and durations — but stored as interleaved Msg rows. foldTurnRecord(msgs, turnId) assembles ONE turn's rows into a self-contained record, exposed as AgentSession.record(turnId): - opening: the user text, or a proactive trigger attributed to `by` - steps: model steps (text, toolCalls, usage?, durationMs?) and tool steps with the input recovered from the requesting model step, so the record stands alone — a replay needs no join back - status: "completed" iff the last model step has no tool calls (the live engine's own terminal condition); everything the log cannot distinguish (in-flight / failed / cancelled / suspended — live-only outcomes) is "incomplete", never guessed - usage: aggregated only when EVERY model step claimed it, so a cost report can't silently undercount This record is the atom the observability and eval lines share: a trace exporter folds on turn-terminal and ships one document; an eval dataset curates records from production and replays them against the frozen tool results. Pure over Msg[], beside foldTranscript. Claude-Session: https://claude.ai/code/session_01CQ4hXUQHnkRPfPDSdQZsEx
…ool duration, naming Findings from two independent review passes, applied: - ModelUsage semantic contract pinned: inputTokens is the TOTAL input the call consumed, cache reads/writes INCLUDED. Providers disagree (OpenAI/Gemini headline counts include cached tokens; Anthropic's input_tokens excludes its separately-reported cache fields), so the adapter now sums them back in and surfaces the split via normalized cachedInputTokens / cacheCreationInputTokens — cross-provider cost sums no longer silently undercount cache-heavy turns, and the split a cost report needs (cache reads ~10%, writes ~125%) no longer hides in raw. - Sync local tools record no durationMs: they do no I/O, and workerd's request-frozen clock only advances on I/O — the measurement would read 0 on the edge. Recording nothing beats recording a lie; remote (async) tools keep their wall-clock. - AgentSession.record(turnId) renamed turnRecord(turnId): record reads as a write verb (channels already use feedback.record for one); the noun getter aligns with transcript()/snapshot(). - Aggregate-usage honesty rule now pinned by a test: one model step without a claim drops the aggregate while per-step usage stays. - Changesets bumped patch → minor (new public API surface, matching the google-drive precedent) and reworded to the corrected semantics. Claude-Session: https://claude.ai/code/session_01CQ4hXUQHnkRPfPDSdQZsEx
Contributor
Author
|
Two independent review passes (correctness/durability + API design) ran against this branch; findings applied in bdac6fa:
What the reviews confirmed correct and unchanged: exactly-once/replay determinism, byte-shape spread discipline, the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two small, stacked additions to the agent core — the shared foundation for the observability and eval lines:
1. Execution metadata on the durable log (
usage+durationMs)ModelDelta's terminaldonegainsusage?: ModelUsage—{ inputTokens, outputTokens, raw? }, normalized across providers, the provider's own usage object preserved asraw(mirroringModelFinish.raw). Any adapter can claim it; no claim = no field (spread, never assignedundefined, so the pre-existing delta shape stays byte-identical).Msgat commit time, together with a newdurationMs(model-call wall-clock, measured before the commit so storage time never inflates it). Every executed toolMsgrecords its owndurationMs; the synthetic results a cancelled batch commits carry none — nothing ran. On a crash-replay the cached step skips, so the persisted numbers honestly describe the run that actually paid.Msgas JSON).finalMessage().usagenow maps throughusageFromAnthropic(exported besidefinishFromStopReason; partial claims are dropped whole — no half-truth for a cost report to trust).replyStreamaccepts an optional thirdusageargument for scripted models.2.
foldTurnRecord(msgs, turnId)+AgentSession.record(turnId)The durable log already IS a complete account of a turn — this fold assembles ONE turn's interleaved rows into a self-contained record: opening (user text or attributed proactive trigger), steps (model steps with usage/duration; tool steps with the input recovered from the requesting model step, so the record stands alone), terminal status by the live engine's own condition, final text, and an aggregate usage that exists only when every model step claimed it (a cost report can't silently undercount). Anything the log cannot distinguish (failed / cancelled / suspended — live-only outcomes) folds to
"incomplete", never guessed.Why
This record is the atom the observability and eval lines share:
recordFeedback(turnId, …)) anchors on the same record.Neither change touches the server↔core contract shape (additive optional fields; the server constructs none of them), so no
RUNTIME_API_VERSIONbump.Tests
usageFromAnthropicmapping (full claim + raw passthrough; partial/absent dropped whole), done-delta carries normalized usage, byte-identical shape when the transport omits it (existing tests unchanged).usageproperty; a cancelled batch's synthetic results carry nodurationMs.undefined, proactive opening attribution.Full suite: 1181 pass / 0 fail.
https://claude.ai/code/session_01CQ4hXUQHnkRPfPDSdQZsEx