-
Notifications
You must be signed in to change notification settings - Fork 196
fix(workflow): stop node state writes from being rolled back mid-run #636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
277165d
8e050d7
5cc73fc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -117,9 +117,10 @@ export class NodeContext { | |
| this.actions = opts.actions ?? createEventActions(); | ||
| // Writes via `ctx.state` accumulate into `actions.stateDelta`, mirroring | ||
| // Python's `ctx.state` -> `ctx.actions.state_delta` behaviour. | ||
| this._state = new State( | ||
| opts.invocationContext.session.state, | ||
| this._state = new NodeStateView( | ||
| invocationOverlay(opts.invocationContext), | ||
| this.actions.stateDelta, | ||
| opts.invocationContext.session.state, | ||
| ); | ||
| } | ||
|
|
||
|
|
@@ -182,3 +183,112 @@ export class NodeContext { | |
| return executeChildNode({parent: this, node, input, options}); | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Per-invocation record of every `ctx.state` write made by the nodes of one | ||
| * invocation, keyed by the session's live state object. | ||
| * | ||
| * `session.state` object identity is stable for the duration of a turn (the | ||
| * session services mutate it in place, and hand out a fresh object per | ||
| * `getSession`), so the WeakMap entry naturally scopes to one turn. | ||
| * | ||
| * The `invocationId` guard covers *sequential* reuse of one state object: a | ||
| * later invocation replaces the entry rather than inheriting the earlier one's | ||
| * writes. It does not let two *live* invocations share a state object — they | ||
| * would evict each other's entry and fall back to reading committed state. | ||
| * Nothing does that today: `getSession` hands out a fresh state object per | ||
| * turn, and a sub-invocation (`AgentTool`) re-fetches the session, so it keys a | ||
| * different entry. | ||
| */ | ||
| const invocationOverlays = new WeakMap< | ||
| Record<string, unknown>, | ||
| {invocationId: string; values: Record<string, unknown>} | ||
| >(); | ||
|
|
||
| /** Returns the write overlay for `ic`'s invocation, creating it on first use. */ | ||
| function invocationOverlay(ic: InvocationContext): Record<string, unknown> { | ||
| const sessionState = ic.session.state; | ||
| const existing = invocationOverlays.get(sessionState); | ||
| if (existing && existing.invocationId === ic.invocationId) { | ||
| return existing.values; | ||
| } | ||
| const created = {invocationId: ic.invocationId, values: {}}; | ||
| invocationOverlays.set(sessionState, created); | ||
| return created.values; | ||
| } | ||
|
|
||
| /** | ||
| * The state view a workflow node sees: its own pending delta, then the | ||
| * invocation's write overlay, then the session's committed state. | ||
| * | ||
| * The overlay exists to keep node-to-node reads honest. `session.state` is | ||
| * mutated from two directions during a run: nodes write through it | ||
| * immediately, while the runner separately re-applies each event's | ||
| * `actions.stateDelta` as it commits that event — and that commit lags node | ||
| * execution. Re-applying an earlier node's delta therefore rolls back a later | ||
| * node's write, and any node reading in that window observes the stale value: | ||
| * | ||
| * a: set('attempts', 0) | ||
| * b: get -> 0, set('attempts', 1) | ||
| * commit(a) re-applies attempts=0 <- rolls back b's write | ||
| * c: get -> 0 <- wrong; b already set 1 | ||
| * commit(b) re-applies attempts=1 <- rolls forward, too late | ||
| * | ||
| * Reads are served from the overlay, which only ever moves forward, so `c` | ||
| * sees `1`. Writes still land in `session.state` as well, so consumers that | ||
| * read it directly — notably `{key}` instruction templating in an agent node — | ||
| * behave exactly as before. | ||
| * | ||
| * Two gaps remain. They are mirror images of each other, share the root cause | ||
| * above, and are both only really fixable at the source (version-stamping keys | ||
| * as the session services apply a delta), so neither is closed here: | ||
| * | ||
| * - Workflow writes still reach an outside reader out of order. `session.state` | ||
| * is written through *and* re-applied by the event commit, so an agent | ||
| * resolving `{key}` for a key two nodes wrote can observe the stale value | ||
| * inside the same window. | ||
| * - Outside writes no longer reach a workflow reader. Once a node writes `k`, | ||
| * every later read of `k` in this invocation is served from the overlay, so a | ||
| * tool or callback that writes `k` straight to `session.state` mid-run is | ||
| * invisible to those reads for the rest of the invocation. Before the overlay | ||
| * they saw it, subject to the rollback race being fixed. | ||
| */ | ||
| class NodeStateView extends State { | ||
| constructor( | ||
| overlay: Record<string, unknown>, | ||
| delta: Record<string, unknown>, | ||
| private readonly committed: Record<string, unknown>, | ||
| ) { | ||
| super(overlay, delta); | ||
| } | ||
|
|
||
| override get<T>(key: string, defaultValue?: T): T | undefined { | ||
| if (super.has(key)) { | ||
| return super.get<T>(key, defaultValue); | ||
| } | ||
| return key in this.committed ? (this.committed[key] as T) : defaultValue; | ||
| } | ||
|
Comment on lines
+265
to
+270
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit. The overlay also shadows writes coming the other way, which is worth naming next to the known residual. Once a node writes override get<T>(key: string, defaultValue?: T): T | undefined {
if (super.has(key)) {
return super.get<T>(key, defaultValue);
}
return key in this.committed ? (this.committed[key] as T) : defaultValue;
}A tool or callback that writes the same key straight to This is the mirror of the residual you already document: that one is workflow writes not reaching an outside reader, this one is outside writes not reaching a workflow reader. Same root cause, same real fix (version-stamping). Worth a sentence in the same place, so the next person does not have to derive it.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, and you framed it better than I would have. Documented both directions together in the Worth stating plainly that this second one is a behaviour change, not just an untouched gap: before the overlay those reads did see the outside write (racily). I think it's the right trade — a mid-run outside writer to a key the workflow also owns is already in undefined-ordering territory, whereas node-to-node read-modify-write is the documented pattern — but it is a trade, and the doc now says so. I'll also carry both into the PR description's Known residual section. |
||
|
|
||
| override has(key: string): boolean { | ||
| return super.has(key) || key in this.committed; | ||
| } | ||
|
|
||
| // Both of `State`'s write paths (`set` and `update`) keep writing through to | ||
| // the session, so readers of `session.state` (agent instruction templates, | ||
| // callbacks, tools) see the write immediately, as they did before the overlay | ||
| // existed. | ||
|
|
||
| override set(key: string, value: unknown): void { | ||
| super.set(key, value); | ||
| this.committed[key] = value; | ||
| } | ||
|
Comment on lines
+281
to
+284
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not a nit. override set(key: string, value: unknown): void {
super.set(key, value);
this.committed[key] = value;
}
update(delta: Record<string, unknown>) {
Object.assign(this.delta, delta);
Object.assign(this.value, delta); // overlay only
}So Nothing calls override update(delta: Record<string, unknown>): void {
super.update(delta);
Object.assign(this.committed, delta);
}
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch — fixed in a00c183. Added the override exactly as you wrote it: override update(delta: Record<string, unknown>): void {
super.update(delta);
Object.assign(this.committed, delta);
}Also moved the write-through comment above both overrides so it reads as a property of Pinned with a test ( |
||
|
|
||
| override update(delta: Record<string, unknown>): void { | ||
| super.update(delta); | ||
| Object.assign(this.committed, delta); | ||
| } | ||
|
|
||
| override toRecord(): Record<string, unknown> { | ||
| return {...this.committed, ...super.toRecord()}; | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a nit. A third path misses the overlay:
llm_agent_wrapper.ts:141writes the delta directly.That write skips the overlay and skips
session.state, so the next node readsundefineduntil the event commits. Usectx.state.set(agent.outputKey, output), which writes the same delta key, plus the overlay and the session. The fault is older than this PR, and no workflow test coversoutputKey.llm_agent.ts:708repeats the pattern outside this package.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 5cc73fc — and you undersold it: that write does not just miss the overlay, it lands nowhere at all.
ctx.actions.stateDeltais only ever drained onto an event byFunctionNode.toEvent(function_node.ts:182, viapendingStateDelta).BaseNode.toEventdoesn't,enrichEventdoesn't, and nothing constructs aNodeContextwith a sharedactions, so for this node the object is write-only. Task-modeoutputKeytherefore reached neither the overlay, norsession.state, nor committed state — it was dropped, not merely late.Measured with the new test, which asserts all three (later node reads it, it is on
session.state, it is in the committed session):session.statectx.actions.stateDelta[k] = v(before)ctx.state.set(k, v)(as suggested)So it needs the write-through and the event stamp:
Test:
an agent node's outputKey is visible to the next node— a task-modeLlmAgentover a mock model that callsfinish_task, feeding a node that reads the key. First workflow coverage ofoutputKey, as you said.On
llm_agent.ts:708: left alone deliberately, and it is not the same severity. I probed a single_turn agent node with anoutputKeyfollowed by a reader node — the reader does see the value today, because the runner commits the agent's event before the next node starts. That path is exposed to the commit-lag window (the first residual in theNodeStateViewdoc comment), not to a dropped write, and closing it properly is the version-stamping change rather than a one-liner here.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a nit. I retract two parts of my comment above; your framing is correct.
The value never committed, so "reads
undefineduntil the event commits" was wrong. Nothing drainsctx.actionsfor this node:BaseNode.toEventnever reads it (base_node.ts:191).enrichEventonly writesagentState(node_runner.ts:250).actionsto aNodeContext.ctx.state.setalone was also incomplete — committed state would stay empty. The landed pair is right, and the new test pins all three effects.