fix(workflow): scope node rehydration to the current invocation - #641
fix(workflow): scope node rehydration to the current invocation#641mohass1927 wants to merge 1 commit into
Conversation
Workflow rehydration matched prior session events by node path alone, so a completed node from one invocation was fast-forwarded into a later, unrelated invocation on the same session. A workflow that gates protected work behind a node (an authorization or credential check, as in auth_gate_test) would skip that node on the next message and continue with the earlier decision. The Python implementation this module is ported from does not have this: it threads the invocation id into reconstruction and skips events belonging to another invocation (workflow/utils/_rehydration_utils.py), passing ic.invocation_id from the scheduler, node runner and replay manager. The port kept the path matching but dropped the invocation scoping, and three pieces were missing to make it work: 1. Interrupt events were not stamped with an invocation id. createRequestInputEvent and createAuthRequestEvent were the only workflow event constructors not setting invocationId, so the events that carry longRunningToolIds could not be attributed to an invocation at all. Every other workflow event site already sets it. 2. A resume did not continue the invocation it resumed. Runner always minted a new invocation id, so rehydration could not have been scoped without breaking resume. Python resolves the id from the resume message before building the context; resolveResumedInvocationId does the same, matching a function response to the interrupt that raised it and otherwise continuing an invocation that still has an unanswered interrupt so plain-text replies keep working. 3. Reconstruction did not filter. reconstructNodeStates and reconstructNodeStatesByPath now take an optional invocation id and skip events from other invocations, mirroring the Python filter including its "no id means no filtering" utility behaviour. The three production call sites pass the current invocation. Two unit tests seeded a prior event without an invocation id and relied on it being visible to a different invocation. They now seed the same invocation, so they still cover fast-forward and plain-text resume, which is what they are about. Adds invocation_scoping_test.ts: a node completed in one invocation is not reused by another, its own invocation still sees it, the unscoped utility form is unchanged, a second message re-runs a gate rather than inheriting its result, and the resume-id resolver adopts, continues or declines an invocation.
|
Thanks for this, and apologies that it sat for eleven days with no reply — the write-up in the description is unusually careful and deserved a faster response than it got. The bug you describe is real, but it was fixed on main two days after you opened this, by #637 (merged 2026-08-11, I want to be straight about the disagreement, because your description anticipated it. #637's doc comment explicitly considers the invocation-id filter and rejects it:
Your point 2 is exactly that premise, and you changed it — the Runner half of this PR makes a resume continue the invocation it resumed, Python-style. That's a real fork in the road and it isn't wrong on its face. But it's the larger of the two changes, since it alters The tree has also moved out from under the branch. It no longer merges (
So I'm closing this as superseded. One genuine ask: if you can still reproduce stale node state on current main — particularly the |
Link to Issue or Description of Change
2. Or, if no issue exists, describe the change:
Problem:
Workflow rehydration matches prior session events by node path alone, so a node
completed in one invocation is fast-forwarded into a later, unrelated invocation
on the same session. A workflow that gates protected work behind a node — an
authorization or credential check, the shape
auth_gate_test.tscovers — skipsthat node on the next message and continues with the earlier decision. If the
caller's entitlement changed in between, the workflow proceeds on the stale one.
The Python implementation this module is a port of does not behave this way. It
threads the invocation id into reconstruction and skips events from another
invocation:
and passes
ic.invocation_idfrom_dynamic_node_scheduler.py,_node_runner.pyandutils/_replay_manager.py. The port kept the pathmatching but dropped the scoping.
Three pieces were missing, which is why the filter could not simply be added:
createRequestInputEventandcreateAuthRequestEventwere the only workflow event constructors notsetting
invocationId, so the events holdinglongRunningToolIdscould notbe attributed to an invocation. Every other workflow event site already sets
it.
Runneralwaysminted a fresh invocation id, so scoping rehydration would have broken every
resume. Python resolves the id from the resume message before building the
context (
runners.py).Solution:
createRequestInputEvent/createAuthRequestEventtake an optionalinvocationIdand stamp it; the two production callers pass theirs.resolveResumedInvocationId(events, newMessage)matches a function responseto the interrupt that raised it, and otherwise continues an invocation that
still has an unanswered interrupt so plain-text replies keep resuming.
Runneruses it, falling back to a new id when nothing is pending.
reconstructNodeStates/reconstructNodeStatesByPathtake an optionalinvocation id and skip events from other invocations, mirroring Python's
filter including its "no id means no filtering" utility behaviour. The three
production call sites pass the current invocation.
Scoping alone would have been wrong: assertion-style resume, HITL resume and the
auth gate all depend on reaching the prior turn's events, which is why the
invocation-continuity half is included rather than left as follow-up.
Testing Plan
Unit Tests:
New
core/test/workflow/invocation_scoping_test.tscovers: a node completed inone invocation is not reused by another; its own invocation still sees it; the
unscoped utility form is unchanged; a second message re-runs a gate rather than
inheriting its result; and the resolver adopts, continues or declines an
invocation.
Two existing unit tests seeded a prior event with no invocation id and relied on
it being visible to a different invocation. They now seed the same invocation,
so they still cover fast-forward and plain-text resume, which is what they are
for. Flagging that explicitly rather than burying it — no assertion was
weakened. The Runner-driven resume tests (
resume_test.ts,dynamic_resume_test.ts,auth_gate_test.ts) were not touched and pass as-is,which is the check that resume still works end to end.
npm run ts:check,npx eslintandnpx prettier --checkare clean on thechanged files;
ts:checkreports the same zero errors before and after.Checklist
Additional context
Based on
aee56e07a47df35bece844a91099c8ee760885aa. Behaviour change worthcalling out for review: a node completed in a previous invocation is no longer
fast-forwarded into a new one. Within an invocation, and across a genuine
resume, fast-forward is unchanged.