Skip to content

fix(workflow): scope node rehydration to the current invocation - #641

Closed
mohass1927 wants to merge 1 commit into
google:mainfrom
mohass1927:fix/scope-workflow-rehydration-to-invocation
Closed

fix(workflow): scope node rehydration to the current invocation#641
mohass1927 wants to merge 1 commit into
google:mainfrom
mohass1927:fix/scope-workflow-rehydration-to-invocation

Conversation

@mohass1927

Copy link
Copy Markdown

Link to Issue or Description of Change

2. Or, if no issue exists, describe the change:

Problem:

Workflow rehydration matches prior session events by node path alone, so a node
completed in one invocation is fast-forwarded into a later, unrelated invocation
on the same session. A workflow that gates protected work behind a node — an
authorization or credential check, the shape auth_gate_test.ts covers — skips
that node on the next message and continues with the earlier decision. If the
caller's entitlement changed in between, the workflow proceeds on the stale one.

The Python implementation this module is a port of does not behave this way. It
threads the invocation id into reconstruction and skips events from another
invocation:

# workflow/utils/_rehydration_utils.py
def _reconstruct_node_states(events, base_path, invocation_id, ...):
  for event in events:
    if invocation_id and event.invocation_id != invocation_id:
      continue

and passes ic.invocation_id from _dynamic_node_scheduler.py,
_node_runner.py and utils/_replay_manager.py. The port kept the path
matching but dropped the scoping.

Three pieces were missing, which is why the filter could not simply be added:

  1. Interrupt events carried no invocation id. createRequestInputEvent and
    createAuthRequestEvent were the only workflow event constructors not
    setting invocationId, so the events holding longRunningToolIds could not
    be attributed to an invocation. Every other workflow event site already sets
    it.
  2. A resume did not continue the invocation it resumed. Runner always
    minted a fresh invocation id, so scoping rehydration would have broken every
    resume. Python resolves the id from the resume message before building the
    context (runners.py).
  3. Reconstruction did not filter.

Solution:

  • createRequestInputEvent / createAuthRequestEvent take an optional
    invocationId and stamp it; the two production callers pass theirs.
  • resolveResumedInvocationId(events, newMessage) matches a function response
    to the interrupt that raised it, and otherwise continues an invocation that
    still has an unanswered interrupt so plain-text replies keep resuming. Runner
    uses it, falling back to a new id when nothing is pending.
  • reconstructNodeStates / reconstructNodeStatesByPath take an optional
    invocation id and skip events from other invocations, mirroring Python's
    filter including its "no id means no filtering" utility behaviour. The three
    production call sites pass the current invocation.

Scoping alone would have been wrong: assertion-style resume, HITL resume and the
auth gate all depend on reaching the prior turn's events, which is why the
invocation-continuity half is included rather than left as follow-up.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.
npx vitest run --project unit:core --project unit:dev
Test Files  219 passed (219)
     Tests  2996 passed (2996)

New core/test/workflow/invocation_scoping_test.ts covers: a node completed in
one invocation is not reused by another; its own invocation still sees it; the
unscoped utility form is unchanged; a second message re-runs a gate rather than
inheriting its result; and the resolver adopts, continues or declines an
invocation.

Two existing unit tests seeded a prior event with no invocation id and relied on
it being visible to a different invocation. They now seed the same invocation,
so they still cover fast-forward and plain-text resume, which is what they are
for. Flagging that explicitly rather than burying it — no assertion was
weakened. The Runner-driven resume tests (resume_test.ts,
dynamic_resume_test.ts, auth_gate_test.ts) were not touched and pass as-is,
which is the check that resume still works end to end.

npm run ts:check, npx eslint and npx prettier --check are clean on the
changed files; ts:check reports the same zero errors before and after.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

Based on aee56e07a47df35bece844a91099c8ee760885aa. Behaviour change worth
calling out for review: a node completed in a previous invocation is no longer
fast-forwarded into a new one. Within an invocation, and across a genuine
resume, fast-forward is unchanged.

Workflow rehydration matched prior session events by node path alone, so a
completed node from one invocation was fast-forwarded into a later, unrelated
invocation on the same session. A workflow that gates protected work behind a
node (an authorization or credential check, as in auth_gate_test) would skip
that node on the next message and continue with the earlier decision.

The Python implementation this module is ported from does not have this: it
threads the invocation id into reconstruction and skips events belonging to
another invocation (workflow/utils/_rehydration_utils.py), passing
ic.invocation_id from the scheduler, node runner and replay manager. The port
kept the path matching but dropped the invocation scoping, and three pieces
were missing to make it work:

1. Interrupt events were not stamped with an invocation id.
   createRequestInputEvent and createAuthRequestEvent were the only workflow
   event constructors not setting invocationId, so the events that carry
   longRunningToolIds could not be attributed to an invocation at all. Every
   other workflow event site already sets it.

2. A resume did not continue the invocation it resumed.
   Runner always minted a new invocation id, so rehydration could not have been
   scoped without breaking resume. Python resolves the id from the resume
   message before building the context; resolveResumedInvocationId does the
   same, matching a function response to the interrupt that raised it and
   otherwise continuing an invocation that still has an unanswered interrupt so
   plain-text replies keep working.

3. Reconstruction did not filter.
   reconstructNodeStates and reconstructNodeStatesByPath now take an optional
   invocation id and skip events from other invocations, mirroring the Python
   filter including its "no id means no filtering" utility behaviour. The three
   production call sites pass the current invocation.

Two unit tests seeded a prior event without an invocation id and relied on it
being visible to a different invocation. They now seed the same invocation, so
they still cover fast-forward and plain-text resume, which is what they are
about.

Adds invocation_scoping_test.ts: a node completed in one invocation is not
reused by another, its own invocation still sees it, the unscoped utility form
is unchanged, a second message re-runs a gate rather than inheriting its result,
and the resume-id resolver adopts, continues or declines an invocation.
@ScottMansfield

Copy link
Copy Markdown
Member

Thanks for this, and apologies that it sat for eleven days with no reply — the write-up in the description is unusually careful and deserved a faster response than it got.

The bug you describe is real, but it was fixed on main two days after you opened this, by #637 (merged 2026-08-11, 3e4f770). That landed eventsForCurrentRun in core/src/workflow/utils/rehydration_utils.ts, which addresses the same stale-fast-forward problem from the other direction: rather than scoping by invocation id, it delimits runs by pausing — an invocation that raised an interrupt belongs to the run still in progress; an invocation that emitted node events without raising one ran to completion and is the boundary, so it and everything before it are dropped.

I want to be straight about the disagreement, because your description anticipated it. #637's doc comment explicitly considers the invocation-id filter and rejects it:

google/adk-python scopes this by invocation id, because there a resumed invocation keeps its id. The TypeScript runner mints a fresh invocation id for every turn (Runner.runAsync), so a run that spans a pause covers several invocations and an id filter would discard the very outputs resume needs.

Your point 2 is exactly that premise, and you changed it — the Runner half of this PR makes a resume continue the invocation it resumed, Python-style. That's a real fork in the road and it isn't wrong on its face. But it's the larger of the two changes, since it alters Runner invocation-id semantics for every agent rather than just workflows, and main already has the bug closed — so the trade no longer favours it.

The tree has also moved out from under the branch. It no longer merges (mergeable: false, dirty):

CONFLICT (content):       core/src/runner/runner.ts
CONFLICT (content):       core/src/workflow/dynamic_node_scheduler.ts
CONFLICT (content):       core/src/workflow/utils/rehydration_utils.ts
CONFLICT (modify/delete): core/src/workflow/workflow_agent.ts        — deleted on main
CONFLICT (modify/delete): core/test/workflow/workflow_agent_test.ts  — deleted on main

workflow_agent.ts is gone from main entirely, so two of the eight source files this touches no longer exist.

So I'm closing this as superseded. One genuine ask: if you can still reproduce stale node state on current main — particularly the auth_gate_test.ts shape you called out, which is the case I'd most want to be certain about — please open an issue with a failing test. That would be valuable and I'll pick it up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants