feat(tools): FunctionTool require_confirmation — HITL approval (Part 7) - #594
feat(tools): FunctionTool require_confirmation — HITL approval (Part 7)#594kalenkevich wants to merge 3 commits into
Conversation
AmaadMartin
left a comment
There was a problem hiding this comment.
Verified against the head tree at c7aac5b. The gate itself is correct for an LlmAgent turn: it runs after schema validation so the predicate sees validated args, it fails closed, denial returns a clean {error: 'This tool call is rejected.'} without executing, and the default is opt-in (requireConfirmation ?? false) with no false convenience knob — which matches Python and is the right default for a generic function wrapper. Two things I would want resolved before this lands: requireConfirmation is not enforced end-to-end for a workflow ToolNode (the request is dropped and the node returns an error string as ordinary output — details inline), and the new plain-text approval fallback lets a single ok approve every pending confirmation in the session, on any surface, not just the CLI.
c7aac5b to
e3e3da5
Compare
Addresses the security/API review on FunctionTool require_confirmation: - The plain-text confirmation fallback no longer runs on every LlmAgent invocation. It is now opt-in via a new `RunConfig.plainTextToolConfirmation` flag (default off), which the interactive `adk run` CLI sets — so on a web/API surface an ordinary chat message is never silently reinterpreted as a tool-gate decision. The structured FunctionResponse path is unchanged. - Harden the fallback itself: resolve only the SINGLE most-recent pending confirmation (never a broadcast across every unanswered gate), require the reply to IMMEDIATELY follow the request (no intervening user turn), and treat unrecognized text as NO decision — the gate stays pending instead of being silently denied (only explicit negatives deny). - Extract a `RequireConfirmation<TParameters>` type with a `toolContext` (not snake_case `tool_context`) parameter, reuse it for both the option and the field, and export it from common.ts. - Correct the `requireConfirmation` doc: the HITL gate is enforced on the LlmAgent path; a workflow ToolNode does not yet route through it (it returns the "requires confirmation" error as node output rather than pausing). - Inline the redundant `await` in runAsync and drop the stale comment.
- Add end-to-end tests that drive a session event list back through RequestConfirmationLlmRequestProcessor with a real LlmAgent + real FunctionTool (no mocks) and assert the original tool is actually re-invoked with the right decision — the step where an id mismatch on resume would show up, and the first coverage of the plain-text fallback: opt-in gating, single-gate binding, unrecognized-text-stays-pending, and no cross-gate broadcast. - Replace the `agent: ... as never` fixture with a real LlmAgent instance so it breaks if InvocationContext's contract changes.
e3e3da5 to
34ed5af
Compare
AmaadMartin
left a comment
There was a problem hiding this comment.
Re-checked at 34ed5af2. All seven findings are resolved, and the two security ones were fixed at the cause rather than papered over: the plain-text path is now opt-in behind runConfig.plainTextToolConfirmation (off everywhere except the CLI), and mapPlainTextConfirmation resolves exactly one gate — the most recent pending id, only when the reply immediately follows it, with unrecognized text left as no decision instead of a silent denial. I traced the immediacy scan by hand: it breaks at any intervening user turn, so a stale gate can't be resolved by a later message.
The round-trip tests are the ones that were missing. They drive a real event list through the real processor with a real LlmAgent and FunctionTool — no mocks — and the no-broadcast case asserts orig-2 is invoked and orig-1 is not, which is the assertion that would have caught the original bug. as never is gone in favour of a real agent, and RequireConfirmation is named, camelCased and exported.
CI: the earlier macOS/Windows reds were timeout flakes (AgentLoader discovery, 40s) in a package this PR doesn't touch; both passed on re-run and all three platforms are green now.
One cosmetic note below. LGTM.
| newMessage: {role: 'user', parts: [{text: query}]}, | ||
| // Interactive CLI: let a plain-text "yes"/"no" resolve a pending tool | ||
| // confirmation (opt-in; off by default on non-interactive surfaces). | ||
| runConfig: {plainTextToolConfirmation: true}, |
There was a problem hiding this comment.
Nit, optional. runFromInputFile is the one call site where "Interactive CLI" isn't true.
// Interactive CLI: let a plain-text "yes"/"no" resolve a pending tool
// confirmation (opt-in; off by default on non-interactive surfaces).
runConfig: {plainTextToolConfirmation: true},This is the --input_file replay path — queries come from a JSON file, nobody is at a prompt. Enabling the flag here is still the right call (the file is authored by whoever runs the CLI, so it's the same trust boundary), but the comment says the opposite of what this function does, and it's the justification a future reader will lean on. The copy at :153 in runInteractively is accurate as written.
Something like "Local CLI surface (interactive prompt or replayed input file)" covers both.
Addresses the security/API review on FunctionTool require_confirmation: - The plain-text confirmation fallback no longer runs on every LlmAgent invocation. It is now opt-in via a new `RunConfig.plainTextToolConfirmation` flag (default off), which the interactive `adk run` CLI sets — so on a web/API surface an ordinary chat message is never silently reinterpreted as a tool-gate decision. The structured FunctionResponse path is unchanged. - Harden the fallback itself: resolve only the SINGLE most-recent pending confirmation (never a broadcast across every unanswered gate), require the reply to IMMEDIATELY follow the request (no intervening user turn), and treat unrecognized text as NO decision — the gate stays pending instead of being silently denied (only explicit negatives deny). - Extract a `RequireConfirmation<TParameters>` type with a `toolContext` (not snake_case `tool_context`) parameter, reuse it for both the option and the field, and export it from common.ts. - Correct the `requireConfirmation` doc: the HITL gate is enforced on the LlmAgent path; a workflow ToolNode does not yet route through it (it returns the "requires confirmation" error as node output rather than pausing). - Inline the redundant `await` in runAsync and drop the stale comment.
- Add end-to-end tests that drive a session event list back through RequestConfirmationLlmRequestProcessor with a real LlmAgent + real FunctionTool (no mocks) and assert the original tool is actually re-invoked with the right decision — the step where an id mismatch on resume would show up, and the first coverage of the plain-text fallback: opt-in gating, single-gate binding, unrecognized-text-stays-pending, and no cross-gate broadcast. - Replace the `agent: ... as never` fixture with a real LlmAgent instance so it breaks if InvocationContext's contract changes.
34ed5af to
64511a1
Compare
64511a1 to
07f828c
Compare
Addresses the security/API review on FunctionTool require_confirmation: - The plain-text confirmation fallback no longer runs on every LlmAgent invocation. It is now opt-in via a new `RunConfig.plainTextToolConfirmation` flag (default off), which the interactive `adk run` CLI sets — so on a web/API surface an ordinary chat message is never silently reinterpreted as a tool-gate decision. The structured FunctionResponse path is unchanged. - Harden the fallback itself: resolve only the SINGLE most-recent pending confirmation (never a broadcast across every unanswered gate), require the reply to IMMEDIATELY follow the request (no intervening user turn), and treat unrecognized text as NO decision — the gate stays pending instead of being silently denied (only explicit negatives deny). - Extract a `RequireConfirmation<TParameters>` type with a `toolContext` (not snake_case `tool_context`) parameter, reuse it for both the option and the field, and export it from common.ts. - Correct the `requireConfirmation` doc: the HITL gate is enforced on the LlmAgent path; a workflow ToolNode does not yet route through it (it returns the "requires confirmation" error as node output rather than pausing). - Inline the redundant `await` in runAsync and drop the stale comment.
- Add end-to-end tests that drive a session event list back through RequestConfirmationLlmRequestProcessor with a real LlmAgent + real FunctionTool (no mocks) and assert the original tool is actually re-invoked with the right decision — the step where an id mismatch on resume would show up, and the first coverage of the plain-text fallback: opt-in gating, single-gate binding, unrecognized-text-stays-pending, and no cross-gate broadcast. - Replace the `agent: ... as never` fixture with a real LlmAgent instance so it breaks if InvocationContext's contract changes.
07f828c to
6e6d170
Compare
6e6d170 to
1e448ca
Compare
Addresses the security/API review on FunctionTool require_confirmation: - The plain-text confirmation fallback no longer runs on every LlmAgent invocation. It is now opt-in via a new `RunConfig.plainTextToolConfirmation` flag (default off), which the interactive `adk run` CLI sets — so on a web/API surface an ordinary chat message is never silently reinterpreted as a tool-gate decision. The structured FunctionResponse path is unchanged. - Harden the fallback itself: resolve only the SINGLE most-recent pending confirmation (never a broadcast across every unanswered gate), require the reply to IMMEDIATELY follow the request (no intervening user turn), and treat unrecognized text as NO decision — the gate stays pending instead of being silently denied (only explicit negatives deny). - Extract a `RequireConfirmation<TParameters>` type with a `toolContext` (not snake_case `tool_context`) parameter, reuse it for both the option and the field, and export it from common.ts. - Correct the `requireConfirmation` doc: the HITL gate is enforced on the LlmAgent path; a workflow ToolNode does not yet route through it (it returns the "requires confirmation" error as node output rather than pausing). - Inline the redundant `await` in runAsync and drop the stale comment.
- Add end-to-end tests that drive a session event list back through RequestConfirmationLlmRequestProcessor with a real LlmAgent + real FunctionTool (no mocks) and assert the original tool is actually re-invoked with the right decision — the step where an id mismatch on resume would show up, and the first coverage of the plain-text fallback: opt-in gating, single-gate binding, unrecognized-text-stays-pending, and no cross-gate broadcast. - Replace the `agent: ... as never` fixture with a real LlmAgent instance so it breaks if InvocationContext's contract changes.
… approval) Part 7/9 of the feature/workflows split. - tools/function_tool: a `requireConfirmation` option so a FunctionTool pauses for human approval before executing. - agents/processors/request_confirmation_llm_request_processor: handles the confirmation request/resume round-trip for such tools. This tool-approval HITL is independent of the workflow engine (it works for any FunctionTool), so it is a small, self-contained slice. Tests: tools/function_tool_confirmation_test (5). Full core suite green (2481), docs:check + tsc clean.
Addresses the security/API review on FunctionTool require_confirmation: - The plain-text confirmation fallback no longer runs on every LlmAgent invocation. It is now opt-in via a new `RunConfig.plainTextToolConfirmation` flag (default off), which the interactive `adk run` CLI sets — so on a web/API surface an ordinary chat message is never silently reinterpreted as a tool-gate decision. The structured FunctionResponse path is unchanged. - Harden the fallback itself: resolve only the SINGLE most-recent pending confirmation (never a broadcast across every unanswered gate), require the reply to IMMEDIATELY follow the request (no intervening user turn), and treat unrecognized text as NO decision — the gate stays pending instead of being silently denied (only explicit negatives deny). - Extract a `RequireConfirmation<TParameters>` type with a `toolContext` (not snake_case `tool_context`) parameter, reuse it for both the option and the field, and export it from common.ts. - Correct the `requireConfirmation` doc: the HITL gate is enforced on the LlmAgent path; a workflow ToolNode does not yet route through it (it returns the "requires confirmation" error as node output rather than pausing). - Inline the redundant `await` in runAsync and drop the stale comment.
- Add end-to-end tests that drive a session event list back through RequestConfirmationLlmRequestProcessor with a real LlmAgent + real FunctionTool (no mocks) and assert the original tool is actually re-invoked with the right decision — the step where an id mismatch on resume would show up, and the first coverage of the plain-text fallback: opt-in gating, single-gate binding, unrecognized-text-stays-pending, and no cross-gate broadcast. - Replace the `agent: ... as never` fixture with a real LlmAgent instance so it breaks if InvocationContext's contract changes.
1e448ca to
b55fa3f
Compare
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Problem: A
FunctionToolmay perform sensitive actions that should pause for human approval before executing.Solution — Part 7 of 8. Stacked on Part 6.
Included:
tools/function_tool.ts— arequireConfirmationoption so aFunctionToolpauses for human approval before running.agents/processors/request_confirmation_llm_request_processor.ts— handles the confirmation request/resume round-trip for such tools.This tool-approval HITL is independent of the workflow engine (it works for any
FunctionTool), so it's a small, self-contained slice and could equally targetmaindirectly.Testing Plan
Tests:
tools/function_tool_confirmation_test(5). Full core suite 2481 green; docs:check + tsc clean.Manual E2E: N/A (unit-level; the request/resume round-trip is covered by the test).
Checklist
Additional context
Stacked split — merge in order (…Part 6 → Part 7 → Part 8). Diff: 3 files, +317.