feat(flows): ground the builder in the LIVE Composio catalog + enforce tool contracts#4605
Conversation
…e tool contracts
The builder authored graphs against GUESSED tool contracts — hallucinated slugs
(SLACK_POST_MESSAGE_TO_CHANNEL → runtime 404), guessed output paths (split_out
path=items vs Gmail's real data.messages), guessed args/fields — because
search_tool_catalog only searched a STATIC curated subset and nothing validated
what the builder wrote. Proven live: a classify-emails flow nulled its whole chain
(split found no items) and 404'd on a fake Slack slug.
Systemic fix — one ToolContract sourced from the FULL live Composio catalog per
named app, grounded AND enforced:
Grounding (PR1):
- caps.rs: ToolContract {slug, required_args, input/output_schema, output_fields,
primary_array_path, is_curated} + unified LIVE_CATALOG_CACHE. fetch_live_toolkit_
catalog() pulls a toolkit's REAL actions via direct_list_tools / backend list_tools
(bypassing the curated filter), degrading gracefully when output_parameters is
absent. compute_primary_array_path() derives the nested array path (data.messages).
composio_required_args/response_fields become thin projections; the two old caches
are removed.
- builder_tools.rs: search_tool_catalog now searches the LIVE catalog (curated-first
ranking, never hides uncurated matches) and returns required_args/output_fields/
primary_array_path/featured; new get_tool_contract tool returns the full contract.
Registered in the builder tool belt + prompt.md teaches: discover live, wire from
required_args/output_fields, split via json.<primary_array_path>.
Enforcement (PR2):
- ops.rs validate_tool_contracts: HARD-reject an unreal slug (kills the 404) or a
missing required arg; advisory warnings for bad output-field bindings + split paths
(suggests json.<primary_array_path>). Wired into propose/revise/save.
- caps.rs flow_tool_allowed: Path B now requires live-connection AND the slug existing
in the toolkit's live catalog, then applies classify_unknown scope gating — strictly
narrower, never loosened.
Subsumes tinyhumansai#4591 (its response-fields cache folds into LIVE_CATALOG_CACHE); orthogonal
to tinyhumansai#4590. cargo check/fmt clean; flows/tinyflows/composio/tools 1281 tests pass.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 7 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughIntroduces a live Composio tool-catalog cache ( ChangesLive Tool-Contract Validation Feature
Estimated code review effort: 4 (Complex) | ~75 minutes Sequence Diagram(s)sequenceDiagram
participant Agent as Workflow Builder Agent
participant SearchTool as search_tool_catalog
participant ContractTool as get_tool_contract
participant LiveCache as LIVE_CATALOG_CACHE
participant SaveTool as save_workflow
Agent->>SearchTool: query for candidate slugs
SearchTool->>LiveCache: fetch_live_toolkit_catalog(toolkit)
LiveCache-->>SearchTool: ToolContract matches
SearchTool-->>Agent: slug, required_args, output_fields
Agent->>ContractTool: get_tool_contract{slug}
ContractTool->>LiveCache: fetch_live_toolkit_catalog(toolkit)
LiveCache-->>ContractTool: full ToolContract
ContractTool-->>Agent: required_args, output_fields, primary_array_path
Agent->>SaveTool: save_workflow(graph)
SaveTool->>LiveCache: validate_tool_contracts(config, graph)
LiveCache-->>SaveTool: contract errors (if any)
alt errors found
SaveTool-->>Agent: reject with node/slug/missing-arg details
else no errors
SaveTool-->>Agent: workflow persisted
end
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d00b5a171
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let Some(contract) = catalog.iter().find(|c| c.slug.eq_ignore_ascii_case(slug)) else { | ||
| tracing::warn!( | ||
| target: "flows", | ||
| node = %node.id, | ||
| %slug, | ||
| %toolkit, | ||
| "[flows] tool-contract check: slug is not a real action in the live catalog — rejecting" | ||
| ); | ||
| errors.push(format!( | ||
| "Node '{}': `{slug}` is not a real action in the `{toolkit}` toolkit's live \ | ||
| Composio catalog — use search_tool_catalog {{ query: ..., toolkit: \"{toolkit}\" \ | ||
| }} to find a real action slug.", | ||
| node.id | ||
| )); | ||
| continue; | ||
| }; |
There was a problem hiding this comment.
When a live contract is found here, validate_tool_contracts lets the graph pass regardless of contract.is_curated, and search_tool_catalog now returns those uncurated live actions. For toolkits that already have a static catalog (e.g. Gmail/Slack), the runtime gate in flow_tool_allowed still takes Path A and rejects any slug not found by find_curated, so a real but uncurated action can be proposed/saved successfully and then fail every run as tool not permitted. Either keep these actions out of authoring/validation for statically cataloged toolkits or update the runtime gate to allow the same live-contract set.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 14e289e — confirmed valid by reading flow_tool_allowed's Path A (src/openhuman/tinyflows/caps.rs): for a toolkit with a static curated catalog (gmail/slack/notion/...), the runtime dispatch gate hard-rejects any slug find_curated doesn't recognize, even a real live action. validate_tool_contracts only checked "is this a real live-catalog action", never contract.is_curated, so a real-but-uncurated action on a curated toolkit could pass authoring/save and then fail every run.
Took your first suggested direction (tighten authoring to match the runtime gate, rather than loosen the runtime gate — the runtime gate is the security-relevant one here). validate_tool_contracts now rejects a tool_call whose toolkit has a static curated catalog (via the same get_provider(...).curated_tools().or_else(catalog_for_toolkit(...)) lookup flow_tool_allowed uses) unless the matched contract is curated. search_tool_catalog/get_tool_contract are unchanged — they still surface uncurated real actions for discovery, per ToolContract::is_curated's doc (ranking signal only) — only the hard save-time gate now matches runtime enforcement.
Added a regression test (validate_tool_contracts_rejects_a_real_but_uncurated_action_on_a_statically_catalogued_toolkit in ops_tests.rs) and updated the existing seeded_slack_send_contract() fixture to is_curated: true (it was arbitrarily false before, which no longer reflects a real curated Slack action now that the gate checks it).
coderabbitai
Bot
added
bug
feature
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
src/openhuman/flows/ops.rs (1)
236-300: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoffConsider memoizing the per-toolkit catalog across the wiring passes.
fetch_live_toolkit_catalogreturns an ownedVec<ToolContract>, so each call clones the whole toolkit catalog out of the cache. Here it's invoked once per matching expression, and the same pattern repeats per-node invalidate_tool_contractsand per-edge ingraph_split_out_path_warnings. For a graph with severaltool_callnodes/bindings in the same toolkit,graph_wiring_warningsnow performs many full-catalog clones per authoring call. A smallHashMap<toolkit, Vec<ToolContract>>memo (or fetching once per distinct toolkit up front) would collapse this to one clone per toolkit per pass.Advisory only — graphs are usually small, so this is a nice-to-have rather than a hot path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhuman/flows/ops.rs` around lines 236 - 300, The wiring checks in graph_output_field_warnings, validate_tool_contracts, and graph_split_out_path_warnings are repeatedly calling fetch_live_toolkit_catalog for the same toolkit, causing unnecessary full catalog clones. Add a small per-pass memoization layer keyed by toolkit (for example a HashMap from toolkit to Vec<ToolContract>, or prefetch distinct toolkits before iterating) and reuse the cached catalog within each pass so repeated tool_call bindings/edges only trigger one clone per toolkit.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhuman/flows/agents/workflow_builder/prompt.md`:
- Around line 197-229: Clarify the workflow guidance so `get_tool_contract` is
described as a grounding step for a known slug, not a discovery/search
primitive. In the prompt text around `search_tool_catalog` and
`get_tool_contract`, separate the responsibilities clearly: use
`search_tool_catalog` to find candidate slugs, then use `get_tool_contract` to
resolve that slug’s toolkit and fetch the full contract for wiring required
args, output fields, and primary_array_path.
In `@src/openhuman/flows/builder_tools_tests.rs`:
- Around line 205-216: The live-catalog seed in this test is using the shared
gmail cache key, which can be overwritten by other parallel tests and make the
ranking assertion flaky. Update the setup in
search_live_catalog_ranks_curated_before_uncurated_without_hiding_either to seed
a unique toolkit key via seed_live_catalog_cache and query search_live_catalog
with that same unique key so the test is isolated from other #[tokio::test]
cases.
In `@src/openhuman/tinyflows/tests.rs`:
- Around line 426-433: The shared LIVE_CATALOG_CACHE seed key is reused by
multiple tests, which can cause flaky preflight assertions under parallel
execution. Update preflight_fails_before_dispatch_naming_the_missing_field and
preflight_invoker_gates_the_mock_tool_path to use distinct toolkit/slug values
in their seed_live_catalog_cache setup, or explicitly clear the cache between
tests so each test gets an isolated schema. Use the existing
seed_live_catalog_cache and seeded_required_args_contract calls as the places to
adjust.
---
Nitpick comments:
In `@src/openhuman/flows/ops.rs`:
- Around line 236-300: The wiring checks in graph_output_field_warnings,
validate_tool_contracts, and graph_split_out_path_warnings are repeatedly
calling fetch_live_toolkit_catalog for the same toolkit, causing unnecessary
full catalog clones. Add a small per-pass memoization layer keyed by toolkit
(for example a HashMap from toolkit to Vec<ToolContract>, or prefetch distinct
toolkits before iterating) and reuse the cached catalog within each pass so
repeated tool_call bindings/edges only trigger one clone per toolkit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 0fee9db1-482c-4a9d-9350-a1a5a9ba4e7b
📒 Files selected for processing (11)
src/openhuman/composio/client.rssrc/openhuman/flows/agents/workflow_builder/agent.tomlsrc/openhuman/flows/agents/workflow_builder/prompt.mdsrc/openhuman/flows/builder_tools.rssrc/openhuman/flows/builder_tools_tests.rssrc/openhuman/flows/ops.rssrc/openhuman/flows/ops_tests.rssrc/openhuman/flows/tools.rssrc/openhuman/tinyflows/caps.rssrc/openhuman/tinyflows/tests.rssrc/openhuman/tools/ops.rs
validate_tool_contracts let a graph pass authoring/save whenever a slug was ANY real live-catalog action, regardless of contract.is_curated. But flow_tool_allowed's Path A (the runtime dispatch gate) hard-rejects any slug find_curated doesn't recognize for a toolkit that ships a static curated catalog (gmail, slack, notion, ...). A real-but-uncurated action on one of those toolkits could therefore pass validation and save successfully, then fail every single run as "tool not permitted". Mirror the runtime gate at authoring time instead of loosening the runtime gate: reject a tool_call whose toolkit has a static curated catalog unless the matched contract is curated. search_tool_catalog still surfaces uncurated real actions (ranking signal only, per ToolContract::is_curated's doc) — this only tightens the hard save-time gate to match what would actually be allowed to run. Addresses @chatgpt-codex-connector on src/openhuman/flows/ops.rs:781.
The "app not connected yet?" bullet described search_tool_catalog and get_tool_contract as if both search the live catalog, blurring the line between discovery (search by keyword) and grounding (resolve a known slug's full contract). Split the two responsibilities out explicitly so the builder agent doesn't treat get_tool_contract as a second search primitive. Addresses @coderabbitai on src/openhuman/flows/agents/workflow_builder/prompt.md:229.
LIVE_CATALOG_CACHE is a process-global cache, shared by every
#[tokio::test] in the same binary. Several tests seeded the SAME
toolkit key ("gmail"/"slack") with DIFFERENT contract content
(different is_curated, different required_args, or a different item
count), so a concurrently-running test could overwrite another's
cache entry between its seed and its assertion:
- search_live_catalog_ranks_curated_before_uncurated_without_hiding_either
needs an exact 2-item, exact-order result but shared the "gmail" key
with ~11 other tests seeding a single curated item.
- search_tool_catalog_degrades_gracefully_when_output_schema_unknown
seeded "slack" with is_curated: false, racing the curated-catalog
gate now enforced in validate_tool_contracts (previous commit) against
ops_tests.rs's "slack"-keyed tests.
- preflight_fails_before_dispatch_naming_the_missing_field and
preflight_invoker_gates_the_mock_tool_path both seeded "gmail" with
DIFFERENT required_args lists under the same slug.
Give each of these its own fictional toolkit/slug (verified none collide
with any other seed_live_catalog_cache key in the repo) instead of
reusing the shared "gmail"/"slack" keys. Verified fixed by stress-running
the affected modules at --test-threads=16 (previously reproduced the
failure within a handful of runs; now stable across 10 consecutive
runs).
Addresses @coderabbitai on src/openhuman/flows/builder_tools_tests.rs:216
and src/openhuman/tinyflows/tests.rs:433.
2 participants
The root cause (one problem behind every symptom)
The builder authored graphs against guessed tool contracts — hallucinated slugs (
SLACK_POST_MESSAGE_TO_CHANNEL→ runtime 404), guessed output paths (split_out.path="items"vs Gmail's realdata.messages), guessed args/fields — becausesearch_tool_catalogonly searched a static curated subset and nothing validated what the builder wrote. Proven live: a classify-emails flow nulled its whole chain (split found no items) and 404'd on a fake Slack slug.The systemic fix: one
ToolContract, from the LIVE catalog, grounded + enforcedGrounding —
search_tool_catalog+ newget_tool_contractpull each named app's full real actions from Composio (list_tool_schemas_v3/backend list), returning required args + input/output schema + computedprimary_array_path(the nested array path, e.g.data.messages). UnifiedLIVE_CATALOG_CACHE(replaces the two per-field caches). Curated list → ranking hint only, never a boundary.Enforcement —
validate_tool_contractsat propose/revise/save: hard-reject an unreal slug (kills the 404) or a missing required arg; warn on bad output-field bindings + split paths (suggestsjson.<primary_array_path>). Runtime gate (flow_tool_allowed) now checks the live catalog for slug existence + preserves scope gating (strictly narrower, never loosened).So the builder can use every real action of any app you name — and can no longer hallucinate a slug, arg, or output field.
Reconciliation
Subsumes #4591 (its response-fields cache folds into
LIVE_CATALOG_CACHE); orthogonal to #4590.Follow-up
Vendored
split_outdotted-path (json.data.messages) + its validation land next (submodule change) — that's the last piece of the nested-path bug; this PR already grounds + suggests the correct path.Verification
cargo check/fmtclean · flows/tinyflows/composio/tools: 1281 tests pass (incl. new tests forcompute_primary_array_path, live catalog fetch/search,get_tool_contract,validate_tool_contracts, and the tightened runtime gate).Summary by CodeRabbit
New Features
Bug Fixes