Toward SearchableToolSet and cross-model ToolSearch #3680

t0yv0 · 2025-12-09T13:03:41Z

Closes Support deferred loading of tools and discovery via tool search tool #3590
Related to feat: Add ToolSearchTool and defer_loading for dynamic tool discovery #3620

DouweM · 2025-12-09T17:36:33Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+class SearchableToolset(AbstractToolset[AgentDepsT]):
+    """A toolset that implements tool search and deferred tool loading."""
+
+    toolset: AbstractToolset[AgentDepsT]


Have a look at WrapperToolset which already handles this + properly forwards __aexit__ and __aenter__!

t0yv0 · 2025-12-11T12:27:35Z

test_searchable_example.py

@@ -0,0 +1,136 @@
+"""Minimal example to test SearchableToolset functionality.


It looks like proper tests need to go into:

test_toolsets.py has space for unit tests

somewhere there are VCR cassettes that record an interaction with an LLM could be useful here

I just wanted to get something quick to iterate with an actual LLM. This ended up working on Claude but took a few iterations on the prompt. The model seemed sensitive to how the "search tool" is called and the content of the description - it would either refuse to load it or start asking for user confirmation before loading it. It took some tweaking to get the current description to pass this simple test.

❯ uv run python test_searchable_example.py ============================================================ Testing SearchableToolset ============================================================ Test 1: Calculation task ------------------------------------------------------------ 2025-12-11 07:20:48,189 - root - DEBUG - SearchableToolset.get_tools 2025-12-11 07:20:48,189 - root - DEBUG - SearchableToolset.get_tools ==> ['load_tools'] Result: I can calculate that for you directly. 123 multiplied by 456 equals **56,088**. Test 2: Database task ------------------------------------------------------------ 2025-12-11 07:20:50,983 - root - DEBUG - SearchableToolset.get_tools 2025-12-11 07:20:50,984 - root - DEBUG - SearchableToolset.get_tools ==> ['load_tools'] 2025-12-11 07:20:54,254 - root - DEBUG - SearchableToolset.call_tool(load_tools, {'regex': 'database|sql|table|query'}) ==> ['fetch_user_data', 'list_database_tables'] 2025-12-11 07:20:54,255 - root - DEBUG - SearchableToolset.get_tools 2025-12-11 07:20:54,255 - root - DEBUG - SearchableToolset.get_tools ==> ['load_tools', 'fetch_user_data', 'list_database_tables'] 2025-12-11 07:20:57,735 - root - DEBUG - SearchableToolset.call_tool(list_database_tables, {}) ==> ['users', 'orders', 'products', 'reviews'] 2025-12-11 07:20:57,735 - root - DEBUG - SearchableToolset.call_tool(fetch_user_data, {'user_id': 42}) ==> {'id': 42, 'name': 'John Doe', 'email': '[email protected]'} 2025-12-11 07:20:57,735 - root - DEBUG - SearchableToolset.get_tools 2025-12-11 07:20:57,736 - root - DEBUG - SearchableToolset.get_tools ==> ['load_tools', 'fetch_user_data', 'list_database_tables'] Result: Perfect! Here are the results: **Database Tables:** - users - orders - products - reviews **User 42 Data:** - ID: 42 - Name: John Doe - Email: [email protected] Test 3: Weather task ------------------------------------------------------------ 2025-12-11 07:21:00,605 - root - DEBUG - SearchableToolset.get_tools 2025-12-11 07:21:00,607 - root - DEBUG - SearchableToolset.get_tools ==> ['load_tools', 'fetch_user_data', 'list_database_tables'] 2025-12-11 07:21:04,597 - root - DEBUG - SearchableToolset.call_tool(load_tools, {'regex': 'weather'}) ==> ['get_weather'] 2025-12-11 07:21:04,598 - root - DEBUG - SearchableToolset.get_tools 2025-12-11 07:21:04,599 - root - DEBUG - SearchableToolset.get_tools ==> ['load_tools', 'get_weather', 'fetch_user_data', 'list_database_tables'] 2025-12-11 07:21:07,769 - root - DEBUG - SearchableToolset.call_tool(get_weather, {'city': 'San Francisco'}) ==> The weather in San Francisco is sunny and 72°F 2025-12-11 07:21:07,770 - root - DEBUG - SearchableToolset.get_tools 2025-12-11 07:21:07,771 - root - DEBUG - SearchableToolset.get_tools ==> ['load_tools', 'get_weather', 'fetch_user_data', 'list_database_tables'] Result: The weather in San Francisco is currently sunny and 72°F - a beautiful day!

t0yv0 · 2025-12-11T12:29:33Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+from ..tools import ToolDefinition
+from .abstract import AbstractToolset, SchemaValidatorProt, ToolsetTool
+
+_SEARCH_TOOL_NAME = 'load_tools'


Another curious bit is that when tool was called "more_tools", I hit a crash:

Traceback (most recent call last): File "/Users/anton/code/pydantic-ai/test_searchable_example.py", line 136, in <module> asyncio.run(main()) File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/asyncio/runners.py", line 195, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/Users/anton/code/pydantic-ai/test_searchable_example.py", line 123, in main result = await agent.run("Can you list the database tables and then fetch user 42?") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/agent/abstract.py", line 226, in run async with self.iter( ^^^^^^^^^^ File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/contextlib.py", line 231, in __aexit__ await self.gen.athrow(value) File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/agent/__init__.py", line 658, in iter async with graph.iter( ^^^^^^^^^^^ File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/contextlib.py", line 231, in __aexit__ await self.gen.athrow(value) File "/Users/anton/code/pydantic-ai/pydantic_graph/pydantic_graph/beta/graph.py", line 270, in iter async with GraphRun[StateT, DepsT, OutputT]( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anton/code/pydantic-ai/pydantic_graph/pydantic_graph/beta/graph.py", line 423, in __aexit__ await self._async_exit_stack.__aexit__(exc_type, exc_val, exc_tb) File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/contextlib.py", line 754, in __aexit__ raise exc_details[1] File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/contextlib.py", line 735, in __aexit__ cb_suppress = cb(*exc_details) ^^^^^^^^^^^^^^^^ File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/Users/anton/code/pydantic-ai/pydantic_graph/pydantic_graph/beta/graph.py", line 978, in _unwrap_exception_groups raise exception File "/Users/anton/code/pydantic-ai/pydantic_graph/pydantic_graph/beta/graph.py", line 750, in _run_tracked_task result = await self._run_task(t_) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anton/code/pydantic-ai/pydantic_graph/pydantic_graph/beta/graph.py", line 779, in _run_task output = await node.call(step_context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anton/code/pydantic-ai/pydantic_graph/pydantic_graph/beta/step.py", line 253, in _call_node return await node.run(GraphRunContext(state=ctx.state, deps=ctx.deps)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py", line 576, in run async with self.stream(ctx): ^^^^^^^^^^^^^^^^ File "/Users/anton/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/contextlib.py", line 217, in __aexit__ await anext(self.gen) File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py", line 590, in stream async for _event in stream: File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py", line 716, in _run_stream async for event in self._events_iterator: File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py", line 677, in _run_stream async for event in self._handle_tool_calls(ctx, tool_calls): File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py", line 732, in _handle_tool_calls async for event in process_tool_calls( File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py", line 925, in process_tool_calls ctx.state.increment_retries(ctx.deps.max_result_retries, model_settings=ctx.deps.model_settings) File "/Users/anton/code/pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py", line 127, in increment_retries raise exceptions.UnexpectedModelBehavior(message) pydantic_ai.exceptions.UnexpectedModelBehavior: Exceeded maximum retries (1) for output validation

Interesting, that suggests that the model was not calling it correctly (wrong args possibly). I suggest adding https://ai.pydantic.dev/logfire/ so you can easily see what's happening behind the scenes in an agent run.

DouweM · 2025-12-11T23:06:14Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+from ..tools import ToolDefinition
+from .abstract import AbstractToolset, SchemaValidatorProt, ToolsetTool
+
+_SEARCH_TOOL_NAME = 'load_tools'


Interesting, that suggests that the model was not calling it correctly (wrong args possibly). I suggest adding https://ai.pydantic.dev/logfire/ so you can easily see what's happening behind the scenes in an agent run.

DouweM · 2025-12-11T23:08:25Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+    regex: str
+
+
+def _search_tool_def() -> ToolDefinition:


Check out Tool.from_schema and the Tool constructor that takes a function (as used by FunctionToolset) for easier ways to construct a single tool. The function approach is the easiest by far

DouweM · 2025-12-11T23:10:37Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+        description="""Search and load additional tools to make them available to the agent.
+
+DO call this to find and load more tools needed for a task.
+NEVER ask the user if you should try loading tools, just try.


Hmm, I see you explained below that this was needed to pass the tests, even for Sonnet 4.5, but tokens are expensive so it'll be worth another iteration on this.

DouweM · 2025-12-11T23:11:51Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+        parameters_json_schema={
+            'type': 'object',
+            'properties': {
+                'regex': {


I like pattern slightly better as an argument name, as we may at some point support different ones. Although it is very helpful to the model in knowing what to put here, in case we remove/shorted the description.

DouweM · 2025-12-11T23:13:58Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+        all_tools: dict[str, ToolsetTool[AgentDepsT]] = {}
+        all_tools[_SEARCH_TOOL_NAME] = _SearchTool(
+            toolset=self,
+            max_retries=1,


We may want to increase this, to give the model a few chances to fix its regex, if it submitted an invalid one the first time

DouweM · 2025-12-11T23:15:20Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+    ) -> Any:
+        if isinstance(tool, _SearchTool):
+            adapter = TypeAdapter(_SearchToolArgs)
+            typed_args = adapter.validate_python(tool_args)


Arguments will/should already have been validated by this point when used through ToolManager/Agent!

Interesting, was not obvious from the types, but sounds like I can just cast this. Thanks.

Can the tool_args parameter be _SearchToolArgs?

DouweM · 2025-12-11T23:16:17Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+        matching_tool_names: list[str] = []
+
+        for tool_name, tool in toolset_tools.items():
+            rx = re.compile(args['regex'])


This'll be more efficient one line up :)

DouweM · 2025-12-11T23:16:40Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+
+        for tool_name, tool in toolset_tools.items():
+            rx = re.compile(args['regex'])
+            if rx.search(tool.tool_def.name) or rx.search(tool.tool_def.description):


For error handling, check out the ModelRetry exception

DouweM · 2025-12-11T23:19:11Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+    """A toolset that implements tool search and deferred tool loading."""
+
+    toolset: AbstractToolset[AgentDepsT]
+    _active_tool_names: set[str] = field(default_factory=set)


The fact that this has instance variables means that it can't be reused across multiple agent runs, even though the same instance is registered to an agent just once... We had a similar issue with DynamicToolset, I suggest having a look at how we handle it there. We could also leverage ctx.run_id to store state on the same SearchableToolset instance, but separate it for each agent run.

This comment was the source of run_id indexing exploration --^

t0yv0 · 2025-12-14T02:49:03Z

Largely following #3620 for the anthropic bulit-in. This can coexist I think with SearchableTool as suggested, no big surprises here. Couple small things to note:

Claude seems ornery about the weather example and does not find the weather tool, refuses to search using its builtin tool. Perhaps that's just overindexing on "weather". This is related to removing the extra instructions for the custom search tool.
Need to have another look if both Tool and ToolDefinition need defer_loading param added or there's a more concise way, perhaps a Toolset combinator that flips the defer_loading bit for all the ToolDefinitions it manages.

DouweM · 2025-12-15T19:39:15Z

pydantic_ai_slim/pydantic_ai/profiles/__init__.py

    This is currently only used by `OpenAIChatModel`, `HuggingFaceModel`, and `GroqModel`.
    """

+    supports_tool_search: bool = False


In #3456, we're getting supported_builtin_tools so we won't need this dedicated field anymore

DouweM · 2025-12-15T19:40:01Z

pydantic_ai_slim/pydantic_ai/profiles/anthropic.py

        thinking_tags=('<thinking>', '</thinking>'),
        supports_json_schema_output=supports_json_schema_output,
        json_schema_transformer=AnthropicJsonSchemaTransformer,
+        supports_tool_search=True,


Note that this is not actually supported by all models: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool

DouweM · 2025-12-15T19:41:26Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+        needs_tool_search = any(tool.get('defer_loading') for tool in tools)
+
+        if needs_tool_search:
+            beta_features.add('advanced-tool-use-2025-11-20')


Note that different providers use different headers: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool

DouweM · 2025-12-15T19:41:52Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

        )
    elif item.name in ('bash_code_execution', 'text_editor_code_execution'):  # pragma: no cover
        raise NotImplementedError(f'Anthropic built-in tool {item.name!r} is not currently supported.')
    elif item.name in ('tool_search_tool_regex', 'tool_search_tool_bm25'):  # pragma: no cover


Let's move this up so that the NotImplementedErrors are last

DouweM · 2025-12-15T19:48:30Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+
+    async def get_tools(self, ctx: RunContext[AgentDepsT]) -> dict[str, ToolsetTool[AgentDepsT]]:
+        # Models that support built-in tool search are exposed to all the tools as-is.
+        if ctx.model.profile.supports_tool_search:


Unfortunately, this is not going to work correctly with FallbackModel, as I realized and edited into #3666 (comment) a few days ago:

(Edit: Checking ctx.model.profile.supports_deferred_loading from inside SearchableToolset.get_tools is not going work right when the model is a FallbackModel, as we wouldn't be checking the underlying models' abilities. So similar to #3212, we likely need to arrange things such that the model classes themselves can determine how to handle the search tool and deferred-loading tools...)

#3212 (comment) sketches out an approach for how a toolset could return both an AbstractBuiltinTool, plus tool definitions to use when that builtin tool is not available, so that the model itself can determine which to send to the API.

In the case of tool search, we'd need to have either the built-in ToolSearchTool + tool defs with defer_loading=True, or the custom search_tool + those tool defs that have already been revealed (without defer_loading=True).

The implementation I sketched in that comment does not yet account for an abstract tool coming with its own set of tool definitions.

DouweM · 2025-12-15T19:51:51Z

pydantic_ai_slim/pydantic_ai/toolsets/function.py

            sequential=sequential,
            requires_approval=requires_approval,
            metadata=metadata,
+            defer_loading=defer_loading,


We need to also add this field to the agent.tool and agent.tool_plain decorators.

DouweM · 2025-12-15T19:53:35Z

pydantic_ai_slim/pydantic_ai/builtin_tools.py

+    To defer loading a tool's definition until the model finds it, mark it as `defer_loading=True`.
+
+    Note that only models with `ModelProfile.supports_tool_search` use this builtin tool. These models receive all tool
+    definitions and natively implement search and loading. All other models rely on `SearchableToolset` instead.


I'd rather explain this in the future Tools docs section on defer_loading, since the primary way users will interact with this feature will be through that field rather than this builtin tool (see above)

DouweM · 2025-12-15T19:57:03Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+            # TODO proper error handling
+            assert tool_name != _SEARCH_TOOL_NAME
+
+            # Expose the tool unless it defers loading and is not yet active.


When a user wraps a toolset in SearchableToolset, I think we should assume they mean for all of them to be defer_loading. Or perhaps we can support bool | None, and have None be interpreted as False normally, but as True here, with an explicit False still being respected here and having it always be surfaced.

DouweM · 2025-12-15T19:59:21Z

pydantic_ai_slim/pydantic_ai/builtin_tools.py

+class ToolSearchTool(AbstractBuiltinTool):
+    """A builtin tool that searches for tools during dynamic tool discovery.
+
+    To defer loading a tool's definition until the model finds it, mark it as `defer_loading=True`.


As mentioned in the penultimate paragraph in #3666 (comment), I envision that users will primarily interact with this feature via the defer_loading=True field on tools (rather than through ToolSearchTool or SearchableToolset), which we'd detect automatically and handle by wrapping those tools in SearchableToolset. Then, as implemented, that would fall back on Anthropic's native functionality when appropriate.

So I'd love to see a test where we register a few @agent.tool(defer_loading=True)s and then test that it works as expected with both Anthropic and OpenAI.

A test is added for anthropic and openai; note that anthropic still wants to have a builtin_tools class so a class probably should stay here, but is not particularly user-facing. Your comment on this being the wrong place to document things stands.

Yeah I think we still want this built-in tool so users can change the settings.

- automatically inject SerchableToolset - taint ToolDefinition by wrapping the with an active model recall flag - all ToolDefinitions get passed down to the model - last-minute filtering is done at the model level - tool active state is indexed on run_id

t0yv0 · 2025-12-20T19:57:26Z

Some progress, we now auto-inject SerchableToolset to handle defer_loading=True automatically.

At the injection point the code does not know if the target model supports builtin tool search or not.

Therefore SearchableToolset now always exposes all the tools. It cannot filter since models without builtin search will
need to see those. Instead, SearchableToolset tags the ToolDefinition objects metadata field with an active
predicate.

The models now need to do a list-minute filtering before sending the tools to the model provider.

Models with builtin search such as Claude drop instances of the SearchTool. They propagate defer_loading=True tools
to the model provider since it will be able to natively take care of placing it on the context at the appropriate time.

Models without builtin search accept the SearchTool but filter out defer_loading=True tool definitions based on the
active metadata marker. Inactive tools are dropped and not exposed to the model provider.

t0yv0 · 2025-12-20T20:18:59Z

Nesting SearchableToolset with auto-injected SearchableToolset hits an error, need to make sure nesting works.
Not only open_ai.py but all models will need to implement appropriate ToolDefinition filtering - this is awkward
The way anthropic.py needs to inject the builtin search tool is a little awkward as well

Perhaps with #3212 can later help with these.

t0yv0 · 2025-12-20T20:33:57Z

pydantic_ai_slim/pydantic_ai/models/openai.py

-    def _get_tools(self, model_request_parameters: ModelRequestParameters) -> list[chat.ChatCompletionToolParam]:
-        return [self._map_tool_definition(r) for r in model_request_parameters.tool_defs.values()]
+    def _run_id(self, messages: list[ModelMessage]) -> str:
+        # Take the last run_id found in messages. TODO can we assume one is always found?


@DouweM this is not obvious from the types but would you say some assumptions here can be made? Can we recover a run_id from list[ModelMessage]? Will it be unique or is taking the last one reasonable? If we cannot recover this, I need to rework to str | None perhaps. Where this is used is making sure the active tool filtering state is indexed on run_id and has the appropriate lifetime. Perhaps there's as more direct way to implement this you can suggest.

ModelRequest.run_id is runtime-guaranteed to have a value here, so we could assert that it exists.

DouweM · 2025-12-22T19:56:12Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

            toolset = CombinedToolset([output_toolset, toolset])

-        return toolset
+        return SearchableToolset(toolset)


This makes sense. I'm thinking now that maybe we should not allow users to explicitly create a SearchableToolset at all, as there can only be one tool search tool without the model getting confused anyway, so it wouldn't make sense to have more than one of these toolsets registered to an agent. So I think this should be the only place we use it, and the toolsets.searchable module should be made private.

This automatically picks up all ToolDefinitions with defer_loading=True, which we've made easy to set using @agent.tool(...) and FunctionToolset function tools, but users will also want to enabled deferred loading of e.g. MCPServer tool definitions. The field could currently be set using a wrapper PreparedToolset, but that'd be a bit awkward, so we may want a new constructor arg on MCPServer (bool | list[str] = None, I think), and/or a new wrapper DeferredLoadingToolset similar to ApprovalRequiredToolset.

Hmm if the user can't create the SearchableToolset manually anymore, where would they set e.g. the search algorithm, once we support multiple?

Yes that is a good question. It seems that we should try to expose SearchableToolset, I'm not sure I can suggest a good alternative to capture the settings if that is not exposed. Another parameter in the Agent() constructor perhaps?

And if we do expose SearchableToolset then it needs to be fully composable as well, so SearchableToolset(SearchableToolset()), I had some notes elsewhere that the current implementation is not robust to that.

Hm, the agent constructor arg would not allow customizing two search algorithms in the same agent.

DouweM · 2025-12-22T19:57:06Z

pydantic_ai_slim/pydantic_ai/builtin_tools.py

+class ToolSearchTool(AbstractBuiltinTool):
+    """A builtin tool that searches for tools during dynamic tool discovery.
+
+    To defer loading a tool's definition until the model finds it, mark it as `defer_loading=True`.


Yeah I think we still want this built-in tool so users can change the settings.

DouweM · 2025-12-22T19:59:06Z

pydantic_ai_slim/pydantic_ai/models/openai.py

-    def _get_tools(self, model_request_parameters: ModelRequestParameters) -> list[chat.ChatCompletionToolParam]:
-        return [self._map_tool_definition(r) for r in model_request_parameters.tool_defs.values()]
+    def _run_id(self, messages: list[ModelMessage]) -> str:
+        # Take the last run_id found in messages. TODO can we assume one is always found?


ModelRequest.run_id is runtime-guaranteed to have a value here, so we could assert that it exists.

DouweM · 2025-12-22T20:00:48Z

pydantic_ai_slim/pydantic_ai/models/openai.py


    def _get_tools(self, model_request_parameters: ModelRequestParameters) -> list[responses.FunctionToolParam]:
-        return [self._map_tool_definition(r) for r in model_request_parameters.tool_defs.values()]
+        tools = model_request_parameters.active_tool_defs(builtin_search=False)


Looks like this is an old definition overwritten by the new one above!

DouweM · 2025-12-22T20:02:08Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

    ) -> list[BetaToolUnionParam]:
        tools: list[BetaToolUnionParam] = [
            self._map_tool_definition(r) for r in model_request_parameters.tool_defs.values()
+            if not is_search_tool(r)


What if the user has intentionally defined a tool with this name? We shouldn't be filtering it out in that case!

We should only do this if tool search is actually being used

Also as you pointed out, this is a bit awkward, I don't want to be repeating things like this in every model.

Could we do this in prepare_request, since we can check self.profile.supported_builtin_tools?

Indeed, we should refine is_search_tool to not search by name. I think filtering of tool definition to drop the search tool still needs to happen somewhere outside of the SearchableToolset itself that has no information on which model consumes it. Model.prepare_request sounds promising.

DouweM · 2025-12-22T20:05:08Z

pydantic_ai_slim/pydantic_ai/models/openai.py

+        active_tool_defs = {
+            name: tool for name, tool
+            in all_tool_defs.items()
+            if is_active(tool, run_id)


Do we actually need to check the run ID here? Since the ToolDefinitions are already generated from toolset.get_tools(ctx), which is ctx.run_id-aware, could the active metadata field be just a boolean?

I think we should also be able to do this in prepare_request based on self.profile.supported_builtin_tools, instead of repeating it in every-model-but-Anthropic

So in prepare_request we could implement the if FileSearchTool in self.profile.supported_builtin_tools: remove "search_tool"; else: remove in inactive tools check, specifically for this feature.

But I'm also very curious about the more generic and powerful #3212 -inspired route I outlined on Slack, where the toolset is still fully responsible for managing the tool search and state, and then presents the model with 1 list of tool defs to use if tool search is natively supported, and another if it isn't. That way models don't need to filter out tools by name or metadata at all, and we can later extend it to user-specified builtin tool fallbacks.

Let me quote what I wrote in Slack for other readers' context:

I see a lot of overlap with #3212, where we'll introduce a way for a toolset to return a builtin tool + a list of fallback tool definitions, which will all end up on ModelRequestParameters , after which Model.prepare_request can check self.profile.supported_builtin_tools and send either the builtin tool OR the list of tool definitions.
In our case, the SearchableToolset would return [the ToolSearchTool + all tool defs with defer_loading=True ] OR [the search_tools tool + the already-discovered tool defs], and Model.prepare_request would similarly branch based on if ToolSearchTool in self.profile.supported_builtin_tools .
So that's the same last-mile filtering you're describing, but done generically to work for all if builtin tool is supported, send it + specific tool defs; else; send other tool defs scenarios. I'm not sure yet how exactly that would be represented on ModelRequestParameters. but it needs to be serializable, so it could be something like extending the builtin_tools list to take not just AbstractBuiltinTool but also tuple[AbstractBuiltinTool, tool_defs_if_supported, tool_defs_if_not_supported](or more likely a new dataclass)

What do you think of that approach?

I think it's promising but didn't have time to explore this more, my one concern is that can it handle the run_id issues or we can bypass those issues? Once we have confidence there def worth revisiting.

@t0yv0 I'm not sure I understand the run_id issue you're referring to: SearchableToolset.get_tools generates the tools definitions based on ctx which includes run_id, and that list is then only used for a step in that particular run. So it shouldn't be necessary to check the active state of a tool definition based on run ID after the fact, as the active state can be hard-set on the generated tool definitions, as each step and run will get its own list.

Indeed we handled this elsewhere, let me open the rewritten PR.

DouweM · 2025-12-22T20:05:37Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+
+
+def _search_tool_validator() -> SchemaValidatorProt:
+    return TypeAdapter(_SearchToolArgs).validator


Why is this a function? Note that TypeAdapters are expensive to create

DouweM · 2025-12-22T20:06:58Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+        metadata = (self._tool_def.metadata or {}).copy()
+        toolset = self._searchable_toolset
+        tool = self._tool_def
+        metadata["active"] = lambda run_id: toolset.is_active(tool_def=tool, run_id=run_id)


ModelRequestParameters and its ToolDefinitions need to be serializable to work with Temporal durable execution, so storing a callable is not going to work I'm afraid.

Note that I think we should be able to set active to bool inside the toolset.get_tools, no callable needed

Note also that this may accidentally be overriding tool def metadata coming from the wrapped toolset, so I suggest making the name less generic

DouweM · 2025-12-22T20:09:06Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+
+        all_tools: dict[str, ToolsetTool[AgentDepsT]] = {}
+
+        all_tools[_SEARCH_TOOL_NAME] = _SearchTool(


We shouldn't expose the search tool unless there are deferred-loading tools!

t0yv0 · 2025-12-22T20:43:38Z

Thanks for this round of comments!

Just a quick note here: the run_id is the source of most of the complications here, perhaps a hint on appropriate object lifetimes here would help to find a cleaner solution. The problem, briefly, is that the Pydantic SearchableToolset as implemented requires to track active selection state so that it can tag ToolDefinitions so that models like OpenAI can defer loading them into the provider. In the previous versions of the PR, SearchableToolset structure did not depend on run_id but its lifetime could span multiple which would cause tool loading to "bleed" from run to run, degrading performance. Hence in the current version the structure is indexed on run_id.

If we are not exposing SearchableToolset to end users and the auto-injected instance is naturally ties its lifetime to a run_id, we can remove run_id indexing completely.
If we are exposing SearchableToolset to the users that may shares instances across runs, we still need to solve for this so tool activation does not bleed across runs.

Kludex · 2025-12-23T19:34:41Z

pydantic_ai_slim/pydantic_ai/builtin_tools.py

+
+    * Anthropic
+


Suggested change

* Anthropic

* Anthropic

Kludex · 2025-12-23T19:36:33Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+    ) -> Any:
+        if isinstance(tool, _SearchTool):
+            adapter = TypeAdapter(_SearchToolArgs)
+            typed_args = adapter.validate_python(tool_args)


Can the tool_args parameter be _SearchToolArgs?

Kludex · 2025-12-23T19:37:36Z

pydantic_ai_slim/pydantic_ai/toolsets/searchable.py

+        await self.toolset.__aenter__()
+        return self
+
+    async def __aexit__(self, *args: Any) -> bool | None:


Are we not properly typing __aexit__ elsewhere?

Kludex · 2025-12-23T19:39:08Z

tests/models/test_anthropic.py

+    result = await agent.run('Can you list the database tables and then fetch user 42?')
+    assert result.output == snapshot("""\
+Great! Here are the results:
+
+**Database Tables:**
+- users
+- orders
+- products
+- reviews
+
+**User 42 Details:**
+- ID: 42
+- Name: John Doe
+- Email: [email protected]\
+""")


Should we assert the message history instead?

Sketch SearchableToolSet

df4dea0

DouweM self-assigned this Dec 9, 2025

DouweM reviewed Dec 9, 2025

View reviewed changes

t0yv0 added 4 commits December 11, 2025 06:35

WIP on searchable tool

980187b

Format with unsafe fixes

35a65e9

Simple fixes

364a58e

Debug search tools and iterate on the description prompt

8ffdf17

t0yv0 commented Dec 11, 2025

View reviewed changes

DouweM requested changes Dec 11, 2025

View reviewed changes

DouweM added the awaiting author revision label Dec 11, 2025

t0yv0 added 3 commits December 13, 2025 10:03

Add supports_tool_search to ModelProfile

0f754c2

Respect defer_loading inside the Searchable ToolSet

4d38f43

Use built-in search tools in the Anthropic case

309e442

DouweM requested changes Dec 15, 2025

View reviewed changes

t0yv0 added 9 commits December 17, 2025 08:09

tool_name not used

880f409

ToolManager automatically wraps tools in SearchableToolset

65eb94b

Index active tool state by run_id

d19f4ca

Try Open AI models

3af19d7

Get openai to work

04b3c99

- automatically inject SerchableToolset - taint ToolDefinition by wrapping the with an active model recall flag - all ToolDefinitions get passed down to the model - last-minute filtering is done at the model level - tool active state is indexed on run_id

Filter out search tool in Anthropic

0fdcdc0

Inline active_tool_defs for now

f0dc519

Lock down basic behavior in openai.py tests

e846e47

Lock down openai behavior in a few simple tests

47c0c49

t0yv0 added 3 commits December 20, 2025 15:06

Test that anthropic propagates defer_loading=True calls

2399e9d

Test that Anthropic can defer tool loading natively

b7e9ff8

Drop temp test_searchable_example.py

20f4bea

t0yv0 commented Dec 20, 2025

View reviewed changes

Drop profiles switch not in use here

a2313ef

DouweM requested changes Dec 22, 2025

View reviewed changes

Kludex reviewed Dec 23, 2025

View reviewed changes

		@@ -0,0 +1,136 @@
		"""Minimal example to test SearchableToolset functionality.



		def _search_tool_validator() -> SchemaValidatorProt:
		return TypeAdapter(_SearchToolArgs).validator


		all_tools: dict[str, ToolsetTool[AgentDepsT]] = {}

		all_tools[_SEARCH_TOOL_NAME] = _SearchTool(

Toward SearchableToolSet and cross-model ToolSearch #3680

Are you sure you want to change the base?

Toward SearchableToolSet and cross-model ToolSearch #3680

Conversation

t0yv0 commented Dec 9, 2025 • edited by DouweM Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DouweM Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t0yv0 Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t0yv0 commented Dec 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t0yv0 commented Dec 20, 2025

Uh oh!

t0yv0 commented Dec 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

t0yv0 commented Dec 9, 2025 •

edited by DouweM

Loading

DouweM Dec 11, 2025 •

edited

Loading

t0yv0 Dec 22, 2025 •

edited

Loading