Tool-call KPI/audit observability only covers MCP tools, not native capability tools

[dimitri-tombroff]

Context

Found during a manual observability test campaign on consolidation (C3 target: docs/swift/platform/OBSERVABILITY-AND-AUDIT.md). Related to #1972 (ReAct execution loop migration to create_agent + platform middleware frame) and OBSERV-03 (id-legend.yaml).

Problem

ContextAwareTool (libs/fred-runtime/fred_runtime/common/context_aware_tool.py) is the only place that emits:

• KPI: agent.tool_latency_ms (histogram) / agent.tool_failed_total (counter)
• Audit: agent.tool.invocation.{started,completed} (the guarantee documented in OBSERVABILITY-AND-AUDIT.md §9 as "True today — emitted... for every actually-executed tool call")

But ContextAwareTool is only constructed in libs/fred-runtime/fred_runtime/common/mcp_toolkit.py:144, i.e. only for MCP-catalog-sourced tools. Native capability tools (e.g. DocumentAccessCapability's search_documents_using_vectorization, a plain @tool-decorated function shipped on the capability's own AgentMiddleware.tools) never pass through ContextAwareTool at all.

Confirmed live (2026-07-18): a chat turn that called search_documents_using_vectorization produced zero agent.tool_latency_ms samples and zero agent.tool.invocation.* audit events, while an MCP-sourced tool call would have produced both. So today:

• The documented "every tool invocation is audited" guarantee is false for capability-native tools.
• A production Grafana panel for "MCP/tool call latency" is blind to capability-native tool calls.

Target fix (not a per-tool patch)

LangChain's AgentMiddleware.awrap_tool_call(request: ToolCallRequest, handler) (langchain/agents/middleware/types.py) intercepts every tool call the graph executes, regardless of source. Add one new platform middleware implementing this hook, insert it into the fixed frame (libs/fred-runtime/fred_runtime/react/middleware/frame.py::build_react_platform_middleware_frame), and remove the now-redundant KPI/audit emission from ContextAwareTool (keep everything else there — context injection, MCP content normalization, error-message formatting stay MCP-specific and untouched).

See the follow-up prompt/PR for the full implementation spec.

[dimitri-tombroff]

Implementation ready for review (not yet committed — working tree only, per repo convention):

• New ToolObservabilityMiddleware (libs/fred-runtime/fred_runtime/react/middleware/tool_observability.py) implements awrap_tool_call, the one chokepoint every tool call goes through (MCP-catalog and capability-native alike). It emits the same agent.tool_latency_ms / agent.tool_failed_total KPI and agent.tool.invocation.{started,completed} audit events ContextAwareTool used to emit, but for every tool call, tagged source="mcp" or source="capability".
• Wired into the fixed frame (react/middleware/frame.py) next to TracingKpiMiddleware.
• Removed the now-redundant KPI/audit emission from ContextAwareTool (_kpi_timer, _emit_tool_call_audit, _kpi_base_dims) — kept context injection, MCP content normalization, and HTTP-status error handling untouched.
• Tests: ported the 6 removed KPI/audit tests' intent into tests/test_tool_observability_middleware.py (10 tests), including an explicit regression test proving a native capability tool (never wrapped by ContextAwareTool) now gets full KPI + audit coverage. test_context_aware_tool.py keeps its 3 context-injection tests unmodified. Updated test_react_middleware_frame.py's frame-order assertions for the new middleware.
• Docs: dated clause added to docs/swift/platform/OBSERVABILITY-AND-AUDIT.md §9.

Known follow-up (not fixed here, flagged in the new middleware's docstring): ContextAwareTool still swallows exceptions from the underlying MCP tool call and returns formatted error text instead of raising — so ToolMessage.status stays "success" for that narrow case, and the new middleware (which only detects failure via a raised exception or ToolMessage.status == "error") can't flag it as failed. This regressed relative to ContextAwareTool's own previous internal instrumentation, which wrapped the raw underlying call directly. Fixing it needs a product decision (raise ToolException instead, or add a structured error signal to the returned content/artifact) — separate follow-up.

make code-quality (lint/format/sast/type-check) and make test (542 passed) both clean in libs/fred-runtime.

[hegu-1]

This is a classic observability blind spot: the documented guarantee says "every actually-executed tool call," but implementation only covers one tool path.

For governance, native capability tools and MCP tools need the same audit/KPI envelope:

• invocation started/completed/failed
• tool identity and action class
• calling agent/run/session
• latency/cost/error metadata
• policy or middleware frame, if present
• correlation id into the surrounding trace

Otherwise teams get a false sense of coverage: the riskiest path may simply be outside the instrumented wrapper.

I have been mapping this as an enterprise AI OS concern: execution surfaces differ, but governance and audit need to wrap all of them consistently: https://github.com/hegu-1/enterprise-ai-os-architecture

[dimitri-tombroff]

thanks for your comment i will come bakc to this and to you @hegu-1 on this asap.

[dimitri-tombroff]

Closing the original MCP-only observability gap as delivered: platform ToolObservabilityMiddleware now covers capability and MCP tool calls. The newly-audited authorization/audit ordering and structural error-status follow-ups are consolidated in #2227.