feat(observability): full-content Langfuse tracing — generations, tools, subagents, provider-qualified models, real costs#4506
Conversation
…ugh agent tracing
- Enable tinyagents PayloadCapture in run_policy_for so ModelCompleted /
ToolCompleted events carry request messages (incl. system prompt),
completions, tool arguments and results. Off-device transmission stays
gated by observability.agent_tracing.capture_content; the durable
journal already redacts secrets via RedactingSink.
- Bridge: project ModelCompleted into ModelCallCompleted for parent AND
subagent scopes (new subagent_task_id attribution), carrying captured
input/output, provider_id, and a real per-call cost. Tool completions
now carry real output text, backfilled arguments, and measured
elapsed_ms (was hardcoded 0 / empty).
- Fix $0 cost: estimate via tier-aware agent::cost (knows chat-v1/
burst-v1/...) instead of the vendor-only cost::catalog lookup.
- Provider identity: new Provider::telemetry_provider_id() (managed /
openai / ollama / claude-code / custom); exporters render the Langfuse
model as {provider_id}.{model} (e.g. managed.chat-v1).
- Collector: generations record content (capture-gated, 25k-char cap);
subagent model calls nest under the subagent iteration and roll
model/usage/cost onto the subagent span (Context Scout model now
visible); subagent spans record delegated prompt + final output; tool
spans record output + backfilled arguments.
- Spawn tools emit full prompt/output on SubagentSpawned/Completed.
- Regression tests for all of the above (collector + bridge).
Claude-Session: https://claude.ai/code/session_01Y66Xr118tDtRnLeFT1owFj
…cache is cold Headless/autonomous cores never warm the auth_get_me cache, so Langfuse traces attributed to the transport client id. Fall back to the stored app-session profile (email, then backend user id) before the client-id last resort. With tests. Claude-Session: https://claude.ai/code/session_01Y66Xr118tDtRnLeFT1owFj
…ntent-tracing # Conflicts: # src/bin/harness_subagent_audit.rs # src/openhuman/agent/progress_tracing.rs # src/openhuman/channels/providers/web/progress_bridge.rs # src/openhuman/tinyagents/mod.rs # src/openhuman/tinyagents/observability.rs
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 34 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThis PR enriches ChangesProgress event content enrichment and provider telemetry
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
senamakel
added
feature
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dad57a1dc1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Snapshot the provider's telemetry id before `provider` moves into the | ||
| // harness assembly — the event bridge stamps it on every per-call | ||
| // generation event (`{provider_id}.{model}` in Langfuse). | ||
| let provider_id = provider.telemetry_provider_id(); |
There was a problem hiding this comment.
When the active provider is a router or fallback wrapper, the provider that serves a call is selected later per request (for example RouterProvider::resolve(model) can dispatch chat-v1/hint:* to a non-default provider, and ReliableProvider can fail over). Caching provider.telemetry_provider_id() once here means every ModelCallCompleted emitted by the bridge is labeled with the default/primary provider even when the call actually went elsewhere, so provider-qualified Langfuse model labels and per-provider trace attribution are wrong for routed/failover runs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct — the label reflects the configured primary provider, not per-call routing. The crate's harness events carry no per-call provider identity today, so the bridge cannot observe which inner provider actually served a given call (RouterProvider::resolve / ReliableProvider failover happen below the event stream). Wrappers delegate to their primary so the common single-provider case is accurate; per-call provider attribution needs the provider identity threaded onto tinyagents' ModelStarted/ModelCompleted events — noting it as a follow-up rather than expanding this PR's surface.
…ntent-tracing # Conflicts: # src/openhuman/tinyagents/observability.rs
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhuman/agent/progress_tracing.rs`:
- Around line 678-690: The TurnContent path in SpanCollector is ignoring the
TraceContext.capture_content setting because SpanCollector::new leaves
capture_content false and the collector never receives the flag. Update the
SpanCollector construction and/or its TurnContent handling in
progress_tracing.rs so the collector uses self.ctx.capture_content (or passes it
through at init) before populating span.input and span.output with
capture_model_content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 23e861de-8dc7-46d9-9547-8f703182d742
📒 Files selected for processing (27)
src/bin/harness_subagent_audit.rssrc/openhuman/agent/harness/session/tool_progress.rssrc/openhuman/agent/progress.rssrc/openhuman/agent/progress_tracing.rssrc/openhuman/agent/progress_tracing/tests.rssrc/openhuman/agent_orchestration/ops.rssrc/openhuman/agent_orchestration/spawn_parallel_graph.rssrc/openhuman/agent_orchestration/tools/agent_prepare_context.rssrc/openhuman/agent_orchestration/tools/continue_subagent.rssrc/openhuman/agent_orchestration/tools/dispatch.rssrc/openhuman/agent_orchestration/tools/spawn_async_subagent.rssrc/openhuman/agent_orchestration/tools/spawn_subagent.rssrc/openhuman/channels/providers/web/progress_bridge.rssrc/openhuman/inference/provider/claude_code/mod.rssrc/openhuman/inference/provider/compatible_provider_impl.rssrc/openhuman/inference/provider/openhuman_backend.rssrc/openhuman/inference/provider/reliable.rssrc/openhuman/inference/provider/router.rssrc/openhuman/inference/provider/traits.rssrc/openhuman/routing/provider.rssrc/openhuman/threads/turn_state/mirror_tests.rssrc/openhuman/tinyagents/mod.rssrc/openhuman/tinyagents/observability.rstests/memory_core_threads_raw_coverage_e2e.rstests/memory_raw_coverage_e2e.rstests/memory_threads_raw_coverage_e2e.rsvendor/tinyagents
…ions tinyagents v1.5 made journal persistence asynchronous (background AppendWorker); read-after-write now requires flush(), so the replay test raced the drain thread and read zero events. Claude-Session: https://claude.ai/code/session_01Y66Xr118tDtRnLeFT1owFj
Review fix (PR tinyhumansai#4506, CodeRabbit): the tinyhumansai#4454 merge left two parallel capture gates — a collector-level flag read by the TurnContent arm and the TraceContext flag read by every other content path. The web bridge only sets the TraceContext flag, so the turn's prompt/reply was always dropped. with_content_capture now delegates to the single ctx gate; regression test covers both construction styles. Claude-Session: https://claude.ai/code/session_01Y66Xr118tDtRnLeFT1owFj
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e65cf04784
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| fn telemetry_provider_id(&self) -> String { | ||
| "custom".to_string() |
There was a problem hiding this comment.
When the configured provider is claude_agent_sdk / claude_agent_sdk:<model>, the factory builds ClaudeAgentSdkProvider, but that provider does not override this new method, so all of its Langfuse generations are labeled custom.<model> and gen_ai.provider=custom rather than the actual SDK provider. Since this change makes provider id part of the exported model/provenance, please add a concrete override for the Claude Agent SDK provider (similar to ClaudeCodeProvider) instead of letting this real provider fall through to the custom default.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ff12f36954
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| span.attributes.insert( | ||
| "gen_ai.usage.cost_usd".to_string(), | ||
| json_f64(prior_cost + cost_usd), | ||
| ); |
There was a problem hiding this comment.
For any subagent that makes an LLM call, this copies the same gen_ai.usage.*/cost data onto the enclosing subagent span after already creating an llm.* generation for that call. spans_to_langfuse_batch promotes any span with these attributes via apply_usage_fields, so the batch will contain both the child generation and the subagent span as generation-create observations with the same cost, double-reporting subagent usage/cost in Langfuse traces.
Useful? React with 👍 / 👎.
1 participant
Summary
llm.*generation records the request messages (system prompt included) and the completion; tool spans record arguments, result text, and real elapsed time; subagent spans record the delegated prompt, final output, and the model that served them.managed.chat-v1,managed.burst-v1,openai.gpt-4o, … via a newProvider::telemetry_provider_id().agent::costtable (charged>estimate precedence from the provider-usage carry is preserved and the generation telemetry reuses the exact resolved figures).auth_get_mecache is cold.Problem
Live staging traces (project
Stagingon fuse.tinyhumans.ai) were missing all content: generations had usage but no input/output (system prompt never logged), the Context Scout flow showed no model and no I/O, tool spans reportedoutput_chars: 0/elapsed_ms: 0with no payloads, per-call cost read $0 for managed tiers, and headless runs attributed traces to socket ids. Root cause was concentrated in the tinyagents→openhuman event bridge: payload capture was never enabled, child model-call events were suppressed, tool completions were hardcoded to zeros, and cost was estimated through the vendor-only catalog that does not know managed tier handles.Solution
PayloadCapture::all()inrun_policy_for— the crate stamps request/response and tool I/O ontoModelCompleted/ToolCompleted. Off-device transmission stays gated byobservability.agent_tracing.capture_content(default off per Trace exporter ignores capture_content — prompts/replies written to NDJSON/app log; Langfuse cached-token double-count #4454); the durable journal already redacts viaRedactingSink.OpenhumanEventBridge: projectModelCompletedintoModelCallCompletedfor parent and child scopes with captured content, provider id, subagent task attribution, and the same resolved cost/cache/reasoning figuresrecord_usagefed the wallet. Tool completions carry output text, backfilled arguments, and measured duration (middleware-recorded, with a bridge-side stamp fallback).SpanCollector: generations record truncated content (25k-char cap); child generations nest under the subagent iteration and roll model/usage/cost onto the subagent span; subagent spans record prompt/output; tool spans record I/O.SubagentSpawned/SubagentCompleted.Verified live against staging Langfuse: a scout-bearing turn audits as fully wired (system prompt visible,
managed.burst-v1scout generations with content, tool I/O + elapsed, non-zerocostDetails, user attributionenamakel@tinyhumans.ai).Submission Checklist
## Related— N/A: no matrix rows touchedCloses #NNNin the## Relatedsection — N/A: no tracked issue; follow-up of Structured tracing export from the agent progress channel #3886/Improve agent harness with LangGraph-style state machine or Polaris-like architecture #4249 observability workImpact
observability.agent_tracing.capture_contentis enabled.AgentProgressevents grew content fields; UI consumers are unaffected (patterns updated).Related
main(macOS debug):turn_dispatches_spawn_subagent_through_full_path,nested_subagent_dispatch_runs_on_a_constrained_worker_stack,agent_turn_runs_long_parallel_subagent_flow_with_many_nested_tool_calls— unrelated to this change.AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
Commit & Branch
Validation Run
pnpm --filter openhuman-app format:check(via pre-push hook)pnpm typecheckcargo test --lib progress_tracing(64 ok),tinyagents::observability(6 ok),turn_state(24 ok),progress_bridge(9 ok),agent_meetings(187 ok)cargo fmt,cargo check --all-targetscleanapp/src-taurichangesValidation Blocked
command:cargo test --lib(full suite)error:three pre-existing nested-subagent stack-overflow aborts (reproduce identically onmainon this machine)impact:unrelated to this diff; all touched-module suites passBehavior Changes
capture_content), provider-qualified model labels, real per-call costs, and real-user attribution on headless runs.https://claude.ai/code/session_01Y66Xr118tDtRnLeFT1owFj
Summary by CodeRabbit
New Features
Bug Fixes