Skip to content

feat(otlp): capture coding-agent behavioral events into agent_telemetry - #548

Merged
njbrake merged 11 commits into
mozilla-ai:mainfrom
arthuursantos:feat/otlp-behavioral-events
Aug 12, 2026
Merged

feat(otlp): capture coding-agent behavioral events into agent_telemetry#548
njbrake merged 11 commits into
mozilla-ai:mainfrom
arthuursantos:feat/otlp-behavioral-events

Conversation

@arthuursantos

@arthuursantos arthuursantos commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Description

Claude Code emits four non-usage log events (tool_result, tool_decision,
user_prompt, api_error) on the OTLP logs signal Otari already receives at
/v1/logs. They were dropped because they carry no token/model attributes. This
PR maps them, content-free, into a new agent_telemetry table kept separate from
usage_logs.

Because behavioral events ride the existing receiver, this PR is cheap to land and
settles the full agent_telemetry schema up front (including the columns the
metrics receiver will use), so the follow-up metrics PR is purely additive with no
second migration.

Only allow-listed, typed fields are persisted (tool name, decision, success,
duration, status code, prompt length); prompts, responses, and user identity are
never stored. Ingestion is idempotent via a natural dedup key.

PR Type

  • New Feature
  • Bug Fix
  • Refactor
  • Documentation
  • Infrastructure / CI

Relevant issues

Part of #429 (paired with the metrics receiver PR, which closes it).

Checklist

  • I understand the code I am submitting.
  • I have added or updated tests that cover my change (tests/unit, tests/integration).
  • I ran the Definition of Done checks locally (make lint, make typecheck, make test).
  • Documentation was updated where necessary. (No user-facing surface changes here; docs land with the metrics PR.)
  • If the API contract changed, I regenerated the OpenAPI spec. (No new endpoint; behavioral capture rides existing /v1/logs, so the spec is unchanged.)

AI Usage

  • No AI was used.
  • AI was used for drafting/refactoring.
  • This is fully AI-generated.

AI Model/Tool used: Claude Code (Claude Opus 4.8)

Any additional AI details you'd like to share:
Used to split an existing feature branch into two stacked PRs and rebase the new migration onto the current main Alembic head.

NOTE:
When responding to reviewer questions, please respond yourself rather than copy/pasting reviewer comments into an AI and pasting back its answer. We want to discuss with you, not your AI :)

  • I am an AI Agent filling out this form (check box if true)

Summary

  • Capture Claude Code behavioral events from the OTLP /v1/logs receiver.
  • Store approved, content-free event details in a new agent_telemetry table.
  • Exclude prompts, responses, and user identity from stored data.
  • Prevent duplicate events through deterministic deduplication.
  • Add gateway and per-key controls for telemetry capture.
  • Add master-key deletion support for telemetry records.
  • Clean up telemetry when a user is deleted.
  • Update API documentation, Postman examples, and integration and unit tests.

This preserves useful agent activity data without storing sensitive content. Usage logging and billing remain unchanged.

Claude Code emits four non-usage log events (tool_result, tool_decision,
user_prompt, api_error) on the existing /v1/logs OTLP receiver; they were
dropped for lacking token/model attributes. Map them, content-free, into a
new agent_telemetry table kept separate from usage_logs.

The full table schema (including the metric columns the /v1/metrics receiver
will use) lands here, so the follow-up metrics PR is purely additive with no
second migration. Refs mozilla-ai#429.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The change adds agent telemetry storage, maps allow-listed behavioral events from OTLP logs, persists validated records with deduplication, adds capture controls, and provides protected deletion with user cleanup.

Changes

Agent telemetry lifecycle

Layer / File(s) Summary
Telemetry storage and capture contract
alembic/versions/..., src/gateway/models/entities.py, src/gateway/core/config.py, src/gateway/api/routes/keys.py, docs/public/otari.postman_collection.json
The migration and models add telemetry storage and per-key capture overrides. Gateway configuration defaults capture to enabled. Key APIs expose tri-state capture settings.
Event mapping and transactional ingestion
src/gateway/services/agent_telemetry_service.py, tests/unit/test_agent_telemetry_*.py
The service validates fields, maps supported events, generates deduplication keys, and handles batched inserts, duplicates, retries, and database failures.
OTLP capture and export handling
src/gateway/api/routes/otlp.py, src/gateway/api/main.py, tests/integration/otlp_*.py, tests/integration/test_otlp_logs_behavioral.py
The OTLP route applies capture settings, combines usage and telemetry limits, ingests behavioral records, and reports combined rejections. Tests cover filtering, parsing, deduplication, user state, limits, and overrides.
Protected deletion and ownership cleanup
src/gateway/services/agent_telemetry_admin_service.py, src/gateway/api/routes/agent_telemetry.py, src/gateway/api/routes/users.py, tests/integration/test_agent_telemetry_admin.py
The admin service and master-key endpoint delete telemetry by IDs or filters. User deletion removes only the deleted user’s telemetry rows.
Telemetry documentation
docs/use-with-claude-code.md, scripts/sdk_codegen/sdk-endpoints.txt
Documentation describes behavioral capture and deletion. The purge endpoint remains excluded from SDK wrapping.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related issues

Possibly related PRs

Suggested reviewers: khaledosman

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 41.38% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title uses a valid Conventional Commit feature prefix, clearly describes the telemetry change, and uses imperative wording; its 71-character length is only slightly above the guideline.
Description check ✅ Passed The description includes the required sections, explains the change, identifies the issue, records AI usage, and includes test and documentation details.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
⚔️ Resolve merge conflicts 💡
  • Resolve merge conflict in branch feat/otlp-behavioral-events
🧪 Generate unit tests (beta)
  • Create PR with unit tests
✨ Simplify code
  • Create PR with simplified code

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai
coderabbitai Bot requested a review from khaledosman August 10, 2026 17:06

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/gateway/services/agent_telemetry_service.py (1)

158-167: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

Batch telemetry inserts instead of flushing each record.

ingest() flushes one AgentTelemetry row at a time inside the record loop and also opens a savepoint per row. Large OTLP exports will create sequential database round trips; batch the compatible rows and keep duplicate counting separate. PostgreSQL can use on_conflict_do_nothing; handle a single bulk insert failure into accepted/duplicate counts.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/gateway/services/agent_telemetry_service.py` around lines 158 - 167,
Update ingest() to collect compatible AgentTelemetry rows and perform one bulk
PostgreSQL insert with on_conflict_do_nothing instead of flushing and creating a
nested transaction per record. Preserve separate accepted and duplicate counts,
and handle a bulk insert failure by assigning those counts appropriately rather
than processing each row with individual database round trips.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/gateway/services/agent_telemetry_service.py`:
- Around line 84-97: Update event_dedup_key so zero-valued duration_ms,
status_code, and prompt_length remain distinct from absent values in the hashed
key. Replace the truthiness-based fallbacks for these fields with explicit None
handling, preserving 0 while still mapping None to the empty-string
representation.

---

Nitpick comments:
In `@src/gateway/services/agent_telemetry_service.py`:
- Around line 158-167: Update ingest() to collect compatible AgentTelemetry rows
and perform one bulk PostgreSQL insert with on_conflict_do_nothing instead of
flushing and creating a nested transaction per record. Preserve separate
accepted and duplicate counts, and handle a bulk insert failure by assigning
those counts appropriately rather than processing each row with individual
database round trips.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7fe57fb0-fabc-41f9-a847-a8cee8a880e1

📥 Commits

Reviewing files that changed from the base of the PR and between 8b7e1d7 and d3cdac2.

📒 Files selected for processing (7)
  • alembic/versions/e8a7c6b5d4f3_add_agent_telemetry.py
  • src/gateway/api/routes/otlp.py
  • src/gateway/models/entities.py
  • src/gateway/services/agent_telemetry_service.py
  • tests/integration/otlp_helpers.py
  • tests/integration/test_otlp_logs_behavioral.py
  • tests/unit/test_agent_telemetry_mapping.py

Comment thread src/gateway/services/agent_telemetry_service.py
Collapse 0 onto "" for duration_ms, status_code, and prompt_length in
event_dedup_key so a zero-valued field no longer hashes identically to an
absent one. Zero is a legitimate value and must remain distinct to avoid
dropping rows on the ON CONFLICT dedup key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@njbrake
njbrake deployed to integration-tests August 10, 2026 18:41 — with GitHub Actions Active
@codecov-commenter

codecov-commenter commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.16058% with 16 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/gateway/services/agent_telemetry_service.py 93.75% 9 Missing ⚠️
.../gateway/services/agent_telemetry_admin_service.py 92.85% 4 Missing ⚠️
...embic/versions/e8a7c6b5d4f3_add_agent_telemetry.py 84.21% 3 Missing ⚠️
Files with missing lines Coverage Δ
src/gateway/api/main.py 100.00% <100.00%> (ø)
src/gateway/api/routes/agent_telemetry.py 100.00% <100.00%> (ø)
src/gateway/api/routes/keys.py 90.00% <100.00%> (ø)
src/gateway/api/routes/otlp.py 92.76% <100.00%> (ø)
src/gateway/api/routes/users.py 92.36% <100.00%> (ø)
src/gateway/core/config.py 89.76% <100.00%> (ø)
src/gateway/models/entities.py 95.47% <100.00%> (ø)
...embic/versions/e8a7c6b5d4f3_add_agent_telemetry.py 84.21% <84.21%> (ø)
.../gateway/services/agent_telemetry_admin_service.py 92.85% <92.85%> (ø)
src/gateway/services/agent_telemetry_service.py 93.75% <93.75%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Three defects in the new agent_telemetry ingest path, each reproduced
against POST /v1/logs:

- Claude Code emits tool_result's success as the string "true"/"false",
  not an OTLP boolValue, so the strict isinstance(bool) check stored NULL
  for the outcome field on every tool result. Read both encodings.
- The user lookup did not filter deleted_at, so a live key whose user was
  soft-deleted kept storing telemetry while the usage path on the same
  request rejected its events. Use get_active_user, the same gate the
  usage path applies.
- Behavioral events bypassed _MAX_EVENTS_PER_EXPORT: an 8 MiB body holds
  roughly 25k of them, each inserting a row inside its own savepoint.
  Apply the same per-export bound the usage path carries.

Adds integration coverage for the first two.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@njbrake
njbrake deployed to integration-tests August 10, 2026 19:33 — with GitHub Actions Active

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/gateway/api/routes/otlp.py (1)

476-477: 🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift

Batch telemetry inserts in agent_telemetry_service.ingest.

This call invokes ingest, which opens a nested transaction and flushes once for every telemetry record. A large valid export therefore creates one database round trip per row. Use a batched, conflict-tolerant insert while preserving accepted, duplicate, and rejected counts.

As per coding guidelines, do not execute queries or deletes inside row loops; batch with IN, bulk operations, or eager loading.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/gateway/api/routes/otlp.py` around lines 476 - 477, Update the telemetry
ingestion flow called by the route around ingest_telemetry and
agent_telemetry_service.ingest to collect records first, then perform a single
batched conflict-tolerant insert instead of opening transactions and flushing
per row. Preserve the existing accepted, duplicate, and rejected counts, and
ensure no database queries or deletes occur inside the record loop.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/gateway/api/routes/otlp.py`:
- Around line 469-472: Update the OTLP export validation around the existing
pairs and telemetry ingestion flow to enforce _MAX_EVENTS_PER_EXPORT against
len(pairs) + len(telemetry), before either ingestion call; remove the
telemetry-only limit and add an integration test covering a mixed export that
exceeds the combined bound.

---

Outside diff comments:
In `@src/gateway/api/routes/otlp.py`:
- Around line 476-477: Update the telemetry ingestion flow called by the route
around ingest_telemetry and agent_telemetry_service.ingest to collect records
first, then perform a single batched conflict-tolerant insert instead of opening
transactions and flushing per row. Preserve the existing accepted, duplicate,
and rejected counts, and ensure no database queries or deletes occur inside the
record loop.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 27fb59c4-33c0-41fb-9b25-813480c459f7

📥 Commits

Reviewing files that changed from the base of the PR and between 3c40d80 and c693211.

📒 Files selected for processing (3)
  • src/gateway/api/routes/otlp.py
  • src/gateway/services/agent_telemetry_service.py
  • tests/integration/test_otlp_logs_behavioral.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/gateway/services/agent_telemetry_service.py

Comment thread src/gateway/api/routes/otlp.py Outdated

@njbrake njbrake left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this review was drafted by Claude via back-and-forth with @njbrake. The reasoning and decisions are his; the prose is Claude's.

Thanks for splitting this out, the shape is right. I pushed three fixes to your branch (c693211) and CI is green on them. The rest below is yours to decide.

What I pushed

  1. tool_result.success was never persisted. Claude Code emits success as the string "true" / "false" (Tool result event, in the Claude Code monitoring docs), so isinstance(success, bool) stored NULL for the outcome field on every tool result. Reproduced through POST /v1/logs. Every other field goes through _bounded_int, which tolerates string encodings; success was the strict one, and it is the field documented as a string.
  2. The user lookup in ingest() did not filter deleted_at, so a live key whose user was soft-deleted kept storing telemetry while the usage path on the same request rejected its events. It now uses get_active_user.
  3. Behavioral events bypassed _MAX_EVENTS_PER_EXPORT. An 8 MiB body holds roughly 25k of them, each inserting a row inside its own savepoint.

Integration coverage added for the first two; both fail on 3c40d80.

What I would like your read on

Dedup key. It hashes the timestamp plus the value fields, but Claude Code ships tool_use_id on tool_result and tool_decision, and event.sequence on every event. Without them, two parallel Read calls with the same duration in one timestamp bucket collide and the second is dropped as a duplicate. Worth settling before merge: dedup_key is stored under a unique constraint, so changing the derivation later leaves existing rows on the old format.

Metric-only columns. kind is always "event", and value, temporality, series_start, series_key are always NULL, with an index on series_key. Also cheaper to decide now. ADD COLUMN <nullable> is metadata-only in PG 11+, so "avoid a second migration" is a weaker argument than it looks.

Batching. Agreeing with CodeRabbit; external_usage_service._insert_rows is the existing pattern. Not blocking now that the cap bounds it.

Docs and opt-out. docs/use-with-claude-code.md tells operators the export carries no content, which stops being true here. There is also no flag to disable capture and no purge path: DELETE /v1/usage does not cover agent_telemetry, and user deletion SET NULLs the owner rather than removing rows. Fine to land with the metrics PR if the two merge together, not fine if this one lands alone.

nathanbrake and others added 6 commits August 10, 2026 20:06
Usage and behavioral events are disjoint by event.name, so checking each
list against _MAX_EVENTS_PER_EXPORT on its own let one export persist
twice the intended number of rows. Bound the total instead. The narrower
check inside _ingest stays for /v1/traces, which has no behavioral path.

Reported by CodeRabbit on c693211.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/use-with-claude-code.md`:
- Around line 137-138: Update the DELETE /v1/agent-telemetry documentation to
include the api_key_id filter and state that filter-based deletion requires
by_filter: true. Ensure the documented request contract or example enables valid
API-key-scoped purges alongside the existing ids and user/date filters.

In `@src/gateway/services/agent_telemetry_service.py`:
- Around line 188-215: Update _insert_same_source_batch at
src/gateway/services/agent_telemetry_service.py:188-215 and its fallback
handling at src/gateway/services/agent_telemetry_service.py:246-250 to use async
with db.begin_nested() and await db.flush() for every tentative batch or
individual-row insert instead of committing. Commit only once after all source
groups in ingest succeed, preserving duplicate detection and rollback behavior;
update commit-based tests and add regression coverage proving a later
SQLAlchemyError leaves no earlier telemetry rows persisted.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 77c9b2d7-4938-4a43-bc17-6feb3868ae85

📥 Commits

Reviewing files that changed from the base of the PR and between 253d875 and f173c4d.

⛔ Files ignored due to path filters (1)
  • docs/public/openapi.json is excluded by !docs/public/openapi.json
📒 Files selected for processing (17)
  • alembic/versions/e8a7c6b5d4f3_add_agent_telemetry.py
  • docs/public/otari.postman_collection.json
  • docs/use-with-claude-code.md
  • scripts/sdk_codegen/sdk-endpoints.txt
  • src/gateway/api/main.py
  • src/gateway/api/routes/agent_telemetry.py
  • src/gateway/api/routes/keys.py
  • src/gateway/api/routes/otlp.py
  • src/gateway/api/routes/users.py
  • src/gateway/core/config.py
  • src/gateway/models/entities.py
  • src/gateway/services/agent_telemetry_admin_service.py
  • src/gateway/services/agent_telemetry_service.py
  • tests/integration/test_agent_telemetry_admin.py
  • tests/integration/test_otlp_logs_behavioral.py
  • tests/unit/test_agent_telemetry_ingest.py
  • tests/unit/test_agent_telemetry_mapping.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/gateway/models/entities.py
  • src/gateway/api/routes/otlp.py

Comment thread docs/use-with-claude-code.md Outdated
Comment thread src/gateway/services/agent_telemetry_service.py
@arthuursantos

Copy link
Copy Markdown
Contributor Author

Note: this comment was drafted by Claude.

Thanks, this was helpful. I took the four follow-ups as:

  1. Dedup key: I agree this needs settling before merge. I widened it with Claude Code’s tool_use_id (tool
    events) and event.sequence (all behavioral events), while retaining the existing fallback when an
    exporter omits them. This preserves retry idempotency without collapsing concurrent, otherwise-identical
    tool calls.

  2. Metric-only columns: I removed the currently-unused metrics shape (kind, value, temporality,
    series_start, series_key, and its index) from this PR’s migration/model. The metrics receiver can add
    those nullable columns in its own migration when it exists, rather than making behavioral rows carry
    permanently-null fields.

  3. Batching: Agreed. Ingestion now batches compatible rows and uses the duplicate fallback only when
    needed, following the existing external-usage insertion pattern rather than doing one savepoint/flush
    per event.

  4. Docs, opt-out, and deletion: I treated these as merge requirements for this PR. The Claude Code
    documentation now accurately describes the retained behavioral metadata and no longer says the export
    carries no content. Capture has a deployment default plus a tri-state per-key override, so operators can
    disable behavioral capture without affecting usage capture. Existing telemetry can be purged through a
    master-key-only DELETE /v1/agent-telemetry endpoint, and deleting a user now removes that user’s
    telemetry rows.

I also added coverage for the new dedup behavior, capture override, purge paths, user cleanup, metric-column removal, and batched ingest behavior.

@khaledosman
khaledosman requested a lite review from Copilot August 12, 2026 12:23
A record repeated inside one export fails the bulk insert on
(source, dedup_key), survives the re-query because nothing is stored yet,
fails the retry, and drops the whole batch into the row-at-a-time
fallback. Measured on a 201-event export carrying one repeat: 204 commits
and 3 rollbacks, against 2 commits for a clean batch of 200. At the
per-export ceiling that is roughly 10,000 sequential commits in one
request.

external_usage_service guards this with a seen_in_batch set before
calling _insert_rows; the port carried _insert_rows over but not the
guard. The stored projection is lossy by design, so two records can
collapse onto one dedup key more readily than on the usage path, where
source_event_id is a real unique id.

Also document capture_agent_telemetry in the configuration settings table
and the PATCH /v1/keys field list, and complete the documented
DELETE /v1/agent-telemetry contract with api_key_id and by_filter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@njbrake njbrake left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this review was drafted by Claude Opus 5 via back-and-forth with @njbrake. The reasoning and decisions are his; the prose is Claude's.

Approving. I pushed one commit to this branch (064e30c) with a performance fix and three doc updates; nothing else is changed.

The fix. A record repeated inside a single export fails the bulk insert on (source, dedup_key), survives the re-query because nothing is stored yet, fails the retry, and drops the whole batch into the row-at-a-time fallback. Measured on a 201-event export carrying one repeat: 204 commits and 3 rollbacks, against 2 commits for a clean batch of 200. At the per-export ceiling that is roughly 10,000 sequential commits in one request. external_usage_service guards this with a seen_in_batch set before calling _insert_rows; the port brought _insert_rows over but not the guard. Added a regression test.

Docs. capture_agent_telemetry was missing from the settings table in configuration.md and from the PATCH /v1/keys field list in api-reference.md, and the documented DELETE /v1/agent-telemetry contract omitted api_key_id and by_filter.

Verified locally against PostgreSQL 18: the telemetry suites plus otlp, keys, users, config, and docs; make lint, make typecheck, make openapi-check, make postman-check; and the migration up, down, and up again with no autogenerate drift.

Two things left for you, neither blocking.

  1. Behavioral events take their timestamp from time_unix_nano only, while usage events go through _resolve_timestamp, which prefers the event.timestamp attribute. An export with no time_unix_nano therefore drops every behavioral event silently while usage still lands, and the two families sit on different clocks, which matters for the cost-per-outcome joins in #429. Changing it changes every dedup key, so it is your call.
  2. capture_agent_telemetry has no dashboard surface, while reject_user_mismatch has a picker on the Keys page. Fine to land with the read view if that is the plan.

Needs a rebase for a one-line conflict in scripts/sdk_codegen/sdk-endpoints.txt.

On CodeRabbit's single-transaction comment: I disagree in scope. _insert_same_source_batch deliberately reproduces external_usage_service._insert_rows, which commits per batch, and telemetry rows carry no budget or spend. If savepoint semantics are wanted, both paths should move together.

Resolves the conflict in scripts/sdk_codegen/sdk-endpoints.txt by keeping
both entries: main's GET /v1/usage/in-flight in the usage section, and
this branch's DELETE /v1/agent-telemetry in its own section below it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@njbrake
njbrake deployed to integration-tests August 12, 2026 13:04 — with GitHub Actions Active
@njbrake
njbrake merged commit 7bd6546 into mozilla-ai:main Aug 12, 2026
10 checks passed
@njbrake

njbrake commented Aug 12, 2026

Copy link
Copy Markdown
Member

@arthuursantos thank you for your work on this, nice job!

arthuursantos added a commit to arthuursantos/otari that referenced this pull request Aug 13, 2026
…of spend

Closes the remaining half of issue mozilla-ai#429. PR mozilla-ai#548 added the behavioral events
on the logs signal; this adds the metrics signal Otari had no receiver for,
plus the read API that gives recorded spend a denominator.

- POST /v1/metrics: records the four outcome counters an agent reports and
  Otari has no other source for (lines of code, commits, pull requests, active
  time), content-free and non-billable. token.usage and cost.usage are skipped
  as already billed from the api_request usage event, and
  code_edit_tool.decision as already captured by tool_decision, so neither
  spend nor edit acceptance is ever double counted. Unrecognized names are
  accepted and skipped, so a newer agent version never breaks reception.
- agent_telemetry gains five nullable metric columns (kind, value, temporality,
  series_start, series_key) and a (series_key, timestamp) index, the ones spec
  002 deferred to this feature's own migration. Points are stored exactly as
  OTLP reported them; the cumulative to delta arithmetic happens at read time,
  split per series generation, so a re-exported total adds nothing and a
  counter reset never reads as negative work.
- GET /v1/agent-telemetry/summary, /count, and /series, master key only,
  mirroring the /v1/usage family. Only summary joins usage, reporting cost per
  commit, per pull request, per line changed, spend per active hour, tool
  acceptance rate, turns per session, and error rate. It filters by session as
  well as user and API key, matching the session on both sides of the join
  (session_label against usage_logs.source_label).
- Ingestion reuses the existing capture_agent_telemetry toggle and the batched
  ingest() pipeline: 1,000 data points cost one INSERT, asserted in the tests.
  The existing purge and user-deletion cleanup already cover metric rows.
- Docs now tell operators to enable OTEL_METRICS_EXPORTER alongside
  OTEL_LOGS_EXPORTER, which are independent settings, and list exactly which
  metric fields are captured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
arthuursantos added a commit to arthuursantos/otari that referenced this pull request Aug 13, 2026
…of spend

Closes the remaining half of issue mozilla-ai#429. PR mozilla-ai#548 added the behavioral events
on the logs signal; this adds the metrics signal Otari had no receiver for,
plus the read API that gives recorded spend a denominator.

- POST /v1/metrics: records the four outcome counters an agent reports and
  Otari has no other source for (lines of code, commits, pull requests, active
  time), content-free and non-billable. token.usage and cost.usage are skipped
  as already billed from the api_request usage event, and
  code_edit_tool.decision as already captured by tool_decision, so neither
  spend nor edit acceptance is ever double counted. Unrecognized names are
  accepted and skipped, so a newer agent version never breaks reception.
- agent_telemetry gains five nullable metric columns (kind, value, temporality,
  series_start, series_key) and a (series_key, timestamp) index, the ones the
  behavioral-events PR reserved for this receiver rather than adding then.
  Points are stored exactly as OTLP reported them; the cumulative to delta
  arithmetic happens at read time, split per series generation, so a
  re-exported total adds nothing and a counter reset never reads as negative
  work.
- GET /v1/agent-telemetry/summary, /count, and /series, master key only,
  mirroring the /v1/usage family. Only summary joins usage, reporting cost per
  commit, per pull request, per line changed, spend per active hour, tool
  acceptance rate, turns per session, and error rate. It filters by session as
  well as user and API key, matching the session on both sides of the join
  (session_label against usage_logs.source_label).
- Ingestion reuses the existing capture_agent_telemetry toggle and the batched
  ingest() pipeline: 1,000 data points cost one INSERT, asserted in the tests.
  The existing purge and user-deletion cleanup already cover metric rows.
- Docs now tell operators to enable OTEL_METRICS_EXPORTER alongside
  OTEL_LOGS_EXPORTER, which are independent settings, and list exactly which
  metric fields are captured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
champ18ion pushed a commit to champ18ion/otari that referenced this pull request Aug 14, 2026
…of spend (mozilla-ai#567)

* feat(telemetry): receive OTLP outcome metrics and read them per unit of spend

Closes the remaining half of issue mozilla-ai#429. PR mozilla-ai#548 added the behavioral events
on the logs signal; this adds the metrics signal Otari had no receiver for,
plus the read API that gives recorded spend a denominator.

- POST /v1/metrics: records the four outcome counters an agent reports and
  Otari has no other source for (lines of code, commits, pull requests, active
  time), content-free and non-billable. token.usage and cost.usage are skipped
  as already billed from the api_request usage event, and
  code_edit_tool.decision as already captured by tool_decision, so neither
  spend nor edit acceptance is ever double counted. Unrecognized names are
  accepted and skipped, so a newer agent version never breaks reception.
- agent_telemetry gains five nullable metric columns (kind, value, temporality,
  series_start, series_key) and a (series_key, timestamp) index, the ones the
  behavioral-events PR reserved for this receiver rather than adding then.
  Points are stored exactly as OTLP reported them; the cumulative to delta
  arithmetic happens at read time, split per series generation, so a
  re-exported total adds nothing and a counter reset never reads as negative
  work.
- GET /v1/agent-telemetry/summary, /count, and /series, master key only,
  mirroring the /v1/usage family. Only summary joins usage, reporting cost per
  commit, per pull request, per line changed, spend per active hour, tool
  acceptance rate, turns per session, and error rate. It filters by session as
  well as user and API key, matching the session on both sides of the join
  (session_label against usage_logs.source_label).
- Ingestion reuses the existing capture_agent_telemetry toggle and the batched
  ingest() pipeline: 1,000 data points cost one INSERT, asserted in the tests.
  The existing purge and user-deletion cleanup already cover metric rows.
- Docs now tell operators to enable OTEL_METRICS_EXPORTER alongside
  OTEL_LOGS_EXPORTER, which are independent settings, and list exactly which
  metric fields are captured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(telemetry): count a cumulative counter's first reading, and bound the summary scan

Review fixes on the OTLP metrics receiver.

A cumulative series was diffed pairwise, so the first reading of every
generation was dropped. An OTel counter exports no data point until its first
measurement, so that reading already carries work: three commits read as two,
and a session whose only export carried one commit read as zero, inflating
cost per commit and cost per line. A generation whose series_start falls inside
the window is now diffed from its own zero, which is what a cumulative counter
reads at its series start. One that began earlier has no known baseline, so its
first in-window reading stays a level, as before.

Also:

- Bound /summary's metric scan at 200k data points and fail closed past it.
  The delta arithmetic runs in Python, so unlike the aggregates beside it this
  read grows with the number of exports rather than with the window.
- Return None from _point_value when a NumberDataPoint sets neither value arm,
  instead of proto3's default 0, which inside a cumulative series reads as a
  reset the series start never announced.
- Pin offset-less date bounds to UTC in core.sql.utc_bound, shared by the
  telemetry read filters and the purge selection so the count an operator
  confirms and the delete that re-derives it mean the same instant. asyncpg
  encodes timestamptz with astimezone, which reads a naive datetime as local.
- Apply _SKIPPED_METRICS, which was defined but never read, so adding an
  already-billed metric to the recorded set cannot double count spend.
- Note in the /summary docstring that the spend side is every usage row in
  scope, not only the agent's.
- Update the capture_agent_telemetry row in docs/configuration.md, which still
  described logs-signal behavioral events alone, list the OTLP metrics receiver
  in docs/external-usage.md, and account for claude_code.session.count in the
  skipped-metrics paragraph.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Nathan Brake <NathanBrake@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants