feat(search): add a billed POST /v1/search pass-through - #473
Conversation
|
Warning Review limit reached
Next review available in: 11 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (18)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review Note: this re-trigger was posted by Claude at @njbrake's request. The earlier review attempt hit the plan's PR review limit and CodeRabbit does not retry on its own. |
|
✅ Action performedReview finished.
|
|
@coderabbitai full review Note: posted by Claude at @njbrake's request. The plain |
|
✅ Action performedFull review finished. Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 46 minutes. |
There was a problem hiding this comment.
Pull request overview
Adds a new standalone-mode, billed search pass-through surface (POST /v1/search and POST /v1/search/{search_tool_name}) to keep direct web-search requests on otari’s normal auth/rate-limit/budget/usage-log path, with Exa wired end-to-end via a new search_tools: config block.
Changes:
- Introduces
/v1/searchroutes plus a provider-neutral search backend with an Exa adapter. - Extends config + access-control validation to support
<provider>:<tool>allow-list entries for search tools. - Updates pricing lookup to optionally skip genai-prices defaults, and adds unit/integration coverage plus docs/OpenAPI/Postman updates.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/gateway/api/routes/search.py |
New /v1/search handlers with reservation/logging scaffold and LiteLLM-compatible wire shape. |
src/gateway/services/search_backend.py |
Provider-neutral search dispatch with Exa request/response translation. |
src/gateway/core/config.py |
Adds search_tools, validation, and provider-prefix support for allow-lists. |
src/gateway/services/model_access.py |
Allows allow-list prefixes for configured search-tool providers. |
src/gateway/services/pricing_service.py |
Adds use_defaults flag to suppress genai-prices fallback when needed. |
src/gateway/api/main.py |
Registers the new search router (standalone mode). |
tests/unit/test_search_backend.py |
Unit tests for tool resolution and Exa adapter translation/behavior. |
tests/integration/test_search_endpoint.py |
Integration tests for auth, allow-lists, budget reservation/settlement, and usage logging. |
tests/integration/test_find_model_pricing.py |
Covers use_defaults=False behavior. |
docs/api-reference.md |
Documents new search endpoints and request/response contract. |
docs/configuration.md |
Documents search_tools configuration and flat per-request pricing convention. |
docs/tools.md |
Cross-links to the standalone search endpoint docs. |
docs/access-control.md |
Notes that allowed_models also gates non-model spend surfaces like search. |
config.example.yml |
Adds commented example search_tools config. |
docs/public/openapi.json |
Regenerated OpenAPI spec including new schemas/routes. |
docs/public/otari.postman_collection.json |
Regenerated Postman collection including new requests. |
Suppressed comments (1)
src/gateway/api/routes/search.py:180
- This docstring is inaccurate about how the
userfield behaves for non-master API keys.resolve_passthrough_user_id(...)always binds usage/billing to the API key’s own user (and rejects or ignores mismatches depending onreject_user_mismatch), so “API key + user field: Use specified user” is not correct.
Authentication modes:
- Master key + user field: Use specified user (must exist)
- API key + user field: Use specified user (must exist)
- API key without user field: Use the shared "default" user
khaledosman
left a comment
There was a problem hiding this comment.
The wire-shape and pricing decisions look right, and the two traps you avoided are the non-obvious ones: use_defaults=False on find_model_pricing and model=None into reserve_budget (with a test pinning the warning, which is how that stays fixed).
Inline: search rejections write no usage row (which contradicts #472 landing alongside), Exa page content cannot be opted out of, the zero-reservation case when no flat rate is configured, and a timeout validation gap.
🤖 Reviewed with Claude Code
Otari exposed web search only as an in-loop tool (otari_web_search), so a
caller that wants a direct search request had to hold its own provider key
and fell off the gateway's billing path entirely. That is the gap blocking
MLPA's move from the LiteLLM proxy onto otari: it proxies an Exa search
endpoint today and needs an equivalent surface here to keep search on one
gateway.
Adds POST /v1/search and POST /v1/search/{search_tool_name}, registered
standalone-only alongside the other pass-throughs. Both run the same
scaffold as a completion (auth, rate limit, budget reservation, usage log,
reconcile or refund) and both log endpoint="/v1/search", so one Activity
filter covers every search. Tools are declared under search_tools: in
config.yml and validated at startup, so an unsupported provider or a
missing API key fails before the first request. Exa is wired end to end.
The request and response follow LiteLLM's /v1/search, itself shaped after
Perplexity's Search API, so a client migrating off that proxy needs no
changes; translation to and from Exa's native shape lives in the backend.
Search bills per request rather than per token, so a usage row carries zero
tokens and a cost taken from Exa's own reported costDollars.total when
present, falling back to a flat per-request rate configured for
<provider>:<tool> under the moderations convention. Since the provider
meters itself, search is exempt from require_pricing like moderations and
audio. find_model_pricing grows a use_defaults flag the route sets to
False: the genai-prices fallback also matches on a bare name, so a search
tool an operator happened to name after a real model would otherwise
inherit that model's per-million-token rate.
No dashboard change is needed for search to be visible: the Activity
endpoint filter is a free-form pass-through rather than an allowlist, and a
zero-token row renders as it should. Credentials come from config.yml only;
dashboard CRUD for search-tool credentials is left for a follow-up.
Fixes #400
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four fixes to the new /v1/search surface, all found by an independent review of the PR. reserve_budget was handed the "<provider>:<tool>" pricing key as its `model`. That argument exists only to drive the free-model shortcut, which splits the string through any-llm, so every search logged a swallowed "Failed to determine provider pricing" warning. The latent half is worse: the shortcut calls find_model_pricing *without* the use_defaults=False guard the route sets one frame up, so a future search provider whose name is also an any-llm provider (perplexity and cohere are both plausible) would run the genai-prices bare-name fallback on the tool name and, on a zero-priced match, skip the reservation outright. Pass model=None: the free-model shortcut is a token-pricing concept with no meaning for a per-request-billed tool. The per-key model allow-list was not enforced, so a key restricted to specific models could spend freely on search. Enforcing it needed a second change to be usable at all: model_access._known_prefix accepts only a configured provider instance or an any-llm provider, so writing "exa:exa-search" to a key was rejected with a 400 and a restricted key would have been permanently denied search with no way to grant it. The prefix check now also accepts the configured search tools' providers. build_exa_payload replaced contents.text wholesale when the caller sent max_tokens_per_page, dropping siblings such as a tool's pinned verbosity; it now merges. The test that was supposed to cover the override was vacuous, since 50 tokens times 4 chars equalled the pinned 200. Also constrains `country` to two characters, and stops claiming callers migrating from LiteLLM need no request changes: `query` must be a single string, and the Perplexity filters otari does not model are ignored rather than rejected. Both are now stated in the API reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six review findings on the new /v1/search surface. A gateway-side refusal wrote no usage row, so an unknown or ambiguous tool name (400) and a tool a key's allowed-models list does not name (403) were invisible in Activity and uncounted by the failure tile. Both now write an error row with a null cost, following the shape the require_pricing 402 rejection already uses in the pass-through scaffold. The row for an unresolved tool carries the requested name (or "unknown") and no provider, since none was resolved. The Exa adapter forced contents.text on every search, so an operator who pinned only highlights still paid Exa's per-page content charge and a caller that wanted ranked URLs had no way to opt out. Pinning contents.text to null (or false) now suppresses page content, and drops an otherwise empty contents block. The opt-out is the operator's, so it holds even for a request carrying max_tokens_per_page. Searches shared no HTTP client: a client per call paid a fresh TCP and TLS handshake and pooled nothing, which is the wrong shape for the sustained traffic this endpoint exists to serve. Provider calls now go through one pooled client for the process, with the per-tool timeout passed on the request; shutdown closes it. search_tools validation accepted a negative timeout, which only blew up inside httpx at request time, and a zero, which was silently swapped for the 30s default when the tool was resolved. Both are now rejected at load. The route docstrings claimed an API key plus a user field bills the named user, and that a key without one bills a shared "default" user. Neither is true: spend always binds to the key's own user, and a mismatch is rejected or ignored per reject_user_mismatch. Corrected, so the OpenAPI description and the Postman collection stop repeating it. Finally, with no flat per-request rate configured for <provider>:<tool> the reservation is $0, so a user just under their cap can overshoot by a search and concurrent searches cannot see each other's holds. Spend stays truthful (the provider's reported charge is reconciled onto the usage row), so this is left as a documented recommendation plus a startup warning naming every unpriced tool, rather than a new 402 that would reject traffic the gateway can bill precisely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… writer #472 landed a shared writer for gateway-side refusals after this branch built its own rows, and the two disagreed on one column that matters: log_gateway_rejection pins counts_toward_budget True, while these rows carried `not budget_exempt`. A key flagged exclude_from_budget would therefore have written a refusal row the dashboard classifies as imported usage and offers for bulk delete and set-price, which must never happen to a row the gateway wrote itself. The two search refusals (unknown or ambiguous tool, allow-list denial) now go through log_gateway_rejection, which also makes the write best-effort: a sick log writer can no longer turn a clean 400 or 403 into a 500. Row shape is otherwise unchanged (status=error, cost NULL, latency, endpoint /v1/search), and the served rows keep their own builder, since those carry a cost and do honor exclude_from_budget. Tests assert counts_toward_budget on both refusal rows, plus a new case for an exempt key hitting the allow-list gate. They read /v1/usage rather than /v1/users/{id}/usage, which does not expose the column. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b5bd57e to
fb1d826
Compare
…assified (mozilla-ai#470) * feat(usage): record a status code on usage logs so failures can be classified UsageLog carried status (success/error) and a free-text error_message, so failures could be counted but not broken down. Any breakdown had to be built on substring matching over provider-specific error prose, which differs per provider and changes without notice. Add a nullable usage_logs.status_code and populate it on every path that already writes an error row: the pipeline's non-streaming and streaming provider failures, the tool-loop cap (422), the missing-pricing rejections (402) in both the pipeline and the pass-through gate, the pass-through provider failure, and batch creation. The recorded code prefers the status the provider returned, which is deliberately not always the status the caller saw. An upstream 401/403 is a provider rejecting the gateway's credentials, so the response stays a generic 502 and never says so; the log keeps the 401, which is what makes "how much of my error rate is my own misconfiguration" answerable. A provider that never answered carries no status, so the row records the gateway's own classification (504 timeout, 502 unreachable) rather than staying unclassifiable. Success rows stay NULL: there is no failure to classify, and a constant 200 would dilute every GROUP BY over the column. Expose it as a status_code filter on the four /v1/usage read endpoints and as errors_by_status_code on the summary, scoped to failures and carrying a coarse error_class (pricing / rate_limit / auth / provider_error / client_error) for display alongside the raw code. Mid-stream failures settle through streaming_generator's on_error, which received a rendered message. It now receives the exception itself so the callback can classify it; a string cannot carry a status. Two things the issue lists are left out. Budget and blocked-user rejections (403) and auth rejections (401) write no usage-log row at all today, so there is nothing to stamp; logging them is mozilla-ai#317's scope, not a status-code change. The OTLP import path hardcodes success and drops token-less events, so populating it would change what gets imported rather than just classifying it. Fixes mozilla-ai#433 * fix(usage): classify the streaming tool-loop cap as 422, not a provider fault The non-streaming path stamps 422 on the tool-loop cap so callers can tell a runaway loop from a real outage, but the streaming path raises the cap while the SSE body is already in flight (run_tool_loop_stream is an async generator, so the raise surfaces mid-iteration). It therefore settles through on_error and took failure_status_code's generic fallback, recording 502. The same gateway-owned cap was landing as client_error on non-streaming traffic and provider_error on streaming traffic, so an operator seeing the 502 spike would go chase a provider outage. Check the cap first, matching how _platform.py already treats it as a gateway-side limit rather than an upstream failure. Also from review: - Type UsageErrorCodeRow.error_class as a closed Literal so the set lands in the OpenAPI schema as an enum and a consumer can switch on it exhaustively, matching the module's existing Bucket literal. - Document the status_code filter and the errors_by_status_code taxonomy in docs/api-reference.md, which enumerates the usage filters. - Stop the UsageSummary comment overclaiming: the taxonomy counts sum to error_count only under the top-N cap, and the tail is omitted rather than folded because a null key would collide with the real "no code recorded" group. * fix(usage): scope a bare status_code filter to failure rows `status_code` is documented as classifying a failure, but the filter applied it on its own, so the query did not carry the invariant its own description states. A bare `status_code=429` now also restricts to `status='error'`: it cannot pick up a non-error row if a future write path ever stamps a code on one, and it is served by the existing (status, timestamp) index instead of scanning the window. An explicit `status` still wins, so passing both stays a literal query rather than a silently contradictory one. That is also why the migration adds no index on the column, which the comment there now records: the filter is error-scoped and every aggregate over it is range-bounded, so a (status, status_code) index would tax every write on an append-heavy table to save a scan over one window's failures. Also corrects the `error_class_for` note: gateway-side budget and blocked-user rejections write no row today, and when mozilla-ai#465 starts recording them they will arrive with no status code and read as `unknown`, so the change that stamps a code on those rows owns deciding what they classify as (a 403 for an over-budget user would otherwise be filed as `auth`). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(usage): spell logged failure codes with fastapi status constants The two literal codes recorded on a usage row in `_pipeline.py` (402 for the missing-pricing rejection, 422 for the tool-loop cap) now use the `fastapi.status` constants, matching `_passthrough.py`, which already writes `status.HTTP_402_PAYMENT_REQUIRED` for the same rejection. The two scaffolds are read side by side, so the same value should not be spelled two ways. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(usage): cover the batch and pass-through status_code stamps Two of the three writers that stamp a code were unexercised: the batch create failure (`log_batch_usage`) and the pass-through provider failure (`_passthrough.py`), both of which settle outside the chat pipeline the rest of these tests drive. Either could have stopped recording a code without failing anything. Adds an embeddings provider failure (upstream 429, caller still sees the generic 502) and a batch create failure (upstream 503, which must reach the taxonomy as `provider_error` rather than `unknown`), plus a test that a bare `status_code` filter returns only failures while an explicit `status` still wins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(usage): describe how gateway rejections classify after mozilla-ai#465 `error_class_for` still described the pre-mozilla-ai#465 world, where budget and blocked-user refusals wrote no row at all. They do now, and after this branch they carry the status they returned, so the docstring records what actually reaches the column: an upstream status or one of the gateway's own rejection codes (403 for a blocked or over-budget user, a user/key mismatch, or a model outside a key's allow-list; 402 for missing pricing; 400 for a selector that no longer resolves). It also states plainly that a budget denial currently files as `auth`, and why splitting it out is a deliberate follow-up: the code alone cannot separate the gateway refusing the caller from a provider refusing the gateway, `provider` is NULL only on the gates that refuse before the selector resolves, and the class names are dashboard-visible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(usage): pin the status code on every gateway rejection row mozilla-ai#465 routes ten gateway-side rejections through one writer, so a missed `status_code` there is invisible: the row still appears in the activity log and still counts as a failure, it just classifies as `unknown`, which reads exactly like a row written before the column existed. These pin the code per gate, on both request scaffolds: over budget (403, and asserted to reach the taxonomy as `auth`), user/key mismatch (403), key allow-list (403), unresolvable selector (400, reaching the taxonomy as `client_error`), and the pass-through counterparts of all three. The gates are set up by importing the helpers from the tests that own them, so the two files cannot drift apart. The best-effort unit test supplies the new required argument and asserts it lands on the row, which is the control for the parameter being load-bearing rather than decorative. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(usage): pin the error taxonomy to the summary dimension selector mozilla-ai#469 made every summary breakdown opt-out through `dimensions`, because each one is its own GROUP BY pass and the dashboard's tiles, timelines, and typeaheads read none of them. The failure taxonomy is another such pass, so the rebase put it behind the same selector (`dimensions=status_code`, the one dimension whose field is not `by_<name>`); this pins that: present by default and by name, absent for a totals-only caller, with the failure still counted in `totals` either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(search): record the status code on a refused or failed search mozilla-ai#473 converged search's refusals onto the shared rejection writer, which now requires the status the refusal returns, so thread it through search's local `log_rejection` helper: 400 for a tool that does not resolve or resolves ambiguously, 403 for a key allow-list denial. The row shape is otherwise untouched, including `counts_toward_budget` pinned True by the shared writer. Search's provider-failure row gets `failure_status_code(exc)` too, the same stamp chat, the pass-through routes, and batches use, so search is not the one billed surface whose outages classify as `unknown`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Description
Otari exposed web search only as an in-loop tool (
otari_web_search), so a caller that wants a direct search request had to hold its own provider key and fell off the gateway's billing path entirely. That is the gap blocking MLPA's move from the LiteLLM proxy onto otari: it proxies an Exa search endpoint today and needs an equivalent surface here to keep search on one gateway.This adds
POST /v1/searchandPOST /v1/search/{search_tool_name}, registered standalone-only alongside the other pass-throughs. Both run the same scaffold as a completion (auth, rate limit, budget reservation, usage log, reconcile or refund) and both logendpoint="/v1/search", so one Activity filter covers every search. Exa is wired end to end.Configuration. Search tools are declared under
search_tools:inconfig.yml, keyed by the name callers pass assearch_tool_name(or in the path), and validated at startup so an unsupported provider or a missingapi_keyfails before the first request:optionsis the escape hatch for provider-native knobs the request body has no field for. Request-derived fields win over it, andqueryalways comes from the caller.Wire shape (issue open question 1): LiteLLM's
/v1/search, itself shaped after Perplexity's Search API, so a client migrating off that proxy needs no request changes. Translation to and from Exa's native shape lives insearch_backend.py; the route never sees a provider payload.Billing (issue open question 2): the same budget as completions, one
UsageLogrow, no separate line item. Search bills per request rather than per token, so the row carries zero tokens and a cost taken from Exa's own reportedcostDollars.totalwhen present, falling back to a flat per-request rate configured for<provider>:<tool>under the same convention moderations uses. Since the provider meters itself, search is exempt fromrequire_pricing, like moderations and audio.find_model_pricinggrows ause_defaultsflag that the route sets toFalse. The genai-prices fallback also matches on a bare name, so a search tool an operator happened to name after a real model would otherwise inherit that model's per-million-token rate and be billed under a per-request convention.Dashboard. No change was needed for search to show up: the Activity endpoint filter is a free-form pass-through (
web/src/api/hooks.ts:653) rather than an allowlist, and a zero-token row renders as it should (formatTokens(0)gives"0",formatCost(0.007)gives"$0.0070"). Credentials come fromconfig.ymlonly; dashboard CRUD for search-tool credentials is left for a follow-up, which the issue allows ("credentials from config or the dashboard").PR Type
Relevant issues
Fixes #400
Checklist
tests/unit,tests/integration).make lint,make typecheck,make test).uv run python scripts/generate_openapi.py).Local results: 1059 unit passed / 1 skipped, 839 integration passed / 9 skipped,
ruffandmypy --strictclean,make openapi-checkandmake postman-checkclean. Note for anyone reproducing locally:test_error_detail_leakageandtest_streaming_error_eventfail ifOPENAI_API_KEYis set in the environment, because they then reach the live API instead of failing locally. Both pass with it unset, which is how CI runs.AI Usage
AI Model/Tool used:
Claude Opus 5 via Claude Code.
Any additional AI details you'd like to share:
Implemented by Claude through back-and-forth with @njbrake. The design decisions (LiteLLM-compatible wire shape, shared budget, config-only credentials for now) are his; the code and prose are Claude's.