feat(dashboard): show requests in flight on the activity page - #549
Conversation
WalkthroughThe gateway now tracks active requests in a process-local registry, exposes them through ChangesIn-flight activity
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Track per-worker in-memory requests entered at the provider-resolving harness points (so refused requests never appear) and exited by ASGI cleanup middleware (so a streaming response stays listed until its body is consumed, and a dropped worker leaks nothing). Surface them on the Activity page as a live panel, capped at the 50 longest-running, that ticks elapsed time between polls. Regenerated the dashboard bundle, OpenAPI spec, and Postman collection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b781fc5 to
f942ff3
Compare
Search passed through to the provider without registering, so a slow search was the one surface invisible to the live in-flight view, and the panel's claim that every served request is listed was not quite true. Enter the request in the in-flight registry after every gate (tool resolution, allowlist, budget) and before the provider call, tagged with the same endpoint/model/provider the usage row carries. Covers both the body-selected and path-selected forms. Also rebuild the committed dashboard bundle from a clean dependency install; the previously committed bundle was built from a stale node_modules and failed the dashboard CI freshness check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Requests still running now appear in the activity table itself, pinned above the settled rows with the status "in progress" and, in place of a total time, a wait that ticks up. When the request lands its row resolves in place into the success or error row it became, which is the reading a separate panel above the table could not give: the operator watched a request leave one list and, seconds later, appear in another. The synthetic row is shaped as a UsageEntry, so the table's columns, widths, selection and detail machinery need no second code path; the outcome fields are null, which the existing formatters already render as their own placeholder. Ordered newest-first like the rest of the table (the endpoint still serves the longest-running first, the order its cap applies in) so a request stays in the slot it appeared in as it settles. A live row is kept out of everything it is not part of: the paginator's count, the bulk delete and reprice selection, and the request detail, none of which exist for a request with no usage row yet. It is dropped from view rather than shown misleadingly wherever the current view could not honestly include it: page 2 onward, a window ending in the past, and any filter on something the request has not got yet (status, priced, tool, source, session). The identity filters do apply, so they are matched client-side. Also from review: * Throttle the settled-request log refetch to once per 10 seconds, matching the log's own staleTime. The registry is per process, so a deployment running several otari processes behind a load balancer answers consecutive polls from different processes and a still-running request reads as settled on every one of them; unthrottled that refetched the log and its COUNT(*) every two seconds for as long as the gateway had any traffic at all. * track_request now drops any id already on the scope before registering, so a second registration on one request cannot strand the first entry for the life of the process. No path reaches it twice today. * Correct the docs: the panel covers completions, embeddings, images, audio and searches rather than literally every request (a batch runs provider-side after its submission returns), gateway_active_requests is per process, and multi-process deployments are several otari processes behind a load balancer, not uvicorn workers, which the CLI refuses. * Name all three registration points in src/gateway/AGENTS.md, so a new dispatch scaffold is known to need its own track_request call. Verified against a live gateway: a streaming request is listed mid-flight with the right identity and a growing wait, its entry clears once the body is consumed, an abandoned stream leaks nothing, a refused request is never listed, and 58 concurrent requests report total=58 with the 50 longest-running serialized in order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to an independent review of this branch. Five findings, one functional and four where the prose asserted more than the code delivers. The settled-request refetch throttle discarded a settle rather than deferring it. Inside the 10 second window the effect returned early, but it had already replaced the seen-id set, so the settled id was gone from both the ref and the next poll and nothing re-detected it. Any two requests landing inside one window left the second one's row missing from the log until an unrelated later settle or a manual refresh, which is the ordinary case on a gateway with traffic and defeats the behaviour the live rows exist for. A `settlePending` ref now carries the settle to the first poll past the window. The effect also keys on `dataUpdatedAt`, because an idle registry answers every poll with a structurally identical empty payload that TanStack's structural sharing returns as the same object, so keying on the payload alone stopped the effect firing once the last request landed and a deferred settle would never have been reconsidered. The multi-replica throttle test still passes: that case re-flags every poll either way and still costs one refetch per interval. The claim that a refused request never appears as in progress was wrong in five places, including the user-facing guide and the bundled dashboard copy of it. `prepare_gateway_tools` runs after the registration point, so a request refused by an input guardrail, an unresolvable MCP id, or a bad tool declaration is already listed, and a guardrail is an outbound call rather than a local check. An unresolvable selector reaches the registration point too. Each site now says which gates the claim covers, and that a later refusal does appear while its check runs, which is the honest reading: the gateway is working on the request by then. `test_a_refused_request_is_never_registered` could not fail for the reason it named. It asserted the registry was empty after the response, and cleanup is unconditional, so that held whether or not anything had been recorded. It now asserts on `InFlightRegistry.begin` call counts, which is what surfaced the unresolvable-selector case above. `gateway_active_requests` was offered as this panel's per-process count in the guide and as a process-wide total in the endpoint docstring. It counts every HTTP scope except `/metrics`, including the dashboard's own poll of this endpoint, so an idle gateway with the Activity page open reports at least one while the table shows nothing. Both sites now say it measures a different, larger population. The pulse dot on an in-progress row animated indefinitely without honouring `prefers-reduced-motion`, which globals.css already respects for two other animations. Regenerated the OpenAPI spec and Postman collection for the endpoint docstring, and the dashboard bundle for the page and guide changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/integration/test_in_flight_requests.py (1)
129-142: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider pinning the serialization cap in this test module.
The dashboard renders
total - requests.lengthas "further requests are in flight", so the endpoint's cap is a contract the UI depends on. This test proves ordering, but nothing here proves thattotalexceedslen(requests)once the cap bites. A test that seeds more than the cap and asserts both the list length and the largertotalwould lock that behaviour down.♻️ Sketch of the extra case
def test_the_response_caps_the_list_and_still_reports_the_true_total( client: TestClient, master_key_header: dict[str, str] ) -> None: registry = _registry(client) ids = [registry.begin(endpoint="/v1/chat/completions", model=f"m{i}") for i in range(60)] try: payload = client.get(IN_FLIGHT, headers=master_key_header).json() finally: for request_id in ids: registry.finish(request_id) assert payload["total"] == 60 assert len(payload["requests"]) == 50 # The cap keeps the longest-running, which is what the panel claims to show. assert [entry["model"] for entry in payload["requests"]] == [f"m{i}" for i in range(50)]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/test_in_flight_requests.py` around lines 129 - 142, Add a test in the in-flight request test module that begins more requests than the endpoint serialization cap, then asserts the response retains the true total while limiting requests to the cap. Verify the returned models are the oldest entries in order, and always finish all registered requests during cleanup.web/src/pages/ActivityPage.test.tsx (1)
1542-1663: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDrive the polling tests with fake timers.
Call
vi.useFakeTimers()per test and restorevi.useRealTimers()during cleanup. Advance the 2-second polling interval and 10-second throttle withawait act(async () => vi.advanceTimersByTimeAsync(...)). Do not rely onwaitForto advance fake time. This makes TanStack Query polling and theDate.now()throttle deterministic.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/src/pages/ActivityPage.test.tsx` around lines 1542 - 1663, Update the three polling tests around “re-reads the log once a tracked request settles” to call vi.useFakeTimers() and restore vi.useRealTimers() during cleanup. Replace timing-dependent waitFor progression with await act(async () => vi.advanceTimersByTimeAsync(...)) for the 2-second polling interval and 10-second throttle, while retaining waitFor only for state assertions; ensure TanStack Query polling and Date.now()-based throttling advance deterministically.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/src/pages/ActivityPage.tsx`:
- Around line 1181-1184: Update the in-flight rows useMemo to return the empty
result whenever inFlight.isError is true, preventing stale successful data from
remaining visible after a failed poll. Add inFlight.isError to the memo
dependency array, and include the in-flight error in the existing ErrorBanner
chain so operators see the polling failure.
---
Nitpick comments:
In `@tests/integration/test_in_flight_requests.py`:
- Around line 129-142: Add a test in the in-flight request test module that
begins more requests than the endpoint serialization cap, then asserts the
response retains the true total while limiting requests to the cap. Verify the
returned models are the oldest entries in order, and always finish all
registered requests during cleanup.
In `@web/src/pages/ActivityPage.test.tsx`:
- Around line 1542-1663: Update the three polling tests around “re-reads the log
once a tracked request settles” to call vi.useFakeTimers() and restore
vi.useRealTimers() during cleanup. Replace timing-dependent waitFor progression
with await act(async () => vi.advanceTimersByTimeAsync(...)) for the 2-second
polling interval and 10-second throttle, while retaining waitFor only for state
assertions; ensure TanStack Query polling and Date.now()-based throttling
advance deterministically.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e2fd7a09-f20e-4089-8943-a05a5e2bd755
⛔ Files ignored due to path filters (1)
docs/public/openapi.jsonis excluded by!docs/public/openapi.json
📒 Files selected for processing (40)
docs/dashboard.mddocs/public/otari.postman_collection.jsonscripts/sdk_codegen/sdk-endpoints.txtsrc/gateway/AGENTS.mdsrc/gateway/api/routes/_passthrough.pysrc/gateway/api/routes/_pipeline.pysrc/gateway/api/routes/search.pysrc/gateway/api/routes/usage.pysrc/gateway/inflight.pysrc/gateway/main.pysrc/gateway/static/dashboard/assets/ActivityPage-BUTTJXqS.jssrc/gateway/static/dashboard/assets/ActivityPage-DVvHOnJV.jssrc/gateway/static/dashboard/assets/BudgetsPage-BRe5t28X.jssrc/gateway/static/dashboard/assets/ConfirmDialog-Ql3FtMnI.jssrc/gateway/static/dashboard/assets/DocsPage-Ci3s5_SN.jssrc/gateway/static/dashboard/assets/KeysPage-Doebe-tT.jssrc/gateway/static/dashboard/assets/ModelScopeControl-Dp3Sygm_.jssrc/gateway/static/dashboard/assets/ModelsPage-CeOCF2a1.jssrc/gateway/static/dashboard/assets/ModelsPage-Dus_If6B.jssrc/gateway/static/dashboard/assets/OverviewPage-C5V0M17G.jssrc/gateway/static/dashboard/assets/ProvidersPage-BGOpAogm.jssrc/gateway/static/dashboard/assets/ProvidersPage-DcqFf570.jssrc/gateway/static/dashboard/assets/RoutingPage-XwtlENNP.jssrc/gateway/static/dashboard/assets/SettingsPage-BsU5fEB5.jssrc/gateway/static/dashboard/assets/TablePagination-CbgAoqkJ.jssrc/gateway/static/dashboard/assets/ToolsGuardrailsPage-CxXrvYsP.jssrc/gateway/static/dashboard/assets/ToolsGuardrailsPage-D3KyltPY.jssrc/gateway/static/dashboard/assets/UsagePage-DP3H5pdP.jssrc/gateway/static/dashboard/assets/UsersPage-CgUvLTG9.jssrc/gateway/static/dashboard/assets/index-94k-6miE.jssrc/gateway/static/dashboard/assets/index-CI-EveRE.csssrc/gateway/static/dashboard/assets/index-CyNKaqo6.jssrc/gateway/static/dashboard/index.htmltests/integration/test_in_flight_requests.pytests/integration/test_search_endpoint.pytests/unit/test_inflight_registry.pyweb/src/api/hooks.tsweb/src/api/types.tsweb/src/pages/ActivityPage.test.tsxweb/src/pages/ActivityPage.tsx
💤 Files with no reviewable changes (5)
- src/gateway/static/dashboard/assets/ProvidersPage-BGOpAogm.js
- src/gateway/static/dashboard/assets/ToolsGuardrailsPage-CxXrvYsP.js
- src/gateway/static/dashboard/assets/ModelsPage-CeOCF2a1.js
- src/gateway/static/dashboard/assets/index-94k-6miE.js
- src/gateway/static/dashboard/assets/ActivityPage-DVvHOnJV.js
|
@coderabbitai review Note: posted by Claude at @njbrake's request. It is a bot trigger, nothing more. |
|
|
Found by CodeRabbit on this branch, and reproduced: with the endpoint answering 503, an in-progress row stayed on screen and its wait climbed from 3s to 9s. The rows were derived from `inFlight.data` alone, and TanStack keeps the last successful payload after a failed refetch. So a gateway restart, an expired session, or a network blip left the table asserting that work was still running when it may have landed minutes ago, with the wait counting up against an anchor that could no longer move. `useInFlightRequests` already refuses to cache that state across mounts, with `staleTime: 0` and no placeholder data, for exactly this reason; reaching it by another route defeats that. The rows are now dropped when the query is in error, and the in-flight error joins the page's error banner last, behind a failure to read the log itself, so a live view that has gone quiet is distinguishable from a gateway that has rather than showing as rows quietly disappearing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
web/src/pages/ActivityPage.test.tsx (1)
1542-1693: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoffConsider fake timers for the throttle/failure tests to speed up the suite.
These three tests wait on real wall-clock time: up to 8s, 20s, and 25s inside
waitFor, with per-test timeouts of 15s, 30s, and 45s. That is a lot of real time spent in CI for one describe block, and it also leaves a small flakiness window if CI is under load and the retry backoff or poll interval runs a bit slower than expected.Using
vi.useFakeTimers()withvi.advanceTimersByTimeAsync(...)would let these tests assert the same throttle and retry behavior deterministically, without actually waiting in real time. Not a blocker since the tests are correct as written, just a nice-to-have if the suite starts to feel slow.Do you want a hand sketching the fake-timer version of one of these tests as a starting point?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/src/pages/ActivityPage.test.tsx` around lines 1542 - 1693, Speed up the timing-sensitive tests “does not re-read the log on every poll when replicas alternate,” “drops the live rows when the in-flight poll starts failing,” and “still picks up a settle that the throttle deferred” by using vi.useFakeTimers and vi.advanceTimersByTimeAsync. Preserve their existing throttle, polling, failure, and deferred-settle assertions while avoiding long real-time waitFor intervals; restore real timers after each test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@web/src/pages/ActivityPage.test.tsx`:
- Around line 1542-1693: Speed up the timing-sensitive tests “does not re-read
the log on every poll when replicas alternate,” “drops the live rows when the
in-flight poll starts failing,” and “still picks up a settle that the throttle
deferred” by using vi.useFakeTimers and vi.advanceTimersByTimeAsync. Preserve
their existing throttle, polling, failure, and deferred-settle assertions while
avoiding long real-time waitFor intervals; restore real timers after each test.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d626c580-3664-423b-a697-63fb7ef170e1
📒 Files selected for processing (19)
src/gateway/static/dashboard/assets/ActivityPage-DOJHPUQ4.jssrc/gateway/static/dashboard/assets/BudgetsPage-CdiIH4HU.jssrc/gateway/static/dashboard/assets/ConfirmDialog-hT_rddGe.jssrc/gateway/static/dashboard/assets/DocsPage-C93UDDOQ.jssrc/gateway/static/dashboard/assets/KeysPage-hJurDqS1.jssrc/gateway/static/dashboard/assets/ModelScopeControl-BD06nruZ.jssrc/gateway/static/dashboard/assets/ModelsPage-BoPl3AYj.jssrc/gateway/static/dashboard/assets/OverviewPage-CA7PVlzB.jssrc/gateway/static/dashboard/assets/ProvidersPage-sUbxKG6D.jssrc/gateway/static/dashboard/assets/RoutingPage-B0PbG8Zg.jssrc/gateway/static/dashboard/assets/SettingsPage-CjDCXBZG.jssrc/gateway/static/dashboard/assets/TablePagination-CLVORqYy.jssrc/gateway/static/dashboard/assets/ToolsGuardrailsPage-Dv1tsNMm.jssrc/gateway/static/dashboard/assets/UsagePage-BDaceVBD.jssrc/gateway/static/dashboard/assets/UsersPage-DqeEPtPm.jssrc/gateway/static/dashboard/assets/index-BM-zs2wX.jssrc/gateway/static/dashboard/index.htmlweb/src/pages/ActivityPage.test.tsxweb/src/pages/ActivityPage.tsx
🚧 Files skipped from review as they are similar to previous changes (2)
- src/gateway/static/dashboard/index.html
- web/src/pages/ActivityPage.tsx
Description
The Activity page only showed settled usage, so a long-running provider request (a local model can take 30s+) was invisible. This adds a live "requests in flight" panel that names the requests the gateway is currently serving and how long each has been running.
In-flight requests are tracked by a per-worker in-memory registry, entered at the request harness points that already resolve provider/model (so a refused request never appears) and exited by ASGI cleanup middleware (so a streaming response stays listed until its body is consumed, and a dropped worker leaks nothing). The panel renders only while requests are running, is capped at the 50 longest-running, ticks elapsed time locally between polls, and the log below it refetches when a tracked request settles.
PR Type
Relevant issues
Fixes #526
Checklist
tests/unit,tests/integration).make lint,make typecheck,make test).uv run python scripts/generate_openapi.py).AI Usage
AI Model/Tool used:
Claude
Any additional AI details you'd like to share:
🤖 Generated with Claude Code
Summary
GET /v1/usage/in-flightendpoint.Benefits
Technical notes