feat(dashboard): evals console — verdict-first, read-focused (R7) - #205
Merged
Conversation
…console backend)
The dashboard console needs to read and author evals with the operator's session
JWT — never a raw API key. Add a control-plane twin of the /v1/evals surface,
authenticated by get_current_identity (Bearer JWT -> Identity{tenant_id, role})
instead of an sk-... key, that REUSES the existing stores and the one verdict core
rather than re-deriving anything (R:LOGIC_FORK):
GET /admin/evals/sets list the session tenant's sets
GET /admin/evals/sets/{set_id} set detail: cases + runs + baseline_run_id
GET /admin/evals/runs/{run_id}/verdict verdict vs pinned baseline (shared core)
GET /admin/evals/runs/{run_id}/cases per-case DIFF rows (snapshot case ⋈ result)
POST /admin/evals/sets author a set (session)
POST /admin/evals/sets/{set_id}/cases author a case (session)
PUT /admin/evals/sets/{set_id}/baseline pin a baseline (session)
There is deliberately NO launch route here (M2, R:RAW_KEY_IN_CONSOLE): launching
dials upstreams and must bill the launching key as live traffic, which needs a raw
key the session must never hold. Launch stays on the /v1 API-key path; the console
links to it. The module never imports _extract_raw_key.
Every store call passes identity.tenant_id; an absent OR cross-tenant set/run is a
uniform 404 (M1, R:CROSS_TENANT). Unlike /v1 (OpenAI {"error":{...}} body for SDK
compat) this control-plane surface raises ProblemError so failures render as native
RFC-9457 problem+json — the shape the dashboard's BffError.problem reads.
To keep /admin and /v1 verdicts byte-identical, the verdict-assembly is extracted
from the /v1 verdict handler into a shared build_verdict_body() that both call — the
scoring/verdict is computed once, in one place. Adds SqlAlchemyEvalRunStore.list_runs
(a set's runs, newest first, tenant-scoped) for the set-detail read. No new table:
EXPECTED_TABLES / migrations / env.py / guardrails allow-list are untouched.
Tests (tests/evals_admin/, 4 red-first CHECKS, proven red by un-registering the
router): session-scoped reads with cross-tenant 404; /admin verdict == /v1 verdict
(reuse); session authoring writes tenant-scoped rows; and a route-table assertion
that there is no launch route and the module never binds a raw key.
author: Tin Dang
…diff (R7 evals-console)
The user-facing half of evals-console: a verdict-first console section for reading
LLM eval regression gates, with the per-case expected-vs-actual DIFF as the signature
element (experience.md's UDD bar — named IA, primary object, signature element; never
a bare CRUD table).
IA: Sets list -> Set detail (cases · runs · pinned baseline) -> Run verdict. Routes
under app/(app)/app/evals/ inherit the group DashboardShell; each page leads with the
verdict, not a row count.
components/evals/VerdictBanner pass/fail/no_baseline/pending, icon+text (never color
alone); a pending case overrides the endpoint verdict
components/evals/CaseDiffRow the signature element — status + assertion(kind+expected)
+ actual response + reason; refused/errored/pending show
status+reason, never a fabricated actual
components/evals/{EvalsListPage,SetDetailPage,RunVerdictPage,CaseDiffList,EvalStatusBadge}
components/evals/{CreateSetDialog,AddCaseDialog,PinBaselineControl} session authoring
Contract invariants held: verdict is the run page's primary object, before any table
(M3, R:VERDICT_NOT_PRIMARY); exactly one h1 via PageHeader, keyboard-navigable rows
(tabIndex/aria-label/Enter+Space/focus-visible ring), axe zero serious/critical across
all three routes (M5); the section renders a visible Loading — never null — while identity
is unresolved (M6, R:NULL_RENDER_LEAK); errors name the failed subsystem + a retry, never
the user's data (M6); no launch button, no raw key in the browser (launch is an
informational "/v1 API key" note). Nav entry in the Insights group, all roles.
Per-case pass/fail is the backend's AUTHORITATIVE `passed` bool (same deterministic scorer
the verdict counts with) — the row renders it directly and never re-derives pass/fail
from the payload. An earlier draft compared expected==actual client-side, which would
fork scoring and mislabel a `contains` case (expected "echo" vs actual "echo:one" is a
PASS the scorer sees but string equality misses); the backend cases endpoint now emits
`passed` and a regression test guards the `contains` case.
Tests: tests/evals-console.test.tsx (8 named vitest CHECKS, MSW + Testing Library) and
e2e-a11y/a11y.spec.ts::test_evals_routes_axe_clean (Playwright + axe). Extends the authed
capture harness (e2e-review/{capture.spec.ts, fixtures.ts}, additively) and updates the
hardcoded nav-link counts in tests-bff/nav-role-filter.test.tsx for the new entry.
ADD gate PASS (evals-console frozen sha256:60cf00293d) — 13 CHECKS bound across pytest +
vitest + playwright, receipt runs/2.md. next build clean, all three routes present.
author: Tin Dang
The list_runs addition for the evals-console set-detail read was lint-clean (ruff check) but not format-clean; `make lint` runs `ruff format --check` and aborted every gateway shard before tests ran. Collapse the signature to ruff format s one-line form. No behaviour change. author: Tin Dang
…im set/case authoring
Per direction, narrow the evals console to its core value — reading the regression
gate verdict with per-case drill-down — and pull in-console set/case authoring back
to a follow-up. Authoring already lives on the /v1 API; keeping it out of the console
tightens the surface and keeps a raw key even further from the browser.
Contract refrozen (sha256:a09b4f0786d59b93):
- M2 is now READ + baseline-pin ONLY (no set/case authoring, no launch). Pinning a
baseline stays: the verdict READ needs a reference run, else every run reads
no_baseline.
- E4: an empty sets list points the operator to the /v1 API, not an in-console create.
- CHECKS renamed: test_admin_evals_authoring_session_writes -> _session_pins_baseline;
test_empty_sets_shows_create_affordance -> _points_to_api.
Backend (src/gateway/evals/console/router.py):
- Remove POST /admin/evals/sets and POST /admin/evals/sets/{id}/cases; the only write
is PUT .../baseline. Drop the now-unused Create*UseCase / name-validation imports.
- The route-table test now asserts the console exposes NO POST route at all and its
only mutating method is the baseline PUT.
Frontend (components/evals):
- Delete CreateSetDialog + AddCaseDialog. EvalsListPage drops the "New eval set"
button; its empty state directs to POST /v1/evals/sets. SetDetailPage drops
"Add case"; keeps the read-only cases table, the runs table, and PinBaselineControl.
- Remove the dead POST fixtures from e2e-review/fixtures.ts (additive-only otherwise).
ADD gate PASS (13 CHECKS bound across pytest + vitest + playwright, receipt runs/3.md).
next build clean; ruff + pyright + guardrails + repo-hygiene green.
author: Tin Dang
TinDang97
changed the title
pilotspacex-byte
approved these changes
Aug 14, 2026
pilotspacex-byte
left a comment
pilotspacex-byte
left a comment
Contributor
There was a problem hiding this comment.
Disclosed byte-approval (operator-directed, NOT independent four-eyes). Read-focused evals-console: ADD gate PASS at sha256:a09b4f07, 13 CHECKS bound across pytest+vitest+playwright; CI green on ci/dashboard/coverage + 4 gateway shards. Owes a genuine second reviewer before the SOC-2 audit.
TinDang97
added a commit
that referenced
this pull request
Aug 14, 2026
Close and archive R7 milestone `evals-regression-gate` (7/7 exit criteria) and cut the release. R7 is the first product-feature milestone since R6's substrate work: a tenant runs a named eval set against a candidate model and gets a scored pass/fail verdict against a pinned baseline, so a model swap is proven safe BEFORE it ships — without touching the hot proxy path, auto-promoting on a verdict, or scoring with an LLM judge. Shipped across the milestone (all merged, all gated PASS): - eval-set-store (#201) — /v1/evals/sets + cases; ZDR disposition frozen first - deterministic-scorers (#202) — exact · contains · regex · json-schema, re-run stable - eval-run-executor (#203) — governed set replay: billed traffic, per-tenant breaker - baseline-and-verdict (#204) — pin baseline, exact integer verdict, no_baseline state - evals-console (#205) — verdict-first, read-focused, reuses the /v1 verdict core Version bumped across all eight sites per the RELEASES.md checklist: pyproject + `__version__` fallback, Chart.yaml version (0.2.1->0.2.2) + appVersion, values.yaml gateway+dashboard tags, values-prod.yaml both `-prod` overrides. Dashboard package.json stays independently versioned (0.1.0) — a gateway release does not ship a phantom UI release. Guards green: tests/release_provenance (served /openapi.json == pyproject == fallback, RELEASES newest >= pyproject) and the production digest-pin / multi-arch image guards (14 passed). Waivers: 0.14.0's suite-stability and pgvector-deploy-runbook carry forward unchanged (both RISK-ACCEPTED, owner Tin Dang, expire 2026-09-30); ci-restoration stays DISCHARGED. No new waivers. REMAINING IS HUMAN-RUN (tagging PUBLISHES images via publish-images.yml): 1. git tag -a v0.15.0 on the merge commit, and push 2. verify the publish-images run is green (reads back the manifest for both images; do not deploy a tag whose publish job is not green) 3. staging first, following the pgvector runbook (dump/restore path) OPEN, carried to R8 `soc2-groundwork` and NOT waived: the `required_approving_review_count: 1` gate was satisfied on all five R7 PRs by an operator-directed self-approval via a second account (`pilotspacex-byte`), disclosed on each PR. That evidences a green required check on the merged artifact; it does NOT evidence independent four-eyes review. Every R7 merge still owes a genuine second human with write access before the SOC 2 audit. author: Tin Dang
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
evals-console (R7 · evals-regression-gate L3)
The operator-facing console for the eval regression gate — a verdict-first, read-focused section with the per-case expected-vs-actual diff as the signature element (experience.md's UDD bar). Last task in the
evals-regression-gatemilestone.Scope (frozen
sha256:a09b4f07): read the verdict + pin the baseline. Set/case authoring and run launch both stay on the/v1API — keeping a raw key out of the browser and the console tight around its core value.Backend — session-authed
/admin/evals/*(control-plane twin of/v1/evals)Authenticated via
get_current_identity(JWT →Identity{tenant_id, role}), reusing the existing stores + the one verdict core — no logic fork:GET /admin/evals/sets·GET /admin/evals/sets/{id}(cases + runs +baseline_run_id)GET /admin/evals/runs/{id}/verdict(sharedbuild_verdict_body) ·GET /admin/evals/runs/{id}/cases(per-case diff rows, with the authoritativepassed)PUT /admin/evals/sets/{id}/baseline— the only writeNo set/case authoring route, no launch route, no raw key (
R:RAW_KEY_IN_CONSOLE). Tenant-scoped uniform 404 (R:CROSS_TENANT). Native problem+json errors (whatBffError.problemreads). The/v1verdict handler was refactored to a sharedbuild_verdict_body()so/adminand/v1verdicts are byte-identical by construction.Frontend — verdict-first console section
Routes under
app/(app)/app/evals/inherit the group shell; each page leads with the verdict, not a table.VerdictBanner(pass/fail/no_baseline/pending, icon+text),CaseDiffRow(status + assertion + actual + reason; refused/errored/pending show status+reason, never a fabricated actual),PinBaselineControl. Empty states direct the operator to the/v1API. Nav entry in the Insights group, all roles.Invariants held: verdict is the primary object (
R:VERDICT_NOT_PRIMARY); one h1, keyboard-navigable rows, axe zero serious/critical (M5); visible Loading — never null — while identity resolves (R:NULL_RENDER_LEAK); errors name the subsystem + a retry (M6).Authoritative per-case pass/fail
The per-case badge renders the backend's
passedbool from the same deterministic scorer the verdict counts with — never re-derived client-side (an earlier draft compared expected==actual, which forks scoring and mislabels acontainscase). Acontainsregression test guards it.Verification
sha256:a09b4f07) — 13 CHECKS bound across pytest + vitest + Playwright in one merged receipt.test_evals_routes_axe_clean;next buildclean (all three routes present); eslint 0 errors.Disclosure
Approved via the
pilotspacex-byteaccount — disclosed, not independent four-eyes; on the SOC-2 tally to settle with a genuine second reviewer before audit.