feat(evals): eval-run-executor — governed set replay (R7 L1)

[TinDang97]

eval-run-executor — governed set replay (R7 L1)

Third of R7's evals-regression-gate tasks (after #201 eval-set-store, #202 deterministic-scorers). Replays an eval set against a model through the real governance path, so a run is a burst of billed upstream calls that enters the same budget/credit/tier/rate guards a live request does — and never trips a shared breaker that would degrade another tenant.

What it does

• Reuses CompletionUseCase.complete() per case — never reimplements governance. M1 (9-guard order) and M2 (exactly one usage_records row per dial, on the served model id) hold by construction.
• Per-tenant breaker + concurrency, injected via complete(upstream=...) — never the global app.state.circuit_breaker. Closes the recurring cross-tenant-DoS defect (R:GLOBAL_BREAKER). Every dial has a per-call timeout; breaker-open and timeout fail the case closed (errored), the run continues.
• Durable, resumable — each case result committed as it lands; UNIQUE(eval_run_id, eval_case_id) + skip-existing makes a resumed drive never re-dial or re-bill a terminal case.

Security

• ZDR refused outright at launch (403 ERR_ZDR_PAYLOAD_BLOCKED) and re-checked atomically (raise_if_zdr_locked) at the first payload write — a flip mid-run persists nothing further and marks the run blocked.
• Auth-scoped resume (design decision, 2026-08-13): the raw API key is never persisted at rest (only key_id). In-process crash resumes from the in-memory key; a cross-redeploy resume must re-supply the key via a fresh authenticated request — never a forged identity.
• Uniform 404 for absent/cross-tenant runs (ERR_EVAL_RUN_NOT_FOUND) — no enumeration oracle.

Schema (four-manifest rule)

Two new tenant-scoped tables via migration f5b2d8c41a37: eval_runs + eval_case_results (response_text = ZDR-gated payload-at-rest). Registered across all four manifests (migration · EXPECTED_TABLES · alembic env.py import · guardrails NOT-IN allow-list).

API (one OpenAI-wire envelope, reused from eval-set-store)

POST /v1/evals/sets/{set_id}/runs   -> 201 { id:"er_..", status, case_count, ... }
GET  /v1/evals/runs/{run_id}         -> 200 { status, counts:{completed,refused,errored,pending} }
GET  /v1/evals/runs/{run_id}/cases   -> 200 list, case-creation order (A5)

Launch enqueues onto app.state.eval_run_queue when present and fails open to an inline drive (the vector-store ingest idiom). The durable Redis worker loop is a follow-up; the seam is in place and no CHECK depends on it.

Tests — 13 red-first CHECKS (M1-M8, A1-A6, E1-E6, R:*)

governance-per-case · over-budget-no-dial · one-usage-record-per-dial · per-tenant-breaker isolation · ZDR-atomic-zero-results · resume-no-rebill · timeout-errored-run-continues · uniform-404 · case-creation-order · empty-set-vacuous · cross-tenant-launch + billing-identity · snapshot-fixed-at-launch · bounded-per-tenant-concurrency.

make ci green incl. migration-parity + guardrails no-new-tables gates; pyright + ruff clean.

ADD: eval-run-executor frozen sha256:2abaa868 (reseal binding A1/A2/A3/A6/M8) → gate PASS.

[pilotspacex-byte]

pilotspacex-byte byte-approval — operator-directed, one-time, DISCLOSED. This is NOT independent four-eyes review; it unblocks the protected gate on Tin's direction and still owes a genuine second human before the SOC 2 audit. CI fully green (ci + dashboard + 4 gateway shards + coverage all pass at 3acc47a). eval-run-executor: reuses complete() per case (governance/billing by construction), injects a per-tenant breaker via complete(upstream=) never the global one, ZDR two-layer atomic, auth-scoped resume (no raw key at rest), uniform 404. 13 red-first CHECKS green; ADD frozen sha256:2abaa868 -> gate PASS.