Context
Policy Studio lets a rule author evaluate fact sets against a ruleset interactively, and the CLI fathom test command runs YAML test cases ({name, facts, expected_decision}) and reports per-case pass/fail. The two do not meet:
GET /studio/api/scenarios (studio_api.py:118) is read-only — it serializes the bundled, hardcoded SCENARIOS tuple from scenarios.py. There is no endpoint to save a fact set as a reusable named test, run a saved suite, or get green/red results.
- The Studio React SPA does have a client-side
saveScn (creem/cs-bench.jsx:95) backed by React useState (creem/cs-shared.jsx:130), but those edits are session-state only (confirmed by the studio_api.py module docstring, line 25) — nothing is persisted server-side and there is no expected-decision / pass-fail concept anywhere in src/fathom/studio/ (grep for expected_decision returns nothing under studio).
- The CLI's decision-comparison logic (
result.decision == expected) is inlined in the test command (cli.py:423-433), not extracted, so any Studio runner would have to duplicate it.
Net: the core authoring loop — edit rule, re-run saved scenarios, confirm no regressions — cannot be closed in the browser.
Task
- Add
POST /studio/api/scenarios to persist a named scenario (facts + expected_decision, reusing the CLI test-case shape).
- Add a run endpoint (e.g.
POST /studio/api/scenarios/run, optionally per-ruleset) that executes saved scenarios against the real engine and returns pass/fail per case.
- Extract the decision-comparison logic out of
cli.py test into a shared helper and call it from both the CLI and the Studio run endpoint (avoid divergence — this is the load-bearing part of the task).
- Persist scenarios file-backed under the ruleset root (
_ruleset_root() in studio_api.py); reuse the existing resolve_ruleset path-jail.
- Surface a UI to list/run scenarios with pass/fail badges. Decide the UI surface first (see Acceptance criteria) — the live Studio is the React SPA under
creem/, while panels.py + templates/ are a parallel HTMX surface; the original proposal assumed HTMX but the SPA already owns scenario authoring.
Where
src/fathom/studio/studio_api.py (read-only /scenarios at line 118; _ruleset_root() line 60; _resolve() line 66; _evaluate() line 220).
src/fathom/studio/scenarios.py (bundled Scenario dataclass + SCENARIOS).
src/fathom/cli.py (test command line 352; comparison logic lines 423-433 to extract).
- UI:
src/fathom/studio/creem/cs-bench.jsx (existing client-side saveScn) and/or src/fathom/studio/panels.py + src/fathom/studio/templates/ — pending the surface decision.
- Tests:
src/fathom/studio/tests/test_studio_api.py; CLI fixtures pattern in tests/test_cli.py.
Acceptance criteria
Size: L
Context
Policy Studio lets a rule author evaluate fact sets against a ruleset interactively, and the CLI
fathom testcommand runs YAML test cases ({name, facts, expected_decision}) and reports per-case pass/fail. The two do not meet:GET /studio/api/scenarios(studio_api.py:118) is read-only — it serializes the bundled, hardcodedSCENARIOStuple fromscenarios.py. There is no endpoint to save a fact set as a reusable named test, run a saved suite, or get green/red results.saveScn(creem/cs-bench.jsx:95) backed by ReactuseState(creem/cs-shared.jsx:130), but those edits are session-state only (confirmed by thestudio_api.pymodule docstring, line 25) — nothing is persisted server-side and there is no expected-decision / pass-fail concept anywhere insrc/fathom/studio/(grepforexpected_decisionreturns nothing under studio).result.decision == expected) is inlined in thetestcommand (cli.py:423-433), not extracted, so any Studio runner would have to duplicate it.Net: the core authoring loop — edit rule, re-run saved scenarios, confirm no regressions — cannot be closed in the browser.
Task
POST /studio/api/scenariosto persist a named scenario (facts+expected_decision, reusing the CLI test-case shape).POST /studio/api/scenarios/run, optionally per-ruleset) that executes saved scenarios against the real engine and returns pass/fail per case.cli.py testinto a shared helper and call it from both the CLI and the Studio run endpoint (avoid divergence — this is the load-bearing part of the task)._ruleset_root()instudio_api.py); reuse the existingresolve_rulesetpath-jail.creem/, whilepanels.py+templates/are a parallel HTMX surface; the original proposal assumed HTMX but the SPA already owns scenario authoring.Where
src/fathom/studio/studio_api.py(read-only/scenariosat line 118;_ruleset_root()line 60;_resolve()line 66;_evaluate()line 220).src/fathom/studio/scenarios.py(bundledScenariodataclass +SCENARIOS).src/fathom/cli.py(testcommand line 352; comparison logic lines 423-433 to extract).src/fathom/studio/creem/cs-bench.jsx(existing client-sidesaveScn) and/orsrc/fathom/studio/panels.py+src/fathom/studio/templates/— pending the surface decision.src/fathom/studio/tests/test_studio_api.py; CLI fixtures pattern intests/test_cli.py.Acceptance criteria
creem/) or the HTMX panels (panels.py/templates/), since both surfaces exist and the SPA already does session-only scenario authoring.fathom testand the new Studio run endpoint (no duplication).fathom teston the same YAML cases.uv run pytest).uv run ruff check src/ tests/anduv run mypy src/are clean.Size: L