Skip to content

Add save-and-run test scenario management to Policy Studio #142

Description

@se-jo-ma

Context

Policy Studio lets a rule author evaluate fact sets against a ruleset interactively, and the CLI fathom test command runs YAML test cases ({name, facts, expected_decision}) and reports per-case pass/fail. The two do not meet:

  • GET /studio/api/scenarios (studio_api.py:118) is read-only — it serializes the bundled, hardcoded SCENARIOS tuple from scenarios.py. There is no endpoint to save a fact set as a reusable named test, run a saved suite, or get green/red results.
  • The Studio React SPA does have a client-side saveScn (creem/cs-bench.jsx:95) backed by React useState (creem/cs-shared.jsx:130), but those edits are session-state only (confirmed by the studio_api.py module docstring, line 25) — nothing is persisted server-side and there is no expected-decision / pass-fail concept anywhere in src/fathom/studio/ (grep for expected_decision returns nothing under studio).
  • The CLI's decision-comparison logic (result.decision == expected) is inlined in the test command (cli.py:423-433), not extracted, so any Studio runner would have to duplicate it.

Net: the core authoring loop — edit rule, re-run saved scenarios, confirm no regressions — cannot be closed in the browser.

Task

  • Add POST /studio/api/scenarios to persist a named scenario (facts + expected_decision, reusing the CLI test-case shape).
  • Add a run endpoint (e.g. POST /studio/api/scenarios/run, optionally per-ruleset) that executes saved scenarios against the real engine and returns pass/fail per case.
  • Extract the decision-comparison logic out of cli.py test into a shared helper and call it from both the CLI and the Studio run endpoint (avoid divergence — this is the load-bearing part of the task).
  • Persist scenarios file-backed under the ruleset root (_ruleset_root() in studio_api.py); reuse the existing resolve_ruleset path-jail.
  • Surface a UI to list/run scenarios with pass/fail badges. Decide the UI surface first (see Acceptance criteria) — the live Studio is the React SPA under creem/, while panels.py + templates/ are a parallel HTMX surface; the original proposal assumed HTMX but the SPA already owns scenario authoring.

Where

  • src/fathom/studio/studio_api.py (read-only /scenarios at line 118; _ruleset_root() line 60; _resolve() line 66; _evaluate() line 220).
  • src/fathom/studio/scenarios.py (bundled Scenario dataclass + SCENARIOS).
  • src/fathom/cli.py (test command line 352; comparison logic lines 423-433 to extract).
  • UI: src/fathom/studio/creem/cs-bench.jsx (existing client-side saveScn) and/or src/fathom/studio/panels.py + src/fathom/studio/templates/ — pending the surface decision.
  • Tests: src/fathom/studio/tests/test_studio_api.py; CLI fixtures pattern in tests/test_cli.py.

Acceptance criteria

  • A design decision is recorded on whether the persistent scenario UI lives in the React SPA (creem/) or the HTMX panels (panels.py/templates/), since both surfaces exist and the SPA already does session-only scenario authoring.
  • Decision-comparison logic is extracted into a single shared helper and used by both fathom test and the new Studio run endpoint (no duplication).
  • A scenario saved via the new endpoint persists server-side (file-backed under the ruleset root) and survives a process restart.
  • Running a suite returns per-case pass/fail consistent with fathom test on the same YAML cases.
  • The chosen UI lists scenarios with run + pass/fail indication.
  • New tests cover save, persist-and-reload, and run-with-pass/fail; existing studio + CLI tests still pass (uv run pytest).
  • uv run ruff check src/ tests/ and uv run mypy src/ are clean.

Size: L

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low priority: doc hygiene, cosmetic, sweepcliCLI (validate, test, bench, info, repl)enhancementNew feature or requestneeds-designDesign not yet settled — please don't open a PR yetsize/L<1 week: cross-cutting feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions