feat: add promptlens compare for run-to-run regression detection

[brainsparker]

What this adds

promptlens compare <baseline.json> <current.json>: run-to-run comparison and regression detection built on the results.json every run already exports.

• Pairs cases by (test case ID, model) and computes per-case judge score deltas, latency drift, and cost drift
• Flags regressions (score drop of at least the threshold, default 0.5 on the 1-5 scale) and new errors (case passed in baseline, errored in current)
• Per-model summary table: baseline avg, current avg, delta, regressed / improved / new error / fixed error counts, total cost and latency drift
• Added or removed test cases are reported separately and never fail the gate, so growing the golden set does not break CI
• --fail-on-regression exits with code 3, distinct from the absolute --fail-under gate (2) and run errors (1), so pipelines can tell the three failure modes apart
• --markdown <path> writes a diff report ready to post as a PR comment; --json <path> emits the full comparison data for downstream tooling

Usage

promptlens compare baselines/main/results.json promptlens_results/latest/results.json \
  --fail-on-regression --markdown regression-report.md

Why this, why now

PromptLens recently shipped JUnit XML export and a --fail-under score gate (#42, #43). That is an absolute gate: it cannot answer "did this prompt change make things worse than main?" when both runs clear the bar. Relative baseline comparison is the defining pre-ship feature across the 2026 eval tooling market:

• Promptfoo's core CI story is failing builds on prompt regressions against prior runs, and it is the consensus pick for CLI-first regression testing (https://www.bearplex.com/compare/promptfoo-vs-braintrust-vs-langsmith, https://thepromptbench.com/evals-and-testing/eval-frameworks-compared/)
• Braintrust's experiment diffs and PR regression visibility are its headline workflow (https://pondero.ai/enterprise/guides/eval-harnesses-braintrust-vs-langsmith-vs-promptfoo-vs-arize-2026/)
• New entrants exist entirely for this job: regtrace is a "regression-first" eval framework whose whole product is baselines plus CI gating (https://github.com/decimozs/regtrace)

PromptLens is positioned as the lightweight local alternative to those platforms, and this was the biggest missing piece of that story. It composes with the existing exit-code scheme instead of adding a parallel one.

Runner-up candidates considered today: new export targets for you.md (existing coverage of Claude Code, Codex, Gemini, Windsurf, Cursor, and AGENTS.md is already strong) and MCP-Profiles harness extensions (smaller user reach).

Implementation notes

• New module promptlens/comparison.py: pydantic models (CaseComparison, ModelComparisonSummary, RunComparison), load_run_result() with clear errors for missing / invalid / wrong-schema files, compare_runs(), and render_markdown()
• New compare CLI command in promptlens/cli.py following the existing click + rich conventions
• Statuses handled per pair: regressed, improved, unchanged, unscored, new_error, fixed_error; fixed errors and unscored pairs never gate
• README: feature bullet, usage section, and a Baseline Comparison in CI section with a GitHub Actions example

Test status

• 28 new tests in tests/test_comparison.py and tests/test_cli_compare.py (pairing, thresholds, error transitions, coverage changes, markdown output, exit codes, invalid input)
• Full suite: 86 passed locally
• Manual end-to-end smoke test of the CLI against synthetic multi-model runs, including the gated exit code path

Built by Daily Feature Builder.