feat: add promptlens compare command for baseline regression detection

[brainsparker]

What this does

Adds promptlens compare BASELINE CURRENT, a run-to-run diff with CI regression gates. PromptLens already has an absolute quality gate (--fail-under, shipped in #43), but nothing catches relative regressions: a prompt change that drops a model from 4.6 to 4.1 still sails past a 3.5 gate even though it made things worse. This closes that gap and checks off the roadmap item "Cross-run comparison and tracking".

Usage

# Run IDs (including "latest") resolved in --output-dir, or direct paths
promptlens compare main-baseline latest
promptlens compare baseline/results.json promptlens_results/latest

# CI gates: exit code 2 on violation
promptlens compare main-baseline latest --max-regression 0 --max-case-regression 1

# Markdown diff sized for a PR comment, plus machine-readable JSON
promptlens compare main-baseline latest --markdown diff.md --json diff.json

Behavior

• Pairs results by (test_case_id, model) and classifies each pair: regressed, improved, unchanged, added, removed, or unscored. Added and removed cases are surfaced so a shrinking golden set cannot hide a regression.
• Rich console summary: per-model table with average deltas, then every regressed case worst first.
• --max-regression gates on model average drops, --max-case-regression gates on single case drops. 0 means any drop fails. Exit codes match run: 0 pass, 2 gate failure, 1 error.
• Warns when the two runs used different golden sets.
• --model restricts the comparison to one model.
• README gains a Baseline regression comparison section with a GitHub Actions flow (baseline artifact from main, compare on the PR, sticky PR comment with the diff).

Why this feature today

Relative regression comparison is the center of gravity for eval tooling in CI right now, and it was the biggest missing piece of the CI story this repo started with the JUnit exporter and --fail-under:

• Promptfoo positions CI/CD eval around catching regressions before deployment, comparing outputs across changes (https://www.promptfoo.dev/docs/integrations/ci-cd/).
• Braintrust's core Evaluate loop is experiment-vs-baseline diffing and regression tracking; it is the headline of their product page (https://www.braintrust.dev/product/evaluate) and their promptfoo comparison (https://www.braintrust.dev/articles/braintrust-vs-promptfoo).
• Practitioner content in 2026 frames "prompt regression testing" as a CI gate pattern: run a fixed set, score, and fail the PR on drops relative to baseline (https://buildpulse.io/blog/llm-ci-prompt-regression-testing-evals).

PromptLens's differentiation is local-first and CI-native without a cloud backend. A baseline compare command keeps that promise: the baseline is just a results.json artifact, no server required.

Runner-up candidates considered today: you.md support for more AGENTS.md-reading tools (largely already shipped in the export command), and MCP structured tool output (outputSchema/structuredContent) for you-aware (good, but narrower immediate value).

Implementation notes

• promptlens/comparison.py: pure pydantic models (CaseComparison, ModelComparison, GateFailure, RunComparison) plus compare_runs() and render_markdown(). No new dependencies.
• promptlens/cli.py: new compare command; run reference resolution accepts a results.json path, a run directory, or a run ID under --output-dir (the existing latest symlink works as a run ID). Also fixes a rich markup bug where model names in brackets were swallowed as markup tags.
• tests/test_comparison.py: 25 tests covering pairing, all six statuses, model filtering, average deltas, golden set mismatch, gate logic for both flags, markdown rendering, and CLI behavior via CliRunner (exit codes 0, 1, 2; markdown and JSON outputs).

Test status

Full suite: 83 passed (58 existing, 25 new). Verified end to end with a manual CLI run: console table, regression listing, markdown diff, and exit code 2 on a failed --max-regression 0 gate.

🤖 Generated with Claude Code