feat: deterministic checks with judge-free runs and a --fail-on-checks CI gate

[brainsparker]

What this adds

Deterministic checks: local, zero-cost assertions declared per test case, evaluated against the model response with reproducible verdicts.

test_cases:
  - id: extract-001
    query: 'Extract name and age as JSON from: "Ada Lovelace, 36"'
    expected_behavior: Return valid JSON with name and age
    checks:
      - type: json_schema
        json_schema:
          type: object
          required: ["name", "age"]
      - type: contains
        value: "Ada Lovelace"
        case_sensitive: true

Six check types: contains, not_contains, regex, exact_match, json_valid, json_schema (a minimal built-in subset: type, required, properties, items, enum; no new dependency). JSON checks tolerate fenced ```json blocks.

How to use:

• Add checks: to any golden set test case. Outcomes land in the console summary, JSON and CSV exports, and the JUnit report (failed check maps to a <failure type="CheckFailure">).
• Gate CI with promptlens run config.yaml --fail-on-checks (exit code 2 on any failed check, consistent with --fail-under).
• Run with no judge API key at all via judge: { enabled: false }. A case whose checks all pass now counts as a JUnit pass even without a judge score.
• New example pair: examples/golden_sets/structured_output.yaml and examples/configs/checks_only.yaml.

Also fixed in passing: output.formats validation rejected "junit" even though the CLI wires a JUnit exporter; junit is now an accepted format.

Why this, why now (market rationale)

PromptLens already has a strong CI story (JUnit export, --fail-under, promptlens compare --fail-on-regression), but every verdict flowed through the LLM judge. That means CI needs a judge API key, every run costs money, and the gate is not reproducible: the same output can score 4 one run and 3 the next.

The field has converged on deterministic assertions as table stakes, with LLM-as-judge reserved for what string matching cannot see:

• A fresh practitioner guide (qtrl.ai, Aug 5, 2026) recommends exact match, regex, and JSON schema grading for structured output and warns against wiring an LLM judge into a merge-blocking gate without calibration, citing 2026 robustness research on judge gameability: https://qtrl.ai/blog/how-to-build-llm-eval-suites
• gatecheck ships exact, contains, regex, json_schema graders and runs CI with --deterministic-only because judge scores "aren't reproducible enough to gate on": https://pypi.org/project/gatecheck/
• Harness Evals categorizes ExactMatch, Contains, Regex, JsonDiff, SchemaValidation as its deterministic tier with --fail-under CI gating: https://github.com/harness/harness-evals
• assertllm markets "zero LLM calls, instant results" assertions as its core: https://docs.assertllm.dev/docs/assertions
• Fabric Harness docs: "Prefer deterministic scorers first": https://harness.fabric.pro/docs/building/evals

This was the largest capability gap between PromptLens and the tools it competes with, and it strengthens the existing differentiators (CI-native gates, local-first, no cloud backend) rather than adding a side feature.

Runner-ups considered today: a GitHub Copilot instructions export target for you.md, and further A2A interop work on youagent (already in flight via the Handler guide PR).

Implementation notes

• promptlens/models/checks.py: Check (validated at load time, so a bad regex or missing field fails promptlens validate, not mid-run) and CheckResult.
• promptlens/checks.py: pure evaluation engine, no I/O.
• EvaluationResult gains check_results plus checks_passed and failed_checks helpers; RunResult.get_check_stats() aggregates per model. Old stored results still parse (fields default empty).
• Runner runs checks before judging and skips judging entirely when judge.enabled is false, so checks-only runs never construct a judge provider.
• JUnit mapping precedence: response error > failed checks > judge below threshold > skipped (only when there is neither a judge score nor checks).

Test status

• 138 tests pass locally (python3 -m pytest tests/), including 53 new tests covering the engine (all six types, case sensitivity, any/all modes, fenced JSON, schema subset edge cases like bool vs integer), model validation, YAML loading, JUnit mapping precedence, CSV columns, the gate helper, and backward compatibility.
• Verified end to end against a local mock OpenAI-compatible endpoint with judge.enabled: false: 4 golden set cases ran with zero API keys, the deliberately failing regex case surfaced as CheckFailure in junit.xml and tripped --fail-on-checks.
• Environment note: on Python 3.9 I hit a pre-existing asyncio issue (the runner's semaphore is created outside the event loop, which 3.9 binds to a different loop). It exists on main and is unrelated to this change; happy to fix separately. README already recommends 3.10+.