feat: multi-sample judging with stability metadata and noise-aware quality gate

[brainsparker]

What this does

LLM-as-judge scores are noisy. The same judge model, prompt, and response can score 4 on one run and 3 on the next, even at temperature 0.3. On a single sample, that noise is indistinguishable from a real quality change, so --fail-under gates flap and teams stop trusting red builds.

This PR adds multi-sample judging and a noise-aware quality gate:

• judge.samples config field and --judge-samples N CLI override (1 to 10, default 1): run the judge N times per response, concurrently.
• Aggregation: the reported score is the median of the samples (robust to a single outlier). Every JudgeScore now carries sample_scores, score_mean, and score_stdev, which flow into the JSON export automatically.
• Resilience: failed judge samples are logged and dropped as long as one succeeds, so a transient judge API error no longer sinks the case.
• Run summary: per-model average judge stdev plus a count of unstable cases (stdev of 1.0 or more, meaning samples disagreed by a full point).
• --noise-aware-gate: lowers each model's --fail-under threshold by its average per-case judge stdev. A model fails only when its score drops below the gate by more than the observed judge noise. The gate output prints the effective threshold and noise margin for each failing model, so the decision is auditable in the CI log.

Usage

promptlens run config.yaml --judge-samples 3 --fail-under 3.5 --noise-aware-gate

or in config:

judge:
  samples: 3

Why now (market rationale)

Judge flakiness is the most complained-about failure mode in the eval category right now, and nobody serves it in the lightweight OSS tier:

• A DeepEval community thread (July 16, 2026) describes the canonical failure: same 50 cases, same config, temperature 0, and Groundedness swings from 0.85 to 0.72 between sequential runs. https://communities.stackinsight.net/community/llm-eval-frameworks/why-is-deepeval-producing-inconsistent-scores-on-same-data/
• A July 8, 2026 write-up documents a faithfulness CI gate scoring 0.79 one day and 0.82 the next with zero code changes; the author's fix is exactly this pattern (k samples, gate on mean adjusted by stdev). https://dev.to/ethanwritesai/your-llm-as-judge-disagrees-with-itself-between-runs-1e3e
• LangSmith launched Tuned Evaluators (August 18, 2026) marketing reduced judge variance, behind a Plus/Enterprise plan. https://www.langchain.com/blog/introducing-langsmith-tuned-evaluators-starting-with-perceived-error
• A standalone judge-calibration library (May 2026) exists because "a single judge gives you a number with no error bar," but it is not integrated into any eval runner. https://github.com/watchtree-19/llm-judge-calibration

DeepEval, promptfoo, and RAGAS all issue a single judge call per case by default. PromptLens shipping variance-aware judging for free, wired directly into its existing CI gate, is a clean differentiator that extends the CI-native story from the recent --fail-under and compare --fail-on-regression work.

Runner-up candidates considered today: an MCP 2026-07-28 conformance checker command (timely, spec is under a month old, but a bigger surface to get right), and delta-gating against saved baselines (real need, but partially covered by the existing promptlens compare --fail-on-regression).

Implementation notes

• New module promptlens/judges/stability.py holds aggregation (aggregate_judge_scores) and orchestration (sample_judge_scores). It works against BaseJudge, so any future judge implementation gets multi-sampling for free. LLMJudge is untouched.
• Explanation and criteria scores come from the representative sample (the one closest to the median), with a stability note appended when samples disagree.
• Backward compatible: samples defaults to 1, new JudgeScore fields have defaults, _check_fail_under keeps its existing shape (existing junit gate tests pass unchanged).
• Uses stdlib statistics only, no new dependencies.
• Cost note: each extra sample adds one judge API call per test case per model; documented in the README.

Test status

• 27 new tests in tests/test_judge_stability.py covering aggregation math (median, outlier robustness, rounding, clamping), sampling orchestration (partial and total failure paths, call counts via a stub judge), config validation bounds, get_average_judge_stdev, and the noise-aware gate.
• Full suite: 110 passed (83 existing plus 27 new), verified locally on Python 3.9.
• CLI smoke-tested: --dry-run with judge.samples: 3, --judge-samples 5 override, and --noise-aware-gate without --fail-under exits 1 with a clear message.

PR opened by Daily Feature Builder. Run log: Hyperagent thread cmta3rei4004x08adr92set04.