Scorer-path validation corpus: labeled short-form fixtures + FNR/exact-zero gates (§4.5)

[devswha]

Summary

Build a labeled corpus + statistical gates for the end-to-end scorer path (short-form / SNS register), as prescribed by the insane-review (GPT-5.6 Pro) §4.5. This is the deferred remainder after landing the scorer-path work on dev.

Already landed on dev

• P0 fix (fix(score): keep hard deterministic evidence floor on short (skipped) text): reconcileScoreOverall no longer discards the hard evidence floor (markup leakage feat: detect model-output leakage (citation markup, chatgpt utm params, explicit 'as an AI' self-id) #332 / structural) on skipped short text. A short AI-leaked snippet went 0 → 90 at LLM 0.
• Scorer benchmark leg (tests/quality/scorer-benchmark.mjs, node tests/quality/scorer-benchmark.mjs): runs scoreDeterministicSignals + reconcileScoreOverall (LLM mocked to 0) and gates on positive_zero_score_rate / false_positive_rate / skipped_evidence_discarded. Wired into the Node-18 CI smoke.
• P1 short-form em-dash signal (src/features/short-form.js, #13 short-form branch): register-gated (social/marketing) weak Low signal; single dash floors an eligible reply to ~1.7. Inert for the default profile.

The above is guarded by hand-written representative fixtures, not a labeled corpus.

What is still missing (this issue)

A real labeled short-form corpus + slice metrics, so we can state how well the scorer catches/avoids these tells, not just that a handful of examples behave:

• Labels per row: generator (human | gpt | claude | …), register (social/marketing/…), perceived_ai_polish (true/false), expected_short_form_tells (em_dash, arbitrary_triad).
• Slices: single em dash, dash density, Rule of Three (arbitrary vs genuine 3-step), context exclusions (quote/code/glossary), combined tell (dash + hype + CTA + triad), and counterfactual pairs (same sentence with the dash inserted vs replaced by a comma/period).
• Metrics as CI gates (beyond the current pass/fail fixtures): short_social_false_negative_rate, short_social_human_false_positive_rate, positive_zero_score_rate, paired_score_delta, short_form_em_dash_recall, per-slice ROC/PR-AUC, and analyzer.hot vs final scoreText().overall disagreement rate.

Why deferred

This requires real data collection + human labeling (genuine human-written and AI-written short social posts), not a code edit — fabricated rows would poison the metrics. It is a research/data task, not on the v6.4 payment/launch critical path.

Acceptance criteria

• A sanitized, hash-only or redistributable short-form corpus (no raw private text committed), mirroring the existing rebaseline manifest hygiene.
• Slice metrics computed and reported; at least positive_zero_score_rate and short_social_false_negative_rate promoted to gates.
• Counterfactual pairs demonstrate a measurable paired_score_delta for the em-dash tell.
• No regression to the published analyzer benchmark (accuracy/F1) or the human false-positive tolerance.

Refs

• Review artifact: insane-review §4 (scorer benchmark blind spot) / §4.5 (corpus).
• Related roadmap: docs/ROADMAP.md Phase 1 (benchmark credibility) / Phase 2 (corpus expansion).