feat(evals): mine FP/FN candidates with Daytona and Jev

[devin-ai-integration]

Summary

Add an FP-first mining loop to the private eval package so maintainers can review likely detector mistakes across a repository corpus:

nr mine → existing Daytona runner → pinned rule catalog + source
        → Jev via Vercel AI Gateway → candidate_fp/candidate_fn → issues.ndjson

--interval-minutes repeats scans; --input reuses a parity/eval artifact. Successful model responses are cached by evidence, detector, rubric, prompt, model and threshold; the issue log deduplicates across restarts. Standalone classify prepare / classify run commands allow independent replay.

Only complete v3 full-scan evidence at pinned revisions is accepted. Jev gets typed questions without the detector verdict; both answer confidence and context sufficiency must reach 0.9. Missing/oversized source becomes review. FN sampling is opt-in (--silent-files); these are review candidates, not measured precision/recall. The candidate limit includes cache hits, so expanding coverage requires a larger limit or changed corpus.

See packages/evals/README.md for usage and artifacts. AI SDK versions are pinned with the explicitly approved version-specific dependency-age exceptions; the private eval package now requires Node 22.13+.

Validation

• All 171 eval tests pass on Node 22 and Node 20, including a subprocess mining/restart test through the actual SDK with mocked HTTP, cache invalidation, malformed answers and pinned-source validation. Fixed the CI Node 20 test harness failure by replacing Promise.withResolvers with an equivalent Promise barrier; assertions are unchanged.
• Repository lint, typecheck, build, formatting and JSON-report smoke pass.
• React Doctor review: replaced repeated rule-array lookups with sets. Two async-await-in-loop warnings remain intentionally: source fetches are already scheduled concurrently, and result writes are serialized to preserve NDJSON record order.
• Full repository tests: one agent-hook test fails (exits quietly when no react-doctor runner is available). Reproduced on untouched base 260b448cd: the test's /usr/bin:/bin PATH still exposes npx, which downloads/runs React Doctor and emits output instead of staying silent. No existing tests were changed.
• Live Jev retry after Gateway credits were added: all four synthetic requests succeeded with validated typed answers (2,686 input tokens). Results: one likely_tp, one candidate_fn, one likely_tn, one review. The strict four-verdict calibration assertion failed: the expected FP was judged valid with probability 0.99, but context sufficiency was 0.88, below the unchanged 0.9 threshold. This is a conservative abstention, not an API failure; the rubric still needs calibration before scaling.
• Replayed those four candidates through classify run: four cache hits, zero errors, zero newly billed input tokens. A fresh Daytona-to-Jev run remains unverified; these synthetic cases are not corpus evidence.

Product brief

Job: Reduce manual review of diagnostics across large React repository corpora.

Reuse: Compose the existing Daytona evaluator and generated rule registry; keep the workflow in private eval tooling.

Signal: JSON summaries expose processed/cache/error/token counts and verdicts by rule. Use the candidate log for human-confirmed candidate yield; no repository/path telemetry is added.

Compatibility: FP-only default; no published CLI, report schema, score or Action change, so no changeset.

Kill metric: Rework or retire the rubric if fewer than 20% of reviewed candidates are confirmed after two corpus cycles with at least 50 reviews.

Link to Devin session: https://app.devin.ai/sessions/490c477d82c54b9786c13f1d5b79ab35
Open in Devin Desktop: https://app.devin.ai/desktop/session/490c477d82c54b9786c13f1d5b79ab35?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from Aiden

wait for react doctor, we have a parity runner, but could we use vercel ai gatewy and typesafe jev to take the codeblock and classify? and find FP/FNs that way at scale. build a thing dor this

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/eslint-plugin-react-doctor@1820

npm i https://pkg.pr.new/oxlint-plugin-react-doctor@1820

npm i https://pkg.pr.new/react-doctor@1820

commit: 7bfc514

[github-actions]

Interactive terminal E2E

Terminal Control verified the built CLI at 7bfc514 in a real PTY:

• selected a project interactively and observed Scanning... before the three-second Git delay completed
• waited for the clean result and exercised the compact report
• opened copy context and the GitHub Actions confirmation, then cancelled safely

Download the edited MP4 and PNG evidence