Strict, local PR review that acts like a senior engineer.
It flags only real defects and stays silent on everything else.

Quick start · Benchmarks · Usage · GitHub Action · Runners · Methodology · 繁體中文

---

Needlefish reviews your diff before merge and reports only real defects — bugs, regressions, security, data loss, migration/upgrade risk, missing validation, duplicate behavior — never style.

• Prefer-zero findings. A strict senior reviewer's bar: if it isn't worth blocking merge, it's dropped. No style nits, no noise.
• Deterministic verdicts. The pass / needs_human / changes_requested verdict comes from fixed rules over the surviving findings, never from model prose.
• Isolated review targets. Reviews run in a throwaway clean clone, checked for tampering after every model call.
• Guarded evals. Every prompt or pipeline change is measured on an 87-scenario harness with active anti-cheat guards before it ships (see Benchmarks).

Small PRs get a review pass plus an adversarial critic; large PRs add map and deep passes before the same critic. Codex is the default runner — Claude Code, opencode, OpenAI-compatible HTTP, Grok, pi, and ACP agents are supported too.

_{A real finding from the deployed lane (GPT-5.6 Terra, high effort) on a planted eval fixture — raw report.}

Quick start

Locally — from inside any git repo you want reviewed. Requires Node 20+ and one authed runner CLI (codex, claude, or opencode) on PATH:

npx needlefish

On every PR — add .github/workflows/needlefish.yml to the target repo:

name: needlefish
on:
  pull_request:
    types: [opened, synchronize, reopened]
permissions:
  contents: read
  pull-requests: write
  checks: write
jobs:
  review:
    # Fork PRs don't receive secrets; skip them instead of failing at model auth.
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # full history: needlefish needs the merge base
      - uses: frankekn/needlefish@v0
        env:
          CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}

Set one secret — CODEX_AUTH_JSON (the contents of a logged-in Codex CLI's ~/.codex/auth.json) or CODEX_API_KEY — and open a PR. Findings arrive as inline review comments anchored to the diff; pushes update the same review in place (fresh / still-open / resolved) instead of stacking new ones.

Cost: 2 model calls per review on small PRs (gpt-5.6-terra at high effort by default), 1 map + N deep calls + 1 critic on large ones. Docs-only PRs and unchanged heads skip the model entirely.

Benchmarks

Which model, agent harness, provider route, and effort catches real PR defects without blocking clean changes? The tables below are generated from the same guarded report JSON as the benchmark page (eval/gen-readme.ts); the curated chronology and confirmation re-runs live in eval/RESULTS.md. The benchmark page generates its leaderboard from the same guarded reports and is never hand-edited.

Reading the columns: Balanced is the primary score — the arithmetic mean of anchored recall and usable specificity. Tier-1 is recall on must-find defects and a hard qualification gate. FP is clean PRs blocked. Noise/review is extra findings per positive review (gate at 0.12). Rows compare only when prompt, fixture-set, and scorer hashes and the anti-cheat version all match; provider failures and unavailable subscription models are operational outcomes, not zero model scores.

Updated 2026-09-10 — measured 2026-09-06; all 11 published lanes at 87 scenarios × 3 draws, sealed holdouts included, Class R gate, anti-cheat v2; commit a5a0c68, prompt e62d0889fc704541, fixture set e9923bbc7753a04a, scorer 8bbc6152d8b45a43; every report has cheatDetectedCount: 0.

Ranked lanes. Deployed lane: DeepSeek V4.1 Flash (deepseek-v4.1-flash-expires-on-0910 at high effort) (selected 2026-09-10) — what the hosted action and reusable workflow run by default. Bold marks the deployed lane and the best value in each column.

Rank	Lane	Harness	Effort	Balanced	95% CI	Tier-1	FP	Noise/review	Mean
1	DeepSeek V4.1 Flash (deployed)	Pi 0.85.1	high	97.81%	95.6%–100.0%	100%	0.00%	0.027	31s
1	Qwen 3.8 Flash	Pi 0.85.1	max	95.63%	92.7%–98.6%	95%	0.00%	0.055	118s
1	Grok 4.6	Grok CLI 1.0.13	xhigh	95.48%	92.2%–98.8%	100%	1.39%	0.011	230s
4	GLM-5.3-Flash	Pi 0.85.1	max	94.93%	91.6%–98.3%	100%	1.39%	0.038	208s
4	GLM-5.3-Flash	Pi 0.85.1	max	94.81%	91.5%–98.1%	95%	0.00%	0.022	164s
6	DeepSeek V4 Flash Vision Exp	Pi 0.85.1	max	91.66%	87.7%–95.6%	95%	1.39%	0.022	116s
6	GPT-5.6 Terra	Pi 0.85.1	xhigh	91.61%	86.7%–96.5%	95%	6.94%	0.077	76s
6	GPT-5.6 Terra	Codex CLI 0.153.4	high	89.95%	83.8%–96.1%	100%	9.72%	0.077	63s
6	GPT-5.6 Sol	Codex CLI 0.153.4	medium	88.41%	81.1%–95.7%	100%	13.89%	0.077	75s

Unranked lanes, same run — the interim gate requires at least 20/21 Tier-1 successes, at least 2/3 on every fixture, and positive noise no higher than 0.12; a later x3 confirmation is recorded but does not restore the rank.

Lane	Balanced	Gate missed in the full report
GPT-5.6 Terra xhigh	90.39%	Tier-1 90.48%: real-pr1-self-review-tool-checkout 2/3, t1-inverted-guard 2/3; noise 0.1202 > 0.12
GPT-5.6 Luna max	88.43%	Tier-1 76.19%: t1-inverted-guard 0/3, real-pr1-codex-no-sandbox-flag 2/3, real-pr1-self-review-tool-checkout 2/3; noise 0.1311 > 0.12

Reading: Grok 4.6 leads on accuracy and noise but is 3.7× slower per review and needs the Grok CLI authenticated on the runner, so it stays a candidate. Terra high and Sol are statistically unresolved against each other; Terra high is faster and cleaner on clean fixtures.

Deployed-lane change, Terra xhigh → Terra high — same model, subscription, harness, and run. Bold marks the better value in each row.

	xhigh (before)	high (now)	Δ
Balanced	90.39%	89.95%	−0.4 pt
Tier-1 recall	90.48%	100%	+9.5 pt
Anchored recall	86.34%	89.62%	+3.3 pt
Tier-3 recall	72.22%	77.78%	+5.6 pt
Usable specificity	94.44%	90.28%	−4.2 pt
False positives (of 72 clean draws)	4.17% (3)	9.72% (7)	+4 draws
Positive noise / review	0.1202	0.0765	−0.0437
Invalid output	0.38%	0%	−1 draw
Mean review time	80s	63s	−21%

Reading: the switch buys Tier-1 completeness, recall, and speed at the cost of four more blocked clean draws. Sol medium was the alternative at rank 2; it has higher recall but nearly double the false-positive rate of Terra high.

The full methodology, per-fixture matrices, and chronological experiment record live in eval/RESULTS.md and RESULTS_HISTORY.md; raw reports under eval/results/.

Usage

Local mode is read-only: Markdown to stdout, no GitHub writes.

Committed work — run from inside the target repo, or point --repo at it from anywhere. The default range is merge-base…HEAD (see base detection):

needlefish --repo /path/to/some-repo
needlefish --repo /path/to/some-repo --focus security
needlefish --repo /path/to/some-repo --deep
needlefish --repo /path/to/some-repo --base develop
needlefish --repo /path/to/some-repo --branch  # force merge-base..HEAD review

Uncommitted work — if the working tree is dirty, or the repo has no commits yet, needlefish reviews your uncommitted changes, untracked files included. Not a git repo yet? Run git init first.

needlefish --repo /path/to/some-repo --uncommitted  # force working-tree review

Pull requests:

needlefish --repo /path/to/some-repo --pr 123  # attach PR metadata to the local diff
needlefish pr 123 --repo /path/to/some-repo    # review the PR ref itself

Runner and model selection:

needlefish --repo /path/to/some-repo --runner claude
needlefish --repo /path/to/some-repo --runner opencode --model zai-coding-plan/glm-5.2
NEEDLEFISH_ACP_BIN=/path/to/acp-agent needlefish --repo /path/to/some-repo --runner acp

Output is Markdown on stdout, with the same review cached as JSON at ~/.cache/needlefish/<repo>/last-review.json. Pass --json to print the ReviewResult JSON to stdout instead:

needlefish --repo . --json | jq .verdict

Machine interface

needlefish --repo <path> --json and needlefish pr <number> --json print a versioned ReviewResult JSON object to stdout — the same object the local cache stores. Within a schemaVersion, fields are only added, never changed or removed; breaking shape changes require a new schemaVersion and a changelog entry.

Field	Shape
schemaVersion	Literal 1.
verdict	pass, needs_human, or changes_requested.
reviewTarget	Optional review target string.
findings[]	Finding objects with severity, title, category, file, lineStart, lineEnd, confidence, whyItBreaks, suggestedFix, and validation.
findings[].consumerFile	Optional downstream file affected by the finding.
findings[].consumerLine	Optional downstream line affected by the finding.
residualRisks[]	Residual-risk objects with text and blocks.
checked[]	Evidence strings describing what the review examined.
stats	Optional per-runner-call timing and attempt stats.
totalDurationMs	Optional total review duration in milliseconds.

Base detection

--base → origin/HEAD → main. Pass --base <ref> to override.

Verdicts

The verdict is derived deterministically — model prose never decides pass/fail:

• any P0 / P1 / P2 finding → changes_requested
• otherwise a blocking residual risk → needs_human
• otherwise → pass

P3-only findings are reported but do not block (the check stays green).

GitHub Action

Two ways to run on every PR: the hosted composite action (zero setup, cold-starts each run) or the self-hosted reusable workflow (low latency, on a machine you control). Both post the same result: a non-sticky COMMENT review with the full rendered review body, plus the authoritative Needlefish check-run as the merge gate.

verdict	review event	check
pass	COMMENT	success
changes_requested	COMMENT	failure
needs_human	COMMENT	neutral
run failed	(none)	failure

All verdict reviews are COMMENT, never approval or blocking-review events: the GITHUB_TOKEN bot cannot formally approve PRs, and a sticky blocking review can outlive a fixed head. The check-run is the merge gate — a failed review never passes a PR because the check goes failure. When a finding includes a validated exact replacement, its inline comment carries a native GitHub suggestion block; failed validation falls back to a plain comment.

Hosted (any repo)

The quick-start workflow above is the whole setup — this repo doubles as a composite action on GitHub-hosted ubuntu-latest.

Runner auth — repo secrets, passed via env on the action step:

runner	secret(s)
codex	CODEX_AUTH_JSON (contents of a logged-in ~/.codex/auth.json) or CODEX_API_KEY
claude	ANTHROPIC_API_KEY
opencode	provider key for the chosen model (e.g. OPENAI_API_KEY)
pi	PI_AUTH_JSON (contents of a logged-in ~/.pi/agent/auth.json)

The hosted install step only accepts codex, claude, opencode, or pi. grok and acp are CLI runners and openai is HTTP — the hosted action installs none of them, and passing runner: grok (or openai / acp) fails that install step; use the self-hosted workflow below for Grok 4.5. Claude's auth vars (ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN) and opencode's OPENAI_API_KEY are allowlisted through to the runner subprocess; other providers' keys need NEEDLEFISH_RUNNER_ENV_PASSTHROUGH=VAR (see subprocess environment).

Inputs (all optional): pr_number (defaults to the event PR), runner (default codex), model, timeout_ms, codex_reasoning_effort, runner_version, repo_path (defaults to the workspace checkout), github_token (defaults to the workflow token).

Runner versions: when runner_version is omitted, the action installs the per-runner pin from action.yml (currently Codex 0.153.4, Claude 2.1.239, OpenCode 1.18.21, pi 0.70.6). A single default cannot be correct for four packages, so the pin is chosen from the selected runner; pass an explicit version — or latest — only when you intentionally want something else.

Cost and behavior:

• Small PRs: 2 model calls (review + critic) at the workflow default, gpt-5.6-terra at high effort. Large PRs: 1 map call + N deep calls (concurrency 3 by default) + 1 critic.
• Docs-only PRs and same-head re-runs cost 0 model calls (force a re-review with --recheck).
• The hosted path cold-starts on every run (pnpm install + runner CLI install, roughly a minute). The self-hosted path below stays the low-latency option.
• Fork PRs don't receive secrets, so the if: gate in the quick start skips them. Avoid pull_request_target — it would hand secrets to workflows triggered by fork code.

Comment commands: the composite action does not add PR comment commands to the consumer repo. This repository's .github/workflows/commands.yml listens for maintainer @needlefish recheck and @needlefish explain <finding> comments (OWNER / MEMBER / COLLABORATOR only): recheck dispatches this repo's review.yml, and explain runs needlefish explain on a self-hosted runner that already has ~/.local/bin/needlefish. Copying that file into another repo only works after you retarget those two jobs.

Self-hosted runner

This installation uses an operator-managed Needlefish bundle. Review workflows never fetch upstream source, resolve an upstream main SHA, or deploy a release. Each consumer vendors a workflow_call-only copy of the reviewed workflow at .github/workflows/needlefish-review-local.yml; its existing caller owns PR and manual triggers, so one event cannot start two reviews.

jobs:
  review:
    uses: ./.github/workflows/needlefish-review-local.yml
    with:
      pr_number: ${{ inputs.pr_number || github.event.pull_request.number }}
      runner: codex
      model: gpt-5.6-terra
      codex_reasoning_effort: xhigh
    secrets: inherit

The caller must retain workflow_dispatch.inputs.pr_number, actions: write, and a run name ending in PR #<number> for bounded reconciliation. Reconcile runs on GitHub-hosted Ubuntu independently of the review runner.

Provision the tested self-managed bundle as the runner service account under ~/.local/share/needlefish-self/releases/<self_version>. Keep release.json, self-managed.patch, and the frozen pnpm-lock.yaml with the bundle; point the operator-owned needlefish-self/current link at that version. Each review resolves that link once, validates its metadata and patch/lockfile digests, and runs only the selected immutable binary. A missing or invalid installation fails closed. The manual needlefish-deploy workflow only checks the installed version; source pushes and upstream releases cannot replace it.

Keep Codex CLI 0.153.4 available as the same service account. The review lane is gpt-5.6-terra / xhigh / fast. The bundle includes the proxy tier forwarding fix: fast is passed to Codex even with a custom provider. Provider acceptance and the service tier actually delivered still require provider evidence.

npm install --global --prefix "$HOME/.local" @openai/codex@0.153.4
export CODEX_BIN="$HOME/.local/bin/codex"
test "$("$CODEX_BIN" --version)" = "codex-cli 0.153.4"

Preserve each caller's existing authentication route. Proxy callers pass codex_proxy_base_url, codex_proxy_required: true, and codex_proxy_api_key; this repository's direct trigger may use the CODEX_PROXY_BASE_URL repository variable. Required proxy mode fails on missing credentials rather than falling back to OAuth. Credentials remain in the child environment. Fork/closed/stale-PR checks, checkout credential isolation, and hosted finalization remain enforced.

Runners

--runner / NEEDLEFISH_RUNNER accepts codex, claude, opencode, openai, grok, pi, or acp; src/shared/codex.ts invokes the selected runner. Common options:

option	env	default
runner	NEEDLEFISH_RUNNER	auto-detects codex, then claude, then opencode
model	NEEDLEFISH_MODEL	runner default
Codex reasoning effort	CODEX_REASONING_EFFORT	medium (composite action and reusable workflow: high for gpt-5.6-terra)
timeout	NEEDLEFISH_TIMEOUT_MS	600000
opencode idle timeout	OPENCODE_IDLE_TIMEOUT_MS	the smaller of the per-call timeout and 600000

The opencode idle deadline resets whenever the CLI emits stdout or stderr. If a provider stream stops producing output, Needlefish terminates that attempt and uses the normal runner retry instead of waiting for an extended per-call timeout.

When neither --runner nor NEEDLEFISH_RUNNER is set and none of codex, claude, or opencode can be found, Needlefish exits with install commands for those three CLIs instead of a stack trace. Auto-detect does not look for grok, pi, openai, or acp.

Per-runner env vars. For CLI runners, binary / model / listed auth vars are in that runner's subprocess allowlist. The openai runner is HTTP and reads its env in-process (its subprocess allowlist is empty). Defaults in parentheses are the executable names used when the *_BIN var is unset:

runner	binary	model / other
codex	CODEX_BIN (codex)	CODEX_MODEL, CODEX_TIMEOUT_MS, CODEX_RETRY_MS, CODEX_REASONING_EFFORT; proxy CODEX_PROXY_BASE_URL, CODEX_PROXY_API_KEY, NEEDLEFISH_CODEX_PROXY_REQUIRED=1
claude	CLAUDE_BIN (claude)	CLAUDE_MODEL; auth ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN
opencode	OPENCODE_BIN (opencode)	OPENCODE_MODEL; auth OPENAI_API_KEY
grok	GROK_BIN (grok)	GROK_MODEL
pi	PI_BIN (pi)	PI_MODEL, PI_PROVIDER (default openai-codex), PI_AUTH_MODE (oauth or proxy; defaults to OAuth for openai-codex, proxy for an explicit provider)
acp	NEEDLEFISH_ACP_BIN (required)	—
openai	none (HTTP, not a CLI)	OPENAI_API_KEY (required), --model / OPENAI_MODEL (required), OPENAI_BASE_URL (default https://api.openai.com/v1)

How each runner is launched

• Codex: --ignore-user-config --ignore-rules --dangerously-bypass-approvals-and-sandbox, so its inspection commands are not blocked by execpolicy rules, approval prompts, or the host sandbox. Reasoning effort defaults to medium; set CODEX_REASONING_EFFORT=high to restore the old default, or xhigh for the highest-effort mode.
• Claude Code: --dangerously-skip-permissions, --safe-mode, and --no-session-persistence.
• Grok: --always-approve --permission-mode bypassPermissions --no-plan --sandbox off.
• opencode: --auto in headless mode, with an inline permission: "allow" override for its global and build-agent permissions.
• pi: --no-session --mode text --provider openai-codex --thinking <level> and its default full toolset.
• ACP: a JSON-RPC 2.0 Agent Client Protocol process over stdio from NEEDLEFISH_ACP_BIN. On timeout Needlefish sends session/cancel, then applies the same process-group kill path as the CLI runners.

Every CLI runner executes inside a throwaway clean clone at the review head commit, with GitHub tokens stripped and the expected HEAD fixed. After each successful model call, Needlefish re-checks the clone with git status --porcelain --untracked-files=all --ignored=matching, verifies HEAD did not move, and rejects any worktree mutation. The clone carries no remote: its origin (which would point at the original repository on the same filesystem) is removed before the runner starts, so an ordinary git push from inside the sandbox cannot create, force-update, or delete branches in the original. This closes the ready-made push route only; it is not an OS-level boundary, and a runner that learns the original path can still write there directly. Closed PRs are skipped before diffing or model invocation.

Subprocess environment

Runner CLIs (codex, claude, opencode, grok, pi, acp) are spawned with an allowlisted environment, not the full parent process.env — only locale/proxy/path basics plus each runner's own _BIN/_MODEL-style variables are passed through. To pass an additional variable to the runner subprocess, set NEEDLEFISH_RUNNER_ENV_PASSTHROUGH=VAR1,VAR2 (comma-separated names). On GitHub Actions, the non-secret RUNNER_TRACKING_ID job marker is retained so the self-hosted runner can terminate detached model processes when a job is cancelled.

ACP env authentication additionally requires an explicit credential declaration: set NEEDLEFISH_ACP_AUTH_ENV_VARS to the credential names and include those same names in NEEDLEFISH_RUNNER_ENV_PASSTHROUGH — for example NEEDLEFISH_ACP_AUTH_ENV_VARS=MY_AGENT_TOKEN with NEEDLEFISH_RUNNER_ENV_PASSTHROUGH=MY_AGENT_TOKEN. Arbitrary passthrough configuration does not prove authentication. Alternatively, set NEEDLEFISH_ACP_AUTH_FILES to comma-separated HOME-relative credential files; Needlefish copies only those files into the disposable runner HOME.

Development

Requires Node 20+, Corepack (recommended) or the pinned pnpm from packageManager, one supported model CLI authed locally (Codex, Claude Code, or opencode), and the GitHub CLI (gh) for --pr, pr, and GitHub Action mode.

git clone https://github.com/frankekn/needlefish
cd needlefish
PNPM_VERSION=$(node -p "require('./package.json').packageManager")
corepack enable
corepack prepare "$PNPM_VERSION" --activate
pnpm install --frozen-lockfile

If Corepack is unavailable, install the pinned package manager directly:

PNPM_VERSION=$(node -p "require('./package.json').packageManager")
npm exec --yes --package "$PNPM_VERSION" -- pnpm install --frozen-lockfile

Dev shim on PATH (optional): the repo keeps a bin/needlefish development shim. Symlink it onto a PATH directory so you can invoke needlefish from any cwd:

ln -sf "$PWD/bin/needlefish" ~/.local/bin/needlefish   # or any PATH dir
needlefish --version

The shim resolves symlinks and runs the repo-local tsx against src/cli.ts, so it survives the repo being linked from elsewhere and works in non-interactive shells (unlike a shell alias). Without this step, invoke via the full path: /path/to/needlefish/node_modules/.bin/tsx /path/to/needlefish/src/cli.ts (cwd is the target).

Status

v0.4.3. Read-only. Shipped: inline review comments, sticky re-review (fresh/open/resolved across pushes), docs-only fast path (no model calls), same-head dedupe, hosted-runner repo inspection (best-effort AppArmor sysctl). --fix stays unimplemented by design. Maintainer @needlefish recheck / @needlefish explain comments exist in this repository's .github/workflows/commands.yml; the published composite action does not install that workflow.