Issues

• Docker sandbox image tag depends only on the task directory name, so concurrent rollouts collide (No such image, wrong image)

  benchflow-ai/benchflow#1149 · Filter by author EYH0602 opened on Oct 2, 2026

• Reviewer stage is not retried after a Daytona PTY transport reset (close_code=1006): rollout ends without a verdict

  benchflow-ai/benchflow#1144 · Filter by author Benjamin-eecs opened on Sep 25, 2026

• bench eval score: Daytona PTY readline timeout stays at the 900 s default regardless of --reviewer-timeout-sec

  benchflow-ai/benchflow#1143 · Filter by author Benjamin-eecs opened on Sep 25, 2026

• Stale .scoring.lock after a reviewer death makes every resume crash in resume_pending_reviews (Event loop stopped before Future completed)

  benchflow-ai/benchflow#1142 · Filter by author Benjamin-eecs opened on Sep 25, 2026

• ACP idle watchdog kills an active session over one pending tool call whose terminal update was lost (per-call grace ignores session activity)

  benchflow-ai/benchflow#1141 · Filter by author Benjamin-eecs opened on Sep 25, 2026

• claude-agent-acp cannot use an Anthropic-compatible third-party endpoint directly: the mandatory LiteLLM bridge rewrites /v1/messages to chat/completions and loses prompt caching (Kimi K3 on Baseten, GLM on Z.ai)

  benchflow-ai/benchflow#1140 · Filter by author Benjamin-eecs opened on Sep 23, 2026

• Daytona PTY transport dies with websocket 1008 on a multi-MB ACP message (Claude Code Read of a PDF): rollout recorded as pipe_closed

  benchflow-ai/benchflow#1138 · Filter by author Benjamin-eecs opened on Sep 23, 2026

• claude-agent-acp pin 0.73.0 bundles Claude Code 2.1.257; claude-opus-5-5 requires 2.1.280 (0.81.0)

  benchflow-ai/benchflow#1137 · Filter by author Benjamin-eecs opened on Sep 23, 2026

• Verifier exec wedges after a long agent phase (no output for the whole timeout, twice), reported as the submission's timeout and the agent is re-run

  benchflow-ai/benchflow#1136 · Filter by author Benjamin-eecs opened on Sep 22, 2026

• Bedrock through the LiteLLM chat/completions bridge loses prompt-cache breakpoints (9% cache hit vs 99% direct, 12-32x cost)

  benchflow-ai/benchflow#1135 · Filter by author Benjamin-eecs opened on Sep 21, 2026

• Host hard deadline ignores the agent.timeout_sec override and reports the kill as INFRA_ERROR

  benchflow-ai/benchflow#1134 · Filter by author Benjamin-eecs opened on Sep 21, 2026

• [BUG] An agent can make a failing pytest task pass by planting a plugin in its workspace

  benchflow-ai/benchflow#1116 · Filter by author tulerfeng opened on Sep 9, 2026

• Mock-service tasks can fall back to operator-hosted account connectors

  benchflow-ai/benchflow#1107 · Filter by author kywch opened on Sep 6, 2026
  4

• Extract OpenClaw shim from BenchFlow core into benchflow-ai/agents

  benchflow-ai/benchflow#1090 · Filter by author kywch opened on Sep 2, 2026
  1

• Harden agent catalog resolution, provenance, and discovery

  benchflow-ai/benchflow#1089 · Filter by author kywch opened on Sep 2, 2026
  1

• OpenClaw ACP shim loses session events when child outlives completed response

  benchflow-ai/benchflow#1087 · Filter by author kywch opened on Sep 2, 2026

• bench eval continue is limited by replay wire protocol, not model openness — and timed-out runs lose their world before they can be salvaged

  benchflow-ai/benchflow#1083 · Filter by author JeremyJC67 opened on Aug 31, 2026

• [Tracking] Define three BenchGuard artifact-evidence contracts

  benchflow-ai/benchflow#1079 · Filter by author kywch opened on Aug 31, 2026

• Verifier hardening does not check post-kill quiescence before verifier startup

  benchflow-ai/benchflow#1078 · Filter by author kywch opened on Aug 31, 2026
  1

• ClawsBench open-source readiness: BenchFlow fixes for review

  benchflow-ai/benchflow#1077 · Filter by author kywch opened on Aug 31, 2026
  2

• Proposal: benchflow-openeval-adapter — export rollout results.jsonl as a portable EvalPort ResultSet

  benchflow-ai/benchflow#1072 · Filter by author adhabnr-ux opened on Aug 31, 2026
  4

• Zero-activity rollouts are recorded as graded 'timeout' (reward 0) with a fabricated elapsed time

  benchflow-ai/benchflow#1071 · Filter by author yuchenlwu opened on Aug 30, 2026

• ACP idle watchdog false-kills long-thinking models: in-flight LLM generation counts as no activity

  benchflow-ai/benchflow#1068 · Filter by author yuchenlwu opened on Aug 30, 2026
  1

• Docker.exec turns a hardening timeout into a bare ProcessLookupError, so the rollout is recorded as "verifier crashed: " with no message

  benchflow-ai/benchflow#1065 · Filter by author Jiankai-Sun opened on Aug 29, 2026
  1

• [BUG] Resume reuses scoreless verifier-errored results so infra-failed tasks never re-run

  benchflow-ai/benchflow#1059 · Filter by author Benjamin-eecs opened on Aug 29, 2026