Skip to content

fix: size the DRACO bench harness's HTTP timeout off the config under test - #146

Open
shrdgn wants to merge 1 commit into
mainfrom
claude/draco-example-total-seconds
Open

fix: size the DRACO bench harness's HTTP timeout off the config under test#146
shrdgn wants to merge 1 commit into
mainfrom
claude/draco-example-total-seconds

Conversation

@shrdgn

@shrdgn shrdgn commented Aug 16, 2026

Copy link
Copy Markdown
Owner

What & why

bench/draco_eval.py calls bench.run._chat (via _chat_with_retry) to hit the openfusion
server under test, without ever passing an explicit timeout=, so every call — the solo
answer, the fusion answer, and every grading call — inherited _chat's shared default of
360s (bench/run.py:97).

examples/draco.yaml.example deliberately widens member_seconds/judge_seconds to
240/180 "because agentic tool loops + long answers are slow" (its own comment). Since
panel members run concurrently and judge synthesis follows sequentially
(panel.py::_call_member's retry loop shares one deadline, so a member's own worst case
is bounded by member_seconds, not a multiple of it), the server-side worst case for a
fusion request against this config is member_seconds + judge_seconds = 420s
comfortably past the bench client's hardcoded 360s HTTP timeout. A DRACO run that
legitimately takes, say, 380s to produce a good answer would get killed client-side by
httpx.ReadTimeout and either get spuriously retried (_chat_with_retry, up to 4 attempts)
or recorded as a task failure (errors.append(...) in run_eval) — silently corrupting the
benchmark's own results for exactly the slow-but-good responses the widened budgets exist to
allow.

This is also the concrete, narrow, self-contained piece of #140 flagged as fair game there:
draco.yaml.example's total_seconds: 350 was itself below its own member_seconds + judge_seconds (420), inconsistent with the field's documented "aggregate wall-clock cap"
semantics (full enforcement of that field is out of scope here, deliberately deferred in
#140 pending a product decision on streaming semantics).

Fix

  • bench/draco_eval.py::run_eval now computes `request_timeout = config.timeouts.member_seconds
    • config.timeouts.judge_seconds + 30.0from the config it just loaded, and passes it explicitly to every_chat_with_retry call (_gradenow takes and forwards atimeout`
      param too). The harness sizes itself off whatever config it's pointed at instead of assuming
      a fixed 360s across every config.
  • examples/draco.yaml.example's total_seconds is raised from 350 to 450 (above the
    420s worst case, with headroom), and the comment now states the real invariant instead of
    the stale "keep under the bench client's 360s timeout" claim (which the harness no longer
    needs, since it derives its timeout from these same values).

Not touched: member_seconds/judge_seconds themselves (240/180) — those are enforced
server-side today and presumably tuned for real agentic tool-loop durations, so shrinking them
is a behavior change with real product trade-offs, out of scope for this fix. Also not touched:
bench/run.py's own _chat default or examples/panel-strongsynth.yaml.example (which has a
narrower, less severe version of the same 360s-vs-member+judge gap — a possible follow-up).

How it was tested

  • ruff check . passes
  • pytest -q passes (491 passed, no live network)
  • New behavior has a test — tests/test_draco_eval.py::test_run_eval_sizes_request_timeout_from_configured_member_and_judge_seconds
    monkeypatches load_config/load_draco/_chat_with_retry and asserts every call receives
    timeout == member_seconds + judge_seconds + 30.
  • Docs updated — not needed; only bench-harness and example-config behavior/comments changed
  • No secrets, prompts, or response bodies added to logs or metrics
  • Bench number — not applicable; this doesn't change model quality/cost, only how long the
    harness itself is willing to wait for a response

Notes for reviewers

Found via an automated repo-review scheduled task, verified by hand: read bench/draco_eval.py
and confirmed it imports _chat from bench/run.py (default timeout=360.0, never overridden
in any _chat_with_retry call), then confirmed panel.py::_call_member's retry loop shares a
single member_seconds deadline (so 240 + 180 = 420 is the real server-side worst case, not an
underestimate), then cross-checked examples/draco.yaml.example's current committed values on
main.


Generated by Claude Code

… test

bench/draco_eval.py called bench.run._chat without an explicit timeout, so
it inherited that function's shared 360s default. examples/draco.yaml.example
deliberately widens member_seconds/judge_seconds to 240/180 (worst case
420s) for slow agentic tool loops, so a legitimately slow-but-successful
DRACO run could be killed client-side by the 360s timeout and misrecorded
as a task failure. run_eval now derives its request timeout from the
loaded config's own member_seconds + judge_seconds, and draco.yaml.example's
total_seconds (previously 350, below its own 420s worst case) is raised to
450 to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SfTJhjkqkDfTMzufuBcLtQ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants