feat: add global timeout to terminal bench

[elisam0]

What does this PR do?

Adds an optional global_agent_timeout config field to AnyTerminalAgentConfig. When set, it overrides the per-task agent_timeout_sec from the dataset for every task, instead of the current default of a per-task timeout falling back to tb_agent_timeout.

This is useful for harness-optimization evals where we want longer traces to observe model behavior — e.g. when running slower models, per-task dataset timeouts can cut off runs before enough signal is collected. Setting global_agent_timeout lets a single config value apply uniformly across all tasks without editing the dataset.

Default behavior (per-task timeout takes precedence) is unchanged when global_agent_timeout is left unset (None).

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format/copyright pass).
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[elisam0]

/claude review

[claude]

SHIP — LGTM, no reliability concerns.

Small, additive change adding an opt-in global_agent_timeout config knob to AnyTerminalAgentConfig.

• global_agent_timeout: Optional[int] = None lives on a Pydantic config model where siblings (tb_agent_timeout: int = 1800) already carry Python defaults — no TypedDict/YAML-defaults convention violation.
• Override guard uses is not None, so global_agent_timeout=0 is honored as an explicit override rather than silently falling through to the per-task path. Correct.
• Default None preserves the prior per-task-timeout behavior exactly; nothing changes for existing configs.
• No verifier, scoring, async, or public-API surface touched. Eval/verifier timeout path is untouched.
• Tests cover all three branches (per-task default, global-overrides-per-task, global-without-per-task) and assert real behavior via _setup_params.

Optional NOTE (author's call): the new field isn't documented in any exemplar YAML under configs/; the sibling timeouts aren't either, so this is consistent, not a regression.

[cmunley1]

This can be set higher than the container's own lifetime, and nothing checks that. The sandbox starts with ttl_s=cfg.tb_sandbox_ttl (default 7200) at L582. In the Docker provider, that makes PID 1 a literal sleep 7200 with --rm (provider.py:337-345), so the container is deleted at t=7200 regardless of the agent timeout.

The tests use global_agent_timeout=7200, exactly equal to the TTL. That leaves no room for startup, staging, or the 300s eval afterward, so the container can disappear before the run finishes. The config comment also points to 12000 as the TB2 max, which is well beyond the TTL.

anyswe_agent derives the TTL instead: swebench_agent_timeout + swebench_tests_timeout + 600 (anyswe_agent/app.py:335). Could we do the same here, or fail at config time if global_agent_timeout + tb_eval_timeout >= tb_sandbox_ttl?

[cmunley1]

When the container is killed by its TTL, we score the task as a real zero instead of masking it.

exec on a removed container returns error_type="sandbox" and does not raise. This line and L570 only check for "timeout", while sandbox_failed is only set in the except block, which never runs. As a result, mask_sample at L628 stays False, and a dead container looks like a genuine failed attempt.

anyswe_agent checks result.error_type in ("timeout", "sandbox") (app.py:411, 458). Maybe worth matching that behavior here

[cmunley1]

Could this use Field(default=None, gt=0)?

Right now, global_agent_timeout: 0 passes validation, and the check on L777 is is not None, so 0 goes straight through as the agent timeout and every task fails immediately with an empty trace. The old code used a truthiness check, which effectively ignored 0.