Retry PTY takeover attach when execd's eviction times out

[hemildesai]

Problem

On terminal-bench 2.1 runs at 1k-sandbox scale, verify-phase re-attaches intermittently fail with:

nemo_gym.sandbox.providers.base.SandboxPtyError: PTY session already has an attached client

even though the attach passes takeover=True.

Root cause

The tb2.1 flow passes one PTY session through three holders — resources server (create) → opencode agent → resources server (verify) — and every handoff relies on takeover-evicting the previous client. execd implements takeover as a cooperative eviction with a timeout (its binary carries takeover timed out for pty session). Eviction is only cheap while the previous client is alive: a socket idled through the whole agent phase is exactly the one the NLB/server-proxy path drops silently, so the next takeover waits out execd's eviction timeout and surfaces as a policy-violation close, which the client maps to "already has an attached client". The stale peer is torn down in the background, so a fresh attempt lands.

Changes

1. Retry the takeover attach (provider.attach_pty): up to three re-dials (2s/5s/10s) on exactly that signature, fresh HTTP client per attempt. Non-takeover rejections and other errors stay definitive.
2. Detach the baton instead of dangling it (tb2.1 seed_session): the creator detaches right after creating the session, so the agent's attach finds nothing to evict; the server-side session keeps running and replays on reattach. Takeover+retry becomes the recovery path for crashed holders, not the routine handoff.
3. Websocket heartbeats on every PTY dial (30s): intermediaries stop idle-reaping healthy sockets, and a dead socket surfaces as a failed ping and re-dials via the existing reattach path within ~a minute instead of dangling half-open until the next takeover times out against it.

For the tb2.1 dev branch (#2175) — same contract, two snippets

The agent-attach and verify-reattach code paths live on the dev branch, so two spots there complete the pattern:

• opencode agent server, when the rollout finishes: await pty_session.detach() — whoever is done with the terminal detaches; takeover is for crashes.
• verify, restoring a client: keep attach(session_id=..., takeover=True) (it now retries), or reattach the stored session when it is merely detached.

Testing

• Unit: tests/unit_tests/test_opensandbox_pty.py — 61 passed (new: takeover-timeout retry lands on re-dial with the failed client released; no retry without takeover; every dial passes the heartbeat).
• E2E against the cell-2 OpenSandbox cluster (real execd through the server proxy), full three-holder flow with this branch's code:
  • creator opened a session, ran a command, detached;
  • agent takeover-attached (nothing to evict), set shell state, detached;
  • verify takeover-attached and ran the check detached across multiple reattach polls (12s command, 4s poll);
  • shell state set by the agent ($AGENT_MARK) was visible in verify — one server-side session across all three handoffs; exit code 0; sandbox deleted cleanly.
  • Separately verified live takeover of an attached client (evictee sees close 4001 "taken over") and that a no-takeover attach against a held session returns exactly the retried signature.

Companion platform ask (separate): execd should force-close the stale socket instead of waiting for a half-open peer, and use a distinct close code for takeover-timeout vs genuinely-attached.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[bxyu-nvidia]

/ok to test 5067394