feat(ultra): 3K NVCF + disaggregated-sandbox pipeclean recipes and launch fixes

[hemildesai]

Overview

Enables the 3K-GPU pipeclean with disaggregated sandboxes + NVCF-hosted judges, stacked on the 6k-pipecleaning lineage (base branch: hemild/saumishr-6k-pipecleaning-base, a snapshot of @saumishr's branch tip).

What the NVCF + disagg arm changes: tool execution moves to prewarmed pods on an external sandbox service, and all three judges (GenRM, NL2Bash, safety) are served from NVCF — so there is no judge hetgroup, no local judge vLLM, and the run is a single 752-node group (256 train / 488 gen / 8 Gym). Measured against the on-allocation arm on the same day/cluster/scale, this removed ~13 minutes of hard init gate (judge bring-up before ray.sub) and reached rollout dispatch ~14 minutes after job start.

Recipes (feat(ultra))

• nvcf_judges.yaml — mergeable fragment routing the judges at NVCF. Deployment-specific function routes and the credential come from the environment (GENRM_NVCF_MODEL, NL2BASH_NVCF_MODEL, SAFETY_NVCF_MODEL, NVIDIA_API_KEY); nothing secret is embedded, and the launcher fails closed if they are missing.
• pipeclean_3k.yaml / pipeclean_3k_sc.yaml — 3K base + SC overlays (halved 6K cohort).
• disagg_sandboxes/launch_3k_nvcf_disagg_sc.sh + pipeclean_3k_nvcf_disagg_sc.yaml — the primary arm.
• disagg_sandboxes/launch_3k_disagg_sc.sh + pipeclean_3k_disagg_sc.yaml — the on-allocation backup arm.

Recipes were authored in @saumishr's RL-6k working tree; this PR publishes them with the NVCF routes externalized.

Launch fixes (fix(launch)) — each hit live at 752 nodes

1. Physical-path poisoning. Slurm records the submit host's physical cwd; on clusters where the login node resolves /lustre through a mount compute nodes cannot see, every path derived from SLURM_SUBMIT_DIR fails (pyxis: couldn't chdir …, all containers dead in ~2 min). Host paths now route through SLURM_WORKDIR (defaults to SLURM_SUBMIT_DIR), ultra_launch gains PROJECT_ROOT_OVERRIDE, and every sbatch component gets --chdir.
2. Service steps starving the ray head. A step that omits --gres/--mem inherits the node's full GPU/memory allocation; the external-vLLM load-balancer steps on the ray-head node blocked the ray-head srun in endless step creation temporarily disabled retries. LB/preflight steps are now pinned to gpu:0 + 16G and the ray-head step overlaps service steps.
3. CPUS_PER_WORKER vs CoreSpecCount. Auto-detection reads the node's physical CPUTot, which exceeds the allocation on clusters that reserve system cores — also an unsatisfiable step. The resolved value is clamped to SLURM_CPUS_ON_NODE.

Unit tests pin all three contracts (tests/unit/tools/test_external_gym_vllm.py).

Validation

• Both arms dry-run clean; the NVCF+disagg arm ran end-to-end through Ray assembly (~3 min for 752 nodes), full Gym spin-up, 768+768 prewarmed sandbox claims, 550B policy instantiation, and weight refit, reaching rollout dispatch before the allocation window closed.
• bash -n on all touched shell files; new unit tests pass by construction against the modified files.

Draft: the mooncake data-plane dispatch stall observed at scale is still under investigation (workaround: DP_BACKEND=simple), and the base branch should be retargeted to @saumishr's branch owner-side before merge.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.