Skip to content

feat: run apply in current Slurm allocation - #360

Draft
anish-shanbhag wants to merge 3 commits into
NVIDIA:mainfrom
anish-shanbhag:codex/current-allocation
Draft

anish-shanbhag wants to merge 3 commits into
NVIDIA:mainfrom
anish-shanbhag:codex/current-allocation

Conversation

@anish-shanbhag

@anish-shanbhag anish-shanbhag commented Aug 28, 2026

Copy link
Copy Markdown

Problem this solves

Teams sometimes already hold a Slurm allocation through salloc, a parent batch job, or an external scheduler, but srtctl apply always submits a new sbatch job. That forces nested scheduling and prevents the normal srun-based orchestration path from using resources the caller already owns. This change lets a single recipe reuse the inherited allocation while preserving the standard service, benchmark, and cleanup lifecycle.

Summary

  • add srtctl apply --current-allocation for running a single recipe under the inherited SLURM_JOB_ID
  • skip sbatch, stage the runtime config under the existing job output directory, and invoke the normal do_sweep orchestrator in the foreground
  • preserve preflight, output-directory, setup-script, environment, Dynamo wheel-prefetch, and --serve-only behavior
  • reject multi-job and alternate execution modes that do not have clear current-allocation semantics
  • document the flag and add focused CLI coverage

Scope

This is intentionally a thin first implementation. It supports one regular recipe at a time. Directories, sweeps, override configs, --bash, --mock, --json, and --tags remain unsupported in this mode. Allocation fields such as account, partition, and time limit are not reapplied because the allocation already exists.

Testing

  • GitHub Actions: copyright, lint, typecheck, unit tests, mock/server tests, and recipe validation all pass
  • pytest tests/test_submit_cli.py tests/test_apply_json.py tests/test_serve_only.py -v — 24 passed
  • focused coverage includes the full run_in_current_allocation helper plus every intentionally rejected execution/input mode
  • ruff check src/srtctl/cli/submit.py tests/test_submit_cli.py
  • ruff format --check src/srtctl/cli/submit.py tests/test_submit_cli.py
  • real Slurm smoke test: ran a one-GPU aggregate vLLM server for Qwen/Qwen3-0.6B inside an existing allocation, passed model-aware health, completed SA-Bench, and exited 0
  • Slurm accounting for the smoke test contained only the original allocation and its srun steps; no nested job was submitted
  • the smoke test produced the staged config, recipe lockfile, benchmark result and rollups, runtime fingerprint, resource snapshot, and dashboard artifacts

The branch also corrects two stale test mock targets introduced by the current main refactor. The same eval-only test failed on unmodified main because it made a real health request instead of using its configured mock.

Signed-off-by: Anish Shanbhag <ashanbhag@nvidia.com>
Signed-off-by: Anish Shanbhag <ashanbhag@nvidia.com>
@codecov-commenter

codecov-commenter commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@00660c9). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #360   +/-   ##
=======================================
  Coverage        ?   72.41%           
=======================================
  Files           ?       98           
  Lines           ?    13593           
  Branches        ?        0           
=======================================
  Hits            ?     9844           
  Misses          ?     3749           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Anish Shanbhag <ashanbhag@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants