Skip to content

CollectiveX: name Slurm allocations after the GHA runner (dash correlation) - #2762

Merged
Oseltamivir merged 3 commits into
mainfrom
collectivex-slurm-job-name
Aug 28, 2026
Merged

CollectiveX: name Slurm allocations after the GHA runner (dash correlation)#2762
Oseltamivir merged 3 commits into
mainfrom
collectivex-slurm-job-name

Conversation

@Oseltamivir

@Oseltamivir Oseltamivir commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

CollectiveX salloc calls carried no --job-name, so its allocations never correlated with GHA jobs on inferencex-dash (the cluster collector samples sacct JobName; the dashboard joins it against ci_jobs.runner_name — the --job-name="$RUNNER_NAME" convention every production runner launcher already follows).

One seam: collx_salloc_jobid now prepends --job-name="$(collx_slurm_job_name)" (caller-supplied --job-name would still win since salloc takes the last occurrence). collx_slurm_job_name uses RUNNER_NAME when it is a plain Slurm-safe token, else falls back to collectivex (hand-driven runs, unsafe names). Covers all three Slurm launchers (single-slurm, gb-nv, mi-amds); mi-tw is Docker and out of scope. Seam-tested in test_runtime.py.

Validation: sweep run on this branch — see checks.


Note

Low Risk
Scheduler labeling and input sanitization only; allocation behavior is unchanged aside from the job name passed to salloc.

Overview
Slurm allocations from CollectiveX now get a --job-name that matches the GitHub Actions runner, so inferencex-dash can join sacct JobName with ci_jobs.runner_name (same convention as production runner launchers).

Adds collx_slurm_job_name, which uses RUNNER_NAME when it passes a Slurm-safe token regex and otherwise falls back to collectivex. collx_salloc_jobid prepends --job-name="$(collx_slurm_job_name)" before other salloc args so an explicit caller --job-name still wins (last flag wins).

Reviewed by Cursor Bugbot for commit d27f05f. Bugbot is set up for automated code reviews on this repo. Configure here.

…d correlation

inferencex-dash's cluster collector samples allocation names (sacct JobName)
and the dashboard joins them against the GHA job's runner_name — the
convention every production runner launcher already follows with
--job-name="$RUNNER_NAME". CollectiveX's salloc calls carried no job name,
so its allocations were invisible to that correlation. Prepend the name at
the collx_salloc_jobid seam (a caller-supplied --job-name still wins; salloc
takes the last occurrence), falling back to 'collectivex' for hand-driven
runs or a runner name that is not a plain Slurm-safe token.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, straightforward and well-tested change.

What was reviewed: the new collx_slurm_job_name() regex validation and fallback logic, and collx_salloc_jobid()'s use of set -- to prepend --job-name (confirmed --no-shell still appends correctly afterward and that salloc's last-occurrence-wins means a caller-supplied --job-name still takes precedence). Checked no CODEOWNERS restriction applies to experimental/CollectiveX/. New tests cover pass-through, unset, and unsafe RUNNER_NAME cases.

Extended reasoning...

Overview

The PR adds a collx_slurm_job_name() helper to experimental/CollectiveX/runtime/common.sh that derives a Slurm --job-name from $RUNNER_NAME, validating it against an anchored, bounded regex (^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$) and falling back to the literal collectivex on empty/unsafe input. collx_salloc_jobid() is updated to prepend --job-name="$(collx_slurm_job_name)" via set --, relying on salloc's documented last-occurrence-wins flag semantics so a caller-supplied --job-name still takes precedence. Corresponding tests in test_runtime.py cover pass-through, unset fallback, and unsafe-input fallback, plus a literal source-text assertion.

Security risks

None identified. The helper rejects unsafe input outright rather than attempting to sanitize or shell-quote it, which avoids injection risk entirely. The value is only ever used as a printf-emitted string interpolated into salloc's argv via set -- (not eval'd or run through a shell), so there is no command-injection vector even for the rejected path, and the regex ensures only a safe token class reaches this point. The change does not touch privilege boundaries, cleanup/scancel logic, or any file I/O correctness (I verified the --no-shell flag remains correctly appended after the new prepended args, and that collx_cleanup_allocation/JOB_ID handling downstream is unaffected).

Level of scrutiny

This is a small, additive, well-isolated helper in a non-production-critical (experimental) path, with clear intent and matching test coverage for the three behaviorally relevant cases (valid, unset, unsafe). No CODEOWNERS entry restricts experimental/CollectiveX/. I verified the code by reading the full diff and the surrounding collx_salloc_jobid function body directly, rather than running the test suite (blocked by tool permissions), but the logic is simple enough to confirm correctness by inspection.

Other factors

The PR description's claims (dashboard correlation via sacct JobName, salloc's last-wins behavior) match the code. The one stylistic nit — a test asserting on the literal set -- line as a string match rather than purely behavior — is brittle but harmless and not something that warrants blocking or an inline comment on its own; it doesn't affect correctness of the shipped behavior. No CLAUDE.md/AGENTS.md conventions (perf-changelog, benchmark_lib.sh conventions, AMD workspace hygiene) are implicated by this non-perf-affecting, non-recipe change.

@Oseltamivir
Oseltamivir merged commit 918524f into main Aug 28, 2026
5 checks passed
@Oseltamivir
Oseltamivir deleted the collectivex-slurm-job-name branch August 28, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant