[balajikumaran-c-s]

Summary

V048, V050, and V052 (_check_slurm_role_ldap_jobs) submitted Slurm LDAP jobs with sbatch --wait --nodelist=<compute>. When a target compute node was down, Slurm kept the job PENDING (ReqNodeNotAvail) and sbatch --wait blocked indefinitely, hanging the FVT run for over 3 hours with no timeout or cleanup.

Root Cause

1. sbatch --parsable --wait --nodelist=<down-node> never returns when the target is down because Slurm holds the job PENDING.
2. remote_command() only has an SSH ConnectTimeout (10 s), not an overall command timeout.
3. The bounded _wait_job_state() (180 s) sits after the blocking sbatch --wait call, so it is never reached.
4. The finally cleanup (scancel + workspace removal) only runs after the call returns, and the job ID is only recorded from sbatch stdout, so both cleanup and the ID are unreachable during the hang.
5. No pre-submission readiness check existed for the LDAP path, unlike V035/V037 which already call _mapped_compute_readiness.

Changes

#	File	Change
1	test/orchestrator/library/functions/_workload_helpers.py	Add mapped_compute_readiness() (moved from slurm_jobs_pxeboot_func.py, unchanged body) so both modules can import it
2	test/orchestrator/library/functions/slurm_auth_pxeboot_func.py	Import mapped_compute_readiness and SLURM_JOB_READY_STATES; add _compute_not_ready_reason() helper; check readiness before submission loop; skip down/drained/unknown nodes with a structured FAIL; drop wait=True so sbatch --parsable returns the job ID immediately; use existing _wait_job_state() for bounded polling
3	test/orchestrator/library/functions/slurm_jobs_pxeboot_func.py	Remove the moved _mapped_compute_readiness() definition; import from _workload_helpers instead
4	test/orchestrator/library/vars/pxeboot_vars.py	Add SLURM_JOB_READY_STATES tuple; change slurm_cancel_job to timeout 30 scancel %s

Why this is safe

• No-wait submission is already used in this module. The PAM job test path already calls _ldap_sbatch_command without --wait, records the job ID, and polls _wait_job_state. This fix reuses that proven pattern.
• Pass criteria are unchanged. The existing checks (submitted and completed and allocated_ok and output_ok and not stderr) are not modified.
• Healthy cluster behavior is identical. On a healthy node, hostname -s completes in seconds, so _wait_job_state returns immediately after the first poll.
• The readiness helper body is unchanged. Only its location moved from a private function in slurm_jobs to the shared _workload_helpers module.
• Existing tests pass. All 50 orchestrator unit tests pass, including ORCH_UT_024 (unmapped LDAP role skip).
• V035/V037 are unaffected. They already import _mapped_compute_readiness from slurm_jobs; after the move they import from _workload_helpers instead.

Test plan

• All 50 orchestrator unit tests pass (test/orchestrator/ut/)
• All four changed files compile cleanly (py_compile)
• No lab values, secrets, or IP addresses in the diff
• bandit static analysis: no new findings
• FVT re-run of V048 on a cluster with a down compute node (manual verification needed)
• FVT re-run of V048/V050/V052 on a healthy cluster (confirm no regression)