[SujalS27]

PR Title

fix(orchestrator): apply PowerVault mount options, redirect CUDA dnf cache to NFS, fix firewall test podman zone

Description of the Solution

This PR fixes three independent FVT failures: (1) PowerVault iSCSI mount ignoring configured mount options from storage_config.yml, (2) CUDA toolkit installation failing due to dnf cache exhausting the LiveOS rootfs overlay, and (3) firewall test falsely expecting container-internal ports to be open on the host-facing default zone. All three were discovered during
FVT runs on a stateless PXE-booted cluster.

Related Issue

#4849

• Fixes # [OMN-DEF] FVT: ORCH_FVT_PREPARE_V006 firewall test checks ports that are intentionally closed on the host firewall 1538
• Fixes # [OMN-DEF] setup_iscsi_storage.sh.j2 mounts PowerVault without mount options -- noatime from fstab/mount_params ignored on first mount 1539
• Fixes # [OMN-DEF] install_cuda_toolkit.sh dnf cache fills LiveOS rootfs, causing CUDA installation to fail 1540

Changes

PowerVault iSCSI Mount (src/orchestrator/roles/mount_config/templates/setup_iscsi_storage.sh.j2)

• Pass MOUNT_OPTS to the mount command so fstab options (noatime, _netdev, etc.) are applied on the initial mount i
  nstead of falling back to kernel defaults (relatime). The fstab entry was already written correctly but the mount comma
  nd was invoked without -o, silently ignoring all user-configured mount options.

CUDA Toolkit Installer (src/orchestrator/roles/provision_common/templates/hpc_tools/install_cuda_toolkit.sh.j2)

• Redirect the dnf download cache to NFS (--setopt=cachedir=$LOCK_ROOT/.dnf-cache/$(hostname -s)) to prevent the ~4 GB RP
  M cache from consuming LiveOS rootfs overlay space. This reduces rootfs requirement from ~15 GB to ~11 GB during install.
• Clean up the NFS-based dnf cache on both success and failure paths.
• Add setup_cuda_env call in the "already installed on VAST" code path (lock acquired, nvcc found) so /etc/profile.d/cud a.sh is always created regardless of how CUDA presence is detected.

Firewall Test (test/orchestrator/library/vars/prepare_vars.py, `test/orchestrator/library/functions/network_prepare_

func.py`)

• Remove 5432/tcp (PostgreSQL), 27778/tcp (unused), and 27779/tcp (SMD) from host-facing FIREWALL_PORTS. These port
  s are intentionally not exposed on the host firewall; services are accessed via HAProxy on 8443/tcp.
• Add new FIREWALL_PODMAN_ZONE_PORTS tuple with 5432/tcp and 27779/tcp, checked against the podman firewall zone fo
  r container-to-container traffic.
• Add firewall_podman_port command to PREPARE_COMMANDS using firewall-cmd --zone=podman --query-port=.
• Drop stale 27778/tcp entirely -- not referenced anywhere in the deploy role.

Files Changed

File	Change Type	Description
src/orchestrator/roles/mount_config/templates/setup_iscsi_storage.sh.j2	Modified	Pass -o "${MOUNT_OPTS}" to mount
command
src/orchestrator/roles/provision_common/templates/hpc_tools/install_cuda_toolkit.sh.j2	Modified	Redirect dnf cache
to NFS, add cache cleanup, add setup_cuda_env to already-installed path
test/orchestrator/library/vars/prepare_vars.py	Modified	Move internal ports to FIREWALL_PODMAN_ZONE_PORTS, add `f
irewall_podman_port` command
test/orchestrator/library/functions/network_prepare_func.py	Modified	Import and check FIREWALL_PODMAN_ZONE_PORTS
against podman zone

Testing

• ORCH_FVT_PXEBOOT_V413 (PowerVault mount options): Previously failed with Missing mount opts noatime on /mnt/slurm.
  Root cause confirmed -- bare mount command ignored fstab options. Fix passes -o "${MOUNT_OPTS}" to apply configured opt ions on first mount.
• ORCH_FVT_PXEBOOT_V056/V502/V507/V508/V512/V514/V522 (CUDA toolkit): Previously failed with At least 1969MB more space needed on the / filesystem. Root cause confirmed -- dnf cache (~4.1 GB) consumed rootfs alongside installed packages (~7.8 GB). Fix redirects cache to NFS. Manual verification: CUDA installed successfully with --setopt=cachedir on NFS, nvcc --version confirmed on nid008, nid009, nid010.
• ORCH_FVT_PREPARE_V006 (Firewall ports): Previously failed with 5432/tcp closed, 27778/tcp closed, 27779/tcp closed. Root cause confirmed -- deploy role intentionally opens these ports only in the podman zone, not the default zone. Fix validates ports against the correct zone. Python import validation passed.

Backward Compatibility

• No breaking changes.
• PowerVault mount behavior changes from kernel defaults to user-configured options on first mount. This is the intended behavior that was previously broken.
• CUDA installer now writes dnf cache to $LOCK_ROOT/.dnf-cache/<hostname> on NFS during installation. This directory is cleaned up after install completes (success or failure). No persistent storage impact.
• Firewall test now checks fewer ports on the host default zone and additionally checks the podman zone. Existing deployments that correctly configure the podman zone will pass. The removed 27778/tcp was never opened by the deploy role in any configuration.