From cfa80cba7d9b6d9eadb381edbede2ad6f6e39f11 Mon Sep 17 00:00:00 2001 From: Kratika_Patidar Date: Tue, 10 Nov 2026 14:22:13 +0530 Subject: [PATCH 1/4] updating omnia auth image tag from 1.2 to 1.3 Signed-off-by: Kratika_Patidar --- src/orchestrator/containers/build_images.sh | 2 +- src/orchestrator/containers/omnia_auth/README.md | 2 +- src/orchestrator/roles/deploy_openldap/vars/main.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/orchestrator/containers/build_images.sh b/src/orchestrator/containers/build_images.sh index 8af51cb406..9d872817d3 100755 --- a/src/orchestrator/containers/build_images.sh +++ b/src/orchestrator/containers/build_images.sh @@ -81,7 +81,7 @@ show_help() { BUILD_TOOL="podman" BUILD_ACTION="load" OMNIA_DOCKER_REGISTERY="docker.io/dellhpcomniaaisolution" -AUTH_TAG="1.2" +AUTH_TAG="1.3" # ============================================================================= # Parse command-line parameters (key=value format) diff --git a/src/orchestrator/containers/omnia_auth/README.md b/src/orchestrator/containers/omnia_auth/README.md index 1993b719f2..5c1eea777e 100644 --- a/src/orchestrator/containers/omnia_auth/README.md +++ b/src/orchestrator/containers/omnia_auth/README.md @@ -35,7 +35,7 @@ helper can build it locally or publish it with Docker Buildx: ```bash cd src/orchestrator/containers -# Local Podman build: omnia_auth:1.2 +# Local Podman build: omnia_auth:1.3 ./build_images.sh # Local Docker build with a custom tag diff --git a/src/orchestrator/roles/deploy_openldap/vars/main.yml b/src/orchestrator/roles/deploy_openldap/vars/main.yml index 3881b211c1..ea3f80f7d0 100644 --- a/src/orchestrator/roles/deploy_openldap/vars/main.yml +++ b/src/orchestrator/roles/deploy_openldap/vars/main.yml @@ -15,7 +15,7 @@ # Container image omnia_auth_image: "docker.io/dellhpcomniaaisolution/omnia_auth" -omnia_auth_tag: "1.2" +omnia_auth_tag: "1.3" omnia_auth_container_name: "omnia_auth" # Numeric ids of the unprivileged ldap user inside the omnia_auth image From ee57f9adf2dd76bc8c1f479f0361e4d30edf90c1 Mon Sep 17 00:00:00 2001 From: Kratika_Patidar Date: Tue, 10 Nov 2026 15:14:40 +0530 Subject: [PATCH 2/4] test(orchestrator): add buildstream marker to IPoIB post-boot checks Select the non-disruptive IPoIB address assignment, NetworkManager profile, routing, peer mesh, Slurm NodeAddr, custom slurm.conf, and Slurm job comm-path checks in BuildStream runs. Negative, reboot, fault-injection, reprovision, and long-running precheck cases stay excluded. Signed-off-by: Kratika_Patidar --- test/orchestrator/fvt/README.md | 16 ++++++++-------- .../ipoib_network/test_ipoib_comm_path.py | 1 + .../pxeboot/ipoib_network/test_ipoib_network.py | 7 +++++++ 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/test/orchestrator/fvt/README.md b/test/orchestrator/fvt/README.md index f062c71951..83f33dd6d5 100644 --- a/test/orchestrator/fvt/README.md +++ b/test/orchestrator/fvt/README.md @@ -744,19 +744,19 @@ and re-addition then skips because no backup is left. | Order | TC ID | Test | Suite | Markers | Validation | Pass criteria | |---:|---|---|---|---|---|---| -| 42501 | `ORCH_FVT_PXEBOOT_V819` | `test_ipoib_mode_address_assignment` | `ipoib_network` | `ipoib`, `non_disruptive` | Verify IB addresses assigned on each node match ib_addr_mode. | Each node carries exactly the IB addresses ib_addr_mode selects. | -| 42502 | `ORCH_FVT_PXEBOOT_V800` | `test_ipoib_address_state` | `ipoib_network` | `ipoib`, `non_disruptive` | Verify each approved IPoIB address is static and healthy. | TC-FVT-015/017: every mode family's approved address has its prefix, is static, is the only global address of its family, and (IPv6) is DAD-clean. | -| 42503 | `ORCH_FVT_PXEBOOT_V801` | `test_ipoib_nm_profile` | `ipoib_network` | `ipoib`, `non_disruptive` | Verify the IPoIB NetworkManager profile matches ib_addr_mode. | TC-FVT-002/009: methods follow the mode, assigned families carry the approved address, excluded families carry none, IPv6 privacy is off when IPv6 is assigned, no gateway. | -| 42504 | `ORCH_FVT_PXEBOOT_V802` | `test_ipoib_routes` | `ipoib_network` | `ipoib`, `non_disruptive` | Verify IPoIB routing is on-link with no gateway or default route. | TC-FVT-009/015: per mode family, no gateway/default route on IB; peers route directly via IB from the approved source. | -| 42505 | `ORCH_FVT_PXEBOOT_V803` | `test_ipoib_peer_mesh` | `ipoib_network` | `ipoib`, `non_disruptive` | Verify every IPoIB node reaches every peer in each ib_addr_mode family. | TC-FVT-023: every IB node reaches every IB peer over each mode family. | -| 42507 | `ORCH_FVT_PXEBOOT_V805` | `test_ipoib_slurm_nodeaddr` | `ipoib_network` | `ipoib`, `non_disruptive`, `slurm` | Verify Slurm NodeAddr follows the IPoIB allocation. | TC-FVT-019/031: Slurm NodeAddr follows the IB allocation. | -| 42508 | `ORCH_FVT_PXEBOOT_V806` | `test_ipoib_custom_slurm_conf` | `ipoib_network` | `ipoib`, `non_disruptive`, `slurm` | Verify custom slurm.conf nodes respond over IPoIB. | TC-FVT-027: custom slurm.conf nodes respond over IB. | +| 42501 | `ORCH_FVT_PXEBOOT_V819` | `test_ipoib_mode_address_assignment` | `ipoib_network` | `buildstream`, `ipoib`, `non_disruptive` | Verify IB addresses assigned on each node match ib_addr_mode. | Each node carries exactly the IB addresses ib_addr_mode selects. | +| 42502 | `ORCH_FVT_PXEBOOT_V800` | `test_ipoib_address_state` | `ipoib_network` | `buildstream`, `ipoib`, `non_disruptive` | Verify each approved IPoIB address is static and healthy. | TC-FVT-015/017: every mode family's approved address has its prefix, is static, is the only global address of its family, and (IPv6) is DAD-clean. | +| 42503 | `ORCH_FVT_PXEBOOT_V801` | `test_ipoib_nm_profile` | `ipoib_network` | `buildstream`, `ipoib`, `non_disruptive` | Verify the IPoIB NetworkManager profile matches ib_addr_mode. | TC-FVT-002/009: methods follow the mode, assigned families carry the approved address, excluded families carry none, IPv6 privacy is off when IPv6 is assigned, no gateway. | +| 42504 | `ORCH_FVT_PXEBOOT_V802` | `test_ipoib_routes` | `ipoib_network` | `buildstream`, `ipoib`, `non_disruptive` | Verify IPoIB routing is on-link with no gateway or default route. | TC-FVT-009/015: per mode family, no gateway/default route on IB; peers route directly via IB from the approved source. | +| 42505 | `ORCH_FVT_PXEBOOT_V803` | `test_ipoib_peer_mesh` | `ipoib_network` | `buildstream`, `ipoib`, `non_disruptive` | Verify every IPoIB node reaches every peer in each ib_addr_mode family. | TC-FVT-023: every IB node reaches every IB peer over each mode family. | +| 42507 | `ORCH_FVT_PXEBOOT_V805` | `test_ipoib_slurm_nodeaddr` | `ipoib_network` | `buildstream`, `ipoib`, `non_disruptive`, `slurm` | Verify Slurm NodeAddr follows the IPoIB allocation. | TC-FVT-019/031: Slurm NodeAddr follows the IB allocation. | +| 42508 | `ORCH_FVT_PXEBOOT_V806` | `test_ipoib_custom_slurm_conf` | `ipoib_network` | `buildstream`, `ipoib`, `non_disruptive`, `slurm` | Verify custom slurm.conf nodes respond over IPoIB. | TC-FVT-027: custom slurm.conf nodes respond over IB. | | 42509 | `ORCH_FVT_PXEBOOT_V807` | `test_ipoib_powervault_coexistence` | `ipoib_network` | `ipoib`, `non_disruptive`, `slurm` | Verify IPoIB and PowerVault iSCSI are both operational on the controller. | TC-FVT-024 S1: IB UP with approved address and PowerVault mounted together. | | 42510 | `ORCH_FVT_PXEBOOT_V808` | `test_ipoib_powervault_iscsi_path` | `ipoib_network` | `ipoib`, `non_disruptive`, `slurm` | Verify PowerVault iSCSI traffic routes via Ethernet, not IPoIB. | TC-FVT-024 S2: iSCSI routes via Ethernet; no IB route captures the portals. | | 42511 | `ORCH_FVT_PXEBOOT_V809` | `test_ipoib_powervault_dns_resilience` | `ipoib_network` | `ipoib`, `non_disruptive`, `slurm` | Verify IPoIB DNS injection does not break iSCSI target resolution. | TC-FVT-024 S4: IB DNS injection does not break iSCSI target discovery. | | 42512 | `ORCH_FVT_PXEBOOT_V810` | `test_ipoib_powervault_nfs_coexistence` | `ipoib_network` | `ipoib`, `non_disruptive`, `slurm` | Verify controller NFS, PowerVault, and IPoIB coexist. | TC-FVT-024 S5: NFS (TCP, Ethernet), PowerVault, and IB coexist without conflict. | | 42513 | `ORCH_FVT_PXEBOOT_V811` | `test_ipoib_powervault_ib_reprovision` | `ipoib_network` | `fault_injection`, `ipoib`, `slurm` | Verify PowerVault binds and Slurm state survive IPoIB re-provisioning. | TC-FVT-024 S3: PowerVault binds and Slurm state survive IB re-provisioning. | -| 42514 | `ORCH_FVT_PXEBOOT_V818` | `test_ipoib_slurm_comm_path` | `ipoib_network` | `ipoib`, `non_disruptive`, `slurm` | Verify Slurm job traffic crosses IPoIB in the ib_addr_mode family, not Ethernet. | TC-FVT-023 ext: every captured slurmd, srun I/O, and slurmctld packet uses the IB interface in the NodeAddr family (IPv4 for ipv4-only, IPv6 otherwise). | +| 42514 | `ORCH_FVT_PXEBOOT_V818` | `test_ipoib_slurm_comm_path` | `ipoib_network` | `buildstream`, `ipoib`, `non_disruptive`, `slurm` | Verify Slurm job traffic crosses IPoIB in the ib_addr_mode family, not Ethernet. | TC-FVT-023 ext: every captured slurmd, srun I/O, and slurmctld packet uses the IB interface in the NodeAddr family (IPv4 for ipv4-only, IPv6 otherwise). | `ipoib_network` (42500) intentionally follows `slurm_lifecycle`, so its NodeAddr check also covers nodes re-added by that suite (TC-FVT-031). diff --git a/test/orchestrator/fvt/pxeboot/ipoib_network/test_ipoib_comm_path.py b/test/orchestrator/fvt/pxeboot/ipoib_network/test_ipoib_comm_path.py index a1ba977852..f807296037 100644 --- a/test/orchestrator/fvt/pxeboot/ipoib_network/test_ipoib_comm_path.py +++ b/test/orchestrator/fvt/pxeboot/ipoib_network/test_ipoib_comm_path.py @@ -29,6 +29,7 @@ from fvt.result import verify_pxeboot +@pytest.mark.buildstream @pytest.mark.ipoib @pytest.mark.slurm @pytest.mark.non_disruptive diff --git a/test/orchestrator/fvt/pxeboot/ipoib_network/test_ipoib_network.py b/test/orchestrator/fvt/pxeboot/ipoib_network/test_ipoib_network.py index 3e8acef4d7..5216abe16e 100644 --- a/test/orchestrator/fvt/pxeboot/ipoib_network/test_ipoib_network.py +++ b/test/orchestrator/fvt/pxeboot/ipoib_network/test_ipoib_network.py @@ -31,6 +31,7 @@ from fvt.result import verify_pxeboot +@pytest.mark.buildstream @pytest.mark.ipoib @pytest.mark.non_disruptive @pytest.mark.order(42501) @@ -41,6 +42,7 @@ def test_ipoib_mode_address_assignment(host): verify_pxeboot(test_log, tc, host, check_ipoib_mode_address_assignment) +@pytest.mark.buildstream @pytest.mark.ipoib @pytest.mark.non_disruptive @pytest.mark.order(42502) @@ -51,6 +53,7 @@ def test_ipoib_address_state(host): verify_pxeboot(test_log, tc, host, check_ipoib_address_state) +@pytest.mark.buildstream @pytest.mark.ipoib @pytest.mark.non_disruptive @pytest.mark.order(42503) @@ -61,6 +64,7 @@ def test_ipoib_nm_profile(host): verify_pxeboot(test_log, tc, host, check_ipoib_nm_profile) +@pytest.mark.buildstream @pytest.mark.ipoib @pytest.mark.non_disruptive @pytest.mark.order(42504) @@ -71,6 +75,7 @@ def test_ipoib_routes(host): verify_pxeboot(test_log, tc, host, check_ipoib_routes) +@pytest.mark.buildstream @pytest.mark.ipoib @pytest.mark.non_disruptive @pytest.mark.order(42505) @@ -81,6 +86,7 @@ def test_ipoib_peer_mesh(host): verify_pxeboot(test_log, tc, host, check_ipoib_peer_mesh) +@pytest.mark.buildstream @pytest.mark.ipoib @pytest.mark.slurm @pytest.mark.non_disruptive @@ -92,6 +98,7 @@ def test_ipoib_slurm_nodeaddr(host): verify_pxeboot(test_log, tc, host, check_ipoib_slurm_nodeaddr) +@pytest.mark.buildstream @pytest.mark.ipoib @pytest.mark.slurm @pytest.mark.non_disruptive From c8824e19ecc7bae1e9007d2d30a94df881babc2d Mon Sep 17 00:00:00 2001 From: Kratika_Patidar Date: Fri, 13 Nov 2026 11:44:16 +0530 Subject: [PATCH 3/4] feat(orchestrator): make NFS mount options configurable for PowerScale Add configurable NFS mount options for Kubernetes/Slurm node sub-mounts and PowerScale CSI volumes to address Input/output errors with RHEL 10.x clients and PowerScale at default 1 MiB transfer sizes. Changes: - Add node_nfs_mnt_opts to storage_config schema and input documentation - Add csi_powerscale_mount_options to omnia_config schema and input documentation - Resolve K8s/Slurm mount options in configure_metadata_svc.yml - Update K8s cloud-init templates to use configurable options - Update Slurm/login cloud-init templates to use configurable options - Update PowerScale CSI StorageClass template to support mountOptions - Add default mount option variables in provision_common/vars - Pass mount options to nfs-client provisioner when configured Backward compatible: both options are optional with sensible defaults matching previous hardcoded values. No behavior change unless keys are set. Related to issue #1445 Signed-off-by: Kratika_Patidar --- src/orchestrator/input/omnia_config.yml | 41 ++---- src/orchestrator/input/storage_config.yml | 29 +++++ .../schema/omnia_config.json | 33 +---- .../schema/storage_config.json | 3 + .../tasks/create_k8s_config_nfs.yml | 18 +-- .../k8s_config/templates/ps_storage_class.j2 | 8 +- .../tasks/configure_metadata_svc.yml | 35 +++--- ...-group-login_compiler_node_aarch64.yaml.j2 | 41 ++---- ...s-group-login_compiler_node_x86_64.yaml.j2 | 43 ++----- .../ms-group-login_node_aarch64.yaml.j2 | 43 ++----- .../ms-group-login_node_x86_64.yaml.j2 | 43 ++----- ...ce_kube_control_plane_first_x86_64.yaml.j2 | 66 ++++------ ...-service_kube_control_plane_x86_64.yaml.j2 | 36 ++---- .../ms-group-service_kube_node_x86_64.yaml.j2 | 34 ++--- ...ms-group-slurm_control_node_x86_64.yaml.j2 | 34 +++-- .../ms-group-slurm_node_aarch64.yaml.j2 | 26 ++-- .../ms-group-slurm_node_x86_64.yaml.j2 | 26 ++-- .../roles/provision_common/vars/main.yml | 119 +----------------- 18 files changed, 215 insertions(+), 463 deletions(-) diff --git a/src/orchestrator/input/omnia_config.yml b/src/orchestrator/input/omnia_config.yml index 17ae4c810c..2d13cd35d5 100644 --- a/src/orchestrator/input/omnia_config.yml +++ b/src/orchestrator/input/omnia_config.yml @@ -60,7 +60,7 @@ # Best for: Mixed hardware environments with different node configurations # - homogeneous: Groups nodes by hardware type for optimized discovery # Best for: Standardized hardware groups (grp0-grp100 in pxe_mapping_file.csv) -# Performance: 0 iDRAC calls (with specs) or 1 call per group (without specs) +# Performance: 0 iDRAC calls (with specs) or 1 call per group (without specs) # Default value is heterogeneous # node_hardware_defaults @@ -119,6 +119,7 @@ slurm_cluster: - cluster_name: slurm_cluster nfs_storage_name: nfs_slurm + vast_storage_name: vast_storage # Optional custom Slurm configuration. Keep skip_merge and config_sources # nested under this slurm_cluster entry. Choose one config_sources format # and remove only the first '#' from each configuration line in that example. @@ -148,17 +149,7 @@ slurm_cluster: # - cluster_name is required field # - deployment: Exactly one entry in both the service_k8s_cluster lists must have deployment set to true to indicate where Kubernetes should be deployed. - -# ----------------------------SERVICE K8S HIGH AVAILABILITY----------------------------------- -# - enable_k8s_ha: (Mandatory on the deployed entry when Kubernetes nodes are mapped) -# Indicates whether to enable HA for the Kubernetes service node. -# Set to 'true' to enable, 'false' to disable. - -# - virtual_ip_address: (Mandatory on the deployed entry when Kubernetes nodes are mapped) -# The virtual IP address for the K8s service node setup. It is used by kube-vip -# even when enable_k8s_ha is false. -# It must sit in the control-plane subnet, and must not collide with an address -# assigned to a mapped node, a DHCP range, an OIM address, or pod_external_ip_range. +# Please ensure corresponding cluster entry is added to high_availability_config.yml if deployment is set to true. # - Kubernetes SDN network.K8s_cni (Mandatory) - It can either be "calico" or "flannel".Default value assigned is "calico". # While setting up Kubernetes plugin for RoCE NIC, ensure that this value is set to "flannel" @@ -186,6 +177,10 @@ slurm_cluster: # File path for the values.yml file which will contain the Powerscale driver configuration parameters. # csi_powerscale_driver_values_file_path: User need to download values.yaml file and fill required data. # Provided the path of the values.yaml file here. +# csi_powerscale_mount_options: Optional comma-separated NFS mount options applied to +# PowerScale CSI volumes through the ps01 StorageClass mountOptions. Leave empty to use +# the kernel defaults. For RHEL 10.x clients that report Input/output errors on PowerScale +# volumes with the default 1 MiB transfer size, use "rsize=1044480,wsize=1044480". # mention configurable values # - k8s_crio_storage_size: Specifies the disk size allocated for CRI-O container storage. @@ -193,36 +188,18 @@ slurm_cluster: # Acceptable formats: "10G", "15G", "50G" (Only positive values in Gigabytes are allowed) # Default value is "20G" -# - k8s_bootstrap_token_ttl: Duration for which kubeadm bootstrap tokens remain valid. -# Bootstrap tokens are used by worker and control-plane nodes to authenticate when -# joining the cluster. After this duration, tokens are automatically deleted and -# operators must regenerate them by running the refresh script on the first control-plane node: -# /generate-control-plane-join.sh -# (e.g. /opt/omnia/k8s_mount/generate-control-plane-join.sh — path matches the 'nfs_k8s' storage mount point) -# Acceptable formats: "1h", "2h", "4h", "24h" (Go duration strings accepted by kubeadm) -# Default value is "2h" - -# - etcd_on_local_disk: When set to true, etcd data is stored on a dedicated local disk (BOSS or -# fallback disk) on each control-plane node, formatted as ext4 and mounted at /var/lib/etcd. -# The disk will be formatted and mounted automatically during provisioning. -# When set to false, etcd data is stored on NFS. If the control-plane node does not have a -# dedicated disk, set this to false so etcd falls back to NFS storage. -# Default value is "true" - service_k8s_cluster: - cluster_name: service_cluster deployment: true - etcd_on_local_disk: true + etcd_on_local_disk: false k8s_cni: "calico" pod_external_ip_range: "172.16.107.170-172.16.107.200" k8s_service_addresses: "10.233.0.0/18" k8s_pod_network_cidr: "10.233.64.0/18" nfs_storage_name: "nfs_k8s" k8s_crio_storage_size: "20G" - k8s_bootstrap_token_ttl: "2h" enable_powerscale_csi: false csi_powerscale_driver_secret_file_path: "" csi_powerscale_driver_values_file_path: "" - enable_k8s_ha: true - virtual_ip_address: "172.16.107.1" + csi_powerscale_mount_options: "" diff --git a/src/orchestrator/input/storage_config.yml b/src/orchestrator/input/storage_config.yml index daf735a23c..aa9b1aa17e 100644 --- a/src/orchestrator/input/storage_config.yml +++ b/src/orchestrator/input/storage_config.yml @@ -47,6 +47,17 @@ # If specified, takes PRIORITY over mount_params profile # - mount_on_oim: Mount this filesystem on the OIM node as well. Optional. Default: false # Ensure storage is network-accessible from OIM before enabling +# - node_nfs_mnt_opts: Mount options for the per-node NFS sub-mounts that Omnia creates +# from the storage entry referenced by nfs_storage_name in omnia_config.yml. +# Optional. Applies to Kubernetes (/var/lib/kubelet, /var/lib/etcd, +# /etc/kubernetes, /var/log/pods, /var/lib/packages) and Slurm/login +# (/etc/slurm, /etc/munge, /var/spool/slurmd, /var/log/slurm, /cert, ...). +# For Kubernetes it is also passed to the nfs-client provisioner. +# Default: "noatime,nolock" (Kubernetes) and "defaults,_netdev" (Slurm) +# PowerScale with RHEL 10.x clients: if Input/output errors occur with the +# default 1 MiB transfer size, set for example +# "noatime,nolock,rsize=1044480,wsize=1044480" (Kubernetes) or +# "defaults,_netdev,rsize=1044480,wsize=1044480" (Slurm) # # Node-specific bind mounts (node_key and node_mount_point are paired): # - node_key: Per-node subdirectory isolation variable. Optional @@ -125,6 +136,24 @@ mounts: mount_on_oim: true functional_group_prefix: ["service_kube"] + # VAST Storage - High-performance NFS for HPC tools and benchmarks + # This storage is critical for the Slurm cluster and must be accessible from OIM. + # Purpose: Serves as the centralized repository for HPC tools (UCX, OpenMPI, CUDA, benchmarks) + # that are bind-mounted to /hpc_tools on compute and login nodes. + # OIM Access: The VAST server (172.16.107.77) must be accessible from OIM to copy hpc_tools + # content during provisioning. OIM mounts this storage to populate the shared + # hpc_tools directory structure before nodes boot. + # Configuration: The slurm_cluster section in omnia_config.yml should reference this storage + # to ensure proper integration with cluster provisioning workflows. + # This standard entry is skipped when vast_storage_name is absent or empty. + # Note: Uses RDMA mount parameters for optimal performance on InfiniBand networks. + - name: "vast_storage" + source: "172.16.107.77:/share/vast" + mount_point: "/mnt/vast" + mount_params: "vast_rdma" + mount_on_oim: true + functional_group_prefix: ["slurm_node", "login"] + # -----------------------------Mount Params (Profiles)------------------------------- # mount_params: Named default profiles for mount configurations. # Profiles are referenced by name from mount entries via the mount_params field. diff --git a/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/omnia_config.json b/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/omnia_config.json index 5039a3708e..134d467628 100644 --- a/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/omnia_config.json +++ b/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/omnia_config.json @@ -103,23 +103,19 @@ "properties": { "sockets": { "type": "integer", - "minimum": 1, - "maximum": 16 + "minimum": 1 }, "cores_per_socket": { "type": "integer", - "minimum": 1, - "maximum": 160 + "minimum": 1 }, "threads_per_core": { "type": "integer", - "minimum": 1, - "maximum": 2 + "minimum": 1 }, "real_memory": { "type": "integer", - "minimum": 1, - "maximum": 8589934592 + "minimum": 1 }, "gres": { "type": "string", @@ -260,12 +256,6 @@ "type": "string", "pattern": "^[1-9][0-9]*G$" }, - "k8s_bootstrap_token_ttl": { - "type": "string", - "description": "Duration for kubeadm bootstrap token validity (e.g. 1h, 2h, 24h). Set to 0 to disable expiry (not recommended).", - "pattern": "^(0|([0-9]+h([0-9]+m)?([0-9]+s)?|[0-9]+m([0-9]+s)?|[0-9]+s))$", - "default": "2h" - }, "enable_powerscale_csi": { "type": "boolean", "default": false @@ -276,20 +266,9 @@ "csi_powerscale_driver_values_file_path": { "$ref": "#/definitions/optionalAbsolutePath" }, - "enable_k8s_ha": { - "type": "boolean" - }, - "virtual_ip_address": { + "csi_powerscale_mount_options": { "type": "string", - "description": "Kubernetes HA virtual IP. Required, non-empty, when the PXE mapping selects Kubernetes (enforced by L2 validation).", - "anyOf": [ - { - "maxLength": 0 - }, - { - "format": "ipv4" - } - ] + "pattern": "^([A-Za-z0-9,=._@/-]+)?$" } }, "allOf": [ diff --git a/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/storage_config.json b/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/storage_config.json index 85dd430b2c..79ffa4ba65 100644 --- a/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/storage_config.json +++ b/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/storage_config.json @@ -171,6 +171,9 @@ "mnt_opts": { "$ref": "#/definitions/mountOptions" }, + "node_nfs_mnt_opts": { + "$ref": "#/definitions/mountOptions" + }, "dump_freq": { "$ref": "#/definitions/dumpFrequency" }, diff --git a/src/orchestrator/roles/k8s_config/tasks/create_k8s_config_nfs.yml b/src/orchestrator/roles/k8s_config/tasks/create_k8s_config_nfs.yml index a08bef336e..675c5101ee 100644 --- a/src/orchestrator/roles/k8s_config/tasks/create_k8s_config_nfs.yml +++ b/src/orchestrator/roles/k8s_config/tasks/create_k8s_config_nfs.yml @@ -18,6 +18,9 @@ - name: Include storage vars ansible.builtin.include_vars: "{{ input_project_dir }}/storage_config.yml" +- name: Include include_high_availability_config vars + ansible.builtin.include_vars: "{{ ha_config_file }}" + - name: Identify deployed service Kubernetes configuration ansible.builtin.set_fact: deployed_service_k8s_cluster_matches: >- @@ -30,7 +33,7 @@ ansible.builtin.set_fact: deployed_service_k8s_cluster: >- {{ deployed_service_k8s_cluster_matches | first - | default({}) }} + | default(service_k8s_cluster | default([]) | first | default({})) }} - name: Set facts for service_k8s ansible.builtin.set_fact: @@ -61,6 +64,10 @@ csi_powerscale_driver_values_file_path: >- {{ deployed_service_k8s_cluster.get( 'csi_powerscale_driver_values_file_path', '') }} + csi_powerscale_mount_options: >- + {{ (deployed_service_k8s_cluster.get('csi_powerscale_mount_options', '') + | default('', true) | string | trim).split(',') + | map('trim') | select | list }} - name: Validate PowerScale CSI input paths ansible.builtin.assert: @@ -133,7 +140,6 @@ community.general.read_csv: path: "{{ hostvars['localhost']['pxe_mapping_file_path'] }}" register: pxe_data_for_cidr - no_log: true delegate_to: localhost connection: local run_once: true @@ -177,12 +183,8 @@ - name: Set HA-related facts ansible.builtin.set_fact: - enable_k8s_ha: >- - {{ hostvars['localhost']['service_k8s_deployed_cluster'] - .enable_k8s_ha | default(false) }} - kube_vip: >- - {{ hostvars['localhost']['service_k8s_deployed_cluster'] - .virtual_ip_address | default('') }} + enable_k8s_ha: "{{ service_k8s_cluster_ha[0].enable_k8s_ha | default(false) }}" + kube_vip: "{{ service_k8s_cluster_ha[0].virtual_ip_address | default('') }}" - name: Set fact for pulp mirror ansible.builtin.set_fact: diff --git a/src/orchestrator/roles/k8s_config/templates/ps_storage_class.j2 b/src/orchestrator/roles/k8s_config/templates/ps_storage_class.j2 index c407073b2e..405660c74d 100644 --- a/src/orchestrator/roles/k8s_config/templates/ps_storage_class.j2 +++ b/src/orchestrator/roles/k8s_config/templates/ps_storage_class.j2 @@ -6,9 +6,15 @@ provisioner: csi-isilon.dellemc.com reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: Immediate +{% if csi_powerscale_mount_options | default([]) | length > 0 %} +mountOptions: +{% for opt in csi_powerscale_mount_options %} + - "{{ opt }}" +{% endfor %} +{% endif %} parameters: AccessZone: {{ ps_access_zone }} Isipath: {{ ps_isipath }} RootClientEnabled: "true" AzServiceIP: {{ ps_azserviceip }} - csi.storage.k8s.io/fstype: "nfs" \ No newline at end of file + csi.storage.k8s.io/fstype: "nfs" diff --git a/src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml b/src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml index 1f527e56d3..b83c2eac0f 100644 --- a/src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml +++ b/src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml @@ -91,6 +91,9 @@ metadata_svc_nfs_path_openldap: "{{ metadata_slurm_nfs_storage_matches[0].source }}/openldap" metadata_svc_slurm_nfs_path: "{{ metadata_slurm_nfs_storage_matches[0].source }}" client_mount_path: "{{ metadata_slurm_nfs_storage_matches[0].mount_point }}" + slurm_nfs_mnt_opts: >- + {{ metadata_slurm_nfs_storage_matches[0].node_nfs_mnt_opts + | default(slurm_nfs_default_mnt_opts, true) }} when: target_category | default('') == 'slurm' - name: Set default effective directories for slurm paths @@ -136,6 +139,12 @@ # metadata-service templates need them during provision_common which # runs before k8s_config. Guard all tasks with target_category check. +- name: Load high availability config for kube_vip + ansible.builtin.include_vars: + file: "{{ input_project_dir }}/high_availability_config.yml" + failed_when: false + when: target_category | default('') == 'kubernetes' + - name: Load orchestrator_config for dns_enabled when: - target_category | default('') == 'kubernetes' @@ -148,7 +157,7 @@ - name: Set dns_enabled fact ansible.builtin.set_fact: - dns_enabled: "{{ _orch_config_ms.dns_enabled | default(true) | bool }}" + dns_enabled: "{{ _orch_config_ms.dns_enabled | default(false) | bool }}" - name: Select deployed service Kubernetes configuration ansible.builtin.set_fact: @@ -157,7 +166,7 @@ | selectattr('deployment', 'defined') | selectattr('deployment', 'sameas', true) | first - | default({}) }} + | default(service_k8s_cluster | default([]) | first | default({})) }} when: target_category | default('') == 'kubernetes' - name: Read version-qualified K8s repository base URLs @@ -178,17 +187,14 @@ quiet: true when: target_category | default('') == 'kubernetes' -- name: Set K8s facts from effective deployed cluster configuration +- name: Set K8s facts from omnia_config and HA config ansible.builtin.set_fact: - kube_vip: >- - {{ hostvars['localhost']['service_k8s_deployed_cluster'] - .virtual_ip_address | default('') }} + kube_vip: "{{ service_k8s_cluster_ha[0].virtual_ip_address | default('') }}" service_k8s_version: "{{ hostvars['localhost']['service_k8s_version'] }}" k8s_pod_network_cidr: "{{ metadata_service_k8s_cluster.k8s_pod_network_cidr }}" k8s_service_addresses: "{{ metadata_service_k8s_cluster.k8s_service_addresses }}" pod_external_ip_range: "{{ metadata_service_k8s_cluster.pod_external_ip_range }}" k8s_crio_storage_size: "{{ metadata_service_k8s_cluster.k8s_crio_storage_size }}" - k8s_bootstrap_token_ttl: "{{ metadata_service_k8s_cluster.k8s_bootstrap_token_ttl | default('2h') }}" etcd_on_local_disk: "{{ metadata_service_k8s_cluster.etcd_on_local_disk | default(false) }}" admin_netmask_bits: "{{ hostvars['localhost']['admin_netmask_bits'] }}" pulp_mirror: "{{ hostvars['localhost']['admin_nic_ip'] }}:{{ hostvars['localhost']['pulp_port'] | default(2225) }}" @@ -222,6 +228,11 @@ ansible.builtin.set_fact: k8s_client_mount_path: "{{ metadata_k8s_storage_matches[0].mount_point }}" k8s_nfs_server_path: "{{ metadata_k8s_storage_matches[0].source }}" + k8s_nfs_mnt_opts: >- + {{ metadata_k8s_storage_matches[0].node_nfs_mnt_opts + | default(k8s_nfs_default_mnt_opts, true) }} + k8s_nfs_custom_mnt_opts: >- + {{ metadata_k8s_storage_matches[0].node_nfs_mnt_opts | default('', true) }} when: target_category | default('') == 'kubernetes' - name: Load K8s packages JSON for template variables @@ -447,13 +458,3 @@ - >- ((_metadata_fg_reconcile.created | default([])) + (_metadata_fg_reconcile.updated | default([]))) | length > 0 - -- name: Track hosts whose per-host metadata changed for {{ fg_name }} - ansible.builtin.set_fact: - omnia_metadata_changed_hostnames: >- - {{ ((omnia_metadata_changed_hostnames | default([])) - + (_metadata_fg_reconcile.inventory_updated | default({}) - | dict2items | map(attribute='value') | flatten | list)) - | unique | list }} - when: - - _metadata_fg_reconcile.inventory_updated | default({}) | length > 0 diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2 index 2a7f56de5b..81f67aafaa 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2 @@ -137,7 +137,7 @@ content: | {{ lookup('template', 'templates/hpc_tools/setup_doca_mpi_env.sh.j2') | indent(12) }} -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -149,11 +149,9 @@ - path: /etc/hosts append: true content: | - # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} - # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -202,7 +200,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -237,7 +235,7 @@ - /usr/local/bin/configure_vast_installation.sh {% endif %} - mkdir -p {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages - - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mount -av - rm -f /etc/slurm/slurm.conf /etc/slurm/slurm.conf.example - cp /cert/pulp_webserver.crt /etc/pki/ca-trust/source/anchors && update-ca-trust @@ -274,13 +272,13 @@ # Add the Slurm SSH child mount after the configured storage root is mounted - mkdir -p {{ client_mount_path }}/slurm/ssh - mkdir -p {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages /hpc_tools - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ trackfile_nfs_path }} /var/log/track nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ trackfile_nfs_path }} /var/log/track nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab - mount -av @@ -355,25 +353,8 @@ PXE_SUBNET="$NETWORK_IP/$NETMASK_BITS" echo "[INFO] Admin IP: $ADMIN_IP, Netmask: /$NETMASK_BITS, PXE Subnet: $PXE_SUBNET" firewall-cmd --zone=trusted --add-source="$PXE_SUBNET" --permanent -{% for extra in hostvars['localhost']['admin_additional_subnets'] | default([]) %} - echo "[INFO] Adding additional admin subnet {{ extra.subnet }}/{{ extra.netmask_bits }} to trusted zone" - firewall-cmd --zone=trusted --add-source="{{ extra.subnet }}/{{ extra.netmask_bits }}" --permanent -{% endfor %} ' - # Configure MPI OOB port range to match firewall-allowed SrunPortRange. - # Uses /etc/profile.d/ environment variables so the setting is picked up - # regardless of the OpenMPI installation prefix. - # OMPI_MCA_ — read by OpenMPI 4.x (DOCA / HPC-X) - # PRTE_MCA_ — read by PRRTE (OpenMPI 5.x runtime) - - | - cat > /etc/profile.d/omnia_mpi_oob.sh <<'MPIEOF' - # Restrict MPI OOB TCP to the Slurm SrunPortRange (firewall-allowed) - export OMPI_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" - export PRTE_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" - MPIEOF - - chmod 0644 /etc/profile.d/omnia_mpi_oob.sh - - firewall-cmd --reload - systemctl enable sshd - systemctl start sshd @@ -390,7 +371,7 @@ {% if hostvars['localhost']['openldap_support'] %} - /usr/local/bin/update_ldap_conf.sh - mkdir /ldapcerts - - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mkdir -pv {{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home - echo "{{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home none bind 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2 index 70e0d6ee20..bd33af81ce 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2 @@ -137,7 +137,7 @@ {{ lookup('template', 'templates/hpc_tools/setup_doca_mpi_env.sh.j2') | indent(12) }} -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -149,11 +149,9 @@ - path: /etc/hosts append: true content: | - # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} - # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -202,7 +200,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -237,7 +235,7 @@ - /usr/local/bin/configure_vast_installation.sh {% endif %} - mkdir -p {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages - - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mount -av - rm -f /etc/slurm/slurm.conf /etc/slurm/slurm.conf.example - cp /cert/pulp_webserver.crt /etc/pki/ca-trust/source/anchors && update-ca-trust @@ -274,13 +272,13 @@ # Add the Slurm SSH child mount after the configured storage root is mounted - mkdir -p {{ client_mount_path }}/slurm/ssh - mkdir -p {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ trackfile_nfs_path }} /var/log/track nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ trackfile_nfs_path }} /var/log/track nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab - mount -a @@ -355,25 +353,8 @@ PXE_SUBNET="$NETWORK_IP/$NETMASK_BITS" echo "[INFO] Admin IP: $ADMIN_IP, Netmask: /$NETMASK_BITS, PXE Subnet: $PXE_SUBNET" firewall-cmd --zone=trusted --add-source="$PXE_SUBNET" --permanent -{% for extra in hostvars['localhost']['admin_additional_subnets'] | default([]) %} - echo "[INFO] Adding additional admin subnet {{ extra.subnet }}/{{ extra.netmask_bits }} to trusted zone" - firewall-cmd --zone=trusted --add-source="{{ extra.subnet }}/{{ extra.netmask_bits }}" --permanent -{% endfor %} ' - - # Configure MPI OOB port range to match firewall-allowed SrunPortRange. - # Uses /etc/profile.d/ environment variables so the setting is picked up - # regardless of the OpenMPI installation prefix. - # OMPI_MCA_ — read by OpenMPI 4.x (DOCA / HPC-X) - # PRTE_MCA_ — read by PRRTE (OpenMPI 5.x runtime) - - | - cat > /etc/profile.d/omnia_mpi_oob.sh <<'MPIEOF' - # Restrict MPI OOB TCP to the Slurm SrunPortRange (firewall-allowed) - export OMPI_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" - export PRTE_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" - MPIEOF - - chmod 0644 /etc/profile.d/omnia_mpi_oob.sh - + - firewall-cmd --reload - systemctl enable sshd - systemctl start sshd @@ -392,7 +373,7 @@ {% if hostvars['localhost']['openldap_support'] %} - /usr/local/bin/update_ldap_conf.sh - mkdir /ldapcerts - - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mkdir -pv {{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home - echo "{{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home none bind 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_aarch64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_aarch64.yaml.j2 index 3e65a32069..548936ce01 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_aarch64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_aarch64.yaml.j2 @@ -101,7 +101,7 @@ {{ lookup('template', 'templates/openldap/update_ldap_conf.sh.j2') | indent(12) }} {% endif %} -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -113,11 +113,9 @@ - path: /etc/hosts append: true content: | - # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} - # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -158,7 +156,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -193,7 +191,7 @@ - /usr/local/bin/configure_vast_installation.sh {% endif %} - mkdir -p {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages - - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mount -av - rm -f /etc/slurm/slurm.conf /etc/slurm/slurm.conf.example - cp /cert/pulp_webserver.crt /etc/pki/ca-trust/source/anchors && update-ca-trust @@ -230,13 +228,13 @@ - mkdir -p {{ client_mount_path }}/slurm/ssh - mkdir -p {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages # Additional Slurm login node mounts (excluding cert which was mounted earlier) - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ trackfile_nfs_path }} /var/log/track nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ trackfile_nfs_path }} /var/log/track nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab - mount -av - /usr/local/bin/check_slurm_controller_status.sh @@ -292,25 +290,8 @@ PXE_SUBNET="$NETWORK_IP/$NETMASK_BITS" echo "[INFO] Admin IP: $ADMIN_IP, Netmask: /$NETMASK_BITS, PXE Subnet: $PXE_SUBNET" firewall-cmd --zone=trusted --add-source="$PXE_SUBNET" --permanent -{% for extra in hostvars['localhost']['admin_additional_subnets'] | default([]) %} - echo "[INFO] Adding additional admin subnet {{ extra.subnet }}/{{ extra.netmask_bits }} to trusted zone" - firewall-cmd --zone=trusted --add-source="{{ extra.subnet }}/{{ extra.netmask_bits }}" --permanent -{% endfor %} ' - - # Configure MPI OOB port range to match firewall-allowed SrunPortRange. - # Uses /etc/profile.d/ environment variables so the setting is picked up - # regardless of the OpenMPI installation prefix. - # OMPI_MCA_ — read by OpenMPI 4.x (DOCA / HPC-X) - # PRTE_MCA_ — read by PRRTE (OpenMPI 5.x runtime) - - | - cat > /etc/profile.d/omnia_mpi_oob.sh <<'MPIEOF' - # Restrict MPI OOB TCP to the Slurm SrunPortRange (firewall-allowed) - export OMPI_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" - export PRTE_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" - MPIEOF - - chmod 0644 /etc/profile.d/omnia_mpi_oob.sh - + - firewall-cmd --reload - systemctl enable sshd - systemctl start sshd @@ -326,7 +307,7 @@ {% if hostvars['localhost']['openldap_support'] %} - /usr/local/bin/update_ldap_conf.sh - mkdir /ldapcerts - - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mkdir -pv {{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home - echo "{{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home none bind 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_x86_64.yaml.j2 index 0ffe8c2f2a..33750496de 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_x86_64.yaml.j2 @@ -100,7 +100,7 @@ content: | {{ lookup('template', 'templates/openldap/update_ldap_conf.sh.j2') | indent(12) }} {% endif %} -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -112,11 +112,9 @@ - path: /etc/hosts append: true content: | - # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} - # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -157,7 +155,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -192,7 +190,7 @@ - /usr/local/bin/configure_vast_installation.sh {% endif %} - mkdir -p {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages - - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mount -av - rm -f /etc/slurm/slurm.conf /etc/slurm/slurm.conf.example - cp /cert/pulp_webserver.crt /etc/pki/ca-trust/source/anchors && update-ca-trust @@ -226,14 +224,14 @@ - mount -av - chmod 1777 /scratch # Additional Slurm login node mounts (excluding cert which was mounted earlier) - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ trackfile_nfs_path }} /var/log/track nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ trackfile_nfs_path }} /var/log/track nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mkdir -p {{ client_mount_path }}/slurm/ssh - - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab - mount -av - /usr/local/bin/check_slurm_controller_status.sh @@ -290,25 +288,8 @@ PXE_SUBNET="$NETWORK_IP/$NETMASK_BITS" echo "[INFO] Admin IP: $ADMIN_IP, Netmask: /$NETMASK_BITS, PXE Subnet: $PXE_SUBNET" firewall-cmd --zone=trusted --add-source="$PXE_SUBNET" --permanent -{% for extra in hostvars['localhost']['admin_additional_subnets'] | default([]) %} - echo "[INFO] Adding additional admin subnet {{ extra.subnet }}/{{ extra.netmask_bits }} to trusted zone" - firewall-cmd --zone=trusted --add-source="{{ extra.subnet }}/{{ extra.netmask_bits }}" --permanent -{% endfor %} ' - - # Configure MPI OOB port range to match firewall-allowed SrunPortRange. - # Uses /etc/profile.d/ environment variables so the setting is picked up - # regardless of the OpenMPI installation prefix. - # OMPI_MCA_ — read by OpenMPI 4.x (DOCA / HPC-X) - # PRTE_MCA_ — read by PRRTE (OpenMPI 5.x runtime) - - | - cat > /etc/profile.d/omnia_mpi_oob.sh <<'MPIEOF' - # Restrict MPI OOB TCP to the Slurm SrunPortRange (firewall-allowed) - export OMPI_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" - export PRTE_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" - MPIEOF - - chmod 0644 /etc/profile.d/omnia_mpi_oob.sh - + - firewall-cmd --reload - systemctl enable sshd - systemctl start sshd @@ -324,7 +305,7 @@ {% if hostvars['localhost']['openldap_support'] %} - /usr/local/bin/update_ldap_conf.sh - mkdir /ldapcerts - - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mkdir -pv {{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home - echo "{{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home none bind 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_first_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_first_x86_64.yaml.j2 index e6e6ce745e..939450c968 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_first_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_first_x86_64.yaml.j2 @@ -312,30 +312,22 @@ content: | #!/bin/bash set -e - # Shared mount path where the join commands will be saved + # Shared mount path where the control-plane join command will be saved K8S_CLIENT_MOUNT_PATH="{{ k8s_client_mount_path }}" mkdir -p "$K8S_CLIENT_MOUNT_PATH" - echo "Generating Kubernetes join commands..." - + echo "Generating Kubernetes control-plane join command..." # Generate certificate key and control plane join command CERT_KEY=$(kubeadm init phase upload-certs --upload-certs | tail -1 | tr -d '\r\n ') if [ -n "$CERT_KEY" ]; then - CONTROL_PLANE_JOIN_CMD=$(kubeadm token create --ttl {{ k8s_bootstrap_token_ttl }} --print-join-command --certificate-key "$CERT_KEY") + CONTROL_PLANE_JOIN_CMD=$(kubeadm token create --ttl 0 --print-join-command --certificate-key "$CERT_KEY") echo "$CONTROL_PLANE_JOIN_CMD" > "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" - chmod 600 "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" - echo "Saved control-plane join command." + chmod 644 "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" + echo "Saved control-plane join command to: ${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" else echo "ERROR: Certificate key is empty! Cannot generate control-plane join command." exit 1 fi - - # Worker join command — always refresh alongside control-plane - WORKER_JOIN_CMD=$(kubeadm token create --ttl {{ k8s_bootstrap_token_ttl }} --print-join-command | sed 's/[[:space:]]*$//') - echo "$WORKER_JOIN_CMD" > "${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" - chmod 600 "${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" - echo "Saved worker join command." - - echo "All join commands refreshed. Nodes may now join the cluster." + echo "Control-plane join script is ready. You can rerun this script anytime to refresh the command." - path: /usr/local/bin/install-helm.sh permissions: '0755' @@ -406,12 +398,12 @@ {% if etcd_on_local_disk | default(false) %} - /usr/local/bin/etcd-fstab-update.sh {% else %} - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/etcd /var/lib/etcd nfs noatime,nolock 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/etcd /var/lib/etcd nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab {% endif %} - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubelet /var/lib/kubelet nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubernetes /etc/kubernetes nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/pod-logs /var/log/pods nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/packages /var/lib/packages nfs noatime,nolock 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubelet /var/lib/kubelet nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubernetes /etc/kubernetes nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/pod-logs /var/log/pods nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/packages /var/lib/packages nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - echo "tmpfs /tmp/crio-storage tmpfs size={{ k8s_crio_storage_size }},noatime,nodev,nosuid 0 0" >> /etc/fstab - sudo swapoff -a - sudo sed -i '/ swap / s/^/#/' /etc/fstab @@ -744,7 +736,7 @@ # Patch: append nameservers after /etc/resolv.conf using Jinja list "dns" sed -i 's|/etc/resolv.conf|/etc/resolv.conf{% for ns in dns | default([], true) %} {{ ns }}{% endfor %}|' "$cfg" -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} # Forward cluster-internal DNS domain to OIM CoreDNS # This allows K8s pods to resolve Slurm/MPI hostnames via CoreDNS python3 - "$cfg" << 'PYEOF' @@ -843,20 +835,18 @@ # Get the certificate key CERT_KEY=$(kubeadm init phase upload-certs --upload-certs | tail -1 | tr -d '\r\n ') if [ -n "$CERT_KEY" ]; then - CONTROL_PLANE_JOIN_CMD=$(kubeadm token create --ttl {{ k8s_bootstrap_token_ttl }} --print-join-command --certificate-key "$CERT_KEY") + CONTROL_PLANE_JOIN_CMD=$(kubeadm token create --ttl 0 --print-join-command --certificate-key "$CERT_KEY") echo "$CONTROL_PLANE_JOIN_CMD" > "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" - chmod 600 "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" - echo "Saved control-plane join command." + echo "Saved control-plane join command to: ${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" else echo "ERROR: Certificate key is empty! Cannot generate control-plane join command." exit 1 fi # For joining worker nodes (regular join command) - WORKER_JOIN_CMD=$(kubeadm token create --ttl {{ k8s_bootstrap_token_ttl }} --print-join-command | sed 's/[[:space:]]*$//') + WORKER_JOIN_CMD=$(kubeadm token create --ttl 0 --print-join-command) echo "$WORKER_JOIN_CMD" > "${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" - chmod 600 "${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" - echo "Saved worker join command." + echo "Saved worker join command to: ${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" export KUBECONFIG=/etc/kubernetes/admin.conf @@ -896,6 +886,9 @@ --namespace default --create-namespace \ --set nfs.server={{ k8s_nfs_server_path.split(':')[0] }} \ --set nfs.path={{ k8s_nfs_server_path.split(':')[1] }} \ +{% if k8s_nfs_custom_mnt_opts | default('') | length > 0 %} + --set "nfs.mountOptions={{ '{' }}{{ k8s_nfs_custom_mnt_opts }}{{ '}' }}" \ +{% endif %} --set storageClass.defaultClass=true \ --set storageClass.reclaimPolicy=Retain echo "Waiting for nfs-subdir-external-provisioner pods to appear..." @@ -914,25 +907,18 @@ echo "Installing Metallb" kubectl create -f {{ k8s_client_mount_path }}/metallb/{{ metallb_package }}.yml echo "Waiting for MetalLB pods to be ready..." - kubectl wait --namespace metallb-system --for=condition=Ready pods --all --timeout=300s || echo "WARNING: Some MetalLB pods are not Ready yet. Continuing initialization." + kubectl wait --namespace metallb-system --for=condition=Ready pods --all --timeout=300s echo "Waiting for MetalLB webhook to be ready..." - METALLB_WEBHOOK_ATTEMPTS=0 - METALLB_WEBHOOK_MAX=120 until kubectl get endpoints metallb-webhook-service -n metallb-system \ -o jsonpath='{.subsets[*].addresses[*].ip}' | grep -qE '[0-9]'; do - METALLB_WEBHOOK_ATTEMPTS=$((METALLB_WEBHOOK_ATTEMPTS + 1)) - if [ "$METALLB_WEBHOOK_ATTEMPTS" -ge "$METALLB_WEBHOOK_MAX" ]; then - echo "WARNING: MetalLB webhook not ready after ${METALLB_WEBHOOK_MAX} attempts. Continuing." - break - fi - echo "Webhook endpoints not ready yet (attempt ${METALLB_WEBHOOK_ATTEMPTS}/${METALLB_WEBHOOK_MAX}). Retrying in 5s..." + echo "Webhook endpoints not ready yet. Retrying in 5s..." sleep 5 done - echo "MetalLB webhook wait completed." + echo "MetalLB webhook is ready." echo "Deploy ipaddress pool" - kubectl create -f {{ k8s_client_mount_path }}/metallb/ipaddress_pool.yaml || echo "WARNING: Failed to create IP address pool. Continuing." + kubectl create -f {{ k8s_client_mount_path }}/metallb/ipaddress_pool.yaml echo "Deploy Layer2 Configuration" - kubectl create -f {{ k8s_client_mount_path }}/metallb/l2advertisement.yaml || echo "WARNING: Failed to create L2 advertisement. Continuing." + kubectl create -f {{ k8s_client_mount_path }}/metallb/l2advertisement.yaml #echo "Deploy Multus" #kubectl apply -f {{ k8s_client_mount_path }}/multus/{{ multus_package }}.yml @@ -959,7 +945,7 @@ kubectl rollout restart deployment coredns -n kube-system echo "Waiting for coredns pods to appear.." sleep 30 - kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=300s || echo "WARNING: CoreDNS pods not fully ready. Continuing." + kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=300s if [ "{{ hostvars['localhost'].get('enable_powerscale_csi', false) | bool | lower }}" == "true" ]; then /usr/local/bin/deploy-powerscale-csi.sh @@ -972,7 +958,7 @@ kubectl rollout restart deployment coredns -n kube-system sleep 30 echo "Waiting for coredns pods to appear.." - kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=300s || echo "WARNING: CoreDNS pods not fully ready. Continuing." + kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=300s # Mark initialization complete so all of above is skipped on reboot! touch "$MARKER" echo "Cloud-Init has completed successfully." diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_x86_64.yaml.j2 index 6e3c3eb13a..bfc65d7834 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_x86_64.yaml.j2 @@ -302,12 +302,12 @@ {% if etcd_on_local_disk | default(false) %} - /usr/local/bin/etcd-fstab-update.sh {% else %} - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/etcd /var/lib/etcd nfs noatime,nolock 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/etcd /var/lib/etcd nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab {% endif %} - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubelet /var/lib/kubelet nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubernetes /etc/kubernetes nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/pod-logs /var/log/pods nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/packages /var/lib/packages nfs noatime,nolock 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubelet /var/lib/kubelet nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubernetes /etc/kubernetes nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/pod-logs /var/log/pods nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/packages /var/lib/packages nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - echo "tmpfs /tmp/crio-storage tmpfs size={{ k8s_crio_storage_size }},noatime,nodev,nosuid 0 0" >> /etc/fstab - sudo swapoff -a - sudo sed -i '/ swap / s/^/#/' /etc/fstab @@ -494,29 +494,9 @@ sleep 10 done echo "kube-vip (${KUBE_VIP}) is reachable. Joining this service_kube_control_plane_x86_64 to the cluster now." - - # Validate file exists and is non-empty - if [ ! -s "$JOIN_CMD_FILE" ]; then - echo "ERROR: Join command file is missing or empty: $JOIN_CMD_FILE" - exit 1 - fi - - # Validate file contains exactly one line - if [ "$(grep -c '' "$JOIN_CMD_FILE")" -ne 1 ]; then - echo "ERROR: Join command file must contain exactly one line." - exit 1 - fi - - # Validate content matches expected kubeadm join format - JOIN_CMD_CONTENT=$(cat "$JOIN_CMD_FILE") - if ! echo "$JOIN_CMD_CONTENT" | grep -qE '^kubeadm join ([0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]{1,5} --token [a-z0-9]{6}\.[a-z0-9]{16} --discovery-token-ca-cert-hash sha256:[a-f0-9]{64} --control-plane --certificate-key [a-f0-9]{64}$'; then - echo "ERROR: Join command file does not match expected kubeadm join format." - echo "Content: $JOIN_CMD_CONTENT" - exit 1 - fi - - echo "Executing join command from: $JOIN_CMD_FILE" - bash -c "$JOIN_CMD_CONTENT --node-name ${NODE_NAME} --apiserver-advertise-address ${NODE_NAME}" + JOIN_CMD="$(cat "$JOIN_CMD_FILE") --node-name ${NODE_NAME} --apiserver-advertise-address ${NODE_NAME}" + echo "Executing: $JOIN_CMD" + eval $JOIN_CMD mkdir -p /root/.kube cp -f /etc/kubernetes/admin.conf /root/.kube/config chown root:root /root/.kube/config diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_node_x86_64.yaml.j2 index e90fa38dc5..e5da922c55 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_node_x86_64.yaml.j2 @@ -196,10 +196,10 @@ {% endfor %} # K8s NFS mount entries - mkdir -pv /tmp/crio-storage {{ k8s_client_mount_path }} /var/lib/kubelet /etc/kubernetes /var/log/pods /var/lib/packages - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubelet /var/lib/kubelet nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubernetes /etc/kubernetes nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/pod-logs /var/log/pods nfs noatime,nolock 0 0" >> /etc/fstab - - echo "{{ k8s_nfs_server_path }}/packages /var/lib/packages nfs noatime,nolock 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubelet /var/lib/kubelet nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/kubernetes /etc/kubernetes nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/{{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}/pod-logs /var/log/pods nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ k8s_nfs_server_path }}/packages /var/lib/packages nfs {{ k8s_nfs_mnt_opts | default(k8s_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - echo "tmpfs /tmp/crio-storage tmpfs size={{ k8s_crio_storage_size }},noatime,nodev,nosuid 0 0" >> /etc/fstab - sudo swapoff -a - sudo sed -i '/ swap / s/^/#/' /etc/fstab @@ -358,29 +358,9 @@ sleep 10 done echo "kube-vip (${KUBE_VIP}) is reachable. Joining this service_kube_node_x86_64 to the cluster now." - - # Validate file exists and is non-empty - if [ ! -s "$JOIN_CMD_FILE" ]; then - echo "ERROR: Join command file is missing or empty: $JOIN_CMD_FILE" - exit 1 - fi - - # Validate file contains exactly one line - if [ "$(grep -c '' "$JOIN_CMD_FILE")" -ne 1 ]; then - echo "ERROR: Join command file must contain exactly one line." - exit 1 - fi - - # Validate content matches expected kubeadm join format - JOIN_CMD_CONTENT=$(cat "$JOIN_CMD_FILE") - if ! echo "$JOIN_CMD_CONTENT" | grep -qE '^kubeadm join ([0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]{1,5} --token [a-z0-9]{6}\.[a-z0-9]{16} --discovery-token-ca-cert-hash sha256:[a-f0-9]{64}$'; then - echo "ERROR: Join command file does not match expected kubeadm join format." - echo "Content: $JOIN_CMD_CONTENT" - exit 1 - fi - - echo "Executing join command from: $JOIN_CMD_FILE" - bash -c "$JOIN_CMD_CONTENT --node-name ${NODE_NAME}" + JOIN_CMD="$(cat "$JOIN_CMD_FILE") --node-name ${NODE_NAME}" + echo "Executing: $JOIN_CMD" + eval $JOIN_CMD sleep 30 echo "Updating the kubelet arguments." sed -i 's/^shutdownGracePeriod:.*/shutdownGracePeriod: 30s/' /var/lib/kubelet/config.yaml diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_control_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_control_node_x86_64.yaml.j2 index 5513c0a3de..685038695e 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_control_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_control_node_x86_64.yaml.j2 @@ -107,7 +107,7 @@ {{ lookup('template', 'templates/openldap/update_ldap_conf.sh.j2') | indent(12) }} {% endif %} -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -119,11 +119,9 @@ - path: /etc/hosts append: true content: | - # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} - # omnia-inventory-end {% endif %} - path: /root/init_slurm_db.sql @@ -292,7 +290,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -326,7 +324,7 @@ # DOCA prerequisites - moved early to ensure RDMA is ready before vendor_data mounts - mkdir -p {{ client_mount_path }}/slurm/ssh - mkdir -p {{ slurm_ctld_log_dir_effective }} {{ slurmdbd_log_dir_effective }} {{ slurm_ctld_pid_dir_effective }} {{ slurmdbd_pid_dir_effective }} {{ slurm_state_save_location_effective }} {% if slurm_sched_log_dir_effective %}{{ slurm_sched_log_dir_effective }} {% endif %}/etc/slurm {{ home_dir }} /etc/my.cnf.d /etc/munge /var/lib/mysql /var/log/mariadb /cert /var/log/track /var/lib/packages - - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mount -av - cp /cert/pulp_webserver.crt /etc/pki/ca-trust/source/anchors && update-ca-trust - sed -i 's/^gpgcheck=1/gpgcheck=0/' /etc/dnf/dnf.conf @@ -351,19 +349,19 @@ - mkdir -pv {{ client_mount_path }}/slurm/ssh # Create directories for nfs and mount all - mkdir -p {{ slurm_ctld_log_dir_effective }} {{ slurmdbd_log_dir_effective }} {{ slurm_ctld_pid_dir_effective }} {{ slurmdbd_pid_dir_effective }} {{ slurm_state_save_location_effective }} {% if slurm_sched_log_dir_effective %}{{ slurm_sched_log_dir_effective }} {% endif %}/etc/slurm {{ home_dir }} /etc/my.cnf.d /etc/munge /var/lib/mysql /var/log/mariadb /cert /var/log/track /var/lib/packages - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm /etc/slurm nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/my.cnf.d /etc/my.cnf.d nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/mariadb /var/log/mariadb nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_ctld_log_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm /etc/slurm nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/my.cnf.d /etc/my.cnf.d nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/mariadb /var/log/mariadb nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_ctld_log_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab {% if slurmdbd_log_dir_effective != slurm_ctld_log_dir_effective %} - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurmdbd_log_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurmdbd_log_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab {% endif %} - - if ! mountpoint -q /var/lib/mysql; then echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/lib/mysql /var/lib/mysql nfs defaults,_netdev 0 0" >> /etc/fstab; fi - - if ! mountpoint -q {{ slurm_state_save_location_effective }}; then echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmctld {{ slurm_state_save_location_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab; fi - - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ trackfile_nfs_path }} /var/log/track nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs defaults,_netdev 0 0" >> /etc/fstab - - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs defaults,_netdev 0 0" >> /etc/fstab + - if ! mountpoint -q /var/lib/mysql; then echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/lib/mysql /var/lib/mysql nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab; fi + - if ! mountpoint -q {{ slurm_state_save_location_effective }}; then echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmctld {{ slurm_state_save_location_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab; fi + - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ trackfile_nfs_path }} /var/log/track nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab - mount -av - chown -R {{ slurm_user }}:{{ slurm_user }} {{ home_dir }} @@ -392,9 +390,9 @@ {% if hostvars['localhost']['openldap_support'] %} - /usr/local/bin/update_ldap_conf.sh - mkdir /ldapcerts - - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab # TODO: This mount may be redundant, remove this - - echo "{{ metadata_svc_nfs_path_openldap }}/ldapuser /home nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path_openldap }}/ldapuser /home nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab - mount -a - yes | cp /ldapcerts/* /etc/openldap/certs diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2 index 690423e04f..83c29f89be 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2 @@ -166,13 +166,13 @@ mkdir -pv {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages echo "[INFO] Updating /etc/fstab with NFS entries for Pulp cert, Slurm and Munge paths" - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ trackfile_nfs_path }} /var/log/track nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs defaults,_netdev 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ trackfile_nfs_path }} /var/log/track nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/cert /cert nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab chmod {{ file_mode }} /etc/fstab @@ -268,7 +268,7 @@ echo "[INFO] Mounting controller slurm.conf from NFS: {{ metadata_svc_nfs_path }}/{{ ctld_list[0] }}/etc/slurm -> $CTLD_SLURM_DIR_MNT" mkdir -p "$CTLD_SLURM_DIR_MNT" - mount -t nfs "{{ metadata_svc_nfs_path }}/{{ ctld_list[0] }}/etc/slurm" "$CTLD_SLURM_DIR_MNT" || { + mount -t nfs -o "{{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }}" "{{ metadata_svc_nfs_path }}/{{ ctld_list[0] }}/etc/slurm" "$CTLD_SLURM_DIR_MNT" || { echo "[WARN] Failed to mount controller slurm.conf directory, falling back to defaults." SRUN_RANGE="$DEFAULT_SRUN_RANGE" SLURMD_PORT="$DEFAULT_SLURMD_PORT" @@ -361,7 +361,7 @@ echo "[INFO] ===== Completed firewall and service configuration (aarch64) =====" -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -373,11 +373,9 @@ - path: /etc/hosts append: true content: | - # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} - # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -430,7 +428,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -480,7 +478,7 @@ {% endfor %} {% endif %} - mkdir -p {{ client_mount_path }}/slurm/ssh - - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab {# mount_config host_mount_map block removed - incompatible with Pongo2 validation #} {# When mount_config bolt-on is needed, implement as a shell script in write_files #} {% set fg_swap = metadata_svc_groups_dict.get(functional_group_name, {}).get('swap', {}) %} @@ -518,7 +516,7 @@ {% if hostvars['localhost']['openldap_support'] %} - /usr/local/bin/update_ldap_conf.sh - mkdir /ldapcerts - - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mkdir -pv {{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home - echo "{{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home none bind 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2 index 82b72ed928..b49afbec52 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2 @@ -148,7 +148,7 @@ {% endif %} {% endif %} -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -160,11 +160,9 @@ - path: /etc/hosts append: true content: | - # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} - # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -199,13 +197,13 @@ mkdir -pv {{ slurm_slurmd_log_dir_effective }} {{ slurm_slurmd_pid_dir_effective }} {{ slurm_slurmd_spool_dir_effective }} {{ slurm_epilog_dirs_all | join(' ') }} /etc/munge /cert /var/log/track /var/lib/packages echo "[INFO] Updating /etc/fstab with NFS entries for Pulp cert, Slurm and Munge paths" - echo "{{ metadata_svc_nfs_path }}/cert /cert nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ trackfile_nfs_path }} /var/log/track nfs defaults,_netdev 0 0" >> /etc/fstab - echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs defaults,_netdev 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/cert /cert nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/log/slurm {{ slurm_slurmd_log_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/var/spool/slurmd {{ slurm_slurmd_spool_dir_effective }} nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/slurm/epilog.d /etc/slurm/epilog.d nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/$(hostname -s)/etc/munge /etc/munge nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ trackfile_nfs_path }} /var/log/track nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab + echo "{{ metadata_svc_nfs_path }}/packages /var/lib/packages nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab chmod {{ file_mode }} /etc/fstab echo "[INFO] Mounting all NFS entries from /etc/fstab" @@ -304,7 +302,7 @@ echo "[INFO] Mounting controller slurm.conf from NFS: {{ metadata_svc_nfs_path }}/{{ ctld_list[0] }}/etc/slurm -> $CTLD_SLURM_DIR_MNT" mkdir -p "$CTLD_SLURM_DIR_MNT" - mount -t nfs "{{ metadata_svc_nfs_path }}/{{ ctld_list[0] }}/etc/slurm" "$CTLD_SLURM_DIR_MNT" || { + mount -t nfs -o "{{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }}" "{{ metadata_svc_nfs_path }}/{{ ctld_list[0] }}/etc/slurm" "$CTLD_SLURM_DIR_MNT" || { echo "[WARN] Failed to mount controller slurm.conf directory, falling back to defaults." SRUN_RANGE="$DEFAULT_SRUN_RANGE" SLURMD_PORT="$DEFAULT_SLURMD_PORT" @@ -438,7 +436,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(true) | bool %} +{% if dns_enabled | default(false) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -488,7 +486,7 @@ {% endfor %} {% endif %} - mkdir -p {{ client_mount_path }}/slurm/ssh - - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path }}/ssh {{ client_mount_path }}/slurm/ssh nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab {# mount_config host_mount_map block removed - incompatible with Pongo2 validation #} {# When mount_config bolt-on is needed, implement as a shell script in write_files #} {% set fg_swap = metadata_svc_groups_dict.get(functional_group_name, {}).get('swap', {}) %} @@ -524,7 +522,7 @@ {% if hostvars['localhost']['openldap_support'] %} - /usr/local/bin/update_ldap_conf.sh - mkdir -p /ldapcerts || echo "LDAP cert directory already exists or creation failed (non-critical)" - - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs defaults,_netdev 0 0" >> /etc/fstab + - echo "{{ metadata_svc_nfs_path_openldap }}/certs /ldapcerts nfs {{ slurm_nfs_mnt_opts | default(slurm_nfs_default_mnt_opts) }} 0 0" >> /etc/fstab - mkdir -pv {{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home - echo "{{ vast_mount_point }}/{{ openldap_dir_name }}/ldapuser /home none bind 0 0" >> /etc/fstab - chmod {{ file_mode }} /etc/fstab diff --git a/src/orchestrator/roles/provision_common/vars/main.yml b/src/orchestrator/roles/provision_common/vars/main.yml index 3b637a2112..97052f9909 100644 --- a/src/orchestrator/roles/provision_common/vars/main.yml +++ b/src/orchestrator/roles/provision_common/vars/main.yml @@ -36,6 +36,11 @@ openchami_hostname_vars_path: "{{ nodes_dir }}/hostname.yaml" ms_defaults_dest: "{{ metadata_svc_dir }}/ms-defaults.yaml" ms_group_common_dest: "{{ metadata_svc_dir }}/ms-group-common.yaml" +# Default options for Omnia-managed per-node NFS sub-mounts. +# Overridden per storage entry by node_nfs_mnt_opts in storage_config.yml. +k8s_nfs_default_mnt_opts: "noatime,nolock" +slurm_nfs_default_mnt_opts: "defaults,_netdev" + # SSH keys ssh_key_path: "{{ ansible_user_dir | default('/root') }}/.ssh/oim_rsa.pub" ssh_private_key_path: "{{ ansible_user_dir | default('/root') }}/.ssh/oim_rsa" @@ -182,117 +187,3 @@ openchami_config_missing_fail_msg: >- This file is generated during the deploy stage. Run the orchestrator deploy stage first: ansible-playbook orchestrator.yml --tags deploy smd_service_fail_msg: "smd service is not running" - -# Actionable error messages surfaced (no_log tasks). Printed with no_log: false. -register_nodes_ensure_mapping_rescue_msg: | - The PXE mapping was not loaded on the OIM for this category. - What to fix: run the stages in order, starting from validate. - Then re-run provisioning. - -register_nodes_reread_mapping_rescue_msg: | - The PXE mapping CSV could not be re-read. - What to fix: check the PXE mapping file exists and has an ADMIN_MAC column. - Then re-run provisioning. - -register_nodes_set_mapping_rescue_msg: | - The PXE mapping could not be applied for this category. - What to fix: check the PXE mapping file is valid. - Then re-run provisioning. - -register_nodes_token_refresh_rescue_msg: | - The OpenCHAMI access token could not be refreshed. - What to fix: confirm the OpenCHAMI target is running and the token helper works on the OIM. - Then re-run provisioning. - -register_nodes_update_auth_rescue_msg: | - The refreshed OpenCHAMI token could not be applied. - What to fix: retry the run. If it repeats, check the token helper output. - Then re-run provisioning. - -register_nodes_check_apis_rescue_msg: | - OpenCHAMI APIs are not reachable, or the CA certificate is not trusted. - What to fix: confirm the OpenCHAMI target is up and the CA certificate is in place. - Then re-run provisioning. - -register_nodes_resolve_identities_rescue_msg: | - Node identities could not be resolved in SMD for this category. - What to fix: confirm SMD is running and the nodes are present in hardware inventory. - Then re-run provisioning. - -register_nodes_publish_mapping_rescue_msg: | - Resolved node identities could not be published to the provisioning mapping. - What to fix: rerun from the registration stage. - Then re-run provisioning. - -register_nodes_delete_group_rescue_msg: | - The old node_registration SMD group could not be removed. - What to fix: confirm SMD is running and reachable. - Then re-run provisioning. - -register_nodes_smd_api_ready_rescue_msg: | - The SMD API did not become ready in time. - What to fix: check the SMD service status and its logs. - Then re-run provisioning. - -register_nodes_discover_retry_rescue_msg: | - SMD rejected node discovery after the retries. - What to fix: check the generated node definitions and the SMD logs. - Then re-run provisioning. - -register_nodes_verify_components_rescue_msg: | - Target nodes did not appear in SMD in time. - What to fix: check SMD discovery status and that the nodes are reachable over the BMC network. - Then re-run provisioning. - -register_nodes_reconcile_instanceinfo_rescue_msg: | - Node hostnames could not be updated in the Metadata Service. - What to fix: check the Metadata Service status. - Then re-run provisioning. - -provision_prune_boot_config_rescue_msg: | - Stale Boot Service configurations could not be cleaned up. - What to fix: confirm the OpenCHAMI target is running. - Then re-run provisioning. - -provision_list_boot_config_rescue_msg: | - Boot Service configurations could not be read. - What to fix: confirm the OpenCHAMI target is running. - Then re-run provisioning. - -provision_read_ssh_key_rescue_msg: | - The OIM SSH public key could not be read. - What to fix: restore the key file on the OIM, or regenerate the OIM SSH key pair. - Then re-run provisioning. - -provision_read_ssh_private_key_rescue_msg: | - The OIM SSH private key could not be read. - What to fix: restore the key file on the OIM, or regenerate the OIM SSH key pair. - Then re-run provisioning. - -provision_password_salt_rescue_msg: | - The password salt could not be derived. - What to fix: confirm the project name, cluster name, and cluster domain are set. - Then re-run provisioning. - -provision_hash_password_rescue_msg: | - The provision password could not be hashed. - What to fix: confirm openssl is installed and the provision password is set. - Then re-run provisioning. - -provision_load_ms_defaults_rescue_msg: | - The metadata-service defaults file could not be read. - What to fix: re-run provisioning so the file is regenerated. - -provision_parse_ms_defaults_rescue_msg: | - The metadata-service defaults file is not valid YAML. - What to fix: re-run provisioning to regenerate it, or fix the template inputs. - -provision_reconcile_cluster_defaults_rescue_msg: | - Metadata-service cluster defaults could not be updated. - What to fix: confirm the Metadata Service is running. - Then re-run provisioning. - -provision_ci_group_common_template_rescue_msg: | - The common cloud-init group template could not be rendered. - What to fix: confirm the template exists and all values it uses are defined. - Then re-run provisioning. From d5280e431615702431c9f1f442e3ef79278e44fb Mon Sep 17 00:00:00 2001 From: Kratika_Patidar Date: Fri, 13 Nov 2026 13:16:32 +0530 Subject: [PATCH 4/4] fix(orchestrator): restore staging content overwritten by NFS mount options commit Commit c8824e19e applied the PowerScale NFS mount options feature by copying whole files from a v2.3.0.0-rc2 based tree, which unintentionally reverted staging-only changes in the same files, including: - enable_k8s_ha, virtual_ip_address and k8s_bootstrap_token_ttl inputs and schema entries in omnia_config - node_hardware_defaults maximum limits in omnia_config schema - provision_common actionable rescue messages in vars/main.yml - other staging template and input documentation updates This commit reverts c8824e19e and re-applies only the intended node_nfs_mnt_opts / csi_powerscale_mount_options changes on top of the current staging content. Related to issue #1445 Signed-off-by: Kratika_Patidar --- src/orchestrator/input/omnia_config.yml | 36 +++++- src/orchestrator/input/storage_config.yml | 18 --- .../schema/omnia_config.json | 33 ++++- .../tasks/create_k8s_config_nfs.yml | 14 ++- .../tasks/configure_metadata_svc.yml | 27 +++-- ...-group-login_compiler_node_aarch64.yaml.j2 | 23 +++- ...s-group-login_compiler_node_x86_64.yaml.j2 | 25 +++- .../ms-group-login_node_aarch64.yaml.j2 | 25 +++- .../ms-group-login_node_x86_64.yaml.j2 | 25 +++- ...ce_kube_control_plane_first_x86_64.yaml.j2 | 53 +++++--- ...-service_kube_control_plane_x86_64.yaml.j2 | 26 +++- .../ms-group-service_kube_node_x86_64.yaml.j2 | 26 +++- ...ms-group-slurm_control_node_x86_64.yaml.j2 | 6 +- .../ms-group-slurm_node_aarch64.yaml.j2 | 6 +- .../ms-group-slurm_node_x86_64.yaml.j2 | 6 +- .../roles/provision_common/vars/main.yml | 114 ++++++++++++++++++ 16 files changed, 380 insertions(+), 83 deletions(-) diff --git a/src/orchestrator/input/omnia_config.yml b/src/orchestrator/input/omnia_config.yml index 2d13cd35d5..b39044ad22 100644 --- a/src/orchestrator/input/omnia_config.yml +++ b/src/orchestrator/input/omnia_config.yml @@ -60,7 +60,7 @@ # Best for: Mixed hardware environments with different node configurations # - homogeneous: Groups nodes by hardware type for optimized discovery # Best for: Standardized hardware groups (grp0-grp100 in pxe_mapping_file.csv) -# Performance: 0 iDRAC calls (with specs) or 1 call per group (without specs) +# Performance: 0 iDRAC calls (with specs) or 1 call per group (without specs) # Default value is heterogeneous # node_hardware_defaults @@ -119,7 +119,6 @@ slurm_cluster: - cluster_name: slurm_cluster nfs_storage_name: nfs_slurm - vast_storage_name: vast_storage # Optional custom Slurm configuration. Keep skip_merge and config_sources # nested under this slurm_cluster entry. Choose one config_sources format # and remove only the first '#' from each configuration line in that example. @@ -149,7 +148,17 @@ slurm_cluster: # - cluster_name is required field # - deployment: Exactly one entry in both the service_k8s_cluster lists must have deployment set to true to indicate where Kubernetes should be deployed. -# Please ensure corresponding cluster entry is added to high_availability_config.yml if deployment is set to true. + +# ----------------------------SERVICE K8S HIGH AVAILABILITY----------------------------------- +# - enable_k8s_ha: (Mandatory on the deployed entry when Kubernetes nodes are mapped) +# Indicates whether to enable HA for the Kubernetes service node. +# Set to 'true' to enable, 'false' to disable. + +# - virtual_ip_address: (Mandatory on the deployed entry when Kubernetes nodes are mapped) +# The virtual IP address for the K8s service node setup. It is used by kube-vip +# even when enable_k8s_ha is false. +# It must sit in the control-plane subnet, and must not collide with an address +# assigned to a mapped node, a DHCP range, an OIM address, or pod_external_ip_range. # - Kubernetes SDN network.K8s_cni (Mandatory) - It can either be "calico" or "flannel".Default value assigned is "calico". # While setting up Kubernetes plugin for RoCE NIC, ensure that this value is set to "flannel" @@ -188,18 +197,37 @@ slurm_cluster: # Acceptable formats: "10G", "15G", "50G" (Only positive values in Gigabytes are allowed) # Default value is "20G" +# - k8s_bootstrap_token_ttl: Duration for which kubeadm bootstrap tokens remain valid. +# Bootstrap tokens are used by worker and control-plane nodes to authenticate when +# joining the cluster. After this duration, tokens are automatically deleted and +# operators must regenerate them by running the refresh script on the first control-plane node: +# /generate-control-plane-join.sh +# (e.g. /opt/omnia/k8s_mount/generate-control-plane-join.sh — path matches the 'nfs_k8s' storage mount point) +# Acceptable formats: "1h", "2h", "4h", "24h" (Go duration strings accepted by kubeadm) +# Default value is "2h" + +# - etcd_on_local_disk: When set to true, etcd data is stored on a dedicated local disk (BOSS or +# fallback disk) on each control-plane node, formatted as ext4 and mounted at /var/lib/etcd. +# The disk will be formatted and mounted automatically during provisioning. +# When set to false, etcd data is stored on NFS. If the control-plane node does not have a +# dedicated disk, set this to false so etcd falls back to NFS storage. +# Default value is "true" + service_k8s_cluster: - cluster_name: service_cluster deployment: true - etcd_on_local_disk: false + etcd_on_local_disk: true k8s_cni: "calico" pod_external_ip_range: "172.16.107.170-172.16.107.200" k8s_service_addresses: "10.233.0.0/18" k8s_pod_network_cidr: "10.233.64.0/18" nfs_storage_name: "nfs_k8s" k8s_crio_storage_size: "20G" + k8s_bootstrap_token_ttl: "2h" enable_powerscale_csi: false csi_powerscale_driver_secret_file_path: "" csi_powerscale_driver_values_file_path: "" csi_powerscale_mount_options: "" + enable_k8s_ha: true + virtual_ip_address: "172.16.107.1" diff --git a/src/orchestrator/input/storage_config.yml b/src/orchestrator/input/storage_config.yml index aa9b1aa17e..a5c4e9714b 100644 --- a/src/orchestrator/input/storage_config.yml +++ b/src/orchestrator/input/storage_config.yml @@ -136,24 +136,6 @@ mounts: mount_on_oim: true functional_group_prefix: ["service_kube"] - # VAST Storage - High-performance NFS for HPC tools and benchmarks - # This storage is critical for the Slurm cluster and must be accessible from OIM. - # Purpose: Serves as the centralized repository for HPC tools (UCX, OpenMPI, CUDA, benchmarks) - # that are bind-mounted to /hpc_tools on compute and login nodes. - # OIM Access: The VAST server (172.16.107.77) must be accessible from OIM to copy hpc_tools - # content during provisioning. OIM mounts this storage to populate the shared - # hpc_tools directory structure before nodes boot. - # Configuration: The slurm_cluster section in omnia_config.yml should reference this storage - # to ensure proper integration with cluster provisioning workflows. - # This standard entry is skipped when vast_storage_name is absent or empty. - # Note: Uses RDMA mount parameters for optimal performance on InfiniBand networks. - - name: "vast_storage" - source: "172.16.107.77:/share/vast" - mount_point: "/mnt/vast" - mount_params: "vast_rdma" - mount_on_oim: true - functional_group_prefix: ["slurm_node", "login"] - # -----------------------------Mount Params (Profiles)------------------------------- # mount_params: Named default profiles for mount configurations. # Profiles are referenced by name from mount entries via the mount_params field. diff --git a/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/omnia_config.json b/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/omnia_config.json index 134d467628..254efa9482 100644 --- a/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/omnia_config.json +++ b/src/orchestrator/plugins/module_utils/orchestrator_validation/schema/omnia_config.json @@ -103,19 +103,23 @@ "properties": { "sockets": { "type": "integer", - "minimum": 1 + "minimum": 1, + "maximum": 16 }, "cores_per_socket": { "type": "integer", - "minimum": 1 + "minimum": 1, + "maximum": 160 }, "threads_per_core": { "type": "integer", - "minimum": 1 + "minimum": 1, + "maximum": 2 }, "real_memory": { "type": "integer", - "minimum": 1 + "minimum": 1, + "maximum": 8589934592 }, "gres": { "type": "string", @@ -256,6 +260,12 @@ "type": "string", "pattern": "^[1-9][0-9]*G$" }, + "k8s_bootstrap_token_ttl": { + "type": "string", + "description": "Duration for kubeadm bootstrap token validity (e.g. 1h, 2h, 24h). Set to 0 to disable expiry (not recommended).", + "pattern": "^(0|([0-9]+h([0-9]+m)?([0-9]+s)?|[0-9]+m([0-9]+s)?|[0-9]+s))$", + "default": "2h" + }, "enable_powerscale_csi": { "type": "boolean", "default": false @@ -269,6 +279,21 @@ "csi_powerscale_mount_options": { "type": "string", "pattern": "^([A-Za-z0-9,=._@/-]+)?$" + }, + "enable_k8s_ha": { + "type": "boolean" + }, + "virtual_ip_address": { + "type": "string", + "description": "Kubernetes HA virtual IP. Required, non-empty, when the PXE mapping selects Kubernetes (enforced by L2 validation).", + "anyOf": [ + { + "maxLength": 0 + }, + { + "format": "ipv4" + } + ] } }, "allOf": [ diff --git a/src/orchestrator/roles/k8s_config/tasks/create_k8s_config_nfs.yml b/src/orchestrator/roles/k8s_config/tasks/create_k8s_config_nfs.yml index 675c5101ee..867a91d287 100644 --- a/src/orchestrator/roles/k8s_config/tasks/create_k8s_config_nfs.yml +++ b/src/orchestrator/roles/k8s_config/tasks/create_k8s_config_nfs.yml @@ -18,9 +18,6 @@ - name: Include storage vars ansible.builtin.include_vars: "{{ input_project_dir }}/storage_config.yml" -- name: Include include_high_availability_config vars - ansible.builtin.include_vars: "{{ ha_config_file }}" - - name: Identify deployed service Kubernetes configuration ansible.builtin.set_fact: deployed_service_k8s_cluster_matches: >- @@ -33,7 +30,7 @@ ansible.builtin.set_fact: deployed_service_k8s_cluster: >- {{ deployed_service_k8s_cluster_matches | first - | default(service_k8s_cluster | default([]) | first | default({})) }} + | default({}) }} - name: Set facts for service_k8s ansible.builtin.set_fact: @@ -140,6 +137,7 @@ community.general.read_csv: path: "{{ hostvars['localhost']['pxe_mapping_file_path'] }}" register: pxe_data_for_cidr + no_log: true delegate_to: localhost connection: local run_once: true @@ -183,8 +181,12 @@ - name: Set HA-related facts ansible.builtin.set_fact: - enable_k8s_ha: "{{ service_k8s_cluster_ha[0].enable_k8s_ha | default(false) }}" - kube_vip: "{{ service_k8s_cluster_ha[0].virtual_ip_address | default('') }}" + enable_k8s_ha: >- + {{ hostvars['localhost']['service_k8s_deployed_cluster'] + .enable_k8s_ha | default(false) }} + kube_vip: >- + {{ hostvars['localhost']['service_k8s_deployed_cluster'] + .virtual_ip_address | default('') }} - name: Set fact for pulp mirror ansible.builtin.set_fact: diff --git a/src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml b/src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml index b83c2eac0f..c9e22efb20 100644 --- a/src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml +++ b/src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml @@ -139,12 +139,6 @@ # metadata-service templates need them during provision_common which # runs before k8s_config. Guard all tasks with target_category check. -- name: Load high availability config for kube_vip - ansible.builtin.include_vars: - file: "{{ input_project_dir }}/high_availability_config.yml" - failed_when: false - when: target_category | default('') == 'kubernetes' - - name: Load orchestrator_config for dns_enabled when: - target_category | default('') == 'kubernetes' @@ -157,7 +151,7 @@ - name: Set dns_enabled fact ansible.builtin.set_fact: - dns_enabled: "{{ _orch_config_ms.dns_enabled | default(false) | bool }}" + dns_enabled: "{{ _orch_config_ms.dns_enabled | default(true) | bool }}" - name: Select deployed service Kubernetes configuration ansible.builtin.set_fact: @@ -166,7 +160,7 @@ | selectattr('deployment', 'defined') | selectattr('deployment', 'sameas', true) | first - | default(service_k8s_cluster | default([]) | first | default({})) }} + | default({}) }} when: target_category | default('') == 'kubernetes' - name: Read version-qualified K8s repository base URLs @@ -187,14 +181,17 @@ quiet: true when: target_category | default('') == 'kubernetes' -- name: Set K8s facts from omnia_config and HA config +- name: Set K8s facts from effective deployed cluster configuration ansible.builtin.set_fact: - kube_vip: "{{ service_k8s_cluster_ha[0].virtual_ip_address | default('') }}" + kube_vip: >- + {{ hostvars['localhost']['service_k8s_deployed_cluster'] + .virtual_ip_address | default('') }} service_k8s_version: "{{ hostvars['localhost']['service_k8s_version'] }}" k8s_pod_network_cidr: "{{ metadata_service_k8s_cluster.k8s_pod_network_cidr }}" k8s_service_addresses: "{{ metadata_service_k8s_cluster.k8s_service_addresses }}" pod_external_ip_range: "{{ metadata_service_k8s_cluster.pod_external_ip_range }}" k8s_crio_storage_size: "{{ metadata_service_k8s_cluster.k8s_crio_storage_size }}" + k8s_bootstrap_token_ttl: "{{ metadata_service_k8s_cluster.k8s_bootstrap_token_ttl | default('2h') }}" etcd_on_local_disk: "{{ metadata_service_k8s_cluster.etcd_on_local_disk | default(false) }}" admin_netmask_bits: "{{ hostvars['localhost']['admin_netmask_bits'] }}" pulp_mirror: "{{ hostvars['localhost']['admin_nic_ip'] }}:{{ hostvars['localhost']['pulp_port'] | default(2225) }}" @@ -458,3 +455,13 @@ - >- ((_metadata_fg_reconcile.created | default([])) + (_metadata_fg_reconcile.updated | default([]))) | length > 0 + +- name: Track hosts whose per-host metadata changed for {{ fg_name }} + ansible.builtin.set_fact: + omnia_metadata_changed_hostnames: >- + {{ ((omnia_metadata_changed_hostnames | default([])) + + (_metadata_fg_reconcile.inventory_updated | default({}) + | dict2items | map(attribute='value') | flatten | list)) + | unique | list }} + when: + - _metadata_fg_reconcile.inventory_updated | default({}) | length > 0 diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2 index 81f67aafaa..4df9e48c69 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2 @@ -137,7 +137,7 @@ content: | {{ lookup('template', 'templates/hpc_tools/setup_doca_mpi_env.sh.j2') | indent(12) }} -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -149,9 +149,11 @@ - path: /etc/hosts append: true content: | + # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} + # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -200,7 +202,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -353,8 +355,25 @@ PXE_SUBNET="$NETWORK_IP/$NETMASK_BITS" echo "[INFO] Admin IP: $ADMIN_IP, Netmask: /$NETMASK_BITS, PXE Subnet: $PXE_SUBNET" firewall-cmd --zone=trusted --add-source="$PXE_SUBNET" --permanent +{% for extra in hostvars['localhost']['admin_additional_subnets'] | default([]) %} + echo "[INFO] Adding additional admin subnet {{ extra.subnet }}/{{ extra.netmask_bits }} to trusted zone" + firewall-cmd --zone=trusted --add-source="{{ extra.subnet }}/{{ extra.netmask_bits }}" --permanent +{% endfor %} ' + # Configure MPI OOB port range to match firewall-allowed SrunPortRange. + # Uses /etc/profile.d/ environment variables so the setting is picked up + # regardless of the OpenMPI installation prefix. + # OMPI_MCA_ — read by OpenMPI 4.x (DOCA / HPC-X) + # PRTE_MCA_ — read by PRRTE (OpenMPI 5.x runtime) + - | + cat > /etc/profile.d/omnia_mpi_oob.sh <<'MPIEOF' + # Restrict MPI OOB TCP to the Slurm SrunPortRange (firewall-allowed) + export OMPI_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" + export PRTE_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" + MPIEOF + - chmod 0644 /etc/profile.d/omnia_mpi_oob.sh + - firewall-cmd --reload - systemctl enable sshd - systemctl start sshd diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2 index bd33af81ce..61ab5f2718 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2 @@ -137,7 +137,7 @@ {{ lookup('template', 'templates/hpc_tools/setup_doca_mpi_env.sh.j2') | indent(12) }} -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -149,9 +149,11 @@ - path: /etc/hosts append: true content: | + # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} + # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -200,7 +202,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -353,8 +355,25 @@ PXE_SUBNET="$NETWORK_IP/$NETMASK_BITS" echo "[INFO] Admin IP: $ADMIN_IP, Netmask: /$NETMASK_BITS, PXE Subnet: $PXE_SUBNET" firewall-cmd --zone=trusted --add-source="$PXE_SUBNET" --permanent +{% for extra in hostvars['localhost']['admin_additional_subnets'] | default([]) %} + echo "[INFO] Adding additional admin subnet {{ extra.subnet }}/{{ extra.netmask_bits }} to trusted zone" + firewall-cmd --zone=trusted --add-source="{{ extra.subnet }}/{{ extra.netmask_bits }}" --permanent +{% endfor %} ' - + + # Configure MPI OOB port range to match firewall-allowed SrunPortRange. + # Uses /etc/profile.d/ environment variables so the setting is picked up + # regardless of the OpenMPI installation prefix. + # OMPI_MCA_ — read by OpenMPI 4.x (DOCA / HPC-X) + # PRTE_MCA_ — read by PRRTE (OpenMPI 5.x runtime) + - | + cat > /etc/profile.d/omnia_mpi_oob.sh <<'MPIEOF' + # Restrict MPI OOB TCP to the Slurm SrunPortRange (firewall-allowed) + export OMPI_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" + export PRTE_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" + MPIEOF + - chmod 0644 /etc/profile.d/omnia_mpi_oob.sh + - firewall-cmd --reload - systemctl enable sshd - systemctl start sshd diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_aarch64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_aarch64.yaml.j2 index 548936ce01..a601d80ad8 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_aarch64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_aarch64.yaml.j2 @@ -101,7 +101,7 @@ {{ lookup('template', 'templates/openldap/update_ldap_conf.sh.j2') | indent(12) }} {% endif %} -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -113,9 +113,11 @@ - path: /etc/hosts append: true content: | + # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} + # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -156,7 +158,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -290,8 +292,25 @@ PXE_SUBNET="$NETWORK_IP/$NETMASK_BITS" echo "[INFO] Admin IP: $ADMIN_IP, Netmask: /$NETMASK_BITS, PXE Subnet: $PXE_SUBNET" firewall-cmd --zone=trusted --add-source="$PXE_SUBNET" --permanent +{% for extra in hostvars['localhost']['admin_additional_subnets'] | default([]) %} + echo "[INFO] Adding additional admin subnet {{ extra.subnet }}/{{ extra.netmask_bits }} to trusted zone" + firewall-cmd --zone=trusted --add-source="{{ extra.subnet }}/{{ extra.netmask_bits }}" --permanent +{% endfor %} ' - + + # Configure MPI OOB port range to match firewall-allowed SrunPortRange. + # Uses /etc/profile.d/ environment variables so the setting is picked up + # regardless of the OpenMPI installation prefix. + # OMPI_MCA_ — read by OpenMPI 4.x (DOCA / HPC-X) + # PRTE_MCA_ — read by PRRTE (OpenMPI 5.x runtime) + - | + cat > /etc/profile.d/omnia_mpi_oob.sh <<'MPIEOF' + # Restrict MPI OOB TCP to the Slurm SrunPortRange (firewall-allowed) + export OMPI_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" + export PRTE_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" + MPIEOF + - chmod 0644 /etc/profile.d/omnia_mpi_oob.sh + - firewall-cmd --reload - systemctl enable sshd - systemctl start sshd diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_x86_64.yaml.j2 index 33750496de..3b6412aeb1 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_node_x86_64.yaml.j2 @@ -100,7 +100,7 @@ content: | {{ lookup('template', 'templates/openldap/update_ldap_conf.sh.j2') | indent(12) }} {% endif %} -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -112,9 +112,11 @@ - path: /etc/hosts append: true content: | + # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} + # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -155,7 +157,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) @@ -288,8 +290,25 @@ PXE_SUBNET="$NETWORK_IP/$NETMASK_BITS" echo "[INFO] Admin IP: $ADMIN_IP, Netmask: /$NETMASK_BITS, PXE Subnet: $PXE_SUBNET" firewall-cmd --zone=trusted --add-source="$PXE_SUBNET" --permanent +{% for extra in hostvars['localhost']['admin_additional_subnets'] | default([]) %} + echo "[INFO] Adding additional admin subnet {{ extra.subnet }}/{{ extra.netmask_bits }} to trusted zone" + firewall-cmd --zone=trusted --add-source="{{ extra.subnet }}/{{ extra.netmask_bits }}" --permanent +{% endfor %} ' - + + # Configure MPI OOB port range to match firewall-allowed SrunPortRange. + # Uses /etc/profile.d/ environment variables so the setting is picked up + # regardless of the OpenMPI installation prefix. + # OMPI_MCA_ — read by OpenMPI 4.x (DOCA / HPC-X) + # PRTE_MCA_ — read by PRRTE (OpenMPI 5.x runtime) + - | + cat > /etc/profile.d/omnia_mpi_oob.sh <<'MPIEOF' + # Restrict MPI OOB TCP to the Slurm SrunPortRange (firewall-allowed) + export OMPI_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" + export PRTE_MCA_oob_tcp_dynamic_ipv4_ports="{{ slurm_conf_dict.SrunPortRange }}" + MPIEOF + - chmod 0644 /etc/profile.d/omnia_mpi_oob.sh + - firewall-cmd --reload - systemctl enable sshd - systemctl start sshd diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_first_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_first_x86_64.yaml.j2 index 939450c968..dcde98249d 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_first_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_first_x86_64.yaml.j2 @@ -312,22 +312,30 @@ content: | #!/bin/bash set -e - # Shared mount path where the control-plane join command will be saved + # Shared mount path where the join commands will be saved K8S_CLIENT_MOUNT_PATH="{{ k8s_client_mount_path }}" mkdir -p "$K8S_CLIENT_MOUNT_PATH" - echo "Generating Kubernetes control-plane join command..." + echo "Generating Kubernetes join commands..." + # Generate certificate key and control plane join command CERT_KEY=$(kubeadm init phase upload-certs --upload-certs | tail -1 | tr -d '\r\n ') if [ -n "$CERT_KEY" ]; then - CONTROL_PLANE_JOIN_CMD=$(kubeadm token create --ttl 0 --print-join-command --certificate-key "$CERT_KEY") + CONTROL_PLANE_JOIN_CMD=$(kubeadm token create --ttl {{ k8s_bootstrap_token_ttl }} --print-join-command --certificate-key "$CERT_KEY") echo "$CONTROL_PLANE_JOIN_CMD" > "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" - chmod 644 "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" - echo "Saved control-plane join command to: ${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" + chmod 600 "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" + echo "Saved control-plane join command." else echo "ERROR: Certificate key is empty! Cannot generate control-plane join command." exit 1 fi - echo "Control-plane join script is ready. You can rerun this script anytime to refresh the command." + + # Worker join command — always refresh alongside control-plane + WORKER_JOIN_CMD=$(kubeadm token create --ttl {{ k8s_bootstrap_token_ttl }} --print-join-command | sed 's/[[:space:]]*$//') + echo "$WORKER_JOIN_CMD" > "${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" + chmod 600 "${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" + echo "Saved worker join command." + + echo "All join commands refreshed. Nodes may now join the cluster." - path: /usr/local/bin/install-helm.sh permissions: '0755' @@ -736,7 +744,7 @@ # Patch: append nameservers after /etc/resolv.conf using Jinja list "dns" sed -i 's|/etc/resolv.conf|/etc/resolv.conf{% for ns in dns | default([], true) %} {{ ns }}{% endfor %}|' "$cfg" -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} # Forward cluster-internal DNS domain to OIM CoreDNS # This allows K8s pods to resolve Slurm/MPI hostnames via CoreDNS python3 - "$cfg" << 'PYEOF' @@ -835,18 +843,20 @@ # Get the certificate key CERT_KEY=$(kubeadm init phase upload-certs --upload-certs | tail -1 | tr -d '\r\n ') if [ -n "$CERT_KEY" ]; then - CONTROL_PLANE_JOIN_CMD=$(kubeadm token create --ttl 0 --print-join-command --certificate-key "$CERT_KEY") + CONTROL_PLANE_JOIN_CMD=$(kubeadm token create --ttl {{ k8s_bootstrap_token_ttl }} --print-join-command --certificate-key "$CERT_KEY") echo "$CONTROL_PLANE_JOIN_CMD" > "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" - echo "Saved control-plane join command to: ${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" + chmod 600 "${K8S_CLIENT_MOUNT_PATH}/control-plane-join-command.sh" + echo "Saved control-plane join command." else echo "ERROR: Certificate key is empty! Cannot generate control-plane join command." exit 1 fi # For joining worker nodes (regular join command) - WORKER_JOIN_CMD=$(kubeadm token create --ttl 0 --print-join-command) + WORKER_JOIN_CMD=$(kubeadm token create --ttl {{ k8s_bootstrap_token_ttl }} --print-join-command | sed 's/[[:space:]]*$//') echo "$WORKER_JOIN_CMD" > "${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" - echo "Saved worker join command to: ${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" + chmod 600 "${K8S_CLIENT_MOUNT_PATH}/worker-join-command.sh" + echo "Saved worker join command." export KUBECONFIG=/etc/kubernetes/admin.conf @@ -907,18 +917,25 @@ echo "Installing Metallb" kubectl create -f {{ k8s_client_mount_path }}/metallb/{{ metallb_package }}.yml echo "Waiting for MetalLB pods to be ready..." - kubectl wait --namespace metallb-system --for=condition=Ready pods --all --timeout=300s + kubectl wait --namespace metallb-system --for=condition=Ready pods --all --timeout=300s || echo "WARNING: Some MetalLB pods are not Ready yet. Continuing initialization." echo "Waiting for MetalLB webhook to be ready..." + METALLB_WEBHOOK_ATTEMPTS=0 + METALLB_WEBHOOK_MAX=120 until kubectl get endpoints metallb-webhook-service -n metallb-system \ -o jsonpath='{.subsets[*].addresses[*].ip}' | grep -qE '[0-9]'; do - echo "Webhook endpoints not ready yet. Retrying in 5s..." + METALLB_WEBHOOK_ATTEMPTS=$((METALLB_WEBHOOK_ATTEMPTS + 1)) + if [ "$METALLB_WEBHOOK_ATTEMPTS" -ge "$METALLB_WEBHOOK_MAX" ]; then + echo "WARNING: MetalLB webhook not ready after ${METALLB_WEBHOOK_MAX} attempts. Continuing." + break + fi + echo "Webhook endpoints not ready yet (attempt ${METALLB_WEBHOOK_ATTEMPTS}/${METALLB_WEBHOOK_MAX}). Retrying in 5s..." sleep 5 done - echo "MetalLB webhook is ready." + echo "MetalLB webhook wait completed." echo "Deploy ipaddress pool" - kubectl create -f {{ k8s_client_mount_path }}/metallb/ipaddress_pool.yaml + kubectl create -f {{ k8s_client_mount_path }}/metallb/ipaddress_pool.yaml || echo "WARNING: Failed to create IP address pool. Continuing." echo "Deploy Layer2 Configuration" - kubectl create -f {{ k8s_client_mount_path }}/metallb/l2advertisement.yaml + kubectl create -f {{ k8s_client_mount_path }}/metallb/l2advertisement.yaml || echo "WARNING: Failed to create L2 advertisement. Continuing." #echo "Deploy Multus" #kubectl apply -f {{ k8s_client_mount_path }}/multus/{{ multus_package }}.yml @@ -945,7 +962,7 @@ kubectl rollout restart deployment coredns -n kube-system echo "Waiting for coredns pods to appear.." sleep 30 - kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=300s + kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=300s || echo "WARNING: CoreDNS pods not fully ready. Continuing." if [ "{{ hostvars['localhost'].get('enable_powerscale_csi', false) | bool | lower }}" == "true" ]; then /usr/local/bin/deploy-powerscale-csi.sh @@ -958,7 +975,7 @@ kubectl rollout restart deployment coredns -n kube-system sleep 30 echo "Waiting for coredns pods to appear.." - kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=300s + kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=300s || echo "WARNING: CoreDNS pods not fully ready. Continuing." # Mark initialization complete so all of above is skipped on reboot! touch "$MARKER" echo "Cloud-Init has completed successfully." diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_x86_64.yaml.j2 index bfc65d7834..3dc9db97d7 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_control_plane_x86_64.yaml.j2 @@ -494,9 +494,29 @@ sleep 10 done echo "kube-vip (${KUBE_VIP}) is reachable. Joining this service_kube_control_plane_x86_64 to the cluster now." - JOIN_CMD="$(cat "$JOIN_CMD_FILE") --node-name ${NODE_NAME} --apiserver-advertise-address ${NODE_NAME}" - echo "Executing: $JOIN_CMD" - eval $JOIN_CMD + + # Validate file exists and is non-empty + if [ ! -s "$JOIN_CMD_FILE" ]; then + echo "ERROR: Join command file is missing or empty: $JOIN_CMD_FILE" + exit 1 + fi + + # Validate file contains exactly one line + if [ "$(grep -c '' "$JOIN_CMD_FILE")" -ne 1 ]; then + echo "ERROR: Join command file must contain exactly one line." + exit 1 + fi + + # Validate content matches expected kubeadm join format + JOIN_CMD_CONTENT=$(cat "$JOIN_CMD_FILE") + if ! echo "$JOIN_CMD_CONTENT" | grep -qE '^kubeadm join ([0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]{1,5} --token [a-z0-9]{6}\.[a-z0-9]{16} --discovery-token-ca-cert-hash sha256:[a-f0-9]{64} --control-plane --certificate-key [a-f0-9]{64}$'; then + echo "ERROR: Join command file does not match expected kubeadm join format." + echo "Content: $JOIN_CMD_CONTENT" + exit 1 + fi + + echo "Executing join command from: $JOIN_CMD_FILE" + bash -c "$JOIN_CMD_CONTENT --node-name ${NODE_NAME} --apiserver-advertise-address ${NODE_NAME}" mkdir -p /root/.kube cp -f /etc/kubernetes/admin.conf /root/.kube/config chown root:root /root/.kube/config diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_node_x86_64.yaml.j2 index e5da922c55..4c0369e398 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-service_kube_node_x86_64.yaml.j2 @@ -358,9 +358,29 @@ sleep 10 done echo "kube-vip (${KUBE_VIP}) is reachable. Joining this service_kube_node_x86_64 to the cluster now." - JOIN_CMD="$(cat "$JOIN_CMD_FILE") --node-name ${NODE_NAME}" - echo "Executing: $JOIN_CMD" - eval $JOIN_CMD + + # Validate file exists and is non-empty + if [ ! -s "$JOIN_CMD_FILE" ]; then + echo "ERROR: Join command file is missing or empty: $JOIN_CMD_FILE" + exit 1 + fi + + # Validate file contains exactly one line + if [ "$(grep -c '' "$JOIN_CMD_FILE")" -ne 1 ]; then + echo "ERROR: Join command file must contain exactly one line." + exit 1 + fi + + # Validate content matches expected kubeadm join format + JOIN_CMD_CONTENT=$(cat "$JOIN_CMD_FILE") + if ! echo "$JOIN_CMD_CONTENT" | grep -qE '^kubeadm join ([0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]{1,5} --token [a-z0-9]{6}\.[a-z0-9]{16} --discovery-token-ca-cert-hash sha256:[a-f0-9]{64}$'; then + echo "ERROR: Join command file does not match expected kubeadm join format." + echo "Content: $JOIN_CMD_CONTENT" + exit 1 + fi + + echo "Executing join command from: $JOIN_CMD_FILE" + bash -c "$JOIN_CMD_CONTENT --node-name ${NODE_NAME}" sleep 30 echo "Updating the kubelet arguments." sed -i 's/^shutdownGracePeriod:.*/shutdownGracePeriod: 30s/' /var/lib/kubelet/config.yaml diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_control_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_control_node_x86_64.yaml.j2 index 685038695e..e66182315b 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_control_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_control_node_x86_64.yaml.j2 @@ -107,7 +107,7 @@ {{ lookup('template', 'templates/openldap/update_ldap_conf.sh.j2') | indent(12) }} {% endif %} -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -119,9 +119,11 @@ - path: /etc/hosts append: true content: | + # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} + # omnia-inventory-end {% endif %} - path: /root/init_slurm_db.sql @@ -290,7 +292,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2 index 83c29f89be..df51db1624 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2 @@ -361,7 +361,7 @@ echo "[INFO] ===== Completed firewall and service configuration (aarch64) =====" -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -373,9 +373,11 @@ - path: /etc/hosts append: true content: | + # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} + # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -428,7 +430,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) diff --git a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2 b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2 index b49afbec52..d172947a75 100644 --- a/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2 +++ b/src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2 @@ -148,7 +148,7 @@ {% endif %} {% endif %} -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - path: /etc/resolv.conf owner: root:root permissions: '0644' @@ -160,9 +160,11 @@ - path: /etc/hosts append: true content: | + # omnia-inventory-begin {% for key in ip_name_map | sort %} {{ ip_name_map[key] }} {{ key }} {% endfor %} + # omnia-inventory-end {% endif %} - path: /etc/profile.d/create_scratch.sh owner: root:root @@ -436,7 +438,7 @@ runcmd: - /usr/local/bin/set-ssh.sh -{% if dns_enabled | default(false) | bool %} +{% if dns_enabled | default(true) | bool %} - | # Restore CoreDNS resolv.conf after set-ssh.sh (nmcli con up may overwrite it) resolv_target=$(readlink -f /etc/resolv.conf 2>/dev/null || true) diff --git a/src/orchestrator/roles/provision_common/vars/main.yml b/src/orchestrator/roles/provision_common/vars/main.yml index 97052f9909..05f064d4cf 100644 --- a/src/orchestrator/roles/provision_common/vars/main.yml +++ b/src/orchestrator/roles/provision_common/vars/main.yml @@ -187,3 +187,117 @@ openchami_config_missing_fail_msg: >- This file is generated during the deploy stage. Run the orchestrator deploy stage first: ansible-playbook orchestrator.yml --tags deploy smd_service_fail_msg: "smd service is not running" + +# Actionable error messages surfaced (no_log tasks). Printed with no_log: false. +register_nodes_ensure_mapping_rescue_msg: | + The PXE mapping was not loaded on the OIM for this category. + What to fix: run the stages in order, starting from validate. + Then re-run provisioning. + +register_nodes_reread_mapping_rescue_msg: | + The PXE mapping CSV could not be re-read. + What to fix: check the PXE mapping file exists and has an ADMIN_MAC column. + Then re-run provisioning. + +register_nodes_set_mapping_rescue_msg: | + The PXE mapping could not be applied for this category. + What to fix: check the PXE mapping file is valid. + Then re-run provisioning. + +register_nodes_token_refresh_rescue_msg: | + The OpenCHAMI access token could not be refreshed. + What to fix: confirm the OpenCHAMI target is running and the token helper works on the OIM. + Then re-run provisioning. + +register_nodes_update_auth_rescue_msg: | + The refreshed OpenCHAMI token could not be applied. + What to fix: retry the run. If it repeats, check the token helper output. + Then re-run provisioning. + +register_nodes_check_apis_rescue_msg: | + OpenCHAMI APIs are not reachable, or the CA certificate is not trusted. + What to fix: confirm the OpenCHAMI target is up and the CA certificate is in place. + Then re-run provisioning. + +register_nodes_resolve_identities_rescue_msg: | + Node identities could not be resolved in SMD for this category. + What to fix: confirm SMD is running and the nodes are present in hardware inventory. + Then re-run provisioning. + +register_nodes_publish_mapping_rescue_msg: | + Resolved node identities could not be published to the provisioning mapping. + What to fix: rerun from the registration stage. + Then re-run provisioning. + +register_nodes_delete_group_rescue_msg: | + The old node_registration SMD group could not be removed. + What to fix: confirm SMD is running and reachable. + Then re-run provisioning. + +register_nodes_smd_api_ready_rescue_msg: | + The SMD API did not become ready in time. + What to fix: check the SMD service status and its logs. + Then re-run provisioning. + +register_nodes_discover_retry_rescue_msg: | + SMD rejected node discovery after the retries. + What to fix: check the generated node definitions and the SMD logs. + Then re-run provisioning. + +register_nodes_verify_components_rescue_msg: | + Target nodes did not appear in SMD in time. + What to fix: check SMD discovery status and that the nodes are reachable over the BMC network. + Then re-run provisioning. + +register_nodes_reconcile_instanceinfo_rescue_msg: | + Node hostnames could not be updated in the Metadata Service. + What to fix: check the Metadata Service status. + Then re-run provisioning. + +provision_prune_boot_config_rescue_msg: | + Stale Boot Service configurations could not be cleaned up. + What to fix: confirm the OpenCHAMI target is running. + Then re-run provisioning. + +provision_list_boot_config_rescue_msg: | + Boot Service configurations could not be read. + What to fix: confirm the OpenCHAMI target is running. + Then re-run provisioning. + +provision_read_ssh_key_rescue_msg: | + The OIM SSH public key could not be read. + What to fix: restore the key file on the OIM, or regenerate the OIM SSH key pair. + Then re-run provisioning. + +provision_read_ssh_private_key_rescue_msg: | + The OIM SSH private key could not be read. + What to fix: restore the key file on the OIM, or regenerate the OIM SSH key pair. + Then re-run provisioning. + +provision_password_salt_rescue_msg: | + The password salt could not be derived. + What to fix: confirm the project name, cluster name, and cluster domain are set. + Then re-run provisioning. + +provision_hash_password_rescue_msg: | + The provision password could not be hashed. + What to fix: confirm openssl is installed and the provision password is set. + Then re-run provisioning. + +provision_load_ms_defaults_rescue_msg: | + The metadata-service defaults file could not be read. + What to fix: re-run provisioning so the file is regenerated. + +provision_parse_ms_defaults_rescue_msg: | + The metadata-service defaults file is not valid YAML. + What to fix: re-run provisioning to regenerate it, or fix the template inputs. + +provision_reconcile_cluster_defaults_rescue_msg: | + Metadata-service cluster defaults could not be updated. + What to fix: confirm the Metadata Service is running. + Then re-run provisioning. + +provision_ci_group_common_template_rescue_msg: | + The common cloud-init group template could not be rendered. + What to fix: confirm the template exists and all values it uses are defined. + Then re-run provisioning.