[SOWJANYAJAGADISH123]

Problem

On diskless PXE-booted Kubernetes control planes with etcd_on_local_disk: true, the kernel reorders /dev/sdX names across reboots. The etcd mount and prepare scripts selected the etcd disk by a device-name scan, so /var/lib/etcd could land on a different physical disk on each boot.

When the orchestrator was re-run and the node re-PXE'd, the PKI on the NFS share was regenerated (new cluster CA) while a stale etcd disk (from an older CA generation) could be mounted. kube-proxy and Calico then failed to verify the API server certificate:

x509: certificate signed by unknown authority

This left the control plane NotReady, with Calico and CoreDNS stuck in ContainerCreating / Init:CrashLoopBackOff. A plain reboot was unaffected because it regenerated neither the CA nor etcd; only an orchestrator re-run exposed the mismatch.

Root cause

• The etcd disk was selected by unstable /dev/sdX ordering, not by a stable identifier.
• The mount script (etcd-fstab-update.sh, every boot) and the prepare script (etcd-disk-setup.sh, cluster-init) used independent selection logic and could choose different disks.
• Multiple disks could carry the same etcd_data label, making label- and scan-based selection ambiguous.
• Nothing persisted the selected disk identity, so later boots had no stable reference.

The certificate error was a downstream symptom of mounting the wrong (stale) etcd disk, not a PKI defect.

Fix

Pin the etcd disk by its stable hardware serial, recorded in a per-node pin file on the NFS share (<k8s_client_mount_path>/<node_ip>/etcd/etcd-disk.conf, containing ETCD_DISK_SERIAL and ETCD_UUID). Every later boot resolves the same physical disk by serial regardless of how the kernel names it.

etcd-disk-setup.sh.j2 (prepare, cluster-init path only)

• Reuse the pinned serial when the pin already exists.
• Resolve the disk by serial and derive the partition from the partition table instead of /dev/sdX naming.
• Strip the etcd_data label from any other disk so only one etcd disk exists.
• After formatting, persist the pin (serial + UUID) to NFS so later boots mount the same disk.
• Fail hard instead of falling back to a device-name scan when an existing pin is present but its serial cannot be resolved uniquely.
• Fail hard if the selected disk has no serial or if the pin cannot be written/read back.

etcd-fstab-update.sh.j2 (mount, every boot)

• Mount strictly by the pinned serial; the device-name scan fallback is removed.
• Validate the partition filesystem type (ext4) and that its UUID matches the pin before mounting.
• Without a pin, adopt only an unambiguous existing etcd disk (the already-mounted one, or a single etcd_data partition), then persist the pin to self-heal clusters provisioned before serial pinning existed. Refuse to guess when the choice is ambiguous.
• Resolve NFS paths from the node IP and jinja variables instead of a hardcoded server IP.

ms-group-service_kube_control_plane_first_x86_64.yaml.j2 (control-plane first boot)

• Add a defensive guard before kubeadm init: refuse to initialize when /var/lib/etcd/member still exists. kubeadm init generates a new cluster CA, so etcd must be empty at that point. The init path already reformats the disk and purges state, so this never triggers on a healthy boot and is not reached on reboot. It makes the "new CA implies fresh etcd" invariant explicit and fails loudly if a future change ever regresses it.

Behavior by scenario

Scenario	Marker	Pin	Behavior
Fresh first boot	absent	absent	Prepare selects/formats a disk, writes the pin, kubeadm init creates matching CA + etcd
Normal reboot	present	present	Mount the same disk by serial; no format, no reset, CA and etcd reused
Reboot after sdX reorder	present	present	Same physical disk resolved by serial
Re-provision / cleanup	recreated	recreated	Fresh etcd and fresh CA created together
Legacy cluster, no pin	present	absent	Adopt the unambiguous existing etcd disk and persist a pin
Invalid/unresolvable pin, blank serial, UUID mismatch, unwritable pin	—	—	Fail loudly; never scan, format, or mount the wrong disk

Why this does not break existing paths

• Fresh provisioning does not require a pre-existing pin; the pin is created during first-boot preparation.
• The kubeadm init guard only runs on the first-boot path, where etcd is already formatted and purged, so it passes; it is never reached on reboot.
• The changes only add deterministic selection and fail-fast guards; no existing successful path is loosened.

Test plan

• Rendered both shell templates with representative values and verified with bash -n.
• Verified the serial-selection logic resolves the same serial when a disk moves between sda and sdb.
• Verified git diff --check (no whitespace errors).
• Fresh provision on hardware: pin written, cluster Ready.
• Reboot: findmnt /var/lib/etcd serial matches the pin even when sdX names change.
• Full cleanup + re-provision: new etcd and new CA created together, cluster Ready.
• Negative cases: blank/duplicate serial, missing pin, UUID mismatch fail safely without touching another disk.
• Confirm lsblk -dno SERIAL returns a non-empty, unique value for each etcd disk on the target nodes.

Notes / follow-ups

• First-boot disk selection still scans when no pin and no inventory serial exist; providing an etcd disk serial via inventory, or requiring exactly one eligible disk, would make first selection fully deterministic.
• The adoption path for legacy clusters (marker present, no pin) should be validated during migration.

[abhishek-sa1]

Technical review: approve

Root cause confirmed. The etcd disk was chosen by /dev/sdX order, which changes on every boot. After a re-provision, a stale etcd disk could be mounted against a newer CA. kube-proxy and Calico then failed with x509: certificate signed by unknown authority. That error was a symptom, not a PKI defect. Pinning the disk by hardware serial fixes the cause.

What the PR does

• etcd-disk-setup.sh.j2 (first boot only):
  • Reuses the pinned serial if one exists, otherwise keeps the BOSS-first scan.
  • Formats the disk and removes the etcd_data label from other disks.
  • Writes serial and UUID to <share>/<node_ip>/etcd/etcd-disk.conf.
  • Stops if the pin can't be resolved, the serial is blank, or the pin can't be written.
• etcd-fstab-update.sh.j2 (every boot):
  • Mounts by pinned serial, finds the partition from the partition table, and checks ext4 and the UUID.
  • No device-name scan.
  • With no pin, adopts a single existing etcd disk and writes the pin. Refuses if several are labelled.
• First-boot template: kubeadm init is refused if /var/lib/etcd/member already exists, so a new CA never starts on old etcd data.

Verified

• Both templates render with Jinja and pass bash -n, with no leftover Jinja tokens (cloud-init renders user-data as Jinja).
• Traced fresh provision, normal reboot, reboot after an sdX reorder, and reprovision with cleanup, without cleanup, and with leftover labelled disks. Each ends with a fresh CA and fresh etcd, or the same disk mounted by serial.
• BOSS selection is unchanged: the BOSS-first break is kept.
• Code and security CI checks pass.

Before merge (process items)

• Link an issue (the "Validate linked issue and IP policy" check is failing).
• Sign off the commits (DCO).
• Update the branch, which is behind staging.