Skip to content

feat: add phys-bits (MAXPHYADDR) as a second live-migration safety axis#3

Open
maxgashkov wants to merge 1 commit into
credativ:mainfrom
maxgashkov:feat/phys-bits-dimension
Open

feat: add phys-bits (MAXPHYADDR) as a second live-migration safety axis#3
maxgashkov wants to merge 1 commit into
credativ:mainfrom
maxgashkov:feat/phys-bits-dimension

Conversation

@maxgashkov

Copy link
Copy Markdown

Add phys-bits (MAXPHYADDR) as a second live-migration safety axis

Problem

ProxCLMC computes the lowest common ISA baseline (x86-64-vN). That's necessary but not sufficient: the guest's advertised physical-address width (phys-bits / MAXPHYADDR) must also be <= every host's. The two axes are independent — a cluster cleanly standardized on x86-64-v2-AES can still corrupt guests on live migration.

An unpinned guest on a named CPU model advertises QEMU's default 40 phys-bits. On a mixed cluster (e.g. Intel nodes at 39-bit, AMD at 48-bit) a VM booted on a node at least 40 bits wide can place its virtio MMIO BARs above a narrower node's limit. The aperture is fixed at boot and travels unchanged across a live migration; when it lands on the narrow node, the next virtqueue kick faults (#PF, reserved bit) and the guest kernel-panics. Offline migration, HA failover and reboots cold-boot on the destination and are unaffected.

Change

  • Parse the address sizes line from /proc/cpuinfo into a per-node phys-bits.
  • Compute the cluster's lowest common phys-bits (undetected nodes are skipped).
  • Show phys-bits as a fourth column in the node listing.
  • When a node sits below the guest default — min < min(GUEST_DEFAULT_PHYS_BITS, max) — pin the reported CPU type to the cluster minimum, e.g. Cluster CPU type: x86-64-v3,phys-bits=39. --list-only carries the same suffix.
  • Output is unchanged for uniform clusters and clusters all at or above the guest default.

GUEST_DEFAULT_PHYS_BITS is hardcoded to 40 (QEMU's default for named models, verified on pve-qemu 11.0.0 via qom-get/query-cpu-model-expansion). If QEMU ever raises that default this can under-report until the constant is bumped.

Testing

  • cargo test — unit tests for cpuinfo parsing, cluster min/max, and the pin predicate.
  • cargo clippy clean.
  • Run live on a 3-node cluster (2x Intel 39-bit, 1x AMD 48-bit): correctly detects the mismatch and recommends phys-bits=39:
./proxclmc
Detected nodes:
prox1 | 10.0.1.10 | x86-64-v3 | 39
prox2 | 10.0.1.11 | x86-64-v3 | 39
prox3 | 10.0.1.12 | x86-64-v3 | 48

Cluster CPU type: x86-64-v3,phys-bits=39

./proxclmc --list-only
x86-64-v3,phys-bits=39

Notes

  • No new CLI flags or dependencies.
  • LLM used (Opus 4.8) but it was heavily steered and the result edited by a human.

ProxCLMC computed only the lowest common ISA baseline (x86-64-vN). That is
necessary but not sufficient for safe live migration: the guest's advertised
physical-address width (phys-bits / MAXPHYADDR) must also be <= every host's.
The two axes are independent — a cluster standardized on x86-64-v2-AES can
still panic guests on live migration when a node is narrower than the guest's
default width (e.g. Intel 39-bit alongside AMD 48-bit).

An unpinned guest on a named CPU model advertises QEMU's default 40 phys-bits.
A VM booted on a node at least that wide can place virtio MMIO BARs above a
narrower node's limit; the aperture is fixed at boot and travels unchanged
across a live migration, so the next virtqueue kick on the narrow node faults
(#PF, reserved bit) and the guest kernel-panics. Offline migration, HA failover
and reboots cold-boot on the destination and are unaffected, which makes this
look random and easy to misdiagnose.

This patch:
- parses the "address sizes" line from /proc/cpuinfo into per-node phys-bits
- computes the cluster's lowest common phys-bits (ignoring undetected nodes)
- pins the reported CPU type to that minimum when a node sits below the guest
  default (min < min(GUEST_DEFAULT_PHYS_BITS, max)), e.g.
  Cluster CPU type: x86-64-v3,phys-bits=39
- --list-only carries the same suffix; output stays single-line and unchanged
  for uniform clusters and clusters all at or above the guest default
- unit tests for parsing (real /proc/cpuinfo address-size lines), cluster
  min/max, and the pin predicate (one case per branch)
- documents the second axis in README, deferring the deep mechanism to QEMU's
  physical-address-space notes
- adds a 1.3.0 changelog fragment

GUEST_DEFAULT_PHYS_BITS is hardcoded to 40 (QEMU's default for named models,
verified on pve-qemu 11.0.0 via qom-get/query-cpu-model-expansion). If QEMU
ever raises that default this can under-report until the constant is bumped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant