Skip to content

fix(cni): quiesce host NICs on stop to end the idle-TAP softirq storm#130

Merged
CMGS merged 4 commits into
masterfrom
fix/cni-stop-quiesce-tap-redirect
Jul 13, 2026
Merged

fix(cni): quiesce host NICs on stop to end the idle-TAP softirq storm#130
CMGS merged 4 commits into
masterfrom
fix/cni-stop-quiesce-tap-redirect

Conversation

@CMGS

@CMGS CMGS commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Problem

A stopped CNI VM keeps its per-VM netns and TAP so vm start can reuse them, but stopping the VM kills the VMM, which drops the TAP's carrier. The CNI veth (eth0) stays UP on the host bridge (cni0) receiving LAN broadcast, and the tc mirred ingress redirect (eth0 → TAP, from tcRedirectInNS) fires against the now-down TAP for every packet:

tc mirred to Houston: device tapXXXXXXXX-0 is down

Under a bridge carrying a physical uplink this storms softirqs (NOHZ tick-stop, pending TASKLET/RCU) until the host soft-locks and reboots. The netns/TAP/tc were only torn down on vm rm, never on stop — so any stopped CNI VM was a host-crash landmine. Reproduced on hardware with a plain Ubuntu VM (not a workload): after vm stop it stormed dmesg with thousands of these messages and rebooted the host.

Fix

Add Quiesce/Unquiesce to the network.Network interface:

  • CNI — bring every recorded veth (ethN) down/up inside the netns. Downing eth0 stops cni0 from forwarding LAN broadcast into the netns, so the redirect never targets a carrier-less TAP. The netns and TAP are kept for the fast restart.
  • Bridge — no-op: bridge TAPs sit directly on the host bridge with no tc redirect to storm.

Wired symmetrically with the existing lifecycle: vm stop quiesces after hyper.Stop (counterpart to vm rm's netProvider.Delete); vm start's recoverNetwork unquiesces on the fast-restart (Verify-passes) path.

Verification (hardware)

Phase eth0 Evidence
running UP normal
after vm stop DOWN quiesce applied
8s stopped, under LAN broadcast mirred messages: 1 (was thousands)
after vm start UP unquiesce applied
after restart VM re-acquired its DHCP IP; host stable

Build + vet + golangci-lint clean on linux and darwin. Unit tests added for CNI Quiesce/Unquiesce (toggles every NIC; skips a VM with no records).

CMGS added 4 commits July 14, 2026 01:32
A stopped CNI VM keeps its netns/TAP for a fast restart, but the VMM exit
drops the TAP's carrier while its veth stays up on the host bridge; the tc
mirred redirect then fires against the down TAP for every LAN broadcast
packet ("mirred to Houston: device is down"), storming softirqs (NOHZ
tick-stop) until the host soft-locks and reboots. Teardown only ran on vm
rm, so any stopped CNI VM was a host-crash landmine. Bring the veths down
on stop (Quiesce) and back up on start (Unquiesce) so the redirect never
targets a carrier-less TAP.
vm rm --force kills the VMM (dropping the TAP's carrier) but only removes
the netns/TC redirect afterwards via netProvider.Delete, leaving the same
idle-TAP softirq storm window vm stop had — long enough to crash a host on
a busy bridge. Extract stopAndQuiesce and reuse it from rm --force before
the delete.
@CMGS CMGS merged commit 0f6c21b into master Jul 13, 2026
4 checks passed
@CMGS CMGS deleted the fix/cni-stop-quiesce-tap-redirect branch July 13, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant