fix(cni): quiesce host NICs on stop to end the idle-TAP softirq storm#130
Merged
Conversation
A stopped CNI VM keeps its netns/TAP for a fast restart, but the VMM exit
drops the TAP's carrier while its veth stays up on the host bridge; the tc
mirred redirect then fires against the down TAP for every LAN broadcast
packet ("mirred to Houston: device is down"), storming softirqs (NOHZ
tick-stop) until the host soft-locks and reboots. Teardown only ran on vm
rm, so any stopped CNI VM was a host-crash landmine. Bring the veths down
on stop (Quiesce) and back up on start (Unquiesce) so the redirect never
targets a carrier-less TAP.
vm rm --force kills the VMM (dropping the TAP's carrier) but only removes the netns/TC redirect afterwards via netProvider.Delete, leaving the same idle-TAP softirq storm window vm stop had — long enough to crash a host on a busy bridge. Extract stopAndQuiesce and reuse it from rm --force before the delete.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A stopped CNI VM keeps its per-VM netns and TAP so
vm startcan reuse them, but stopping the VM kills the VMM, which drops the TAP's carrier. The CNI veth (eth0) stays UP on the host bridge (cni0) receiving LAN broadcast, and the tcmirredingress redirect (eth0 → TAP, fromtcRedirectInNS) fires against the now-down TAP for every packet:Under a bridge carrying a physical uplink this storms softirqs (NOHZ tick-stop, pending TASKLET/RCU) until the host soft-locks and reboots. The netns/TAP/tc were only torn down on
vm rm, never on stop — so any stopped CNI VM was a host-crash landmine. Reproduced on hardware with a plain Ubuntu VM (not a workload): aftervm stopit stormed dmesg with thousands of these messages and rebooted the host.Fix
Add
Quiesce/Unquiesceto thenetwork.Networkinterface:ethN) down/up inside the netns. Downingeth0stopscni0from forwarding LAN broadcast into the netns, so the redirect never targets a carrier-less TAP. The netns and TAP are kept for the fast restart.Wired symmetrically with the existing lifecycle:
vm stopquiesces afterhyper.Stop(counterpart tovm rm'snetProvider.Delete);vm start'srecoverNetworkunquiesces on the fast-restart (Verify-passes) path.Verification (hardware)
vm stopmirredmessages: 1 (was thousands)vm startBuild +
vet+golangci-lintclean onlinuxanddarwin. Unit tests added for CNIQuiesce/Unquiesce(toggles every NIC; skips a VM with no records).