test: cover the async delete create event received while CNS runs

[Quang Nguyen (nddq)]

Follows the review comment on #4768: the async delete e2e test misses the fsnotify create event and covers only the startup scan.

What the test misses today

The script stops CNS on the node, deletes a Pod, and then starts CNS again. The pending delete file is therefore always on disk before CNS starts. CNS reads it in the startup directory scan at fsnotify.go:133, which keys the map by the file name and has always been correct.

The other path never runs. When CNS is up but does not answer, the CNI writes the file while the watcher listens, and the watcher takes the create event at fsnotify.go:166. That line held the defect that #4768 fixes. The e2e test passed on the broken code for 3 years.

What this adds

A second stage that drives a real CNI delete against a CNS that never restarts:

1. Find the CNS process from the debug container and send it SIGSTOP. The socket stays open, so the CNI request times out instead of failing fast.
2. Delete a busybox Pod. The CNI DEL waits out the 15 second CNS request timeout, then writes the pending delete file.
3. Check that the file appeared. Without this check an unexercised path would look like a pass.
4. SIGCONT. The kernel queues the inotify event while the process is stopped and delivers it on resume, so the watcher gets a live create event.
5. Wait up to 60 seconds for the directory to drain. The release ticker runs every 15 seconds.
6. Fail if the file remains, or if the CNS container restarted. A restart would prove only the startup scan again.

shareProcessNamespace: true on the CNS test daemonset is what lets the debug container signal CNS. It is needed because the CNI delete path and the CNS watcher both use http://localhost:10090 — cni/network/network.go:1108 and cns/service/main.go:1076 each call cnsclient.New("") — so no port or address change can break one without breaking the other, and the debug container drops all capabilities.

Validation

Run on a live swift-byocni AKS cluster, 2 nodes, CNS installed from these manifests.

CNS build	Stage 1 (startup scan)	Stage 2 (create event)
v1.8.12-25-g6013330eb (master, defect present)	pass	fail
same commit with #4768 cherry-picked	pass	pass

Stage 1 passing on the defective build is the point: it is why this never showed up.

CNS logs from the failing run, with the doubled path the defect produces:

msg="successfully released IP for missed delete"
  containerID=/var/run/azure-vnet/deleteIDs/11a88a41...
msg="failed to remove file for missed delete"
  error="remove /var/run/azure-vnet/deleteIDs//var/run/azure-vnet/deleteIDs/11a88a41...: no such file or directory"

The same stage on the fixed build, container restart count unchanged at 0:

msg="received create event" event=/var/run/azure-vnet/deleteIDs/a3fc8d96...
msg="processing pending missed deletes" count=1
msg="releasing IP for missed delete" podInterfaceID=a3fc8d96-eth0 containerID=a3fc8d96...
msg="successfully released IP for missed delete"

Notes

• Merge after fix: key async delete watcher by container ID, not full path #4768. The new stage fails on a CNS that still holds the defect, which is the point of the test.
• The stage adds about 40 seconds: 15 for the CNI timeout and up to 60 for one or two release ticks.
• sh -n and bash -n both pass. No new dependency, no new manifest, no pipeline change.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds coverage to the async-delete integration test to exercise the live fsnotify create-event path (when CNS is running but unresponsive) rather than only the startup directory scan path, by pausing the CNS process and validating pending-delete file creation/cleanup without a CNS restart.

Changes:

• Extend async-delete-test.sh with a second stage that SIGSTOPs CNS, drives a real CNI DEL, and waits for the watcher-driven cleanup.
• Update the CNS test DaemonSet manifest to enable shareProcessNamespace so the debug container can signal the CNS process.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
hack/scripts/async-delete-test.sh	Adds a second-phase test that pauses CNS to force pending-delete file creation while the fsnotify watcher is active, then verifies cleanup without restart.
test/integration/manifests/cns/daemonset-linux.yaml	Enables process namespace sharing so the debug container can signal CNS for the new test stage.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

hack/scripts/async-delete-test.sh:83

• After SIGSTOP succeeds, cancelling or interrupting this test before the explicit resume leaves CNS stopped indefinitely; this DaemonSet has no liveness probe to recover node networking. Register cleanup immediately after stopping CNS (including EXIT, HUP, INT, and TERM) and clear it only after SIGCONT succeeds.

kubectl exec -i $cns_pod -c debug -n kube-system -- sh -c "kill -STOP $cns_pid"

[Evan Baker (rbtr)]

entering the CNS container ns and pausing the process is a fascinating way to test this - totally valid and very thorough. i would have stubbed some fake deletes out on disk to see if they would be processed instead of such surgery because i don't think i could pull it off

[Evan Baker (rbtr)]

/azp run Azure Container Networking PR

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).