Repository navigation
fix: Retry network interface write races instead of erroring - #884
Merged
Merged
Conversation
The claim controller, the galactic-networkinterface controller and Galactic all write the same NetworkInterface, claim and VPCAttachment status from cached copies. Losing that race, or a create racing the cache, is normal convergence, but each loss was returned as a reconcile error. Requeue on conflict and on already-exists, as the claim controller does, and treat a not-found write to an interface deleted mid-pass, including the finalizer add, as nothing left to do.
ecv
approved these changes
Nov 6, 2026
ecv
left a comment
Contributor
There was a problem hiding this comment.
Approve.
Same pattern as the claim controller and galactic#1375: a conflict, a cache-race already-exists, or a write to an object deleted mid-pass stops counting as a reconcile error, and every other error still surfaces. That takes routine races out of the error-floor alert without hiding real faults.
- Already-exists can only come from the two
CreateOrUpdatecalls (attachment, NAD), where it means the cache missed an object that exists. The requeue'sGetthen finds it and updates, so it cannot loop on a real collision. - NotFound is swallowed only on the finalizer add and the three status writes, all after the object is gone; deletion handling and finalizer removal are untouched.
- The
networkinterfacecontroller reusesnetworkInterfaceClaimWriteRace, which matches conflicts only; the galactic one also takes already-exists, which fits its creates.
Follow-ups:
- Requeues log at V(1) only and carry no counter, so a pair of writers fighting forever would be invisible once the error floor stops seeing it. A conflict-requeue metric, as on galactic#1375, would keep that visible until the server-side apply move.
kevwilliams
approved these changes
Nov 9, 2026
kevwilliams
left a comment
Contributor
There was a problem hiding this comment.
Verified Requeue:true goes through the rate limiter so backoff holds, and each new test fails without its fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Scot Schuchert-Wells
What
The
networkinterfaceandgalactic-networkinterfacecontrollers report reconcile errors for races that resolve on their own:galactic-networkinterfaceand Galactic all update status on the same NetworkInterface, claim and VPCAttachment, each from a cached copy. Whoever writes second gets a conflict.These errors keep the reconcile error floor alert firing. They don't point to any real fault.
Change
This follows the pattern the claim controller already uses:
Tests cover each case and confirm that other errors still surface. Each new test fails without the fix.
A later step could move each writer to server-side apply status with its own field manager. That would avoid the conflicts outright. It's left out here because it needs a managed-fields migration across two repos.
A companion Galactic change, datum-cloud/galactic#1375, handles the same races on its side.