Skip to content

fix: Retry network interface write races instead of erroring - #884

Merged
scotwells merged 2 commits into
mainfrom
fix/networkinterface-reconcile-noise
Nov 10, 2026
Merged

scotwells merged 2 commits into
mainfrom
fix/networkinterface-reconcile-noise

Conversation

@scotwells

@scotwells scotwells commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Scot Schuchert-Wells

What

The networkinterface and galactic-networkinterface controllers report reconcile errors for races that resolve on their own:

  • The claim controller, galactic-networkinterface and Galactic all update status on the same NetworkInterface, claim and VPCAttachment, each from a cached copy. Whoever writes second gets a conflict.
  • Short-lived guests are deleted while a reconcile is still running. The finalizer add, or a status write, then hits an object that no longer exists.
  • A create can race the cache and return already-exists.

These errors keep the reconcile error floor alert firing. They don't point to any real fault.

Change

This follows the pattern the claim controller already uses:

  • A conflict or already-exists now requeues with backoff instead of returning an error.
  • A write to an interface, attachment or claim that was deleted mid-pass is a no-op. That includes the finalizer add.
  • Every other error still surfaces.

Tests cover each case and confirm that other errors still surface. Each new test fails without the fix.

A later step could move each writer to server-side apply status with its own field manager. That would avoid the conflicts outright. It's left out here because it needs a managed-fields migration across two repos.

A companion Galactic change, datum-cloud/galactic#1375, handles the same races on its side.

The claim controller, the galactic-networkinterface controller and Galactic
all write the same NetworkInterface, claim and VPCAttachment status from
cached copies. Losing that race, or a create racing the cache, is normal
convergence, but each loss was returned as a reconcile error.

Requeue on conflict and on already-exists, as the claim controller does, and
treat a not-found write to an interface deleted mid-pass, including the
finalizer add, as nothing left to do.
@scotwells
scotwells requested review from ecv and kevwilliams November 6, 2026 22:38
@scotwells
scotwells marked this pull request as ready for review November 6, 2026 23:05
@scotwells
scotwells requested a review from a team as a code owner November 6, 2026 23:05

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

Same pattern as the claim controller and galactic#1375: a conflict, a cache-race already-exists, or a write to an object deleted mid-pass stops counting as a reconcile error, and every other error still surfaces. That takes routine races out of the error-floor alert without hiding real faults.

  • Already-exists can only come from the two CreateOrUpdate calls (attachment, NAD), where it means the cache missed an object that exists. The requeue's Get then finds it and updates, so it cannot loop on a real collision.
  • NotFound is swallowed only on the finalizer add and the three status writes, all after the object is gone; deletion handling and finalizer removal are untouched.
  • The networkinterface controller reuses networkInterfaceClaimWriteRace, which matches conflicts only; the galactic one also takes already-exists, which fits its creates.

Follow-ups:

  • Requeues log at V(1) only and carry no counter, so a pair of writers fighting forever would be invisible once the error floor stops seeing it. A conflict-requeue metric, as on galactic#1375, would keep that visible until the server-side apply move.

@kevwilliams kevwilliams left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified Requeue:true goes through the rate limiter so backoff holds, and each new test fails without its fix.

@scotwells
scotwells merged commit 9961ba0 into main Nov 10, 2026
10 checks passed
@scotwells
scotwells deleted the fix/networkinterface-reconcile-noise branch November 10, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants