Repository navigation
reconciler: Fix bugs in retry handling - #215
Merged
Merged
Conversation
joamaki
requested review from
dylandreimerink
and removed request for
a team
November 19, 2026 10:48
|
dylandreimerink
approved these changes
Nov 19, 2026
Force a concurrent status write during a retry and check that successful results are committed and failed attempts remain scheduled. Cover both pending and refreshing objects, with single and batch initial operations. Also verify that committing the retry preserves the other reconciler's status and does not overwrite a newer pending generation. AIL: 3 Signed-off-by: Jussi Maki <jussi@isovalent.com>
A retry can race with another reconciler's status write. Its result then fails CompareAndSwap while the current status is Error, which the fallback previously rejected. Failed attempts were left without a scheduled retry, and successful attempts could leave the object stuck in Error. Preserve the reconciliation request's status identifier across Done and Error transitions. Accept matching Error and Refreshing statuses as well as Pending when merging a result after a revision conflict. A new pending request still has a different identifier and rejects the stale result. Carry the merged object into subsequent retries so they preserve the other reconciler's fields. Keep the original status passed to retry operations so refresh hints survive retries. AIL: 3 Signed-off-by: Jussi Maki <jussi@isovalent.com>
Keep two objects failing across repeated retries and verify that the reported low watermark stays at the oldest unresolved desired revision. When that object recovers, the watermark must advance to the second object's original revision and then clear once both have recovered. Check that new pending work for a reused key starts a new watermark, both after recovery and when superseding a failure. Exercise single and batch initial operations with virtual time. AIL: 3 Signed-off-by: Jussi Maki <jussi@isovalent.com>
Each failed update commits an Error status at a new table revision. Requeueing previously replaced the retry item's original revision with that newer status revision, allowing the low watermark to advance even though the original desired state had never reconciled successfully. Set the original revision only when creating a retry item. Requeueing continues to update the CAS revision, object, error and backoff while preserving the oldest unresolved revision. Clearing the item allows a new pending request for the same key to establish its own watermark. AIL: 3 Signed-off-by: Jussi Maki <jussi@isovalent.com>
joamaki
force-pushed
the
pr/joamaki/fix-retry-status-conflict
branch
from
November 19, 2026 13:30
30a7719 to
65ceba1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes two AI discovered issues in how the reconciler handled retrying while the object was updated externally.
A retry could lose the result if the object was updated while it was being retried, e.g. if the retry succeeded but previous status was Error it wouldn't get replaced due to CompareAndSwap not succeeding, or conversely the status write of a failed retry could've caused the object not to be queued again for retry.
The second issue fixes an issue in preserving the original revision. This affected how the "low watermark" for reconciliation was advanced and could have caused
WaitForRevisionto succeed before the object was properly reconciled.AIL:3