Skip to content

reconciler: Fix bugs in retry handling - #215

Merged
joamaki merged 4 commits into
mainfrom
pr/joamaki/fix-retry-status-conflict
Nov 19, 2026
Merged

joamaki merged 4 commits into
mainfrom
pr/joamaki/fix-retry-status-conflict

Conversation

@joamaki

@joamaki joamaki commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

This fixes two AI discovered issues in how the reconciler handled retrying while the object was updated externally.

A retry could lose the result if the object was updated while it was being retried, e.g. if the retry succeeded but previous status was Error it wouldn't get replaced due to CompareAndSwap not succeeding, or conversely the status write of a failed retry could've caused the object not to be queued again for retry.

The second issue fixes an issue in preserving the original revision. This affected how the "low watermark" for reconciliation was advanced and could have caused WaitForRevision to succeed before the object was properly reconciled.

AIL:3

@joamaki
joamaki requested a review from a team as a code owner November 19, 2026 10:48
@joamaki
joamaki requested review from dylandreimerink and removed request for a team November 19, 2026 10:48
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
$ make
go build ./...
go: downloading github.com/cilium/hive v1.0.4
go: downloading golang.org/x/time v0.15.0
go: downloading github.com/spf13/cobra v1.10.2
go: downloading github.com/spf13/pflag v1.0.10
go: downloading go.yaml.in/yaml/v3 v3.0.4
go: downloading github.com/cilium/stream v0.0.1
go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de
go: downloading github.com/spf13/viper v1.18.2
go: downloading go.uber.org/dig v1.17.1
go: downloading golang.org/x/term v0.16.0
go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading golang.org/x/sys v0.17.0
go: downloading golang.org/x/tools v0.17.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.14.0
STATEDB_VALIDATE=1 go test ./... -cover -vet=all -test.count 1
go: downloading github.com/stretchr/testify v1.11.1
go: downloading go.uber.org/goleak v1.3.0
go: downloading golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
ok  	github.com/cilium/statedb	250.139s	coverage: 79.5% of statements
ok  	github.com/cilium/statedb/index	0.005s	coverage: 48.6% of statements
ok  	github.com/cilium/statedb/internal	0.036s	coverage: 54.1% of statements
ok  	github.com/cilium/statedb/lpm	3.328s	coverage: 76.4% of statements
ok  	github.com/cilium/statedb/part	64.958s	coverage: 86.7% of statements
ok  	github.com/cilium/statedb/reconciler	0.346s	coverage: 94.2% of statements
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
go test -race ./... -test.count 1
ok  	github.com/cilium/statedb	31.601s
ok  	github.com/cilium/statedb/index	1.017s
ok  	github.com/cilium/statedb/internal	1.022s
ok  	github.com/cilium/statedb/lpm	2.050s
ok  	github.com/cilium/statedb/part	27.000s
ok  	github.com/cilium/statedb/reconciler	1.421s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 9V74 80-Core Processor                
BenchmarkDB_WriteTxn_1-4                      	 1304020	       921.4 ns/op	   1085304 objects/sec	     632 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_10-4                     	 3071186	       401.5 ns/op	   2490723 objects/sec	     349 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_100-4                    	 3622038	       329.0 ns/op	   3039273 objects/sec	     317 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_1000-4                   	 3342626	       361.4 ns/op	   2767293 objects/sec	     318 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4     	 1867297	       568.3 ns/op	   1759512 objects/sec	     419 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_100Tables-4   	 2099985	       561.2 ns/op	    1088 B/op	       3 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_1Table-4      	 2863780	       418.2 ns/op	     200 B/op	       3 allocs/op
BenchmarkDB_NewWriteTxn-4                     	 3084464	       387.9 ns/op	     200 B/op	       3 allocs/op
BenchmarkDB_WriteTxnCommit100-4               	 2165784	       553.4 ns/op	    1072 B/op	       3 allocs/op
BenchmarkDB_NewReadTxn-4                      	729798128	         1.643 ns/op	       0 B/op	       0 allocs/op
BenchmarkDB_Modify-4                          	    2833	    406403 ns/op	   2460611 objects/sec	  342483 B/op	    6072 allocs/op
BenchmarkDB_GetInsert-4                       	    2601	    444742 ns/op	   2248497 objects/sec	  326480 B/op	    6072 allocs/op
BenchmarkDB_ListInsert-4                      	    2044	    589721 ns/op	   1695717 objects/sec	  486512 B/op	   11072 allocs/op
BenchmarkDB_RandomInsert-4                    	    3181	    378568 ns/op	   2641533 objects/sec	  318475 B/op	    5072 allocs/op
BenchmarkDB_RandomReplace-4                   	    1598	    771734 ns/op	   1295783 objects/sec	  432292 B/op	    8088 allocs/op
BenchmarkDB_SequentialInsert-4                	    3285	    364949 ns/op	   2740112 objects/sec	  318475 B/op	    5072 allocs/op
BenchmarkDB_SequentialInsert_Prefix-4         	     682	   1733957 ns/op	    576715 objects/sec	 2793439 B/op	   41795 allocs/op
BenchmarkDB_Changes_Baseline-4                	    2614	    460822 ns/op	   2170037 objects/sec	  388220 B/op	    6162 allocs/op
BenchmarkDB_Changes-4                         	    1369	    886868 ns/op	   1127564 objects/sec	  603949 B/op	    9323 allocs/op
BenchmarkDB_RandomLookup-4                    	   37106	     32394 ns/op	  30870294 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_SequentialLookup-4                	   39188	     30658 ns/op	  32618331 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_Prefix_SecondaryIndex-4           	   38838	     30654 ns/op	  32622177 objects/sec	     193 B/op	       6 allocs/op
BenchmarkDB_LowerBound_SecondaryIndex-4       	   35905	     32510 ns/op	  30759351 objects/sec	     506 B/op	       8 allocs/op
BenchmarkDB_FullIteration_All-4               	     644	   1904794 ns/op	  52499116 objects/sec	     104 B/op	       4 allocs/op
BenchmarkDB_FullIteration_Prefix-4            	     610	   1902299 ns/op	  52567966 objects/sec	     136 B/op	       5 allocs/op
BenchmarkDB_FullIteration_Get-4               	     180	   6585488 ns/op	  15184903 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_Get_Secondary-4     	      85	  13522733 ns/op	   7394955 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_ReadTxnGet-4        	     176	   6745446 ns/op	  14824818 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4                	 1318810	       917.5 ns/op	         7.000 50th_µs	         9.000 90th_µs	        30.00 99th_µs	     755 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_1_DefaultMetrics-4       	 1264258	       948.1 ns/op	     632 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_1_ExpVarMetrics-4        	  916941	      1206 ns/op	     656 B/op	      15 allocs/op
BenchmarkDB_WriteTxn_100_LPMIndex-4           	  696980	      1762 ns/op	    567632 objects/sec	    1571 B/op	      35 allocs/op
BenchmarkDB_WriteTxn_1_LPMIndex-4             	  187027	     13571 ns/op	     73689 objects/sec	   14154 B/op	      66 allocs/op
BenchmarkDB_LPMIndex_Get-4                    	     177	   6690721 ns/op	   1494607 objects/sec	       0 B/op	       0 allocs/op
BenchmarkWatchSet_4-4                         	 2932486	       402.9 ns/op	     296 B/op	       4 allocs/op
BenchmarkWatchSet_16-4                        	  901227	      1260 ns/op	    1096 B/op	       5 allocs/op
BenchmarkWatchSet_128-4                       	  105590	     11380 ns/op	    8904 B/op	       5 allocs/op
BenchmarkWatchSet_1024-4                      	   10000	    112214 ns/op	   73742 B/op	       5 allocs/op
PASS
ok  	github.com/cilium/statedb	49.807s
PASS
ok  	github.com/cilium/statedb/index	0.004s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/internal
cpu: AMD EPYC 9V74 80-Core Processor                
Benchmark_SortableMutex-4   	 7558468	       158.8 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/internal	1.204s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/lpm
cpu: AMD EPYC 9V74 80-Core Processor                
Benchmark_txn_insert/batchSize=1-4         	    2562	    453587 ns/op	   2204648 objects/sec	  822424 B/op	   13975 allocs/op
Benchmark_txn_insert/batchSize=10-4        	    4519	    267568 ns/op	   3737368 objects/sec	  369202 B/op	    6668 allocs/op
Benchmark_txn_insert/batchSize=100-4       	    4874	    249894 ns/op	   4001691 objects/sec	  329619 B/op	    6027 allocs/op
Benchmark_txn_delete/batchSize=1-4         	    2055	    579365 ns/op	   1726028 objects/sec	 1270473 B/op	   13976 allocs/op
Benchmark_txn_delete/batchSize=10-4        	    4321	    286971 ns/op	   3484674 objects/sec	  356419 B/op	    5769 allocs/op
Benchmark_txn_delete/batchSize=100-4       	    4935	    244197 ns/op	   4095048 objects/sec	  270754 B/op	    5038 allocs/op
Benchmark_LPM_Lookup-4                     	   10000	    107661 ns/op	   9288381 objects/sec	       0 B/op	       0 allocs/op
Benchmark_LPM_All-4                        	  165397	      7225 ns/op	 138405206 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_Prefix-4                     	  167782	      7159 ns/op	 139686169 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_LowerBound-4                 	  327753	      3821 ns/op	 130866070 objects/sec	     288 B/op	       2 allocs/op
PASS
ok  	github.com/cilium/statedb/lpm	11.962s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 9V74 80-Core Processor                
Benchmark_Set_Singleton_Create-4              	27953902	        40.97 ns/op	      24 B/op	       1 allocs/op
Benchmark_Set_Singleton_Has-4                 	122522752	         9.805 ns/op	       0 B/op	       0 allocs/op
Benchmark_StringMap_Txn_Insert-4              	   12217	     98089 ns/op	  10194864 items/sec	   97737 B/op	    1305 allocs/op
Benchmark_Uint64Map_Random-4                  	    2587	    487787 ns/op	   2050074 items/sec	 1322706 B/op	    6063 allocs/op
Benchmark_Uint64Map_Sequential-4              	    2559	    441642 ns/op	   2264279 items/sec	 1702577 B/op	    5752 allocs/op
Benchmark_Uint64Map_Sequential_Insert-4       	    2973	    402891 ns/op	   2482061 items/sec	 1694573 B/op	    4751 allocs/op
Benchmark_Uint64Map_Sequential_Txn_Insert-4   	   13807	     87348 ns/op	  11448504 items/sec	   89952 B/op	    2030 allocs/op
Benchmark_Uint64Map_Random_Insert-4           	    2794	    409592 ns/op	   2441452 items/sec	 1313192 B/op	    5031 allocs/op
Benchmark_Uint64Map_Random_Txn_Insert-4       	    8710	    135713 ns/op	   7368500 items/sec	  116794 B/op	    2413 allocs/op
Benchmark_Insert_RootOnlyWatch-4              	   14119	     85526 ns/op	  11692291 objects/sec	   75040 B/op	    2035 allocs/op
Benchmark_Insert-4                            	   14508	     82954 ns/op	  12054934 objects/sec	   75584 B/op	    2046 allocs/op
Benchmark_WatchReplace-4                      	   16173	     74117 ns/op	  13492240 objects/sec	   57305 B/op	    1006 allocs/op
Benchmark_Modify-4                            	   16747	     71558 ns/op	  13974746 objects/sec	   58056 B/op	    1007 allocs/op
Benchmark_GetInsert-4                         	   13736	     87201 ns/op	  11467695 objects/sec	   58056 B/op	    1007 allocs/op
Benchmark_Replace-4                           	21633190	        55.44 ns/op	  18037211 objects/sec	      48 B/op	       1 allocs/op
Benchmark_Replace_RootOnlyWatch-4             	 9067713	       144.6 ns/op	   6914731 objects/sec	      94 B/op	       1 allocs/op
Benchmark_txn_1-4                             	 7912226	       144.0 ns/op	   6942200 objects/sec	      64 B/op	       3 allocs/op
Benchmark_txn_10-4                            	13236304	        87.07 ns/op	  11485165 objects/sec	      76 B/op	       2 allocs/op
Benchmark_txn_100-4                           	16032511	        75.56 ns/op	  13234722 objects/sec	      67 B/op	       2 allocs/op
Benchmark_txn_1000-4                          	13881721	        84.87 ns/op	  11782527 objects/sec	      65 B/op	       2 allocs/op
Benchmark_txn_delete_1-4                      	 6862005	       177.5 ns/op	   5632330 objects/sec	     632 B/op	       3 allocs/op
Benchmark_txn_delete_10-4                     	14120068	        84.10 ns/op	  11890972 objects/sec	     103 B/op	       1 allocs/op
Benchmark_txn_delete_100-4                    	16581313	        70.98 ns/op	  14088388 objects/sec	      35 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4                   	16588492	        72.28 ns/op	  13834183 objects/sec	      28 B/op	       1 allocs/op
BenchmarkDeleteParentCache/depth_8-4          	 6064176	       195.1 ns/op	      48 B/op	       1 allocs/op
BenchmarkDeleteParentCache/depth_100-4        	  648424	      1849 ns/op	      48 B/op	       1 allocs/op
Benchmark_Get-4                               	   82029	     14765 ns/op	  67726620 objects/sec	       0 B/op	       0 allocs/op
Benchmark_GetWatch-4                          	   65217	     18405 ns/op	  54332038 objects/sec	       0 B/op	       0 allocs/op
Benchmark_All-4                               	  202770	      5842 ns/op	 171188010 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_All-4                      	  215322	      5586 ns/op	 179019591 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_Next-4                     	  195980	      5983 ns/op	 167136579 objects/sec	     896 B/op	       1 allocs/op
Benchmark_LowerBound-4                        	   16046	     75064 ns/op	  13322035 objects/sec	  104000 B/op	    2000 allocs/op
Benchmark_LowerBound_Random-4                 	   10423	    114948 ns/op	   8699557 objects/sec	   96384 B/op	    1002 allocs/op
Benchmark_Hashmap_Insert-4                    	   21812	     54864 ns/op	  18226758 objects/sec	   74264 B/op	      20 allocs/op
Benchmark_Hashmap_Get_Uint64-4                	  196080	      6079 ns/op	 164490249 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4                 	  161516	      7408 ns/op	 134997918 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Delete_Random-4                     	      66	  17470487 ns/op	   5723939 objects/sec	 2539401 B/op	  102756 allocs/op
Benchmark_find16-4                            	312965786	         3.835 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex16-4                       	100000000	        11.47 ns/op	       0 B/op	       0 allocs/op
Benchmark_find64-4                            	365453121	         3.285 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_hit-4                   	363749815	         3.288 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_miss-4                  	348716197	         3.328 ns/op	       0 B/op	       0 allocs/op
Benchmark_find4-4                             	545178751	         2.204 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex4-4                        	438255895	         2.742 ns/op	       0 B/op	       0 allocs/op
BenchmarkSmallWriteTxn/updates_1-4            	  524886	      2236 ns/op	    3529 B/op	       4 allocs/op
BenchmarkSmallWriteTxn/updates_2-4            	  360393	      3100 ns/op	    4742 B/op	       6 allocs/op
BenchmarkSmallWriteTxn/updates_4-4            	  244488	      4788 ns/op	    7155 B/op	      11 allocs/op
BenchmarkSmallWriteTxn/updates_8-4            	  145458	      8546 ns/op	   11927 B/op	      20 allocs/op
BenchmarkSmallWriteTxn/updates_16-4           	   68024	     16561 ns/op	   21290 B/op	      38 allocs/op
BenchmarkFreshWriteTxn-4                      	 7343902	       171.4 ns/op	     185 B/op	       6 allocs/op
BenchmarkAtomicWatchPointerFirstChannel-4     	25717723	        47.54 ns/op	     120 B/op	       2 allocs/op
BenchmarkAtomicWatchPointerChannel-4          	627269310	         1.913 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/part	64.637s
PASS
ok  	github.com/cilium/statedb/reconciler	0.004s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 1.57 seconds (batch size 1000)
Throughput 637776.91 objects per second
506MB total allocated, 5007183 in-use objects, 231MB bytes in use

Force a concurrent status write during a retry and check that successful
results are committed and failed attempts remain scheduled. Cover both
pending and refreshing objects, with single and batch initial operations.

Also verify that committing the retry preserves the other reconciler's
status and does not overwrite a newer pending generation.

AIL: 3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
A retry can race with another reconciler's status write. Its result then
fails CompareAndSwap while the current status is Error, which the fallback
previously rejected. Failed attempts were left without a scheduled retry,
and successful attempts could leave the object stuck in Error.

Preserve the reconciliation request's status identifier across Done and
Error transitions. Accept matching Error and Refreshing statuses as well
as Pending when merging a result after a revision conflict. A new pending
request still has a different identifier and rejects the stale result.

Carry the merged object into subsequent retries so they preserve the
other reconciler's fields. Keep the original status passed to retry
operations so refresh hints survive retries.

AIL: 3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
Keep two objects failing across repeated retries and verify that the
reported low watermark stays at the oldest unresolved desired revision.
When that object recovers, the watermark must advance to the second
object's original revision and then clear once both have recovered.

Check that new pending work for a reused key starts a new watermark,
both after recovery and when superseding a failure. Exercise single and
batch initial operations with virtual time.

AIL: 3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
Each failed update commits an Error status at a new table revision.
Requeueing previously replaced the retry item's original revision with
that newer status revision, allowing the low watermark to advance even
though the original desired state had never reconciled successfully.

Set the original revision only when creating a retry item. Requeueing
continues to update the CAS revision, object, error and backoff while
preserving the oldest unresolved revision. Clearing the item allows a
new pending request for the same key to establish its own watermark.

AIL: 3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
@joamaki
joamaki force-pushed the pr/joamaki/fix-retry-status-conflict branch from 30a7719 to 65ceba1 Compare November 19, 2026 13:30
@joamaki
joamaki merged commit 860a59c into main Nov 19, 2026
1 check passed
@joamaki
joamaki deleted the pr/joamaki/fix-retry-status-conflict branch November 19, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants