Skip to content

Improve iteration over non-unique secondary indexes - #211

Merged
joamaki merged 2 commits into
mainfrom
pr/joamaki/non-unique-dedup-with-revision
Oct 15, 2026
Merged

joamaki merged 2 commits into
mainfrom
pr/joamaki/non-unique-dedup-with-revision

Conversation

@joamaki

@joamaki joamaki commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Non-unique secondary indexes may refer to the same object with multiple keys. When iterating over such index StateDB de-duplicates and makes sure the same object isn't emitted multiple times.

This PR improves the performance and reduces allocations of such queries by using the revision instead of the primary key to track visited objects and by using sync.Pool to reuse allocation of the visited hash map.

See commits for benchmark results. tl;dr:

Before:
BenchmarkDB_Prefix_SecondaryIndex-6        18668             63668 ns/op          15706356 objects/sec     74458 B/op         26 allocs/op

After:
BenchmarkDB_Prefix_SecondaryIndex-6        51578             23400 ns/op          42735799 objects/sec       195 B/op          6 allocs/op

AIL:3

Using the primary key is costly to figure out if an object was
already visited when iterating over a non-unique secondary index.
We can instead use the revision as that is also guaranteed unique.

Before:
BenchmarkDB_Prefix_SecondaryIndex-6 13364  89515 ns/op  11171333 objects/sec 108954 B/op  26 allocs/op

After:
BenchmarkDB_Prefix_SecondaryIndex-6 19248  63163 ns/op  15832108 objects/sec 74457 B/op   26 allocs/op

AIL:3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
@joamaki
joamaki requested a review from a team as a code owner October 13, 2026 15:35
@joamaki
joamaki requested review from bimmlerd and removed request for a team October 13, 2026 15:35
Prefix and lower-bound scans of non-unique indexes allocate a visited map for
tracking objects that appear under multiple secondary keys. Reuse these maps
across scans with sync.Pool, and discard maps that have grown beyond 4096
entries. Return maps after All completes or Next reaches the end of iteration.

Benchmarks: BenchmarkDB_Prefix_SecondaryIndex and
BenchmarkDB_LowerBound_SecondaryIndex, linux/arm64, Go 1.25.10,
benchtime=200ms, count=10. Results from benchstat:

                       │       old       │      new       │ vs base
                       │     sec/op      │     sec/op     │
Prefix                │ 72.53µs ± 51%  │ 23.69µs ± 3%  │ -67.34% (p=0.000 n=10)
LowerBound            │ 72.57µs ± 25%  │ 24.74µs ± 2%  │ -65.91% (p=0.000 n=10)

                       │       old       │      new       │ vs base
                       │      B/op       │      B/op      │
Prefix                │ 74457 ± 0%    │ 199 ± 4%     │ -99.73% (p=0.000 n=10)
LowerBound            │ 74745 ± 0%    │ 504 ± 0%     │ -99.33% (p=0.000 n=10)

                       │       old       │      new       │ vs base
                       │    allocs/op    │    allocs/op   │
Prefix                │ 26.000 ± 0%   │ 6.000 ± 0%   │ -76.92% (p=0.000 n=10)
LowerBound            │ 28.000 ± 0%   │ 8.000 ± 0%   │ -71.43% (p=0.000 n=10)

AIL:3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
@joamaki
joamaki force-pushed the pr/joamaki/non-unique-dedup-with-revision branch from 943a515 to e754238 Compare October 13, 2026 15:40
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
$ make
go build ./...
go: downloading go.yaml.in/yaml/v3 v3.0.4
go: downloading github.com/cilium/hive v1.0.4
go: downloading golang.org/x/time v0.15.0
go: downloading github.com/spf13/cobra v1.10.2
go: downloading github.com/spf13/pflag v1.0.10
go: downloading github.com/cilium/stream v0.0.1
go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de
go: downloading github.com/spf13/viper v1.18.2
go: downloading go.uber.org/dig v1.17.1
go: downloading golang.org/x/term v0.16.0
go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading golang.org/x/sys v0.17.0
go: downloading golang.org/x/tools v0.17.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.14.0
STATEDB_VALIDATE=1 go test ./... -cover -vet=all -test.count 1
go: downloading github.com/stretchr/testify v1.11.1
go: downloading go.uber.org/goleak v1.3.0
go: downloading golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
ok  	github.com/cilium/statedb	343.941s	coverage: 79.5% of statements
ok  	github.com/cilium/statedb/index	0.006s	coverage: 48.6% of statements
ok  	github.com/cilium/statedb/internal	0.015s	coverage: 54.1% of statements
ok  	github.com/cilium/statedb/lpm	4.518s	coverage: 76.4% of statements
ok  	github.com/cilium/statedb/part	80.732s	coverage: 86.5% of statements
ok  	github.com/cilium/statedb/reconciler	0.307s	coverage: 93.3% of statements
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
go test -race ./... -test.count 1
ok  	github.com/cilium/statedb	41.651s
ok  	github.com/cilium/statedb/index	1.016s
ok  	github.com/cilium/statedb/internal	1.024s
ok  	github.com/cilium/statedb/lpm	2.896s
ok  	github.com/cilium/statedb/part	38.434s
ok  	github.com/cilium/statedb/reconciler	1.406s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkDB_WriteTxn_1-4                      	  925006	      1285 ns/op	    778063 objects/sec	     632 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_10-4                     	 2180254	       551.8 ns/op	   1812332 objects/sec	     349 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_100-4                    	 2537392	       474.4 ns/op	   2108140 objects/sec	     317 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_1000-4                   	 2151728	       556.1 ns/op	   1798125 objects/sec	     318 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4     	 1492345	       801.6 ns/op	   1247440 objects/sec	     419 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_100Tables-4   	 1502701	       797.3 ns/op	    1088 B/op	       3 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_1Table-4      	 2063737	       581.8 ns/op	     200 B/op	       3 allocs/op
BenchmarkDB_NewWriteTxn-4                     	 2276582	       526.8 ns/op	     200 B/op	       3 allocs/op
BenchmarkDB_WriteTxnCommit100-4               	 1507288	       795.9 ns/op	    1072 B/op	       3 allocs/op
BenchmarkDB_NewReadTxn-4                      	548142546	         2.195 ns/op	       0 B/op	       0 allocs/op
BenchmarkDB_Modify-4                          	    2066	    573748 ns/op	   1742926 objects/sec	  342486 B/op	    6072 allocs/op
BenchmarkDB_GetInsert-4                       	    1849	    645012 ns/op	   1550359 objects/sec	  326482 B/op	    6072 allocs/op
BenchmarkDB_ListInsert-4                      	    1441	    822106 ns/op	   1216389 objects/sec	  486514 B/op	   11072 allocs/op
BenchmarkDB_RandomInsert-4                    	    2238	    529631 ns/op	   1888106 objects/sec	  318476 B/op	    5072 allocs/op
BenchmarkDB_RandomReplace-4                   	    1188	   1022706 ns/op	    977798 objects/sec	  432301 B/op	    8088 allocs/op
BenchmarkDB_SequentialInsert-4                	    2278	    518880 ns/op	   1927227 objects/sec	  318476 B/op	    5072 allocs/op
BenchmarkDB_SequentialInsert_Prefix-4         	     541	   2216847 ns/op	    451091 objects/sec	 2793440 B/op	   41795 allocs/op
BenchmarkDB_Changes_Baseline-4                	    1860	    629365 ns/op	   1588903 objects/sec	  388222 B/op	    6162 allocs/op
BenchmarkDB_Changes-4                         	     967	   1220639 ns/op	    819243 objects/sec	  604149 B/op	    9324 allocs/op
BenchmarkDB_RandomLookup-4                    	   27606	     43792 ns/op	  22835270 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_SequentialLookup-4                	   35379	     33990 ns/op	  29420403 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_Prefix_SecondaryIndex-4           	   27062	     44215 ns/op	  22616763 objects/sec	     194 B/op	       6 allocs/op
BenchmarkDB_LowerBound_SecondaryIndex-4       	   25346	     47420 ns/op	  21088295 objects/sec	     507 B/op	       8 allocs/op
BenchmarkDB_FullIteration_All-4               	     920	   1274286 ns/op	  78475333 objects/sec	     104 B/op	       4 allocs/op
BenchmarkDB_FullIteration_Prefix-4            	     850	   1460373 ns/op	  68475667 objects/sec	     136 B/op	       5 allocs/op
BenchmarkDB_FullIteration_Get-4               	     234	   5053056 ns/op	  19790005 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_Get_Secondary-4     	     116	  10229704 ns/op	   9775454 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_ReadTxnGet-4        	     238	   4838675 ns/op	  20666814 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4                	  932140	      1310 ns/op	        10.00 50th_µs	        14.00 90th_µs	        42.00 99th_µs	     756 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_1_DefaultMetrics-4       	  937092	      1340 ns/op	     632 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_1_ExpVarMetrics-4        	  717510	      1727 ns/op	     656 B/op	      15 allocs/op
BenchmarkDB_WriteTxn_100_LPMIndex-4           	  504783	      2396 ns/op	    417351 objects/sec	    1571 B/op	      35 allocs/op
BenchmarkDB_WriteTxn_1_LPMIndex-4             	  152044	     13756 ns/op	     72698 objects/sec	   13640 B/op	      64 allocs/op
BenchmarkDB_LPMIndex_Get-4                    	     338	   3455191 ns/op	   2894196 objects/sec	       0 B/op	       0 allocs/op
BenchmarkWatchSet_4-4                         	 2332461	       509.4 ns/op	     296 B/op	       4 allocs/op
BenchmarkWatchSet_16-4                        	  725600	      1602 ns/op	    1096 B/op	       5 allocs/op
BenchmarkWatchSet_128-4                       	   85315	     13983 ns/op	    8904 B/op	       5 allocs/op
BenchmarkWatchSet_1024-4                      	    8410	    138461 ns/op	   73743 B/op	       5 allocs/op
PASS
ok  	github.com/cilium/statedb	50.579s
PASS
ok  	github.com/cilium/statedb/index	0.004s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/internal
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_SortableMutex-4   	 6323605	       188.9 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/internal	1.199s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/lpm
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_txn_insert/batchSize=1-4         	    1837	    646221 ns/op	   1547458 objects/sec	  822406 B/op	   13975 allocs/op
Benchmark_txn_insert/batchSize=10-4        	    3279	    378550 ns/op	   2641656 objects/sec	  369195 B/op	    6668 allocs/op
Benchmark_txn_insert/batchSize=100-4       	    3316	    351245 ns/op	   2847017 objects/sec	  329613 B/op	    6027 allocs/op
Benchmark_txn_delete/batchSize=1-4         	    1551	    818626 ns/op	   1221559 objects/sec	 1270475 B/op	   13976 allocs/op
Benchmark_txn_delete/batchSize=10-4        	    3232	    369607 ns/op	   2705577 objects/sec	  356419 B/op	    5769 allocs/op
Benchmark_txn_delete/batchSize=100-4       	    3655	    331757 ns/op	   3014257 objects/sec	  270754 B/op	    5038 allocs/op
Benchmark_LPM_Lookup-4                     	    9312	    128244 ns/op	   7797650 objects/sec	       0 B/op	       0 allocs/op
Benchmark_LPM_All-4                        	  136833	      8880 ns/op	 112615465 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_Prefix-4                     	  131928	      9224 ns/op	 108411273 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_LowerBound-4                 	  247786	      4822 ns/op	 103691910 objects/sec	     288 B/op	       2 allocs/op
PASS
ok  	github.com/cilium/statedb/lpm	12.105s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_Set_Singleton_Create-4              	21451015	        55.57 ns/op	      24 B/op	       1 allocs/op
Benchmark_Set_Singleton_Has-4                 	100000000	        11.38 ns/op	       0 B/op	       0 allocs/op
Benchmark_StringMap_Txn_Insert-4              	    7249	    153117 ns/op	   6530956 items/sec	   97737 B/op	    1305 allocs/op
Benchmark_Uint64Map_Random-4                  	    1848	    645128 ns/op	   1550081 items/sec	 1320548 B/op	    6022 allocs/op
Benchmark_Uint64Map_Sequential-4              	    2030	    614516 ns/op	   1627296 items/sec	 1702578 B/op	    5752 allocs/op
Benchmark_Uint64Map_Sequential_Insert-4       	    2162	    561299 ns/op	   1781582 items/sec	 1694573 B/op	    4751 allocs/op
Benchmark_Uint64Map_Sequential_Txn_Insert-4   	   10000	    121626 ns/op	   8221948 items/sec	   89952 B/op	    2030 allocs/op
Benchmark_Uint64Map_Random_Insert-4           	    2145	    571352 ns/op	   1750235 items/sec	 1314667 B/op	    5030 allocs/op
Benchmark_Uint64Map_Random_Txn_Insert-4       	    5968	    192969 ns/op	   5182178 items/sec	  116616 B/op	    2409 allocs/op
Benchmark_Insert_RootOnlyWatch-4              	    8895	    121615 ns/op	   8222694 objects/sec	   75040 B/op	    2035 allocs/op
Benchmark_Insert-4                            	   10000	    119836 ns/op	   8344737 objects/sec	   75584 B/op	    2046 allocs/op
Benchmark_WatchReplace-4                      	   10000	    107814 ns/op	   9275239 objects/sec	   57306 B/op	    1006 allocs/op
Benchmark_Modify-4                            	   12114	     99063 ns/op	  10094601 objects/sec	   58056 B/op	    1007 allocs/op
Benchmark_GetInsert-4                         	   10000	    117277 ns/op	   8526800 objects/sec	   58056 B/op	    1007 allocs/op
Benchmark_Replace-4                           	15859034	        75.25 ns/op	  13289673 objects/sec	      48 B/op	       1 allocs/op
Benchmark_Replace_RootOnlyWatch-4             	 7105677	       149.0 ns/op	   6711369 objects/sec	      95 B/op	       1 allocs/op
Benchmark_txn_1-4                             	 7682929	       153.8 ns/op	   6503799 objects/sec	      64 B/op	       3 allocs/op
Benchmark_txn_10-4                            	 9757281	       125.5 ns/op	   7966522 objects/sec	      76 B/op	       2 allocs/op
Benchmark_txn_100-4                           	10572033	       113.4 ns/op	   8819559 objects/sec	      67 B/op	       2 allocs/op
Benchmark_txn_1000-4                          	 9162811	       128.4 ns/op	   7786996 objects/sec	      65 B/op	       2 allocs/op
Benchmark_txn_delete_1-4                      	 4600780	       251.0 ns/op	   3983290 objects/sec	     632 B/op	       3 allocs/op
Benchmark_txn_delete_10-4                     	 8990119	       128.7 ns/op	   7767543 objects/sec	     103 B/op	       1 allocs/op
Benchmark_txn_delete_100-4                    	10959636	       106.9 ns/op	   9353469 objects/sec	      35 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4                   	11434470	       107.6 ns/op	   9291259 objects/sec	      28 B/op	       1 allocs/op
BenchmarkDeleteParentCache/depth_8-4          	 4354657	       274.9 ns/op	      48 B/op	       1 allocs/op
BenchmarkDeleteParentCache/depth_100-4        	  460184	      2604 ns/op	      48 B/op	       1 allocs/op
Benchmark_Get-4                               	   60381	     19764 ns/op	  50597744 objects/sec	       0 B/op	       0 allocs/op
Benchmark_GetWatch-4                          	   47290	     25377 ns/op	  39406333 objects/sec	       0 B/op	       0 allocs/op
Benchmark_All-4                               	  137886	      8488 ns/op	 117815681 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_All-4                      	  157831	      7596 ns/op	 131642480 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_Next-4                     	  150411	      8004 ns/op	 124944513 objects/sec	     896 B/op	       1 allocs/op
Benchmark_LowerBound-4                        	   10000	    110813 ns/op	   9024189 objects/sec	  104000 B/op	    2000 allocs/op
Benchmark_LowerBound_Random-4                 	    6968	    155533 ns/op	   6429516 objects/sec	   96384 B/op	    1002 allocs/op
Benchmark_Hashmap_Insert-4                    	   14228	     84252 ns/op	  11869142 objects/sec	   74264 B/op	      20 allocs/op
Benchmark_Hashmap_Get_Uint64-4                	  134469	      8918 ns/op	 112127975 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4                 	  109810	     10927 ns/op	  91513290 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Delete_Random-4                     	      48	  24891220 ns/op	   4017481 objects/sec	 2539541 B/op	  102756 allocs/op
Benchmark_find16-4                            	225065844	         5.317 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex16-4                       	85120888	        13.77 ns/op	       0 B/op	       0 allocs/op
Benchmark_find64-4                            	295795668	         4.054 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_hit-4                   	295931631	         4.053 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_miss-4                  	294855267	         4.061 ns/op	       0 B/op	       0 allocs/op
Benchmark_find4-4                             	427338337	         2.811 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex4-4                        	319727314	         3.751 ns/op	       0 B/op	       0 allocs/op
BenchmarkSmallWriteTxn/updates_1-4            	  413175	      2777 ns/op	    3529 B/op	       4 allocs/op
BenchmarkSmallWriteTxn/updates_2-4            	  299924	      3787 ns/op	    4742 B/op	       6 allocs/op
BenchmarkSmallWriteTxn/updates_4-4            	  198404	      5907 ns/op	    7155 B/op	      11 allocs/op
BenchmarkSmallWriteTxn/updates_8-4            	  118963	     10150 ns/op	   11927 B/op	      20 allocs/op
BenchmarkSmallWriteTxn/updates_16-4           	   63620	     18630 ns/op	   21289 B/op	      38 allocs/op
BenchmarkFreshWriteTxn-4                      	 5423113	       221.4 ns/op	     185 B/op	       6 allocs/op
BenchmarkAtomicWatchPointerFirstChannel-4     	19543236	        61.45 ns/op	     120 B/op	       2 allocs/op
BenchmarkAtomicWatchPointerChannel-4          	481079133	         2.494 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/part	63.530s
PASS
ok  	github.com/cilium/statedb/reconciler	0.005s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 1.68 seconds (batch size 1000)
Throughput 595273.21 objects per second
506MB total allocated, 5007214 in-use objects, 231MB bytes in use

Comment thread part_index.go
@@ -662,7 +659,7 @@ func (it *nonUniquePartIterator) All(yield func([]byte, object) bool) {
//
// This of course makes iterating over a non-unique index with a prefix
// (or lowerbound search) about 20x slower than normal!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is it still 20x slower? 🙃

@joamaki
joamaki merged commit a74dbdc into main Oct 15, 2026
1 check passed
@joamaki
joamaki deleted the pr/joamaki/non-unique-dedup-with-revision branch October 15, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants