Skip to content

part: represent compressed prefixes as full-key ranges - #213

Draft
joamaki wants to merge 1 commit into
mainfrom
pr/joamaki/compact-leaf
Draft

joamaki wants to merge 1 commit into
mainfrom
pr/joamaki/compact-leaf

Conversation

@joamaki

@joamaki joamaki commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Reduce the size of the leaf node by storing a full key pointer and prefix start/end offsets in every node. This shrinks leaf[T] from 64 bytes to 48 bytes while the header struct size remains the same.

With room in the padding also cache prefix length (up to 256 bytes) and the node key.

Benchmarks versus a74dbdc: linux/arm64, Go 1.25.10, cpu=1, benchtime=200ms, count=10. Benchstat results; RandomReplace updates 1000 objects per operation:

                              old B/op  new B/op    change
WriteTxn_1                         632       600    -5.06%
WriteTxn_100_SecondaryIndex         419       371   -11.46%
RandomReplace                  422.2Ki   375.2Ki   -11.14%
PrefixMerge                        432       288   -33.33%

PrefixMerge allocations: 7 -> 6 per operation. Write and merge timings are statistically unchanged in these runs.

Interleaved lookup benchmarks show the read-path tradeoff:

                         old sec/op  new sec/op    change
RandomLookup                 30.07us     30.79us    +2.39%
SequentialLookup             22.21us     23.11us    +4.06%

reconciler/benchmark versus a74dbdc, same platform and Go version: GOMAXPROCS=6, -quiet -objects=1000000 -batchsize=1000 -incrbatchsize=1000, ten runs each with alternating execution order. Medians (memory rounded down to MiB by the benchmark):

                              old       new       change
Throughput (objects/sec)   779.6k    789.9k    ~ (p=0.280)
Total allocated (MiB)         506       445       -12.06%
Live heap after GC (MiB)      231       200       -13.42%

pkg/loadbalancer/benchmark from cilium/cilium:

Before:
Min: Allocated 356830kB in total, 1589696 objects /  66402kB still reachable (per service:  31 objs,  7307B alloc,  1359B in-use)

After:
Min: Allocated 347871kB in total, 1565079 objects /  61223kB still reachable (per service:  31 objs,  7124B alloc,  1253B in-use)

AIL:3

Reduce the size of the leaf node by storing a full key pointer and
prefix start/end offsets in every node. This shrinks `leaf[T]` from
64 bytes to 48 bytes while the header struct size remains the same.

With room in the padding also cache prefix length (up to 256 bytes)
and the node key.

Benchmarks versus a74dbdc: linux/arm64, Go 1.25.10, cpu=1,
benchtime=200ms, count=10. Benchstat results; RandomReplace updates
1000 objects per operation:

                              old B/op  new B/op    change
WriteTxn_1                         632       600    -5.06%
WriteTxn_100_SecondaryIndex         419       371   -11.46%
RandomReplace                  422.2Ki   375.2Ki   -11.14%
PrefixMerge                        432       288   -33.33%

PrefixMerge allocations: 7 -> 6 per operation. Write and merge timings
are statistically unchanged in these runs.

Interleaved lookup benchmarks show the read-path tradeoff:
                         old sec/op  new sec/op    change
RandomLookup                 30.07us     30.79us    +2.39%
SequentialLookup             22.21us     23.11us    +4.06%

reconciler/benchmark versus a74dbdc, same platform and Go version:
GOMAXPROCS=6, -quiet -objects=1000000 -batchsize=1000
-incrbatchsize=1000, ten runs each with alternating execution order.
Medians (memory rounded down to MiB by the benchmark):

                              old       new       change
Throughput (objects/sec)   779.6k    789.9k    ~ (p=0.280)
Total allocated (MiB)         506       445       -12.06%
Live heap after GC (MiB)      231       200       -13.42%

AIL:3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
$ make
go build ./...
go: downloading go.yaml.in/yaml/v3 v3.0.4
go: downloading github.com/cilium/hive v1.0.4
go: downloading golang.org/x/time v0.15.0
go: downloading github.com/spf13/cobra v1.10.2
go: downloading github.com/spf13/pflag v1.0.10
go: downloading github.com/cilium/stream v0.0.1
go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de
go: downloading github.com/spf13/viper v1.18.2
go: downloading go.uber.org/dig v1.17.1
go: downloading golang.org/x/term v0.16.0
go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading golang.org/x/sys v0.17.0
go: downloading golang.org/x/tools v0.17.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.14.0
STATEDB_VALIDATE=1 go test ./... -cover -vet=all -test.count 1
go: downloading github.com/stretchr/testify v1.11.1
go: downloading go.uber.org/goleak v1.3.0
go: downloading golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
ok  	github.com/cilium/statedb	337.661s	coverage: 79.5% of statements
ok  	github.com/cilium/statedb/index	0.008s	coverage: 48.6% of statements
ok  	github.com/cilium/statedb/internal	0.012s	coverage: 54.1% of statements
ok  	github.com/cilium/statedb/lpm	3.954s	coverage: 76.1% of statements
ok  	github.com/cilium/statedb/part	84.787s	coverage: 86.7% of statements
ok  	github.com/cilium/statedb/reconciler	0.302s	coverage: 93.3% of statements
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
go test -race ./... -test.count 1
ok  	github.com/cilium/statedb	39.195s
ok  	github.com/cilium/statedb/index	1.021s
ok  	github.com/cilium/statedb/internal	1.034s
ok  	github.com/cilium/statedb/lpm	2.904s
ok  	github.com/cilium/statedb/part	37.938s
ok  	github.com/cilium/statedb/reconciler	1.399s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkDB_WriteTxn_1-4                      	 1012426	      1185 ns/op	    843814 objects/sec	     600 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_10-4                     	 2206195	       543.3 ns/op	   1840495 objects/sec	     317 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_100-4                    	 2561696	       467.4 ns/op	   2139334 objects/sec	     285 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_1000-4                   	 2383324	       498.9 ns/op	   2004594 objects/sec	     286 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4     	 1403050	       766.9 ns/op	   1303959 objects/sec	     371 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_100Tables-4   	 1656992	       726.6 ns/op	    1088 B/op	       3 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_1Table-4      	 2145640	       557.7 ns/op	     200 B/op	       3 allocs/op
BenchmarkDB_NewWriteTxn-4                     	 2315709	       517.5 ns/op	     200 B/op	       3 allocs/op
BenchmarkDB_WriteTxnCommit100-4               	 1658212	       722.8 ns/op	    1072 B/op	       3 allocs/op
BenchmarkDB_NewReadTxn-4                      	548142811	         2.190 ns/op	       0 B/op	       0 allocs/op
BenchmarkDB_Modify-4                          	    2100	    569411 ns/op	   1756201 objects/sec	  310323 B/op	    6068 allocs/op
BenchmarkDB_GetInsert-4                       	    1862	    638330 ns/op	   1566588 objects/sec	  294319 B/op	    6068 allocs/op
BenchmarkDB_ListInsert-4                      	    1464	    815114 ns/op	   1226823 objects/sec	  454355 B/op	   11068 allocs/op
BenchmarkDB_RandomInsert-4                    	    2310	    525470 ns/op	   1903059 objects/sec	  286311 B/op	    5068 allocs/op
BenchmarkDB_RandomReplace-4                   	    1208	    992455 ns/op	   1007602 objects/sec	  384129 B/op	    8084 allocs/op
BenchmarkDB_SequentialInsert-4                	    2418	    507830 ns/op	   1969162 objects/sec	  286311 B/op	    5068 allocs/op
BenchmarkDB_SequentialInsert_Prefix-4         	     538	   2216767 ns/op	    451107 objects/sec	 2761309 B/op	   41795 allocs/op
BenchmarkDB_Changes_Baseline-4                	    1965	    618790 ns/op	   1616058 objects/sec	  355841 B/op	    6151 allocs/op
BenchmarkDB_Changes-4                         	    1004	   1330972 ns/op	    751330 objects/sec	  539423 B/op	    9302 allocs/op
BenchmarkDB_RandomLookup-4                    	   26583	     44747 ns/op	  22347852 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_SequentialLookup-4                	   34671	     34629 ns/op	  28877645 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_Prefix_SecondaryIndex-4           	   27416	     43699 ns/op	  22883957 objects/sec	     194 B/op	       6 allocs/op
BenchmarkDB_LowerBound_SecondaryIndex-4       	   24760	     47910 ns/op	  20872558 objects/sec	     502 B/op	       8 allocs/op
BenchmarkDB_FullIteration_All-4               	    1018	   1230782 ns/op	  81249131 objects/sec	     104 B/op	       4 allocs/op
BenchmarkDB_FullIteration_Prefix-4            	     972	   1210699 ns/op	  82596933 objects/sec	     136 B/op	       5 allocs/op
BenchmarkDB_FullIteration_Get-4               	     286	   4176180 ns/op	  23945330 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_Get_Secondary-4     	     138	   8667860 ns/op	  11536873 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_ReadTxnGet-4        	     277	   4310413 ns/op	  23199635 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4                	  947595	      1263 ns/op	        10.00 50th_µs	        12.00 90th_µs	        41.00 99th_µs	     691 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_1_DefaultMetrics-4       	  982678	      1205 ns/op	     600 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_1_ExpVarMetrics-4        	  765878	      1583 ns/op	     624 B/op	      15 allocs/op
BenchmarkDB_WriteTxn_100_LPMIndex-4           	  534813	      2333 ns/op	    428612 objects/sec	    1539 B/op	      35 allocs/op
BenchmarkDB_WriteTxn_1_LPMIndex-4             	  157272	     13496 ns/op	     74098 objects/sec	   13702 B/op	      65 allocs/op
BenchmarkDB_LPMIndex_Get-4                    	     398	   3000989 ns/op	   3332235 objects/sec	       0 B/op	       0 allocs/op
BenchmarkWatchSet_4-4                         	 2313274	       513.4 ns/op	     296 B/op	       4 allocs/op
BenchmarkWatchSet_16-4                        	  748930	      1549 ns/op	    1096 B/op	       5 allocs/op
BenchmarkWatchSet_128-4                       	   90130	     13437 ns/op	    8904 B/op	       5 allocs/op
BenchmarkWatchSet_1024-4                      	    8792	    134426 ns/op	   73743 B/op	       5 allocs/op
PASS
ok  	github.com/cilium/statedb	50.745s
PASS
ok  	github.com/cilium/statedb/index	0.004s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/internal
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_SortableMutex-4   	 6382561	       187.4 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/internal	1.201s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/lpm
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_txn_insert/batchSize=1-4         	    1760	    649539 ns/op	   1539554 objects/sec	  822403 B/op	   13975 allocs/op
Benchmark_txn_insert/batchSize=10-4        	    3028	    383709 ns/op	   2606143 objects/sec	  369193 B/op	    6668 allocs/op
Benchmark_txn_insert/batchSize=100-4       	    3355	    354882 ns/op	   2817840 objects/sec	  329613 B/op	    6027 allocs/op
Benchmark_txn_delete/batchSize=1-4         	    1544	    757448 ns/op	   1320223 objects/sec	 1270473 B/op	   13976 allocs/op
Benchmark_txn_delete/batchSize=10-4        	    3301	    383367 ns/op	   2608465 objects/sec	  356418 B/op	    5769 allocs/op
Benchmark_txn_delete/batchSize=100-4       	    3739	    324611 ns/op	   3080614 objects/sec	  270754 B/op	    5038 allocs/op
Benchmark_LPM_Lookup-4                     	    9172	    130608 ns/op	   7656502 objects/sec	       0 B/op	       0 allocs/op
Benchmark_LPM_All-4                        	  141510	      9278 ns/op	 107784354 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_Prefix-4                     	  132949	      9159 ns/op	 109183515 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_LowerBound-4                 	  245887	      4836 ns/op	 103395703 objects/sec	     288 B/op	       2 allocs/op
PASS
ok  	github.com/cilium/statedb/lpm	12.076s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkPrefixMerge-4                        	 2800934	       429.5 ns/op	     288 B/op	       6 allocs/op
Benchmark_Set_Singleton_Create-4              	20961394	        54.97 ns/op	      24 B/op	       1 allocs/op
Benchmark_Set_Singleton_Has-4                 	100000000	        11.34 ns/op	       0 B/op	       0 allocs/op
Benchmark_StringMap_Txn_Insert-4              	    7798	    144922 ns/op	   6900265 items/sec	   80137 B/op	    1305 allocs/op
Benchmark_Uint64Map_Random-4                  	    1874	    631409 ns/op	   1583761 items/sec	 1302453 B/op	    6028 allocs/op
Benchmark_Uint64Map_Sequential-4              	    1986	    592863 ns/op	   1686730 items/sec	 1686514 B/op	    5752 allocs/op
Benchmark_Uint64Map_Sequential_Insert-4       	    2234	    539696 ns/op	   1852893 items/sec	 1678509 B/op	    4751 allocs/op
Benchmark_Uint64Map_Sequential_Txn_Insert-4   	    9774	    125295 ns/op	   7981187 items/sec	   73888 B/op	    2030 allocs/op
Benchmark_Uint64Map_Random_Insert-4           	    2095	    561690 ns/op	   1780341 items/sec	 1295715 B/op	    5044 allocs/op
Benchmark_Uint64Map_Random_Txn_Insert-4       	    6276	    190734 ns/op	   5242911 items/sec	   98635 B/op	    2410 allocs/op
Benchmark_Insert_RootOnlyWatch-4              	   10000	    117503 ns/op	   8510428 objects/sec	   58960 B/op	    2035 allocs/op
Benchmark_Insert-4                            	   10000	    117268 ns/op	   8527448 objects/sec	   59504 B/op	    2046 allocs/op
Benchmark_WatchReplace-4                      	   12536	     95660 ns/op	  10453642 objects/sec	   41305 B/op	    1006 allocs/op
Benchmark_Modify-4                            	   12639	     94997 ns/op	  10526654 objects/sec	   42056 B/op	    1007 allocs/op
Benchmark_GetInsert-4                         	    9811	    122056 ns/op	   8192933 objects/sec	   42056 B/op	    1007 allocs/op
Benchmark_Replace-4                           	16686122	        71.53 ns/op	  13981010 objects/sec	      32 B/op	       1 allocs/op
Benchmark_Replace_RootOnlyWatch-4             	 6913635	       173.7 ns/op	   5757116 objects/sec	      80 B/op	       1 allocs/op
Benchmark_txn_1-4                             	 7780539	       167.7 ns/op	   5963173 objects/sec	      48 B/op	       3 allocs/op
Benchmark_txn_10-4                            	 9823430	       110.6 ns/op	   9045085 objects/sec	      60 B/op	       2 allocs/op
Benchmark_txn_100-4                           	12124132	        96.65 ns/op	  10346524 objects/sec	      51 B/op	       2 allocs/op
Benchmark_txn_1000-4                          	10676995	       112.0 ns/op	   8930604 objects/sec	      49 B/op	       2 allocs/op
Benchmark_txn_delete_1-4                      	 4992835	       236.5 ns/op	   4227457 objects/sec	     632 B/op	       3 allocs/op
Benchmark_txn_delete_10-4                     	10815831	       110.2 ns/op	   9071335 objects/sec	     100 B/op	       1 allocs/op
Benchmark_txn_delete_100-4                    	12627682	        92.32 ns/op	  10831694 objects/sec	      35 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4                   	13037852	        93.78 ns/op	  10663087 objects/sec	      28 B/op	       1 allocs/op
BenchmarkDeleteParentCache/depth_8-4          	 4110040	       290.9 ns/op	      32 B/op	       1 allocs/op
BenchmarkDeleteParentCache/depth_100-4        	  418074	      2824 ns/op	      32 B/op	       1 allocs/op
Benchmark_Get-4                               	   60242	     20098 ns/op	  49756331 objects/sec	       0 B/op	       0 allocs/op
Benchmark_GetWatch-4                          	   42559	     27268 ns/op	  36672485 objects/sec	       0 B/op	       0 allocs/op
Benchmark_All-4                               	  123138	     10279 ns/op	  97289500 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_All-4                      	  151135	      8379 ns/op	 119343080 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_Next-4                     	  159156	      7467 ns/op	 133928869 objects/sec	     896 B/op	       1 allocs/op
Benchmark_LowerBound-4                        	   10000	    107242 ns/op	   9324737 objects/sec	  104000 B/op	    2000 allocs/op
Benchmark_LowerBound_Random-4                 	    7486	    159873 ns/op	   6254954 objects/sec	   96384 B/op	    1002 allocs/op
Benchmark_Hashmap_Insert-4                    	   14845	     80816 ns/op	  12373850 objects/sec	   74264 B/op	      20 allocs/op
Benchmark_Hashmap_Get_Uint64-4                	  134619	      8938 ns/op	 111875924 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4                 	  108256	     11091 ns/op	  90162632 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Delete_Random-4                     	      78	  16341071 ns/op	   6119550 objects/sec	 2530108 B/op	  102362 allocs/op
Benchmark_find16-4                            	225228301	         5.318 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex16-4                       	100000000	        11.25 ns/op	       0 B/op	       0 allocs/op
Benchmark_find64-4                            	274352971	         4.377 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_hit-4                   	274068309	         4.377 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_miss-4                  	256256490	         4.683 ns/op	       0 B/op	       0 allocs/op
Benchmark_find4-4                             	421970191	         2.852 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex4-4                        	275010360	         4.364 ns/op	       0 B/op	       0 allocs/op
BenchmarkSmallWriteTxn/updates_1-4            	  438772	      2707 ns/op	    3513 B/op	       4 allocs/op
BenchmarkSmallWriteTxn/updates_2-4            	  325777	      3760 ns/op	    4710 B/op	       6 allocs/op
BenchmarkSmallWriteTxn/updates_4-4            	  209287	      5740 ns/op	    7091 B/op	      11 allocs/op
BenchmarkSmallWriteTxn/updates_8-4            	  120405	      9745 ns/op	   11799 B/op	      20 allocs/op
BenchmarkSmallWriteTxn/updates_16-4           	   64688	     18054 ns/op	   21033 B/op	      38 allocs/op
BenchmarkFreshWriteTxn-4                      	 5635270	       208.1 ns/op	     169 B/op	       6 allocs/op
BenchmarkAtomicWatchPointerFirstChannel-4     	20343154	        58.66 ns/op	     120 B/op	       2 allocs/op
BenchmarkAtomicWatchPointerChannel-4          	549488230	         2.182 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/part	65.150s
PASS
ok  	github.com/cilium/statedb/reconciler	0.005s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 1.84 seconds (batch size 1000)
Throughput 542860.07 objects per second
445MB total allocated, 5007183 in-use objects, 200MB bytes in use

@thejesh23

This comment was marked as low quality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants