Skip to content

Proposal: Query pipeline - #212

Draft
joamaki wants to merge 2 commits into
mainfrom
pr/joamaki/pipeline
Draft

joamaki wants to merge 2 commits into
mainfrom
pr/joamaki/pipeline

Conversation

@joamaki

@joamaki joamaki commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

With Go v1.27 we can now do generic methods. This would allow for something like this:

type UserOrder struct {
  User  *User
  Order *Order
}

rows := statedb.From(users, usersByRegion.Query(region)).
  Where(func(user *User) bool { return user.Active }).
  JoinList(orders, func(user *User) statedb.Query[*Order] {
    return ordersByUser.Query(user.ID)
  }, func(user *User, order *Order) UserOrder {
    return UserOrder{User: user, Order: order}
  }).
  Select(func(row UserOrder) string {
    return row.User.Name + ": " + row.Order.ID
  })

watchSet := statedb.NewWatchSet()
for description := range rows.AllWatch(txn, watchSet) {
  // ...
}

// After consuming the full result sequence, wait for any source or join
// query to be invalidated.
_, err := watchSet.Wait(ctx, 0)

Unclear if we've got a use-case for it though.

Signed-off-by: Jussi Maki <jussi@isovalent.com>
AIL: 3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
@github-actions

Copy link
Copy Markdown
$ make
go build ./...
go: downloading go1.27.0 (linux/amd64)
go: downloading go.yaml.in/yaml/v3 v3.0.4
go: downloading github.com/cilium/hive v1.0.4
go: downloading golang.org/x/time v0.15.0
go: downloading github.com/spf13/cobra v1.10.2
go: downloading github.com/spf13/pflag v1.0.10
go: downloading github.com/cilium/stream v0.0.1
go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de
go: downloading github.com/spf13/viper v1.18.2
go: downloading go.uber.org/dig v1.17.1
go: downloading golang.org/x/term v0.16.0
go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading golang.org/x/sys v0.17.0
go: downloading golang.org/x/tools v0.17.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.14.0
STATEDB_VALIDATE=1 go test ./... -cover -vet=all -test.count 1
go: downloading github.com/stretchr/testify v1.11.1
go: downloading go.uber.org/goleak v1.3.0
go: downloading golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
ok  	github.com/cilium/statedb	310.069s	coverage: 81.0% of statements
ok  	github.com/cilium/statedb/index	0.006s	coverage: 48.6% of statements
ok  	github.com/cilium/statedb/internal	0.014s	coverage: 51.3% of statements
ok  	github.com/cilium/statedb/lpm	3.835s	coverage: 79.6% of statements
ok  	github.com/cilium/statedb/part	76.087s	coverage: 88.1% of statements
ok  	github.com/cilium/statedb/reconciler	0.288s	coverage: 95.3% of statements
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
go test -race ./... -test.count 1
ok  	github.com/cilium/statedb	36.987s
ok  	github.com/cilium/statedb/index	1.013s
ok  	github.com/cilium/statedb/internal	1.031s
ok  	github.com/cilium/statedb/lpm	2.683s
ok  	github.com/cilium/statedb/part	39.953s
ok  	github.com/cilium/statedb/reconciler	1.355s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkDB_WriteTxn_1-4                      	  974185	      1227 ns/op	    815095 objects/sec	     632 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_10-4                     	 2161594	       539.0 ns/op	   1855228 objects/sec	     349 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_100-4                    	 2569803	       478.7 ns/op	   2089072 objects/sec	     317 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_1000-4                   	 2413452	       497.3 ns/op	   2010816 objects/sec	     318 B/op	       5 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4     	 1621435	       790.6 ns/op	   1264921 objects/sec	     419 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_100Tables-4   	 1463067	       820.6 ns/op	    1088 B/op	       3 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_1Table-4      	 2078714	       577.6 ns/op	     200 B/op	       3 allocs/op
BenchmarkDB_NewWriteTxn-4                     	 2202991	       544.4 ns/op	     200 B/op	       3 allocs/op
BenchmarkDB_WriteTxnCommit100-4               	 1458914	       819.7 ns/op	    1072 B/op	       3 allocs/op
BenchmarkDB_NewReadTxn-4                      	1000000000	         0.7572 ns/op	       0 B/op	       0 allocs/op
BenchmarkDB_Modify-4                          	    2173	    546358 ns/op	   1830302 objects/sec	  342488 B/op	    6072 allocs/op
BenchmarkDB_GetInsert-4                       	    1952	    598932 ns/op	   1669639 objects/sec	  326480 B/op	    6072 allocs/op
BenchmarkDB_ListInsert-4                      	    1501	    791333 ns/op	   1263690 objects/sec	  486535 B/op	   11073 allocs/op
BenchmarkDB_RandomInsert-4                    	    2426	    500871 ns/op	   1996521 objects/sec	  318476 B/op	    5072 allocs/op
BenchmarkDB_RandomReplace-4                   	    1262	    948143 ns/op	   1054693 objects/sec	  432297 B/op	    8088 allocs/op
BenchmarkDB_SequentialInsert-4                	    2436	    490287 ns/op	   2039621 objects/sec	  318475 B/op	    5072 allocs/op
BenchmarkDB_SequentialInsert_Prefix-4         	     592	   2039424 ns/op	    490334 objects/sec	 2793439 B/op	   41795 allocs/op
BenchmarkDB_Changes_Baseline-4                	    2054	    574847 ns/op	   1739593 objects/sec	  388221 B/op	    6162 allocs/op
BenchmarkDB_Changes-4                         	    1056	   1138834 ns/op	    878091 objects/sec	  603832 B/op	    9323 allocs/op
BenchmarkDB_RandomLookup-4                    	   29659	     40441 ns/op	  24727343 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_SequentialLookup-4                	   35936	     33425 ns/op	  29917522 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_Prefix_SecondaryIndex-4           	   31573	     38045 ns/op	  26284969 objects/sec	     194 B/op	       6 allocs/op
BenchmarkDB_LowerBound_SecondaryIndex-4       	   29774	     40364 ns/op	  24774493 objects/sec	     498 B/op	       8 allocs/op
BenchmarkDB_FullIteration_All-4               	    1170	   1009094 ns/op	  99098794 objects/sec	     104 B/op	       4 allocs/op
BenchmarkDB_FullIteration_Prefix-4            	    1197	    965230 ns/op	 103602280 objects/sec	     136 B/op	       5 allocs/op
BenchmarkDB_FullIteration_Get-4               	     304	   3911119 ns/op	  25568129 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_Get_Secondary-4     	     139	   8533872 ns/op	  11718010 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_ReadTxnGet-4        	     298	   4022512 ns/op	  24860086 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4                	  796311	      1266 ns/op	        10.00 50th_µs	        14.00 90th_µs	        61.00 99th_µs	     755 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_1_DefaultMetrics-4       	  927400	      1282 ns/op	     632 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_1_ExpVarMetrics-4        	  740486	      1642 ns/op	     656 B/op	      15 allocs/op
BenchmarkDB_WriteTxn_100_LPMIndex-4           	  553560	      1941 ns/op	    515213 objects/sec	    1571 B/op	      35 allocs/op
BenchmarkDB_WriteTxn_1_LPMIndex-4             	  157450	     11413 ns/op	     87617 objects/sec	   13738 B/op	      65 allocs/op
BenchmarkDB_LPMIndex_Get-4                    	     460	   2553185 ns/op	   3916676 objects/sec	       0 B/op	       0 allocs/op
BenchmarkWatchSet_4-4                         	 2127003	       557.0 ns/op	     296 B/op	       4 allocs/op
BenchmarkWatchSet_16-4                        	  755037	      1583 ns/op	    1096 B/op	       5 allocs/op
BenchmarkWatchSet_128-4                       	   90128	     13359 ns/op	    8904 B/op	       5 allocs/op
BenchmarkWatchSet_1024-4                      	    8820	    132978 ns/op	   73743 B/op	       5 allocs/op
PASS
ok  	github.com/cilium/statedb	48.423s
PASS
ok  	github.com/cilium/statedb/index	0.003s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/internal
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_SortableMutex-4   	 6372267	       187.8 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/internal	1.201s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/lpm
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_txn_insert/batchSize=1-4         	    2689	    458524 ns/op	   2180911 objects/sec	  774427 B/op	   12975 allocs/op
Benchmark_txn_insert/batchSize=10-4        	    4192	    289701 ns/op	   3451833 objects/sec	  364400 B/op	    6568 allocs/op
Benchmark_txn_insert/batchSize=100-4       	    4383	    274332 ns/op	   3645216 objects/sec	  329136 B/op	    6017 allocs/op
Benchmark_txn_delete/batchSize=1-4         	    1858	    632675 ns/op	   1580591 objects/sec	 1222470 B/op	   12976 allocs/op
Benchmark_txn_delete/batchSize=10-4        	    3978	    309868 ns/op	   3227185 objects/sec	  351618 B/op	    5669 allocs/op
Benchmark_txn_delete/batchSize=100-4       	    4347	    276972 ns/op	   3610477 objects/sec	  270273 B/op	    5028 allocs/op
Benchmark_LPM_Lookup-4                     	   10000	    117340 ns/op	   8522250 objects/sec	       0 B/op	       0 allocs/op
Benchmark_LPM_All-4                        	  146700	      8159 ns/op	 122567287 objects/sec	       0 B/op	       0 allocs/op
Benchmark_LPM_Prefix-4                     	  144963	      8233 ns/op	 121459680 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_LowerBound-4                 	  259263	      4710 ns/op	 106161764 objects/sec	     288 B/op	       2 allocs/op
PASS
ok  	github.com/cilium/statedb/lpm	12.060s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_Set_Singleton_Create-4              	25771888	        45.66 ns/op	      24 B/op	       1 allocs/op
Benchmark_Set_Singleton_Has-4                 	100000000	        10.91 ns/op	       0 B/op	       0 allocs/op
Benchmark_StringMap_Txn_Insert-4              	    9128	    129497 ns/op	   7722215 items/sec	   97737 B/op	    1305 allocs/op
Benchmark_Uint64Map_Random-4                  	    1983	    612100 ns/op	   1633720 items/sec	 1320987 B/op	    6046 allocs/op
Benchmark_Uint64Map_Sequential-4              	    2047	    588885 ns/op	   1698125 items/sec	 1702570 B/op	    5752 allocs/op
Benchmark_Uint64Map_Sequential_Insert-4       	    2307	    598019 ns/op	   1672188 items/sec	 1694566 B/op	    4751 allocs/op
Benchmark_Uint64Map_Sequential_Txn_Insert-4   	   10000	    106957 ns/op	   9349547 items/sec	   89952 B/op	    2030 allocs/op
Benchmark_Uint64Map_Random_Insert-4           	    2104	    541845 ns/op	   1845545 items/sec	 1313085 B/op	    5017 allocs/op
Benchmark_Uint64Map_Random_Txn_Insert-4       	    7376	    169369 ns/op	   5904257 items/sec	  117211 B/op	    2409 allocs/op
Benchmark_Insert_RootOnlyWatch-4              	   10000	    100636 ns/op	   9936753 objects/sec	   75040 B/op	    2035 allocs/op
Benchmark_Insert-4                            	   10000	    101793 ns/op	   9823886 objects/sec	   75584 B/op	    2046 allocs/op
Benchmark_WatchReplace-4                      	   13938	     86228 ns/op	  11597099 objects/sec	   57305 B/op	    1006 allocs/op
Benchmark_Modify-4                            	   14234	     84269 ns/op	  11866706 objects/sec	   58056 B/op	    1007 allocs/op
Benchmark_GetInsert-4                         	   10000	    105223 ns/op	   9503657 objects/sec	   58056 B/op	    1007 allocs/op
Benchmark_Replace-4                           	18731790	        64.11 ns/op	  15597299 objects/sec	      48 B/op	       1 allocs/op
Benchmark_Replace_RootOnlyWatch-4             	 8727679	       120.2 ns/op	   8319701 objects/sec	      96 B/op	       1 allocs/op
Benchmark_txn_1-4                             	 9255091	       128.3 ns/op	   7793721 objects/sec	      64 B/op	       3 allocs/op
Benchmark_txn_10-4                            	11776526	        99.71 ns/op	  10029437 objects/sec	      76 B/op	       2 allocs/op
Benchmark_txn_100-4                           	13439996	        87.59 ns/op	  11416972 objects/sec	      67 B/op	       2 allocs/op
Benchmark_txn_1000-4                          	11864286	       101.0 ns/op	   9899599 objects/sec	      65 B/op	       2 allocs/op
Benchmark_txn_delete_1-4                      	 4801674	       249.6 ns/op	   4007064 objects/sec	     632 B/op	       3 allocs/op
Benchmark_txn_delete_10-4                     	 9712550	       123.6 ns/op	   8090691 objects/sec	     103 B/op	       1 allocs/op
Benchmark_txn_delete_100-4                    	12780760	        92.06 ns/op	  10862798 objects/sec	      35 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4                   	12777826	        92.99 ns/op	  10754144 objects/sec	      28 B/op	       1 allocs/op
BenchmarkDeleteParentCache/depth_8-4          	 4613088	       258.6 ns/op	      48 B/op	       1 allocs/op
BenchmarkDeleteParentCache/depth_100-4        	  442120	      2582 ns/op	      48 B/op	       1 allocs/op
Benchmark_Get-4                               	   67459	     17786 ns/op	  56224557 objects/sec	       0 B/op	       0 allocs/op
Benchmark_GetWatch-4                          	   48019	     24984 ns/op	  40025493 objects/sec	       0 B/op	       0 allocs/op
Benchmark_All-4                               	  127730	      9334 ns/op	 107132224 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_All-4                      	  134996	      8907 ns/op	 112265515 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_Next-4                     	  139488	      8436 ns/op	 118544719 objects/sec	     896 B/op	       1 allocs/op
Benchmark_LowerBound-4                        	   10000	    105863 ns/op	   9446187 objects/sec	  104000 B/op	    2000 allocs/op
Benchmark_LowerBound_Random-4                 	    8414	    141463 ns/op	   7068978 objects/sec	   96384 B/op	    1002 allocs/op
Benchmark_Hashmap_Insert-4                    	   15524	     77503 ns/op	  12902706 objects/sec	   74264 B/op	      20 allocs/op
Benchmark_Hashmap_Get_Uint64-4                	  120378	      9935 ns/op	 100658617 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4                 	  111213	     10762 ns/op	  92916607 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Delete_Random-4                     	      84	  15470025 ns/op	   6464114 objects/sec	 2539393 B/op	  102755 allocs/op
Benchmark_find16-4                            	213039579	         5.625 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex16-4                       	100000000	        11.28 ns/op	       0 B/op	       0 allocs/op
Benchmark_find64-4                            	296030258	         4.051 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_hit-4                   	273795272	         4.372 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_miss-4                  	296000295	         4.051 ns/op	       0 B/op	       0 allocs/op
Benchmark_find4-4                             	384141837	         3.120 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex4-4                        	274824865	         4.366 ns/op	       0 B/op	       0 allocs/op
BenchmarkSmallWriteTxn/updates_1-4            	  474502	      2436 ns/op	    3529 B/op	       4 allocs/op
BenchmarkSmallWriteTxn/updates_2-4            	  320079	      3441 ns/op	    4742 B/op	       6 allocs/op
BenchmarkSmallWriteTxn/updates_4-4            	  221832	      5333 ns/op	    7155 B/op	      11 allocs/op
BenchmarkSmallWriteTxn/updates_8-4            	  125901	      9009 ns/op	   11927 B/op	      20 allocs/op
BenchmarkSmallWriteTxn/updates_16-4           	   70963	     16516 ns/op	   21290 B/op	      38 allocs/op
BenchmarkFreshWriteTxn-4                      	 6556486	       183.2 ns/op	     185 B/op	       6 allocs/op
BenchmarkAtomicWatchPointerFirstChannel-4     	23425803	        53.14 ns/op	     120 B/op	       2 allocs/op
BenchmarkAtomicWatchPointerChannel-4          	481014962	         2.494 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/part	63.036s
PASS
ok  	github.com/cilium/statedb/reconciler	0.005s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 1.53 seconds (batch size 1000)
Throughput 652391.32 objects per second
506MB total allocated, 5007212 in-use objects, 231MB bytes in use

@dylandreimerink

Copy link
Copy Markdown
Member

I definitely like the Where and Select/Map, especially for situations where you want to "Query" the DB for something but it does not warrant an index. This provides a much nicer way to get a useful iterator or set of objects.

I am not so sure about the Join family of pipeline ops, this seems very expensive and would have more limited uses.

@joamaki

joamaki commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

I definitely like the Where and Select/Map, especially for situations where you want to "Query" the DB for something but it does not warrant an index. This provides a much nicer way to get a useful iterator or set of objects.

For those cases it would likely make more sense to have utilities around iter.Seq instead of providing that functionality from StateDB. I'm sure there will be a popular library for this soon.

I am not so sure about the Join family of pipeline ops, this seems very expensive and would have more limited uses.

Yeah indeed. I'm still leaning towards open-coding these as it makes it more obvious what the cost of the join might be.
But let's keep the idea around for awhile and if we start seeing use-cases where writing it with a helper like this would improve readability we could give this more attention.

@thejesh23

This comment was marked as spam.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants