Repository navigation
Improve iteration over non-unique secondary indexes - #211
Merged
Merged
Conversation
Using the primary key is costly to figure out if an object was already visited when iterating over a non-unique secondary index. We can instead use the revision as that is also guaranteed unique. Before: BenchmarkDB_Prefix_SecondaryIndex-6 13364 89515 ns/op 11171333 objects/sec 108954 B/op 26 allocs/op After: BenchmarkDB_Prefix_SecondaryIndex-6 19248 63163 ns/op 15832108 objects/sec 74457 B/op 26 allocs/op AIL:3 Signed-off-by: Jussi Maki <jussi@isovalent.com>
Prefix and lower-bound scans of non-unique indexes allocate a visited map for
tracking objects that appear under multiple secondary keys. Reuse these maps
across scans with sync.Pool, and discard maps that have grown beyond 4096
entries. Return maps after All completes or Next reaches the end of iteration.
Benchmarks: BenchmarkDB_Prefix_SecondaryIndex and
BenchmarkDB_LowerBound_SecondaryIndex, linux/arm64, Go 1.25.10,
benchtime=200ms, count=10. Results from benchstat:
│ old │ new │ vs base
│ sec/op │ sec/op │
Prefix │ 72.53µs ± 51% │ 23.69µs ± 3% │ -67.34% (p=0.000 n=10)
LowerBound │ 72.57µs ± 25% │ 24.74µs ± 2% │ -65.91% (p=0.000 n=10)
│ old │ new │ vs base
│ B/op │ B/op │
Prefix │ 74457 ± 0% │ 199 ± 4% │ -99.73% (p=0.000 n=10)
LowerBound │ 74745 ± 0% │ 504 ± 0% │ -99.33% (p=0.000 n=10)
│ old │ new │ vs base
│ allocs/op │ allocs/op │
Prefix │ 26.000 ± 0% │ 6.000 ± 0% │ -76.92% (p=0.000 n=10)
LowerBound │ 28.000 ± 0% │ 8.000 ± 0% │ -71.43% (p=0.000 n=10)
AIL:3
Signed-off-by: Jussi Maki <jussi@isovalent.com>
joamaki
force-pushed
the
pr/joamaki/non-unique-dedup-with-revision
branch
from
October 13, 2026 15:40
943a515 to
e754238
Compare
|
bimmlerd
approved these changes
Oct 14, 2026
| @@ -662,7 +659,7 @@ func (it *nonUniquePartIterator) All(yield func([]byte, object) bool) { | |||
| // | |||
| // This of course makes iterating over a non-unique index with a prefix | |||
| // (or lowerbound search) about 20x slower than normal! | |||
Member
There was a problem hiding this comment.
nit: is it still 20x slower? 🙃
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Non-unique secondary indexes may refer to the same object with multiple keys. When iterating over such index StateDB de-duplicates and makes sure the same object isn't emitted multiple times.
This PR improves the performance and reduces allocations of such queries by using the revision instead of the primary key to track visited objects and by using
sync.Poolto reuse allocation of the visited hash map.See commits for benchmark results. tl;dr:
AIL:3