Skip to content

larger block size/cache & use bloom filters in pebble#47

Merged
KiKoS0 merged 1 commit into
mainfrom
riadh/reduce-block-cache-thrashing
May 4, 2026
Merged

larger block size/cache & use bloom filters in pebble#47
KiKoS0 merged 1 commit into
mainfrom
riadh/reduce-block-cache-thrashing

Conversation

@KiKoS0
Copy link
Copy Markdown
Contributor

@KiKoS0 KiKoS0 commented May 4, 2026

I noticed lots of cache thrashing when pods memory usage is nearing the
limit, that puts the pod in a cycle of reading/invalidating indexes from
disk and aggregate evaluation taking minutes sometimes.


2 @ 0x485cee 0x46313d 0x463114 0x4877e5 0x14bb6cc 0x14bb6af 0x14bb025 0x14efb57 0x1559b14 0x1559a3b 0x1558d05 0x1647eb4 0x1647bb3 0x165f9cd 0x165e6a5 0x162266c 0x16220dd 0x236c350 0x2370a9e 0x236953f 0x2ab9086 0x2ab70df 0x50fe4b2 0x50fafb8 0x21ff766 0x48e821
#	0x4877e4	sync.runtime_SemacquireRWMutexR+0x24									/usr/local/go/src/runtime/sema.go:100
#	0x14bb6cb	sync.(*RWMutex).RLock+0x4b										/usr/local/go/src/sync/rwmutex.go:74
#	0x14bb6ae	github.com/cockroachdb/pebble/v2/internal/cache.(*shard).getWithMaybeReadEntry+0x2e			/app/vendor/github.com/cockroachdb/pebble/v2/internal/cache/clockpro.go:139
#	0x14bb024	github.com/cockroachdb/pebble/v2/internal/cache.(*Handle).GetWithReadHandle+0xa4			/app/vendor/github.com/cockroachdb/pebble/v2/internal/cache/cache.go:299
#	0x14efb56	github.com/cockroachdb/pebble/v2/sstable/block.(*Reader).Read+0x176					/app/vendor/github.com/cockroachdb/pebble/v2/sstable/block/block.go:505
#	0x1559b13	github.com/cockroachdb/pebble/v2/sstable.(*Reader).readIndexBlock+0x373					/app/vendor/github.com/cockroachdb/pebble/v2/sstable/reader.go:357
#	0x1559a3a	github.com/cockroachdb/pebble/v2/sstable.(*twoLevelIterator[...]).loadSecondLevelIndexBlock+0x29a	/app/vendor/github.com/cockroachdb/pebble/v2/sstable/reader_iter_two_lvl.go:70
#	0x1558d04	github.com/cockroachdb/pebble/v2/sstable.(*twoLevelIterator[...]).SeekPrefixGE+0x484			/app/vendor/github.com/cockroachdb/pebble/v2/sstable/reader_iter_two_lvl.go:481
#	0x1647eb3	github.com/cockroachdb/pebble/v2.(*getIter).Next+0x293							/app/vendor/github.com/cockroachdb/pebble/v2/get_iter.go:134
#	0x1647bb2	github.com/cockroachdb/pebble/v2.(*getIter).First+0x12							/app/vendor/github.com/cockroachdb/pebble/v2/get_iter.go:72
#	0x165f9cc	github.com/cockroachdb/pebble/v2.(*Iterator).iterFirstWithinBounds+0x8c					/app/vendor/github.com/cockroachdb/pebble/v2/iterator.go:2096
#	0x165e6a4	github.com/cockroachdb/pebble/v2.(*Iterator).First+0xe4							/app/vendor/github.com/cockroachdb/pebble/v2/iterator.go:1648
#	0x162266b	github.com/cockroachdb/pebble/v2.(*DB).getInternal+0x54b						/app/vendor/github.com/cockroachdb/pebble/v2/db.go:629
#	0x16220dc	github.com/cockroachdb/pebble/v2.(*DB).Get+0x1c								/app/vendor/github.com/cockroachdb/pebble/v2/db.go:547
#	0x236c34f	github.com/inngest/expr.(*EvalKV[...]).Get+0x6f								/app/vendor/github.com/inngest/expr/kvdb.go:72
#	0x2370a9d	github.com/inngest/expr.(*aggregator[...]).Evaluate+0x79d						/app/vendor/github.com/inngest/expr/expr.go:404
#	0x236953e	github.com/inngest/inngest/pkg/expressions/expragg.(*aggregator).EvaluateAsyncEvent+0xd7e		/app/vendor/github.com/inngest/inngest/pkg/expressions/expragg/expragg.go:174
#	0x2ab9085	github.com/inngest/inngest/pkg/execution/executor.(*executor).handleAggregatePauses+0x345		/app/vendor/github.com/inngest/inngest/pkg/execution/executor/executor.go:2535
#	0x2ab70de	github.com/inngest/inngest/pkg/execution/executor.(*executor).HandlePauses+0x3be			/app/vendor/github.com/inngest/inngest/pkg/execution/executor/executor.go:2398
#	0x50fe4b1	github.com/tonyhb/datos/pkg/execution/pauses.mgr.handlePauseEvent+0xaf1					/app/pkg/execution/pauses/pauses.go:531
#	0x50fafb7	github.com/tonyhb/datos/pkg/execution/pauses.(*svc).Run.mgr.Run.func2+0x117				/app/pkg/execution/pauses/pauses.go:397
#	0x21ff765	github.com/inngest/inngest/pkg/execution/queue.(*queueProcessor).ProcessItem.func5+0xa85		/app/vendor/github.com/inngest/inngest/pkg/execution/queue/process.go:367

2 @ 0x485cee 0x46313d 0x463114 0x487845 0x49a2cb 0x14bb8a6 0x14be4fc 0x14efe74 0x14efe64 0x155d2c5 0x155d1ee 0x155c539 0x155c036 0x1558e05 0x1647eb4 0x1647bb3 0x165f9cd 0x165e6a5 0x162266c 0x16220dd 0x236c350 0x2370a9e 0x236953f 0x2ab9086 0x2ab70df 0x50fe4b2 0x50fafb8 0x21ff766 0x48e821
#	0x487844	sync.runtime_SemacquireRWMutex+0x24								/usr/local/go/src/runtime/sema.go:105
#	0x49a2ca	sync.(*RWMutex).Lock+0x6a									/usr/local/go/src/sync/rwmutex.go:155
#	0x14bb8a5	github.com/cockroachdb/pebble/v2/internal/cache.(*shard).set+0x65				/app/vendor/github.com/cockroachdb/pebble/v2/internal/cache/clockpro.go:166
#	0x14be4fb	github.com/cockroachdb/pebble/v2/internal/cache.(*readEntry).setReadValue+0x3b			/app/vendor/github.com/cockroachdb/pebble/v2/internal/cache/read_shard.go:304
#	0x14efe73	github.com/cockroachdb/pebble/v2/internal/cache.ReadHandle.SetReadValue+0x493			/app/vendor/github.com/cockroachdb/pebble/v2/internal/cache/read_shard.go:372
#	0x14efe63	github.com/cockroachdb/pebble/v2/sstable/block.(*Reader).Read+0x483				/app/vendor/github.com/cockroachdb/pebble/v2/sstable/block/block.go:536
#	0x155d2c4	github.com/cockroachdb/pebble/v2/sstable.(*Reader).readDataBlock+0x404				/app/vendor/github.com/cockroachdb/pebble/v2/sstable/reader.go:372
#	0x155d1ed	github.com/cockroachdb/pebble/v2/sstable.(*singleLevelIterator[...]).loadDataBlock+0x32d	/app/vendor/github.com/cockroachdb/pebble/v2/sstable/reader_iter_single_lvl.go:493
#	0x155c538	github.com/cockroachdb/pebble/v2/sstable.(*singleLevelIterator[...]).seekGEHelper+0x318		/app/vendor/github.com/cockroachdb/pebble/v2/sstable/reader_iter_single_lvl.go:752
#	0x155c035	github.com/cockroachdb/pebble/v2/sstable.(*singleLevelIterator[...]).seekPrefixGE+0x1f5		/app/vendor/github.com/cockroachdb/pebble/v2/sstable/reader_iter_single_lvl.go:860
#	0x1558e04	github.com/cockroachdb/pebble/v2/sstable.(*twoLevelIterator[...]).SeekPrefixGE+0x584		/app/vendor/github.com/cockroachdb/pebble/v2/sstable/reader_iter_two_lvl.go:554
#	0x1647eb3	github.com/cockroachdb/pebble/v2.(*getIter).Next+0x293						/app/vendor/github.com/cockroachdb/pebble/v2/get_iter.go:134
#	0x1647bb2	github.com/cockroachdb/pebble/v2.(*getIter).First+0x12						/app/vendor/github.com/cockroachdb/pebble/v2/get_iter.go:72
#	0x165f9cc	github.com/cockroachdb/pebble/v2.(*Iterator).iterFirstWithinBounds+0x8c				/app/vendor/github.com/cockroachdb/pebble/v2/iterator.go:2096
#	0x165e6a4	github.com/cockroachdb/pebble/v2.(*Iterator).First+0xe4						/app/vendor/github.com/cockroachdb/pebble/v2/iterator.go:1648
#	0x162266b	github.com/cockroachdb/pebble/v2.(*DB).getInternal+0x54b					/app/vendor/github.com/cockroachdb/pebble/v2/db.go:629
#	0x16220dc	github.com/cockroachdb/pebble/v2.(*DB).Get+0x1c							/app/vendor/github.com/cockroachdb/pebble/v2/db.go:547
#	0x236c34f	github.com/inngest/expr.(*EvalKV[...]).Get+0x6f							/app/vendor/github.com/inngest/expr/kvdb.go:72
#	0x2370a9d	github.com/inngest/expr.(*aggregator[...]).Evaluate+0x79d					/app/vendor/github.com/inngest/expr/expr.go:404
#	0x236953e	github.com/inngest/inngest/pkg/expressions/expragg.(*aggregator).EvaluateAsyncEvent+0xd7e	/app/vendor/github.com/inngest/inngest/pkg/expressions/expragg/expragg.go:174
#	0x2ab9085	github.com/inngest/inngest/pkg/execution/executor.(*executor).handleAggregatePauses+0x345	/app/vendor/github.com/inngest/inngest/pkg/execution/executor/executor.go:2535
#	0x2ab70de	github.com/inngest/inngest/pkg/execution/executor.(*executor).HandlePauses+0x3be		/app/vendor/github.com/inngest/inngest/pkg/execution/executor/executor.go:2398
#	0x50fe4b1	github.com/tonyhb/datos/pkg/execution/pauses.mgr.handlePauseEvent+0xaf1				/app/pkg/execution/pauses/pauses.go:531
#	0x50fafb7	github.com/tonyhb/datos/pkg/execution/pauses.(*svc).Run.mgr.Run.func2+0x117			/app/pkg/execution/pauses/pauses.go:397
#	0x21ff765	github.com/inngest/inngest/pkg/execution/queue.(*queueProcessor).ProcessItem.func5+0xa85	/app/vendor/github.com/inngest/inngest/pkg/execution/queue/process.go:367

I noticed lots of cache thrashing when pods memory usage is nearing the
limit, that puts the pod in a cycle of reading/invalidating indexes from
disk and aggregate evaluation taking minutes sometimes.

```
2 @ 0x485cee 0x46313d 0x463114 0x4877e5 0x14bb6cc 0x14bb6af 0x14bb025 0x14efb57 0x1559b14 0x1559a3b 0x1558d05 0x1647eb4 0x1647bb3 0x165f9cd 0x165e6a5 0x162266c 0x16220dd 0x236c350 0x2370a9e 0x236953f 0x2ab9086 0x2ab70df 0x50fe4b2 0x50fafb8 0x21ff766 0x48e821

2 @ 0x485cee 0x46313d 0x463114 0x487845 0x49a2cb 0x14bb8a6 0x14be4fc 0x14efe74 0x14efe64 0x155d2c5 0x155d1ee 0x155c539 0x155c036 0x1558e05 0x1647eb4 0x1647bb3 0x165f9cd 0x165e6a5 0x162266c 0x16220dd 0x236c350 0x2370a9e 0x236953f 0x2ab9086 0x2ab70df 0x50fe4b2 0x50fafb8 0x21ff766 0x48e821
```
@KiKoS0 KiKoS0 merged commit f1f238f into main May 4, 2026
2 checks passed
@KiKoS0 KiKoS0 deleted the riadh/reduce-block-cache-thrashing branch May 4, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants