Read Events through an uncached APIReader instead of a cluster-wide informer#948
Open
LipatovAlexander wants to merge 1 commit into
Open
Read Events through an uncached APIReader instead of a cluster-wide informer#948LipatovAlexander wants to merge 1 commit into
LipatovAlexander wants to merge 1 commit into
Conversation
…nformer The failureModeDetection path listed Events for a single pod through the cached client. Because controller-runtime instantiates an informer for every type read through the cache, this one namespaced, field-selected lookup forced a cluster-wide Event informer — Events being the highest-cardinality, highest-churn resource in a cluster (27k+ objects observed on a mid-size production cluster). In cluster-scoped deployments this informer is a major memory driver and contributes to OOM kills at startup. Wire mgr.GetAPIReader() into the reconciler and ReconciliationContext and use it for the Event lookup: involvedObject.name is a server-supported field selector for Events, so the live read is cheap and behavior-identical. Drop the Event field index that existed only to serve this lookup from the cache (and eagerly created the informer at startup). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexander Lipatov <lipatov@mindbox.cloud>
|
No linked issues found. Please add the corresponding issues in the pull request description. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Reads Events through an uncached
mgr.GetAPIReader()instead of the cached client, and drops thecorev1.Eventfield index.The index existed to serve a single lookup (
failureModeDetectionreads the events of one pod), but registering it eagerly instantiated a cluster-wide Event informer at startup — Events being the highest-cardinality, highest-churn resource (27k+ objects on a mid-size cluster). In cluster-scoped deployments this informer is a major OOM driver. The field selectorinvolvedObject.nameis server-side supported for Events, so the live read is cheap and behavior-identical.APIReaderis threaded throughCassandraDatacenterReconcilerandReconciliationContextso future high-cardinality reads can use the same path.Which issue(s) this PR fixes:
Part of #947 (first of two PRs; the second bounds the rest of the cache)
Checklist