Repository navigation
tonic-xds: stop per-cluster tasks when the channel is dropped - #3019
Merged
Merged
Conversation
## Motivation The per-cluster CDS watch (`ClusterConnector`) and EDS diff loop (`EndpointManager`) tasks outlived their channel. The CDS task held an `Arc<XdsCache>` that could keep the cache alive forever, and the diff loop only noticed a dropped receiver on its next send. ## Solution - `ClusterConnector` aborts its task on drop and subscribes before spawning, handing the task a `CacheWatch` instead of the cache. - Add `TaskStream<T>`, an mpsc stream that aborts its producer task on drop, and run the EDS diff loop on it.
YutaoMa
marked this pull request as ready for review
November 10, 2026 17:47
ankurmittal
reviewed
Nov 10, 2026
ankurmittal
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The per-cluster CDS watch (
ClusterConnector) and EDS diff loop (EndpointManager) tasks outlived their channel. The CDS task held anArc<XdsCache>that could keep the cache alive forever, and the diff loop only noticed a dropped receiver on its next send.(The task can't exit until the sender drops, and the sender can't drop until the task releases the cache. Since the runtime owns the task, dropping the channel doesn't break the loop.)
This leak so far manifests only when the
XdsChannelis dropped, so the impact is memory growth on apps creating and discarding channels. But because we plan to address #2939 cluster removal bug soon, this leak will create a race window that the discovery task may subscribe to orphaned cache entry and wait forever, thus leaking even when channel is still alive. To simplify the upcoming cluster removal PR, I'm addressing it separately first.Solution
ClusterConnectoraborts its task on drop and subscribes before spawning, handing the task aCacheWatchinstead of the cache.TaskStream<T>, an mpsc stream that aborts its producer task on drop, and run the EDS diff loop on it.