Skip to content

fix: stop destroyed pool clients from dialing again - #615

Open
guiguan wants to merge 1 commit into
milvus-io:mainfrom
guiguan:fix/pool-destroy-redial
Open

guiguan wants to merge 1 commit into
milvus-io:mainfrom
guiguan:fix/pool-destroy-redial

Conversation

@guiguan

@guiguan guiguan commented Oct 3, 2026 •

Copy link
Copy Markdown

The channel pool's destroy closes each client and then calls client.getChannel().getConnectivityState(true). On a closed channel, grpc-js (1.14.4) still acts on tryToConnect: it restarts name resolution and dials a new connection. Nothing references that channel any more, so the connection is never closed.

Each pooled client that is destroyed therefore leaves one connection behind. That happens on closeConnection() and when a topology change replaces the pool, which holds at least 2 clients by default.

  • With TLS: every client has its own credentials, so every stray connection is a separate socket.
  • Without TLS: grpc-js shares a single socket across these clients, which hides the leak.

We found this through Attu v3.0.1, which ships this SDK (3.0.6). Against a TLS Milvus, with a browser tab left open, the proxies held ~590 established connections from the one Attu pod, growing by about one a minute.

Fix

destroy only closes the client. generic-pool ignores the value destroy resolves to, so nothing depends on the removed call. It dates from the original closeConnection(), which returned the post-close state.

Test

test/utils/ChannelPool.spec.ts runs a local HTTP/2 TLS server with the repo's test certificates and needs no Milvus. It does the following, then expects 0 sessions on the server:

  1. creates the real channel pool with tls.skipCertCheck;
  2. connects one client;
  3. releases it and drains the pool.
  • Before the fix: 1 session remains.
  • After the fix: 0.

test/utils Grpc, Connection and Data specs still pass.

Standalone reproduction with grpc-js 1.14.4 directly, 5 create/close cycles each:

setup server sessions left open
close() only, TLS or plaintext 0
close() + getConnectivityState(true), plaintext 1
close() + getConnectivityState(true), TLS 5

Signed-off-by: Guan Gui <guiguan@gmail.com>
@sre-ci-robot

Copy link
Copy Markdown

Welcome @guiguan! It looks like this is your first PR to milvus-io/milvus-sdk-node 🎉

@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: guiguan, shanghaikid

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants