Repository navigation
Conversation
… re-entered The gateway's reconnect watcher re-enters connect(is_reconnect=True) while the previous connection is still live (gateway/run_adapters.py). connect() overwrote self._ws, self._session and self._rest_session and started a second _listen_loop without closing or cancelling what it replaced, so every reconnect: - leaked the websocket ClientSession and the REST ClientSession: aiohttp reports "Unclosed client session" once GC collects them, and the sockets stay open until then; - left two listeners running against the same adapter, each dispatching every watched state_changed event (duplicate notifications from one HA event). Close and cancel the superseded connection before dialling again, using the existing _cleanup_ws()/_close() teardown path, so a first connect (nothing to close) is unchanged. Observable on a long-lived gateway as repeated "Unclosed client session" / "Unclosed connector" errors, roughly one pair per reconnect. tests/test_homeassistant_connect_reconnect.py fails on main, passes with this fix.
teknium1
left a comment
There was a problem hiding this comment.
Thanks @mmhorda. The patch is tidy and its tests pass, but we can't reproduce the bug it fixes through Hermes's real reconnect path, so we'd like a repro before merging.
On current Hermes, the gateway's reconnect watcher builds a fresh adapter for every attempt and disposes failed ones with disconnect() (gateway/run_adapters.py, the reconnect loop and _dispose_unused_adapter). So connect() is never called a second time on a live adapter instance. The new tests reach that state by calling connect() twice by hand on one instance. The Reconnection failed: 502 line in the description comes from the adapter's own _listen_loop reconnect, which already calls _cleanup_ws() before every redial.
Could you share either:
- the gateway log around the
Unclosed client sessionwarning (with the Hermes version), or - the code path that calls
connect()on an already-connected adapter?
If it's the failure path inside _ws_connect, that leak is already covered by #6 (bounded connect/teardown), so it may be fixed once that lands.
|
Confirmed on my side — I re-audited the current gateway and the premise doesn't hold:
Closing. I'm keeping the idempotent-connect hardening as a local patch on my install (cheap insurance against a future in-place reconnect refactor), but there's no reproducible bug for upstream to merge, so a hardening-only PR isn't worth your review bandwidth. Thanks for the careful check. |
What
[homeassistant] fix: close the superseded connection whenconnect()is re-entered.Why
connect()is not one-shot: the gateway's reconnect watcher re-enters it withis_reconnect=Truewhile the previous connection is still live. On that path the old codecalled
_ws_connect()(which overwritesself._session/self._ws) and then overwroteself._rest_sessionandself._listen_task, without closing or cancelling anything itreplaced. Two things leak on every reconnect:
aiohttp.ClientSessions are orphaned. Nothing references them any more, sothey are only cleaned up when GC collects them — until then aiohttp logs
Unclosed client session/Unclosed connector, and their sockets and file descriptorsstay open. Where this was found the gateway log held 183
[Homeassistant] Connected to …against 187
[Homeassistant] Disconnectedlines, i.e. the watcher normally completes ateardown before re-dialling, so the leak is latent rather than one per reconnect: it
fires whenever
connect()is re-entered without a finished teardown (a second watchertick, a teardown that raised, the lost-connection path), and the log did show
Unclosed client sessionerrors on the days with reconnect churn. The isolated test aboveis what pins the mechanism down; the log says how often it can bite.
The clearest sighting, from the log on the machine where this was found:
_ws_connect()assignsself._sessionbefore it dials, so a handshake that fails (here a 502 fromthe proxy) leaves that session unreferenced with nothing closing it on the failure path. With this
patch the next
connect()closes it via_cleanup_ws(); #6 additionally closes the freshly-createdsession inside
_ws_connect()'s own failure path. The two cover different halves of the same leak,which is why they are complementary.
2. Two
_listen_looptasks run against one adapter. The superseded listener keepsreading its own (still-open) websocket and dispatching events, so a single HA
state_changedevent can produce duplicate agent turns.How
Close and cancel the superseded connection before dialling again, reusing the existing
teardown path (
_cleanup_ws()+_close()), and drop the stale listener before startingthe new one:
18 lines in
connect(). A first connect has nothing to close, so that path is unchanged.Relationship to #6
Complementary, not a duplicate. #6 ("bound every wedgeable await during Home Assistant
connect and teardown") bounds the teardown awaits and makes
_close()bounded, and itrewrites
_ws_connect()to hold the new socket in locals — but itsconnect()still assignsself._session = session/self._ws = wsat the end without closing the previous pair, sothe reconnect orphaning described here still happens on top of #6. This patch and #6 touch
different lines of
connect()/_ws_connect()and should merge cleanly in either order;whichever lands second should re-run both suites.
Tests
tests/test_homeassistant_connect_reconnect.py(new, offline — connection primitives stubbed,no real HA instance):
test_reconnect_closes_superseded_sessions— both superseded sessions are closed, thesuperseded listener is done, the live one is a new task;
test_repeated_reconnects_do_not_accumulate_sessions— 3 reconnects leave exactly the liveconnection's two sessions open;
test_cold_connect_then_disconnect_closes_everything— the cold path still tears down.Verified on this commit:
(The new tests fail on
mainand pass with the fix, which is the point of adding them.)