Skip to content

[homeassistant] fix: close the superseded connection when connect() is re-entered - #64

Closed
mmhorda wants to merge 1 commit into
NousResearch:mainfrom
mmhorda:fix/ha-reconnect-session-leak
Closed

mmhorda wants to merge 1 commit into
NousResearch:mainfrom
mmhorda:fix/ha-reconnect-session-leak

Conversation

@mmhorda

@mmhorda mmhorda commented Oct 12, 2026 •

Copy link
Copy Markdown

What

[homeassistant] fix: close the superseded connection when connect() is re-entered.

Why

connect() is not one-shot: the gateway's reconnect watcher re-enters it with
is_reconnect=True while the previous connection is still live. On that path the old code
called _ws_connect() (which overwrites self._session/self._ws) and then overwrote
self._rest_session and self._listen_task, without closing or cancelling anything it
replaced. Two things leak on every reconnect:

  1. The two aiohttp.ClientSessions are orphaned. Nothing references them any more, so
    they are only cleaned up when GC collects them — until then aiohttp logs
    Unclosed client session / Unclosed connector, and their sockets and file descriptors
    stay open. Where this was found the gateway log held 183 [Homeassistant] Connected to …
    against 187 [Homeassistant] Disconnected lines, i.e. the watcher normally completes a
    teardown before re-dialling, so the leak is latent rather than one per reconnect: it
    fires whenever connect() is re-entered without a finished teardown (a second watcher
    tick, a teardown that raised, the lost-connection path), and the log did show
    Unclosed client session errors on the days with reconnect churn. The isolated test above
    is what pins the mechanism down; the log says how often it can bite.

The clearest sighting, from the log on the machine where this was found:

2026-10-12 14:01:31 WARNING hermes_plugins.homeassistant.adapter: [Homeassistant] Reconnection failed: 502, message='Invalid response status', url='wss://mmhome…'
2026-10-12 14:02:17 ERROR   asyncio: Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x…>

_ws_connect() assigns self._session before it dials, so a handshake that fails (here a 502 from
the proxy) leaves that session unreferenced with nothing closing it on the failure path. With this
patch the next connect() closes it via _cleanup_ws(); #6 additionally closes the freshly-created
session inside _ws_connect()'s own failure path. The two cover different halves of the same leak,
which is why they are complementary.
2. Two _listen_loop tasks run against one adapter. The superseded listener keeps
reading its own (still-open) websocket and dispatching events, so a single HA
state_changed event can produce duplicate agent turns.

How

Close and cancel the superseded connection before dialling again, reusing the existing
teardown path (_cleanup_ws() + _close()), and drop the stale listener before starting
the new one:

await self._cleanup_ws()
await self._close(self._rest_session)
self._rest_session = None
if self._listen_task is not None and not self._listen_task.done():
    self._listen_task.cancel()
    try:
        await self._listen_task
    except asyncio.CancelledError:
        pass
    except Exception:  # a dying listener must not block the reconnect
        logger.debug("[%s] stale listener exited with an error", self.name, exc_info=True)
self._listen_task = None

18 lines in connect(). A first connect has nothing to close, so that path is unchanged.

Relationship to #6

Complementary, not a duplicate. #6 ("bound every wedgeable await during Home Assistant
connect and teardown") bounds the teardown awaits and makes _close() bounded, and it
rewrites _ws_connect() to hold the new socket in locals — but its connect() still assigns
self._session = session / self._ws = ws at the end without closing the previous pair, so
the reconnect orphaning described here still happens on top of #6. This patch and #6 touch
different lines of connect()/_ws_connect() and should merge cleanly in either order;
whichever lands second should re-run both suites.

Tests

tests/test_homeassistant_connect_reconnect.py (new, offline — connection primitives stubbed,
no real HA instance):

  • test_reconnect_closes_superseded_sessions — both superseded sessions are closed, the
    superseded listener is done, the live one is a new task;
  • test_repeated_reconnects_do_not_accumulate_sessions — 3 reconnects leave exactly the live
    connection's two sessions open;
  • test_cold_connect_then_disconnect_closes_everything — the cold path still tears down.

Verified on this commit:

tests/test_homeassistant_connect_reconnect.py::TestConnectReconnectLifecycle::test_reconnect_closes_superseded_sessions FAILED
tests/test_homeassistant_connect_reconnect.py::TestConnectReconnectLifecycle::test_repeated_reconnects_do_not_accumulate_sessions FAILED
2 failed, 1 passed        # against main, before the fix

66 passed, 2 skipped      # full plugin suite from homeassistant/ with the fix

(The new tests fail on main and pass with the fix, which is the point of adding them.)

… re-entered

The gateway's reconnect watcher re-enters connect(is_reconnect=True) while the previous
connection is still live (gateway/run_adapters.py). connect() overwrote self._ws,
self._session and self._rest_session and started a second _listen_loop without closing or
cancelling what it replaced, so every reconnect:

- leaked the websocket ClientSession and the REST ClientSession: aiohttp reports
  "Unclosed client session" once GC collects them, and the sockets stay open until then;
- left two listeners running against the same adapter, each dispatching every watched
  state_changed event (duplicate notifications from one HA event).

Close and cancel the superseded connection before dialling again, using the existing
_cleanup_ws()/_close() teardown path, so a first connect (nothing to close) is unchanged.

Observable on a long-lived gateway as repeated "Unclosed client session" / "Unclosed
connector" errors, roughly one pair per reconnect.

tests/test_homeassistant_connect_reconnect.py fails on main, passes with this fix.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mmhorda. The patch is tidy and its tests pass, but we can't reproduce the bug it fixes through Hermes's real reconnect path, so we'd like a repro before merging.

On current Hermes, the gateway's reconnect watcher builds a fresh adapter for every attempt and disposes failed ones with disconnect() (gateway/run_adapters.py, the reconnect loop and _dispose_unused_adapter). So connect() is never called a second time on a live adapter instance. The new tests reach that state by calling connect() twice by hand on one instance. The Reconnection failed: 502 line in the description comes from the adapter's own _listen_loop reconnect, which already calls _cleanup_ws() before every redial.

Could you share either:

  1. the gateway log around the Unclosed client session warning (with the Hermes version), or
  2. the code path that calls connect() on an already-connected adapter?

If it's the failure path inside _ws_connect, that leak is already covered by #6 (bounded connect/teardown), so it may be fixed once that lands.

@mmhorda

mmhorda commented Oct 13, 2026

Copy link
Copy Markdown
Author

Confirmed on my side — I re-audited the current gateway and the premise doesn't hold:

  • The only adapter.connect() call site is _connect_adapter_bounded, and both callers (startup in run_startup.py, _reconnect_failed_platform in run_adapters.py) always connect a freshly created adapter. Failed candidates are disposed via _dispose_unused_adapter → disconnect(), and the fatal handler queues before disconnecting. My tests reached the re-entry state by calling connect() twice by hand, as you said.
  • I also chased the Unclosed client session warnings that motivated this PR (21 occurrences over 10 days in errors.log, Hermes v0.21.5+9957, gateway + agent processes): they persist on days with zero HA reconnect events and appear in agent-side process contexts. The actual source is the hindsight memory plugin's teardown stranding sibling providers' aiohttp sessions on the shared loop — the failure mode documented against #11923 — not this adapter.
  • The _ws_connect failure-path leak is indeed the remaining hazard here, and I agree it's covered by [homeassistant] fix: bound every wedgeable await during Home Assistant connect and teardown #6.

Closing. I'm keeping the idempotent-connect hardening as a local patch on my install (cheap insurance against a future in-place reconnect refactor), but there's no reproducible bug for upstream to merge, so a hardening-only PR isn't worth your review bandwidth.

Thanks for the careful check.

@mmhorda mmhorda closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants