Skip to content

Commit 8f3b9c9

Browse files
authored
fix(background_tasks): warm the invalidation publish connection from the listener (#882)
* docs: invalidation warm publish design and plan (#880) Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4 * fix(background_tasks): warm the invalidation publish connection from the listener The client's command connection opened lazily, so the first publish after boot or after an outage paid the connect inside its 1 s request-path budget and lost the message under load. The listener now PINGs after every successful SUBSCRIBE; a failed warm-up is logged at DEBUG and never restarts it. Closes #880 Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4 * fix: address code review findings (round 1, pass 1) Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
1 parent dc72198 commit 8f3b9c9

6 files changed

Lines changed: 321 additions & 3 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ All notable changes to this project are documented in this file. The format is b
1414
1515
## [Unreleased]
1616

17+
### Fixed
18+
19+
- `background_tasks`: the Redis invalidation listener now opens the connection a publish uses (one `PING` after every successful subscribe), so the first invalidation after boot or after a Redis outage no longer spends part of its 1 s budget connecting. Under load that connect was cancelled and the message was lost; other workers kept the stale value until its TTL (#880).
20+
1721
### Changed
1822

1923
- Faster tests: `simple_module_test` gives every in-memory SQLite engine made through `init_db` with the same dialect options one shared dialect and compiled-statement cache, so a statement compiled in one test is reused by the next instead of being recompiled for every fresh engine. The databases stay separate. The patch is process-wide; set `SM_TEST_SEPARATE_COMPILE_CACHES=1` to skip it. `make test-py` drops from about 250 s to 232 s on 8 workers (test bodies 566 s to 511 s of worker time).
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Plan: warm the invalidation publish connection (GH #880)
2+
3+
Design: `docs/superpowers/specs/2026-11-07-invalidation-warm-publish-design.md`.
4+
5+
1. **Fake-client tests first.** Create `modules/background_tasks/tests/core/invalidation/test_background_tasks_invalidation_warmup.py`.
6+
- Its own small fake with `ping()`, because the shared fake lives in a 293-line file. Copy only what the tests need.
7+
- Cover: a ping after subscribe; another after a reconnect; a failing ping and a hanging ping both leave the listener delivering.
8+
- Done when they fail against `main`.
9+
2. **Implement** `_warm_publish_connection` in `modules/background_tasks/background_tasks/core/invalidation.py` and call it after a successful subscribe. Add one sentence to the `PUBLISH_TIMEOUT_SECONDS` docstring. Done when the step 1 tests pass.
10+
3. **Real-Redis test** in the same new file: once subscribed, `CLIENT LIST` shows a client whose last command is `ping`, and a publish does not add a client. Done when it passes against `SM_TEST_REDIS_URL`.
11+
4. **CHANGELOG** `[Unreleased]` entry under Fixed, citing #880.
12+
5. `make lint`, `make test-py` (with `SM_TEST_REDIS_URL`), `make test-js`.
13+
14+
Tasks 1–3 are sequential and touch the same files; none is marked hard.
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Warm the invalidation publish connection (GH #880)
2+
3+
## Goal and scope
4+
5+
A publish on `RedisInvalidationTransport` runs inside `asyncio.timeout(PUBLISH_TIMEOUT_SECONDS)`
6+
(1 s), in the response cycle of the request that made the write. The client's command
7+
connection is opened lazily by the first command, so the first publish after boot, or after
8+
Redis came back from an outage, also pays the TCP connect and handshake inside that second.
9+
Under load the connect is cancelled (seen in CI, #865), the bus logs "Invalidation broadcast
10+
failed", and other workers keep the stale value until its TTL.
11+
12+
In scope: make the command connection already open when a publish needs it, in the cases we
13+
can see coming. Out of scope: any change to the 1 s budget, the bus's swallow-and-log
14+
contract, or delivery guarantees. Invalidation stays best-effort over a TTL floor.
15+
16+
## Approach
17+
18+
**Chosen: warm from the listener.** Each time the listener task's SUBSCRIBE succeeds (first
19+
boot, and every reconnect after an outage), it sends one `PING` on the command client, bounded
20+
by `SUBSCRIBE_TIMEOUT_SECONDS`. That opens a pooled connection, which the next publish reuses.
21+
`start()` still never waits on Redis.
22+
23+
Rejected:
24+
- **A separate connect budget** (say 1 s connect + 1 s publish). It doubles the worst case a
25+
request can be held, and it still loses the message when the connect is slower than that.
26+
- **Publish from a background sender queue with retry.** This takes publish out of the request
27+
path entirely. But it changes the transport contract (`publish` would no longer raise for
28+
the bus to log), queued messages are lost at shutdown, and it is more machinery than a
29+
best-effort accelerator needs.
30+
31+
## Design
32+
33+
In `modules/background_tasks/background_tasks/core/invalidation.py`, right after
34+
`pubsub.subscribe(...)` succeeds in `_listen`:
35+
36+
```python
37+
await self._warm_publish_connection()
38+
```
39+
40+
```python
41+
async def _warm_publish_connection(self) -> None:
42+
"""Open the command connection a publish will use, off the request path."""
43+
try:
44+
async with asyncio.timeout(SUBSCRIBE_TIMEOUT_SECONDS):
45+
await self._client.ping()
46+
except Exception as exc: # the listener stays up; a publish just connects cold
47+
logger.debug("Invalidation publish connection not warmed: %s", exc)
48+
```
49+
50+
- A failed warm-up never fails or restarts the listener. The subscription is what keeps this
51+
worker receiving; the warm-up only saves the next publish a connect.
52+
- Cancellation (`stop()`) still propagates, because `CancelledError` is not an `Exception`.
53+
- The constants and docstrings say why: `PUBLISH_TIMEOUT_SECONDS` gains one sentence on the warm
54+
connection.
55+
56+
## Decisions made
57+
58+
- Warm on every successful subscribe, not only the first. A Redis restart drops both
59+
connections, and the listener notices it first (health check within ~15 s).
60+
- Bound the PING by `SUBSCRIBE_TIMEOUT_SECONDS` (5 s), not the publish budget: it is not in a
61+
request's way.
62+
- Failures log at DEBUG. The listener already warns once per outage, and a second warning for
63+
the same outage would be noise.
64+
- No new setting or environment variable.
65+
- The #881 test fixture that raises the publish timeout in the real-Redis tests stays. Those
66+
tests publish right after `start()`, before a warm-up can finish.
67+
68+
## Testing
69+
70+
New file `modules/background_tasks/tests/core/invalidation/test_background_tasks_invalidation_warmup.py`
71+
(the existing `test_background_tasks_invalidation.py` is at 293 of the 300-line cap):
72+
73+
- With the fake client: after the listener subscribes, the client receives one `ping`; after a
74+
scripted subscribe failure and a reconnect, it receives another.
75+
- With the fake client: a `ping` that raises, or hangs (bounded by a patched short timeout),
76+
leaves the listener subscribed and delivering.
77+
- With a real `redis-server`: once subscribed, the server reports a second client for this
78+
transport whose last command is `ping` (`CLIENT LIST`). A publish then reuses it, and the
79+
server's client count does not grow.
80+
81+
The new file carries its own small fake, with an `async def ping()`. Full `make test-py` with
82+
`SM_TEST_REDIS_URL`, plus `make lint`. No browser QA (`--skip-browser`): nothing user-visible changes.
83+
84+
## Risks
85+
86+
- redis-py might discard idle pool connections, so the warmed connection would not be reused.
87+
It doesn't (`ConnectionPool` keeps released connections until `disconnect`); the real-Redis
88+
test pins this.
89+
- The health check on an idle command connection runs inside the next publish's budget, but
90+
that is a PING round trip on an open socket, not a connect.

‎modules/background_tasks/background_tasks/core/invalidation.py‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,9 @@
8383
One second is far more than a local round trip needs and far less than a user
8484
waits. Exceeding it raises, which the bus logs and swallows: the local eviction
8585
has already happened, so the cost of the timeout is the same staleness window
86-
that existed before this transport.
86+
that existed before this transport. The listener opens the connection a publish
87+
uses ahead of time (``_warm_publish_connection``), so the budget normally covers
88+
a round trip on an open socket, not a connect.
8789
"""
8890

8991

@@ -174,6 +176,7 @@ async def _listen(self) -> None:
174176
pubsub = self._client.pubsub()
175177
async with asyncio.timeout(SUBSCRIBE_TIMEOUT_SECONDS):
176178
await pubsub.subscribe(self._channel)
179+
await self._warm_publish_connection()
177180
if self._warned:
178181
logger.info("Invalidation listener reconnected to %r", self._channel)
179182
self._warned = False
@@ -198,6 +201,22 @@ async def _listen(self) -> None:
198201
with contextlib.suppress(Exception):
199202
await pubsub.aclose()
200203

204+
async def _warm_publish_connection(self) -> None:
205+
"""Open the command connection the next publish will use, off the request path.
206+
207+
The client opens it lazily, so without this the first publish after boot,
208+
or after Redis came back, paid the TCP connect inside its 1 s budget and
209+
lost the message under load (GH #880). Runs after every successful
210+
SUBSCRIBE because an outage drops both connections. A failure here leaves
211+
the listener up: the subscription is what keeps this worker receiving, and
212+
a cold publish still connects on its own.
213+
"""
214+
try:
215+
async with asyncio.timeout(SUBSCRIBE_TIMEOUT_SECONDS):
216+
await self._client.ping()
217+
except Exception as exc:
218+
logger.debug("Invalidation publish connection not warmed: %s", exc)
219+
201220
def _log_failure(self, exc: Exception) -> None:
202221
"""Warn once per outage, then stay quiet about the same outage."""
203222
if self._warned:

‎modules/background_tasks/tests/core/invalidation/test_background_tasks_invalidation_redis.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,9 @@ def _patient_publish(monkeypatch):
3838
"""Give a publish the delivery wait's budget instead of production's 1 s.
3939
4040
``PUBLISH_TIMEOUT_SECONDS`` bounds the whole publish, and a publisher's first
41-
publish opens its command connection (the listener's pub/sub connection does
42-
not warm it). In CI that connect was cancelled by the 1 s timeout inside
41+
publish may open its command connection. The listener warms it after SUBSCRIBE
42+
(GH #880), but these tests publish right after ``start()``, before that can
43+
finish. In CI that connect was cancelled by the 1 s timeout inside
4344
``create_connection``, the bus logged "Invalidation broadcast failed", and the
4445
message was lost (GH #865). ``start()`` also reads it as
4546
``socket_connect_timeout``, so this fixture must run before a transport starts,
Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
"""The listener warms the connection a publish will use (GH #880).
2+
3+
``publish`` runs under a 1 s budget in the request's response cycle, and the
4+
client's command connection opens lazily. Without a warm-up the first publish
5+
after boot, or after an outage, paid the TCP connect inside that second and lost
6+
the message under load. The listener PINGs the command client after every
7+
successful SUBSCRIBE, off the request path, so that connection is already open.
8+
"""
9+
10+
from __future__ import annotations
11+
12+
import asyncio
13+
import os
14+
import uuid
15+
16+
import pytest
17+
from background_tasks.core import invalidation as bg_invalidation
18+
from background_tasks.core.invalidation import RedisInvalidationTransport
19+
from simple_module_core.invalidation import Invalidation, InvalidationBus
20+
from simple_module_test import wait_for
21+
22+
_CHANNEL = "sm.invalidation.warmup"
23+
24+
25+
class _PubSub:
26+
def __init__(self, frames, subscribe_error) -> None:
27+
self._frames = list(frames)
28+
self._subscribe_error = subscribe_error
29+
self.subscribed = False
30+
31+
async def subscribe(self, channel: str) -> None:
32+
if self._subscribe_error is not None:
33+
raise self._subscribe_error
34+
self.subscribed = True
35+
36+
async def get_message(self, ignore_subscribe_messages: bool = False, timeout=None):
37+
if self._frames:
38+
return self._frames.pop(0)
39+
await asyncio.sleep(timeout or 0) # intentional-sleep: fake idle-channel poll interval
40+
return None
41+
42+
async def aclose(self) -> None: ...
43+
44+
45+
class _Redis:
46+
"""Records PINGs; the first ``fail_subscribes`` subscriptions raise."""
47+
48+
def __init__(self, frames=(), fail_subscribes=0, ping_error=None, ping_hangs=False) -> None:
49+
self.pings = 0
50+
self.pubsubs: list[_PubSub] = []
51+
self._frames = frames
52+
self._fail_subscribes = fail_subscribes
53+
self._ping_error = ping_error
54+
self._ping_hangs = ping_hangs
55+
56+
def pubsub(self) -> _PubSub:
57+
error = None
58+
if self._fail_subscribes:
59+
self._fail_subscribes -= 1
60+
error = ConnectionError("redis down")
61+
ps = _PubSub(self._frames, error)
62+
self.pubsubs.append(ps)
63+
return ps
64+
65+
async def ping(self) -> bool:
66+
self.pings += 1
67+
if self._ping_hangs:
68+
await asyncio.Event().wait()
69+
if self._ping_error is not None:
70+
raise self._ping_error
71+
return True
72+
73+
async def aclose(self) -> None: ...
74+
75+
76+
@pytest.fixture
77+
def fake_redis(monkeypatch):
78+
import redis.asyncio as aioredis
79+
80+
def install(**kwargs) -> _Redis:
81+
fake = _Redis(**kwargs)
82+
monkeypatch.setattr(aioredis, "from_url", lambda *a, **kw: fake)
83+
return fake
84+
85+
monkeypatch.setattr(bg_invalidation, "INITIAL_BACKOFF_SECONDS", 0.01)
86+
monkeypatch.setattr(bg_invalidation, "LISTEN_POLL_SECONDS", 0.01)
87+
return install
88+
89+
90+
def _frame(key: str) -> dict:
91+
wire = Invalidation(channel=_CHANNEL, key=key, origin="elsewhere").to_wire()
92+
return {"type": "message", "data": wire.encode()}
93+
94+
95+
class TestWarmUp:
96+
async def test_the_listener_pings_once_it_has_subscribed(self, fake_redis):
97+
fake = fake_redis()
98+
transport = RedisInvalidationTransport(InvalidationBus(), "redis://fake", _CHANNEL)
99+
await transport.start()
100+
try:
101+
await wait_for(lambda: fake.pings == 1)
102+
assert fake.pubsubs[0].subscribed
103+
finally:
104+
await transport.stop()
105+
106+
async def test_a_reconnect_warms_again(self, fake_redis):
107+
"""A Redis restart drops both connections; the resubscribe re-warms."""
108+
fake = fake_redis(fail_subscribes=1)
109+
transport = RedisInvalidationTransport(InvalidationBus(), "redis://fake", _CHANNEL)
110+
await transport.start()
111+
try:
112+
await wait_for(lambda: fake.pings == 1)
113+
assert [ps.subscribed for ps in fake.pubsubs] == [False, True]
114+
finally:
115+
await transport.stop()
116+
117+
async def test_no_ping_before_a_subscribe_succeeds(self, fake_redis):
118+
fake = fake_redis(fail_subscribes=10**6)
119+
transport = RedisInvalidationTransport(InvalidationBus(), "redis://fake", _CHANNEL)
120+
await transport.start()
121+
try:
122+
await wait_for(lambda: len(fake.pubsubs) >= 3)
123+
assert fake.pings == 0
124+
finally:
125+
await transport.stop()
126+
127+
128+
class TestAFailedWarmUpIsHarmless:
129+
@pytest.mark.parametrize("ping", ["raises", "hangs"])
130+
async def test_the_listener_still_delivers(self, fake_redis, monkeypatch, ping):
131+
monkeypatch.setattr(bg_invalidation, "SUBSCRIBE_TIMEOUT_SECONDS", 0.05)
132+
if ping == "raises":
133+
fake = fake_redis(frames=[_frame("k")], ping_error=ConnectionError("no"))
134+
else:
135+
fake = fake_redis(frames=[_frame("k")], ping_hangs=True)
136+
bus = InvalidationBus()
137+
seen: list[Invalidation] = []
138+
bus.subscribe(_CHANNEL, seen.append)
139+
transport = RedisInvalidationTransport(bus, "redis://fake", _CHANNEL)
140+
await transport.start()
141+
try:
142+
await wait_for(lambda: bool(seen))
143+
finally:
144+
await transport.stop()
145+
146+
assert [i.key for i in seen] == ["k"]
147+
assert len(fake.pubsubs) == 1, "a failed warm-up must not restart the listener"
148+
149+
150+
class TestAgainstARealServer:
151+
async def test_a_publish_reuses_the_warmed_connection(self, redis_server):
152+
"""The server sees the PING'd connection, and the publish arrives on it.
153+
154+
Clients are found by a per-test ``client_name`` (parsed from the URL by
155+
redis-py): under xdist every worker shares one Redis, so counting all of
156+
the server's clients would race the other workers.
157+
"""
158+
import redis.asyncio as aioredis
159+
160+
name = f"sm-warmup-{os.getpid()}-{uuid.uuid4().hex[:8]}"
161+
sep = "&" if "?" in redis_server else "?"
162+
transport = RedisInvalidationTransport(
163+
InvalidationBus(), f"{redis_server}{sep}client_name={name}", _CHANNEL
164+
)
165+
probe = aioredis.from_url(redis_server)
166+
167+
async def ours() -> list[dict]:
168+
return [c for c in await probe.client_list() if c.get("name") == name]
169+
170+
async def warmed() -> list[dict]:
171+
clients = await ours()
172+
return clients if any(c["cmd"] == "ping" for c in clients) else []
173+
174+
await transport.start()
175+
try:
176+
before = await wait_for(warmed, timeout=5.0)
177+
await transport.publish(
178+
Invalidation(channel=_CHANNEL, key="k", origin="elsewhere").to_wire()
179+
)
180+
after = await ours()
181+
finally:
182+
await transport.stop()
183+
await probe.aclose()
184+
185+
assert len(before) == 2, "expected the subscription plus one warmed command connection"
186+
warmed_id = next(c["id"] for c in before if c["cmd"] == "ping")
187+
assert {c["id"] for c in after} == {c["id"] for c in before}, (
188+
"publish opened a new connection"
189+
)
190+
assert next(c["cmd"] for c in after if c["id"] == warmed_id) == "publish"

0 commit comments

Comments
 (0)