Repository navigation
Description
Problem
When the http worker that hosts a node's replication subscriptions stops yielding (its event loop is pinned), replication to that peer stays down until something restarts the process. The main thread detects both the pinned worker and the disconnected subscriptions, but its only response is to re-drive the subscriptions on that same worker, which cannot run them.
Observed on a two-node internal cluster running harper-pro 5.3.2 (2026-10-20, UTC):
| Time | Event |
|---|---|
| 05:31:43 | Worker 17 sends its last resource report |
| 05:32:14 | Main: Worker thread 17 (http) event loop utilization has been pinned at 100% for 30s; the worker may be wedged |
| 05:33:30 | Both subscriptions to the peer (system, data) corrected connected -> down, close code 1006. They lived on worker 17 |
| 05:33 → 11:58 | Reconciling replication subscriptions for nodes wedged disconnected on a live worker every 30 s; 553 Reconciling 2 wedged subscription(s) lines, each classed redundant (counter 213 at 07:19 and rising) |
| 11:58:05 | Main at ack timeout: worker 17's last resource report was 23,181,711 ms (6.4 h) ago; OS thread state=R, +0 voluntary context switches |
| 11:58:23 | An unrelated deploy restarts the workers: Thread did not voluntarily terminate, terminating from the outside 17 |
| 11:58:24 | Subscription for data/system ... has no live worker; reassigning; both reconnect within seconds |
So for 6.4 hours one node did not receive its peer's writes. The node stayed up, /health answered from the other worker, and nothing escalated. The application on top saw it as a silent split (two nodes making conflicting decisions from different data).
The spin itself is very likely HarperFast/harper#3308 (DatabaseTransaction.abort() drain loop), whose production report shows the same signature. That is the trigger; this issue is about the recovery path, which should survive any spinning worker (#756 and #788 describe other ways a worker wedges).
This is not new, only newly long: the same node logged the wedged-disconnected reconcile in 27 earlier episodes since 2026-08-13, each cleared within one or two reconciles.
Where
replication/subscriptionManager.ts(~1965): the wedged branch re-drives existingconnected:falseentries on the worker that already owns them. A worker that never returns to its event loop never receives the re-drive.replication/replicationConnection.tsfire counters: the reconcile already classifies these fires asredundantand counts them, but nothing acts on the count.- core
server/threads/manageThreads.js(~2456): the pinned-ELU detector only warns. The ack-timeout snapshot (~2378) also only logs, even with a resource report hours stale.
Expected
A subscription whose owning worker has stopped making progress is reassigned to a healthy worker within a bounded time, and the stuck worker is recycled (or the node otherwise reports itself unhealthy), without an external restart.
Possible shape, for whoever picks this up:
- Treat "owning worker has not sent a resource report / ack within N" the same as "pointing at an exited worker": reassign its subscriptions (the existing
staleNodeUrlspath already does this correctly, as the 11:58:24 lines show). - Escalate after K consecutive
redundantwedge fires for the same entries instead of re-firing indefinitely. - Recycle a worker whose event loop has been pinned past a threshold, or at least fail the node's health check so the orchestrator restarts it.
Test
Unit level: a fake worker that stops answering (no resource reports, never processes the re-drive) while its subscriptions read connected:false; assert the subscriptions are reassigned to another worker within the bound, and that the reconcile does not re-fire on the dead worker past K attempts.
Related: HarperFast/harper#3308 (likely trigger), #788, #756, #810, HarperFast/dispatch#566 (the application-side guard added after this incident).
Filed by Claude Opus 5.5 (Claude Code) for @kriszyp.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity