Skip to content

Replication stays down for hours when the worker hosting its subscriptions stops yielding: wedge reconcile re-drives the dead worker instead of reassigning #1090

Open

Description

@kriszyp

Problem

When the http worker that hosts a node's replication subscriptions stops yielding (its event loop is pinned), replication to that peer stays down until something restarts the process. The main thread detects both the pinned worker and the disconnected subscriptions, but its only response is to re-drive the subscriptions on that same worker, which cannot run them.

Observed on a two-node internal cluster running harper-pro 5.3.2 (2026-10-20, UTC):

Time Event
05:31:43 Worker 17 sends its last resource report
05:32:14 Main: Worker thread 17 (http) event loop utilization has been pinned at 100% for 30s; the worker may be wedged
05:33:30 Both subscriptions to the peer (system, data) corrected connected -> down, close code 1006. They lived on worker 17
05:33 → 11:58 Reconciling replication subscriptions for nodes wedged disconnected on a live worker every 30 s; 553 Reconciling 2 wedged subscription(s) lines, each classed redundant (counter 213 at 07:19 and rising)
11:58:05 Main at ack timeout: worker 17's last resource report was 23,181,711 ms (6.4 h) ago; OS thread state=R, +0 voluntary context switches
11:58:23 An unrelated deploy restarts the workers: Thread did not voluntarily terminate, terminating from the outside 17
11:58:24 Subscription for data/system ... has no live worker; reassigning; both reconnect within seconds

So for 6.4 hours one node did not receive its peer's writes. The node stayed up, /health answered from the other worker, and nothing escalated. The application on top saw it as a silent split (two nodes making conflicting decisions from different data).

The spin itself is very likely HarperFast/harper#3308 (DatabaseTransaction.abort() drain loop), whose production report shows the same signature. That is the trigger; this issue is about the recovery path, which should survive any spinning worker (#756 and #788 describe other ways a worker wedges).

This is not new, only newly long: the same node logged the wedged-disconnected reconcile in 27 earlier episodes since 2026-08-13, each cleared within one or two reconciles.

Where

  • replication/subscriptionManager.ts (~1965): the wedged branch re-drives existing connected:false entries on the worker that already owns them. A worker that never returns to its event loop never receives the re-drive.
  • replication/replicationConnection.ts fire counters: the reconcile already classifies these fires as redundant and counts them, but nothing acts on the count.
  • core server/threads/manageThreads.js (~2456): the pinned-ELU detector only warns. The ack-timeout snapshot (~2378) also only logs, even with a resource report hours stale.

Expected

A subscription whose owning worker has stopped making progress is reassigned to a healthy worker within a bounded time, and the stuck worker is recycled (or the node otherwise reports itself unhealthy), without an external restart.

Possible shape, for whoever picks this up:

  1. Treat "owning worker has not sent a resource report / ack within N" the same as "pointing at an exited worker": reassign its subscriptions (the existing staleNodeUrls path already does this correctly, as the 11:58:24 lines show).
  2. Escalate after K consecutive redundant wedge fires for the same entries instead of re-firing indefinitely.
  3. Recycle a worker whose event loop has been pinned past a threshold, or at least fail the node's health check so the orchestrator restarts it.

Test

Unit level: a fake worker that stops answering (no resource reports, never processes the re-drive) while its subscriptions read connected:false; assert the subscriptions are reassigned to another worker within the bound, and that the reconcile does not re-fire on the dead worker past K attempts.

Related: HarperFast/harper#3308 (likely trigger), #788, #756, #810, HarperFast/dispatch#566 (the application-side guard added after this incident).

Filed by Claude Opus 5.5 (Claude Code) for @kriszyp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P1

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions