Skip to content

Replication W1: Connection & health — single source of truth #431

Open
5 of 18 issues completed
Open

Description

@kriszyp

Workstream W1 of #430 · Foundation 1 · the linchpin for reliability and adaptive behavior

Status update (2026-07-01)

Since filing, the connection-truth gap has been patched by layered edge-triggered recovery mechanisms rather than a source of truth: the open-but-idle receive watchdog (#420 → #424), the copy-progress watchdog (issue #453), a separate copy-phase receive timeout, the pause-stall watchdog + wedge re-drive + never-connected backstop (#466), the reconcile-level fallback for connected:true copy stalls (#463), and the empty-subscription delayed-close repair (#471 → #475). That's ~6 overlapping watchdog/fallback layers (~97 watchdog references in replicationConnection.ts), and they've already produced their first interaction bug — the #466 false-positive force-reconnect in the wedge re-drive.

Two consequences for this workstream:

  1. The current-state risk is no longer "wedges with no recovery" — it's "overlapping recovery layers with subtle interactions." Every new watchdog raises the cost of reasoning about the next one.
  2. New acceptance criterion: once level-triggered truth exists, the point-fix watchdogs demote to telemetry/assertions — they should detect invariant violations and alert, not be load-bearing recovery paths. W13 (Replication W13: Base-copy & catch-up path (consolidation & correctness) #510) owns the copy-side consolidation.

The underlying claims below still verify on main: back-pressure is still read only by clusterStatus.ts (backPressurePercent), and the per-(db,node) worker.on('exit') listener is still attached in subscriptionManager.ts (#357 open).

Summary

Connection and health state has no single source of truth. The main-thread orchestrator (subscriptionManager.ts connectionReplicationMap) keeps an edge-triggered, inferred mirror of whether each peer is connected; the real sockets live on HTTP worker threads; live metrics (latency, back-pressure) live in shared-memory Float64Array buffers that the main thread never reads. These three views desync, which is the direct cause of a recurring class of connection-truth bugs — and it's the missing bridge that adaptive routing (#218) needs.

Root cause / current state

Design direction

  1. Move connection state into shared memory. Add connection-state, last-error-code, and a heartbeat-timestamp slot to the per-(db,node) status buffer (slots 9–15 are currently reserved), so the main thread reads truth directly instead of inferring it. This is the same channel Adaptive replication routing: dynamic fan-out under egress back-pressure #218 needs for back-pressure.
  2. A liveness watchdog owns connected. No bytes + no pong within the ping timeout ⇒ down; a successful ping ⇒ up — independent of whether 'close' ever fired.
  3. One 'exit' handler per worker, consulting the registry, instead of one per subscription.
  4. Idempotent resubscribe, with one-shot intents detached from connection lifetime (no binding to persistent listeners).
  5. Reset connection state on restart so outbound connections re-derive truth at boot.
  6. remove_node synchronously closes sockets and drops registry entries; cluster_status reports real membership (replace the hardcoded is_enabled:true).

Scope

Retires / advances

Dependencies

None upstream. Strongly recommended before W5/W6.

Effort / risk

L / medium. Highest leverage in the epic.

Acceptance criteria

  • connected reflects reachability within the ping timeout regardless of whether 'close' fired.
  • A cluster with >10 databases produces no MaxListeners warning.
  • A removed node reports removed and its sockets are closed without a restart.
  • The main thread can read connection state and back-pressure from shared memory.
  • The layered point-fix watchdogs are no longer load-bearing recovery paths — they fire as telemetry/assertions only.

🤖 Filed by Claude on behalf of Kris.

Activity

  1. added this to the v5.2 milestone on Jun 20, 2026
  2. self-assigned this
    on Jun 20, 2026
  3. kriszyp commented on Jul 6, 2026

    @kriszyp
    MemberAuthor

    Design proposal: level-triggered connection truth via the shared status buffer

    Grounded in a code-level mapping of the current machinery at 714c0743 (v5.1.14). Reviewed cross-model before posting.

    The defect in code terms

    entry.connected is written in exactly two places — subscriptionManager.ts:751-752 (→false) and :827-828 (→true) — both edge-triggered off disconnected-from-node / connected-to-node IPC. There is no periodic re-assertion: between edges the bit is frozen, and it desyncs whenever a terminal state is reached without the expected transition event. The real liveness signal (ws._socket.bytesRead) lives on the worker where main can't read it; the ~7 watchdogs are edge-triggered recovery layers compensating for the missing level-triggered truth, and they've begun interacting (#466's false-positive force-reconnect).

    Foundation: extend the per-(db,node) shared buffer

    The buffer (knownNodes.ts:94-112) is 16 Float64 slots; 0–8 in use, 9–15 confirmed free (no constant, read, or write touches an index ≥ 9). Proposed layout:

    Slot Constant Written on Purpose
    9 LIVENESS_HEARTBEAT_TIME WS open, every pong, every inbound frame level-triggered proof-of-life main can read
    10 CONNECTION_EPOCH each successful (re)connect (increment) flap detection; "still alive" vs "reconnected"; feeds #218
    11 LAST_CLOSE_CODE socket close/error last-error for cluster_status + classification
    12 LAST_STATE_CHANGE_TIME connect/disconnect transition replaces main's disconnectedAt ?? createdAt inference
    13–15 reserved — adaptive routing (W5), send-side liveness, dedicated-thread id (W6)

    Ownership rule (load-bearing): slots 9–12 are written exclusively by the subscription (receive) connection — the same isSubscriptionConnection guard that gates connected-to-node today (replicationConnection.ts:3534-3540). The buffer is shared with the send side (slot 3) and a send-side socket must never keep the heartbeat fresh while the subscription socket is dead. Single-writer also makes the non-atomic epoch++ safe. Clock rule: Date.now() throughout — performance.now() origins differ per thread.

    Main-thread derivation

    In the existing 5s reconcileWorkers tick:

    derivedConnected = heartbeat > 0 && (Date.now() - heartbeat) < PING_TIMEOUT
    

    A flat PING_TIMEOUT threshold suffices — no copy-mode widening. sendPing fires unconditionally every 30s and the pong handler stamps regardless of pauseReasons, so a live socket's heartbeat stays fresh even during copy back-pressure pauses. Heartbeat = socket proof-of-life; progress stalls remain the job of the progress watchdogs. A silently-dead socket flips derived-down within one tick even if close never fires (retires the #233/#420 class at the root); a stale heartbeat also catches a wedged worker, which no current watchdog does; reset-on-restart falls out naturally (heartbeat starts 0 → down until first proof-of-life).

    Watchdog demotion (the acceptance criterion)

    Coordination notes

    Sequencing (each step its own PR)

    1. Additive slots — constants + subscription-connection stamping of 9–12. No reader; abandonable; unit-testable in isolation. (Starting now.)
    2. Shadow mode — main derives derivedConnected alongside the edge bit, logs disagreements, changes no behavior. Validates the derivation against production reality before it drives anything.
    3. Cutover — reconnect decisions keyed on derived truth; fix(replication): reconcile-level fallback for connected:true/Receiving copy-stall wedge #463 demoted to assertion.
    4. Consolidation — remaining watchdog demotion; expose new slots in cluster_status (W8 Tier 1 overlap).

    🤖 Design by Claude (Opus 4.8, cross-reviewed by Fable 5) on behalf of Kris.

  4. kriszyp commented on Jul 6, 2026

    @kriszyp
    MemberAuthor

    Correction to the design comment above (twice-corrected — final state, verified against origin/main bcd9c8f5):

    Already landed by PR #445 (merged 2026-07-02):

    • The shared-memory slots, with a better layout than the table proposed above: CONNECTION_STATE_POSITION=9 paired with LAST_LIVENESS_TIME_POSITION=10 (state AND fresh liveness must agree, so a dead/wedged worker cannot leave a stale CONNECTED), plus LAST_ERROR_CODE_POSITION=11 / LAST_ERROR_TIME_POSITION=12. No epoch slot; 13–15 reserved. Treat Make replication connection state authoritative via shared memory (W1, #431) #445's layout as canonical; the comment's slot table is superseded.
    • Worker-side stamping on handshake / pong / receive / close, and the pure deriveConnectionTruth() / readConnectionTruth() helpers.
    • The down-direction consumer: reconcileWorkers already corrects a connected:true/undefined entry to false when shared-memory truth says down (subscriptionManager.ts in the 5s tick, ahead of the early-return), feeding the existing wedge re-drive. So the "shadow mode first" step in the design is moot for this direction — the cutover is live.

    Genuinely remaining in W1 core (branch kris/w1-truth-shadow, in progress):

    1. Up-direction reconcile — the loop skips connected === false entries, so a lost/unprocessed connect edge (Replication: ~9/11 outgoing peers show connected:false in connectionReplicationMap after restart despite being reachable #289, the founding W1 bug) is still only healed by the wedge reconcile force-reconnecting a healthy link after 30s. Truth says CONNECTED with fresh liveness ⇒ correct the bit in place, no disruptive reconnect.
    2. Correction telemetry — both directions should log when the edge bit disagreed with truth (they're transitions, self-limiting), so truth-vs-edge quality is observable in production before deeper watchdog demotion leans on it.

    Then the larger scope items stand as before: cluster_status exposure of the truth slots (W8 overlap), watchdog demotion once truth drives recovery, reset-on-restart, remove_node sync close, idempotent resubscribe.

    🤖 Claude (Fable 5) on behalf of Kris.

  5. kriszyp commented on Jul 6, 2026

    @kriszyp
    MemberAuthor

    Design: watchdog demotion (the W1 acceptance criterion)

    With the truth foundation in (#445), both reconcile directions + telemetry (#523), and liveness age visible in cluster_status (#524), here is the staged plan for demoting the layered watchdogs from load-bearing recovery to telemetry/assertions. Anchors verified against origin/main bcd9c8f5.

    Principle

    Preserve every detector; demote independent actuation. The risk today is ~6 recovery layers each independently deciding to tear down connections, with their first interaction bug already observed (#466 false-positive). The goal is one recovery driver keyed on level-triggered truth, with the other layers detecting and reporting — not acting — unless their condition is provably invisible to truth.

    Classification — what truth can and cannot see

    The dividing line: truth = socket proof-of-life (CONNECTION_STATE + LAST_LIVENESS, stamped on handshake/pong/receive, owner-gated by #523). It sees dead/wedged sockets and dead/wedged workers (stale liveness). It deliberately does not see progress semantics — a link can be ping-alive with fresh liveness while a copy or apply pipeline is stalled (the pause path even refreshes liveness on purpose, replicationConnection.ts:1592).

    Mechanism Keys on Truth sees it? Disposition
    Wedge reconcile (30s, subscriptionManager.ts:104) connected !== true past threshold ✅ (bit is now truth-reconciled both directions) Promote to the single main-thread recovery driver. Optionally key directly on deriveConnectionTruth instead of the reconciled bit to drop one 5s lag layer.
    Byte-silence receive watchdog (#420/#424, createReceiveWatchdog:761) bytesRead frozen past PING_TIMEOUT/COPY_TIMEOUT Partially — it is the worker-local twin of liveness Keep as the fast local actuator; align its window with LIVENESS_STALE_MS so worker-local terminate and main-derived down can't disagree for long.
    Pause-stall watchdog (#466, createPauseStallWatchdog:861) zero consumer progress while paused ❌ (liveness deliberately fresh during pause) Keep worker-local. Not demotable.
    Copy-progress watchdog (#453/#454) copy app-frame progress ❌ (ping-alive) Keep; W13 (#510) owns copy-side consolidation.
    Receive-stall reconcile net (#463, 15min, :117) fresh RECEIVED_TIME frozen while Receiving ❌ for progress, ✅ for its worker-death subcase Demote candidate — the only true demotion in this plan. Its residual value is "copy-progress watchdog failed to fire"; evidence-gated below.
    Empty-subscription delayed close (#471/#475) subscription lifecycle n/a Out of scope — a lifecycle decision, not a recovery net.
    IPC edges (connected-to-node/disconnected-from-node) transition events superseded Demote to fast-path hint + latency carrier. Already effectively true post-#523: the reconcile corrects both directions every 5s.

    The gap truth exposes: the blocked-worker recovery hole

    Truth detects a wedged worker (stale liveness across all its links) — something no current watchdog does. But every main-thread recovery action is a postMessage to the owning worker (subscribe-to-node / force-reconnect-node), which a blocked worker never processes, and findStaleNodeUrls only reassigns entries of exited workers. So detection improves but recovery doesn't reach this case. The truth-driven actuator to add: when liveness for a worker's entries goes stale but the worker hasn't exited (and the peer is reachable from elsewhere), escalate — reassign the entries to a live worker. This needs duplicate-connection care (the wedged worker may still hold sockets that revive), so it should be designed as its own small step: likely terminate-then-reassign via worker.terminate() after a bounded grace, making the exited-worker net do the rest. Flagged for its own PR + review, not prescribed here.

    Staged plan (each stage its own PR, evidence-gated)

    • T1 — Structured fire telemetry. Every net logs a structured record on fire: which mechanism, the truth snapshot at fire time (state, liveness age), and whether another mechanism had already scheduled recovery for the same (db, peer) within its window. Cheap: they all already log; this adds the snapshot + cross-check. Gives the demotion evidence and immediately turns "watchdog fired while truth said connected+fresh" into a visible invariant violation.
    • T2 — Single-driver keying. Wedge reconcile keys directly on derived truth; the byte watchdog's window is aligned with LIVENESS_STALE_MS. No mechanism removed.
    • T3 — Evidence-gated demotion of fix(replication): reconcile-level fallback for connected:true/Receiving copy-stall wedge #463. Demote the receive-stall net to alert-only once T1 telemetry shows, over an agreed soak (proposal: 4 weeks fleet + the stress suite), zero fires where it was the sole detector (i.e., every fire was preceded by the copy-progress watchdog or truth-driven recovery). If the soak shows sole-detector fires, it stays — the telemetry then tells us exactly which condition the primary layers miss.
    • T4 — Blocked-worker actuator (the reassign escalation above), separately designed and reviewed.

    Also folded in

    Non-goals

    Worker-local progress watchdogs stay actuators permanently (truth cannot see their conditions). Copy-side consolidation is W13's (#510). The empty-subscription close is lifecycle, untouched.

    🤖 Design by Claude (Fable 5) on behalf of Kris.

  6. kriszyp commented on Jul 6, 2026

    @kriszyp
    MemberAuthor

    Design: T2 (single-driver keying — revised) and T4 (blocked-worker actuator)

    Follow-up to the demotion design above; premises verified against the current W1 stack head (kris/w1-t1-fire-telemetry, i.e. #523+#525 applied). One honest revision and one new latent interaction found while grounding T2.


    T2 — revised: the original keying idea is moot; a real tolerance mismatch takes its place

    Dropping "key the wedge reconcile directly on derived truth." Post-#523 the truth-correction loop runs at the top of reconcileWorkers, before findWedgedNodeUrls in the same 5s tick — so the connected bit the wedge predicate reads is already truth-fresh at evaluation time. Direct keying would add a truth-unavailable fallback path (DB unmounting, never-connected entries with lastLiveness=0) for zero behavioral gain. The edge-IPC demotion is likewise already de facto complete: edges are now a fast-path hint + latency carrier, with disagreements corrected and logged every tick. No code needed for either.

    T2a (the real item) — copy-aware liveness window. Found while checking the alignment premise: deriveConnectionTruth uses a flat LIVENESS_STALE_MS = max(120s, 2×PING_TIMEOUT), but the worker's own byte-silence tolerance is mode-dependent — PING_TIMEOUT (60s) normally, COPY_TIMEOUT (300s) during a base copy (#460: a sender stalled in write back-pressure mid-copy must not be timed out on the 60s window). The mismatch: a copy whose leader goes silent (leader-side stall — compaction, slow blob read; follower not paused, so the pause-liveness stamp doesn't apply) reads liveness-stale at 120s → down-correction ≈t+125s → wedge re-drive ≈t+155s — preempting the deliberate 300s copy tolerance by half, forcing a reconnect + copy-resume cycle every ~155s for as long as the leader stalls. Latent since #445's down-correction; #523 didn't change it (down-corrections were always the safe direction — this is a tolerance inconsistency, not data risk).

    Fix: the owner declares its current expected-max-silence window in the shared buffer — slot 13, LIVENESS_WINDOW_POSITION — written (owner-gated, like slots 9–12) on mode transitions: COPY_TIMEOUT at COPY_START, back to PING_TIMEOUT×2 on copy end / normal operation. deriveConnectionTruth then uses max(LIVENESS_STALE_MS, declaredWindow × 1.25) (margin so the worker-local actuator always gets first chance, preserving the layering invariant: local actuator fires before the main net). Main stays generic — it never learns what "copy mode" is, it just honors the owner's declared tolerance. Zero-value slot (never written, e.g. pre-upgrade buffers) falls back to LIVENESS_STALE_MS, so mixed-version workers degrade to today's behavior.

    T2b — cleanups riding along: the DRY fire-telemetry helper from the #525 review (same code region), and a PING_TIMEOUT ≤ LIVENESS_STALE_MS structural assertion comment (already true by construction, worth pinning).

    Scope: one small PR, stacked on #525. Uses one reserved slot (13); 14–15 remain for W5/W6.


    T4 — blocked-worker actuator (terminate-then-reassign)

    The hole (from the demotion design): truth detects a blocked-but-not-exited worker — all its (db, peer) entries go liveness-stale — but every recovery action is a postMessage the blocked worker never processes, and findStaleNodeUrls only reassigns entries of exited workers. Detection without recovery.

    Detection & the critical discriminator. Stale liveness across a worker's entries has two very different causes: the worker's event loop is blocked, or the network to those peers is down. Terminating a healthy worker over a network partition would be the classic watchdog-interaction bug this epic exists to end, so the discriminator is load-bearing:

    • A network-down worker still writes: close events stamp DOWN + LAST_ERROR_TIME, retries update state — shared memory moves.
    • A blocked worker writes nothing: every slot it owns is frozen, and it doesn't process re-drives.

    Primary signal: a per-worker event-loop heartbeat — each worker stamps a per-worker shared slot from a timer (the same getUserSharedBuffer machinery, keyed per thread rather than per (db, peer)); a blocked event loop stops stamping within one interval. This measures exactly the condition (event-loop health), not a proxy. Open question first: core already runs a "JavaScript execution has taken too long" monitor — if that signal is exposed (or can be) to the main thread, reuse it instead of a new heartbeat; if core's monitor already terminates blocked workers, T4 collapses to verifying the replication-side reassignment machinery (#446 exit handler + stale-worker net) picks up the pieces, and no new actuator is needed at all. Resolve before building anything.

    Escalation ladder (each stage observable before the next acts):

    1. Suspect — worker heartbeat stale > H (H ≈ 2× heartbeat interval) and ≥1 of its entries derived-down and a re-drive was posted with no shared-memory movement since. Log loud with the T1 snapshot format (mechanism=worker-wedge-suspect).
    2. Confirm — condition persists across a second reconcile window (blocks the transient-GC-pause false positive).
    3. Actuate — worker.terminate(). Thread death destroys its sockets (no zombie/duplicate connections — this is why terminate-then-reassign beats reassign-in-place, which would race a wedged-but-revivable worker holding live sockets), fires 'exit', and the existing machinery does the rest: the per-worker exit handler (Bound replication worker exit listeners (#357); membership-aware cluster_status.is_enabled (#217) #446) clears its entries, the stale-worker net reassigns them to live workers. T4 adds no new reassignment path — it only converts "blocked" into "exited," which the system already heals from.

    Blast-radius controls (non-negotiable):

    • A replication worker is also an HTTP worker; terminating one kills in-flight requests. Mitigation: a worker whose event loop is blocked isn't serving HTTP either — but the heartbeat threshold must be well above worst-case legitimate synchronous work (large sync decode, GC). Propose H ≥ 30s stale + the stage-2 confirm.
    • Rate limit: at most one termination per process per cooldown (10 min), so a systemic cause (e.g. shared-resource stall blocking all workers) can't cascade into terminating the fleet.
    • Telemetry-first rollout, mirroring T1→T3: stage 3 ships behind replication_workerWedgeRecovery (default off = stages 1–2 log-only). Enable after the soak shows suspect/confirm fires correlate with real wedges and zero false positives. The demotion epic's whole lesson is that un-soaked actuators interact; T4 doesn't get to skip its own medicine.

    Explicit non-triggers: network partition (heartbeat fresh → never suspect), worker busy-but-progressing (heartbeat stamps between work units), main-thread stalls (out of scope — nothing here runs off-main), copy stalls (T2a's territory; connection-level, worker healthy).

    Sequencing: T4 design-gates on the core-monitor question; implementation after T2a and only once T1 telemetry is flowing in a release (its stage-1/2 logs ride the same soak).

    🤖 Design by Claude (Fable 5) on behalf of Kris.

  7. kriszyp commented on Jul 6, 2026

    @kriszyp
    MemberAuthor

    T4 open question resolved (checked before anyone builds on it): core's "JavaScript execution has taken too long" monitor is core/server/throttle.ts:62-74 — a log-only, self-observing check. The blocked worker's own setInterval measures event-queue lag and warns — which means it can only fire after the loop resumes (a setInterval can't run while its thread is blocked), it never terminates anything, and it writes nothing the main thread can read. It's the inverse of what T4 needs (an external observer of a thread that has stopped responding), so:

    • Reuse is not viable — the per-worker heartbeat slot stands as T4's detection mechanism.
    • T4 does not collapse — no existing mechanism terminates a blocked worker; the actuator is genuinely missing.
    • One small synergy: when T4's stage-1 suspect fires and the worker later recovers, core's monitor will have logged the lag from the inside — the two log lines cross-corroborate in the soak analysis.

    🤖 Claude (Fable 5) on behalf of Kris.

  8. 57 remaining items

  9. modified the milestones: v5.2, v5.4 on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:replicationReplication, cluster sync, peer connectionsenhancementNew feature or request

Type

Fields

Priority

P2

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions