Repository navigation
Description
Workstream W1 of #430 · Foundation 1 · the linchpin for reliability and adaptive behavior
Status update (2026-07-01)
Since filing, the connection-truth gap has been patched by layered edge-triggered recovery mechanisms rather than a source of truth: the open-but-idle receive watchdog (#420 → #424), the copy-progress watchdog (issue #453), a separate copy-phase receive timeout, the pause-stall watchdog + wedge re-drive + never-connected backstop (#466), the reconcile-level fallback for connected:true copy stalls (#463), and the empty-subscription delayed-close repair (#471 → #475). That's ~6 overlapping watchdog/fallback layers (~97 watchdog references in replicationConnection.ts), and they've already produced their first interaction bug — the #466 false-positive force-reconnect in the wedge re-drive.
Two consequences for this workstream:
- The current-state risk is no longer "wedges with no recovery" — it's "overlapping recovery layers with subtle interactions." Every new watchdog raises the cost of reasoning about the next one.
- New acceptance criterion: once level-triggered truth exists, the point-fix watchdogs demote to telemetry/assertions — they should detect invariant violations and alert, not be load-bearing recovery paths. W13 (Replication W13: Base-copy & catch-up path (consolidation & correctness) #510) owns the copy-side consolidation.
The underlying claims below still verify on main: back-pressure is still read only by clusterStatus.ts (backPressurePercent), and the per-(db,node) worker.on('exit') listener is still attached in subscriptionManager.ts (#357 open).
Summary
Connection and health state has no single source of truth. The main-thread orchestrator (subscriptionManager.ts connectionReplicationMap) keeps an edge-triggered, inferred mirror of whether each peer is connected; the real sockets live on HTTP worker threads; live metrics (latency, back-pressure) live in shared-memory Float64Array buffers that the main thread never reads. These three views desync, which is the direct cause of a recurring class of connection-truth bugs — and it's the missing bridge that adaptive routing (#218) needs.
Root cause / current state
connectedis set only by edge-triggered IPC (connected-to-node/disconnected-from-node). It desyncs whenever a terminal/idle state is reached without firing the expected transition: a silently-dead socket never emits'close'(Outbound replication WebSocket can silently die without firing 'close', leaving (peer, db) stuck with no retry until process restart #233/Replication wedges permanently after simultaneous cluster restart (reconciler skips open-but-idle sockets) → blocks replicated deploys #420); a worker-load race arms a permanent "no subscriptions" close that pinsconnected:falseeven though the peer is reachable (Replication: ~9/11 outgoing peers show connected:false in connectionReplicationMap after restart despite being reachable #289).- Back-pressure is written by the sender (
updateBackPressureRatioinreplicationConnection.ts) and read only byclusterStatus.ts— it never reaches the orchestrator. - One
worker.on('exit')is attached per (db,node) subscription on a shared worker object (subscriptionManager.ts, see theworker?.on('exit', ...)in the subscription path) →MaxListenersExceededWarningpast ~10 databases (subscriptionManager registers a worker.on('exit') per database subscription → MaxListenersExceededWarning with >10 databases #357). The same anti-pattern (logical entity registers a listener on a long-lived shared emitter, nobody removes it deterministically) drivesforceResubscribeleaking into persistentsubscriptions-updatedlisteners (Replication: 5.0.31 wedge-reconcile spins at 100% CPU re-subscribing an unreachable peer (forceResubscribe leaks into persistent listeners) #349).
Design direction
- Move connection state into shared memory. Add connection-state, last-error-code, and a heartbeat-timestamp slot to the per-(db,node) status buffer (slots 9–15 are currently reserved), so the main thread reads truth directly instead of inferring it. This is the same channel Adaptive replication routing: dynamic fan-out under egress back-pressure #218 needs for back-pressure.
- A liveness watchdog owns
connected. No bytes + no pong within the ping timeout ⇒ down; a successful ping ⇒ up — independent of whether'close'ever fired. - One
'exit'handler per worker, consulting the registry, instead of one per subscription. - Idempotent resubscribe, with one-shot intents detached from connection lifetime (no binding to persistent listeners).
- Reset connection state on restart so outbound connections re-derive truth at boot.
remove_nodesynchronously closes sockets and drops registry entries;cluster_statusreports real membership (replace the hardcodedis_enabled:true).
Scope
- Extend shared-memory status schema: connection-state, last-error, heartbeat slots
- Watchdog-owned
connectedstate machine (liveness-derived) - Per-worker exit/reassign handler (replace per-subscription listeners)
- Idempotent resubscribe; detach
subscriptions-updatedfrom connection lifetime - Reset connection state on restart
-
remove_node→ synchronous socket close + entry removal + real membership flag - Demote the point-fix watchdogs (fix(replication): recover open-but-idle wedged subscriptions via watchdog-driven reconnect #424, a customer cluster preprod 5.1.7: replicated deploys wedge after rolling upgrade — stalled system blob send (#450/harper#1443) blocks COPY_COMPLETE; fixes absent from 5.1.7 (needs 5.1.8) #453-family, fix(replication): reconcile-level fallback for connected:true/Receiving copy-stall wedge #463, Replication: outbound subscription to a restarted peer wedges connected:false with no reconnect attempt (no SYN), reconcile never re-drives — base-copy + post-restart TLS window #466) to telemetry/assertions once level-triggered truth owns recovery
Retires / advances
- Replication: ~9/11 outgoing peers show connected:false in connectionReplicationMap after restart despite being reachable #289 — outgoing peers show connected:false after restart despite reachable
- Outbound replication WebSocket can silently die without firing 'close', leaving (peer, db) stuck with no retry until process restart #233 / Replication wedges permanently after simultaneous cluster restart (reconciler skips open-but-idle sockets) → blocks replicated deploys #420 — outbound WS dies without
'close'(closed via edge-triggered watchdogs fix(replication): recover open-but-idle wedged subscriptions via watchdog-driven reconnect #424/Replication: outbound subscription to a restarted peer wedges connected:false with no reconnect attempt (no SYN), reconcile never re-drives — base-copy + post-restart TLS window #466 — this workstream replaces those layers with level-triggered truth and retires the bug class) - subscriptionManager registers a worker.on('exit') per database subscription → MaxListenersExceededWarning with >10 databases #357 — worker.on('exit') per subscription → MaxListenersExceededWarning
- Replication: 5.0.31 wedge-reconcile spins at 100% CPU re-subscribing an unreachable peer (forceResubscribe leaks into persistent listeners) #349 — forceResubscribe leaks into persistent listeners (lifecycle half)
- cluster_status on a removed node should clearly indicate removal and close sockets #217 — removed node should indicate removal + close sockets
- Enables: Adaptive replication routing: dynamic fan-out under egress back-pressure #218 (W5), W6 (dedicated threads), W13 watchdog consolidation (Replication W13: Base-copy & catch-up path (consolidation & correctness) #510)
Dependencies
None upstream. Strongly recommended before W5/W6.
Effort / risk
L / medium. Highest leverage in the epic.
Acceptance criteria
connectedreflects reachability within the ping timeout regardless of whether'close'fired.- A cluster with >10 databases produces no MaxListeners warning.
- A removed node reports removed and its sockets are closed without a restart.
- The main thread can read connection state and back-pressure from shared memory.
- The layered point-fix watchdogs are no longer load-bearing recovery paths — they fire as telemetry/assertions only.
🤖 Filed by Claude on behalf of Kris.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
Make replication connection state authoritative via shared memory (W1, …
Make replication connection state authoritative via shared memory (W1, …
kriszyp commentedon Jul 6, 2026
Design proposal: level-triggered connection truth via the shared status buffer
Grounded in a code-level mapping of the current machinery at
714c0743(v5.1.14). Reviewed cross-model before posting.The defect in code terms
entry.connectedis written in exactly two places —subscriptionManager.ts:751-752(→false) and:827-828(→true) — both edge-triggered offdisconnected-from-node/connected-to-nodeIPC. There is no periodic re-assertion: between edges the bit is frozen, and it desyncs whenever a terminal state is reached without the expected transition event. The real liveness signal (ws._socket.bytesRead) lives on the worker where main can't read it; the ~7 watchdogs are edge-triggered recovery layers compensating for the missing level-triggered truth, and they've begun interacting (#466's false-positive force-reconnect).Foundation: extend the per-(db,node) shared buffer
The buffer (
knownNodes.ts:94-112) is 16 Float64 slots; 0–8 in use, 9–15 confirmed free (no constant, read, or write touches an index ≥ 9). Proposed layout:LIVENESS_HEARTBEAT_TIMEopen, everypong, every inbound frameCONNECTION_EPOCHLAST_CLOSE_CODELAST_STATE_CHANGE_TIMEdisconnectedAt ?? createdAtinferenceOwnership rule (load-bearing): slots 9–12 are written exclusively by the subscription (receive) connection — the same
isSubscriptionConnectionguard that gatesconnected-to-nodetoday (replicationConnection.ts:3534-3540). The buffer is shared with the send side (slot 3) and a send-side socket must never keep the heartbeat fresh while the subscription socket is dead. Single-writer also makes the non-atomicepoch++safe. Clock rule:Date.now()throughout —performance.now()origins differ per thread.Main-thread derivation
In the existing 5s
reconcileWorkerstick:A flat
PING_TIMEOUTthreshold suffices — no copy-mode widening.sendPingfires unconditionally every 30s and the pong handler stamps regardless ofpauseReasons, so a live socket's heartbeat stays fresh even during copy back-pressure pauses. Heartbeat = socket proof-of-life; progress stalls remain the job of the progress watchdogs. A silently-dead socket flips derived-down within one tick even ifclosenever fires (retires the #233/#420 class at the root); a stale heartbeat also catches a wedged worker, which no current watchdog does; reset-on-restart falls out naturally (heartbeat starts 0 → down until first proof-of-life).Watchdog demotion (the acceptance criterion)
Coordination notes
subscriptions-updatedlifecycle half is already handled on main (removed atreplicationConnection.ts:1064and:1148) — scope item effectively done.Sequencing (each step its own PR)
derivedConnectedalongside the edge bit, logs disagreements, changes no behavior. Validates the derivation against production reality before it drives anything.cluster_status(W8 Tier 1 overlap).🤖 Design by Claude (Opus 4.8, cross-reviewed by Fable 5) on behalf of Kris.
kriszyp commentedon Jul 6, 2026
Correction to the design comment above (twice-corrected — final state, verified against origin/main
bcd9c8f5):Already landed by PR #445 (merged 2026-07-02):
CONNECTION_STATE_POSITION=9paired withLAST_LIVENESS_TIME_POSITION=10(state AND fresh liveness must agree, so a dead/wedged worker cannot leave a stale CONNECTED), plusLAST_ERROR_CODE_POSITION=11/LAST_ERROR_TIME_POSITION=12. No epoch slot; 13–15 reserved. Treat Make replication connection state authoritative via shared memory (W1, #431) #445's layout as canonical; the comment's slot table is superseded.deriveConnectionTruth()/readConnectionTruth()helpers.reconcileWorkersalready corrects aconnected:true/undefined entry tofalsewhen shared-memory truth says down (subscriptionManager.tsin the 5s tick, ahead of the early-return), feeding the existing wedge re-drive. So the "shadow mode first" step in the design is moot for this direction — the cutover is live.Genuinely remaining in W1 core (branch
kris/w1-truth-shadow, in progress):connected === falseentries, so a lost/unprocessed connect edge (Replication: ~9/11 outgoing peers show connected:false in connectionReplicationMap after restart despite being reachable #289, the founding W1 bug) is still only healed by the wedge reconcile force-reconnecting a healthy link after 30s. Truth says CONNECTED with fresh liveness ⇒ correct the bit in place, no disruptive reconnect.Then the larger scope items stand as before: cluster_status exposure of the truth slots (W8 overlap), watchdog demotion once truth drives recovery, reset-on-restart, remove_node sync close, idempotent resubscribe.
🤖 Claude (Fable 5) on behalf of Kris.
fix(replication): gate connection-truth slots to the owning subscript…
fix(replication): also gate the back-pressure-pause liveness write (#431
kriszyp commentedon Jul 6, 2026
Design: watchdog demotion (the W1 acceptance criterion)
With the truth foundation in (#445), both reconcile directions + telemetry (#523), and liveness age visible in cluster_status (#524), here is the staged plan for demoting the layered watchdogs from load-bearing recovery to telemetry/assertions. Anchors verified against origin/main
bcd9c8f5.Principle
Preserve every detector; demote independent actuation. The risk today is ~6 recovery layers each independently deciding to tear down connections, with their first interaction bug already observed (#466 false-positive). The goal is one recovery driver keyed on level-triggered truth, with the other layers detecting and reporting — not acting — unless their condition is provably invisible to truth.
Classification — what truth can and cannot see
The dividing line: truth = socket proof-of-life (
CONNECTION_STATE+LAST_LIVENESS, stamped on handshake/pong/receive, owner-gated by #523). It sees dead/wedged sockets and dead/wedged workers (stale liveness). It deliberately does not see progress semantics — a link can be ping-alive with fresh liveness while a copy or apply pipeline is stalled (the pause path even refreshes liveness on purpose,replicationConnection.ts:1592).subscriptionManager.ts:104)connected !== truepast thresholdderiveConnectionTruthinstead of the reconciled bit to drop one 5s lag layer.createReceiveWatchdog:761)bytesReadfrozen pastPING_TIMEOUT/COPY_TIMEOUTLIVENESS_STALE_MSso worker-local terminate and main-derived down can't disagree for long.createPauseStallWatchdog:861):117)RECEIVED_TIMEfrozen while Receivingconnected-to-node/disconnected-from-node)The gap truth exposes: the blocked-worker recovery hole
Truth detects a wedged worker (stale liveness across all its links) — something no current watchdog does. But every main-thread recovery action is a
postMessageto the owning worker (subscribe-to-node/force-reconnect-node), which a blocked worker never processes, andfindStaleNodeUrlsonly reassigns entries of exited workers. So detection improves but recovery doesn't reach this case. The truth-driven actuator to add: when liveness for a worker's entries goes stale but the worker hasn't exited (and the peer is reachable from elsewhere), escalate — reassign the entries to a live worker. This needs duplicate-connection care (the wedged worker may still hold sockets that revive), so it should be designed as its own small step: likely terminate-then-reassign viaworker.terminate()after a bounded grace, making the exited-worker net do the rest. Flagged for its own PR + review, not prescribed here.Staged plan (each stage its own PR, evidence-gated)
state, liveness age), and whether another mechanism had already scheduled recovery for the same (db, peer) within its window. Cheap: they all already log; this adds the snapshot + cross-check. Gives the demotion evidence and immediately turns "watchdog fired while truth said connected+fresh" into a visible invariant violation.LIVENESS_STALE_MS. No mechanism removed.Also folded in
RECEIVED_TIME/RECEIVING_STATUSwrites are not owner-gated, so a retrieval connection may be able to mask a subscription receive-stall inisReceiveStalled. Needs verification first (does a retrieval connection ever traverse the audit-record apply path?); if real, the T1 PR gates those writes the same way fix(replication): reconcile connected bit UP from shared-memory truth; log corrections (#431) #523 gated the truth slots.Non-goals
Worker-local progress watchdogs stay actuators permanently (truth cannot see their conditions). Copy-side consolidation is W13's (#510). The empty-subscription close is lifecycle, untouched.
🤖 Design by Claude (Fable 5) on behalf of Kris.
kriszyp commentedon Jul 6, 2026
Design: T2 (single-driver keying — revised) and T4 (blocked-worker actuator)
Follow-up to the demotion design above; premises verified against the current W1 stack head (
kris/w1-t1-fire-telemetry, i.e. #523+#525 applied). One honest revision and one new latent interaction found while grounding T2.T2 — revised: the original keying idea is moot; a real tolerance mismatch takes its place
Dropping "key the wedge reconcile directly on derived truth." Post-#523 the truth-correction loop runs at the top of
reconcileWorkers, beforefindWedgedNodeUrlsin the same 5s tick — so theconnectedbit the wedge predicate reads is already truth-fresh at evaluation time. Direct keying would add a truth-unavailable fallback path (DB unmounting, never-connected entries withlastLiveness=0) for zero behavioral gain. The edge-IPC demotion is likewise already de facto complete: edges are now a fast-path hint + latency carrier, with disagreements corrected and logged every tick. No code needed for either.T2a (the real item) — copy-aware liveness window. Found while checking the alignment premise:
deriveConnectionTruthuses a flatLIVENESS_STALE_MS = max(120s, 2×PING_TIMEOUT), but the worker's own byte-silence tolerance is mode-dependent —PING_TIMEOUT(60s) normally,COPY_TIMEOUT(300s) during a base copy (#460: a sender stalled in write back-pressure mid-copy must not be timed out on the 60s window). The mismatch: a copy whose leader goes silent (leader-side stall — compaction, slow blob read; follower not paused, so the pause-liveness stamp doesn't apply) reads liveness-stale at 120s → down-correction ≈t+125s → wedge re-drive ≈t+155s — preempting the deliberate 300s copy tolerance by half, forcing a reconnect + copy-resume cycle every ~155s for as long as the leader stalls. Latent since #445's down-correction; #523 didn't change it (down-corrections were always the safe direction — this is a tolerance inconsistency, not data risk).Fix: the owner declares its current expected-max-silence window in the shared buffer — slot 13,
LIVENESS_WINDOW_POSITION— written (owner-gated, like slots 9–12) on mode transitions:COPY_TIMEOUTat COPY_START, back toPING_TIMEOUT×2on copy end / normal operation.deriveConnectionTruththen usesmax(LIVENESS_STALE_MS, declaredWindow × 1.25)(margin so the worker-local actuator always gets first chance, preserving the layering invariant: local actuator fires before the main net). Main stays generic — it never learns what "copy mode" is, it just honors the owner's declared tolerance. Zero-value slot (never written, e.g. pre-upgrade buffers) falls back toLIVENESS_STALE_MS, so mixed-version workers degrade to today's behavior.T2b — cleanups riding along: the DRY fire-telemetry helper from the #525 review (same code region), and a
PING_TIMEOUT ≤ LIVENESS_STALE_MSstructural assertion comment (already true by construction, worth pinning).Scope: one small PR, stacked on #525. Uses one reserved slot (13); 14–15 remain for W5/W6.
T4 — blocked-worker actuator (terminate-then-reassign)
The hole (from the demotion design): truth detects a blocked-but-not-exited worker — all its (db, peer) entries go liveness-stale — but every recovery action is a
postMessagethe blocked worker never processes, andfindStaleNodeUrlsonly reassigns entries of exited workers. Detection without recovery.Detection & the critical discriminator. Stale liveness across a worker's entries has two very different causes: the worker's event loop is blocked, or the network to those peers is down. Terminating a healthy worker over a network partition would be the classic watchdog-interaction bug this epic exists to end, so the discriminator is load-bearing:
DOWN+LAST_ERROR_TIME, retries update state — shared memory moves.Primary signal: a per-worker event-loop heartbeat — each worker stamps a per-worker shared slot from a timer (the same
getUserSharedBuffermachinery, keyed per thread rather than per (db, peer)); a blocked event loop stops stamping within one interval. This measures exactly the condition (event-loop health), not a proxy. Open question first: core already runs a "JavaScript execution has taken too long" monitor — if that signal is exposed (or can be) to the main thread, reuse it instead of a new heartbeat; if core's monitor already terminates blocked workers, T4 collapses to verifying the replication-side reassignment machinery (#446 exit handler + stale-worker net) picks up the pieces, and no new actuator is needed at all. Resolve before building anything.Escalation ladder (each stage observable before the next acts):
mechanism=worker-wedge-suspect).worker.terminate(). Thread death destroys its sockets (no zombie/duplicate connections — this is why terminate-then-reassign beats reassign-in-place, which would race a wedged-but-revivable worker holding live sockets), fires'exit', and the existing machinery does the rest: the per-worker exit handler (Bound replication worker exit listeners (#357); membership-aware cluster_status.is_enabled (#217) #446) clears its entries, the stale-worker net reassigns them to live workers. T4 adds no new reassignment path — it only converts "blocked" into "exited," which the system already heals from.Blast-radius controls (non-negotiable):
replication_workerWedgeRecovery(default off = stages 1–2 log-only). Enable after the soak shows suspect/confirm fires correlate with real wedges and zero false positives. The demotion epic's whole lesson is that un-soaked actuators interact; T4 doesn't get to skip its own medicine.Explicit non-triggers: network partition (heartbeat fresh → never suspect), worker busy-but-progressing (heartbeat stamps between work units), main-thread stalls (out of scope — nothing here runs off-main), copy stalls (T2a's territory; connection-level, worker healthy).
Sequencing: T4 design-gates on the core-monitor question; implementation after T2a and only once T1 telemetry is flowing in a release (its stage-1/2 logs ride the same soak).
🤖 Design by Claude (Fable 5) on behalf of Kris.
kriszyp commentedon Jul 6, 2026
T4 open question resolved (checked before anyone builds on it): core's "JavaScript execution has taken too long" monitor is
core/server/throttle.ts:62-74— a log-only, self-observing check. The blocked worker's ownsetIntervalmeasures event-queue lag and warns — which means it can only fire after the loop resumes (asetIntervalcan't run while its thread is blocked), it never terminates anything, and it writes nothing the main thread can read. It's the inverse of what T4 needs (an external observer of a thread that has stopped responding), so:🤖 Claude (Fable 5) on behalf of Kris.
feat(replication): expose connection-truth liveness age in cluster_st…
57 remaining items
Record what a worker-exit truth stamp did not land, and which writer …