Skip to content

Hold a durable subscription's position below every local write still in flight - #3635

Merged
kriszyp merged 21 commits into
mainfrom
fix/durable-position-below-inflight-writes
Oct 30, 2026
Merged

kriszyp merged 21 commits into
mainfrom
fix/durable-position-below-inflight-writes

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 27, 2026 •

Copy link
Copy Markdown
Member
Kris on belay slides the reached chalk mark to just below his daughter, who is still mid-move in the crack

⊙ Problem

A RocksDB transaction takes its local log key at its first staged write and commits later. A durable MQTT session saved the dispatched watermark, which could already sit above such a key. When that transaction committed, its message was delivered below the saved position, so a reconnect skipped it (harper#3112). A subscription joining meanwhile started there too.

🤔 Fixed at the watermark, not only in the session

The fix changes the database's watermark, which a joining subscription's registeredThrough reads too. The issue asks about the saved position.

💡 Solution

Each cross-thread notify pass captures the lowest outstanding local reservation before it drains and, after a complete drain, certifies min(raw watermark, that ceiling). progress() and registeredThrough return the certified mark. A session also stays below a late event still queued behind higher keys: the mark certifies dispatch, not delivery. QoS 2 receipt compaction takes the same bound.

⚖️ Alternatives

Framing-Verdict: chosen-approach-sound

Planning records: initial 289902ca7eb2, queued-late bound f85afec45842, receipt-compaction recheck a43258e7, scalar replay 9030b3163b84.

Rejected candidates, measured
Approach Result Why not
Cap at the published floor 15/20 durable-resume tests lags up to 5 s; acknowledged messages resent after a fast reconnect
Fresh registry scan per advance 20/20 5.7–15.6 µs per advance at one advance per write, and it certifies a released, undispatched key
Scan at most every N ms 19/20 the final save is stale unless forced; saves nothing over a 100 ns scan
Bound only at persist time — every advancePositions is followed by a persist, and receipts are pruned before it
Clamp only the replay floor Safe, but stalls a held registration sample never catches the quiet replay’s already acknowledged higher key after release
Outstanding-reservation counter 20/20 an atomic add and sub on every write to skip a scan that binds 0–26% of the time on one thread

🔧 Changes

🤔 A queued late event holds only its own session

The send wrapper records late sends by ordinal; the session stays below the lowest it has not consumed. Holding the shared mark until delivery would tie every subscriber to the slowest one. Each send prunes the list by queue length. This keeps it bounded for a consumer that never asks. So the optional interface method requires a sequential consumer. MQTT’s for-await loop, its only in-tree caller, is one.

🤔 Capture before the drain, never a scan per advance

A reservation is released a microtask after its commit settles, before the next pass reads its entry. So a scan after a drain or at read time certifies past it; a session test reproduces it.

🤔 An aborted hold waits for the capture tick

An aborted hold clears at the next capture tick, up to 5 s later. A committed one clears with the commit's pass or its one follow-up. A longer hold waits too: a delay, never a skip. Kris chose this over a rate-limited read-triggered pass.

🤔 Same-thread registries get the minimum

crossThreads: false progress consumers (none in-tree) take this thread's own lowest reservation as the ceiling; their re-capture timer can arm and clear once per commit, accepted.

🤔 QoS 2 receipts wait for the verdict and certified progress

Before a resumed replay's verdict, a lower key can still be unread anywhere in the rest of the log, so no receipt collapses. After it, receipts collapse only below certified progress, which an open local write holds. Past 1,000 receipts kept either way, a disconnect resends the overflow: a duplicate, not a lost key. The limit and its overflow rule predate this PR.

🤔 Completed replay still needs reservation certification

A scalar replay's walked key is capped by its registration sample. Its observed key becomes a raw candidate, certified after a ceiling-bounded drain. Clamping forever would stall quiet catch-up. Same-thread replay uses only its live certificate because replay reads all threads. goLive runs synchronously before verdict callbacks; pre-verdict positions are not checkpointable. Record resumes retain the requested start, not the replay’s walked key.

🤔 Registration defines the live-delivery interval

New subscriptions receive commits made after registration. A low reservation-bounded start can replay earlier keys after an early reconnect; it does not require their live delivery. Replay candidates may pass pre-registration keys while post-registration commits remain covered by the complete-drain gate. Existing durable topics reconnect through their own checked replay and cannot save beyond its start before verification.

🤔 Keep isolated layout fixtures

The slot-limit restore is confined to synchronous fixtures. Unit workers are terminated in afterEach; the benchmark starts workers only in its later throughput case. Restoring the fixture’s prior limit can hide no concurrent claimer. Hard-coded offsets deliberately pin the shared layout. The benchmark comment explains the contention being measured; retaining it needs no production test API.

🤔 #3248 claims header word 4; WORD_SLOT_LIMIT takes word 5

Merged #3248 puts WORD_INITIAL_CEILING at word 4, so this PR puts WORD_SLOT_LIMIT at word 5, raises HEADER_WORDS from 5 to 8, updates its fixture offsets, and asserts that all header word constants are distinct and below HEADER_WORDS.

🤔 A worker-only restart fails loudly

A worker running this code in a process whose buffer has an earlier header (5 words on main, 4 words in v5.4.0-alpha.1) throws as it opens the log, telling the operator to restart the process. It never misreads slots silently. Only main and the prerelease v5.4.0-alpha.1 have the old layout, so no upgrade from a GA release hits this. The rollout note below asks for a full-process restart.

🤔 Preserve a safe prefix after a failed read

A failed read stops further certification and makes deliveryCertified false. The previously certified prefix remains safe, so it stays available rather than being revoked. subscribeProgress.test.js checks that the mark stays frozen after a failed read. A joining subscriber cannot certify delivery through the failed range.

🤔 Keep the registration capture on every progress subscription

The registration drain can clear the last certified mark, and a scalar replay needs a sample taken before its cursor opens. Each registration retains that sample even when a shared certificate exists. This adds one capture per SUBSCRIBE, with no new work in the per-write path; reconnect-burst cost remains a measured tradeoff rather than a throughput claim.

🤔 Keep database-level lag diagnostics

The existing reservation holders and floor-lag diagnostics describe the shared database cause. Per-session warnings would repeat that cause for every affected subscriber. Additional alerting remains separate work; this fix preserves the safety bound while a hold remains open.

Accessor and cancellation boundaries

🤔 Keep the certified accessor name

certifiedThrough replaces the former internal dispatchedThrough accessor, and all core consumers use it. The root API does not export that accessor. This review covers the authorized core checkout; private-file imports in downstream repositories remain outside that verification.

🤔 Closed replay remains uncheckpointable

A close on the final replay delivery settles verification false. If goLive then encounters the detached registration, the closed-subscription handler absorbs that rejection and progress() returns undefined. No checkpoint advances. An activity guard would be cleanup with no change to the cancellation outcome.

The simplified pass (transactionBroadcast.ts:530):

notifyFromTransactionData(subscriptions, range) {
  if (freshPass) passCeiling = captureCeiling(auditStore, sameThread)   // ⚠ before the first read
  for (const entry of range) { … advanceProgress(dispatchedThrough) … }   // ⏳ a yield keeps passCeiling
  if (drained && !failed) certifiedThrough = max(certifiedThrough, min(dispatchedThrough, passCeiling))
  if (passCeiling < dispatchedThrough && (readEntries || heldByReservation)) recheckPending = true  // ↻ one follow-up
}
Product and architecture tour

Product and architecture tour

Certified watermark

What may a durable subscription save as its position, and when does it move?

Every local write at or below the certified mark committed since a subscriber registered was dispatched to it, and none can still land there. Completed replay covers earlier history for a resumed subscriber. A key below the pass's ceiling was committed before the capture, so the drain read it. A key reserved after the capture is above the capture's clock sample.

  • The follow-up pass re-captures and certifies without another commit. An explicit transaction releases its reservation after its commit settles, so the commit's pass can capture first (certifyProgress, scheduled from the pass's cleanup).
  • Registration samples the reservation bound before the registration drain: localStartBound across threads, threadStartBound on the same thread. It prefers the certified mark when one exists; the registration drain can end the last consumer and clear the mark.
  • progress() on a table subscription reads the certified mark; the raw dispatchedThrough field stays internal. Completed scalar replay retains a reservation-bounded private floor and submits its observed key to the same certification loop; it cannot override a lower outstanding reservation.
Before After
A saved position could pass a key still in flight on another thread; the late commit was skipped on resume. The position holds below every outstanding local key and every committed key not yet dispatched; a held, aborted write releases it at the next capture tick.

Slot limit

Why does the bounded scan still see every claimed slot?

originClosedFloor.ts keeps one past the highest claimed slot in a header word. A claimer raises it right after it owns the slot, before reserve reads proposed; a sampler raises proposed before it loads the limit, so a claim the scan misses admits only keys above the sampler's value. minBound and holders stop at the limit, which never falls. The header grows to a full line, so the floor test and the bench follow the new offsets.

Same-thread registries

What does a crossThreads: false progress consumer get?

threadStartBound reads this thread's own lowest reservation after raising proposed; that registry runs no capture tick, and its follow-up and abort re-capture timer read an empty log, so the shared log range stays the cross-thread registry's to drain. No in-tree code combines crossThreads: false with reportProgress.

Queued late events

Why can a certified position still be too high for one session?

progress() certifies that an event reached the subscription's queue. A late event is sent below a key sent earlier (a commit that took its key early, or a resumed replay reading the log in commit order). It can wait in the queue while the session delivers higher keys, so keyBefore passes it. lowestLateUnconsumed returns the lowest such key the session has not consumed; nextPosition and compactReceived both take it. Every other queued event is at or above the last delivered key.

✅ Verification

  • Focused RocksDB: 153 passing (3 pending); LMDB: 54 passing (14 pending); broadcaster neighbours: 41 passing.
  • MQTT integration under uWS: 20/20. Build, strict lint, formatting, design-doc budgets and unit partition pass.
  • The scalar-replay reconnect regression fails before this fix and passes now. Resource tests cover quiet catch-up, shared subscribers, same-thread exclusion, yielded drains and failed reads.

🤔 Two tests force their window

The release-before-dispatch test holds the pass by setting notifyScheduled: a reservation is released a microtask after its commit, so no real schedule opens that window reliably. A worker thread parked mid-reservation covers a real cross-thread release. The unverified-replay test stalls on a promise, not on its 50 ms sleep, and ends by checking that a reconnect delivers the late key.

The other new tests, the QA reproductions, and the cost
  • Docs: none needed; reportProgress keeps its shape.
  • PR QA: prior request qa-pr-harper-3635-d42d2e8 covered the old head; current CI follows this push.

🤖 Generated by Claude Fable 5.1, Claude Opus 5.5 (Claude Code), OpenAI Codex, and Google Gemini (maintenance); posted via @kriszyp.

Release target: v5.4 — builds on #3602 and #3629, both v5.4; the certified mark changes subscription position behavior

Related PRs: #3602 overlaps (in the base; the registration start this certifies), #3629 overlaps (in the base; the pending-slot bound the capture uses), #3137 overlaps (certifiedThrough consumer in Table.ts progress()), #3248 overlaps (merged into the base; admission order in originClosedFloor.ts), #3643 overlaps (merged into the base; new durable MQTT topic start on the same reconnect path), #3778 overlaps (session verification/position gates in DurableSubscriptionsSession.ts), #3733 overlaps (durable MQTT section in server/DESIGN.md), #3728 independent (replication), #3775 independent (replication), #3479 independent (durable MQTT re-subscribe QoS; closed), #3714 overlaps (merged into the base; full-text fixtures and readiness), #3638 overlaps (merged into the base; QA-769 readiness), #3469 overlaps (merged into the base; full PUT replay semantics), #2155 independent (expiry sweeps), #3527 independent (schema sharding)
Complexity: complicated

Origin — the dispatch brief this PR was written from

CONSULT FIRST: open a needs-input consultation with your design and measurements before implementing. Investigate harper#3112: a durable MQTT session can save a position (via the subscription's progress(), resources/Table.ts, or the dispatched-through watermark in resources/transactionBroadcast.ts) above a local transaction that has taken its log key but not committed; when it commits, its delivery is below the saved position and a resume skips it. Deliverable: measurements plus a design that keeps saved positions below every in-flight local write without redelivering acknowledged messages after a fast reconnect and without a material write-throughput cost. Stop at the design for Kris's sign-off.

Acceptance

A needs-input consultation containing: measured frequency of outstanding reservations under a steady write load; cost per advance and write-throughput impact for each candidate considered (same put-loop method: ~77k writes/s baseline); behavior of the two durable-resume integration tests under each; the recommended design with its invariant and rejected alternatives with the fact that rules each out. No PR until Kris signs off on the design.

Dispatch: task harper-3112-live-position-cap-design · queued by kris-session · ran by claude/fable/high · worker kzyp-xps-1

Late Grok findings

8 surviving (highest minor) — Grok reviewed b1e51a1
  • minor — resources/transactionBroadcast.ts:489 — every fresh notify pass with a progress consumer now calls localStartBound, which raises WORD_PROPOSED on each commit burst. A write whose handle was created before the latest raise then needs a second freshKey(). The cost moved from the 5 s tick onto the commit path. Grading follows the recor
  • minor — resources/transactionBroadcast.ts:301 — only one follow-up pass per ceiling (recheckedCeiling). If the reservation is held on another thread whose event loop is busy past that follow-up, certified progress stays held until the next commit or the 5 s capture tick. This is delay, not loss
  • minor — resources/transactionBroadcast.ts:311 — the same-thread re-capture timer branch is reached only by a sameThread registry with a progress consumer. The open record says no production caller exists. Unverified here; resolved by grepping for crossThreads: false subscribers that use progress
  • minor — resources/transactionBroadcast.ts:275 — export dispatchedThrough was replaced by certifiedThrough. Any external caller (harper-pro) of the old name breaks. Unverified here; resolved by grepping harper-pro
  • minor — unitTests/server/durableSessionResume.test.js:2700 — the test sets notifyScheduled itself to stall the pipeline, so it proves the hold logic under a forced state rather than through a real cross-thread late release
  • nit — resources/originClosedFloor.ts:32 — header grew from 4 to 8 words, so a buffer with the old layout would be misread. Disputed: the buffer is shared within one process only, so a rolling restart cannot mix layouts
  • nit — resources/originClosedFloor.ts:479 — retireOriginFloorSlots scans all SLOTS instead of stopping at WORD_SLOT_LIMIT. Runs only on thread exit, so the cost is negligible
  • nit — resources/Table.ts:7995 — comments and naming, one finding: the local dispatched now holds the certified mark; the localCeiling JSDoc (resources/transactionBroadcast.ts:279-284) duplicates the DESIGN.md paragraph; tests hard-code setOriginCaptureIntervalForTests(5000) instead of using ORIGIN_FLOOR_TICK_MS

Not standing: 1 merged. Grok ran 19m (20m from launch to report); the fold adjudication (cursor-grok@b1e51a10ddab-9ef01b69, claude) took 1m.

Late Grok findings

1 surviving (highest minor) — Grok reviewed 2dd4afa
  • minor — resources/transactionBroadcast.ts:273 — a subscribe({ reportProgress: true, crossThreads: false }) registry never publishes a watermark. Line 441 sets passCeiling to undefined for the same-thread registry, certifyProgress returns early at line 276, and dispatchedThrough() at line 255 reads only certifiedThrough. Once the subsc

Not standing: 1 merged. Grok ran 18m (18m from launch to report); the fold adjudication (cursor-grok@2dd4afa814a5-4e51aebc, claude) took 1m.

Deferred findings

  • f68453d746a8eefa nit reliability — server/DurableSubscriptionsSession.ts — MQTT lacks session-level visibility into database reservation holds — NOT FILED

Late Grok findings

3 surviving (highest major) — Grok reviewed cdee71f, ruled against d42d2e8
  • major — resources/transactionBroadcast.ts:505 — with durable consumers tracking progress, every fresh notify pass now runs captureCeiling/localStartBound. Follow-up passes do too. Each capture raises the shared proposed-clock word that writer threads read and scans the slot high-water range. No benchmark measures this broadcast path: `originC
  • minor — server/DurableSubscriptionsSession.ts:1122 — while a resumed replay awaits its verdict, compactReceived returns early. Every in-order PUBREC for a new key then adds one receivedRuns entry. Past 1,000 distinct keys, addReceived drops receipts (:1054). The position is also frozen while unverified (:1398). So a disconnect before the ver
  • minor — resources/transactionBroadcast.ts:326 — partially resolved. A request that certification already satisfied is now cleared (:316-318). The fallback recapture timer is still armed on every held same-thread pass, even when a follow-up pass is already pending, so timers are armed and cancelled once per hold

Not standing: none. Grok ran 18m (18m from launch to report); the fold adjudication (cursor-grok@cdee71fd9265-394afeb3, claude) took 1m.

Late Grok findings

6 surviving (highest major) — Grok reviewed ded928e
  • major — server/DurableSubscriptionsSession.ts:1122 — A held reservation or unverified replay blocks receipt compaction; exceeding 1,000 acknowledged transaction keys can cause duplicates after reconnect
  • minor — resources/transactionBroadcast.ts:513 — Fresh progress passes and progress registrations add clock sampling, shared-word updates and potentially repeated reservation scans under concurrent writes
  • minor — resources/originClosedFloor.ts:95 — Worker replacement against the previous shared-buffer layout rejects initialization; full-process restart guidance remains missing
  • nit — unitTests/resources/originClosedFloor.bench.js:154 — Added comments narrate benchmark and test setup; aggregate cleanup includes unitTests/server/durableSessionResume.test.js:2937 and unitTests/server/durableSessionResume.test.js:2990
  • nit — server/DurableSubscriptionsSession.ts:1122 — MQTT sessions provide no session-level explanation when database reservations hold progress; retain the existing observability follow-up
  • major — server/DurableSubscriptionsSession.ts:1119 — Compaction holds increase exposure to the QoS 2 receipt cap (raised by the CLI: unmeasured hot path)

Not standing: 1 separate-issue. Grok ran 23m (23m from launch to report); the fold adjudication (cursor-grok@ded928e02a4e-180c9e29, codex) took 5m.

Late Grok findings

6 surviving (highest major) — Grok reviewed 9d6b40e
  • major — server/DurableSubscriptionsSession.ts:1119 — An unrelated held transaction prevents receipt compaction; after 1,000 higher-key QoS 2 receipts, reconnect can redeliver acknowledged messages under new packet IDs
  • major — resources/Table.ts:8734 — Resuming from scalar H above an outstanding reservation L preserves H as the progress floor, allowing a later reconnect to skip L; this override predates the diff
  • minor — resources/transactionBroadcast.ts:512 — Busy progress-enabled databases repeatedly sample reservations and write the shared proposed word, adding contention with concurrent writers
  • minor — resources/transactionBroadcast.ts:69 — A reconnect storm repeats synchronous reservation sampling for every progress registration, including registrations with an existing certified mark
  • minor — resources/originClosedFloor.ts:95 — Loading this layout over an existing process buffer throws; full-process-restart rollout guidance and overlapping-layout coordination remain unverified
  • nit — unitTests/resources/originClosedFloor.bench.js:154 — Added comments narrate benchmark and fixture setup; aggregate cleanup also covers unitTests/resources/subscribeProgress.test.js:48-50 and unitTests/server/durableSessionResume.test.js:2937

Not standing: 1 dropped-factually-wrong, 1 separate-issue. Grok ran 23m (24m from launch to report); the fold adjudication (cursor-grok@9d6b40e10e87-fb603899, codex) took 5m.

Late Grok findings

8 surviving (highest major) — Grok reviewed 432005d
  • major — resources/transactionBroadcast.ts:265 — Disputed: another subscriber’s replay can certify H that this subscriber never received; H predates registration, so whether it is owed remains a contract question
  • major — resources/transactionBroadcast.ts:160 — Disputed: a joining subscriber starts below held L, never receives earlier H, then checkpoints through H after L is acknowledged
  • major — server/DurableSubscriptionsSession.ts:1122 — An unverified scalar replay with over 1,000 distinct PUBREC’d keys exhausts receipt storage; interruption can cause acknowledged deliveries to replay
  • major — resources/ResourceInterface.ts:375 — Batched dequeuing can erase an unconsumed late-event minimum, violating the published contract
  • minor — resources/transactionBroadcast.ts:71 — Reconnect storms and fresh notify passes repeatedly sample shared reservation state, adding synchronous scans and shared-word contention
  • minor — resources/originClosedFloor.ts:95 — Recycling workers after a layout-changing binary swap rejects local reservations until full-process restart, without rollout guidance
  • minor — unitTests/resources/originClosedFloor.test.js:202 — Disputed: a concurrent first slot claim during the synthetic fixture can be hidden when its stale slot limit is restored
  • nit — unitTests/resources/originClosedFloor.bench.js:154 — Redundant setup narration remains here and in unitTests/server/durableSessionResume.test.js:2809-2831 and unitTests/server/durableSessionResume.test.js:3007-3015

Not standing: none. Grok ran 13m (13m from launch to report); the fold adjudication (cursor-grok@432005d83d95-0238f9ce, codex) took 5m.

Late Grok findings

7 surviving (highest major) — Grok reviewed dc704a4
  • major — server/DurableSubscriptionsSession.ts:1122 — During unverified replay or unavailable progress, 1,001 distinct PUBRECs exhaust receipt retention; disconnecting can replay acknowledged deliveries under new packet IDs
  • major — resources/Table.ts:8698 — Dequeuing several events before counting consumption can prune unconsumed key 50 and let a checkpoint skip it; the narrowed sequential-consumer contract leaves this behavior intact
  • minor — resources/transactionBroadcast.ts:70 — Reconnect storms and fresh progress passes repeat shared reservation CAS operations and slot scans against active writers
  • minor — unitTests/resources/originClosedFloor.test.js:202 — If another worker claims a slot during the synthetic fixture, restoring the old limit hides its reservation; the benchmark repeats this restoration
  • nit — resources/transactionBroadcast.ts:299 — Removing dispatchedThrough breaks any remaining deep importer; none was found in the available harper-pro checkout
  • nit — resources/Table.ts:8742 — Closing from the final collection-replay listener leaves goLive dereferencing a detached registration; its rejection is caught and suppressed
  • nit — unitTests/resources/originClosedFloor.bench.js:154 — Added benchmark/test narration merits one aggregate cleanup, preserving comments that explain otherwise invisible ordering invariants

Not standing: none. Grok ran 16m (17m from launch to report); the fold adjudication (cursor-grok@dc704a480cc3-6d7e6600, codex) took 6m.

Late Grok findings

8 surviving (highest minor) — Grok reviewed e66646e
  • minor — resources/transactionBroadcast.ts:333 — when another thread's reservation is released late, a held pass gets at most one follow-up per ceiling, so certified progress waits for the next commit or the 5 s capture tick (the position stays lower, so this is a delay, not data loss)
  • minor — resources/transactionBroadcast.ts:530 — localStartBound now runs on every fresh notify pass, on every reportProgress registration (:71) and on follow-up passes; each call raises WORD_PROPOSED and scans the slots, which adds cost to the commit path
  • minor — resources/transactionBroadcast.ts:348 — the same-thread (crossThreads: false) progress certification and its recapture timer have no production caller; the only crossThreads: false subscription (resources/Table.ts:2442) does not set reportProgress
  • nit — resources/transactionBroadcast.ts:299 — the internal export dispatchedThrough was renamed to certifiedThrough; its only consumer is resources/Table.ts:111
  • nit — resources/originClosedFloor.ts:553 — retireOriginFloorSlots compare-and-swaps all SLOTS instead of stopping at slotLimit; it runs only on thread exit
  • nit — unitTests/server/durableSessionResume.test.js:2771 — the test stalls the pipeline by setting the internal notifyScheduled flag, which ties it to that implementation detail
  • nit — unitTests/server/durableSessionResume.test.js:3024 — a fixed 50 ms sleep before asserting that nothing was compacted; this asserts a non-event, which AGENTS.md permits, but a slow runner could let the assertion pass without exercising the case
  • nit — resources/Table.ts:8847 — comments and naming: the local dispatched now holds the certified mark; the JSDoc on captureCeiling/certifyProgress (resources/transactionBroadcast.ts:303-308, :329-332, :341, :348-349) repeats the DESIGN.md text; 5000 is hard-coded in tests

Not standing: none. Grok ran 25m (25m from launch to report); the fold adjudication (cursor-grok@e66646e1c86b-893cd6c0, claude) took 1m.

Review-Coverage: authored=gemini; ran=cursor-composer,codex,cursor-grok,claude; late=cursor-grok:adjudicated; adjudicated=domain; blocked=claude(fallback)(out-of-budget); declined=muse; rounds=21; full=5 @ 336ff8d

Review-Attention: deep ~40m (critical: Table.ts, transactionBroadcast.ts; hot: DurableSubscriptionsSession.ts; decisions: do-less-alternative, same-thread-progress-support, replay-candidate-sharing, hold-release-latency, export-layout-constants, release-target-vs-evidence) @ 336ff8d

@kriszyp kriszyp added this to the v5.4 milestone Oct 27, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates transaction progress tracking and subscription watermarks to use a certified local watermark (certifiedThrough instead of dispatchedThrough), ensuring that late-committing transactions are not skipped on resume. It also introduces threadStartBound for same-thread subscriptions and updates the relevant design documentation and tests. The review feedback highlights two critical issues in resources/transactionBroadcast.ts where the use of a module-level auditStore variable instead of subscriptions.auditStore could lead to cross-database pollution or runtime TypeErrors in multi-database environments. Code suggestions were provided to resolve these issues by safely querying subscriptions.auditStore.

Comment thread resources/transactionBroadcast.ts Resolved
Comment thread resources/transactionBroadcast.ts Resolved
@kriszyp kriszyp closed this Oct 27, 2026
@kriszyp kriszyp reopened this Oct 27, 2026
@kriszyp
kriszyp force-pushed the fix/durable-position-below-inflight-writes branch 2 times, most recently from 06a965b to b1e51a1 Compare October 28, 2026 00:50
@kriszyp
kriszyp marked this pull request as ready for review October 28, 2026 11:51
@kriszyp
kriszyp requested a review from dawsontoth October 28, 2026 11:51
kriszyp and others added 14 commits October 29, 2026 19:23
…in flight

A RocksDB transaction takes its `local` log key at its first staged write and commits later, so the
dispatched watermark a durable MQTT session saves could pass a key still in flight; the late commit
was then delivered below the saved position and skipped on resume (harper#3112). Each cross-thread
notify pass now captures `localStartBound` before it drains and, after a complete drain, certifies
`min(raw watermark, that ceiling)`; `progress()` and a joining subscription's `registeredThrough`
read the certified mark. The registry's bounds scan stops at a slot high-water mark instead of
visiting all 1024 slots.

Fixes #3112

Dispatch-Task: harper-3112-live-position-cap-design
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…s release outran, and sample a start the registration drain cleared

Review round 1: a `crossThreads: false` progress subscription never certified (ceiling now Infinity
there); an explicit transaction's reservation is released after its commit's notify pass can capture,
so one follow-up pass per hold re-captures; a registration whose drain ended the last consumer got no
start at all (pre-existing). Rename the accessor to `certifiedThrough`, cap the slot limit at SLOTS,
scan holders to the limit too, propagate bench worker failures, refresh stale design notes.

Dispatch-Task: harper-3112-live-position-cap-design
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…ollow-up per dispatching pass, and give a same-thread registry its own ceiling

Review round 2: the registration start is now always sampled before the drain (the certified mark
is only a preference over it); a held, dispatching pass gets one follow-up pass each time, and a
registration drain's pending follow-up is scheduled too; a `crossThreads: false` registry runs no
capture tick (its pass drained the cross-thread log range) and takes its ceiling from this thread's
own lowest outstanding reservation (`threadStartBound`).

Dispatch-Task: harper-3112-live-position-cap-design
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
… log

Review round 3: a same-thread follow-up pass read the cross-thread registry's shared log range,
taking entries its subscribers never get; a same-thread hold that ended in an abort was never
re-captured, since that registry runs no capture tick. Both passes now read an empty log, the abort
case on a timer at the capture cadence. Tests: a held same-thread registry leaves every write to the
cross-thread subscriber on its thread; the abort re-capture.

Dispatch-Task: harper-3112-live-position-cap-design
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…s scheduled

Review round 4: a range replaced between the two would be drained stale.

Dispatch-Task: harper-3112-live-position-cap-design
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: harper-3112-live-position-cap-design
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…mit case from Not covered

Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
… deliveries

progress() certifies dispatch into a subscription's queue, not delivery. A
late event (sent below a key sent before it: a late local commit, or a
resumed replay reading the log in commit order) could wait in the queue
while deliveries of higher keys moved keyBefore above it, so a disconnect
or a re-SUBSCRIBE saved a position past it and the resume skipped it.

The progress send wrapper records late sends by ordinal; nextPosition lowers
the boundary below the lowest one the session has not consumed. The list is
pruned on send to what the consumer has not taken, so a consumer that never
reads it stays bounded by its queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd
Dispatch-Task: harper-3635-f1019-f1024
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…ate list a sliding-window minimum

compactReceived collapses receipts below its bound into receivedThrough, a
second replay-skipping frontier: with a late lower key still queued behind
PUBREC'd higher deliveries it moved receivedThrough past that key, and the
replay after a reconnect classified it ALREADY_RECEIVED. It now takes the
same lowest-unconsumed-late bound as nextPosition.

The late list drops an entry a later, lower one follows (it leaves the window
first), so the lowest is at the head and pruning advances an index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd
Dispatch-Task: harper-3635-f1019-f1024
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
A pass that read nothing and whose ceiling was a clock sample re-armed its
follow-up whenever the sample rose, so a replicated key ahead of this
node's clock kept notify passes following each other at setImmediate rate
until the clock caught up. Every local key such a pass had read is below
its sample, so only time certifies the hold; the next commit's pass or the
capture tick does. A pass held by an outstanding reservation keeps its
re-arm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd
Dispatch-Task: harper-3635-f1019-f1024
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
… registration-sample invariant

The shared buffer lives as long as the process, and its header grew; a
thread loading this layout over one a running thread allocated would read
owner and bound words at shifted offsets. Its size is now checked at
acquisition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd
Dispatch-Task: harper-3635-f1019-f1024
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
kriszyp and others added 7 commits October 29, 2026 19:27
…certifies anything

A resumed replay reads the log in commit order, and until its verdict its
progress() is the replay cursor, which can pass a late key not yet read;
with no generation bound, progress() is undefined and set no bound at all.
Either way PUBRECs during the replay collapsed receipts past that key,
and the next resume took it as ALREADY_RECEIVED. A certified subscription
now compacts only once verified and while progress() certifies a key, as
nextPosition already does for the scalar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd
Dispatch-Task: harper-3635-f1019-f1024
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…e in the design notes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd
Dispatch-Task: harper-3635-f1019-f1024
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
A synchronous drain (a same-thread commit, or a registration drain on the
cross-thread registry) that ran while a follow-up pass was pending left
recheckPending set; the follow-up then certified through the mark and its
cleanup still scheduled one more empty pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd
Dispatch-Task: harper-3635-f1019-f1024
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Bound the replay floor by its registration sample and let the existing notify pass certify observed replay progress. Cover reconnect loss, quiet catch-up and shared registry safety.

Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Require the short replay to complete before the outstanding live batch finishes, and document the sequential-consumer precondition for the late-event minimum.

Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…line budget

origin/main sits at the 1000-line ceiling in resources/DESIGN.md, so fold the certified-watermark and scalar-replay notes into the Shared state bullet without dropping content.

Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0

Co-Authored-By: Gemini <noreply@google.com>
…ick reset

Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0
Co-Authored-By: Gemini <noreply@google.com>
@kriszyp
kriszyp force-pushed the fix/durable-position-below-inflight-writes branch from dc704a4 to 336ff8d Compare October 30, 2026 02:11
@kriszyp
kriszyp merged commit 0378959 into main Oct 30, 2026
103 of 105 checks passed
@kriszyp
kriszyp deleted the fix/durable-position-below-inflight-writes branch October 30, 2026 02:45
kriszyp added a commit that referenced this pull request Oct 30, 2026
…e onto #3635

With #3635 holding the local scalar below open local transactions and #3778 keeping QoS 1 receipts for deliveries acknowledged above a held position:
- In durableSessionResume.test.js, expect the acknowledged first record of a multi-batch late transaction not to be resent on resume, and commit the unacknowledged higher-key and late writes while the client is away when testing late-key replay across a re-subscribe.
- In durableSessionCheckpoint.test.js, test cross-generation checkpoint resume with a foreign scalar above the late commit, assert that an open local transaction holds both the checkpoint and scalar while receipts suppress the acknowledged delivery on scalar resume, and test first-checkpoint adoption against a floor sampled before an unheld acknowledged delivery.

Dispatch-Task: pr-maint-264505d2bc891d3aae41ef11111406de
Co-Authored-By: Gemini <noreply@google.com>
kriszyp added a commit that referenced this pull request Oct 30, 2026
…e their position once the storage engine issues checkpoints (#3778)

* Add regressions for durable MQTT resume from a lagging checkpoint

With a syncWal-capable engine the session resumes from a checkpoint token
that lags its acknowledged position by the floor cadence: an acknowledged
QoS 1 delivery is sent again, and a topic whose first token predates its
SUBSCRIBE replays that earlier history. Both reproduce on the pinned
rocksdb-js 2.12.0 with syncWal stood in and the floor held. Red until the
resume contract is settled.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Resume a durable MQTT session from its own position on this node, and keep QoS 1 acknowledgements

Once the storage engine offers syncWal, checkpoint tokens are issued, and
a durable session resumed from its token and capped its position to it.
The token trails what the session reached by the floor cadence, so a
reconnect re-sent acknowledged QoS 1 versions, a quiet topic's position
stopped advancing, and a new topic replayed history from before its
SUBSCRIBE.

- Within the database generation that bound the scalar position, the
  scalar is authoritative: resume, a replacing SUBSCRIBE and the check
  before CONNACK use it, and it is no longer capped by the token. The
  token resumes a session saved in another generation (another node, a
  restored copy). A fresh topic's start is bound when its first token is
  adopted.
- A PUBACK'd QoS 1 delivery is kept as a receipt, as a PUBREC'd QoS 2
  one is, so a replay (behind an unacknowledged delivery, or from a
  token) skips it.
- A late local commit below a same-node scalar stays uncovered
  (harper#2928), as on every shipped release.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Defer to the checkpoint only for a scalar bound to another generation, and keep QoS 1 receipts in O(1)

An unbound scalar is a fresh topic's own start on this node, so it resumes
from the scalar rather than the lagging token; only a restored scalar bound
to another generation defers to the token. This replaces binding the start
when the first token arrives, which could bind a start progress never
certified.

A PUBACK only adds its receipt (positions prune it), and QoS 1 receipts stop
at half the session's receipt limit, so they never crowd out QoS 2's.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a foreign scalar marked foreign across saves, and let receipts a resume here reads win at the QoS 1 limit

A restored scalar bound to another generation is now saved back with that
generation until this node moves it, so a reconnect that moved nothing still
resumes from the checkpoint. At the QoS 1 receipt share, a topic's receipts
at or below its scalar, which only a checkpoint replay elsewhere reads, are
pruned before a new one is refused.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Prune QoS 1 receipts at the scalar beside a checkpoint, and save one only when the position keeps it

Keeping QoS 1 receipts down to a checkpoint's floor made every save rewrite
up to 500 of them, about three times the save cost under steady QoS 1
traffic, to spare a checkpoint resume on another node some at-least-once
duplicates. A QoS 1 receipt is now kept only above the scalar, where a
resume here reads it, and an acknowledgement forces a save only when such a
receipt survives the position.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a QoS 1 receipt only while the position may stay below it

An in-order PUBACK the next position passes needs no receipt, so it records
none; one held behind an older or same-key unacknowledged delivery, a still
queued message, or progress below its key is kept as before.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Adopt a topic's first checkpoint only once its floor passes every receipt it lacks, and write QoS 1 receipts below the scalar in the final save

A first checkpoint whose floor sits below a receipt the topic already pruned
or never kept, or below its start, would have a replay elsewhere resend
acknowledged deliveries or history from before the SUBSCRIBE; it now waits
for floors above them (prunedThrough). With a checkpoint kept or certified,
every QoS 1 acknowledgement is a receipt again, but the ones at or below the
scalar, which only a checkpoint replay on another node reads, are written
only by the final save, as in-flight QoS 2 deliveries are, so a steady
stream of saves stays small. At the QoS 1 share those give way first, once
per position pass. The receipt gate now follows nextPosition's holds.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Never understate a topic's missing-receipt bound, and write routine saves' receipts in one pass

The bound a first checkpoint must pass is now raised, never assigned: over a
compacted watermark the adoption would erase, over the replaced topic's
bound and any receipts a replacing SUBSCRIBE forgets, and across a scalar
that moves down. A routine save filters QoS 1 receipts as it writes them
instead of copying the maps first.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a foreign scalar's receipts and marker until this node certifies a position of its own

A routine save no longer drops QoS 1 receipts at or below a foreign scalar,
which name nothing in this generation, and an uncertified hold no longer
clears the foreign marker. An acknowledgement forces a save only for a
receipt that save writes.

Dispatch-Task: harper-mqtt-durable-resume-syncwal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a QoS 2 receipt above the scalar when a checkpoint floor passes it

A same-node resume replays from the scalar, which an unacknowledged
delivery can hold below a late delivery PUBRECed since. Pruning at the
checkpoint floor dropped that receipt, so the resume sent the late
delivery again under a new packet id. Prune at the lower of the floor and
the scalar, and keep receivedThrough at first adoption: the gate already
keeps it below the floor, where no checkpoint replay reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: late-review-kriszyp_harper_3778-aa79e71d76ca-bb778ae1

* Reconcile durable session checkpoint and late-write tests after rebase onto #3635

With #3635 holding the local scalar below open local transactions and #3778 keeping QoS 1 receipts for deliveries acknowledged above a held position:
- In durableSessionResume.test.js, expect the acknowledged first record of a multi-batch late transaction not to be resent on resume, and commit the unacknowledged higher-key and late writes while the client is away when testing late-key replay across a re-subscribe.
- In durableSessionCheckpoint.test.js, test cross-generation checkpoint resume with a foreign scalar above the late commit, assert that an open local transaction holds both the checkpoint and scalar while receipts suppress the acknowledged delivery on scalar resume, and test first-checkpoint adoption against a floor sampled before an unheld acknowledged delivery.

Dispatch-Task: pr-maint-264505d2bc891d3aae41ef11111406de
Co-Authored-By: Gemini <noreply@google.com>

* Format durableSessionResume.test.js with Prettier

Co-Authored-By: Gemini <noreply@google.com>
Dispatch-Task: pr-maint-264505d2bc891d3aae41ef11111406de

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Gemini <noreply@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants