Repository navigation
Hold a durable subscription's position below every local write still in flight - #3635
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates transaction progress tracking and subscription watermarks to use a certified local watermark (certifiedThrough instead of dispatchedThrough), ensuring that late-committing transactions are not skipped on resume. It also introduces threadStartBound for same-thread subscriptions and updates the relevant design documentation and tests. The review feedback highlights two critical issues in resources/transactionBroadcast.ts where the use of a module-level auditStore variable instead of subscriptions.auditStore could lead to cross-database pollution or runtime TypeErrors in multi-database environments. Code suggestions were provided to resolve these issues by safely querying subscriptions.auditStore.
06a965b to
b1e51a1
Compare
ded928e to
d6a4e26
Compare
d6a4e26 to
dc704a4
Compare
…in flight A RocksDB transaction takes its `local` log key at its first staged write and commits later, so the dispatched watermark a durable MQTT session saves could pass a key still in flight; the late commit was then delivered below the saved position and skipped on resume (harper#3112). Each cross-thread notify pass now captures `localStartBound` before it drains and, after a complete drain, certifies `min(raw watermark, that ceiling)`; `progress()` and a joining subscription's `registeredThrough` read the certified mark. The registry's bounds scan stops at a slot high-water mark instead of visiting all 1024 slots. Fixes #3112 Dispatch-Task: harper-3112-live-position-cap-design Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…s release outran, and sample a start the registration drain cleared Review round 1: a `crossThreads: false` progress subscription never certified (ceiling now Infinity there); an explicit transaction's reservation is released after its commit's notify pass can capture, so one follow-up pass per hold re-captures; a registration whose drain ended the last consumer got no start at all (pre-existing). Rename the accessor to `certifiedThrough`, cap the slot limit at SLOTS, scan holders to the limit too, propagate bench worker failures, refresh stale design notes. Dispatch-Task: harper-3112-live-position-cap-design Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…ollow-up per dispatching pass, and give a same-thread registry its own ceiling Review round 2: the registration start is now always sampled before the drain (the certified mark is only a preference over it); a held, dispatching pass gets one follow-up pass each time, and a registration drain's pending follow-up is scheduled too; a `crossThreads: false` registry runs no capture tick (its pass drained the cross-thread log range) and takes its ceiling from this thread's own lowest outstanding reservation (`threadStartBound`). Dispatch-Task: harper-3112-live-position-cap-design Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
… log Review round 3: a same-thread follow-up pass read the cross-thread registry's shared log range, taking entries its subscribers never get; a same-thread hold that ended in an abort was never re-captured, since that registry runs no capture tick. Both passes now read an empty log, the abort case on a timer at the capture cadence. Tests: a held same-thread registry leaves every write to the cross-thread subscriber on its thread; the abort re-capture. Dispatch-Task: harper-3112-live-position-cap-design Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…s scheduled Review round 4: a range replaced between the two would be drained stale. Dispatch-Task: harper-3112-live-position-cap-design Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: harper-3112-live-position-cap-design Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…mit case from Not covered Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: GPT-5 Codex <noreply@openai.com>
… deliveries progress() certifies dispatch into a subscription's queue, not delivery. A late event (sent below a key sent before it: a late local commit, or a resumed replay reading the log in commit order) could wait in the queue while deliveries of higher keys moved keyBefore above it, so a disconnect or a re-SUBSCRIBE saved a position past it and the resume skipped it. The progress send wrapper records late sends by ordinal; nextPosition lowers the boundary below the lowest one the session has not consumed. The list is pruned on send to what the consumer has not taken, so a consumer that never reads it stays bounded by its queue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd Dispatch-Task: harper-3635-f1019-f1024 Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…ate list a sliding-window minimum compactReceived collapses receipts below its bound into receivedThrough, a second replay-skipping frontier: with a late lower key still queued behind PUBREC'd higher deliveries it moved receivedThrough past that key, and the replay after a reconnect classified it ALREADY_RECEIVED. It now takes the same lowest-unconsumed-late bound as nextPosition. The late list drops an entry a later, lower one follows (it leaves the window first), so the lowest is at the head and pruning advances an index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd Dispatch-Task: harper-3635-f1019-f1024 Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
A pass that read nothing and whose ceiling was a clock sample re-armed its follow-up whenever the sample rose, so a replicated key ahead of this node's clock kept notify passes following each other at setImmediate rate until the clock caught up. Every local key such a pass had read is below its sample, so only time certifies the hold; the next commit's pass or the capture tick does. A pass held by an outstanding reservation keeps its re-arm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd Dispatch-Task: harper-3635-f1019-f1024 Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
… registration-sample invariant The shared buffer lives as long as the process, and its header grew; a thread loading this layout over one a running thread allocated would read owner and bound words at shifted offsets. Its size is now checked at acquisition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd Dispatch-Task: harper-3635-f1019-f1024 Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…certifies anything A resumed replay reads the log in commit order, and until its verdict its progress() is the replay cursor, which can pass a late key not yet read; with no generation bound, progress() is undefined and set no bound at all. Either way PUBRECs during the replay collapsed receipts past that key, and the next resume took it as ALREADY_RECEIVED. A certified subscription now compacts only once verified and while progress() certifies a key, as nextPosition already does for the scalar. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd Dispatch-Task: harper-3635-f1019-f1024 Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…e in the design notes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd Dispatch-Task: harper-3635-f1019-f1024 Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
A synchronous drain (a same-thread commit, or a registration drain on the cross-thread registry) that ran while a follow-up pass was pending left recheckPending set; the follow-up then certified through the mark and its cleanup still scheduled one more empty pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xoBABkHr2qdjuq31B7Nqd Dispatch-Task: harper-3635-f1019-f1024 Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Bound the replay floor by its registration sample and let the existing notify pass certify observed replay progress. Cover reconnect loss, quiet catch-up and shared registry safety. Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Require the short replay to complete before the outstanding live batch finishes, and document the sequential-consumer precondition for the late-event minimum. Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
…line budget origin/main sits at the 1000-line ceiling in resources/DESIGN.md, so fold the certified-watermark and scalar-replay notes into the Shared state bullet without dropping content. Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: Gemini <noreply@google.com>
…ick reset Dispatch-Task: pr-maint-3ae157eef85ded9fd0fb9fc6cbeb35d0 Co-Authored-By: Gemini <noreply@google.com>
dc704a4 to
336ff8d
Compare
…e onto #3635 With #3635 holding the local scalar below open local transactions and #3778 keeping QoS 1 receipts for deliveries acknowledged above a held position: - In durableSessionResume.test.js, expect the acknowledged first record of a multi-batch late transaction not to be resent on resume, and commit the unacknowledged higher-key and late writes while the client is away when testing late-key replay across a re-subscribe. - In durableSessionCheckpoint.test.js, test cross-generation checkpoint resume with a foreign scalar above the late commit, assert that an open local transaction holds both the checkpoint and scalar while receipts suppress the acknowledged delivery on scalar resume, and test first-checkpoint adoption against a floor sampled before an unheld acknowledged delivery. Dispatch-Task: pr-maint-264505d2bc891d3aae41ef11111406de Co-Authored-By: Gemini <noreply@google.com>
…e their position once the storage engine issues checkpoints (#3778) * Add regressions for durable MQTT resume from a lagging checkpoint With a syncWal-capable engine the session resumes from a checkpoint token that lags its acknowledged position by the floor cadence: an acknowledged QoS 1 delivery is sent again, and a topic whose first token predates its SUBSCRIBE replays that earlier history. Both reproduce on the pinned rocksdb-js 2.12.0 with syncWal stood in and the floor held. Red until the resume contract is settled. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Resume a durable MQTT session from its own position on this node, and keep QoS 1 acknowledgements Once the storage engine offers syncWal, checkpoint tokens are issued, and a durable session resumed from its token and capped its position to it. The token trails what the session reached by the floor cadence, so a reconnect re-sent acknowledged QoS 1 versions, a quiet topic's position stopped advancing, and a new topic replayed history from before its SUBSCRIBE. - Within the database generation that bound the scalar position, the scalar is authoritative: resume, a replacing SUBSCRIBE and the check before CONNACK use it, and it is no longer capped by the token. The token resumes a session saved in another generation (another node, a restored copy). A fresh topic's start is bound when its first token is adopted. - A PUBACK'd QoS 1 delivery is kept as a receipt, as a PUBREC'd QoS 2 one is, so a replay (behind an unacknowledged delivery, or from a token) skips it. - A late local commit below a same-node scalar stays uncovered (harper#2928), as on every shipped release. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Defer to the checkpoint only for a scalar bound to another generation, and keep QoS 1 receipts in O(1) An unbound scalar is a fresh topic's own start on this node, so it resumes from the scalar rather than the lagging token; only a restored scalar bound to another generation defers to the token. This replaces binding the start when the first token arrives, which could bind a start progress never certified. A PUBACK only adds its receipt (positions prune it), and QoS 1 receipts stop at half the session's receipt limit, so they never crowd out QoS 2's. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep a foreign scalar marked foreign across saves, and let receipts a resume here reads win at the QoS 1 limit A restored scalar bound to another generation is now saved back with that generation until this node moves it, so a reconnect that moved nothing still resumes from the checkpoint. At the QoS 1 receipt share, a topic's receipts at or below its scalar, which only a checkpoint replay elsewhere reads, are pruned before a new one is refused. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Prune QoS 1 receipts at the scalar beside a checkpoint, and save one only when the position keeps it Keeping QoS 1 receipts down to a checkpoint's floor made every save rewrite up to 500 of them, about three times the save cost under steady QoS 1 traffic, to spare a checkpoint resume on another node some at-least-once duplicates. A QoS 1 receipt is now kept only above the scalar, where a resume here reads it, and an acknowledgement forces a save only when such a receipt survives the position. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep a QoS 1 receipt only while the position may stay below it An in-order PUBACK the next position passes needs no receipt, so it records none; one held behind an older or same-key unacknowledged delivery, a still queued message, or progress below its key is kept as before. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Adopt a topic's first checkpoint only once its floor passes every receipt it lacks, and write QoS 1 receipts below the scalar in the final save A first checkpoint whose floor sits below a receipt the topic already pruned or never kept, or below its start, would have a replay elsewhere resend acknowledged deliveries or history from before the SUBSCRIBE; it now waits for floors above them (prunedThrough). With a checkpoint kept or certified, every QoS 1 acknowledgement is a receipt again, but the ones at or below the scalar, which only a checkpoint replay on another node reads, are written only by the final save, as in-flight QoS 2 deliveries are, so a steady stream of saves stays small. At the QoS 1 share those give way first, once per position pass. The receipt gate now follows nextPosition's holds. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Never understate a topic's missing-receipt bound, and write routine saves' receipts in one pass The bound a first checkpoint must pass is now raised, never assigned: over a compacted watermark the adoption would erase, over the replaced topic's bound and any receipts a replacing SUBSCRIBE forgets, and across a scalar that moves down. A routine save filters QoS 1 receipts as it writes them instead of copying the maps first. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep a foreign scalar's receipts and marker until this node certifies a position of its own A routine save no longer drops QoS 1 receipts at or below a foreign scalar, which name nothing in this generation, and an uncertified hold no longer clears the foreign marker. An acknowledgement forces a save only for a receipt that save writes. Dispatch-Task: harper-mqtt-durable-resume-syncwal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep a QoS 2 receipt above the scalar when a checkpoint floor passes it A same-node resume replays from the scalar, which an unacknowledged delivery can hold below a late delivery PUBRECed since. Pruning at the checkpoint floor dropped that receipt, so the resume sent the late delivery again under a new packet id. Prune at the lower of the floor and the scalar, and keep receivedThrough at first adoption: the gate already keeps it below the floor, where no checkpoint replay reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: late-review-kriszyp_harper_3778-aa79e71d76ca-bb778ae1 * Reconcile durable session checkpoint and late-write tests after rebase onto #3635 With #3635 holding the local scalar below open local transactions and #3778 keeping QoS 1 receipts for deliveries acknowledged above a held position: - In durableSessionResume.test.js, expect the acknowledged first record of a multi-batch late transaction not to be resent on resume, and commit the unacknowledged higher-key and late writes while the client is away when testing late-key replay across a re-subscribe. - In durableSessionCheckpoint.test.js, test cross-generation checkpoint resume with a foreign scalar above the late commit, assert that an open local transaction holds both the checkpoint and scalar while receipts suppress the acknowledged delivery on scalar resume, and test first-checkpoint adoption against a floor sampled before an unheld acknowledged delivery. Dispatch-Task: pr-maint-264505d2bc891d3aae41ef11111406de Co-Authored-By: Gemini <noreply@google.com> * Format durableSessionResume.test.js with Prettier Co-Authored-By: Gemini <noreply@google.com> Dispatch-Task: pr-maint-264505d2bc891d3aae41ef11111406de --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Gemini <noreply@google.com>
⊙ Problem
A RocksDB transaction takes its
locallog key at its first staged write and commits later. A durable MQTT session saved the dispatched watermark, which could already sit above such a key. When that transaction committed, its message was delivered below the saved position, so a reconnect skipped it (harper#3112). A subscription joining meanwhile started there too.🤔 Fixed at the watermark, not only in the session
The fix changes the database's watermark, which a joining subscription's
registeredThroughreads too. The issue asks about the saved position.💡 Solution
Each cross-thread notify pass captures the lowest outstanding local reservation before it drains and, after a complete drain, certifies
min(raw watermark, that ceiling).progress()andregisteredThroughreturn the certified mark. A session also stays below a late event still queued behind higher keys: the mark certifies dispatch, not delivery. QoS 2 receipt compaction takes the same bound.⚖️ Alternatives
Framing-Verdict: chosen-approach-sound
Planning records: initial
289902ca7eb2, queued-late boundf85afec45842, receipt-compaction rechecka43258e7, scalar replay9030b3163b84.Rejected candidates, measured
advancePositionsis followed by a persist, and receipts are pruned before it🔧 Changes
🤔 A queued late event holds only its own session
The send wrapper records late sends by ordinal; the session stays below the lowest it has not consumed. Holding the shared mark until delivery would tie every subscriber to the slowest one. Each send prunes the list by queue length. This keeps it bounded for a consumer that never asks. So the optional interface method requires a sequential consumer. MQTT’s for-await loop, its only in-tree caller, is one.
🤔 Capture before the drain, never a scan per advance
A reservation is released a microtask after its commit settles, before the next pass reads its entry. So a scan after a drain or at read time certifies past it; a session test reproduces it.
🤔 An aborted hold waits for the capture tick
An aborted hold clears at the next capture tick, up to 5 s later. A committed one clears with the commit's pass or its one follow-up. A longer hold waits too: a delay, never a skip. Kris chose this over a rate-limited read-triggered pass.
🤔 Same-thread registries get the minimum
crossThreads: falseprogress consumers (none in-tree) take this thread's own lowest reservation as the ceiling; their re-capture timer can arm and clear once per commit, accepted.🤔 QoS 2 receipts wait for the verdict and certified progress
Before a resumed replay's verdict, a lower key can still be unread anywhere in the rest of the log, so no receipt collapses. After it, receipts collapse only below certified progress, which an open local write holds. Past 1,000 receipts kept either way, a disconnect resends the overflow: a duplicate, not a lost key. The limit and its overflow rule predate this PR.
🤔 Completed replay still needs reservation certification
A scalar replay's walked key is capped by its registration sample. Its observed key becomes a raw candidate, certified after a ceiling-bounded drain. Clamping forever would stall quiet catch-up. Same-thread replay uses only its live certificate because replay reads all threads.
goLiveruns synchronously before verdict callbacks; pre-verdict positions are not checkpointable. Record resumes retain the requested start, not the replay’s walked key.🤔 Registration defines the live-delivery interval
New subscriptions receive commits made after registration. A low reservation-bounded start can replay earlier keys after an early reconnect; it does not require their live delivery. Replay candidates may pass pre-registration keys while post-registration commits remain covered by the complete-drain gate. Existing durable topics reconnect through their own checked replay and cannot save beyond its start before verification.
🤔 Keep isolated layout fixtures
The slot-limit restore is confined to synchronous fixtures. Unit workers are terminated in
afterEach; the benchmark starts workers only in its later throughput case. Restoring the fixture’s prior limit can hide no concurrent claimer. Hard-coded offsets deliberately pin the shared layout. The benchmark comment explains the contention being measured; retaining it needs no production test API.🤔 #3248 claims header word 4;
WORD_SLOT_LIMITtakes word 5Merged #3248 puts
WORD_INITIAL_CEILINGat word 4, so this PR putsWORD_SLOT_LIMITat word 5, raisesHEADER_WORDSfrom 5 to 8, updates its fixture offsets, and asserts that all header word constants are distinct and belowHEADER_WORDS.🤔 A worker-only restart fails loudly
A worker running this code in a process whose buffer has an earlier header (5 words on
main, 4 words in v5.4.0-alpha.1) throws as it opens the log, telling the operator to restart the process. It never misreads slots silently. Onlymainand the prerelease v5.4.0-alpha.1 have the old layout, so no upgrade from a GA release hits this. The rollout note below asks for a full-process restart.🤔 Preserve a safe prefix after a failed read
A failed read stops further certification and makes
deliveryCertifiedfalse. The previously certified prefix remains safe, so it stays available rather than being revoked.subscribeProgress.test.jschecks that the mark stays frozen after a failed read. A joining subscriber cannot certify delivery through the failed range.🤔 Keep the registration capture on every progress subscription
The registration drain can clear the last certified mark, and a scalar replay needs a sample taken before its cursor opens. Each registration retains that sample even when a shared certificate exists. This adds one capture per SUBSCRIBE, with no new work in the per-write path; reconnect-burst cost remains a measured tradeoff rather than a throughput claim.
🤔 Keep database-level lag diagnostics
The existing reservation holders and floor-lag diagnostics describe the shared database cause. Per-session warnings would repeat that cause for every affected subscriber. Additional alerting remains separate work; this fix preserves the safety bound while a hold remains open.
Accessor and cancellation boundaries
🤔 Keep the certified accessor name
certifiedThroughreplaces the former internaldispatchedThroughaccessor, and all core consumers use it. The root API does not export that accessor. This review covers the authorized core checkout; private-file imports in downstream repositories remain outside that verification.🤔 Closed replay remains uncheckpointable
A close on the final replay delivery settles verification false. If
goLivethen encounters the detached registration, the closed-subscription handler absorbs that rejection andprogress()returns undefined. No checkpoint advances. An activity guard would be cleanup with no change to the cancellation outcome.The simplified pass (transactionBroadcast.ts:530):
resources/DESIGN.md: the certified watermark and the slot limit's ordering, with theprogress()wording and the Not covered list corrected, andserver/DESIGN.mdupdated.lowestLateUnconsumed, optional for other resources.originClosedFloor-thread.js: the writer mode can run unpaced for the bench.Product and architecture tour
Product and architecture tour
Certified watermark
What may a durable subscription save as its position, and when does it move?
localStartBoundacross threads,threadStartBoundon the same thread. It prefers the certified mark when one exists; the registration drain can end the last consumer and clear the mark.progress()on a table subscription reads the certified mark; the rawdispatchedThroughfield stays internal. Completed scalar replay retains a reservation-bounded private floor and submits its observed key to the same certification loop; it cannot override a lower outstanding reservation.Slot limit
Why does the bounded scan still see every claimed slot?
originClosedFloor.tskeeps one past the highest claimed slot in a header word. A claimer raises it right after it owns the slot, beforereservereadsproposed; a sampler raisesproposedbefore it loads the limit, so a claim the scan misses admits only keys above the sampler's value.minBoundandholdersstop at the limit, which never falls. The header grows to a full line, so the floor test and the bench follow the new offsets.Same-thread registries
What does a
crossThreads: falseprogress consumer get?threadStartBoundreads this thread's own lowest reservation after raisingproposed; that registry runs no capture tick, and its follow-up and abort re-capture timer read an empty log, so the shared log range stays the cross-thread registry's to drain. No in-tree code combinescrossThreads: falsewithreportProgress.Queued late events
Why can a certified position still be too high for one session?
progress()certifies that an event reached the subscription's queue. A late event is sent below a key sent earlier (a commit that took its key early, or a resumed replay reading the log in commit order). It can wait in the queue while the session delivers higher keys, sokeyBeforepasses it.lowestLateUnconsumedreturns the lowest such key the session has not consumed;nextPositionandcompactReceivedboth take it. Every other queued event is at or above the last delivered key.✅ Verification
🤔 Two tests force their window
The release-before-dispatch test holds the pass by setting
notifyScheduled: a reservation is released a microtask after its commit, so no real schedule opens that window reliably. A worker thread parked mid-reservation covers a real cross-thread release. The unverified-replay test stalls on a promise, not on its 50 ms sleep, and ends by checking that a reconnect delivers the late key.The other new tests, the QA reproductions, and the cost
Rebased onto
ba3cc2198(includes Stamp a record lock's writes above the version it was admitted on, even when that version is ahead of the clock #3248, movingWORD_SLOT_LIMITto word 5 and asserting distinct header word constants). Current focused validation passes 153/153 on RocksDB.The ahead-of-clock follow-up unit test injects the raw watermark; it does not exercise a real remote-origin replicated key.
Capture measurements above do not resolve multi-writer cache-line contention; the put-loop A/B was too noisy to establish a throughput delta.
QoS 2: receipts stay below a queued late key, and no compaction while a resumed replay is unverified; both deliver the late key after a reconnect.
The late list's minimum and pruning, one follow-up while another origin's key is ahead of this clock, and a buffer of another layout refused.
Earlier: the issue's sequence, the release-before-dispatch gap, a late transaction spanning notify batches, registration during a hold, an abort cleared by the tick, a thread parked mid-reservation, and the same-thread holds in
subscribeProgress.test.js.QA-5012 burst spec (4 workers, 8 writers with every tenth write held, 20 durable QoS 1 clients per cycle): 10 cycles, 0 missing in every cycle, run twice. QA measured 1–6 lost per cycle on
b1e51a10d.QA-5018 hold, release and fast reconnect, n = 60: the held write was redelivered 60/60, and the saved position was below its key 60/60.
integrationTests/mqtt/durable-resume.test.ts: 20/20.Cost, in-process against
b1e51a10d(turbo off, 4 interleaved rounds of 2 M calls, medians): a send through the progress wrapper 35.0 → 45.8 ns, or 32.3 → 46.2 ns with every other key late;nextPositionunchanged at 23.1 ns with everything consumed, 24.5 → 58.4 ns with a send outstanding. A durable QoS 1 put loop costs 200–300 µs per write here, so this is under 0.01%. The per-pass capture (localStartBound) costs 430–500 ns, at most once per commit. A put-loop A/B resolved neither: ±50% spread at load 12–17.reportProgresskeeps its shape.qa-pr-harper-3635-d42d2e8covered the old head; current CI follows this push.🤖 Generated by Claude Fable 5.1, Claude Opus 5.5 (Claude Code), OpenAI Codex, and Google Gemini (maintenance); posted via @kriszyp.
Release target: v5.4 — builds on #3602 and #3629, both v5.4; the certified mark changes subscription position behavior
Related PRs: #3602 overlaps (in the base; the registration start this certifies), #3629 overlaps (in the base; the pending-slot bound the capture uses), #3137 overlaps (certifiedThrough consumer in Table.ts progress()), #3248 overlaps (merged into the base; admission order in originClosedFloor.ts), #3643 overlaps (merged into the base; new durable MQTT topic start on the same reconnect path), #3778 overlaps (session verification/position gates in DurableSubscriptionsSession.ts), #3733 overlaps (durable MQTT section in server/DESIGN.md), #3728 independent (replication), #3775 independent (replication), #3479 independent (durable MQTT re-subscribe QoS; closed), #3714 overlaps (merged into the base; full-text fixtures and readiness), #3638 overlaps (merged into the base; QA-769 readiness), #3469 overlaps (merged into the base; full PUT replay semantics), #2155 independent (expiry sweeps), #3527 independent (schema sharding)
Complexity: complicated
Origin — the dispatch brief this PR was written from
CONSULT FIRST: open a needs-input consultation with your design and measurements before implementing. Investigate harper#3112: a durable MQTT session can save a position (via the subscription's
progress(), resources/Table.ts, or the dispatched-through watermark in resources/transactionBroadcast.ts) above a local transaction that has taken its log key but not committed; when it commits, its delivery is below the saved position and a resume skips it. Deliverable: measurements plus a design that keeps saved positions below every in-flight local write without redelivering acknowledged messages after a fast reconnect and without a material write-throughput cost. Stop at the design for Kris's sign-off.Acceptance
A needs-input consultation containing: measured frequency of outstanding reservations under a steady write load; cost per advance and write-throughput impact for each candidate considered (same put-loop method: ~77k writes/s baseline); behavior of the two durable-resume integration tests under each; the recommended design with its invariant and rejected alternatives with the fact that rules each out. No PR until Kris signs off on the design.
Dispatch: task
harper-3112-live-position-cap-design· queued by kris-session · ran by claude/fable/high · worker kzyp-xps-1Late Grok findings
8 surviving (highest minor) — Grok reviewed b1e51a1
resources/transactionBroadcast.ts:489— every fresh notify pass with a progress consumer now callslocalStartBound, which raisesWORD_PROPOSEDon each commit burst. A write whose handle was created before the latest raise then needs a secondfreshKey(). The cost moved from the 5 s tick onto the commit path. Grading follows the recorresources/transactionBroadcast.ts:301— only one follow-up pass per ceiling (recheckedCeiling). If the reservation is held on another thread whose event loop is busy past that follow-up, certified progress stays held until the next commit or the 5 s capture tick. This is delay, not lossresources/transactionBroadcast.ts:311— the same-thread re-capture timer branch is reached only by asameThreadregistry with a progress consumer. The open record says no production caller exists. Unverified here; resolved by grepping forcrossThreads: falsesubscribers that use progressresources/transactionBroadcast.ts:275— exportdispatchedThroughwas replaced bycertifiedThrough. Any external caller (harper-pro) of the old name breaks. Unverified here; resolved by grepping harper-prounitTests/server/durableSessionResume.test.js:2700— the test setsnotifyScheduleditself to stall the pipeline, so it proves the hold logic under a forced state rather than through a real cross-thread late releaseresources/originClosedFloor.ts:32— header grew from 4 to 8 words, so a buffer with the old layout would be misread. Disputed: the buffer is shared within one process only, so a rolling restart cannot mix layoutsresources/originClosedFloor.ts:479—retireOriginFloorSlotsscans allSLOTSinstead of stopping atWORD_SLOT_LIMIT. Runs only on thread exit, so the cost is negligibleresources/Table.ts:7995— comments and naming, one finding: the localdispatchednow holds the certified mark; thelocalCeilingJSDoc (resources/transactionBroadcast.ts:279-284) duplicates the DESIGN.md paragraph; tests hard-codesetOriginCaptureIntervalForTests(5000)instead of usingORIGIN_FLOOR_TICK_MSNot standing: 1 merged. Grok ran 19m (20m from launch to report); the fold adjudication (
cursor-grok@b1e51a10ddab-9ef01b69, claude) took 1m.Late Grok findings
1 surviving (highest minor) — Grok reviewed 2dd4afa
resources/transactionBroadcast.ts:273— asubscribe({ reportProgress: true, crossThreads: false })registry never publishes a watermark. Line 441 setspassCeilingtoundefinedfor the same-thread registry,certifyProgressreturns early at line 276, anddispatchedThrough()at line 255 reads onlycertifiedThrough. Once the subscNot standing: 1 merged. Grok ran 18m (18m from launch to report); the fold adjudication (
cursor-grok@2dd4afa814a5-4e51aebc, claude) took 1m.Deferred findings
f68453d746a8eefanit reliability —server/DurableSubscriptionsSession.ts— MQTT lacks session-level visibility into database reservation holds — NOT FILEDLate Grok findings
3 surviving (highest major) — Grok reviewed cdee71f, ruled against d42d2e8
resources/transactionBroadcast.ts:505— with durable consumers tracking progress, every fresh notify pass now runscaptureCeiling/localStartBound. Follow-up passes do too. Each capture raises the shared proposed-clock word that writer threads read and scans the slot high-water range. No benchmark measures this broadcast path: `originCserver/DurableSubscriptionsSession.ts:1122— while a resumed replay awaits its verdict,compactReceivedreturns early. Every in-order PUBREC for a new key then adds onereceivedRunsentry. Past 1,000 distinct keys,addReceiveddrops receipts (:1054). The position is also frozen while unverified (:1398). So a disconnect before the verresources/transactionBroadcast.ts:326— partially resolved. A request that certification already satisfied is now cleared (:316-318). The fallback recapture timer is still armed on every held same-thread pass, even when a follow-up pass is already pending, so timers are armed and cancelled once per holdNot standing: none. Grok ran 18m (18m from launch to report); the fold adjudication (
cursor-grok@cdee71fd9265-394afeb3, claude) took 1m.Late Grok findings
6 surviving (highest major) — Grok reviewed ded928e
server/DurableSubscriptionsSession.ts:1122— A held reservation or unverified replay blocks receipt compaction; exceeding 1,000 acknowledged transaction keys can cause duplicates after reconnectresources/transactionBroadcast.ts:513— Fresh progress passes and progress registrations add clock sampling, shared-word updates and potentially repeated reservation scans under concurrent writesresources/originClosedFloor.ts:95— Worker replacement against the previous shared-buffer layout rejects initialization; full-process restart guidance remains missingunitTests/resources/originClosedFloor.bench.js:154— Added comments narrate benchmark and test setup; aggregate cleanup includesunitTests/server/durableSessionResume.test.js:2937andunitTests/server/durableSessionResume.test.js:2990server/DurableSubscriptionsSession.ts:1122— MQTT sessions provide no session-level explanation when database reservations hold progress; retain the existing observability follow-upserver/DurableSubscriptionsSession.ts:1119— Compaction holds increase exposure to the QoS 2 receipt cap (raised by the CLI: unmeasured hot path)Not standing: 1 separate-issue. Grok ran 23m (23m from launch to report); the fold adjudication (
cursor-grok@ded928e02a4e-180c9e29, codex) took 5m.Late Grok findings
6 surviving (highest major) — Grok reviewed 9d6b40e
server/DurableSubscriptionsSession.ts:1119— An unrelated held transaction prevents receipt compaction; after 1,000 higher-key QoS 2 receipts, reconnect can redeliver acknowledged messages under new packet IDsresources/Table.ts:8734— Resuming from scalar H above an outstanding reservation L preserves H as the progress floor, allowing a later reconnect to skip L; this override predates the diffresources/transactionBroadcast.ts:512— Busy progress-enabled databases repeatedly sample reservations and write the shared proposed word, adding contention with concurrent writersresources/transactionBroadcast.ts:69— A reconnect storm repeats synchronous reservation sampling for every progress registration, including registrations with an existing certified markresources/originClosedFloor.ts:95— Loading this layout over an existing process buffer throws; full-process-restart rollout guidance and overlapping-layout coordination remain unverifiedunitTests/resources/originClosedFloor.bench.js:154— Added comments narrate benchmark and fixture setup; aggregate cleanup also coversunitTests/resources/subscribeProgress.test.js:48-50andunitTests/server/durableSessionResume.test.js:2937Not standing: 1 dropped-factually-wrong, 1 separate-issue. Grok ran 23m (24m from launch to report); the fold adjudication (
cursor-grok@9d6b40e10e87-fb603899, codex) took 5m.Late Grok findings
8 surviving (highest major) — Grok reviewed 432005d
resources/transactionBroadcast.ts:265— Disputed: another subscriber’s replay can certify H that this subscriber never received; H predates registration, so whether it is owed remains a contract questionresources/transactionBroadcast.ts:160— Disputed: a joining subscriber starts below held L, never receives earlier H, then checkpoints through H after L is acknowledgedserver/DurableSubscriptionsSession.ts:1122— An unverified scalar replay with over 1,000 distinct PUBREC’d keys exhausts receipt storage; interruption can cause acknowledged deliveries to replayresources/ResourceInterface.ts:375— Batched dequeuing can erase an unconsumed late-event minimum, violating the published contractresources/transactionBroadcast.ts:71— Reconnect storms and fresh notify passes repeatedly sample shared reservation state, adding synchronous scans and shared-word contentionresources/originClosedFloor.ts:95— Recycling workers after a layout-changing binary swap rejects local reservations until full-process restart, without rollout guidanceunitTests/resources/originClosedFloor.test.js:202— Disputed: a concurrent first slot claim during the synthetic fixture can be hidden when its stale slot limit is restoredunitTests/resources/originClosedFloor.bench.js:154— Redundant setup narration remains here and inunitTests/server/durableSessionResume.test.js:2809-2831andunitTests/server/durableSessionResume.test.js:3007-3015Not standing: none. Grok ran 13m (13m from launch to report); the fold adjudication (
cursor-grok@432005d83d95-0238f9ce, codex) took 5m.Late Grok findings
7 surviving (highest major) — Grok reviewed dc704a4
server/DurableSubscriptionsSession.ts:1122— During unverified replay or unavailable progress, 1,001 distinct PUBRECs exhaust receipt retention; disconnecting can replay acknowledged deliveries under new packet IDsresources/Table.ts:8698— Dequeuing several events before counting consumption can prune unconsumed key 50 and let a checkpoint skip it; the narrowed sequential-consumer contract leaves this behavior intactresources/transactionBroadcast.ts:70— Reconnect storms and fresh progress passes repeat shared reservation CAS operations and slot scans against active writersunitTests/resources/originClosedFloor.test.js:202— If another worker claims a slot during the synthetic fixture, restoring the old limit hides its reservation; the benchmark repeats this restorationresources/transactionBroadcast.ts:299— Removing dispatchedThrough breaks any remaining deep importer; none was found in the available harper-pro checkoutresources/Table.ts:8742— Closing from the final collection-replay listener leaves goLive dereferencing a detached registration; its rejection is caught and suppressedunitTests/resources/originClosedFloor.bench.js:154— Added benchmark/test narration merits one aggregate cleanup, preserving comments that explain otherwise invisible ordering invariantsNot standing: none. Grok ran 16m (17m from launch to report); the fold adjudication (
cursor-grok@dc704a480cc3-6d7e6600, codex) took 6m.Late Grok findings
8 surviving (highest minor) — Grok reviewed e66646e
resources/transactionBroadcast.ts:333— when another thread's reservation is released late, a held pass gets at most one follow-up per ceiling, so certified progress waits for the next commit or the 5 s capture tick (the position stays lower, so this is a delay, not data loss)resources/transactionBroadcast.ts:530—localStartBoundnow runs on every fresh notify pass, on everyreportProgressregistration (:71) and on follow-up passes; each call raisesWORD_PROPOSEDand scans the slots, which adds cost to the commit pathresources/transactionBroadcast.ts:348— the same-thread (crossThreads: false) progress certification and its recapture timer have no production caller; the onlycrossThreads: falsesubscription (resources/Table.ts:2442) does not setreportProgressresources/transactionBroadcast.ts:299— the internal exportdispatchedThroughwas renamed tocertifiedThrough; its only consumer isresources/Table.ts:111resources/originClosedFloor.ts:553—retireOriginFloorSlotscompare-and-swaps allSLOTSinstead of stopping atslotLimit; it runs only on thread exitunitTests/server/durableSessionResume.test.js:2771— the test stalls the pipeline by setting the internalnotifyScheduledflag, which ties it to that implementation detailunitTests/server/durableSessionResume.test.js:3024— a fixed 50 ms sleep before asserting that nothing was compacted; this asserts a non-event, which AGENTS.md permits, but a slow runner could let the assertion pass without exercising the caseresources/Table.ts:8847— comments and naming: the localdispatchednow holds the certified mark; the JSDoc oncaptureCeiling/certifyProgress(resources/transactionBroadcast.ts:303-308,:329-332,:341,:348-349) repeats the DESIGN.md text; 5000 is hard-coded in testsNot standing: none. Grok ran 25m (25m from launch to report); the fold adjudication (
cursor-grok@e66646e1c86b-893cd6c0, claude) took 1m.Review-Coverage: authored=gemini; ran=cursor-composer,codex,cursor-grok,claude; late=cursor-grok:adjudicated; adjudicated=domain; blocked=claude(fallback)(out-of-budget); declined=muse; rounds=21; full=5 @ 336ff8d
Review-Attention: deep ~40m (critical: Table.ts, transactionBroadcast.ts; hot: DurableSubscriptionsSession.ts; decisions: do-less-alternative, same-thread-progress-support, replay-candidate-sharing, hold-release-latency, export-layout-constants, release-target-vs-evidence) @ 336ff8d