Repository navigation
Keep a replicated put that shares its transaction with a same-key publish or relocate from being skipped as a duplicate - #3499
Conversation
…ped as a duplicate The head-tie skip decided identity from the commit base, which reads through the transaction. A publish or relocate saves when it is added, so a later same-key write in the source transaction had already restamped that base when the put committed, and the put was dropped. Identity is now read from the committed record, outside the transaction, which also makes the earlier-write guard unnecessary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ate only for a key written twice Only another write to the same key in the transaction can restamp the base without staging it, so the common single-write duplicate keeps the in-memory check and pays no read. Adds tests for the logged put, the read on a twice-written key, and a competing delivery that commits during a staged publish. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yet saved A redelivered put and patch to one key now skips both without a read; only a same-key write that already saved unskipped, such as an eager publish or relocate, sends the decision to committed state. An uncached committed read measured 2.6 us against 39 us for a two-write redelivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ined key id A re-delivered relocate restamped a tie, so with a non-resident residency list it committed a stub over the full record its own transaction's put had written, while that put was skipped. The head-tie check now reuses the key id linkWrite already computed instead of re-encoding it per duplicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ish's own ref can't mask a put When the log key differs from the version, an eager same-key publish or relocate adds its ref to the commit base, and the put matched it and was skipped on first delivery. The ref check now uses the same record identityBase gives the head-tie skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… unchained key The head-tie and ref checks share one identityBase result, so a put with an eager same-key sibling reads committed state once. A write with no per-key chain has no sibling that could have restamped its base, so it keeps the base, which leaves LMDB unchanged. Source-applied [message, put] measured about 3 us per transaction above main (49.8-77.3 vs 47.7-75.4 us, 1 KB records), where main skips the put. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release cherry-pick
|
There was a problem hiding this comment.
Code Review
This pull request updates the transaction replication and duplicate-detection logic to use the committed identity before audit I/O. Specifically, it introduces identityBase to determine the commit base, checking if another write to the same key in the transaction has saved without being skipped, and falls back to reading from the committed record outside the transaction if needed. It also adds helper functions committedEntry and isHeadIdentity to refine head-identity checks and prevent re-delivered relocates from restamping the committed head. Corresponding unit tests have been added to cover these scenarios, including competing deliveries and re-delivered relocates. There are no review comments, so I have no feedback to provide.
…lish or relocate from being skipped as a duplicate (#3499) Port of 5d94a9a to v5.3. The head-tie identity now comes from the committed record whenever another write to the key already saved unskipped, and a source-applied relocate that ties with the committed head is skipped. v5.3 has a single txnLogKey (no origin-closed floor), so main's receivedLogKey/appendedLogKey checks map to `txnLogKey === txnTime`; the relocate keeps v5.3's plain return on an older version (#3183's skipped marking is not on v5.3), and the ref check keeps v5.3's form without main's priorStaged guard. (cherry picked from commit 5d94a9a) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SbvHaBBdQ7Cp7yVDuSD5GS Dispatch-Task: cherry-resolve-kriszyp_harper_3476-123ff057
#3499 removed the helper with its last caller; the audit:false out-of-order arms still use it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Dispatch-Task: pr-maint-563ceb51d0967c6c19dbcab60690aaa4
…hout reading the transaction log (conflicts → v5.3) (#3478) * Skip relayed duplicates of a record's committed head without reading the transaction log A replicated write that matches the committed record's origin, version and log key (no explicit refs) is already applied. That was only recognized on a retry; on a first delivery the keyed dedup looked for the head's log entry, and when the receiver had none (a base-copied record) the exact-start lookup scanned to end-of-log and the write was applied and logged again, so every peer was re-sent it (harper#3467). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep a put that follows an unstaged same-key write out of the head-tie skip A publish or relocate earlier in the same source transaction restamps the record at the transaction's version without staging it, so the put after it tied with that restamp and was dropped. The skip now requires every earlier write to the key in the transaction to have been skipped, which still lets a re-delivered multi-write transaction skip all of its writes from the record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cover a put after a same-key relocate, and keep removalBefore's documentation on it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the relocate case in the head-tie design note Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep only the constraint in the head-tie comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Resolve the v5.3 cherry-pick of #3476 v5.3 has a single log key, so the head-tie skip keeps txnLogKey === txnTime and adds priorWritesSkipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep a replicated put that shares its transaction with a same-key publish or relocate from being skipped as a duplicate (#3499) Port of 5d94a9a to v5.3. The head-tie identity now comes from the committed record whenever another write to the key already saved unskipped, and a source-applied relocate that ties with the committed head is skipped. v5.3 has a single txnLogKey (no origin-closed floor), so main's receivedLogKey/appendedLogKey checks map to `txnLogKey === txnTime`; the relocate keeps v5.3's plain return on an older version (#3183's skipped marking is not on v5.3), and the ref check keeps v5.3's form without main's priorStaged guard. (cherry picked from commit 5d94a9a) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SbvHaBBdQ7Cp7yVDuSD5GS Dispatch-Task: cherry-resolve-kriszyp_harper_3476-123ff057 --------- Co-authored-by: Kris Zyp <kriszyp@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#3499 removed the helper with its last caller; the audit:false out-of-order arms still use it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Dispatch-Task: pr-maint-563ceb51d0967c6c19dbcab60690aaa4
#3499 removed the helper with its last caller; the audit:false out-of-order arms still use it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Dispatch-Task: pr-maint-563ceb51d0967c6c19dbcab60690aaa4
#3499 removed the helper with its last caller; the audit:false out-of-order arms still use it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Dispatch-Task: pr-maint-563ceb51d0967c6c19dbcab60690aaa4
⊙ Problem
Since #3476, a replicated transaction that writes a record and also publishes to or relocates it can lose the write: the receiver keeps the old value and logs nothing. A publish or relocate saves as soon as it is added, updating the record before the put's commit checks for a duplicate, so the put matches it and is skipped.
🤔 Why does this reach v5.3 through #3478 rather than its own cherry-pick?
#3478 carries #3476 to v5.3 and would ship the same loss, so this fix is folded into it once merged; v5.3 never gets #3476 alone. This PR is therefore unmilestoned, which keeps the workflow from picking it separately.
💡 Solution
⚖️ Alternatives
🤔 Is the committed read cheap enough?
I think so. It runs only when another write to the key has saved unskipped, which in practice means a publish or relocate in the same transaction. A source-applied
[message, put]measured about 3 µs abovemain(49.8–77.3 against 47.7–75.4 µs, 1 KB records), andmainthere skips the put. A single write, or a re-delivered put and patch, reads nothing.🔧 Changes
resources/Table.ts:identityBasepicks the record both duplicate checks read;_writeRelocateskips at the committed head;priorWritesSkippedis removed.resources/DatabaseTransaction.ts:linkWritekeeps the key id on the write, so the check does not re-encode the key.resources/DESIGN.md: the head-tie note says where identity comes from.unitTests/resources/relayedHeadTieDuplicate.test.js: ordering, re-delivery, read-count, relocate and competing-delivery cases.✅ Verification
main: a put before a same-key message or relocate keeps the old value; a put with a message whose log key differs from its version keeps the old value; a re-delivered put and message never created the record. All pass with the fix.test:unit:resources: 4637 passing. The [put, delete] case inreplicatedDeleteRedelivery.test.jsfails alone onmaintoo; an HNSW allow-set test failed once in the full run and passes alone with and without this change.🤖 Generated by Claude Code (Claude Opus 5.5); posted via @kriszyp.
Release target: main — reaches v5.3 folded into #3478 (the #3476 backport), not as a separate cherry-pick
Related PRs: #3476 overlaps (fixes its regression), #3478 overlaps (v5.3 backport of #3476 that must carry this), #3453 overlaps (same design-note bullet and dedup path), #3375 overlaps (changes relocate, which now skips at the committed head), #3401 overlaps (same commit handler), 7 others independent
Complexity: complicated
Deferred findings
f8d14ea99889b0ddminor data-integrity —resources/Table.ts— same-version restamp committed in a separate receiver txn masks a later put — NOT FILED0f8b86123de7ea9dminor data-integrity —resources/Table.ts— retry re-saves an eager relocate after the deferred put, so it stubs over the staged put — NOT FILED87aefe2d0b974466nit maintainability —resources/DatabaseTransaction.ts— priorWrite/chainsStagedState field comments restate linkWrite — NOT FILEDLate Grok findings
3 surviving (highest major) — Grok reviewed c916aaa, ruled against 4a47e38
resources/Table.ts:9584— On a replicated redelivery, a key written more than once in one transaction pays an uncached synchronous primary read and record decode for every write that ties the stored version. The read also runs before theadditionalAuditRefsrejection atresources/Table.ts:9594, so writes that check thenresources/Table.ts:9584— If a separate receiver transaction commits a same-version restamp, a later write to that key reads it as the head and its put is skipped. This was deferred earlier on this branch and is unchanged at this HEAD.resources/DESIGN.md:150— The note still says identity is "read from the committed record, outside the transaction". That is no longer always true: a key's only write in the transaction is now decided from its commit base, and only keys written more than once get the separate committed read (resources/Table.ts:9584-9588).Not standing: 1 merged. Grok ran 26m (26m from launch to report); the fold adjudication (
cursor-grok@c916aaa5cee3-05ba8ae3, claude) took 1m.Late Grok findings
1 surviving (highest major) — Grok reviewed 2be36df
resources/Table.ts:9639— When a source transaction with a put and a same-key relocate is delivered a second time, and the relocate's residency list excludes this host, the receiver ends up with the relocate stub while peers that got the transaction once keep the full record. On the second delivery the relocate is not recognNot standing: none. Grok ran 14m (14m from launch to report); the fold adjudication (
cursor-grok@2be36dfd8308-3ea09e43, claude) took 2m.Review-Coverage: authored=claude; ran=cursor-composer,codex,gemini,cursor-grok; adjudicated=domain; declined=muse; rounds=8; full=4 @ 5f8bfe9
Review-Attention: study ~20m (critical: DatabaseTransaction.ts, Table.ts; decisions: keyid-cached-on-every-write, committed-read-vs-stashed-base, relocate-implicit-head-only, release-target-vs-evidence) @ 5f8bfe9