Skip to content

Keep a replicated put that shares its transaction with a same-key publish or relocate from being skipped as a duplicate - #3499

Merged
kriszyp merged 8 commits into
mainfrom
kris/3476-committed-head-tie
Oct 23, 2026
Merged

kriszyp merged 8 commits into
mainfrom
kris/3476-committed-head-tie

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 23, 2026 •

Copy link
Copy Markdown
Member
Is this a pigeon meme redraw where a puzzled robot clerk points at a fresh envelope after a stamp already checked the ledger

⊙ Problem

Since #3476, a replicated transaction that writes a record and also publishes to or relocates it can lose the write: the receiver keeps the old value and logs nothing. A publish or relocate saves as soon as it is added, updating the record before the put's commit checks for a duplicate, so the put matches it and is skipped.

🤔 Why does this reach v5.3 through #3478 rather than its own cherry-pick?

#3478 carries #3476 to v5.3 and would ship the same loss, so this fix is folded into it once merged; v5.3 never gets #3476 alone. This PR is therefore unmilestoned, which keeps the workflow from picking it separately.

💡 Solution

⚖️ Alternatives

🤔 Is the committed read cheap enough?

I think so. It runs only when another write to the key has saved unskipped, which in practice means a publish or relocate in the same transaction. A source-applied [message, put] measured about 3 µs above main (49.8–77.3 against 47.7–75.4 µs, 1 KB records), and main there skips the put. A single write, or a re-delivered put and patch, reads nothing.

🔧 Changes

✅ Verification

🤖 Generated by Claude Code (Claude Opus 5.5); posted via @kriszyp.

Release target: main — reaches v5.3 folded into #3478 (the #3476 backport), not as a separate cherry-pick
Related PRs: #3476 overlaps (fixes its regression), #3478 overlaps (v5.3 backport of #3476 that must carry this), #3453 overlaps (same design-note bullet and dedup path), #3375 overlaps (changes relocate, which now skips at the committed head), #3401 overlaps (same commit handler), 7 others independent
Complexity: complicated

Deferred findings

  • f8d14ea99889b0dd minor data-integrity — resources/Table.ts — same-version restamp committed in a separate receiver txn masks a later put — NOT FILED
  • 0f8b86123de7ea9d minor data-integrity — resources/Table.ts — retry re-saves an eager relocate after the deferred put, so it stubs over the staged put — NOT FILED
  • 87aefe2d0b974466 nit maintainability — resources/DatabaseTransaction.ts — priorWrite/chainsStagedState field comments restate linkWrite — NOT FILED

Late Grok findings

3 surviving (highest major) — Grok reviewed c916aaa, ruled against 4a47e38
  • major — resources/Table.ts:9584 — On a replicated redelivery, a key written more than once in one transaction pays an uncached synchronous primary read and record decode for every write that ties the stored version. The read also runs before the additionalAuditRefs rejection at resources/Table.ts:9594, so writes that check then
  • minor — resources/Table.ts:9584 — If a separate receiver transaction commits a same-version restamp, a later write to that key reads it as the head and its put is skipped. This was deferred earlier on this branch and is unchanged at this HEAD.
  • nit — resources/DESIGN.md:150 — The note still says identity is "read from the committed record, outside the transaction". That is no longer always true: a key's only write in the transaction is now decided from its commit base, and only keys written more than once get the separate committed read (resources/Table.ts:9584-9588).

Not standing: 1 merged. Grok ran 26m (26m from launch to report); the fold adjudication (cursor-grok@c916aaa5cee3-05ba8ae3, claude) took 1m.

Late Grok findings

1 surviving (highest major) — Grok reviewed 2be36df
  • major — resources/Table.ts:9639 — When a source transaction with a put and a same-key relocate is delivered a second time, and the relocate's residency list excludes this host, the receiver ends up with the relocate stub while peers that got the transaction once keep the full record. On the second delivery the relocate is not recogn

Not standing: none. Grok ran 14m (14m from launch to report); the fold adjudication (cursor-grok@2be36dfd8308-3ea09e43, claude) took 2m.

Review-Coverage: authored=claude; ran=cursor-composer,codex,gemini,cursor-grok; adjudicated=domain; declined=muse; rounds=8; full=4 @ 5f8bfe9

Review-Attention: study ~20m (critical: DatabaseTransaction.ts, Table.ts; decisions: keyid-cached-on-every-write, committed-read-vs-stashed-base, relocate-implicit-head-only, release-target-vs-evidence) @ 5f8bfe9

kriszyp and others added 8 commits October 23, 2026 10:20
…ped as a duplicate

The head-tie skip decided identity from the commit base, which reads through the transaction. A
publish or relocate saves when it is added, so a later same-key write in the source transaction
had already restamped that base when the put committed, and the put was dropped. Identity is now
read from the committed record, outside the transaction, which also makes the earlier-write guard
unnecessary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ate only for a key written twice

Only another write to the same key in the transaction can restamp the base without staging it, so the
common single-write duplicate keeps the in-memory check and pays no read. Adds tests for the logged
put, the read on a twice-written key, and a competing delivery that commits during a staged publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yet saved

A redelivered put and patch to one key now skips both without a read; only a same-key write that
already saved unskipped, such as an eager publish or relocate, sends the decision to committed state.
An uncached committed read measured 2.6 us against 39 us for a two-write redelivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ined key id

A re-delivered relocate restamped a tie, so with a non-resident residency list it committed a stub
over the full record its own transaction's put had written, while that put was skipped. The head-tie
check now reuses the key id linkWrite already computed instead of re-encoding it per duplicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ish's own ref can't mask a put

When the log key differs from the version, an eager same-key publish or relocate adds its ref to the
commit base, and the put matched it and was skipped on first delivery. The ref check now uses the same
record identityBase gives the head-tie skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… unchained key

The head-tie and ref checks share one identityBase result, so a put with an eager same-key sibling
reads committed state once. A write with no per-key chain has no sibling that could have restamped
its base, so it keeps the base, which leaves LMDB unchanged. Source-applied [message, put] measured
about 3 us per transaction above main (49.8-77.3 vs 47.7-75.4 us, 1 KB records), where main skips the put.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kriszyp kriszyp added this to the v5.3 milestone Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Release cherry-pick v5.3: cancelled

Cherry-pick branch cherry-pick/v5.3/pr-3499 was deleted — this PR no longer targets v5.3 (milestone is now <none>).

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the transaction replication and duplicate-detection logic to use the committed identity before audit I/O. Specifically, it introduces identityBase to determine the commit base, checking if another write to the same key in the transaction has saved without being skipped, and falls back to reading from the committed record outside the transaction if needed. It also adds helper functions committedEntry and isHeadIdentity to refine head-identity checks and prevent re-delivered relocates from restamping the committed head. Corresponding unit tests have been added to cover these scenarios, including competing deliveries and re-delivered relocates. There are no review comments, so I have no feedback to provide.

@kriszyp kriszyp removed this from the v5.3 milestone Oct 23, 2026
@kriszyp
kriszyp marked this pull request as ready for review October 23, 2026 20:32
@kriszyp
kriszyp merged commit 5d94a9a into main Oct 23, 2026
92 of 95 checks passed
@kriszyp
kriszyp deleted the kris/3476-committed-head-tie branch October 23, 2026 20:32
kriszyp added a commit that referenced this pull request Oct 23, 2026
…lish or relocate from being skipped as a duplicate (#3499)

Port of 5d94a9a to v5.3. The head-tie identity now comes from the committed record whenever
another write to the key already saved unskipped, and a source-applied relocate that ties with the
committed head is skipped. v5.3 has a single txnLogKey (no origin-closed floor), so main's
receivedLogKey/appendedLogKey checks map to `txnLogKey === txnTime`; the relocate keeps v5.3's
plain return on an older version (#3183's skipped marking is not on v5.3), and the ref check keeps
v5.3's form without main's priorStaged guard.

(cherry picked from commit 5d94a9a)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SbvHaBBdQ7Cp7yVDuSD5GS
Dispatch-Task: cherry-resolve-kriszyp_harper_3476-123ff057
kriszyp added a commit that referenced this pull request Oct 23, 2026
#3499 removed the helper with its last caller; the audit:false out-of-order
arms still use it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: pr-maint-563ceb51d0967c6c19dbcab60690aaa4
kriszyp added a commit that referenced this pull request Oct 23, 2026
…hout reading the transaction log (conflicts → v5.3) (#3478)

* Skip relayed duplicates of a record's committed head without reading the transaction log

A replicated write that matches the committed record's origin, version and log key (no explicit refs) is
already applied. That was only recognized on a retry; on a first delivery the keyed dedup looked for the
head's log entry, and when the receiver had none (a base-copied record) the exact-start lookup scanned to
end-of-log and the write was applied and logged again, so every peer was re-sent it (harper#3467).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a put that follows an unstaged same-key write out of the head-tie skip

A publish or relocate earlier in the same source transaction restamps the record at the transaction's
version without staging it, so the put after it tied with that restamp and was dropped. The skip now
requires every earlier write to the key in the transaction to have been skipped, which still lets a
re-delivered multi-write transaction skip all of its writes from the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cover a put after a same-key relocate, and keep removalBefore's documentation on it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Name the relocate case in the head-tie design note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep only the constraint in the head-tie comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Resolve the v5.3 cherry-pick of #3476

v5.3 has a single log key, so the head-tie skip keeps txnLogKey === txnTime and adds priorWritesSkipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a replicated put that shares its transaction with a same-key publish or relocate from being skipped as a duplicate (#3499)

Port of 5d94a9a to v5.3. The head-tie identity now comes from the committed record whenever
another write to the key already saved unskipped, and a source-applied relocate that ties with the
committed head is skipped. v5.3 has a single txnLogKey (no origin-closed floor), so main's
receivedLogKey/appendedLogKey checks map to `txnLogKey === txnTime`; the relocate keeps v5.3's
plain return on an older version (#3183's skipped marking is not on v5.3), and the ref check keeps
v5.3's form without main's priorStaged guard.

(cherry picked from commit 5d94a9a)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SbvHaBBdQ7Cp7yVDuSD5GS
Dispatch-Task: cherry-resolve-kriszyp_harper_3476-123ff057

---------

Co-authored-by: Kris Zyp <kriszyp@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Oct 24, 2026
#3499 removed the helper with its last caller; the audit:false out-of-order
arms still use it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: pr-maint-563ceb51d0967c6c19dbcab60690aaa4
kriszyp added a commit that referenced this pull request Oct 27, 2026
#3499 removed the helper with its last caller; the audit:false out-of-order
arms still use it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: pr-maint-563ceb51d0967c6c19dbcab60690aaa4
kriszyp added a commit that referenced this pull request Oct 27, 2026
#3499 removed the helper with its last caller; the audit:false out-of-order
arms still use it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: pr-maint-563ceb51d0967c6c19dbcab60690aaa4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant