Skip to content

Skip relayed duplicates of a record's committed head without reading the transaction log - #3476

Merged
kriszyp merged 5 commits into
mainfrom
kris/3467-head-tie-dedup
Oct 23, 2026
Merged

kriszyp merged 5 commits into
mainfrom
kris/3467-head-tie-dedup

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 23, 2026 •

Copy link
Copy Markdown
Member
Approve-reject meme where a robot clerk checks one index card and recycles a duplicate envelope instead of scanning an endless scroll

⊙ Problem

Replication receivers spend seconds per write on duplicates they already hold (#3467). The check looks up the write in the origin's transaction log. A base-copied record has no log entry, so on RocksDB the lookup reads to the end of the log. The duplicate is then logged again and re-sent to every peer, so an old stretch replays indefinitely.

🤔 Does this belong beside #3453?

Yes. #3453 hardens the deeper retry-path dedup. This PR covers first deliveries, which #3453 does not reach. Both edit the same design-note bullet, so whichever merges second keeps both guarantees.

💡 Solution

  • A replicated write that matches the stored record's origin, version and log key, with no explicit refs, is skipped on first delivery as well as retry. It reads nothing from the log and appends nothing.
  • The skip is not taken after an applied earlier write to the same key in the transaction. A publish or relocate updates the record's version without staging it.

⚖️ Alternatives

  • Bound the exact-start lookup in rocksdb-js — still worth doing. A suffix-minimum index (rocksdb-js#676) would not bound this case: the re-appended segments hold keys from across the whole stretch.
  • Purge an inactive origin's last log file — the stretch is re-written into new files, so retention never reaches it while duplicates keep being re-logged.

🤔 Should a base-copied record count as logged under its version?

I chose yes. A copy drops the record's head ref, so a distinct write from the same origin, whose version equals the copied write's log key, would be skipped. That needs a source-reported version equal to a commit timestamp. Carrying the head's log key in copy frames would close it.

🔧 Changes

✅ Verification

🤖 Generated by Claude Code (Claude Opus 5.5); posted via @kriszyp.

Release target: v5.3 — fixes code from #3366, a backport to v5.3; the affected cluster runs 5.3
Related PRs: #3453 overlaps (same design-note bullet), #3375 overlaps (changes relocate, which priorWritesSkipped relies on), 6 others independent
Complexity: complicated

Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,cursor-grok; adjudicated=domain; declined=muse; rounds=5; full=2 @ ab34002

Review-Attention: study ~15m (critical: Table.ts; decisions: head-tie-on-first-delivery, skip-without-append, copy-row-log-identity, sibling-write-types; raised: degraded review) @ ab34002

@kriszyp kriszyp added this to the v5.3 milestone Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Release cherry-pick v5.3: conflict

Cherry-pick onto v5.3 produced conflicts on commit(s): 6c10cca7992015539e9457d0c7df3139e5dce8e2 f8f32b712f2fbe5efe889b7951c0d6151c6139a5 2eb7630c5fa62c44bfe5008fed22a7db28091954

The conflict markers are committed on branch cherry-pick/v5.3/pr-3476.
A pull request has been opened to land this patch: #3478

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Harper design documentation and implementation to improve the handling of replicated duplicates. Specifically, it introduces a priorWritesSkipped helper function in Table.ts to ensure that replicated writes are correctly identified and processed, and adds comprehensive test coverage in relayedHeadTieDuplicate.test.js to verify these scenarios. There are no review comments to address for this pull request.

kriszyp and others added 5 commits October 23, 2026 08:04
…the transaction log

A replicated write that matches the committed record's origin, version and log key (no explicit refs) is
already applied. That was only recognized on a retry; on a first delivery the keyed dedup looked for the
head's log entry, and when the receiver had none (a base-copied record) the exact-start lookup scanned to
end-of-log and the write was applied and logged again, so every peer was re-sent it (harper#3467).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e skip

A publish or relocate earlier in the same source transaction restamps the record at the transaction's
version without staging it, so the put after it tied with that restamp and was dropped. The skip now
requires every earlier write to the key in the transaction to have been skipped, which still lets a
re-delivered multi-write transaction skip all of its writes from the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…entation on it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kriszyp
kriszyp force-pushed the kris/3467-head-tie-dedup branch from ab34002 to b81bc92 Compare October 23, 2026 14:05
kriszyp added a commit that referenced this pull request Oct 23, 2026
v5.3 has a single log key, so the head-tie skip keeps txnLogKey === txnTime and adds priorWritesSkipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kriszyp
kriszyp marked this pull request as ready for review October 23, 2026 14:32
@kriszyp
kriszyp merged commit c7feeff into main Oct 23, 2026
52 of 53 checks passed
@kriszyp
kriszyp deleted the kris/3467-head-tie-dedup branch October 23, 2026 14:32
kriszyp added a commit that referenced this pull request Oct 23, 2026
v5.3 has a single log key, so the head-tie skip keeps txnLogKey === txnTime and adds priorWritesSkipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Oct 23, 2026
…hout reading the transaction log (conflicts → v5.3) (#3478)

* Skip relayed duplicates of a record's committed head without reading the transaction log

A replicated write that matches the committed record's origin, version and log key (no explicit refs) is
already applied. That was only recognized on a retry; on a first delivery the keyed dedup looked for the
head's log entry, and when the receiver had none (a base-copied record) the exact-start lookup scanned to
end-of-log and the write was applied and logged again, so every peer was re-sent it (harper#3467).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a put that follows an unstaged same-key write out of the head-tie skip

A publish or relocate earlier in the same source transaction restamps the record at the transaction's
version without staging it, so the put after it tied with that restamp and was dropped. The skip now
requires every earlier write to the key in the transaction to have been skipped, which still lets a
re-delivered multi-write transaction skip all of its writes from the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cover a put after a same-key relocate, and keep removalBefore's documentation on it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Name the relocate case in the head-tie design note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep only the constraint in the head-tie comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Resolve the v5.3 cherry-pick of #3476

v5.3 has a single log key, so the head-tie skip keeps txnLogKey === txnTime and adds priorWritesSkipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a replicated put that shares its transaction with a same-key publish or relocate from being skipped as a duplicate (#3499)

Port of 5d94a9a to v5.3. The head-tie identity now comes from the committed record whenever
another write to the key already saved unskipped, and a source-applied relocate that ties with the
committed head is skipped. v5.3 has a single txnLogKey (no origin-closed floor), so main's
receivedLogKey/appendedLogKey checks map to `txnLogKey === txnTime`; the relocate keeps v5.3's
plain return on an older version (#3183's skipped marking is not on v5.3), and the ref check keeps
v5.3's form without main's priorStaged guard.

(cherry picked from commit 5d94a9a)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SbvHaBBdQ7Cp7yVDuSD5GS
Dispatch-Task: cherry-resolve-kriszyp_harper_3476-123ff057

---------

Co-authored-by: Kris Zyp <kriszyp@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant