Skip to content

Confirm a transaction whose only write is a losing relocation on the stored entry, not an unwritten log key - #3183

Merged
kriszyp merged 4 commits into
mainfrom
fix/losing-relocate-skipped
Oct 19, 2026
Merged

kriszyp merged 4 commits into
mainfrom
fix/losing-relocate-skipped

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

A transaction that requests replicatedConfirmation waits on a log position that was never written when every confirmable write in it is a relocation that loses to a newer version (harper#3182). Peers never acknowledge that key, so the caller blocks until unrelated traffic replicates, or until the confirmation times out.

Cause: _writeRelocate's commit handler returned early on precedesExistingVersion(...) < 0 without setting write.skipped. Staging reads a falsy skipped as "this write appended an audit entry" and records appendedTo, so confirmationTarget chose the committed log key instead of falling back to the stored entry's version. Main only: the log-key target came from Count only a record's residents toward replicatedConfirmation (#3148), which no release carries yet.

❓ Your call: as specified, yes. The trigger is narrow (a confirmed transaction whose only confirmable writes are losing relocations), but the fix is the convention every other commit handler already follows. Milestoned v5.4, not v5.3: the _writeRelocate early return is also on v5.3, but nothing there reads skipped for a relocation (no appendedTo, no saved blobs).

💡 Solution

_writeRelocate's commit handler now resets write.skipped = false at the start of each round and sets it to true before the losing return, the same way _writeInvalidate and _writeDelete do: _writeRelocate marks its losing return skipped and resets it each round — the one runtime line pair to read. That is the whole runtime change. The per-round reset matters as well as the mark: when an open iterator forces the commit to re-stage its writes on a fresh transaction, a relocation that lost when first staged can apply on the re-stage, and a stale true would hide its append.

Audit of the other commit handlers in resources/Table.ts: none had the same gap. Every version-conflict early return in _writeInvalidate, _writeUpdate, _writeDelete and the source-resolution write already marks skipped; _writeUpdate's return writeCommit(false) paths do append an audit-only entry, so leaving them unmarked is correct; _writePublish has no early return; the reload-marker and lock-control writes set skipReplicationConfirmation.

Side effect checked: removalBefore now passes over a losing relocation when a later delete to the same key looks for what left it empty, as it already did for a losing invalidate and as its doc comment says. The relocation stored nothing, so the stored entry is the right answer there.

resources/DESIGN.md now says that staging reads skipped as "appended nothing", so an unmarked early return also breaks replicatedConfirmation, not just blob cleanup.

⚖️ Alternatives

Planning review: Framing-Verdict: chosen-approach-sound.

  • Have staging learn "appended" from the audit write itself (updateRecord / the transaction log reporting that it wrote) instead of inferring it from skipped. That removes this whole class of bug: a forgotten flag would fail safe. Not done here because it changes the bookkeeping contract for every handler and the sticky appendedAuditEntry retry dedup, which is more than this fix needs.

    ❓ Your call: worth a follow-up issue? Any future early return that forgets skipped brings this bug back without any visible error.

  • Set skipReplicationConfirmation on a losing relocation. Rejected: the caller would then get no confirmation of the surviving entry, which the existing fallback gives them.

✅ Verification

  • New tests in unitTests/resources/residencyConfirmation.test.js:
    • a confirmed transaction whose only write is a losing relocation confirms on the stored entry's version, leaves the record unchanged, and confirms once;
    • a losing relocation next to an applied write still confirms on the committed log key;
    • a relocation that loses when staged but applies when an open iterator forces re-staging (its base evicted in between) confirms on its own audit entry's log key. This test covers the per-round reset.
  • Fails on base: with origin/main's resources/Table.ts built into dist, the first test fails (confirms on the log key, version − 60000). With the fix but without the reset line, the re-staging test fails (0 == 1: no confirmation issued). Both pass with the fix.
  • npm run test:unit:resources: 4257 passing, 0 failing (RocksDB).
  • Under HARPER_STORAGE_ENGINE=lmdb, residencyConfirmation, localOnly, dualClockAuditRecord, indexOrphanStagingOrder and subscriptionSuperseded pass (45 passing). residencyConfirmation skips itself on LMDB, which keeps its own confirmation registry, so the new tests are RocksDB-only.
  • oxlint and prettier clean on the changed files.
  • End-to-end route: not observable from core alone; actual peer acknowledgement runs in harper-pro's replication layer. The unit tests check the value core hands to the replication callback, and the callback resolves immediately in them.

Refs #3182

— Claude Opus 5.5

🤖 Generated with Claude Code

https://claude.ai/code/session_01MquViG8a7UtvqFChsoL7qH

🤖 Generated by Claude Opus 5.5 (Claude Code dev-agent); posted via @kriszyp.

Release target: v5.4 — the log-key confirmation target this fixes came from #3148 (v5.4); v5.3 has no appendedTo, so the unmarked return is harmless there
Related PRs: #2446 independent (MCP session deletion; does not touch _writeRelocate or removalBefore)
Complexity: medium

Closes #3182

Dispatch: task harper-3182-losing-relocate-skipped · queued by unknown · ran by claude/opus/high · worker kzyp-xps-1

Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=2; full=1 @ db4669b

Review-Attention: study ~9m (critical: Table.ts; decisions: skipped-flag-as-append-proof, all-lost-confirmation-target) @ db4669b

kriszyp and others added 4 commits October 12, 2026 16:47
…stored entry, not an unwritten log key

_writeRelocate's commit handler returned early when a newer version already
existed without marking the write skipped, so staging recorded appendedTo for
it and confirmationTarget waited on a log key nothing was appended at. Reset
and set write.skipped the way the other commit handlers do.

Refs #3182

Dispatch-Task: harper-3182-losing-relocate-skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MquViG8a7UtvqFChsoL7qH
A relocation that loses when first staged but applies when an open iterator
forces the commit to re-stage it must report the append; a losing relocation
next to an applied write still confirms on the log key.

Refs #3182

Dispatch-Task: harper-3182-losing-relocate-skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MquViG8a7UtvqFChsoL7qH
…nded nothing"

Refs #3182

Dispatch-Task: harper-3182-losing-relocate-skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MquViG8a7UtvqFChsoL7qH
…ects, and fail on a missing audit entry

Refs #3182

Dispatch-Task: harper-3182-losing-relocate-skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MquViG8a7UtvqFChsoL7qH
@kriszyp kriszyp added this to the v5.4 milestone Oct 12, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that early-return paths in the Table commit handlers, specifically when a write precedes an existing version, correctly set write.skipped to true to prevent replication confirmation from waiting indefinitely (harper#3182). The changes include updates to resources/Table.ts to set this flag, corresponding documentation updates in resources/DESIGN.md, and new unit tests in unitTests/resources/residencyConfirmation.test.js to validate various losing relocation scenarios. There are no review comments, and I have no feedback to provide as the implementation is correct and adheres to the repository style guide.

@kriszyp
kriszyp marked this pull request as ready for review October 19, 2026 00:09
@kriszyp
kriszyp merged commit 809fa3c into main Oct 19, 2026
66 of 67 checks passed
@kriszyp
kriszyp deleted the fix/losing-relocate-skipped branch October 19, 2026 00:09
kriszyp added a commit that referenced this pull request Oct 23, 2026
…lish or relocate from being skipped as a duplicate (#3499)

Port of 5d94a9a to v5.3. The head-tie identity now comes from the committed record whenever
another write to the key already saved unskipped, and a source-applied relocate that ties with the
committed head is skipped. v5.3 has a single txnLogKey (no origin-closed floor), so main's
receivedLogKey/appendedLogKey checks map to `txnLogKey === txnTime`; the relocate keeps v5.3's
plain return on an older version (#3183's skipped marking is not on v5.3), and the ref check keeps
v5.3's form without main's priorStaged guard.

(cherry picked from commit 5d94a9a)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SbvHaBBdQ7Cp7yVDuSD5GS
Dispatch-Task: cherry-resolve-kriszyp_harper_3476-123ff057
kriszyp added a commit that referenced this pull request Oct 23, 2026
…hout reading the transaction log (conflicts → v5.3) (#3478)

* Skip relayed duplicates of a record's committed head without reading the transaction log

A replicated write that matches the committed record's origin, version and log key (no explicit refs) is
already applied. That was only recognized on a retry; on a first delivery the keyed dedup looked for the
head's log entry, and when the receiver had none (a base-copied record) the exact-start lookup scanned to
end-of-log and the write was applied and logged again, so every peer was re-sent it (harper#3467).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a put that follows an unstaged same-key write out of the head-tie skip

A publish or relocate earlier in the same source transaction restamps the record at the transaction's
version without staging it, so the put after it tied with that restamp and was dropped. The skip now
requires every earlier write to the key in the transaction to have been skipped, which still lets a
re-delivered multi-write transaction skip all of its writes from the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cover a put after a same-key relocate, and keep removalBefore's documentation on it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Name the relocate case in the head-tie design note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep only the constraint in the head-tie comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Resolve the v5.3 cherry-pick of #3476

v5.3 has a single log key, so the head-tie skip keeps txnLogKey === txnTime and adds priorWritesSkipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a replicated put that shares its transaction with a same-key publish or relocate from being skipped as a duplicate (#3499)

Port of 5d94a9a to v5.3. The head-tie identity now comes from the committed record whenever
another write to the key already saved unskipped, and a source-applied relocate that ties with the
committed head is skipped. v5.3 has a single txnLogKey (no origin-closed floor), so main's
receivedLogKey/appendedLogKey checks map to `txnLogKey === txnTime`; the relocate keeps v5.3's
plain return on an older version (#3183's skipped marking is not on v5.3), and the ref check keeps
v5.3's form without main's priorStaged guard.

(cherry picked from commit 5d94a9a)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SbvHaBBdQ7Cp7yVDuSD5GS
Dispatch-Task: cherry-resolve-kriszyp_harper_3476-123ff057

---------

Co-authored-by: Kris Zyp <kriszyp@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

replicatedConfirmation waits on an unwritten log key when a relocation loses to a newer version

1 participant