[kriszyp]

⊙ Problem

When a transaction writes its log batch and then fails, readers still treat that entry as committed. So Harper never gives up on a logged retry: under contention it retries without limit. A retried replicated write also skips duplicate checks, so it can apply an already-committed duplicate twice (#3453, #3513).

🤔 Is this the right requirement?

Yes. It is the 5.4 plan from #3253. Setting it to false keeps today's behavior.

💡 Solution

• storage.rocks.transactionLogAbortMarkers is now on by default. rocksdb-js marks a failed batch aborted, and every reader skips it.
• A logged commit can now fail with a 500 after 41 attempts that conflicted at commit (noteRetryBoundary), or a 503 at the deadline. A chain link after a committed store still retries until it commits (mustRetryLoggedCommit; ⚠️ look hardest here).
• New storage.rocks.transactionLogWriteIntents (default off, latchWriteIntents) refuses a commit before its log write while an older transaction holds a write intent. The retry then re-logs.
• A retried replicated write runs the normal duplicate checks, including past the 1000-entry walk cap (mayReadOwnOrphan; ⚠️ and here).

🤔 After a restart, retry dedup on a log stays off until its older files are purged

The owner accepted this window of about 3 days, during which a twin delivery can double an increment or CRDT add. That matches main but regresses v5.3.5 (#3524). rocksdb-js #980 and the walk cap bound the scan cost; #3941 would make certification survive restarts.

⚖️ Alternatives

Six alternatives rejected

• Fall back to Stop retried replicated writes from re-applying a duplicate that already committed #3453's committed-link walk while a log is uncertified: rejected by owner ruling. It brings back the provenance machinery this PR retires, to cover a window that only matches main.
• Durable per-segment marker stamps from rocksdb-js: needs a log-format change outside this PR. Tracked in Keep retry dedup on across restarts by recording which transaction-log files were written with abort markers #3941.
• A Harper-side persisted record of which history is marker-era: a downgrade and back leaves it stale.
• The prototype as written (treat no logged batch as durable when markers are on): frees blobs that a write-once retry still references.
• Flip the default only: logged retries stay unbounded, and A retried replicated write can re-apply a committed duplicate buried beyond the audit walk's depth cap #3513 stays open.
• Write intents on by default: the owner ruled against it; it costs throughput and does not cover aborts after the log write.

🤔 Lease expiry and commit suspension still wait on a logged retry

Refusing one write in a write-once batch would still publish that write's entry. Giving up the whole handle is tracked in #3905.

🔧 Changes

• resources/DatabaseTransaction.ts: noteRetryBoundary counts conflicts found at commit and re-logs refused attempts. mustRetryLoggedCommit decides the cap and the deadline. abortedWithoutLog lets rollback and blob cleanup run only when the abort left nothing a reader can apply: no batch, or a batch that was marked. An abort that cannot be marked logs an error, and a deadline 503 then reports retryable: false.
• resources/Table.ts: mayReadOwnOrphan now gates the five audit-log duplicate checks. readsOwnStaging is unchanged.
• resources/auditStore.ts: establishAbortMarkedHistory and auditHistoryAbortMarked record, per log, which files predate this process.
• resources/transactionLogAbortMarkers.ts (new; both settings), utility/hdbTerms.ts, databases.ts, residencyPolicy.ts: the default, set once per process and passed to the workers the main thread starts afterwards.
• resources/originClosedFloor.ts: rekeyLocalRetry. RocksTransactionLogStore.ts: the re-log gate. RecordEncoder.ts and stagesLog: a re-log staging counts as a fresh one, so a write claims its blob again only once its entry is staged.
• .github/workflows/review-coverage.yml: lists the new module among the storage-binding framing paths, as the policy test requires.
• resources/DESIGN.md, resources/MAP.md: notes for both modes.

✅ Verification

• New abortMarkersAdoption.test.js and the ported Stop retried replicated writes from re-applying a duplicate that already committed #3453 set in replicatedRetryDuplicate.test.js (run with markers on and off) cover real conflicts, a real write-intent refusal, the 500, the 503, chain links, the derived-index runtime, the walk cap and pre-marker orphans. The tests of the new behavior fail on the markers-off (main) path.
• All three test:unit:resources shards are green, and 81 of 81 integration tests in a related subset pass. The hot-key commit bench shows markers on vs off within noise.
• PR QA: Do concurrent conflicting writes across workers keep every acknowledged write exactly once? — not finished in 90m @ c8dfa03 (queued behind the fleet's QA lane), qa-pr-harper-3928-c8dfa03
• Release note: a 5.3 build older than the rocksdb-js 2.13.0 pin must not read logs a 5.4 node wrote.
• Docs: HarperFast/documentation#780 covers the default change, the downgrade requirement and the write-intents setting. Release-note tracking: #3942.

Details

• Fails on base: with markers off (main's code paths), the cap, 503 and re-log tests hang or fail. The past-cap duplicate is applied twice, and Stop retried replicated writes from re-applying a duplicate that already committed #3453's six residual cases fail. Forcing the history to certified drops the write in the pre-marker orphan case.
• Pinned off: suites that fake conflicts without RocksDB (noAbortAfterTxnLog, requestPathRetryExhaustion, one test in commitConflictDeadline, and the forced-conflict helper in immediateTransactionConflictRetry) now pin markers off, where their assertions hold. shardingSupport now sets the setting off explicitly where it expects a refusal.
• test:unit:main: 1242 passing and 0 failing when a 50-minute local cap stopped it; CI runs the full gate. The only resources-shard failures were full-text and subscription timing tests, which also fail with markers off or pass on a rerun.
• Bench, 2 runs each (8 threads, commits/s, off vs on): hot4 2134 and 2247 vs 2089 and 2373; cold1 9468 and 7530 vs 8901 and 7763; cold4 6723 and 6215 vs 6484 and 6579. 0 errors, and every count check passed.

Supersedes #3453. Closes #3513: with markers on, the default, a retry past the walk cap now deduplicates once its log's pre-process files are purged. Refs #3253.

_{🤖 Generated by Claude Opus 5.5 with Claude Code; posted via @kriszyp.}

Release target: main — the 5.4 default change; v5.3 already reads markers (#3828) and keeps them opt-in
Related PRs: #3453 supersedes, #3524 overlaps (the v5.3 port of #3453; main gets this PR instead), #3728 overlaps (also changes retry staging in resources/RecordEncoder.ts), #3910 overlaps (open; also edits resources/Table.ts and resources/DESIGN.md), #3856 overlaps (merged; the DESIGN.md cut this branch rebased onto), 13 others independent
Complexity: complicated

Deferred findings

• 357dfee6eed04dd2 minor reliability — resources/DESIGN.md — logged retries still bypass suspension/lease refusal with markers on, prolonging drain — filed With abort markers on, let suspension and lease expiry give up a logged retry #3905
• 2e694142be678601 minor data-integrity — resources/Table.ts — audit-only writeCommit(false) appends a blob-bearing audit entry without setting blobsAuditReferenced; a later skip/supersede deletes the bl — filed An audit-only staging can lose a blob its committed audit entry references #3912
• 8e3b71330068c463 nit data-integrity — resources/transactionLogAbortMarkers.ts — default-on markers write aborted batches that older log readers treat as live — NOT FILED
• 968b54995b7248e9 nit documentation — resources/transactionLogAbortMarkers.ts — default-on, restart-only abort-marker (and opt-in write-intent) setting lacks operator documentation — NOT FILED

Late Grok findings

4 surviving (highest major) — Grok reviewed 77121e9

• major — resources/databases.ts:502 — abort markers now default on for every audited RocksDB commit and every transaction-log reader, and the PR includes no throughput or latency measurement
• minor — resources/transactionLogAbortMarkers.ts:17 — a worker created before the main thread latches resolves the setting itself. When that worker opens a database, it can write a different value into the process-global native setting
• nit — resources/residencyPolicy.ts:39 — the attribute-sharding gate reads the live config, not the latched value
• nit — static/defaultConfig.yaml:87 — operators are not told about this setting: storage.rocks has no transactionLogAbortMarkers entry and no operator doc mentions it

Not standing: none. Grok ran 19m (19m from launch to report); the fold adjudication (cursor-grok@77121e91fdca-dbd456fb, claude) took 1m.

Late Grok findings

8 surviving (highest major) — Grok reviewed f3e5cae

• major — resources/transactionLogAbortMarkers.ts:17 — a worker that loads before main publishes the latch picks its own policy, and resources/databases.ts:502 writes that choice into process-global native config
• minor — resources/DatabaseTransaction.ts:3040 — abandonScope clears rotatedAfterCommit (:3798) while a chain commit is in flight, so the pending link can be abandoned after an earlier segment committed, and the deadline error reports retryable
• minor — resources/DatabaseTransaction.ts:405 — at the retry cap, ERR_TRANSACTION_ABORT_UNRESOLVED is swallowed and both cap exits (:3376-3378, :3595-3597) return the generic 500 while crash replay can still apply the batch
• minor — resources/DatabaseTransaction.ts:1622 — a re-logged retry that ends up skipped keeps blobsAuditReferenced/appendedTo; the same-handle exemption at :3490 leaks the blob file
• minor — resources/residencyPolicy.ts:39 — attribute-sharding admission reads live config while native storage follows the latched value
• nit — resources/originClosedFloor.ts:420 — rekeyLocalRetry declines caller-value reservations, so the retry re-appends at its original minted key
• nit — resources/Table.ts:6205 — added comments repeat the orphan rationale and narrate code and test steps
• nit — resources/transactionLogAbortMarkers.ts:13 — nothing tells operators about the new default, the false opt-out, or the whole-process restart requirement

Not standing: 7 merged. Grok ran 10m (11m from launch to report); the fold adjudication (cursor-grok@f3e5cae0b9fd-6c0a7878, claude) took 1m.

Late Grok findings

4 surviving (highest nit) — Grok reviewed 1564a87

• nit — resources/residencyPolicy.ts:39 — the attribute-sharding gate reads the live config value instead of the latched setting
• nit — resources/DatabaseTransaction.ts:4050 — the "Abandoning a write transaction" error log is built before the abort runs, so it says neither "not retryable" nor "outcome unknown" when the abort then fails to mark the batch
• nit — unitTests/resources/abortMarkersAdoption.test.js:317 — the blob regression test releases the holder on a delay(100) timer instead of after an observed pre-log refusal
• nit — resources/transactionLogAbortMarkers.ts:13 — operators have no guidance yet: the setting is now on by default, false opts out, a change needs a whole-process restart, and a batch this process aborts is read as committed by an older reader

Not standing: 1 merged. Grok ran 16m (16m from launch to report); the fold adjudication (cursor-grok@1564a874b25d-060511aa, claude) took 1m.

_{Ready-For-Promotion: 5fb516d urgency=normal late=folded}

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=12; full=4 @ 5fb516d}

_{Review-Attention: deep ~50m (critical: DatabaseTransaction.ts, RecordEncoder.ts +5; sensitive: review-coverage.yml; decisions: write-intents-opt-in, dedup-certification-window, markers-default-on, latch-local-fallback, do-less-alternative, release-target-unclear) @ 5fb516d}

[gemini-code-assist]

Code Review

This pull request implements support for transaction-log abort markers in RocksDB, introducing process-global latching of the setting, tracking abort markers on aborted transactions, and adjusting retry boundaries, conflict budgets, and deduplication logic accordingly. The review identified two critical issues in resources/DatabaseTransaction.ts: a ReferenceError caused by using an undefined txn variable in abortNativeTransaction, and a potential TypeError in abortedWithoutLog when transaction is nullish but error?.hasLog is true.