[ldt1996]

⊙ Problem

On 5.4, after a restart, worker restart or replication reconnect, a node delivers replicated MQTT messages to its live subscribers a second time: its resume re-sends the log below the last certified floor (harper-pro#1092), and a re-sent publish was logged and delivered again.

💡 Solution

Stacked on #4384, which keeps a record's version from moving back.

• A replicated publish is skipped when its origin's log here already holds it (same key, table, record key and version): nothing is logged or delivered (check, lookup).
• Only a message at or below its topic's record version is looked up, so an in-order publish reads no log; a missing record is looked up only on a table that can drop records (guard).
• A message the node never applied is still applied once; local publishes and LMDB are unchanged. Release note; DESIGN.md.

⚖️ Alternatives

• A rocksdb-js committed-identity lookup (planning review: option set too narrow) — would also close the concurrent-apply race, but needs a native release; that race only duplicates (Two concurrent applies of one replicated transaction can both log it, because re-delivery checks read only up to the log's committed-read watermark #4340).
• harper-pro's leading-duplicate fast-skip — armed only for the subscription's own node; core's apply owns duplicate detection.
• Dedupe at delivery — lost on a node restart, and the copies would still be logged and relayed.
• Resume at the applied position — loses chore: bump lmdb to 3.5.5 #1092's re-send of a late commit below it.

🤔 Where a re-sent message is still applied again

A duplicate, never a loss: two applies of one origin transaction racing (#4340); an entry purged by audit retention (#4348); markers configured off; a topic record an earlier version moved back, or a deleted topic whose tombstone was cleaned up.

🤔 Decision: one narrow loss case is accepted

5.3 wrote no abort markers. If a 5.3 apply logged a message and failed for good just before the upgrade, and that transaction is re-sent after another write moved its topic's record past it, the message is skipped. Trusting no pre-restart file would close this, but would re-apply every superseded message after each restart until those files are purged.

✅ Verification

• replicatedPublishRedelivery.test.js: re-delivery cases fail on main and pass; new cases check that an in-order publish to a held topic and the first publish to a new topic read no log (test). relayedHeadTieDuplicate.test.js expects a re-delivered publish to read its entry once.
• test:unit:resources 8,194 pass on RocksDB, test:unit:lmdb 6,098; failing in both runs only rocksOpenConfig, LMDB's startTime replay handoff and Replay a long run of patches to a record in linear time #4337's bounded-history case, which fail on main too, and originClosedFloor, which passes in isolation.
• Two harper-pro nodes on one host, QoS 0 at 20 messages/s: base gave 111 subscriber duplicates after the publisher restarted and logged 103–111 messages twice after a worker or node restart; the fix gave 0. Deliver replicated messages to subscribers once after a restart or replication reconnect harper-pro#1362 covers the same restarts.

Performance: no cost on in-order publishes (min µs/op over 7×20k replicated publishes, two rounds): held topic 17.5–18.0 vs main 17.9–18.0; new topic 16.8–16.9 vs 16.8–16.9. The unstacked check cost 20.3–20.7 on the same held-topic run.

PR QA: skipped @ 6a06693 — dispatch is not available in this session

_{🤖 Generated by Claude Opus 5.5 (Claude Code); posted via @ldt1996.}

Release target: v5.4 — fixes a regression in the 5.4 betas (harper-pro#1092's floor-only resume); 5.3 has no floor-only resume

Framing-Verdict: option-set-too-narrow (c11e475ee1be)

Related PRs: #4384 overlaps (this PR's base: a record's version never moves back); #3137 overlaps (publish delivery in resources/Table.ts); 2 others independent
Complexity: complicated

_{Review-Coverage: authored=claude; ran=gemini,codex,gemini-carbon; late=gemini-carbon:adjudicated; adjudicated=domain; blocked=cursor-composer(not-installed); declined=cursor-grok,muse; rounds=4; full=2 @ 6a06693}

_{Review-Attention: deep ~30m (critical: Table.ts; decisions: do-less-alternative, requirement-shift, uncertified-history-trust, perf-cost; raised: degraded review) @ 6a06693}

[kriszyp]

Thanks for running this down. Probing the origin's log at the incoming key is the right way to tell a duplicate from a late message, and the restart and reconnect numbers make the case well.

But the premise the per-publish probe rests on: "a late one lowers the record's version". On RocksDB, _writePublish restamps the record unconditionally: recordUpdater stores newVersion as the record's version and log reference (RecordEncoder.ts, "with rocksdb, we simplify to just storing the singular version/timestamp"). A replicated message also runs no precedesExistingVersion check. So a late publish moves the record's version and log reference backward. I think that backward move is the defect, and the duplicates are one symptom of it:

The record then points at an older log entry than the latest message for that topic, until the next publish.
A later replicated put compares its version with the lowered one. Example: a put at 200 is applied, a late publish at 150 lowers the record to 150, then a late put at 170 wins on this node (precedesExistingVersion, txnTime <= existingEntry.version), while the origin keeps the 200 value. We're checking this with a repro now, so treat it as unconfirmed.
If a publish only ever moves the record's version forward (keep the maximum, while still logging and delivering the late message), dedup can check the version first:

a message newer than the record's version is new, so no probe;
one at or below it is late or a duplicate, so probe the origin's log exactly as this PR does.
That takes the 2.3 µs off every in-order publish and makes #4341 unnecessary. It would also cover the "a later-arriving message lowered the record" case listed under "Where a re-sent message is still applied again". The open question is whether walking a topic's message history through its previous-version links still finds a late message once the record's reference never points at it.

I think it would be difficult to justify this significant of performance regression, and so I think we should address the root cause instead. Let me know if you want any help with this. I'm also drafting a PR with a test to reproduce the lower-level issue.

Drafted with a little bit of help from Opus 5.5.

[kriszyp]

[major] Retained old files also allow duplicate delivery of messages committed into newer files

[kriszyp]

Here is a reproduction demonstrating this is a inbound replication handling issue, not a subscription issue: #4360

[kriszyp]

_{🤖 Reviewed with Codex}

[kriszyp]

[major] Newly committed messages can still be delivered twice after restart

The missing-topic trigger from the prior finding remains. On an expiring or evicting audited table, retain an origin's pre-restart file, roll into a new file, commit message A after restart onto a populated topic, then evict that record without purging A's log entry. Re-send A: this branch refuses the lookup because auditHistoryAbortMarked() remains false while the older file survives (resources/auditStore.ts:1913-1922). A is logged and delivered again even though its own batch was written with abort markers and committed in the newer file.

Preserving record versions fixes the late-publish trigger, but cannot supply evidence after eviction removes the record. Scope trust to the matched frame's file/sequence, potentially through a rocksdb-js committed-identity primitive. The discriminating check is that restart/rollover/apply/evict/re-send sequence, expecting one delivery of A. The simulated old-history case at unitTests/resources/replicatedPublishRedelivery.test.js:283-302 retains the topic record, so it does not exercise this branch. This remaining outcome is supported by code tracing; I could not run a database repro in this read-only checkout.

[kriszyp]

[major] Newly committed messages can still be delivered twice after restart

_{(resources/Table.ts:11158 is not part of this PR's diff — anchored to the nearest line this PR's diff can hold)}