[kriszyp]

⊙ Problem

A publish sets a record's version to its own, with no ordering check: _writePublish passes its txnTime to updateRecord, and recordUpdater stores it as the record's version. Puts and patches compare against that record version. A publish below the record's version moves it backward. This can be a late replicated publish or a local one on a node whose clock is behind a peer's.

💡 Solution

No product code changes. New tests reproduce each consequence on the engine where it occurs, and check LMDB where it differs, applying replicated writes through the methods replication calls. Each failing case pins today's wrong result, so CI stays green until something changes it; RUN_KNOWN_FAILURES=1 asserts the correct result instead.

🔎 Findings

1. Divergence: confirmed, both engines. On B@200, a late publish@150 then a late put C@170 leaves C@170. A node that got the publish first holds B, since there 170 loses to 200.
2. Version and log head: confirmed. The version drops to 150. On RocksDB it is also the head log key, so reads starting from the record miss newer entries. Previous-version links still reach every message, out of version order.
3. LMDB: same divergence; its head is local time, so reads are unaffected.

Observed values, every case (offsets in ms from the case's base)

Case	Expected	RocksDB	LMDB
put B@200, late publish@150, late put C@170	B@200	C@170	C@170
put B@200, late put C@170, late publish@150, late put D@160	B@200	D@160	D@160
tie: put B@200, late publish@150, put C@150 from the peer that wins ties	B@200	C@150	C@150
peer puts B at now+60 s, local publish, peer puts C at now+30 s	B	C, 30 s behind B	C, 30 s behind B
record version after put@200 and late publish@150	200	150	150
getHistoryOfRecord after messages 1@100, 3@300, then late 2@200	1, 2, 3	1, 2	1, 2, 3 ✔
getHistoryOfRecord after put@200 and late publish@150	put, message	message	✔
subscribe({ previousCount: 1 }), same messages	3	2	skipped: newest is the last to arrive
subscribe({ previousCount: 3 })	1, 2, 3	✔	✔
subscribe({ startTime: 250 })	3	nothing	skipped: startTime is local time
previous-version links from the record	all three	✔ (200 → 300 → 100)	n/a
control: put B@200, late put C@170	B@200	✔	✔
in-order publish@300 after B@200, then late put or patch @250	applied	✔, version stays 300	✔, version 250

On RocksDB, getHistoryOfRecord starts at the record's head and follows a previous link only below its window. The late entry links up to 300, so the walk stops. A startTime backfill walks only when the head is above startTime. A count backfill follows links to the end, which is why previousCount: 3 finds all three.

Fix sketch: restamp the record only forward

In _writePublish's commit, order the publish against the record with precedesExistingVersion. When it precedes, log and deliver it but keep the record's version and node id (recordUpdater already takes a separate audit version, options.recordVersion). Then a replicated publish strictly above the record's version cannot be a re-delivery, so #4339's log probe is needed only for late and tied ones.

What the forward-only restamp would break

• RocksDB history from the record. Nothing would link to the late message: the next write's previous version is the head the record kept. getHistoryOfRecord and the previousCount/startTime backfills would miss the late message instead of the newer ones they miss today. Keeping it reachable means listing its log key in the record's additionalAuditRefs, as out-of-order puts do. resolveAuditHead still picks the entry whose version equals the record's, so the head stays right. The cost is one ref per late message until the next in-order write, after which it survives only as previousAdditionalAuditRefs.
• The audit previous-version chain. Today a late message puts the chain out of version order. With forward-only it stays in order but skips the late entry unless the refs carry it. Value reconstruction (the out-of-order merge walk, getRecordAtTime) is unaffected, since a message does not change the value. Anything that lists messages from the chain is affected.
• Ties. A publish at the record's exact version from another node needs the same node-name rule as puts. Otherwise the record's node id moves without a value change, and a later put's tie-break is decided against the wrong node.
• Local publishes. The same rule must cover them, since a local publish behind a peer's clock moves the version backward too. The alternative, setting local times above the record's version, is a larger clock change.
• Kept behavior, pinned here. An in-order publish ahead of a late put or patch still lets that write apply. RocksDB keeps the head at the publish's version, and LMDB's merge stores the write's own version (LMDB: an out-of-order merge on an audit:false record stores the older timestamp as its version #3388).
• LMDB only needs the stored version kept at the maximum. Its local-time head already advances on every write.

🤔 Known failures pinned in CI rather than skipped

Instead of it.skip: a fix, or any other change here, turns CI red, so whoever changes the behavior updates the test. The cost is green tests titled with the correct behavior.

🤔 Unit-level replication, not a two-node cluster

The tests call _writePublish/_writeUpdate with a source-apply context, as #4339's test does: the methods replication uses. These orders on two real nodes are inferred, not run.

✅ Verification

• The new file: 14 passing on RocksDB, 11 passing and 3 pending on LMDB, 14 under type stripping. With RUN_KNOWN_FAILURES=1, 9 fail on RocksDB and 5 on LMDB, with the table's values.
• test:unit:resources:core:a-n (this file's shard): 3434 passing, 0 failing. Lint, prettier and check:unit-partition clean.
• Not run: test:unit:main and integration; no product code changed.
• PR QA: skipped @ 1d69f5d — test-only PR.
• Docs: none needed; no behavior changes.

Refs #4339

_{🤖 Generated by Claude Opus 5.5 (Claude Code); posted via @kriszyp.}

Release target: main — test-only, no product change

Related PRs: none found
Complexity: easy

Deferred findings

• bdb95d67e2deef59 nit test-coverage — unitTests/resources/latePublishVersionRegression.test.js — LMDB runs h1/h2 as plain it(), assuming LMDB history parity — NOT FILED
• 8b53ed623607d829 nit maintainability — unitTests/resources/latePublishVersionRegression.test.js — suggest a comment explaining why previousCount: 3 stays green — NOT FILED
• deb63d6a8805443d nit maintainability — unitTests/resources/latePublishVersionRegression.test.js — suggest cross-referencing p1 from h1 so p1 passing is not read as h1 should pass — NOT FILED
• efd63e8b68bfdb87 nit maintainability — unitTests/resources/latePublishVersionRegression.test.js — added comments narrate code or describe current pre-fix behavior — NOT FILED

Origin — the dispatch brief this PR was written from

Reproduce: a late replicated publish moves a record's version backward on RocksDB, letting an older replicated put win

Test-only investigation in HarperFast/harper. Change no product code; deliver a draft PR containing the reproducing test(s) and a findings write-up in the body.

Hypothesis (from code reading, not yet confirmed):

• On RocksDB, Table.ts _writePublish commits through updateRecord(id, value, existingEntry, txnTime, …, 'message'). recordUpdater (resources/RecordEncoder.ts, the if (isRocksDB) timestampNextEncoding = newVersion branch) stores the publish's txnTime as the record's single version and transaction-log reference, unconditionally.
• A replicated message is applied with no precedesExistingVersion check. So a replicated publish that arrives late, with a timestamp below the record's current version, moves the record's version backward.

Consequences to test:

1. Divergence on the receiving node:
   • apply a replicated put at version 200 (value B);
   • then a late replicated publish to the same record at version 150;
   • then a late replicated put at version 170 (value C).
     Expected, if the hypothesis holds: this node ends with C at version 170, while last-writer-wins says B at 200 must stay. Also check the reverse order and a timestamp tie.
2. After the late publish, the record's version and log reference point at the 150 entry, not the latest. Show what a reader that finds a topic's latest messages through the record sees, and whether walking history through previous-version links still finds every message.
3. Run the same cases on LMDB and say whether it differs.

Simulate replicated applies the way unitTests/resources/replicatedPublishRedelivery.test.js on #4339 does: a source-apply context with an origin nodeId and timestamp. A two-node harper-pro cluster test is not needed unless the unit-level simulation cannot reach the path.

Write the tests so they fail on main if the bug is real, and mark them as expected failures so CI stays green. In the PR body, state whether each consequence is confirmed, with the observed versions and values. Then sketch a fix: a publish restamps the record only forward (keep the maximum version) while still logging and delivering the late message. Say what that would break, especially history walks and the audit previous-version chain. Context: harper#4339 dedupes re-sent publishes by probing the origin's log on every replicated publish, because the record's version can move backward; a forward-only restamp would let a version check skip that probe for in-order messages.

Acceptance

A draft PR with tests that fail on main for each confirmed consequence, marked as expected failures, or a clear explanation of why a hypothesis does not hold. The PR body includes the fix sketch and its risks.

_{Dispatch: task harper-late-publish-version-regression-repro · queued by kris-session · ran by claude/opus/high · worker kris-mac-m3-fleet-18b9c6}

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=3; full=2 @ 1d69f5d}

_{Review-Attention: skim ~2m (decisions: pin-exact-wrong-result, private-write-fixture) @ 1d69f5d}