Skip to content

Keep a replicated write with no expiry from taking its transaction's first expiry - #4391

Draft
DavidCockerill wants to merge 8 commits into
mainfrom
david/4390-replica-expiresat-inherit
Draft

DavidCockerill wants to merge 8 commits into
mainfrom
david/4390-replica-expiresat-inherit

Conversation

@DavidCockerill

@DavidCockerill DavidCockerill commented Nov 13, 2026 •

Copy link
Copy Markdown
Member
A cow cashier stamps the milk's expiry date onto a skillet, a brick and a gnome in the same bag, as a replicated row takes the expiry of the first row in its transaction

⊙ Problem

A transaction writes a row with an expiry, then a row without one. A node applying it from a source gives the second row the first row's expiry. A replica applies every origin transaction this way. After that expiry the replica hides the row, and its cleanup scan can delete it. The origin keeps it. Present in 5.3.6 and 5.4.0.

Closes #4390

💡 Solution

🤔 Should this also repair rows already stamped with a borrowed expiry?

No. It changes new writes only. A replica row that already borrowed an expiry keeps it until rewritten. A row already evicted needs a resync from the origin. Nothing on the row distinguishes a borrowed expiry from its own.

⚖️ Alternatives

  • Send -1 ("no expiry") from harper-pro's receiver — skips the receiving table's own expiration and @expiresAt; custom sources stay broken.
  • Give each source transaction a context of its own instead of its first event — a dozen transaction fields (log timestamp, origin, commit hooks) would need copying, and expiry is the only per-write field that leaked.
  • Let any expiresAt key in the options override the context — changes _writeUpdate's contract for every caller, not only writes applied from sources.
Left out: two related gaps that predate this PR
  • An explicit -1 is not replicated. An origin write with expiresAt: -1 stores no expiry, so a replica applies its own table default. Fixing it needs an audit-format change.
  • A write triggered by a source event without the source flag still shares its transaction's context and expiry, as on main.

🤔 Should a transaction event's own expiresAt cover the writes in it?

No: each write uses only its own expiresAt. A custom source that set one only on the wrapper no longer gets it on those writes; they fall back to their @expiresAt field or table expiration. Nothing documented the old behavior, and restoring it is one condition.

🔧 Changes

Product and architecture tour: which expiry a source-applied write stores, and where the borrowed one came from (one diagram)

Product and architecture tour

Which expiry a source-applied write stores

Which expiry does a write applied from a source or a replication peer get when its event carries none?

Before After
A write with no expiresAt in a source transaction took the expiry of the first write, or of the transaction event. It takes its record's @expiresAt field, then its table's expiration, then none.

A source-applied write's expiry comes from its own event, else its record's @expiresAt field, else its table's expiration. It never comes from another write or from the transaction event.

Where the borrowed expiry came from

The apply loop reuses a transaction's first event as the context for every write in it. Expiry was the only per-write field that fell back to that context.

flowchart LR
    E1[First event, expiry T] -->|becomes| C[Transaction context]
    E2[Later event, no expiry] --> W[_writeUpdate]
    C --> W
    W -->|before: falls back to T| H[Row hidden after T]
    W -->|after: field or table default| K[Row kept]
Loading

Every other per-write field (nodeId, version, residencyId) is read from the write's own options only, so nothing else leaks the same way. isNotification is set only by the apply loop's writeUpdate, which also passes each event's own expiresAt. A -1 still means "never expires", and a write's own deadline still wins.

Table.ts:6834

- expiresAt = options?.expiresAt ?? context?.expiresAt
+ expiresAt = options?.expiresAt ?? undefined
+ if (expiresAt === undefined) {
+   contextExpiresAt = context?.expiresAt
+   if (contextExpiresAt != undefined && !options?.isNotification) expiresAt = contextExpiresAt
+ }
  …
  if (expiresAt == undefined) expiresAt = fieldDeadline ?? tableDefault ?? -1   // @expiresAt, then expiration

Files

✅ Verification

  • sourceTxnExpiry.test.js fails on base in 4 of 6 cases, on RocksDB and LMDB. All 6 pass with the fix.
  • End to end: HarperFast/harper-pro#1410 adds a two-node test of this case. With core at base the replica loses Plain; with this branch's head it keeps it.
  • Gates: every local failure also fails on base, or passes alone (details below). test:unit:main is left to CI.
  • Post-review delta: one comment in Table.ts reworded, not re-reviewed.
  • No release-note fragment: this ships as a 5.4 patch, which GitHub releases describe.
  • Performance: _writeUpdate runs on every write. A write with no expiry in its options or context reads what it read before; only one that would take the context's expiry reads options.isNotification too. No allocation, lock or await.
  • PR QA: skipped @ 57b9017 — dispatch unavailable (HTTP 409: no registered worker reports the general QA lane)
Test cases
Case On base
A write with no expiry after an expiring one, on a table with no expiration: no stored expiry, still readable after the first expires fails
The same on a table with expiration: 3600: gets the table default; a -1 write stays without expiry fails
A put and a patch with an @expiresAt field: each keeps its field's deadline fails
A transaction event with its own expiresAt: the write inside gets none fails
Two writes with their own expiries keep them passes
An ordinary write in a context with expiresAt still gets it passes
Gate results

Full unit gates ran at e990466 and the integration gate at fddf5ea, on harper-pro main 43599092 for the cluster run (base core 4f20ed5). Lint, format, design-doc and unit-partition checks pass. test:unit:main cannot finish in one process on this Mac. Later commits change only the form of the expiry expression and comments; the focused suites (sourceTxnExpiry, sourceTxnStreams, outOfOrderWriteExpiry, expirationWarning, caching, replicatedApplyFailure) were rerun on both engines at 16acead.

Gate Result Failures
test:unit:resources (RocksDB) 8159 passing, 2 failing rocksOpenConfig writeBufferManagerFlushPolicy fails the same on base; originClosedFloor passes alone
test:unit:resources (LMDB) 6092 passing, 2 failing the same rocksOpenConfig case; historyReader "stops search windows at the lower retained-log boundary" fails on base too (2 of 2 runs)
test:integration:all 4033 passing, 49 failing, 35 cancelled 22 suites; every failing subtest fails identically on a base build (UDS mirror path length and the single HTTP worker on this Mac)
A related warning with a different cause

A 5.4.0-beta.2 replica also logged the "per-record expiresAt was set" warning for tables that do have an expiration. That is a separate defect: a table a node received through add_node lost its expiration, fixed for 5.4.0 by HarperFast/harper-pro#1233.

🤖 Generated by Claude Code (Anthropic Claude Opus); posted via @DavidCockerill.

Release target: v5.4 — the issue's milestone; the code is also on v5.3, but no customer needs a v5.3 patch
Related PRs: #2155 overlaps (edits the same expiry line in _writeUpdate; its version keeps the context fallback)
Complexity: complicated

Deferred findings

  • 378124750da8e5ed major data-integrity — resources/RecordEncoder.ts — -1 never-expire override is not encoded in record metadata or audit, so replicas apply their table default — NOT FILED
  • 9f628a1ecdd9c2b6 minor data-integrity — resources/Table.ts — a write sharing a source transaction's context without isNotification (derived or side-table write) still inherits the first event's expires — NOT FILED

Review-Coverage: authored=claude; ran=gemini,codex,gemini-carbon; late=gemini-carbon:adjudicated; adjudicated=domain; declined=cursor-grok,cursor-composer,muse; rounds=7; full=3 @ 57b9017

Review-Attention: study ~11m (critical: Table.ts; decisions: wrapper-expiry-semantics, gate-layer, suppress-at-writeupdate, perf-cost) @ 57b9017

DavidCockerill and others added 8 commits November 13, 2026 16:10
…first expiry

The source apply loop stages every write of a source transaction under its
first event (or the transaction wrapper) as the context, and _writeUpdate
resolved options.expiresAt ?? context.expiresAt. A write whose event carries
no expiry therefore took the first event's, so a replica (or any table fed
by a subscribing source) later hid and evicted a row the origin kept.

A source-applied write (options.isNotification) now takes its expiry only
from its own event, falling back to its @expiresAt field and the table
default as before.

Closes #4390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…test comment

Refs #4390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the no-expiry assertions

An ordinary write with no expiry now reads only what it read before.

Refs #4390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he read-hiding test a wider window

A null options.expiresAt collapsed to undefined through ?? before; keep that,
since an audit-only commit would otherwise encode it as expiry 0.

Refs #4390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A replicated write with no expiry takes the expiry of the first write in its transaction, so the replica later hides and evicts the row

1 participant