Skip to content

A replicated write with no expiry takes the expiry of the first write in its transaction, so the replica later hides and evicts the row #4390

Closed

Description

@DavidCockerill

Summary

When one transaction writes a row to a table with an expiration and then a row to a table without one, the receiving node stores the second row with the first row's expiry. Once that time passes, the receiver hides the row from reads (observed), and its cleanup scan can later delete it (read from code: on a table with no expiration that scan is best-effort and runs about daily). The originating node keeps the row. Nothing reports the divergence. The only signal is the warning A per-record expiresAt was set on table "<table>" … on the receiver, logged at most once per table per worker thread. It is logged only when the table has no expiration, no @expiresAt attribute, and no source. A table that has one of those can be affected with no warning at all (read from code, untested).

  • Verified: reproduced on harper-pro v5.3.6 and v5.4.0 with a two-node cluster (repro below).
  • Verified: the same code is on core main at 4f20ed583.
  • Severity: silent loss of rows on replicas, in shipped versions. The trigger needs an expiring write earlier in the same origin transaction.

Expected vs actual

Row in one transaction Origin Replica, after the first row's expiry
1. Exp row (table expiration: 10) expires (expected) expires (expected)
2. Plain row (no expiration) kept hidden from reads; later deleted by the cleanup scan, per the code (expected: kept)

Cause

All line links are at the v5.4.0 release commit c5d82cff7. main has the same code. v5.3.6 (eac08b16a) has the same context handling and fallback, with the expiry check written inline instead of in isHiddenExpiry.

  1. The replica's apply loop uses the transaction's first event as the transaction context: transaction(event, …) and txnInProgress = stream.txn = event. Later events in the transaction are staged with stageWrite(event, txnInProgress).
  2. Each write passes expiresAt: event.expiresAt, with the comment "use per-event expiresAt: batched txn context only holds the first event's expiration". For a row with no expiry, event.expiresAt is undefined (the sender's audit entry carries no expiration).
  3. _writeUpdate resolves options?.expiresAt ?? context?.expiresAt. undefined falls through to the context, which is the first event, so the row is stored with the first event's expiry.
  4. After that time, isHiddenExpiry hides the row on a table without a source, and the cleanup scan the write path arms (scheduleCleanup()) evicts it.

Possible fix

Stop a write in this apply loop from taking another event's expiry through the shared context. Untested. Two options, each with a catch:

Option Catch
expiresAt: event.expiresAt ?? -1 at Table.ts#L2984 The same loop also applies source.subscribe feeds. With -1, _writeUpdate skips its table-default and @expiresAt fallback (L7830–L7838), so a subscription-fed row on a table with an expiration would never expire.
Leave options.expiresAt undefined, but skip the context?.expiresAt fallback at L6827 for writes staged by this loop The loop needs a flag on the write's options to say so, and any code that relies on the batch context's expiry would change.

Repro

A harper-pro cluster integration test (integrationTests/cluster/, same harness as addNodeTableExpiration.test.mjs): two nodes, the component below deployed on both, B joined to A with add_node.

schema.graphql:

type Exp @table(expiration: 10) @export {
	id: ID @primaryKey
	v: Int
}

type Plain @table @export {
	id: ID @primaryKey
	v: Int
}

resources.js: one request is one transaction.

export class WriteExpFirst extends Resource {
	static loadAsInstance = false;
	async get(target) {
		target.checkPermission = false;
		const id = String(target.id);
		await tables.Exp.put(id, { id, v: 1 });
		await tables.Plain.put(id, { id, v: 1 });
		return { wrote: id };
	}
}
// WritePlainFirst: the same two puts in the opposite order (control).

Steps:

  1. GET /WriteExpFirst/x1 and GET /WritePlainFirst/y1 on A.
  2. Wait until Plain x1 and y1 are readable on B (search_by_id).
  3. Wait 25 seconds (more than the 10-second Exp expiration).
  4. Read Plain x1 and y1 on both nodes.

Results (observed: the first case failed the same way in three runs on v5.4.0; one run on v5.3.6):

Case v5.3.6 v5.4.0
Plain x1 (written after Exp), on A present present
Plain x1 (written after Exp), on B missing missing
Plain y1 (written before Exp), on B present present
20 pairs of Exp then Plain as separate upsert operations, on B 20 of 20 present 20 of 20 present
Plain.create() or Plain.patch(), then Exp.put(), in one transaction, on B present present

B logged A per-record expiresAt was set on table "Plain" … in both versions.

Open items

  • Untested: several transactions from the same origin arriving in one replication message. On RocksDB the receiver starts a new transaction only per message or when the origin changes (replicationConnection.ts#L11643), so they would share the first event's context. The 20-pair case above was not confirmed to have been batched into one message.
  • Untested: LMDB, where a change of origin does not start a new receiver transaction.
  • Untested: relayed writes (A → B → C) and the base-copy path.
  • Untested: tables with an @expiresAt attribute or eviction config, which can be affected without the warning (see Summary).
  • Not this bug: the same warning also fires for system.hdb_session, on nodes that save sessions with an expiry. security/auth.ts saves them with getSessionTable().put(session, { expiresAt }) on a table declared without an expiration. Those rows are meant to expire.

🤖 Generated by Claude Code (Anthropic Claude Opus); posted via @DavidCockerill.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Fields

Priority

P1

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions