Repository navigation
Description
Summary
When one transaction writes a row to a table with an expiration and then a row to a table without one, the receiving node stores the second row with the first row's expiry. Once that time passes, the receiver hides the row from reads (observed), and its cleanup scan can later delete it (read from code: on a table with no expiration that scan is best-effort and runs about daily). The originating node keeps the row. Nothing reports the divergence. The only signal is the warning A per-record expiresAt was set on table "<table>" … on the receiver, logged at most once per table per worker thread. It is logged only when the table has no expiration, no @expiresAt attribute, and no source. A table that has one of those can be affected with no warning at all (read from code, untested).
- Verified: reproduced on harper-pro v5.3.6 and v5.4.0 with a two-node cluster (repro below).
- Verified: the same code is on core
mainat 4f20ed583. - Severity: silent loss of rows on replicas, in shipped versions. The trigger needs an expiring write earlier in the same origin transaction.
Expected vs actual
| Row in one transaction | Origin | Replica, after the first row's expiry |
|---|---|---|
1. Exp row (table expiration: 10) |
expires (expected) | expires (expected) |
2. Plain row (no expiration) |
kept | hidden from reads; later deleted by the cleanup scan, per the code (expected: kept) |
Cause
All line links are at the v5.4.0 release commit c5d82cff7. main has the same code. v5.3.6 (eac08b16a) has the same context handling and fallback, with the expiry check written inline instead of in isHiddenExpiry.
- The replica's apply loop uses the transaction's first event as the transaction context:
transaction(event, …)andtxnInProgress = stream.txn = event. Later events in the transaction are staged withstageWrite(event, txnInProgress). - Each write passes
expiresAt: event.expiresAt, with the comment "use per-event expiresAt: batched txn context only holds the first event's expiration". For a row with no expiry,event.expiresAtisundefined(the sender's audit entry carries no expiration). _writeUpdateresolvesoptions?.expiresAt ?? context?.expiresAt.undefinedfalls through to the context, which is the first event, so the row is stored with the first event's expiry.- After that time,
isHiddenExpiryhides the row on a table without a source, and the cleanup scan the write path arms (scheduleCleanup()) evicts it.
Possible fix
Stop a write in this apply loop from taking another event's expiry through the shared context. Untested. Two options, each with a catch:
| Option | Catch |
|---|---|
expiresAt: event.expiresAt ?? -1 at Table.ts#L2984 |
The same loop also applies source.subscribe feeds. With -1, _writeUpdate skips its table-default and @expiresAt fallback (L7830–L7838), so a subscription-fed row on a table with an expiration would never expire. |
Leave options.expiresAt undefined, but skip the context?.expiresAt fallback at L6827 for writes staged by this loop |
The loop needs a flag on the write's options to say so, and any code that relies on the batch context's expiry would change. |
Repro
A harper-pro cluster integration test (integrationTests/cluster/, same harness as addNodeTableExpiration.test.mjs): two nodes, the component below deployed on both, B joined to A with add_node.
schema.graphql:
type Exp @table(expiration: 10) @export {
id: ID @primaryKey
v: Int
}
type Plain @table @export {
id: ID @primaryKey
v: Int
}resources.js: one request is one transaction.
export class WriteExpFirst extends Resource {
static loadAsInstance = false;
async get(target) {
target.checkPermission = false;
const id = String(target.id);
await tables.Exp.put(id, { id, v: 1 });
await tables.Plain.put(id, { id, v: 1 });
return { wrote: id };
}
}
// WritePlainFirst: the same two puts in the opposite order (control).Steps:
GET /WriteExpFirst/x1andGET /WritePlainFirst/y1on A.- Wait until
Plainx1andy1are readable on B (search_by_id). - Wait 25 seconds (more than the 10-second
Expexpiration). - Read
Plainx1andy1on both nodes.
Results (observed: the first case failed the same way in three runs on v5.4.0; one run on v5.3.6):
| Case | v5.3.6 | v5.4.0 |
|---|---|---|
Plain x1 (written after Exp), on A |
present | present |
Plain x1 (written after Exp), on B |
missing | missing |
Plain y1 (written before Exp), on B |
present | present |
20 pairs of Exp then Plain as separate upsert operations, on B |
20 of 20 present | 20 of 20 present |
Plain.create() or Plain.patch(), then Exp.put(), in one transaction, on B |
present | present |
B logged A per-record expiresAt was set on table "Plain" … in both versions.
Open items
- Untested: several transactions from the same origin arriving in one replication message. On RocksDB the receiver starts a new transaction only per message or when the origin changes (replicationConnection.ts#L11643), so they would share the first event's context. The 20-pair case above was not confirmed to have been batched into one message.
- Untested: LMDB, where a change of origin does not start a new receiver transaction.
- Untested: relayed writes (A → B → C) and the base-copy path.
- Untested: tables with an
@expiresAtattribute or eviction config, which can be affected without the warning (see Summary). - Not this bug: the same warning also fires for
system.hdb_session, on nodes that save sessions with an expiry.security/auth.tssaves them withgetSessionTable().put(session, { expiresAt })on a table declared without an expiration. Those rows are meant to expire.
🤖 Generated by Claude Code (Anthropic Claude Opus); posted via @DavidCockerill.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity