You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
@expiresAt per-record TTL never evicts on RocksDB (default engine) — index.getValues() is LMDB-only #1481
A table with an @expiresAt field (per-record absolute-expiry timestamp — the session/token/cache pattern) never evicts expired records on RocksDB (the default engine). The per-record eviction sweep throws on every pass and is silently swallowed, so records past their expiresAt accumulate forever and remain live + readable indefinitely. Works correctly on LMDB.
Mechanism
resources/Table.ts:5293runRecordExpirationEviction() calls index.getValues(key) — an LMDB-specific dupsort API that does not exist on RocksDB index stores. On RocksDB every ~60s sweep throws TypeError: index.getValues is not a function; the error is caught + logged but otherwise swallowed, so the sweep makes no progress.
There is no lazy-on-read fallback for an @expiresAt-only table: the entry-metadata expiresAt is -1 (no @table(expiration:)), and resources/RecordEncoder.ts:708-709 only sets HAS_EXPIRATION when expiresAt >= 0. So on RocksDB the records are neither swept nor lazily expired.
Impact
Storage leak:@expiresAt records grow unbounded on RocksDB.
Correctness / security-adjacent: a "session"/"token" table using @expiresAt keeps serving records that should have expired — an expired token never expires.
Affects the default engine, so it hits most deployments.
Fix
Replace the LMDB-only index.getValues(key) with a RocksDB-compatible index.getRange(...) iteration that reads entry.value as the primary key (the RocksDB index model), so the per-record sweep works on both engines.
Secondary footgun (both engines)
A full-record PUT that omits the expiresAt field clears it (the record then never expires). A PATCH of a non-expiry field correctly preserves it. (And @table(expiration:N) + @expiresAt together → the table-level TTL wins.) Worth documenting that a session-renewal-via-PUT must re-include expiresAt.
Reproduction
cd /home/kzyp/dev/harper # main @ 28db4fde4 (v5.1.12)
npm run build
npm run test:integration -- "integrationTests/qa-scratch/qa365-expiresat.test.ts" # RocksDB → broken (records never expire)
HARPER_STORAGE_ENGINE=lmdb npm run test:integration -- "integrationTests/qa-scratch/qa365-expiresat.test.ts" # LMDB → works
Found by the qa-explorer exploratory campaign. — filed by KrAIs (Claude) on Kris's behalf
Related: #1810 tracks the record-@expiresAt-not-authoritative-over-table-default half (the "secondary footgun" noted here), with a fix that feeds the field into the write-time expiry metadata. Because read-hiding and the getRange-based cleanup sweep are engine-agnostic, that fix also enforces field-derived expiry on RocksDB for new writes — so it addresses the correctness half of this issue without relying on the LMDB-only pruning sweep. This issue stays open for the pruning-sweep index.getValues→getRange fix (covers pre-existing records + removes the per-pass RocksDB throw).
Still present on 5.3.0 (harper-pro 5.3.0, rocksdb-js 2.10.0), and it bites a production customer.
Observed today on an 8-node RocksDB cluster (every node, before and after a 5.2.6 -> 5.3.0 upgrade): the expiration timer fires once a minute per worker and throws every time:
[http/N] [error]: Error in evicting old records TypeError: index.getValues is not a function or its return value is not iterable
at Timeout._onTimeout (.../@harperfast/harper-pro/core/resources/Table.ts:9610:30)
resources/Table.ts at v5.3.0 line 9610 is still for (const id of index.getValues(key)), the LMDB-only call this issue describes.
Impact: the customer's schema declares expiration on several tables (7200s, 86400s, 604800s) and none of it takes effect; the tables only shrink through explicit deletes. Nothing in the API or logs other than this error tells them their TTLs are inert.
Worth bumping the priority: it has been the top error line by volume on that cluster for a month, and it hides any real eviction problem behind the noise.
Activity
kriszyp commentedon Jul 14, 2026
Related: #1810 tracks the record-
@expiresAt-not-authoritative-over-table-default half (the "secondary footgun" noted here), with a fix that feeds the field into the write-time expiry metadata. Because read-hiding and thegetRange-based cleanup sweep are engine-agnostic, that fix also enforces field-derived expiry on RocksDB for new writes — so it addresses the correctness half of this issue without relying on the LMDB-only pruning sweep. This issue stays open for the pruning-sweepindex.getValues→getRangefix (covers pre-existing records + removes the per-pass RocksDB throw).Make @expiresAt attribute authoritative over the table expiration def…
Make @expiresAt attribute authoritative over the table expiration def…
Make @expiresAt attribute authoritative over the table expiration def…
maurice-harper commentedon Oct 1, 2026
Still present on 5.3.0 (harper-pro 5.3.0, rocksdb-js 2.10.0), and it bites a production customer.
Observed today on an 8-node RocksDB cluster (every node, before and after a 5.2.6 -> 5.3.0 upgrade): the expiration timer fires once a minute per worker and throws every time:
resources/Table.tsat v5.3.0 line 9610 is stillfor (const id of index.getValues(key)), the LMDB-only call this issue describes.Impact: the customer's schema declares
expirationon several tables (7200s, 86400s, 604800s) and none of it takes effect; the tables only shrink through explicit deletes. Nothing in the API or logs other than this error tells them their TTLs are inert.Worth bumping the priority: it has been the top error line by volume on that cluster for a month, and it hides any real eviction problem behind the noise.