Skip to content

@expiresAt per-record TTL never evicts on RocksDB (default engine) — index.getValues() is LMDB-only #1481

Description

@kriszyp

Summary

A table with an @expiresAt field (per-record absolute-expiry timestamp — the session/token/cache pattern) never evicts expired records on RocksDB (the default engine). The per-record eviction sweep throws on every pass and is silently swallowed, so records past their expiresAt accumulate forever and remain live + readable indefinitely. Works correctly on LMDB.

Mechanism

resources/Table.ts:5293 runRecordExpirationEviction() calls index.getValues(key) — an LMDB-specific dupsort API that does not exist on RocksDB index stores. On RocksDB every ~60s sweep throws TypeError: index.getValues is not a function; the error is caught + logged but otherwise swallowed, so the sweep makes no progress.

There is no lazy-on-read fallback for an @expiresAt-only table: the entry-metadata expiresAt is -1 (no @table(expiration:)), and resources/RecordEncoder.ts:708-709 only sets HAS_EXPIRATION when expiresAt >= 0. So on RocksDB the records are neither swept nor lazily expired.

Impact

  • Storage leak: @expiresAt records grow unbounded on RocksDB.
  • Correctness / security-adjacent: a "session"/"token" table using @expiresAt keeps serving records that should have expired — an expired token never expires.
  • Affects the default engine, so it hits most deployments.

Fix

Replace the LMDB-only index.getValues(key) with a RocksDB-compatible index.getRange(...) iteration that reads entry.value as the primary key (the RocksDB index model), so the per-record sweep works on both engines.

Secondary footgun (both engines)

A full-record PUT that omits the expiresAt field clears it (the record then never expires). A PATCH of a non-expiry field correctly preserves it. (And @table(expiration:N) + @expiresAt together → the table-level TTL wins.) Worth documenting that a session-renewal-via-PUT must re-include expiresAt.

Reproduction

cd /home/kzyp/dev/harper   # main @ 28db4fde4 (v5.1.12)
npm run build
npm run test:integration -- "integrationTests/qa-scratch/qa365-expiresat.test.ts"          # RocksDB → broken (records never expire)
HARPER_STORAGE_ENGINE=lmdb npm run test:integration -- "integrationTests/qa-scratch/qa365-expiresat.test.ts"   # LMDB → works

Found by the qa-explorer exploratory campaign. — filed by KrAIs (Claude) on Kris's behalf

Activity

  1. kriszyp commented on Jul 14, 2026

    @kriszyp
    MemberAuthor

    Related: #1810 tracks the record-@expiresAt-not-authoritative-over-table-default half (the "secondary footgun" noted here), with a fix that feeds the field into the write-time expiry metadata. Because read-hiding and the getRange-based cleanup sweep are engine-agnostic, that fix also enforces field-derived expiry on RocksDB for new writes — so it addresses the correctness half of this issue without relying on the LMDB-only pruning sweep. This issue stays open for the pruning-sweep index.getValues→getRange fix (covers pre-existing records + removes the per-pass RocksDB throw).

  2. added this to the v5.2 milestone on Aug 7, 2026
  3. added theissue type on Aug 7, 2026
  4. maurice-harper commented on Oct 1, 2026

    @maurice-harper
    Contributor

    Still present on 5.3.0 (harper-pro 5.3.0, rocksdb-js 2.10.0), and it bites a production customer.

    Observed today on an 8-node RocksDB cluster (every node, before and after a 5.2.6 -> 5.3.0 upgrade): the expiration timer fires once a minute per worker and throws every time:

    [http/N] [error]: Error in evicting old records TypeError: index.getValues is not a function or its return value is not iterable
        at Timeout._onTimeout (.../@harperfast/harper-pro/core/resources/Table.ts:9610:30)
    

    resources/Table.ts at v5.3.0 line 9610 is still for (const id of index.getValues(key)), the LMDB-only call this issue describes.

    Impact: the customer's schema declares expiration on several tables (7200s, 86400s, 604800s) and none of it takes effect; the tables only shrink through explicit deletes. Nothing in the API or logs other than this error tells them their TTLs are inert.

    Worth bumping the priority: it has been the top error line by volume on that cluster for a month, and it hides any real eviction problem behind the noise.

  5. modified the milestones: v5.2, v5.4 on Oct 22, 2026
  6. assigned and unassigned on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P1

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions