You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cleanup scans of loaded tables: none under threads: 0 until a write, and only daily on threads that load the table #2831
Two gaps in how resources/Table.ts arms the cleanup scan (the scheduleCleanup timer that removes rows past their metadata expiry) of a table loaded from its catalog. Reads are unaffected, since they already treat a row past its expiry as absent, but the rows stay on disk.
1. Under threads: 0, no scan until the first write after startup
bin/run.tsinitialize() loads, and may declare, tables on the main thread before startHTTPThreads calls setMainIsWorker(true) (server/threads/socketRouter.ts). #2823 changed scheduleCleanup to record lastCleanupInterval only once its thread has a worker index, so the first expiring write or declaration after promotion arms the scan, as the first expiring write already did on main. Nothing re-evaluates the tables loaded before promotion, though: a single-threaded node that writes nothing to such a table after a restart never scans it.
A fix would re-evaluate every loaded table's scan once the main thread becomes worker 0, for example through a databases.ts helper that the threads: 0 path calls right after setMainIsWorker(true). It has to include databases.system explicitly: ensureDB() defines it with Object.defineProperty and no enumerable, so Object.values(databases) skips it, and system tables are the ones that rely on a load-armed scan (system.hdb_oidc_token_use since #2823, and the certificate-verification tables moving to a table-level expiration).
2. A thread that only loads a table scans daily, whatever its expiration
A table loaded from its catalog calls setTTLExpiration(expirationMs / 1000) (in makeTable, around ttlFromLoad = true). The number form keeps the initial cleanupInterval = 86400000, so a thread that loads rather than declares the table scans once a day regardless of expiration; only a declaring thread gets (expiration + eviction) / 4. The catalog does not store scanInterval. Decide whether this is intentional before changing it.
Related: #1339 (per-record runtime expiresAt is not reliably evicted without a table-level expiration). dataLayer/DESIGN.md "System table bootstrap" documents both limits.
After #3938, leg 2 is fixed: a catalog-loaded table now scans every scanInterval, else (expiration + eviction) / 4.
Leg 1 now shows in query results. Under threads: 0, after a restart with no write, search, REST list queries and SQL return a runtime TTL table's expired rows indefinitely (QA: all 200 rows still returned 41 s after expiry, on RocksDB and LMDB; on the base commit they were hidden). get and search_by_id still hide them, so "Reads are unaffected" no longer holds for queries. Worth re-checking the P3 if TTL system tables are read through query paths.
The catalog still doesn't store scanInterval. That's accepted: only a declaration can set it, and the declaration runs again at startup.
🤖 Generated by Claude Opus 5.5 (Claude Code); posted via @kriszyp.
Activity
kriszyp commentedon Nov 4, 2026
After #3938, leg 2 is fixed: a catalog-loaded table now scans every
scanInterval, else (expiration + eviction) / 4.Leg 1 now shows in query results. Under
threads: 0, after a restart with no write,search, REST list queries and SQL return a runtime TTL table's expired rows indefinitely (QA: all 200 rows still returned 41 s after expiry, on RocksDB and LMDB; on the base commit they were hidden).getandsearch_by_idstill hide them, so "Reads are unaffected" no longer holds for queries. Worth re-checking the P3 if TTL system tables are read through query paths.The catalog still doesn't store
scanInterval. That's accepted: only a declaration can set it, and the declaration runs again at startup.🤖 Generated by Claude Opus 5.5 (Claude Code); posted via @kriszyp.