Skip to content

Table background work (expiration timer, delete/cleanup callbacks) not disposed when a database's stores are closed #1811

Closed

Description

@cb1kenobi

Summary

When a database's stores are closed and later reloaded, per-table background work is not torn down. This is pre-existing (it predates the RocksDB backup/restore work) and was surfaced by a cross-model review of that change.

Details

Two code paths close and reload a loaded database in place:

  • resetDatabases() in resources/databases.ts — invoked on every schema change via the ITC schema handler (server/itc/serverHandlers.js).
  • closeDatabase() / closeLoadedDatabases() in resources/databases.ts — the restore_backup flow and job-worker exit.

Neither disposes table-level background work before/after closing the stores. In particular:

  • The record-expiration eviction timer — setInterval(...) in runRecordExpirationEviction (resources/Table.ts, ~line 5537, thread 0 only) — is not stored in a handle and never cleared.
  • Delete-removal / cleanup callbacks registered against the audit store (addDeleteRemoval → auditStore.addDeleteRemovalCallback, resources/Table.ts ~line 5528) are not deregistered on close.

Impact

After a close, these timers/callbacks keep referencing the now-closed stores (erroring on the next tick, or silently retaining them); a subsequent reload can register a second set. Bounded, but real — most likely to accumulate on instances with frequent schema changes and tables that use expiration/eviction.

Proposed fix

Add an idempotent table-disposal method that clears the table's timers/intervals and deregisters its delete/cleanup callbacks, and invoke it before closing the stores. Wire it into both resetDatabases() and closeDatabase() so both close paths are covered.

References

  • resources/Table.ts — runRecordExpirationEviction (expiration interval), addDeleteRemoval (delete-removal callback)
  • resources/databases.ts — closeDatabase, closeLoadedDatabases, resetDatabases

Surfaced via cross-model review (codex) of the RocksDB backup/restore change; filed by Claude.

Activity

  1. added this to the v5.3 milestone on Aug 7, 2026
  2. added theissue type on Aug 7, 2026
  3. kriszyp commented on Sep 25, 2026

    @kriszyp
    Member

    Confirmed still fully open on current main: Table.cleanup() (resources/Table.ts:7105) exists
    and is wired into the schema-recreate and discard-unpublished-table paths
    (resources/databases.ts:1451, :3730, :3751), but closeDatabase()/closeLoadedDatabases()
    (resources/databases.ts:2223/:2318 — the restore and job-worker-exit paths this issue names)
    still don't call it.

    PR #2721 (closed, superseded by #2766 for the LMDB segfault this repo just fixed) prototyped
    wiring Table.cleanup() into that close path, and hit two things worth flagging for whoever
    implements this issue's proposed fix:

    • HNSW alias-backend strand. Table.cleanup() settles/deletes a table's native-HNSW backend
      unconditionally. If it's ever called per-alias (one name of a multi-alias table closing while a
      sibling alias of the same table stays loaded) rather than for the whole database at once, the
      surviving alias's writes silently stop reaching the index — no error, just drift. Not reachable
      today (both in-tree close paths close a whole database's tables together), but a real
      constraint on the fix: don't retire a table's HNSW backend from Table.cleanup() unless every
      alias of that table is going away with it.
    • derivedIndexRuntime.close() is fire-and-forget (resources/Table.ts:7108, .catch only) —
      fine for the general case, but the restore path specifically needs the physical store close to
      wait for it (a job worker's closeLoadedDatabases() can otherwise drop the process-wide RocksDB
      refcount to 0 before an in-flight HNSW flush finishes). Close one database alias without tearing down the store its other aliases share #2721 added an explicit await barrier
      before the physical close in the restore path only, leaving the fire-and-forget behavior
      everywhere else. closeLoadedDatabases() doesn't await it either way today — same gap this
      issue already names for the worker-exit path.

    Reference (unmerged, won't ship): branch fix/alias-close-shared-root-store at 840d14f26,
    resources/databases.ts's closeDatabaseWithAliases and Table.ts's awaited-stop changes.

    See also #2802 (RocksDB alias handles not released together on restore_backup — the
    shared-store-identity half of this problem; different gap, same close paths).

  4. modified the milestones: v5.3, v5.4 on Oct 10, 2026
  5. cb1kenobi commented on Nov 14, 2026

    @cb1kenobi
    MemberAuthor

    Fixed by #3920 Publish every restore_backup through one crash-safe protocol, merged 2026-11-05 for v5.4. Closing a database disposes its tables' expiration timers and cleanup callbacks, and a request made during the close gets a retryable closing error.

    🤖 Generated by Claude Opus 5.5; posted via @cb1kenobi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions