Skip to content

After an online restore_backup, a component's table class answers 503 DATABASE_CLOSING until Harper restarts #4395

Open

Description

@cb1kenobi

Problem

After an online restore_backup of a database, a component that took a table class from databases when its module loaded answers every request that uses the class with 503 DATABASE_CLOSING. It keeps doing so until Harper is restarted. This happens whether the restore succeeds or is refused.

Taking the table class when the module loads is the usual way to write a component:

const { RestoreRestartItem } = databases.restore_on_restart;

Harper's own REST and Operations API paths are fine. They look the table up again after the reload, so they serve the restored data while the component is down.

Reproduction

The fixture is integrationTests/database/restore-on-restart/, installed as a component with no other changes.

  1. create_backup of restore_on_restart.
  2. restore_backup database=restore_on_restart backup_id=<id> with Harper running. The job finishes COMPLETE.
  3. Call the component (POST /RestoreRestartBlobs/ with {"action":"snapshot"}). Every call returns 503, DATABASE_CLOSING, until harper stop; harper start.

The same happens when the online restore is refused because a component holds its own RocksDB handle. The job ends in ERROR with … held open by a loaded component … Pass on_restart=true …, and the component answers 503 afterwards. That was reproduced four times over 10 seconds, with the Harper PID unchanged.

Stack from hdb.log:

DatabaseClosingError … at RestoreRestartItem.closedResource (dist/resources/Table.js:1198)
  at RestoreRestartItem.applyContext [as search] (dist/resources/Resource.js:721)
  at RestoreRestartBlobs.post (…/components/restore-on-restart/resources.js:29)

A variant of the fixture that looks up databases.restore_on_restart.RestoreRestartItem on each request serves the restored data (HTTP 200) after both a successful and a refused online restore. That isolates the cause to the class the component kept from load time.

Cause

An online restore closes the database on every thread before it applies anything (dataLayer/rocksdbBackup.ts, online.close(), broadcasting the 'close' schema event). Each thread runs closeDatabase → closeDatabaseGraph, which calls table.cleanup({ closing: true }) on every table (resources/databases.ts). That sets the class's getResource to closedResource (resources/Table.ts), which throws DatabaseClosingError.

When the restore publishes, or releases its fence after a refusal, the 'reload' broadcast runs resetDatabases(). That registers a new table class under databases.<db>. Nothing redirects the old, disposed class that a component is still holding, so the component keeps calling a closed class.

The disposal (cleanup({ closing: true }) / closedResource) arrived with #3920. The close-then-verify order of an online restore is older, from the original managed-backup operations. Before #3920, closeDatabase closed the table's primary store and unregistered the database without disposing the class. How a captured class behaved after a reload then has not been checked. A component could already have failed then too, in a different way.

restore_backup on_restart=true (#3987) isn't affected, because the restart loads every component again.

Versions

Coverage gap

integrationTests/database/restore-backup-online.test.ts reads through REST, and restore-on-restart.test.ts only exercises on_restart. No test reads through a component's captured class after an online restore.

Possible fixes

  • Redirect a disposed class's static entry points to the class the reload registers for the same table, so a captured class keeps working.
  • Or restart Harper after any online in-place restore of a database that a loaded component references, as a successful online takeover already does when it lifts restore safe mode.
  • Documenting that components must look up tables lazily would only push the problem onto every component author.

Workaround

Run harper stop; harper start after an online restore, or use restore_backup on_restart=true.

🤖 Generated by Claude Opus 5.5; posted via @cb1kenobi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Fields

    Priority

    P1

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions