Repository navigation
Description
Problem
After an online restore_backup of a database, a component that took a table class from databases when its module loaded answers every request that uses the class with 503 DATABASE_CLOSING. It keeps doing so until Harper is restarted. This happens whether the restore succeeds or is refused.
Taking the table class when the module loads is the usual way to write a component:
const { RestoreRestartItem } = databases.restore_on_restart;Harper's own REST and Operations API paths are fine. They look the table up again after the reload, so they serve the restored data while the component is down.
Reproduction
The fixture is integrationTests/database/restore-on-restart/, installed as a component with no other changes.
create_backupofrestore_on_restart.restore_backup database=restore_on_restart backup_id=<id>with Harper running. The job finishesCOMPLETE.- Call the component (
POST /RestoreRestartBlobs/with{"action":"snapshot"}). Every call returns503,DATABASE_CLOSING, untilharper stop; harper start.
The same happens when the online restore is refused because a component holds its own RocksDB handle. The job ends in ERROR with … held open by a loaded component … Pass on_restart=true …, and the component answers 503 afterwards. That was reproduced four times over 10 seconds, with the Harper PID unchanged.
Stack from hdb.log:
DatabaseClosingError … at RestoreRestartItem.closedResource (dist/resources/Table.js:1198)
at RestoreRestartItem.applyContext [as search] (dist/resources/Resource.js:721)
at RestoreRestartBlobs.post (…/components/restore-on-restart/resources.js:29)
A variant of the fixture that looks up databases.restore_on_restart.RestoreRestartItem on each request serves the restored data (HTTP 200) after both a successful and a refused online restore. That isolates the cause to the class the component kept from load time.
Cause
An online restore closes the database on every thread before it applies anything (dataLayer/rocksdbBackup.ts, online.close(), broadcasting the 'close' schema event). Each thread runs closeDatabase → closeDatabaseGraph, which calls table.cleanup({ closing: true }) on every table (resources/databases.ts). That sets the class's getResource to closedResource (resources/Table.ts), which throws DatabaseClosingError.
When the restore publishes, or releases its fence after a refusal, the 'reload' broadcast runs resetDatabases(). That registers a new table class under databases.<db>. Nothing redirects the old, disposed class that a component is still holding, so the component keeps calling a closed class.
The disposal (cleanup({ closing: true }) / closedResource) arrived with #3920. The close-then-verify order of an online restore is older, from the original managed-backup operations. Before #3920, closeDatabase closed the table's primary store and unregistered the database without disposing the class. How a captured class behaved after a reload then has not been checked. A component could already have failed then too, in a different way.
restore_backup on_restart=true (#3987) isn't affected, because the restart loads every component again.
Versions
- Affected: v5.4.0. Every online in-place
restore_backupof a database a loaded component uses. - Not affected: v5.3.6, which doesn't contain Publish every restore_backup through one crash-safe protocol: receipts, a proving restore process, and a membership fence #3920.
Coverage gap
integrationTests/database/restore-backup-online.test.ts reads through REST, and restore-on-restart.test.ts only exercises on_restart. No test reads through a component's captured class after an online restore.
Possible fixes
- Redirect a disposed class's static entry points to the class the reload registers for the same table, so a captured class keeps working.
- Or restart Harper after any online in-place restore of a database that a loaded component references, as a successful online takeover already does when it lifts restore safe mode.
- Documenting that components must look up tables lazily would only push the problem onto every component author.
Workaround
Run harper stop; harper start after an online restore, or use restore_backup on_restart=true.
🤖 Generated by Claude Opus 5.5; posted via @cb1kenobi.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity