Repository navigation
Description
Part of #2632 (item 5) — the create-if-absent PR. Design note §6 and §5.7 in docs/proposals/archive-restore.md on branch design/archive-restore. Depends on #2634, #2636, #2640, and either #2451 or minting a generation here.
Problem
target_database is refused while Harper runs (dataLayer/rocksdbBackup.ts:500, :537) by policy, not because the runtime cannot publish a new name — create_database → database() → signalSchemaChange does that today (dataLayer/harperBridge/ResourceBridge.ts createSchema, resources/databases.ts:1897-1935). Restore-into-a-copy is the safest disaster-recovery move (inspect the copy, then swap or copy across), and on Fabric it is currently unreachable.
Simply lifting the refusal is not enough, because today's publication is weak in three ways:
- The restore marker is a check, not an exclusion.
database()callsthrowIfBlockedByRestoreand then opens without holding the lock (resources/databases.ts:1932-1937), so an opener that passes the check, pauses, and resumes after the marker lands opens the tree being restored; the startup scan snapshots its blocked set once and then opens each directory (:577-608); andrestoreBackupOfflinechecks the target is absent before it reserves (dataLayer/rocksdbBackup.ts:1008-1020), so a concurrentcreate_databasecan lose the database it just made. - The current online
restoreBackupclears its marker and lets the ordinary reload publish the directory (:605-611) — no private open, no awaited replay, no check that the tables the backup should contain actually opened. "No exception thrown" is the False-green health surfaces: eight instances in two weeks share one design gap #2095 false-green shape. initStoresrecreates a missing column family as an empty store when the catalogue still names it (resources/databases.ts:1091-1096), so "opened successfully" cannot prove a family existed.
What to build: one publication protocol
Used by online restore into a new name, by in-place online restore of an unheld database, and (item 6) by the restart route.
- Reservation first.
beginRestore(targetDir)before the first byte lands; the target is re-checked for absence (or, in place, for closure) inside the reservation. On-demand opens and the scan's per-directory open take the per-database restore lock in shared mode (tryFileLock(file, true)) across check-and-open, releasing once the handle is registered; restore holds it exclusive.dropDatabasekeeps serializing on the same lock. All of this is on cold open paths — no per-request work. - Apply without finishing. Split
restoreBackupOfflineinto an apply helper (lock, marker, purge, engine restore, blob restore) that does not callcompleteRestore, so the marker survives a failed validation; the CLI keeps its current shape by calling apply and then completing. - Inventory, then private open. Read the physical column-family inventory from the restored directory's
OPTIONS-*file and check every family the catalogue requires is present, before any create-capable open. TheninitStores(path, root, db, { destination, openedStores })and check the expected table set is a subset of what enumerated and every store opened. - Restore-strict elected replay, in the job worker (never on a serving thread —
replayLogsruns synchronously behind its Promise,resources/replayLogs.ts:60-136, and athreads: 0install serves from main).await replayLogs(root, destination, { elected: true, policy: 'restore' }): elected mode already rejects a failing tail but still tolerates undecodable entries (skipped++,:183-201), so the restore policy rejects whenskipped > 0ordiscardedWrites > 0, and gatespurgeAgedLogsoff (it otherwise deletes a restored log older thanlogging.auditRetentionbefore replay,:83). - Blob completeness. A blob reclaimed before its directory was enumerated is absent from the source with no marker (
dataLayer/blobBackup.ts:30-40). Replacement restores run the O(data) blob-reference scan unless the request carriedaccept_unverified_blobs: true; the result recordsblob_integrity: verified | unverified | failed.target_databasereports it and does not require it. - Peer fence and re-check. Core exposes "a restore reservation is held for X";
add_node/set_node/add_node_backand routes-config changes refuse while it is; peers are re-checked after replay and beforecompleteRestore. - Durability barriers. Close private handles and surface close errors; fsync restored blob files and their directories; fsync manifests. Today's copies do not sync (
dataLayer/blobBackup.ts:59-80,:192-211;dataLayer/backupManifest.ts:47-57). State the Windows directory-fsync limit rather than hide it. - Finalize:
completeRestore, then the rescan that publishes. Mint or consume the generation (restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451) between steps 2 and 8.
Then lift the target_database refusal when the target directory is absent; the offline form's "never purge an existing database of that name" (:1008-1013) applies unchanged. Engine-only backups follow the single rule in #2640.
Acceptance
restore_backup database=X target_database=Yonline, with blobs, records and blobs readable afterwards; roles are not created (a new name has no grants — expected).- An opener paused between
throwIfBlockedByRestoreand its open while a restore reserves the name is held off; a rescan and acreate_tableracing an online restore into a new name lose to the reservation, not the other way round. - A backup whose catalogue names a dropped column family fails validation; a log with valid framing and an undecodable value is rejected before publication; a blob missing without a marker is caught by the scan and reported under the waiver.
- A peer or route added after online admission is refused while the name is reserved, and caught by the pre-publication re-check if it slips through.
- Unrelated request latency measured during publication and replay, including on a
threads: 0install.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
3 remaining items
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
Make the restore marker an exclusion, not a check before an open
cb1kenobi commentedon Nov 14, 2026
Fixed by #3920 Publish every restore_backup through one crash-safe protocol, merged 2026-11-05 for v5.4. Every
restore_backupnow goes through one publication protocol: a receipt, a separate restore process that applies the backup and proves it, then the publication. Online restore into a new database withtarget_databaseworks while Harper runs.🤖 Generated by Claude Opus 5.5; posted via @cb1kenobi.