Skip to content

restore_backup: shared publication protocol and online restore into a new database (create-if-absent) #2637

Closed

Description

@cb1kenobi

Part of #2632 (item 5) — the create-if-absent PR. Design note §6 and §5.7 in docs/proposals/archive-restore.md on branch design/archive-restore. Depends on #2634, #2636, #2640, and either #2451 or minting a generation here.

Problem

target_database is refused while Harper runs (dataLayer/rocksdbBackup.ts:500, :537) by policy, not because the runtime cannot publish a new name — create_database → database() → signalSchemaChange does that today (dataLayer/harperBridge/ResourceBridge.ts createSchema, resources/databases.ts:1897-1935). Restore-into-a-copy is the safest disaster-recovery move (inspect the copy, then swap or copy across), and on Fabric it is currently unreachable.

Simply lifting the refusal is not enough, because today's publication is weak in three ways:

  • The restore marker is a check, not an exclusion. database() calls throwIfBlockedByRestore and then opens without holding the lock (resources/databases.ts:1932-1937), so an opener that passes the check, pauses, and resumes after the marker lands opens the tree being restored; the startup scan snapshots its blocked set once and then opens each directory (:577-608); and restoreBackupOffline checks the target is absent before it reserves (dataLayer/rocksdbBackup.ts:1008-1020), so a concurrent create_database can lose the database it just made.
  • The current online restoreBackup clears its marker and lets the ordinary reload publish the directory (:605-611) — no private open, no awaited replay, no check that the tables the backup should contain actually opened. "No exception thrown" is the False-green health surfaces: eight instances in two weeks share one design gap #2095 false-green shape.
  • initStores recreates a missing column family as an empty store when the catalogue still names it (resources/databases.ts:1091-1096), so "opened successfully" cannot prove a family existed.

What to build: one publication protocol

Used by online restore into a new name, by in-place online restore of an unheld database, and (item 6) by the restart route.

  1. Reservation first. beginRestore(targetDir) before the first byte lands; the target is re-checked for absence (or, in place, for closure) inside the reservation. On-demand opens and the scan's per-directory open take the per-database restore lock in shared mode (tryFileLock(file, true)) across check-and-open, releasing once the handle is registered; restore holds it exclusive. dropDatabase keeps serializing on the same lock. All of this is on cold open paths — no per-request work.
  2. Apply without finishing. Split restoreBackupOffline into an apply helper (lock, marker, purge, engine restore, blob restore) that does not call completeRestore, so the marker survives a failed validation; the CLI keeps its current shape by calling apply and then completing.
  3. Inventory, then private open. Read the physical column-family inventory from the restored directory's OPTIONS-* file and check every family the catalogue requires is present, before any create-capable open. Then initStores(path, root, db, { destination, openedStores }) and check the expected table set is a subset of what enumerated and every store opened.
  4. Restore-strict elected replay, in the job worker (never on a serving thread — replayLogs runs synchronously behind its Promise, resources/replayLogs.ts:60-136, and a threads: 0 install serves from main). await replayLogs(root, destination, { elected: true, policy: 'restore' }): elected mode already rejects a failing tail but still tolerates undecodable entries (skipped++, :183-201), so the restore policy rejects when skipped > 0 or discardedWrites > 0, and gates purgeAgedLogs off (it otherwise deletes a restored log older than logging.auditRetention before replay, :83).
  5. Blob completeness. A blob reclaimed before its directory was enumerated is absent from the source with no marker (dataLayer/blobBackup.ts:30-40). Replacement restores run the O(data) blob-reference scan unless the request carried accept_unverified_blobs: true; the result records blob_integrity: verified | unverified | failed. target_database reports it and does not require it.
  6. Peer fence and re-check. Core exposes "a restore reservation is held for X"; add_node/set_node/add_node_back and routes-config changes refuse while it is; peers are re-checked after replay and before completeRestore.
  7. Durability barriers. Close private handles and surface close errors; fsync restored blob files and their directories; fsync manifests. Today's copies do not sync (dataLayer/blobBackup.ts:59-80, :192-211; dataLayer/backupManifest.ts:47-57). State the Windows directory-fsync limit rather than hide it.
  8. Finalize: completeRestore, then the rescan that publishes. Mint or consume the generation (restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451) between steps 2 and 8.

Then lift the target_database refusal when the target directory is absent; the offline form's "never purge an existing database of that name" (:1008-1013) applies unchanged. Engine-only backups follow the single rule in #2640.

Acceptance

  • restore_backup database=X target_database=Y online, with blobs, records and blobs readable afterwards; roles are not created (a new name has no grants — expected).
  • An opener paused between throwIfBlockedByRestore and its open while a restore reserves the name is held off; a rescan and a create_table racing an online restore into a new name lose to the reservation, not the other way round.
  • A backup whose catalogue names a dropped column family fails validation; a log with valid framing and an undecodable value is rejected before publication; a blob missing without a marker is caught by the scan and reported under the waiver.
  • A peer or route added after online admission is refused while the name is reserved, and caught by the pre-publication re-check if it slips through.
  • Unrelated request latency measured during publication and replay, including on a threads: 0 install.

Activity

  1. self-assigned this
    on Sep 16, 2026
  2. added
    enhancementNew feature or request
    area:storageStorage engine, LMDB/RocksDB, compaction
    area:operations-apiOperations API: handlers, validation, response shape
    on Sep 16, 2026
  3. 3 remaining items

  4. cb1kenobi commented on Nov 14, 2026

    @cb1kenobi
    MemberAuthor

    Fixed by #3920 Publish every restore_backup through one crash-safe protocol, merged 2026-11-05 for v5.4. Every restore_backup now goes through one publication protocol: a receipt, a separate restore process that applies the backup and proves it, then the publication. Online restore into a new database with target_database works while Harper runs.

    🤖 Generated by Claude Opus 5.5; posted via @cb1kenobi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:operations-apiOperations API: handlers, validation, response shapearea:storageStorage engine, LMDB/RocksDB, compactionenhancementNew feature or request

Fields

Priority

P2

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions