Skip to content

restore_backup: on_restart route to replace a component-held database at the next startup #2638

Closed

Description

@cb1kenobi

Part of #2632 (item 6) — the replace-existing PR. Design note §5.6, §5.7, §5.8 in docs/proposals/archive-restore.md on branch design/archive-restore. Depends on #2635, #2636, #2637, #2640, and either #2451 or minting a generation here.

Problem

restore_backup closes the database across worker threads and returns 409 when a handle remains (verifyDatabaseClosed, dataLayer/rocksdbBackup.ts:624), pointing at the offline CLI. Any database a component declares tables in is component-held, and on Fabric the operator cannot stop the server, so restore_backup cannot restore the databases people actually use there. This is independent of archives; it is #1831's Fabric gap.

What to build

on_restart: true on restore_backup, explicit rather than automatic on 409 (a restart drops availability; the 409 message names the flag). The request validates as today (backup complete, blob roots compatible, no peers, room for the pre-restore backup), writes an intent file in state pending, inserts the job row, returns the job id, drains workers through the rolling path (restart today posts SHUTDOWN and terminates immediately — server/threads/manageThreads.js:983-995), and restarts.

The intent file lives beside the restore marker under the reserved `restore` directory, written temp → fsync → rename, versioned, and holds: operation id, database, backup_id, job id, retain_previous and the pinned pre-restore backup id, expected table set, blob-completeness decision, generation, requester, attempt count, and state. It is not the marker: beginRestore() truncates the marker before rewriting it (#2634). Malformed or torn intent fails closed (name blocked).

state discovery of the name next boot
pending blocked run the hook
applying blocked (marker too) run the hook; apply is rerunnable
committed open finish finalization only — never re-apply
terminal blocked boot in safe mode; wait for resume/supersede/cancel

The boot hook runs in server/loadRootComponents.js right after getTables() and before the root component loads — system is open (peer check), the target is unpublished (the intent blocks discovery like databasesBlockedByRestore), no worker exists, and no plugin has loaded. Sequence per intent, attempt counter fsynced first:

  1. Peers, read by core from system.hdb_nodes (rows other than getThisNodeName()) and replication.routes — no pro predicate, which cannot be registered yet. Peers → terminal.
  2. Pre-restore backup exactly once, while pending (createBackupOffline refuses a marked database, :965-969, so an applying retry must reuse the pinned id, never recapture). Taken here, after the drain, it captures the final acknowledged write; durable and pinned before step 3. Skipped for retain_previous: false and for a supersede/rollback.
  3. State → applying. The apply helper (restore_backup: shared publication protocol and online restore into a new database (create-if-absent) #2637) in the engine child process: lock, marker, purge, engine restore, blob restore — no LOCK probe at boot (the hook is the server; a probe that aborts on a corrupt old CURRENT would otherwise make the database unrecoverable through any resume/supersede), and no marker clear.
    4–7. Inventory check, private open, expected-table check, restore-strict elected replay, blob completeness, durability barriers — the publication protocol from restore_backup: shared publication protocol and online restore into a new database (create-if-absent) #2637.
  4. State → committed, fsynced, before the discovery block lifts. Then finalize idempotently: clear marker, publish via the ordinary load, reconcile the job row, delete the intent. A crash after committed only re-runs finalization.

Failure never exits. Under HARPER_EXIT_ON_RESTART a restart is process.exit(0) and the orchestrator loops (bin/restart.ts:162-166); on self-managed installs there is no supervisor at all (harper start forks and its restart handler exits with the launcher — utility/processManagement/processManagement.js:36-78). A failed attempt records the failure, keeps marker and intent, and continues boot with the database blocked. Retry on the next restart or resume: true; after the attempt limit, terminal. A terminal intent triggers safe mode by file for that boot (HARPER_SAFE_MODE exists but is env-only — bin/run.ts:43-44, server/loadRootComponents.js:48,61,77-82 — and a container cannot set its own env for the next boot); component load failures are already contained per component (components/componentLoader.ts:1165-1174). Document HARPER_SAFE_MODE.

API on a pending or terminal intent, super-user, working without the database loaded (#2636): resume=true re-arms and restarts; backup_id=<other> on_restart=true supersedes (this is rollback to the pre-restore id — no new pre-restore backup is taken); cancel=true removes the intent (unblocks only if pending; otherwise the marker keeps the block and the response says so). A committed intent can only be finished.

Retained copy: retain_previous defaults to true; declining is explicit and recorded in the job; the pre-restore backup is a normal managed backup, pinned while an intent names it, pruned by purge_backups afterwards. Its cost (one more N in the repository) is real under a quota, which is why it is an operator decision on the record rather than a heuristic.

Acceptance

  • get_backup → import_backup → restore_backup on_restart=true on a component-held database → supervised exit and restart → records and blobs readable; the job row shows the outcome; the pre-restore backup is listed and pinned.
  • Kill injection at every step: between pre-restore backup completion and pin (retry reuses the pinned id, never recaptures); between pin and first destructive write; during apply; between committed and the job-row write followed by new writes and another restart — nothing re-applies and the marker is gone.
  • A corrupt old destination at boot is replaced on the next attempt (no probe at the hook).
  • A peer appearing between request and boot lands in terminal; the node serves the operations API in safe mode; recovery entirely via supersede to the pre-restore id, then a writable restart.
  • Unsupervised (no orchestrator) startup after a failed attempt stays up with the database blocked.
  • Rollback content asserted equal to the last acknowledged pre-restart write.
  • retain_previous=false recorded in the job; cancel after destructive work leaves the marker.

Activity

  1. self-assigned this
    on Sep 16, 2026
  2. added
    enhancementNew feature or request
    area:storageStorage engine, LMDB/RocksDB, compaction
    area:operations-apiOperations API: handlers, validation, response shape
    on Sep 16, 2026
  3. cb1kenobi commented on Nov 14, 2026

    @cb1kenobi
    MemberAuthor

    Fixed by #3987 Restore a database a component holds open at the next startup with restore_backup on_restart=true, merged 2026-11-14 for v5.4. restore_backup on_restart=true restores a database a component holds open at the next startup, before components load, keeping a rollback copy unless retain_previous=false. resume=true and cancel=true handle a restore that can't finish. See Restoring at the next startup in the backup operations reference.

    🤖 Generated by Claude Opus 5.5; posted via @cb1kenobi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:operations-apiOperations API: handlers, validation, response shapearea:storageStorage engine, LMDB/RocksDB, compactionenhancementNew feature or request

Fields

Priority

P2

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions