Repository navigation
Description
Part of #2632 (item 6) — the replace-existing PR. Design note §5.6, §5.7, §5.8 in docs/proposals/archive-restore.md on branch design/archive-restore. Depends on #2635, #2636, #2637, #2640, and either #2451 or minting a generation here.
Problem
restore_backup closes the database across worker threads and returns 409 when a handle remains (verifyDatabaseClosed, dataLayer/rocksdbBackup.ts:624), pointing at the offline CLI. Any database a component declares tables in is component-held, and on Fabric the operator cannot stop the server, so restore_backup cannot restore the databases people actually use there. This is independent of archives; it is #1831's Fabric gap.
What to build
on_restart: true on restore_backup, explicit rather than automatic on 409 (a restart drops availability; the 409 message names the flag). The request validates as today (backup complete, blob roots compatible, no peers, room for the pre-restore backup), writes an intent file in state pending, inserts the job row, returns the job id, drains workers through the rolling path (restart today posts SHUTDOWN and terminates immediately — server/threads/manageThreads.js:983-995), and restarts.
The intent file lives beside the restore marker under the reserved `restore` directory, written temp → fsync → rename, versioned, and holds: operation id, database, backup_id, job id, retain_previous and the pinned pre-restore backup id, expected table set, blob-completeness decision, generation, requester, attempt count, and state. It is not the marker: beginRestore() truncates the marker before rewriting it (#2634). Malformed or torn intent fails closed (name blocked).
| state | discovery of the name | next boot |
|---|---|---|
pending |
blocked | run the hook |
applying |
blocked (marker too) | run the hook; apply is rerunnable |
committed |
open | finish finalization only — never re-apply |
terminal |
blocked | boot in safe mode; wait for resume/supersede/cancel |
The boot hook runs in server/loadRootComponents.js right after getTables() and before the root component loads — system is open (peer check), the target is unpublished (the intent blocks discovery like databasesBlockedByRestore), no worker exists, and no plugin has loaded. Sequence per intent, attempt counter fsynced first:
- Peers, read by core from
system.hdb_nodes(rows other thangetThisNodeName()) andreplication.routes— no pro predicate, which cannot be registered yet. Peers →terminal. - Pre-restore backup exactly once, while
pending(createBackupOfflinerefuses a marked database,:965-969, so anapplyingretry must reuse the pinned id, never recapture). Taken here, after the drain, it captures the final acknowledged write; durable and pinned before step 3. Skipped forretain_previous: falseand for a supersede/rollback. - State →
applying. The apply helper (restore_backup: shared publication protocol and online restore into a new database (create-if-absent) #2637) in the engine child process: lock, marker, purge, engine restore, blob restore — noLOCKprobe at boot (the hook is the server; a probe that aborts on a corrupt oldCURRENTwould otherwise make the database unrecoverable through anyresume/supersede), and no marker clear.
4–7. Inventory check, private open, expected-table check, restore-strict elected replay, blob completeness, durability barriers — the publication protocol from restore_backup: shared publication protocol and online restore into a new database (create-if-absent) #2637. - State →
committed, fsynced, before the discovery block lifts. Then finalize idempotently: clear marker, publish via the ordinary load, reconcile the job row, delete the intent. A crash aftercommittedonly re-runs finalization.
Failure never exits. Under HARPER_EXIT_ON_RESTART a restart is process.exit(0) and the orchestrator loops (bin/restart.ts:162-166); on self-managed installs there is no supervisor at all (harper start forks and its restart handler exits with the launcher — utility/processManagement/processManagement.js:36-78). A failed attempt records the failure, keeps marker and intent, and continues boot with the database blocked. Retry on the next restart or resume: true; after the attempt limit, terminal. A terminal intent triggers safe mode by file for that boot (HARPER_SAFE_MODE exists but is env-only — bin/run.ts:43-44, server/loadRootComponents.js:48,61,77-82 — and a container cannot set its own env for the next boot); component load failures are already contained per component (components/componentLoader.ts:1165-1174). Document HARPER_SAFE_MODE.
API on a pending or terminal intent, super-user, working without the database loaded (#2636): resume=true re-arms and restarts; backup_id=<other> on_restart=true supersedes (this is rollback to the pre-restore id — no new pre-restore backup is taken); cancel=true removes the intent (unblocks only if pending; otherwise the marker keeps the block and the response says so). A committed intent can only be finished.
Retained copy: retain_previous defaults to true; declining is explicit and recorded in the job; the pre-restore backup is a normal managed backup, pinned while an intent names it, pruned by purge_backups afterwards. Its cost (one more N in the repository) is real under a quota, which is why it is an operator decision on the record rather than a heuristic.
Acceptance
get_backup→import_backup→restore_backup on_restart=trueon a component-held database → supervised exit and restart → records and blobs readable; the job row shows the outcome; the pre-restore backup is listed and pinned.- Kill injection at every step: between pre-restore backup completion and pin (retry reuses the pinned id, never recaptures); between pin and first destructive write; during apply; between
committedand the job-row write followed by new writes and another restart — nothing re-applies and the marker is gone. - A corrupt old destination at boot is replaced on the next attempt (no probe at the hook).
- A peer appearing between request and boot lands in
terminal; the node serves the operations API in safe mode; recovery entirely viasupersedeto the pre-restore id, then a writable restart. - Unsupervised (no orchestrator) startup after a failed attempt stays up with the database blocked.
- Rollback content asserted equal to the last acknowledged pre-restart write.
retain_previous=falserecorded in the job;cancelafter destructive work leaves the marker.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
Restore a database a component holds open at the next startup with re…
cb1kenobi commentedon Nov 14, 2026
Fixed by #3987 Restore a database a component holds open at the next startup with restore_backup on_restart=true, merged 2026-11-14 for v5.4.
restore_backup on_restart=truerestores a database a component holds open at the next startup, before components load, keeping a rollback copy unlessretain_previous=false.resume=trueandcancel=truehandle a restore that can't finish. See Restoring at the next startup in the backup operations reference.🤖 Generated by Claude Opus 5.5; posted via @cb1kenobi.