Repository navigation
Description
Design note: docs/proposals/archive-restore.md on branch design/archive-restore (four planning-review rounds, the last two by codex gpt-6-astra at extra-high effort; both returned chosen-approach-sound). This epic tracks the work the note breaks down in its §9. It supersedes the rationale on #995 (closed 2026-07-31, "we don't want to introduce restoring backups while Harper is running"); #1831 landed restore_backup doing exactly that the next day.
The problem
get_backup streams a full point-in-time snapshot of a database as .tar/.tar.gz (createBackupStream, dataLayer/rocksdbBackup.ts:689). Nothing consumes it: the archive's embedded README tells the operator to stop Harper and hand-copy files into the database directory and each blob root. On Fabric that is unusable three ways — the operator has no filesystem access, target_database is refused while Harper runs (:500, :537), and "stop the server" is a host-manager action. The third point is not specific to archives: any database a component declares tables in is component-held, verifyDatabaseClosed (:624) returns 409 for it, and the 409 points at a CLI the Fabric operator cannot run. So restore_backup itself cannot restore the databases people actually use on Fabric today.
The approach
A get_backup archive and a managed backup differ only in format, and the conversion is two existing calls: an extracted stream backup "unpacks into a directory that opens directly as a RocksDB database" (rocksdb-js docs/backups.md), and db.backup(repo) on an opened directory produces a managed backup — which is exactly what createBackupOffline already does (dataLayer/rocksdbBackup.ts:959-990). Everything else an archive restore needs is already the managed restore path: lock + marker, verifyDatabaseClosed, backups.restore(purgeAllFiles), index-aware blob restore, verify_backup, and the offline CLI. What that path lacks — a way to run for a held database on Fabric — it lacks for managed backups too.
So:
import_backupconverts an archive (multipart upload or a pull from S3/HTTPS) into a verified managed backup id. No restart.- A shared publication protocol under which a restored database becomes visible: the restore lock as a real name reservation, private open, expected-table and column-family inventory checks, a restore-strict elected transaction-log replay, a blob-completeness decision, durability barriers, then publication. It carries restore into a new name online (create-if-absent) and upgrades in-place
restore_backupfor unheld databases. - A restart route on
restore_backup(on_restart: true) for component-held databases: a durable intent state machine, a boot hook that runs the restore before components load, a pre-restore backup captured once as the rollback source, no-exit retries (self-managed installs have no supervisor), a terminal state that boots in safe mode, andresume/supersede/cancelentirely through the API.
The invariant every piece serves (note §3): an acknowledged restore publishes exactly one complete, recoverable generation — engine data, schema, replayed transactions, index-addressed blobs — and no reader or creator can observe an intermediate one; the recovery source stays available until completion is durable; a restore under a new name holds an exclusive reservation from the first byte; a retry preserves the original rollback identity and never overwrites a published generation.
Restore is refused on any node with peers (hdb_nodes rows other than self, or replication.routes entries), re-checked at the boot hook and fenced during online restores. The system database, LMDB archives, and v4 migration are out of scope.
Work items
| # | Issue | Depends on |
|---|---|---|
| 1 | #2633 — capability manifest as the archive's first entry, and in managed backup manifests | — |
| 2 | #2634 — beginRestore() truncates the marker; write temp → fsync → rename |
— |
| 3 | #2635 — boot-time job reconciliation with process-instance ownership | — |
| 4 | #2636 — repository operations without a loaded database, per-repository management lock, restore pins | — |
| 5 | #2637 — publication protocol and online create-if-absent (target_database online) |
2, 4, 8, 9 |
| 6 | #2638 — on_restart route for component-held databases |
3, 4, 5, 8, 9 |
| 7 | #2639 — import_backup |
1, 4, 9 |
| 8 | #2451 — database generation on restore (or minted by 5 and 6 at the same point) | — |
| 9 | #2640 — refuse an engine-only restore over live blob roots without an explicit opt-in | — |
Items 1–4 and 9 each fix something on main today. The two PRs that carry the user-visible feature are item 5 (create-if-absent) and item 6 (replace an existing, held database). #2031 is a dependency only in the minimum form item 4 carries; #2033 is adjacent, not a dependency.
Rollout
Producer manifest → repository operations and pins → publication protocol with restore into a new name (stands alone, and is the safest mode) → restart route → feature-gated import_backup → Fabric canary evidence, including GTM behavior across the drained restart.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
cb1kenobi commentedon Sep 21, 2026
Design rationale, moved here from the closed PR #2643
The design note lived in
docs/proposals/archive-restore.mdon a PR of its own. Kris's review: "I think this should go in issues, not in source." That is consistent with the repo — everydocs/design/*.mdonmain(e.g.analytics-aggregation-resume-marker.mdin #2692,audit-walk-retention-floor.mdin #2684) lands in the same commit as the implementation it documents, whereas this was a plan for work that is mostly unbuilt. It also decayed: several of itsfile:linecitations went stale asmainmoved beneath the branch, which is exactly the failure mode an in-tree plan invites.The per-item scope already lives on the nine child issues. What follows is the part that lived only in the note: the invariant the work is measured against, and the option set with the reason the first approach was withdrawn.
The invariant
Every clause is load-bearing:
create_backupreporting abackup_idfor a purged engine backup (create_backup: serialize backup management ops to close a finalize/purge race #2031) andcopy-dbexiting 0 on a non-restorable copy (copy-db produces a silently corrupt, non-restorable copy and exits 0 — four independent channels in bin/copyDb.ts #2048) are the same shape: success reported on an artifact nobody proved whole.committedrecord only ever finishes bookkeeping (restore_backup: on_restart route to replace a component-held database at the next startup #2638).Approaches considered, and why the first was withdrawn
Chosen:
import_backupinto the managed backup repository, plus a restart route onrestore_backup. An extracted archive already is a database directory, and two existing calls turn one into a managed backup, so the archive path reduces to a format conversion followed by the restore path that already exists.Withdrawn: a bespoke rename-swap protocol. Extract per destination volume, validate out of process, write a versioned journal of every intended rename, and have the next startup swap the engine tree and each blob tree into place, retaining the displaced tree under a reserved name for a separate reclamation operation. Every piece of it exists in another form in the managed-backup path, and the original grounds for rejecting that path were wrong. Withdrawing it removed the journal, the per-volume rename protocol, and a new reclamation operation from the design, and let create-if-absent and
target_databasego back online.The tradeoff is real and worth recording: the rename swap writes ~1N and peaks at 2N occupancy, against ~3N written and 3N peak for the managed route (live + staging + repository). Under a Fabric disk quota that difference matters, so if the managed route's disk cost proves prohibitive, the rename swap is the fallback — and it would have to provide the journal, the per-volume protocol, and the reclamation operation that were dropped with it.
Also considered and rejected as whole answers: doing nothing and requiring filesystem access (Fabric operators have none); provisioning a new Fabric instance from the archive and cutting routing over (operationally safer, but changes instance identity and control-plane state, needs orchestration support per restore, and does not serve self-managed installs); and
target_databaseinto a copy, which is a useful rollout stage but not the whole answer because Fabric's component-held databases are the case that needs replacing in place.Review history
Six rounds: two by Claude, two by codex
gpt-6-astraat extra-high effort, one operator review, and one cross-model round (codex + Gemini + cursor-grok) that found the note had become self-inconsistent across edits. Round 2 replaced the chosen approach outright (rename-swap → managed repository). Rounds 3 and 4 returnedchosen-approach-soundand produced 22 findings against the protocol rather than the architecture. Kris's review then found two design defects that are recorded on #2639 and #2633.Generated by Barber AI