Skip to content

[Epic] Restore a get_backup archive into a running Harper (incl. Fabric) #2632

Open
9 of 12 issues completed
Open

Description

@cb1kenobi

Design note: docs/proposals/archive-restore.md on branch design/archive-restore (four planning-review rounds, the last two by codex gpt-6-astra at extra-high effort; both returned chosen-approach-sound). This epic tracks the work the note breaks down in its §9. It supersedes the rationale on #995 (closed 2026-07-31, "we don't want to introduce restoring backups while Harper is running"); #1831 landed restore_backup doing exactly that the next day.

The problem

get_backup streams a full point-in-time snapshot of a database as .tar/.tar.gz (createBackupStream, dataLayer/rocksdbBackup.ts:689). Nothing consumes it: the archive's embedded README tells the operator to stop Harper and hand-copy files into the database directory and each blob root. On Fabric that is unusable three ways — the operator has no filesystem access, target_database is refused while Harper runs (:500, :537), and "stop the server" is a host-manager action. The third point is not specific to archives: any database a component declares tables in is component-held, verifyDatabaseClosed (:624) returns 409 for it, and the 409 points at a CLI the Fabric operator cannot run. So restore_backup itself cannot restore the databases people actually use on Fabric today.

The approach

A get_backup archive and a managed backup differ only in format, and the conversion is two existing calls: an extracted stream backup "unpacks into a directory that opens directly as a RocksDB database" (rocksdb-js docs/backups.md), and db.backup(repo) on an opened directory produces a managed backup — which is exactly what createBackupOffline already does (dataLayer/rocksdbBackup.ts:959-990). Everything else an archive restore needs is already the managed restore path: lock + marker, verifyDatabaseClosed, backups.restore(purgeAllFiles), index-aware blob restore, verify_backup, and the offline CLI. What that path lacks — a way to run for a held database on Fabric — it lacks for managed backups too.

So:

  1. import_backup converts an archive (multipart upload or a pull from S3/HTTPS) into a verified managed backup id. No restart.
  2. A shared publication protocol under which a restored database becomes visible: the restore lock as a real name reservation, private open, expected-table and column-family inventory checks, a restore-strict elected transaction-log replay, a blob-completeness decision, durability barriers, then publication. It carries restore into a new name online (create-if-absent) and upgrades in-place restore_backup for unheld databases.
  3. A restart route on restore_backup (on_restart: true) for component-held databases: a durable intent state machine, a boot hook that runs the restore before components load, a pre-restore backup captured once as the rollback source, no-exit retries (self-managed installs have no supervisor), a terminal state that boots in safe mode, and resume/supersede/cancel entirely through the API.

The invariant every piece serves (note §3): an acknowledged restore publishes exactly one complete, recoverable generation — engine data, schema, replayed transactions, index-addressed blobs — and no reader or creator can observe an intermediate one; the recovery source stays available until completion is durable; a restore under a new name holds an exclusive reservation from the first byte; a retry preserves the original rollback identity and never overwrites a published generation.

Restore is refused on any node with peers (hdb_nodes rows other than self, or replication.routes entries), re-checked at the boot hook and fenced during online restores. The system database, LMDB archives, and v4 migration are out of scope.

Work items

# Issue Depends on
1 #2633 — capability manifest as the archive's first entry, and in managed backup manifests —
2 #2634 — beginRestore() truncates the marker; write temp → fsync → rename —
3 #2635 — boot-time job reconciliation with process-instance ownership —
4 #2636 — repository operations without a loaded database, per-repository management lock, restore pins —
5 #2637 — publication protocol and online create-if-absent (target_database online) 2, 4, 8, 9
6 #2638 — on_restart route for component-held databases 3, 4, 5, 8, 9
7 #2639 — import_backup 1, 4, 9
8 #2451 — database generation on restore (or minted by 5 and 6 at the same point) —
9 #2640 — refuse an engine-only restore over live blob roots without an explicit opt-in —

Items 1–4 and 9 each fix something on main today. The two PRs that carry the user-visible feature are item 5 (create-if-absent) and item 6 (replace an existing, held database). #2031 is a dependency only in the minimum form item 4 carries; #2033 is adjacent, not a dependency.

Rollout

Producer manifest → repository operations and pins → publication protocol with restore into a new name (stands alone, and is the safest mode) → restart route → feature-gated import_backup → Fabric canary evidence, including GTM behavior across the drained restart.

Activity

  1. self-assigned this
    on Sep 16, 2026
  2. added
    area:storageStorage engine, LMDB/RocksDB, compaction
    area:operations-apiOperations API: handlers, validation, response shape
    on Sep 16, 2026
  3. cb1kenobi commented on Sep 21, 2026

    @cb1kenobi
    MemberAuthor

    Design rationale, moved here from the closed PR #2643

    The design note lived in docs/proposals/archive-restore.md on a PR of its own. Kris's review: "I think this should go in issues, not in source." That is consistent with the repo — every docs/design/*.md on main (e.g. analytics-aggregation-resume-marker.md in #2692, audit-walk-retention-floor.md in #2684) lands in the same commit as the implementation it documents, whereas this was a plan for work that is mostly unbuilt. It also decayed: several of its file:line citations went stale as main moved beneath the branch, which is exactly the failure mode an in-tree plan invites.

    The per-item scope already lives on the nine child issues. What follows is the part that lived only in the note: the invariant the work is measured against, and the option set with the reason the first approach was withdrawn.

    The invariant

    An acknowledged restore publishes exactly one complete, recoverable generation — engine data, schema, replayed transactions, and index-addressed blobs — and no reader or creator can observe or mutate an intermediate generation. The recovery source stays available until a durable completion decision is recorded. A restore under a new name holds an exclusive reservation on that name from the first byte. A retry preserves the original rollback identity and never overwrites a generation that has already been published.

    Every clause is load-bearing:

    Approaches considered, and why the first was withdrawn

    Chosen: import_backup into the managed backup repository, plus a restart route on restore_backup. An extracted archive already is a database directory, and two existing calls turn one into a managed backup, so the archive path reduces to a format conversion followed by the restore path that already exists.

    Withdrawn: a bespoke rename-swap protocol. Extract per destination volume, validate out of process, write a versioned journal of every intended rename, and have the next startup swap the engine tree and each blob tree into place, retaining the displaced tree under a reserved name for a separate reclamation operation. Every piece of it exists in another form in the managed-backup path, and the original grounds for rejecting that path were wrong. Withdrawing it removed the journal, the per-volume rename protocol, and a new reclamation operation from the design, and let create-if-absent and target_database go back online.

    The tradeoff is real and worth recording: the rename swap writes ~1N and peaks at 2N occupancy, against ~3N written and 3N peak for the managed route (live + staging + repository). Under a Fabric disk quota that difference matters, so if the managed route's disk cost proves prohibitive, the rename swap is the fallback — and it would have to provide the journal, the per-volume protocol, and the reclamation operation that were dropped with it.

    Also considered and rejected as whole answers: doing nothing and requiring filesystem access (Fabric operators have none); provisioning a new Fabric instance from the archive and cutting routing over (operationally safer, but changes instance identity and control-plane state, needs orchestration support per restore, and does not serve self-managed installs); and target_database into a copy, which is a useful rollout stage but not the whole answer because Fabric's component-held databases are the case that needs replacing in place.

    Review history

    Six rounds: two by Claude, two by codex gpt-6-astra at extra-high effort, one operator review, and one cross-model round (codex + Gemini + cursor-grok) that found the note had become self-inconsistent across edits. Round 2 replaced the chosen approach outright (rename-swap → managed repository). Rounds 3 and 4 returned chosen-approach-sound and produced 22 findings against the protocol rather than the architecture. Kris's review then found two design defects that are recorded on #2639 and #2633.

    Generated by Barber AI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:operations-apiOperations API: handlers, validation, response shapearea:storageStorage engine, LMDB/RocksDB, compactionenhancementNew feature or request

Type

No type

Fields

Priority

P2

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions