Repository navigation
Description
Part of #2632 (item 1). Design note §7.1 in docs/proposals/archive-restore.md on branch design/archive-restore.
Problem
A get_backup archive contains only human-readable READMEs (streamedBackupReadme, dataLayer/rocksdbBackup.ts:928; blobsReadmeContent, dataLayer/blobBackup.ts:228). Nothing machine-readable identifies the producing Harper version or the storage format, so a future importer cannot refuse an incompatible archive it cannot identify. Every archive produced today is unidentifiable, and those are the archives people will try to restore — the set only grows, which is why this ships first.
The managed backup manifest has the same gap: dataLayer/backupManifest.ts records only { backupId, blobs, completedAt }.
What to build
- A manifest as the first tar entry of every
get_backuparchive. It must be first: reaching the end of a.tar.gzrequires inflating the whole stream, while the first entry is readable after a few KB. - Fields: archive-schema version; producing Harper version (for logging); rocksdb-js major; transaction-log format version; blob-format features present (compression, marker types — deflate-compressed blob bodies from Enable compression for file-backed blobs (opt-in, per-content-type), and stream-inflate on read #2443 are unreadable by a Harper without that support and no engine check catches them); source database name; whether blobs are included; blob root count; the names of roles holding grants on the database (names only — the archive carries no role catalogue, and the restore side reports which names are absent locally rather than creating anything).
- Compatibility rule: capability flags, not a scalar version. The reader refuses when the target lacks any capability the archive declares. (An earlier draft cited 5.2.0 upgrade is one-way and the older binary fails silently: no log line, no error, just stops after the version check #2046 as "the 5.2.0 upgrade is one-way"; that issue's resolution says the migration is additive and the binary was blocked on a hidden downgrade prompt, and it concerns
system.hdb_info, not user-database format. The real hazards are RocksDBformat_version/OPTIONS, structon struct-mode records (DESIGN.md:17-25), the transaction-log format version, and blob file format.) - The same fields on the managed backup manifest. A managed backup whose manifest predates them is accepted (same-instance lineage) and reported as
format: unidentified. - Pre-manifest archives: accepted only under an explicit super-user override that records the operator's assertion of provenance; the result reports the archive as unidentified. Refusing them outright would make the importer useless for exactly the backups already on operators' disks.
Implementation note: this unifies createBackupStream's two branches. The excludeBlobs path hands the whole archive to the binding, which gzips it directly (dataLayer/rocksdbBackup.ts:704-709), so there is no tar-stream pack to prepend an entry to; both paths route through the plain-tar-then-gzip assembly streamBackupWithBlobs already uses.
Acceptance
- Every
get_backuparchive (with and withoutexclude_blobs) starts with the manifest entry;tar tfshows it first. - Unit: capability-flag refusal matrix, including a pre-manifest archive with and without the override, and a managed manifest without the new fields.
create_backupmanifests carry the producer fields;list_backupssurfaces them.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
Identify a get_backup archive with a manifest as its first entry
Identify a get_backup archive with a manifest as its first entry
Identify a get_backup archive with a manifest as its first entry
cb1kenobi commentedon Sep 21, 2026
Design defect found in review of #2643 (Kris), carried here before that PR closes.
The archive manifest's capability list needs to carry the external compression dictionary as an identity, not a presence flag:
Background on how this got here: the dictionary was first recorded in the manifest's provenance block as a boolean, which was the wrong half of the document — provenance is explicitly never gated on, while data compressed with an external dictionary genuinely cannot be read without that file (
resources/databases.ts, theSTORAGE_COMPRESSION_DICTIONARYhandling that reads the file intoLMDB_COMPRESSION_OPTS.dictionary). Moving it intorequireswas necessary but not sufficient, becauserequirestokens are matched by name: a reader either has the capability or does not. A dictionary is not a capability — it is a specific artifact, and two instances can both "have a dictionary" while having different ones.Resolution: the token carries the dictionary's algorithm and SHA-256 digest, and admission compares that against the digest of the dictionary configured on the target. No configured dictionary, or a digest mismatch, refuses the archive. An exact-match requirement is the safe default here — silently decoding with the wrong dictionary produces corrupt records rather than an error.
Note this also applies to the managed-backup manifest, not only to archives, if a dictionary can change between a backup and its restore on the same instance.
Generated by Barber AI
Identify a get_backup archive with a manifest as its first entry
Identify a get_backup archive with a manifest as its first entry
Identify a get_backup archive with a manifest as its first entry