Skip to content

restore_backup claim admission: specific 409s erased, foreign source can take over a reserved target, rerun drops pins #4164

Open

Description

@kriszyp

Summary

Residual of #3920 (merged 2026-11-05). rocksdbBackup.ts beginRestoreForDatabase (~850-866) rewrites every 409 into the generic lock message, hiding admitOverEarlierReceipt's errors (restorePublication.ts ~190-209); assertTargetClaimable (~1367-1389) lets a different repository claim an empty target reserved by an interrupted restore; restorePublication.ts ~287 drops inherited pins on immutable_source or marker-less applying reruns, stranding a pin so delete_backup/purge_backups 409. Recoverable wedge; patch verified by QA.

Triage (finding-triage wave 50) re-verified the mechanism by reading harper main c7572a5. Detail from the QA finding(s) follows; line numbers there may predate main.

QA finding F-1179

Summary

In dataLayer/rocksdbBackup.ts:850-866, 1367-1389 and dataLayer/restorePublication.ts:111-123, 282-301 (introduced/modified in harper#3920 f5819f76e666d0ec8b81e17897960c4e7ede7b1d), three defects in restore_backup claim admission and pin inheritance across interrupted restores cause:

  1. Erased 409 operator recovery diagnostics (beginRestoreForDatabase): Unreadable .intent receipts (admitOverEarlierReceipt), committed receipts, and incomplete .dropping markers (databaseDropMarkerPresent) when no lock is held have their specific 409 messages overwritten with Cannot claim database '<name>': a restore, a drop, or a database open holds its lock; retry once that finishes.
  2. Foreign repository target_database takeover & stranded pins (assertTargetClaimable): When a restore from sourceA into target_database (inPlace: false) is interrupted while target.databaseDir is still missing or empty (preexisting === true, ours === false) or while a marker-less 'applying' receipt reserves target_database (preexisting === false, recorded.kind === 'valid'), assertTargetClaimable falls through to if (isMissingOrEmptyDir(target.databaseDir)) return; and admits a foreign repository sourceB (backupDirB !== backupDirA), overwriting <meta>/<key>.restoring and <meta>/<key>.intent with sourceB, locking sourceA out of rerunning if sourceB fails after writing debris, and stranding sourceA's backup pin (409 Cannot remove backups ... in use).
  3. Dropped superseded_pin_ids and stranded per-operation pins (publishUnderFence): Because publishUnderFence (restorePublication.ts:287) gates const held = preexisting && pinId ? ... : [] on both preexisting and pinId, an immutable_source: true rerun (pinId === undefined) overwrites the .intent receipt with source_pin_id: undefined and superseded_pin_ids: undefined (making releaseReceiptPins's superseded_pin_ids retirement at :113-119 dead code and permanently stranding any per-operation minted pin ${pinId}-${operation_id} across a subsequent writable recovery, which then falsely blocks delete_backup/purge_backups with 409 whenever databaseDir is next marked/recorded), and a rerun from another backup_id over a marker-less 'applying' receipt (preexisting === false && earlier !== undefined) overwrites restorePinId(databaseDir) in place instead of preserving the earlier backup's pin in superseded_pin_ids.

Mechanism (file:line)

  1. dataLayer/rocksdbBackup.ts:850-866 (beginRestoreForDatabase) vs dataLayer/restorePublication.ts:190-209, 256-257 (admitOverEarlierReceipt) & dataLayer/restoreMarker.ts:400-404 (claimRestore):
    • publishUnderFence (restorePublication.ts:256-257) calls beginRestoreForDatabase(databaseDir, targetDatabase, backupDir, (preexisting, claimLock) => { const earlier = admitOverEarlierReceipt(databaseDir, targetDatabase); ... }).
    • Inside claimRestore (restoreMarker.ts:387-425), databaseDropMarkerPresent(dbPath) (:400-404) throws Error("Database at <dbPath> has an incomplete drop; retry drop_database first") with error.statusCode = 409; admitOverEarlierReceipt (restorePublication.ts:193-197) throws ClientError("Database '<name>' has a restore record that cannot be read (<reason>); remove <receiptPath> before restoring again", 409) on an unreadable receipt and ClientError("Database '<name>' is finishing an earlier restore; retry once it has been published", 409) (:200-204) on a committed receipt.
    • beginRestoreForDatabase (rocksdbBackup.ts:859-863) catches if (error.statusCode === 409) unconditionally and throws new BackupInProgressError("Cannot claim database '<name>': a restore, a drop, or a database open holds its lock; retry once that finishes") without cause, telling the operator to wait for a lock that is not held (restoreLockHeldExclusively(databaseDir) === false).
  2. dataLayer/rocksdbBackup.ts:1367-1389 (assertTargetClaimable) & dataLayer/restoreMarker.ts:406-419 (claimRestore):
    • When target.inPlace === false:
      • If preexisting === true (a .restoring marker exists from sourceA) and ours === false (backupDir is foreign sourceB), if (ours) return; (:1383) does not return, and execution falls through to if (isMissingOrEmptyDir(target.databaseDir)) return; (:1385). In dataLayer/restoreChildCommands.ts:120-128, announceDestruction() publishes .restoring before backups.restore(..., { mode: 'purgeAllFiles' }) populates target.databaseDir, so an interruption before target files are written (or after purge empties target.databaseDir) leaves isMissingOrEmptyDir(target.databaseDir) === true and admits sourceB.
      • If preexisting === false (an interrupted claim from sourceA wrote <meta>/<key>.intent in state: 'applying' and pinned backupA before markRestoreDestructive), assertTargetClaimable skips if (preexisting) (:1373) completely and returns at :1385 (isMissingOrEmptyDir), ignoring readRestoreReceipt(target.databaseDir).
    • Once admitted, claimRestore (restoreMarker.ts:412-414) rewrites <meta>/<key>.restoring with restoreSourceId(backupDirB) and publishUnderFence (restorePublication.ts:296-301) overwrites <meta>/<key>.intent with backup_dir: backupDirB. If sourceB fails after writing debris, sourceA is locked out of rerunning its own restore (400 target_database ... already exists) while sourceA's backup remains pinned (pinIsLive === true at dataLayer/backupRepository.ts:237-244), and even after sourceB commits, releaseReceiptPins (:111-123) only unpins backupDirB, stranding sourceA's pin file on disk.
  3. dataLayer/restorePublication.ts:111-123 (releaseReceiptPins) & :282-301 (publishUnderFence):
    • At restorePublication.ts:287: const held = preexisting && pinId ? readBackupPins(backupDir).filter((pin) => inherited.has(pin.pin_id)) : [];.
    • When request.immutableSource === true, pinId is undefined (:243), so held is [] and supersededPinIds is [] (:289). writeRestoreReceipt (:296-301) overwrites .intent with source_pin_id: undefined and superseded_pin_ids: undefined.
      • This makes releaseReceiptPins's for (const superseded of receipt.superseded_pin_ids ?? []) (:113-119, commented // immutable_source pins nothing of its own, but still retires what the attempts it took over pinned) dead code when immutable_source: true commits.
      • Worse, if an earlier rerun minted a per-operation pin ${pinId}-${operation_id} (:292) and an immutable_source: true attempt then runs and fails, the receipt loses ${pinId}-${operation_id}. When a subsequent writable (immutable_source: false) recovery runs and commits, inherited (:282-284) only knows [restorePinId(databaseDir), earlier?.source_pin_id, ...(earlier?.superseded_pin_ids ?? [])], so ${pinId}-${operation_id}.json is never unpinned and remains in backupDir/pins/ with database_path: databaseDir. Whenever databaseDir next has an active or incomplete restore, pinIsLive (dataLayer/backupRepository.ts:237-244) treats that orphaned pin as live and assertBackupsUnpinned (:255-281) rejects delete_backup/purge_backups with 409 Cannot remove backups ... in use.
    • When preexisting === false and earlier !== undefined (a rerun from backup 2 over a marker-less 'applying' receipt that pinned backup 1), held is also [] because :287 checks preexisting instead of preexisting || earlier !== undefined, so pinBackup (:294) overwrites restorePinId(databaseDir) in place with backup 2 instead of preserving backup 1 in superseded_pin_ids and minting ${pinId}-${fresh.operation_id}.

Reproduction (45519a70df369e4fdc1192fb3af4a7cbc05d3bc4)

# Contract assertions fail 3/4 on unmodified origin/main (45519a70df369e4fdc1192fb3af4a7cbc05d3bc4):
QA_ASSERT_MODE=contract HARPER_STORAGE_ENGINE=rocksdb timeout -k 10 120 npx mocha unitTests/dataLayer/restorePublicationClaimAndPinInheritance.test.js --exit
QA_ASSERT_MODE=contract HARPER_STORAGE_ENGINE=lmdb timeout -k 10 120 npx mocha unitTests/dataLayer/restorePublicationClaimAndPinInheritance.test.js --exit

Proposed Minimal Fix (archive/QA-5302/counterfactual.patch)

  1. In dataLayer/rocksdbBackup.ts:850-866 (beginRestoreForDatabase), only rewrite non-ClientError 409s from acquireRestoreLock (if (error.statusCode === 409 && !(error instanceof ClientError))), preserving ClientErrors from admitOverEarlierReceipt and replacing at ${databaseDir} with '${databaseName}' for incomplete-drop / child-lock 409s.
  2. In dataLayer/rocksdbBackup.ts:1367-1389 (assertTargetClaimable), check const recorded = readRestoreReceipt(target.databaseDir); if (preexisting || recorded.kind === 'valid') { ... if (ours) return; } else if (isMissingOrEmptyDir(target.databaseDir)) { return; } so a foreign repository (!ours) cannot fall through to isMissingOrEmptyDir(target.databaseDir) when preexisting or recorded.kind === 'valid' holds.
  3. In dataLayer/restorePublication.ts:287 (publishUnderFence), compute const held = preexisting || earlier !== undefined ? readBackupPins(backupDir).filter((pin) => inherited.has(pin.pin_id)) : []; so inherited pins are preserved in superseded_pin_ids across immutable_source: true and marker-less 'applying' receipts.
  • Verified with archive/QA-5302/counterfactual.patch: restorePublicationClaimAndPinInheritance.test.js flips from 1 passing, 3 failing to 4 passing, 0 failing under QA_ASSERT_MODE=contract on both rocksdb (6s) and lmdb (4s), and all 5 existing restore/backup suites (restorePublication.test.js, restoreMarker.test.js, restoreReceipt.test.js, backupRepository.test.js, restoreGeneration.test.js) pass 105/105 (15s).

Existing Work Searched

  • Searched HarperFast/harper and HarperFast/harper-pro issues and PRs (open, closed, merged) for restorePublication, assertTargetClaimable, beginRestoreForDatabase, releaseReceiptPins, superseded_pin_ids, immutable_source, admitOverEarlierReceipt, 3920, 3825, 3816, 3107, 3106, 2965, 2031: related to merged PRs harper#3920, harper#3825, harper#3107 and closed issue harper#3106, but no open issue or PR tracks these three defects (attach=related:harper#3920,harper#3825,harper#3107).

Scope & What This Does NOT Prove

  • Applies to restore_backup (both online job and offline CLI routes) across dataLayer/rocksdbBackup.ts and dataLayer/restorePublication.ts.
  • Does not claim a failure when immutable_source: true is used on a clean target with no prior interrupted writable restore pins (Test 1 of rocksdbBackup.test.js describe('immutable_source')), nor does it claim releaseReceiptPins fails when backupDir is mounted EROFS (since releaseReceiptPins catches unpinBackup errors at restorePublication.ts:120-122).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions