Repository navigation
Description
Summary
Residual of #3920 (merged 2026-11-05). rocksdbBackup.ts beginRestoreForDatabase (~850-866) rewrites every 409 into the generic lock message, hiding admitOverEarlierReceipt's errors (restorePublication.ts ~190-209); assertTargetClaimable (~1367-1389) lets a different repository claim an empty target reserved by an interrupted restore; restorePublication.ts ~287 drops inherited pins on immutable_source or marker-less applying reruns, stranding a pin so delete_backup/purge_backups 409. Recoverable wedge; patch verified by QA.
Triage (finding-triage wave 50) re-verified the mechanism by reading harper main c7572a5. Detail from the QA finding(s) follows; line numbers there may predate main.
QA finding F-1179
Summary
In dataLayer/rocksdbBackup.ts:850-866, 1367-1389 and dataLayer/restorePublication.ts:111-123, 282-301 (introduced/modified in harper#3920 f5819f76e666d0ec8b81e17897960c4e7ede7b1d), three defects in restore_backup claim admission and pin inheritance across interrupted restores cause:
- Erased 409 operator recovery diagnostics (
beginRestoreForDatabase): Unreadable.intentreceipts (admitOverEarlierReceipt),committedreceipts, and incomplete.droppingmarkers (databaseDropMarkerPresent) when no lock is held have their specific409messages overwritten withCannot claim database '<name>': a restore, a drop, or a database open holds its lock; retry once that finishes. - Foreign repository
target_databasetakeover & stranded pins (assertTargetClaimable): When a restore fromsourceAintotarget_database(inPlace: false) is interrupted whiletarget.databaseDiris still missing or empty (preexisting === true,ours === false) or while a marker-less'applying'receipt reservestarget_database(preexisting === false,recorded.kind === 'valid'),assertTargetClaimablefalls through toif (isMissingOrEmptyDir(target.databaseDir)) return;and admits a foreign repositorysourceB(backupDirB !== backupDirA), overwriting<meta>/<key>.restoringand<meta>/<key>.intentwithsourceB, lockingsourceAout of rerunning ifsourceBfails after writing debris, and strandingsourceA's backup pin (409 Cannot remove backups ... in use). - Dropped
superseded_pin_idsand stranded per-operation pins (publishUnderFence): BecausepublishUnderFence(restorePublication.ts:287) gatesconst held = preexisting && pinId ? ... : []on bothpreexistingandpinId, animmutable_source: truererun (pinId === undefined) overwrites the.intentreceipt withsource_pin_id: undefinedandsuperseded_pin_ids: undefined(makingreleaseReceiptPins'ssuperseded_pin_idsretirement at:113-119dead code and permanently stranding any per-operation minted pin${pinId}-${operation_id}across a subsequent writable recovery, which then falsely blocksdelete_backup/purge_backupswith409wheneverdatabaseDiris next marked/recorded), and a rerun from anotherbackup_idover a marker-less'applying'receipt (preexisting === false && earlier !== undefined) overwritesrestorePinId(databaseDir)in place instead of preserving the earlier backup's pin insuperseded_pin_ids.
Mechanism (file:line)
dataLayer/rocksdbBackup.ts:850-866(beginRestoreForDatabase) vsdataLayer/restorePublication.ts:190-209, 256-257(admitOverEarlierReceipt) &dataLayer/restoreMarker.ts:400-404(claimRestore):publishUnderFence(restorePublication.ts:256-257) callsbeginRestoreForDatabase(databaseDir, targetDatabase, backupDir, (preexisting, claimLock) => { const earlier = admitOverEarlierReceipt(databaseDir, targetDatabase); ... }).- Inside
claimRestore(restoreMarker.ts:387-425),databaseDropMarkerPresent(dbPath)(:400-404) throwsError("Database at <dbPath> has an incomplete drop; retry drop_database first")witherror.statusCode = 409;admitOverEarlierReceipt(restorePublication.ts:193-197) throwsClientError("Database '<name>' has a restore record that cannot be read (<reason>); remove <receiptPath> before restoring again", 409)on an unreadable receipt andClientError("Database '<name>' is finishing an earlier restore; retry once it has been published", 409)(:200-204) on acommittedreceipt. beginRestoreForDatabase(rocksdbBackup.ts:859-863) catchesif (error.statusCode === 409)unconditionally and throwsnew BackupInProgressError("Cannot claim database '<name>': a restore, a drop, or a database open holds its lock; retry once that finishes")withoutcause, telling the operator to wait for a lock that is not held (restoreLockHeldExclusively(databaseDir) === false).
dataLayer/rocksdbBackup.ts:1367-1389(assertTargetClaimable) &dataLayer/restoreMarker.ts:406-419(claimRestore):- When
target.inPlace === false:- If
preexisting === true(a.restoringmarker exists fromsourceA) andours === false(backupDiris foreignsourceB),if (ours) return;(:1383) does not return, and execution falls through toif (isMissingOrEmptyDir(target.databaseDir)) return;(:1385). IndataLayer/restoreChildCommands.ts:120-128,announceDestruction()publishes.restoringbeforebackups.restore(..., { mode: 'purgeAllFiles' })populatestarget.databaseDir, so an interruption before target files are written (or after purge emptiestarget.databaseDir) leavesisMissingOrEmptyDir(target.databaseDir) === trueand admitssourceB. - If
preexisting === false(an interrupted claim fromsourceAwrote<meta>/<key>.intentinstate: 'applying'and pinnedbackupAbeforemarkRestoreDestructive),assertTargetClaimableskipsif (preexisting)(:1373) completely and returns at:1385(isMissingOrEmptyDir), ignoringreadRestoreReceipt(target.databaseDir).
- If
- Once admitted,
claimRestore(restoreMarker.ts:412-414) rewrites<meta>/<key>.restoringwithrestoreSourceId(backupDirB)andpublishUnderFence(restorePublication.ts:296-301) overwrites<meta>/<key>.intentwithbackup_dir: backupDirB. IfsourceBfails after writing debris,sourceAis locked out of rerunning its own restore (400 target_database ... already exists) whilesourceA's backup remains pinned (pinIsLive === trueatdataLayer/backupRepository.ts:237-244), and even aftersourceBcommits,releaseReceiptPins(:111-123) only unpinsbackupDirB, strandingsourceA's pin file on disk.
- When
dataLayer/restorePublication.ts:111-123(releaseReceiptPins) &:282-301(publishUnderFence):- At
restorePublication.ts:287:const held = preexisting && pinId ? readBackupPins(backupDir).filter((pin) => inherited.has(pin.pin_id)) : [];. - When
request.immutableSource === true,pinIdisundefined(:243), soheldis[]andsupersededPinIdsis[](:289).writeRestoreReceipt(:296-301) overwrites.intentwithsource_pin_id: undefinedandsuperseded_pin_ids: undefined.- This makes
releaseReceiptPins'sfor (const superseded of receipt.superseded_pin_ids ?? [])(:113-119, commented// immutable_source pins nothing of its own, but still retires what the attempts it took over pinned) dead code whenimmutable_source: truecommits. - Worse, if an earlier rerun minted a per-operation pin
${pinId}-${operation_id}(:292) and animmutable_source: trueattempt then runs and fails, the receipt loses${pinId}-${operation_id}. When a subsequent writable (immutable_source: false) recovery runs and commits,inherited(:282-284) only knows[restorePinId(databaseDir), earlier?.source_pin_id, ...(earlier?.superseded_pin_ids ?? [])], so${pinId}-${operation_id}.jsonis never unpinned and remains inbackupDir/pins/withdatabase_path: databaseDir. WheneverdatabaseDirnext has an active or incomplete restore,pinIsLive(dataLayer/backupRepository.ts:237-244) treats that orphaned pin as live andassertBackupsUnpinned(:255-281) rejectsdelete_backup/purge_backupswith409 Cannot remove backups ... in use.
- This makes
- When
preexisting === falseandearlier !== undefined(a rerun frombackup 2over a marker-less'applying'receipt that pinnedbackup 1),heldis also[]because:287checkspreexistinginstead ofpreexisting || earlier !== undefined, sopinBackup(:294) overwritesrestorePinId(databaseDir)in place withbackup 2instead of preservingbackup 1insuperseded_pin_idsand minting${pinId}-${fresh.operation_id}.
- At
Reproduction (45519a70df369e4fdc1192fb3af4a7cbc05d3bc4)
# Contract assertions fail 3/4 on unmodified origin/main (45519a70df369e4fdc1192fb3af4a7cbc05d3bc4):
QA_ASSERT_MODE=contract HARPER_STORAGE_ENGINE=rocksdb timeout -k 10 120 npx mocha unitTests/dataLayer/restorePublicationClaimAndPinInheritance.test.js --exit
QA_ASSERT_MODE=contract HARPER_STORAGE_ENGINE=lmdb timeout -k 10 120 npx mocha unitTests/dataLayer/restorePublicationClaimAndPinInheritance.test.js --exitProposed Minimal Fix (archive/QA-5302/counterfactual.patch)
- In
dataLayer/rocksdbBackup.ts:850-866(beginRestoreForDatabase), only rewrite non-ClientError409s fromacquireRestoreLock(if (error.statusCode === 409 && !(error instanceof ClientError))), preservingClientErrors fromadmitOverEarlierReceiptand replacingat ${databaseDir}with'${databaseName}'for incomplete-drop / child-lock409s. - In
dataLayer/rocksdbBackup.ts:1367-1389(assertTargetClaimable), checkconst recorded = readRestoreReceipt(target.databaseDir); if (preexisting || recorded.kind === 'valid') { ... if (ours) return; } else if (isMissingOrEmptyDir(target.databaseDir)) { return; }so a foreign repository (!ours) cannot fall through toisMissingOrEmptyDir(target.databaseDir)whenpreexistingorrecorded.kind === 'valid'holds. - In
dataLayer/restorePublication.ts:287(publishUnderFence), computeconst held = preexisting || earlier !== undefined ? readBackupPins(backupDir).filter((pin) => inherited.has(pin.pin_id)) : [];so inherited pins are preserved insuperseded_pin_idsacrossimmutable_source: trueand marker-less'applying'receipts.
- Verified with
archive/QA-5302/counterfactual.patch:restorePublicationClaimAndPinInheritance.test.jsflips from1 passing, 3 failingto4 passing, 0 failingunderQA_ASSERT_MODE=contracton bothrocksdb(6s) andlmdb(4s), and all 5 existing restore/backup suites (restorePublication.test.js,restoreMarker.test.js,restoreReceipt.test.js,backupRepository.test.js,restoreGeneration.test.js) pass105/105(15s).
Existing Work Searched
- Searched
HarperFast/harperandHarperFast/harper-proissues and PRs (open, closed, merged) forrestorePublication,assertTargetClaimable,beginRestoreForDatabase,releaseReceiptPins,superseded_pin_ids,immutable_source,admitOverEarlierReceipt,3920,3825,3816,3107,3106,2965,2031: related to merged PRsharper#3920,harper#3825,harper#3107and closed issueharper#3106, but no open issue or PR tracks these three defects (attach=related:harper#3920,harper#3825,harper#3107).
Scope & What This Does NOT Prove
- Applies to
restore_backup(both online job and offline CLI routes) acrossdataLayer/rocksdbBackup.tsanddataLayer/restorePublication.ts. - Does not claim a failure when
immutable_source: trueis used on a clean target with no prior interrupted writable restore pins (Test 1ofrocksdbBackup.test.jsdescribe('immutable_source')), nor does it claimreleaseReceiptPinsfails whenbackupDiris mountedEROFS(sincereleaseReceiptPinscatchesunpinBackuperrors atrestorePublication.ts:120-122).
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity