Repository navigation
Description
Summary
Residual of #3920. restoreChildCommands.ts (~19-24) replays rootPath via setProperty, which does not update installProps (environmentManager.ts ~128-130), so getHdbBasePath stays stale; prove() (~182) does not pin databasePaths to args.blobRoots (openBranchDatabase does, databases.ts ~2693); childExecArgv (restoreChildRunner.ts ~84-104) leaves orphan values for --max-old-space-size-percentage and --inspect-publish-uid, keeps --debug-port and attached -e/-p. Fails in prove after apply purged the target, leaving 'incomplete'; a rerun recovers. Arm 3 alone is P3.
Triage (finding-triage wave 50) re-verified the mechanism by reading harper main c7572a5. Detail from the QA finding(s) follows; line numbers there may predate main.
QA finding F-1182
Summary
In the restore-child architecture introduced by harper#3920 (f5819f76e666d0ec8b81e17897960c4e7ede7b1d), three coupled defects in utility/environment/environmentManager.ts, dataLayer/restoreChildCommands.ts, resources/databases.ts, and dataLayer/restoreChildRunner.ts cause restore_database / restoreBackupOffline / publishRestore to fail after phase 'apply' has already purged the target database directory (mode: 'purgeAllFiles', leaving the target wedged in checkRestoreState(srcDir) === 'incomplete') or prevent runRestoreChild from spawning the restore child at all:
-
Arm 1 (
initializeRestoreChildEnvironment+setPropertyleavesgetHdbBasePath()unsynced on canonicalizedrootPathoverrides):setProperty(propName, value)(utility/environment/environmentManager.ts:97-133) canonicalizes'HDB_ROOT'(HDB_SETTINGS_NAMES.HDB_ROOT_KEYatutility/hdbTerms.ts:467) viaCONFIG_PARAM_MAP['hdb_root'](utility/hdbTerms.ts:901-903) to'rootPath'(CONFIG_PARAMS.ROOTPATHatutility/hdbTerms.ts:795), sogetConfigOverrides()(utility/environment/environmentManager.ts:141-148) returns{ rootPath: isolatedRoot }withoverrides.HDB_ROOT === undefined.- However,
setProperty(utility/environment/environmentManager.ts:128-130) only updatesinstallProps[propName]wheninstallPropsToSave[propName](utility/environment/environmentManager.ts:20-25) is truthy ('HDB_ROOT', not'rootPath'). - Unlike worker threads (where
initSync()atutility/environment/environmentManager.ts:300-324replaysworkerData.configOverridesviaapplyInheritedConfigOverrides()before syncinginstallProps[HDB_ROOT_KEY] = configUtils.getConfigValue(HDB_ROOT_KEY)at:319-323), the restore child is afork()ed child process (workerData === null). initializeRestoreChildEnvironment(spec)(dataLayer/restoreChildCommands.ts:19-24, called fromdataLayer/restoreChild.ts:67-70) callsinitSync()(:20) before replayingspec.configOverrides(for (const [name, value] of Object.entries(spec.configOverrides ?? {})) setProperty(name, value);at:21). Becausespec.configOverridescarries canonical key'rootPath'rather than'HDB_ROOT', replayingsetProperty('rootPath', isolatedRoot)updatesconfigUtils(get('rootPath') === isolatedRoot) while leavinginstallProps[HDB_ROOT_KEY](getHdbBasePath()atutility/environment/environmentManager.ts:48-50) pinned to the pre-replay on-diskrootPath(unlessutility/logging/harper_logger.ts:926-943's unrelated 100mslogRotator.tstimer happens to fire first whenlogging.rotationis enabled on disk).- When
restoreBackupOffline(dataLayer/rocksdbBackup.ts:1278-1313) /publishRestore(dataLayer/restorePublication.ts:230-381) restores a blob-bearing database under anHDB_ROOT/rootPathoverride, phase'apply'(dataLayer/restoreChildCommands.ts:120-136) purgesargs.databasePath(backups.restore(..., { mode: 'purgeAllFiles' })at:128) and extracts the blob snapshot into<isolatedRoot>/blobs/<db>(restoreBlobSnapshot(..., args.blobRoots, { durable: true })at:130-132), and phase'prove'(dataLayer/restoreChildCommands.ts:181-208) resolvesgetBlobPathsForDatabaseName(args.databaseName)(resources/blob.ts:2818-2824) against the child's stalegetHdbBasePath(), misclassifying every restored blob asstate: 'missing'and aborting with409 ClientError(phase: 'prove'), leaving the target database destroyed and wedged withcheckRestoreState(srcDir) === 'incomplete'(dataLayer/restorePublication.ts:394-455).
-
Arm 2 (
prove()andopenRestoredGeneration()do not pingeneration.rootStoretoargs.blobRootsindatabasePaths):- In
applyAndProve(args)(dataLayer/restoreChildCommands.ts:108-138), phase'apply'restores the blob snapshot intoargs.blobRoots(restoreBlobSnapshot(args.backupDir, args.backupId, args.sourceDatabase, args.blobRoots, { durable: true })at:130-132), whereas phase'prove'(dataLayer/restoreChildCommands.ts:181-223) callsopenRestoredGeneration(args.databasePath, args.databaseName)(:182) without passingargs.blobRoots. - Unlike
openBranchDatabase(resources/databases.ts:2685-2690), which pinsif (blobRoots) databasePaths.set(rootStore as unknown as RootDatabase, blobRoots);(cleaned up bycloseBranchHandlesatresources/databases.ts:2824, whichcloseRestoredGenerationatresources/databases.ts:1961-1963already calls),openRestoredGeneration(resources/databases.ts:1926-1958) leavesdatabasePaths.get(generation.rootStore)undefined. - When
assessBlobReferences(generation.tables)(resources/blob.ts:4022-4082) callsgetFilePath(storageInfo)(resources/blob.ts:2862-2871) ->getRootBlobPathsForDB(generation.rootStore)(resources/blob.ts:2794-2809),databasePaths.get(generation.rootStore)isundefined, so it falls back togetBlobPathsForDatabaseName(args.databaseName)(resources/blob.ts:2808, 2818-2824) instead ofargs.blobRoots. - Whenever
args.blobRootsdiffers fromgetBlobPathsForDatabaseName(args.databaseName), phase'prove'fails with409 ClientError (...: missing)after phase'apply'has already destroyedargs.databasePath(or, withacceptUnverifiedBlobs: true, falsely downgradesevidence.blobs.blob_integrityto'unverified'withwaived: trueandfailures: [{ state: 'missing' }]atdataLayer/restoreChildCommands.ts:194-205even though the restored blob file inargs.blobRootsis intact).
- In
-
Arm 3 (
childExecArgvleaves orphan positional tokens for space-separated flags and keeps attached-e/-pscripts):childExecArgv(parentExecArgv)(dataLayer/restoreChildRunner.ts:84-104) strips any flag starting with--max-old-space-sizeor--inspect(:96), but only skips the following argument token whenarg === '--max-old-space-size' || arg === '--inspect-port'(:98), and only matches-e/-pwhen passed as exact standalone elements ofSCRIPT_FLAGS(:85, :92).- When
parentExecArgvcontains valid space-separated Node flags--inspect-publish-uid stderr,--max-old-space-size-percentage 50, or--debug-port 9229,childExecArgvstrips the flag (or keeps--debug-port 9229) and leaves'stderr'/'50'inexecArgvas a bare positional script path before the restore child script; and whenparentExecArgvcontains attached-e<script>or-p<script>,childExecArgvleaves it inexecArgv. Both causerunRestoreChild(dataLayer/restoreChildRunner.ts:145-256) to fail immediately onfork()(RestoreChildError: The restore child running ... exited with code ... without a result).
Reproduction
Tested at HarperFast/harper origin/main SHA 45519a70df369e4fdc1192fb3af4a7cbc05d3bc4 (with promote-candidates/P-6777/qa5308-restore-child-blob-roots-and-exec-argv.test.js from skills-internal em-state state/general/):
# Default characterization mode (exits 0 on both engines, 4/4 passing):
HARPER_STORAGE_ENGINE=rocksdb npx mocha --no-config --exit --timeout 60000 unitTests/qa-scratch/general/qa5308-restore-child-blob-roots-and-exec-argv.test.js
HARPER_STORAGE_ENGINE=lmdb npx mocha --no-config --exit --timeout 60000 unitTests/qa-scratch/general/qa5308-restore-child-blob-roots-and-exec-argv.test.js
# Contract mode (exits 3 on rocksdb with Arms 1, 2, 3 failing; exits 2 on lmdb with Arms 1, 3 failing):
QA_ASSERT_MODE=contract HARPER_STORAGE_ENGINE=rocksdb npx mocha --no-config --exit --timeout 60000 unitTests/qa-scratch/general/qa5308-restore-child-blob-roots-and-exec-argv.test.js
QA_ASSERT_MODE=contract HARPER_STORAGE_ENGINE=lmdb npx mocha --no-config --exit --timeout 60000 unitTests/qa-scratch/general/qa5308-restore-child-blob-roots-and-exec-argv.test.jsProposed Fix (Verified Counterfactual in archive/QA-5308/counterfactual.patch)
utility/environment/environmentManager.ts(setPropertyat:128-130):
UpdateinstallProps[hdbTerms.HDB_SETTINGS_NAMES.HDB_ROOT_KEY] = valuewhencanonicalName === hdbTerms.CONFIG_PARAMS.ROOTPATHin addition toif (installPropsToSave[propName]), so setting or replaying either'HDB_ROOT'or canonicalized'rootPath'keepsgetHdbBasePath()synchronized.resources/databases.ts(openRestoredGenerationat:1926-1958) anddataLayer/restoreChildCommands.ts(proveat:181-183):
Add optionalblobRoots?: string[]toopenRestoredGeneration(path: string, databaseName: string, blobRoots?: string[]), callif (blobRoots) databasePaths.set(rootStore as unknown as RootDatabase, blobRoots);beforeinitStores(cleaned up automatically bycloseRestoredGeneration->closeBranchHandlesatresources/databases.ts:2824), and passargs.blobRootsinprove()(openRestoredGeneration(args.databasePath, args.databaseName, args.blobRoots)).dataLayer/restoreChildRunner.ts(childExecArgvat:84-104):
Strip attached/^-[ep].+/flags, strip--debug-port*flags alongside--inspect*and--max-old-space-size*, and skip the space-separated value token for--max-old-space-size,--max-old-space-size-percentage,--inspect-port,--debug-port, and--inspect-publish-uid.
With this 4-file patch applied, QA_ASSERT_MODE=contract passes 4/4 on both rocksdb and lmdb, and all 22 existing tests in unitTests/dataLayer/restoreChild.test.js and unitTests/resources/openRestoredGeneration.test.js pass (22/22).
Existing Work Searched
- Searched
HarperFast/harperandHarperFast/harper-proissues and PRs (open and closed) forinitializeRestoreChildEnvironment,childExecArgv,assessBlobReferences,openRestoredGeneration,restoreChild, andaccept_unverified_blobs. - Found PRs
harper#3825andharper#3920(which introducedrestoreChildRunner.ts,restoreChildCommands.ts,restorePublication.ts, andassessBlobReferences), open PRharper#4099(HNSW native index rebuild after restore,F-1164), and open issueharper#2637(epic for shared publication protocol and online restore). No issue or PR tracks any of the three defects inF-1182. - Checked documentation at
HarperFast/documentationd602fab2b54b9cadfc27e103b0ceeba57c58b07eandHarperFast/harper45519a70df369e4fdc1192fb3af4a7cbc05d3bc4:docs/reference/backups/operations.md:141-152, which states that every file-backed blob a record references in a sound backup verifies during proof and only a reference to a blob the backup does not contain fails the proof unlessaccept_unverified_blobsis set.
Scope / What This Does Not Prove
- Arm 1 occurs when
HDB_ROOT/rootPathis overridden in-process viasetProperty()(whichgetConfigOverrides()propagates to worker threads and the restore child as{ rootPath: ... }) rather than whenrootPathis read unmodified from the on-diskharperdb-config.yamlandlogging.rotationhas not yet triggeredlogRotator.ts's 100ms timer. - Arm 2 occurs whenever
args.blobRootspassed toapply-and-provediffers fromgetBlobPathsForDatabaseName(args.databaseName)in the restore child (including via Arm 1 or any caller supplying customblobRoots). - Arm 3 occurs when the parent Node process was launched with space-separated
--inspect-publish-uid <val>,--max-old-space-size-percentage <val>,--debug-port <port>, or attached-e<script>/-p<script>inprocess.execArgv, not when=-joined forms (--inspect-publish-uid=stderr,--max-old-space-size-percentage=50) are used.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity