Skip to content

Restore child: rootPath override, blob roots and exec args not carried into the child process #4165

Open

Description

@kriszyp

Summary

Residual of #3920. restoreChildCommands.ts (~19-24) replays rootPath via setProperty, which does not update installProps (environmentManager.ts ~128-130), so getHdbBasePath stays stale; prove() (~182) does not pin databasePaths to args.blobRoots (openBranchDatabase does, databases.ts ~2693); childExecArgv (restoreChildRunner.ts ~84-104) leaves orphan values for --max-old-space-size-percentage and --inspect-publish-uid, keeps --debug-port and attached -e/-p. Fails in prove after apply purged the target, leaving 'incomplete'; a rerun recovers. Arm 3 alone is P3.

Triage (finding-triage wave 50) re-verified the mechanism by reading harper main c7572a5. Detail from the QA finding(s) follows; line numbers there may predate main.

QA finding F-1182

Summary

In the restore-child architecture introduced by harper#3920 (f5819f76e666d0ec8b81e17897960c4e7ede7b1d), three coupled defects in utility/environment/environmentManager.ts, dataLayer/restoreChildCommands.ts, resources/databases.ts, and dataLayer/restoreChildRunner.ts cause restore_database / restoreBackupOffline / publishRestore to fail after phase 'apply' has already purged the target database directory (mode: 'purgeAllFiles', leaving the target wedged in checkRestoreState(srcDir) === 'incomplete') or prevent runRestoreChild from spawning the restore child at all:

  1. Arm 1 (initializeRestoreChildEnvironment + setProperty leaves getHdbBasePath() unsynced on canonicalized rootPath overrides):

    • setProperty(propName, value) (utility/environment/environmentManager.ts:97-133) canonicalizes 'HDB_ROOT' (HDB_SETTINGS_NAMES.HDB_ROOT_KEY at utility/hdbTerms.ts:467) via CONFIG_PARAM_MAP['hdb_root'] (utility/hdbTerms.ts:901-903) to 'rootPath' (CONFIG_PARAMS.ROOTPATH at utility/hdbTerms.ts:795), so getConfigOverrides() (utility/environment/environmentManager.ts:141-148) returns { rootPath: isolatedRoot } with overrides.HDB_ROOT === undefined.
    • However, setProperty (utility/environment/environmentManager.ts:128-130) only updates installProps[propName] when installPropsToSave[propName] (utility/environment/environmentManager.ts:20-25) is truthy ('HDB_ROOT', not 'rootPath').
    • Unlike worker threads (where initSync() at utility/environment/environmentManager.ts:300-324 replays workerData.configOverrides via applyInheritedConfigOverrides() before syncing installProps[HDB_ROOT_KEY] = configUtils.getConfigValue(HDB_ROOT_KEY) at :319-323), the restore child is a fork()ed child process (workerData === null).
    • initializeRestoreChildEnvironment(spec) (dataLayer/restoreChildCommands.ts:19-24, called from dataLayer/restoreChild.ts:67-70) calls initSync() (:20) before replaying spec.configOverrides (for (const [name, value] of Object.entries(spec.configOverrides ?? {})) setProperty(name, value); at :21). Because spec.configOverrides carries canonical key 'rootPath' rather than 'HDB_ROOT', replaying setProperty('rootPath', isolatedRoot) updates configUtils (get('rootPath') === isolatedRoot) while leaving installProps[HDB_ROOT_KEY] (getHdbBasePath() at utility/environment/environmentManager.ts:48-50) pinned to the pre-replay on-disk rootPath (unless utility/logging/harper_logger.ts:926-943's unrelated 100ms logRotator.ts timer happens to fire first when logging.rotation is enabled on disk).
    • When restoreBackupOffline (dataLayer/rocksdbBackup.ts:1278-1313) / publishRestore (dataLayer/restorePublication.ts:230-381) restores a blob-bearing database under an HDB_ROOT/rootPath override, phase 'apply' (dataLayer/restoreChildCommands.ts:120-136) purges args.databasePath (backups.restore(..., { mode: 'purgeAllFiles' }) at :128) and extracts the blob snapshot into <isolatedRoot>/blobs/<db> (restoreBlobSnapshot(..., args.blobRoots, { durable: true }) at :130-132), and phase 'prove' (dataLayer/restoreChildCommands.ts:181-208) resolves getBlobPathsForDatabaseName(args.databaseName) (resources/blob.ts:2818-2824) against the child's stale getHdbBasePath(), misclassifying every restored blob as state: 'missing' and aborting with 409 ClientError (phase: 'prove'), leaving the target database destroyed and wedged with checkRestoreState(srcDir) === 'incomplete' (dataLayer/restorePublication.ts:394-455).
  2. Arm 2 (prove() and openRestoredGeneration() do not pin generation.rootStore to args.blobRoots in databasePaths):

    • In applyAndProve(args) (dataLayer/restoreChildCommands.ts:108-138), phase 'apply' restores the blob snapshot into args.blobRoots (restoreBlobSnapshot(args.backupDir, args.backupId, args.sourceDatabase, args.blobRoots, { durable: true }) at :130-132), whereas phase 'prove' (dataLayer/restoreChildCommands.ts:181-223) calls openRestoredGeneration(args.databasePath, args.databaseName) (:182) without passing args.blobRoots.
    • Unlike openBranchDatabase (resources/databases.ts:2685-2690), which pins if (blobRoots) databasePaths.set(rootStore as unknown as RootDatabase, blobRoots); (cleaned up by closeBranchHandles at resources/databases.ts:2824, which closeRestoredGeneration at resources/databases.ts:1961-1963 already calls), openRestoredGeneration (resources/databases.ts:1926-1958) leaves databasePaths.get(generation.rootStore) undefined.
    • When assessBlobReferences(generation.tables) (resources/blob.ts:4022-4082) calls getFilePath(storageInfo) (resources/blob.ts:2862-2871) -> getRootBlobPathsForDB(generation.rootStore) (resources/blob.ts:2794-2809), databasePaths.get(generation.rootStore) is undefined, so it falls back to getBlobPathsForDatabaseName(args.databaseName) (resources/blob.ts:2808, 2818-2824) instead of args.blobRoots.
    • Whenever args.blobRoots differs from getBlobPathsForDatabaseName(args.databaseName), phase 'prove' fails with 409 ClientError (...: missing) after phase 'apply' has already destroyed args.databasePath (or, with acceptUnverifiedBlobs: true, falsely downgrades evidence.blobs.blob_integrity to 'unverified' with waived: true and failures: [{ state: 'missing' }] at dataLayer/restoreChildCommands.ts:194-205 even though the restored blob file in args.blobRoots is intact).
  3. Arm 3 (childExecArgv leaves orphan positional tokens for space-separated flags and keeps attached -e/-p scripts):

    • childExecArgv(parentExecArgv) (dataLayer/restoreChildRunner.ts:84-104) strips any flag starting with --max-old-space-size or --inspect (:96), but only skips the following argument token when arg === '--max-old-space-size' || arg === '--inspect-port' (:98), and only matches -e / -p when passed as exact standalone elements of SCRIPT_FLAGS (:85, :92).
    • When parentExecArgv contains valid space-separated Node flags --inspect-publish-uid stderr, --max-old-space-size-percentage 50, or --debug-port 9229, childExecArgv strips the flag (or keeps --debug-port 9229) and leaves 'stderr' / '50' in execArgv as a bare positional script path before the restore child script; and when parentExecArgv contains attached -e<script> or -p<script>, childExecArgv leaves it in execArgv. Both cause runRestoreChild (dataLayer/restoreChildRunner.ts:145-256) to fail immediately on fork() (RestoreChildError: The restore child running ... exited with code ... without a result).

Reproduction

Tested at HarperFast/harper origin/main SHA 45519a70df369e4fdc1192fb3af4a7cbc05d3bc4 (with promote-candidates/P-6777/qa5308-restore-child-blob-roots-and-exec-argv.test.js from skills-internal em-state state/general/):

# Default characterization mode (exits 0 on both engines, 4/4 passing):
HARPER_STORAGE_ENGINE=rocksdb npx mocha --no-config --exit --timeout 60000 unitTests/qa-scratch/general/qa5308-restore-child-blob-roots-and-exec-argv.test.js
HARPER_STORAGE_ENGINE=lmdb npx mocha --no-config --exit --timeout 60000 unitTests/qa-scratch/general/qa5308-restore-child-blob-roots-and-exec-argv.test.js

# Contract mode (exits 3 on rocksdb with Arms 1, 2, 3 failing; exits 2 on lmdb with Arms 1, 3 failing):
QA_ASSERT_MODE=contract HARPER_STORAGE_ENGINE=rocksdb npx mocha --no-config --exit --timeout 60000 unitTests/qa-scratch/general/qa5308-restore-child-blob-roots-and-exec-argv.test.js
QA_ASSERT_MODE=contract HARPER_STORAGE_ENGINE=lmdb npx mocha --no-config --exit --timeout 60000 unitTests/qa-scratch/general/qa5308-restore-child-blob-roots-and-exec-argv.test.js

Proposed Fix (Verified Counterfactual in archive/QA-5308/counterfactual.patch)

  1. utility/environment/environmentManager.ts (setProperty at :128-130):
    Update installProps[hdbTerms.HDB_SETTINGS_NAMES.HDB_ROOT_KEY] = value when canonicalName === hdbTerms.CONFIG_PARAMS.ROOTPATH in addition to if (installPropsToSave[propName]), so setting or replaying either 'HDB_ROOT' or canonicalized 'rootPath' keeps getHdbBasePath() synchronized.
  2. resources/databases.ts (openRestoredGeneration at :1926-1958) and dataLayer/restoreChildCommands.ts (prove at :181-183):
    Add optional blobRoots?: string[] to openRestoredGeneration(path: string, databaseName: string, blobRoots?: string[]), call if (blobRoots) databasePaths.set(rootStore as unknown as RootDatabase, blobRoots); before initStores (cleaned up automatically by closeRestoredGeneration -> closeBranchHandles at resources/databases.ts:2824), and pass args.blobRoots in prove() (openRestoredGeneration(args.databasePath, args.databaseName, args.blobRoots)).
  3. dataLayer/restoreChildRunner.ts (childExecArgv at :84-104):
    Strip attached /^-[ep].+/ flags, strip --debug-port* flags alongside --inspect* and --max-old-space-size*, and skip the space-separated value token for --max-old-space-size, --max-old-space-size-percentage, --inspect-port, --debug-port, and --inspect-publish-uid.

With this 4-file patch applied, QA_ASSERT_MODE=contract passes 4/4 on both rocksdb and lmdb, and all 22 existing tests in unitTests/dataLayer/restoreChild.test.js and unitTests/resources/openRestoredGeneration.test.js pass (22/22).

Existing Work Searched

  • Searched HarperFast/harper and HarperFast/harper-pro issues and PRs (open and closed) for initializeRestoreChildEnvironment, childExecArgv, assessBlobReferences, openRestoredGeneration, restoreChild, and accept_unverified_blobs.
  • Found PRs harper#3825 and harper#3920 (which introduced restoreChildRunner.ts, restoreChildCommands.ts, restorePublication.ts, and assessBlobReferences), open PR harper#4099 (HNSW native index rebuild after restore, F-1164), and open issue harper#2637 (epic for shared publication protocol and online restore). No issue or PR tracks any of the three defects in F-1182.
  • Checked documentation at HarperFast/documentation d602fab2b54b9cadfc27e103b0ceeba57c58b07e and HarperFast/harper 45519a70df369e4fdc1192fb3af4a7cbc05d3bc4:docs/reference/backups/operations.md:141-152, which states that every file-backed blob a record references in a sound backup verifies during proof and only a reference to a blob the backup does not contain fails the proof unless accept_unverified_blobs is set.

Scope / What This Does Not Prove

  • Arm 1 occurs when HDB_ROOT / rootPath is overridden in-process via setProperty() (which getConfigOverrides() propagates to worker threads and the restore child as { rootPath: ... }) rather than when rootPath is read unmodified from the on-disk harperdb-config.yaml and logging.rotation has not yet triggered logRotator.ts's 100ms timer.
  • Arm 2 occurs whenever args.blobRoots passed to apply-and-prove differs from getBlobPathsForDatabaseName(args.databaseName) in the restore child (including via Arm 1 or any caller supplying custom blobRoots).
  • Arm 3 occurs when the parent Node process was launched with space-separated --inspect-publish-uid <val>, --max-old-space-size-percentage <val>, --debug-port <port>, or attached -e<script> / -p<script> in process.execArgv, not when =-joined forms (--inspect-publish-uid=stderr, --max-old-space-size-percentage=50) are used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions