Repository navigation
Description
@harperfast/rocksdb-js deduplicates native store descriptors by canonical path (weakly_canonical) but records only the first-opened spelling in entry.path. branchReferenceCount canonicalizes with realpathSync before comparing getRegistryStatus() paths; three sibling consumers do not, so a database held open under a symlinked spelling is invisible to them. Verified on main 9c43c7790 (triage, 2026-10-30): describeHolderCandidates (resources/longLivedTransactions.ts ~294-310, resolve(database.path) === target), resumeIncompleteDatabaseDrop and dropDatabase's detachedRootHasForeignHandle (resources/databases.ts ~3167 / ~3227, Set.has(entry.path)), and verifyDatabaseClosed (dataLayer/rocksdbBackup.ts ~804).
Severity: Harper opens its own databases with one consistent path string, so this needs a foreign handle (a component) that opened the same directory under a different spelling first; only then can dropDatabase or restore_backup delete or overwrite under a live handle. The describeHolderCandidates half is cosmetic (wrong on <path> suffix, missed self-exclusion). P3.
Suggested fix: one canonicalPath(p) helper (realpathSync, falling back to resolve(p) on ENOENT) next to canonicalOrSelf in resources/branchDatabase.ts or a shared util, applied to both sides at the four sites (for the two Sets, build the Set of canonical paths and compare canonicalPath(entry.path)). A 5-test unit spec (3 red on main, both engines) is staged and can be contributed with the fix.
Summary
@harperfast/rocksdb-js deduplicates native Store descriptors across the process by canonical path (realpath), so opening symlinkDir and realDir shares a single entry in getRegistryStatus() whose entry.path holds whichever path string was passed to the first RocksDatabase.open() call.
In commit 489eebcf0 (HarperFast/harper#3429), four consumers were migrated from registryStatus() to getRegistryStatus(). While branchReferenceCount (resources/branchDatabase.ts:587-608) canonicalizes both branchPath and entry.path via realpathSync (canonicalOrSelf), the other three consumers compare raw strings or path.resolve() strings without resolving symlinks:
describeHolderCandidates(resources/longLivedTransactions.ts:294-310, 326): Comparesresolve(database.path) === resolve(databasePath). Becausepath.resolve()does not resolve symlinks, whendatabase.pathingetRegistryStatus()and the caller'sdatabasePathdiffer by a symlink,sameDatabaseevaluates tofalse. At:309,if (!sameDatabase || handle.id !== excludeNativeId)fails to exclude the stuck transaction's ownexcludeNativeId(so the transaction lists itself as a conflicting holder candidate!) and:326appends a foreignon ${candidate.path}suffix to every handle on that database.resumeIncompleteDatabaseDrop(resources/databases.ts:2982-2985) anddropDatabasedetachedRootHasForeignHandle(resources/databases.ts:3040-3046, 3094-3095): Both checkpaths.has(entry.path)anddetachedRootPathSet.has(entry.path)via rawSet.prototype.hasstring equality withoutrealpathSync. When a detached root or incomplete-drop root is held open (refCount > 0) via a symlinked path,dropDatabaseandresumeIncompleteDatabaseDropfail to throwDatabaseClosingError(statusCode === 503,code === 'DATABASE_CLOSING') and executeawait rm(rootPath, { recursive: true, force: true })(resources/databases.ts:2958, 3095), deleting the database directory on disk underneath the live openRocksDatabasehandle.verifyDatabaseClosed(dataLayer/rocksdbBackup.ts:800-817, called byrestoreBackupat:684, :688): Comparesresolve(instance.path) === resolve(databaseDir)withoutrealpathSync. When a handle holds the target database open (refCount > 0) via a path differing by a symlink fromdatabaseDir,stillOpenevaluates tofalse,verifyDatabaseClosed()returns immediately without polling or throwingBackupInProgressError(409), andrestoreBackup()executesawait backups.restore(backupDir, databaseDir, { backupId, mode: 'purgeAllFiles' })(:691) underneath the live open handle.
Mechanism (file:line on HarperFast/harper d4f9fb948d8103283714ea8738db80618030cd43)
- Positive control (
resources/branchDatabase.ts:587-608):canonicalOrSelf(p)wrapsrealpathSync(resolve(p))with aresolve(p)fallback, andbranchReferenceCountcomparescanonicalOrSelf(entry.path) === resolved. - Defect 1 (
resources/longLivedTransactions.ts:294-310, 326)::294:const target = resolve(databasePath);:301:const sameDatabase = database.path != null && resolve(database.path) === target;:309-310:if (!sameDatabase || handle.id !== excludeNativeId) candidates.push({ id: handle.id, ageMs: handle.ageMs, sameDatabase, path: database.path });
- Defect 2 (
resources/databases.ts:2982-2985, 3040-3046, 3094-3095)::2982-2985:getRegistryStatus().some((entry) => paths.has(entry.path) && entry.refCount > 0 && !incompleteDatabaseDropStores.has(entry.path)):3040-3045:const detachedRootPathSet = new Set(detachedRootPaths); const detachedRootHasForeignHandle = () => getRegistryStatus().some((entry) => detachedRootPathSet.has(entry.path) && entry.refCount > 0 && !incompleteDatabaseDropStores.has(entry.path));:2958, :3095:await rm(rootPath, { recursive: true, force: true });
- Defect 3 (
dataLayer/rocksdbBackup.ts:684, 688, 691, 800-817)::801:const targetPath = resolve(databaseDir);:804-806:const stillOpen = getRegistryStatus().some((instance) => resolve(instance.path) === targetPath && instance.refCount > 0);:691:await backups.restore(backupDir, databaseDir, { backupId, mode: 'purgeAllFiles' });
Reproduction
On HarperFast/harper origin/main d4f9fb948d8103283714ea8738db80618030cd43, copy the staged QA spec (available on request) to unitTests/resources/registryStatusPathCanonicalization.test.js and run:
HARPER_STORAGE_ENGINE=rocksdb npx mocha --exit unitTests/resources/registryStatusPathCanonicalization.test.js
HARPER_STORAGE_ENGINE=lmdb npx mocha --exit unitTests/resources/registryStatusPathCanonicalization.test.js- Unpatched
d4f9fb948:2 passing, 3 failing(307msonrocksdb,316msonlmdb, exit code3). - Counterfactual fix (
canonicalOrResolvedPathviarealpathSync(resolve(p))withresolve(p)fallback inresources/longLivedTransactions.ts:294-301,resources/databases.ts:2982-2985, 3040-3046, anddataLayer/rocksdbBackup.ts:801-806):5 passing, 0 failing(3.3sonrocksdb,3.3sonlmdb, exit code0).
Existing Work Searched
Searched HarperFast/harper and HarperFast/harper-pro issues and PRs (open and closed) for getRegistryStatus, describeHolderCandidates, detachedRootHasForeignHandle, verifyDatabaseClosed, branchReferenceCount, and resumeIncompleteDatabaseDrop, plus the Dispatch Finding queue (qaLane=general, 162 rows):
HarperFast/harper#3359(closed issue) &HarperFast/harper#3429(489eebcf0, merged PR): mechanical migration fromregistryStatus()togetRegistryStatus().HarperFast/harper#2473(merged PR): addeddescribeHolderCandidatesinresources/longLivedTransactions.ts.HarperFast/harper#2802(open issue): tracksrestore_backup409 when a RocksDB database has another loaded alias under a different database name, not symlink path canonicalization inverifyDatabaseClosed.HarperFast/harper#2891(closed issue),HarperFast/harper#2649(merged PR),HarperFast/harper#1831(merged PR): drop fencing and backup/restore lifecycle; none tracksrealpathSynccanonicalization ingetRegistryStatus()path comparisons.
What This Does NOT Prove
- Commit
489eebcf0did not introduce the missingrealpathSynccalls as a new regression; it was a mechanical rename fromregistryStatus()togetRegistryStatus()(#3429), and the pre-existing comparisons inlongLivedTransactions.ts,databases.ts, androcksdbBackup.tsalready lackedrealpathSyncprior to489eebcf0. - Standard single-root Harper table operations where no symlink exists in
storage.path,databases.<name>.path, or component-opened paths are unaffected becauseresolveDatabasePathproduces a consistent path string. - For
describeHolderCandidates, the unit test injects registry entries viasetRegistryStatusForTestsrather than triggering a live multi-transaction commit-conflict timeout inDatabaseTransaction.ts.
— Claude Mythos 5.1 (finding triage)
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity