Branched databases: blob store sharing, allocator, and GC safety #644

Summary

Wire blob handling for branched databases (#642, #643) using a hard-link clone of the blob tree, taken alongside the RocksDB checkpoint. This is the decided design, replacing an earlier shared-store-plus-high-water-mark approach this issue previously described — that approach is not being built.

Design

A branch materializes as a checkpoint of the base's RocksDB store (resources/branchDatabase.ts, materializeBranch). RocksDB's createCheckpoint covers only the SST/WAL files; it does not touch the blob directories that live alongside the database (getRootBlobPathsForDB, resources/blob.ts:2074). This issue adds the missing half: when a branch is materialized, recursively hard-link the base's blob tree into the branch's own blob root.

The mechanism already exists and is proven: dataLayer/blobBackup.ts's linkOrCopy/copyTree (lines 58, 133) already walk a blob tree and hard-link each file, falling back to a real copy on EXDEV/EMLINK/EPERM/ENOTSUP (cross-filesystem, link-count exhaustion, or a filesystem that forbids hard links) — reuse it rather than re-implement it. resources/databases.ts:1299 already notes this same same-filesystem constraint for the RocksDB checkpoint itself, so a branch already degrades to a full byte copy off-volume; the blob clone should follow the same rule for the same reason.

Once cloned, everything else falls out of ordinary filesystem semantics — no allocator sharing, no high-water mark, no gated deletion logic:

• Blob directory: the branch gets its own blob root (<branch's blob path>, resolved the same way getBlobPathsForDatabaseName resolves any other database's), populated by the hard-link clone at branch-creation time. It is a real, independent directory — just one whose existing files share inodes with the base's.
• ID allocator: the branch gets its own allocator, exactly like any other database — no sharing with the base or with other branches. getNextFileId (resources/blob.ts:2179) already seeds by scanning the directory for the highest existing filename-derived ID; since the branch's directory is a full clone (same filenames as the base had at checkpoint time), it naturally seeds above every ID that existed at checkpoint, and its own directory is separate from every other branch's, so two branches allocating new IDs concurrently cannot collide with each other or with the base. No new allocator-sharing code is needed at all.
• Deletion / GC: the OS inode refcount is the reference count. When a branch overwrites or deletes a blob (deleteBlobsInObject, resources/blob.ts:2419), it unlinks its own directory entry; the base's link (and any other branch's) is untouched, and the underlying data is freed by the OS only once every link to it is gone. cleanupOrphans (resources/blob.ts:2792) needs no branch-awareness: it already only removes a directory entry for a blob the database's own records no longer reference, and unlinking that entry is exactly the "release my reference" operation — safe to run on the base while branches are alive, and safe to run on a branch independently.

This is materially simpler than the shared-allocator design this issue previously specified: there is no high-water mark to compute or carry on ApplicationScope, no gating logic in the delete path, and no suspend/resume protocol for orphan GC. The OS already implements the exact refcounting behavior that design built by hand.

Requirements

1. Clone the blob tree at branch materialization

In materializeBranch (resources/branchDatabase.ts), alongside createCheckpoint, hard-link-clone each of the base's blob roots into the corresponding branch blob root, using linkOrCopy/copyTree from dataLayer/blobBackup.ts (promote them to an exported, reusable utility rather than duplicating the walk).

2. Resolve the branch's own blob root

The branch's blob path resolver must key off the branch's own store identity (the same branchStoreName-derived identity openBranchDatabase already uses for the RocksDB store), not the base's — a branch is a real, independent database as far as blob storage is concerned. This is likely already correct by construction (getRootBlobPathsForDB keys off store.databaseName, and a branch's store carries its own identity) — verify rather than assume.

3. No allocator, HWM, or GC changes

None needed, per the design above. If verification finds a case where it is needed (e.g. the same-filesystem fallback degrading to a full copy changes filenames or timestamps in a way getNextFileId's scan cannot handle), that is the thing to fix — not a reason to build the shared-allocator/HWM/suspend-GC machinery this issue previously specified.

4. Replication

Unchanged: out of scope for the branch itself (branches are excluded from replication, per #643). Base replication continues normally; branch-allocated blobs never leave the local instance.

Acceptance criteria

• Branch can read every blob that existed in the base at checkpoint time (via the hard-linked clone, not the base's directory).
• A branch write that creates a new blob produces a file with an ID unique within the branch's own directory — no shared-directory collision to reason about, because the branch's directory is its own.
• A branch write that overwrites/deletes a blob reference removes only the branch's directory entry; the base's copy (and any other branch's) is unaffected, verified both via the base still reading the original content and the underlying file still existing on disk under the base's root.
• cleanupOrphans run on the base while a branch is alive does not remove any blob the branch still references (true by construction once directories are independent — verify, don't just assert).
• cleanupOrphans run on a branch reclaims blobs that branch no longer references, independently of the base's or any other branch's GC state.
• Off-volume / no-hard-link-support fallback: forcing linkOrCopy onto a path that cannot hard-link (mock EXDEV) still produces a fully readable branch blob tree via full copy.

Verification

Extend the integration test from #643 (integrationTests/components/branched-database.test.ts):

1. Base data contains a row whose value includes a blob (e.g. 16 KB payload, above the inline threshold).
2. Two branched apps appA, appB boot from the same base.
3. Both apps read the row; both successfully stream the blob content, and the underlying files are distinct inodes with the same content on disk (or the same inode, if hard-linked — either is correct; the point is no shared mutable directory).
4. appA overwrites the row with a new blob — assert the base's original blob file still exists and is unchanged on disk, and appB's read of the same row is unaffected.
5. appB overwrites the same original row with a different new blob — assert appA's independent overwrite is unaffected, and the base is unaffected by either.
6. Run cleanupOrphans against the base while both branches are alive — assert nothing either branch references is removed.
7. Close appA and run cleanupOrphans against appA's own branch — assert appA's no-longer-referenced blob is reclaimed, independent of appB and the base.

Key files

• resources/branchDatabase.ts — materializeBranch: add the blob-tree clone alongside createCheckpoint
• dataLayer/blobBackup.ts — linkOrCopy (line 58), copyTree (line 133): promote to a reusable exported utility
• resources/blob.ts — getRootBlobPathsForDB (2074), getBlobPathsForDatabaseName (2100), getNextFileId (2179), deleteBlobsInObject (2419), cleanupOrphans (2792): verify each behaves correctly against a branch's own independent, hard-linked directory — no changes expected, but verify rather than assume
• resources/databases.ts:1299 — existing note on the same same-filesystem hard-link constraint, for the checkpoint itself

🤖 Updated by Claude on behalf of Kris — replaces the shared-allocator/high-water-mark design below with the hard-link clone design that was actually decided (2026-08-30). Original issue body preserved in the first comment for reference.