Skip to content

Branched databases: app config, scope proxy, lifecycle wiring #643

Closed

Description

@kriszyp

Summary

Harper-side implementation of branched databases. Subtask of #642; depends on HarperFast/rocksdb-js#577 for the underlying checkpoint primitive.

Scope

Everything inside `core/` that wires the feature together. No native code changes.

1. Application config schema

`core/components/Application.ts` — extend `ApplicationConfig` (lines 28–38) with:

```ts
branchedDatabases?: string[];
```

Validate in `assertApplicationConfig()` (lines 72–109): array of strings; each must name an existing database; reject duplicates.

2. ApplicationScope field

`core/components/ApplicationScope.ts` — add:

```ts
branches: Map<string, string>; // baseDbName → branchDbName
```

3. Branch creation on app load

Extend `prepareApplication()` in `core/components/Application.ts` (lines 429–480). After the `ApplicationScope` is constructed but before any of the app's modules import `harper`:

For each `baseName` in `config.branchedDatabases`:

  • Branch DB name: `${scope.appName}:${baseName}` (colon-namespaced in the `databases` map).
  • Branch path: `<storage_root>/database/${scope.appName}${baseName}` (use `` on disk; colon would be illegal on some filesystems).
  • Resolve the base `RocksDatabase` via `database({ database: baseName })` (`core/resources/databases.ts:740`).
  • `await baseDb.createCheckpoint(branchPath)`.
  • Open the checkpoint as a database via the same factory, override path; register under the namespaced key.
  • Record `scope.branches.set(baseName, '${scope.appName}:${baseName}')`.

The checkpoint is byte-identical to the base, so `internal_dbis/` metadata is correct and `makeTable()` (`core/resources/Table.ts:146`) works unchanged — Table classes get `primaryStore` pointing at the branch.

4. Scoped `databases` proxy in `getHarperExports`

`core/security/jsLoader.ts` (`getHarperExports` lines 678–716). Replace the static `databases` reference with a `Proxy` that consults `scope.branches`:

```ts
const scopedDatabases = scope.branches.size === 0
? databases
: new Proxy(databases, {
get(target, prop, recv) {
if (typeof prop === 'string' && scope.branches.has(prop)) {
return Reflect.get(target, scope.branches.get(prop)!, recv);
}
return Reflect.get(target, prop, recv);
},
has(target, prop) {
return (typeof prop === 'string' && scope.branches.has(prop)) || Reflect.has(target, prop);
},
});
```

Mirror the proxy treatment for the `tables` flat alias when the default database is branched.

5. Branch teardown

  • On `ApplicationScope` dispose / app unload: `branchDb.close()` then `fs.rm(branchPath, { recursive: true })`. Hardlinks mean the base's SST files remain intact.
  • Startup sweep in `core/resources/databases.ts` initialization: remove any directory matching the `__` pattern that doesn't correspond to a currently-loaded app, to clean up after crashes.

6. Replication exclusion

Branch DBs must not participate in cluster replication. Plumb through the existing non-replicating mechanism (cf. `NON_REPLICATING_SYSTEM_TABLES` pattern in `databases.ts:78`) so replication code skips them. Verify there is no code path that auto-discovers databases on disk and replicates them.

Acceptance criteria

  • App with `branchedDatabases: ['data']` boots; its `databases.data` is the branch.
  • Writes from one branched app are invisible to a second branched app and to the base.
  • Reads from an app without `branchedDatabases` see the unchanged base.
  • On clean shutdown, branch directories are removed.
  • On unclean shutdown, startup sweep removes orphaned branch directories.
  • Replication does not propagate branch writes.

Verification

Integration test under the component test suite:

  1. Define base `data` with table `Item` containing 3 rows.
  2. Deploy two test applications `appA` and `appB`, both with `branchedDatabases: ['data']`.
  3. From `appA` write `{id: 'a-only'}`; from `appB` write `{id: 'b-only'}`.
  4. Assert: `appA` reads 3 + `a-only`; `appB` reads 3 + `b-only`; a third unbranched app reads exactly 3.
  5. Restart process; assert branch directories cleaned up and base `data` still has exactly 3 rows.

🤖 Filed by Claude on behalf of Kris

Activity

  1. added this to the v5.3 milestone on Aug 7, 2026
  2. added theissue type on Aug 7, 2026
  3. self-assigned this
    on Aug 22, 2026
  4. kriszyp commented on Aug 22, 2026

    @kriszyp
    MemberAuthor

    Gating requirement for the scope-wiring work, carried over from the opener PR.

    openBranchDatabase (landing separately as the storage-side opener) returns a table graph whose Table classes carry the base's logical name — that is deliberate, so an application's existing schema and code resolve unchanged. The consequence is that a schema operation through a branch table, dropTable() in particular, resolves against the global schema and would delete the live base Table class.

    Row reads and writes are unaffected. Nothing calls the opener today, so there is no live exposure — but the scope wiring in this issue is the first thing that hands a branch to application code, so it must gate schema mutation before it does. Kris's call (2026-08-22) was to gate here rather than widen the opener PR into Table.

    Concretely, this issue's acceptance criteria gain:

    • A schema operation attempted through a branched databases.<name> fails loudly rather than mutating the base's schema — at minimum dropTable, and the same check applied to every schema-mutating path a branch table exposes.
    • A test proves it: dropping a table through a branch leaves the base's Table class intact and registered.

    The constraint is also recorded in the opener's own docstring so it cannot be missed by someone reading only the code.

    🤖 Comment by Claude (Opus 5) on behalf of Kris

  5. kriszyp commented on Aug 31, 2026

    @kriszyp
    MemberAuthor

    Shipped in #2352 — root-config branchedDatabases declaration (including true for every database except system), durable deterministic branch identity adopted across restarts, scoped databases/tables/defineTable bindings that leave unbranched applications on the process-wide singletons by identity, create-once claim coordination across worker threads, the branch→base DDL fence, and branch-scoped relationship hydration.

    Remaining branch work tracked separately: #644 (blob hard-link clone), #2264 (schema declarations landing in the branch), and branch removal on undeploy, all listed on the epic #642.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:componentsComponents / applications subsystemenhancementNew feature or request

Fields

Priority

P2

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions