Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@
"test:unit:resources": "mocha \"unitTests/resources/**/*.js\" --exclude \"unitTests/resources/**/*.bench.js\"",
"test:unit:resources:core": "mocha \"unitTests/resources/**/*.js\" --exclude \"unitTests/resources/**/*.bench.js\" --exclude \"unitTests/resources/**/*{[Ii]ndex,[Hh]nsw,[Vv]ector}*.js\"",
"test:unit:resources:indexes": "mocha \"unitTests/resources/**/*{[Ii]ndex,[Hh]nsw,[Vv]ector}*.js\" --exclude \"unitTests/resources/**/*.bench.js\"",
"bench": "mocha \"unitTests/resources/caching-rocks-database.bench.js\" \"unitTests/resources/indexRangeSelect.bench.js\" \"unitTests/resources/originClosedFloor.bench.js\" \"unitTests/resources/liveSubscriptionReauth.bench.js\"",
"bench": "mocha \"unitTests/resources/caching-rocks-database.bench.js\" \"unitTests/resources/indexRangeSelect.bench.js\" \"unitTests/resources/originClosedFloor.bench.js\" \"unitTests/resources/subscriptionCheckpoint.bench.js\" \"unitTests/resources/liveSubscriptionReauth.bench.js\"",
"test:unit:bin": "mocha \"unitTests/bin/**/*.js\"",
"test:unit:apitests": "node ./dist/bin/harper.js stop && mocha \"unitTests/apiTests/**/*-test.mjs\"",
"test:unit:logging": "mocha \"unitTests/utility/logging/*.js\"",
Expand Down
5 changes: 3 additions & 2 deletions resources/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -473,7 +473,7 @@ A replication receiver feeds one source subscription per database from every con

A transaction-log entry's identity is (`txnLogKey`, `nodeId`): the key alone repeats across origins, and a receiver files a replicated transaction under the origin's key. `RocksTransactionLogStore.put` therefore files an origin's entries in that origin's own log, created on first use, never in the relaying peer's log, so a key stays unique within a log (exact resume, `resumePastExactStart`, depends on it); within `local` it is unique because every key is minted there ("Origin-closed timestamp floor" above). A relayed entry whose origin id has no node name is rejected rather than misfiled; an unnamed id that was not relayed (`viaNodeId` equal to `nodeId` or absent) is not a replication origin and stays in `local`, under a minted key.

A failed transaction on a tagged stream (a non-retryable commit failure, including a segment closed by a later `beginTxn`) is reported at that stream's next `end_txn`: when the `end_txn` carries `onFailure(error, txnLogKey)`, neither its `onCommit` nor its sequence id passes the failure, and `onFailure` returning true _holds_ the stream β€” every later `end_txn` on it commits its writes without `onCommit` or a cursor, because the source is reconnecting to replay from its durable cursor under a new token (a held token is never reused). A held failure is not reported to the apply-failure listeners, since the replay applies it; a failure the source moves past (`onFailure` false, or no `onFailure`) is. An `end_txn` without `onFailure` keeps the older behavior of recording its cursor, so holding is opt-in per source. `txnStream` must be an object: streams are held in a `WeakMap`, so a closed connection's state is collected with its token.
A failed transaction on a tagged stream (a non-retryable commit failure, including a segment closed by a later `beginTxn`) is reported at that stream's next `end_txn`: when the `end_txn` carries `onFailure(error, txnLogKey)`, neither its `onCommit` nor its sequence id passes the failure, and `onFailure` returning true _holds_ the stream β€” every later `end_txn` on it commits its writes without `onCommit` or a cursor, because the source is reconnecting to replay from its durable cursor under a new token (a held token is never reused). A held failure is not reported to the apply-failure listeners, since the replay applies it; a failure the source moves past (`onFailure` false, or no `onFailure`) is. An `end_txn` without `onFailure` keeps the older behavior of recording its cursor, so holding is opt-in per source. When the transaction committed, and any `onCommit` ran, but the sequence-row write itself failed, `onFailure(error, txnLogKey, true)` runs before the error propagates. The third argument tells the source that the commit path completed, so nothing it releases on commit is still held; only what it attached for recording (cursors, floors, route state) was lost. The stream's last sequence id is rolled back, so a repeat of that position writes again. `txnStream` must be an object: streams are held in a `WeakMap`, so a closed connection's state is collected with its token.

An `end_txn` can also carry `originCursors: [nodeId, originLogKey][]`, the highest origin log key the source has durably applied for each origin. The apply loop merges them, keeping the max, into the connection's `[seq, remoteNodeIds[0]]` row as `nodes[].originLogKey`, even when `localTime` did not move (harper-pro#989; the per-origin cursor vector of harper-pro's replication workstream W4, harper-pro#434). A source may attach the array from `onCommit`, and the loop reads it only after awaiting `onCommit`. A held stream records neither the sequence id, origin cursors nor origin floors. A source that sends origin cursors must tag its stream and pass `onFailure`: without them, a failed segment's later `end_txn` still merges its cursor vector, so a cursor would pass the failed transaction; floors from such a source are dropped (below). The same `end_txn` can carry `originFloors: [nodeId, closedFloor, relayable][]` β€” an origin's closed timestamp floor (harper-pro#922 item 2; the producer is "Origin-closed timestamp floor" above), merged by max into `nodes[].closedFloor` with `nodes[].relayable` (whether the source received that origin directly with full table coverage, so a relay may forward the floor). It is kept apart from `originLogKey`: an applied position can sit above a transaction still open at the origin, a floor cannot, so only the floor is proof that resume may start there. A floor-only `end_txn` writes the row like a cursor-only one. Floors are merged only from a tagged stream that passes `onFailure` β€” core enforces for floors what it only documents for cursors, since a stored floor is what resume trusts β€” and an entry whose floor is not a positive finite number is skipped.

Expand Down Expand Up @@ -638,7 +638,8 @@ A primary key declared on an existing keyless table cannot re-key its records. S

A `reportProgress` subscription on RocksDB exposes `checkpoint(holds?)`: the issue's compact form `{ v, scope, time, overlap, exceptions, members }`, where `S = time - overlap` is the inclusive replay boundary of every member not in `exceptions` and `members` is the XOR of `sha256(origin, originGeneration)` over every member, so healthy origins are neither named nor counted while the set is still bound. The per-origin vector stays internal: `B_O = min(F_O, oldest event of O handed to the queue and not yet taken, holds[O])`, encoded with `time = max B`, `S = max(min B, time - MAX_OVERLAP_MS)`; lowering any boundary only adds duplicates. `F_O` is certified by the broadcaster (`startOriginTracking`): every 5 s it captures the qualifying floors (`captureCertifiedFloors`: the own published floor, and every seq-row origin with a level-2 `closedFloor`, `originGeneration`, `fullCoverage`, `certifiedFrom`, no `partialHistory`, `hole` or `floorRevoked`) and schedules a notify pass; the capture a fresh pass began with (`passCapture`) is published as `certifiedFloors` only when that pass drains without a failed read, since every entry below a captured floor was in the log before the capture and the drained pass dispatched it. **Strict availability:** one known origin (own, any seq row, any log) without a qualifying floor means no capture, so `checkpoint()` is undefined and MQTT keeps today's scalar.

- **Admission** (`admitCheckpoint`, `subscribe({ resume })`, 400/409/410): the scope string must match (database, table, table generation, target, delivery mode, `filterKey`); membership is resolved by subtracting at most four candidates whose route here began at the origin's first retained entry (`certifiedFrom === 0`, log retains sequence 1 β€” a node that joined after the checkpoint, admitted at boundary 0) from this node's candidates, at most 100 000 combinations; every member boundary must be at or above the resume floor, its origin not revoked, and held here by a full-coverage route with `certifiedFrom ≀ B_O`. Admission facts are rechecked at each replay yield against `auditStore.originStateEpoch`, which the seq-row merge bumps on any qualifier change. The replay seeks each member's log inclusively (`startByLog`, snapshot false, `includeLogName`), a record target filtered by exact key identity, and the subscription is `ungated`: the broadcaster's `startTime >= key` gate would drop a late lower-key commit, the omission harper#2928 describes. The boundary during the replay is the token's; after it, the certified floors (every entry below one appended before the replay ended was read by it, or dispatched live). **Qualifier lifetimes** (the seq-row merge): a revocation (a downgrade, or an applied-below-floor report) and a hole are sticky within an origin generation; a certificate naming a new origin generation starts that origin over (floor, revocation and `certifiedFrom` cleared, so the route proves its prefix again), and a completed base copy clears `partialHistory` and `hole`. `certifiedFrom` is the start of the route that certified the floor (two whole routes can cover disjoint intervals, so another row's start proves nothing about this floor), and a whole-log replay (a late joiner, a zero boundary) needs that same route to have begun at the origin's first entry, since another whole route can cover a disjoint interval; of two generations of one origin the one whose rows saw the highest floor is current (floors sample the origin's clock; a revoked row keeps the floor it was revoked at; a removed peer's retained row is frozen at its death), and the floor, revocations and holes count only from rows of that generation; a queued event without an origin tag holds every boundary, except a snapshot row, which is state rather than a log position; a repeated copy-completion mark clears a hole whether or not `partialHistory` changed; a certificate without a generation on a row that had one clears its level-2 qualifiers (that route certifies at level 1 now). A certified subscription that resumed from a scalar position delivers twice what its replay read and the live dispatch runs after it went live, and never what committed below its position before then: a scalar resume keeps main's exposure. A route mark or revocation names the generation it was derived under and is ignored once the row's generation moved on. Revocations and holes count only from rows of the current generation, so a dead peer's row for an earlier generation cannot hold the origin back. The epoch lives in a shared buffer (`bumpOriginStateEpoch`, after the row commits), so a recheck on any thread sees it. Every certified subscription is `ungated` once live, not only a checkpoint resume: the scalar-resumed MQTT session a rolling upgrade carries would otherwise drop a late lower-key commit the next floor certifies as delivered.
- **Admission** (`admitCheckpoint`, `subscribe({ resume })`, 400/409/410): the scope string must match (database, table, table generation, target, delivery mode, `filterKey`); membership is resolved by subtracting at most four candidates whose route here began at the origin's first retained entry (`certifiedFrom === 0`, log retains sequence 1 β€” a node that joined after the checkpoint, admitted at boundary 0) from this node's candidates, at most 100 000 combinations; every member boundary must be at or above the resume floor, its origin not revoked, and held here by a full-coverage route with `certifiedFrom ≀ B_O`. Admission facts are rechecked at each replay yield against `auditStore.originStateEpoch`, which the seq-row merge bumps on any qualifier change and a copy-bound writer bumps after its commit. The replay seeks each member's log inclusively (`startByLog`, snapshot false, `includeLogName`), a record target filtered by exact key identity, and the subscription is `ungated`: the broadcaster's `startTime >= key` gate would drop a late lower-key commit, the omission harper#2928 describes. The boundary during the replay is the token's; after it, the certified floors (every entry below one appended before the replay ended was read by it, or dispatched live). **Qualifier lifetimes** (the seq-row merge): a revocation (a downgrade, or an applied-below-floor report) and a hole are sticky within an origin generation; a certificate naming a new origin generation starts that origin over (floor, revocation and `certifiedFrom` cleared, so the route proves its prefix again), and a completed base copy clears `partialHistory` and `hole`. `certifiedFrom` is the start of the route that certified the floor (two whole routes can cover disjoint intervals, so another row's start proves nothing about this floor), and a whole-log replay (a late joiner, a zero boundary) needs that same route to have begun at the origin's first entry, since another whole route can cover a disjoint interval; of two generations of one origin the one whose rows saw the highest floor is current (floors sample the origin's clock; a revoked row keeps the floor it was revoked at; a removed peer's retained row is frozen at its death), and the floor, revocations and holes count only from rows of that generation; a queued event without an origin tag holds every boundary, except a snapshot row, which is state rather than a log position; a repeated copy-completion mark clears a hole whether or not `partialHistory` changed; a certificate without a generation on a row that had one clears its level-2 qualifiers (that route certifies at level 1 now). A certified subscription that resumed from a scalar position delivers twice what its replay read and the live dispatch runs after it went live, and never what committed below its position before then: a scalar resume keeps main's exposure. A route mark or revocation names the generation it was derived under and is ignored once the row's generation moved on. Revocations and holes count only from rows of the current generation, so a dead peer's row for an earlier generation cannot hold the origin back. The epoch lives in a shared buffer (`bumpOriginStateEpoch`, after the row commits), so a recheck on any thread sees it. Every certified subscription is `ungated` once live, not only a checkpoint resume: the scalar-resumed MQTT session a rolling upgrade carries would otherwise drop a late lower-key commit the next floor certifies as delivered.
- **One writer per `[seq, P]` row; copy bounds are applied by the reader.** `updateRecordedSequenceId`, in route P's apply loop, is the only writer of `[seq, P]`: it reads the row and commits it from that snapshot with no cross-thread lock, so any other writer's change can be overwritten (rocksdb-js `occValidation: 'parallel'` does not flag a plain put landing between a commit's conflict check and its write). A source changes its own row only through `end_txn` fields. Route state (`originRouteStates`) is `[id, certifiedFrom, partialHistory, hole, generation?, fullCoverage?]`; `fullCoverage: true` restores a level-2 route's coverage after a copy that ran with partial coverage. A revocation naming no generation against a node with none clears its level-1 floor rather than revoking it, since there is no generation for the revocation to end with. An audit-less base copy (`isCopyApply`) leaves this node's log of each copied origin incomplete below the highest version it wrote, a database-wide fact a route row cannot carry, because the route that certified may be disconnected and has no writer. So the copying source records that version per origin in its own `[copyBound, peerId]` row (`{ [originId]: version }`, committed before the copied row is applied), and `readOriginStates` treats a route's `certifiedFrom` at or below any peer's bound for that origin as absent (`readCopyBounds`; a malformed row throws a 410, so admission refuses and capture issues nothing). A bound for this node's own origin (a copy restoring its own lost rows) raises the own start above it, so an own boundary at or below it, or a whole-log replay, is refused. A bound commit bumps the origin-state epoch, and a capture carries the epoch it read before its scans: `certifiedFloors` drops a capture the epoch has since passed, and an admitted replay rechecks. The two halves ship together: a core without the bound reader admits a route whose start a copy has since voided. Pinned by `subscribeCheckpoint.test.js` and `replicationSeqCursor.test.js`.
- **Not covered:** SSE/WS resume (no transport exposes a token; harper#2448 deferred them too); an origin two hops away (one-hop relay) or reached only over a partial route never qualifies; `eventFilter`/`rowFilter` without `filterKey` issues no token; the own floor qualifies only on an engine with `syncWal`. Pinned by `subscribeCheckpoint.test.js` and `durableSessionCheckpoint.test.js`.

## Subscription version selection (`Table.ts`)
Expand Down
Loading
Loading