Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
8d34e7b
Stamp record-lock holder writes above the state they were admitted on
kriszyp Oct 16, 2026
7405024
Claim a lock floor ahead of the clock as a unique local key
kriszyp Oct 16, 2026
46f9f9a
Persist a ceiling above lock claims so a restart cannot reissue them
kriszyp Oct 16, 2026
6589890
Cover every key the ratchet issues ahead of the clock, and keep past-…
kriszyp Oct 16, 2026
e5a7458
Abort an immediate handle whose key reservation throws, and drop a st…
kriszyp Oct 16, 2026
d4b78eb
Mint acquiredAt, tsR and passed-floor stamps through the local key al…
kriszyp Oct 16, 2026
1d6b673
Keep every lock-derived stamp off the next key the ratchet issues
kriszyp Oct 16, 2026
416dc9f
Keep allocator failures on the paths that can recover from them
kriszyp Oct 16, 2026
57dfe07
Upgrade before detaching on the coalesced path, and document the post…
kriszyp Oct 16, 2026
2fc2d40
Skip the claim-lead wait on the LMDB leg
kriszyp Oct 16, 2026
a4c11f1
Give the native key back if building the lock handle throws
kriszyp Oct 16, 2026
c6357da
Keep a hold's applied version across a retry that ties with its own e…
kriszyp Oct 20, 2026
db55888
Drop the lock stamp ceiling scan: each immediate write reserves its o…
kriszyp Oct 20, 2026
56c83d7
Count a retried write's version as applied only while its own record …
kriszyp Oct 20, 2026
4111fa9
Refuse a holder write that would join a commit reserved below its stamp
kriszyp Oct 20, 2026
d2091cc
Detach a holder write whose key reservation fails, and fail the test …
kriszyp Oct 20, 2026
0936ee8
detach holder write when lock stamp allocation fails
kriszyp Oct 23, 2026
594239b
reject retries after a refused lock write
kriszyp Oct 23, 2026
e818d90
fix(record-locks): reject unsupported admission skew
kriszyp Oct 23, 2026
cf8c598
fix(config): reject overflowing lock skew durations
kriszyp Oct 23, 2026
099fd04
fix(record-locks): fence every refused handle write
kriszyp Oct 23, 2026
8b330ce
fix(record-locks): preserve frozen transaction clocks
kriszyp Oct 25, 2026
1877bf2
test(record-locks): expect distinct local log key
codex Oct 25, 2026
37503d5
fix(record-locks): close reviewed ordering gaps
codex Oct 25, 2026
937d1d6
chore: clear inherited rebase validation blockers
kriszyp Oct 28, 2026
ff7c91d
test: repair inherited stream and startup fixtures
kriszyp Oct 28, 2026
0363ca9
docs(record-locks): align issuance and hold descriptions
kriszyp Oct 28, 2026
dfa2303
fix(record-locks): make issuance coverage durable before publication
kriszyp Oct 28, 2026
52887da
test(record-locks): exercise mapped remote origins
kriszyp Oct 28, 2026
afcc62c
fix(record-locks): contain issuance recovery and refusal failures
kriszyp Oct 28, 2026
d1b762e
fix(record-locks): detach siblings at every refusal site
kriszyp Oct 28, 2026
21c8675
fix(record-locks): honor hold-lock version floors across transactions
kriszyp Oct 29, 2026
b54d488
fix(record-locks): separate committed holder floor and keep refused h…
kriszyp Oct 29, 2026
e8f149e
fix(record-locks): detach sibling deferred hold writes on fixed-clock…
kriszyp Oct 29, 2026
bb52ccc
fix(record-locks): address round-3 review feedback on skew cap, ceili…
kriszyp Oct 29, 2026
3beac7f
fix(audit-store): throttle local issuance ceiling error logs and rest…
kriszyp Oct 29, 2026
9e9a47d
test: isolate qa5043 lock stamp table in its own database
kriszyp Oct 29, 2026
ac849f2
chore(docs): restore merge-base indexes DESIGN.md and fit resources D…
kriszyp Oct 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ A note lives in the `DESIGN.md` of the directory that owns the code it describes

## resources/record-locks.md — `table.lock()`

- [Record locks: the native key lock is the sole authority (`Table`/`DatabaseTransaction`/`recordLock`)](resources/record-locks.md#record-locks-the-native-key-lock-is-the-sole-authority-tabledatabasetransactionrecordlock) — The rocksdb-js process-wide key lock is the sole authority; node scope writes no lock state to the store or the audit log, and cluster scope adds only `lockRelease`/`lockBarrier` control entries.
- [Record locks: the native key lock is the sole authority (`Table`/`DatabaseTransaction`/`recordLock`)](resources/record-locks.md#record-locks-the-native-key-lock-is-the-sole-authority-tabledatabasetransactionrecordlock) — The rocksdb-js process-wide key lock is the sole authority; node scope writes no lock state to the store or the audit log, and cluster scope adds only `lockRelease`/`lockBarrier` control entries. A fresh admission stamps the holder's writes one ulp above the reloaded version and the predecessor positions it waited for, claiming that stamp as a unique local key within `storage.maxLockClockSkew`.
- [Phase 1 design: amortized per-record ownership](resources/record-locks.md#phase-1-design-amortized-per-record-ownership) — Operator-agreed home map, delegations with ordered fencing tokens, drain that revokes capability, and an exclusion-only guarantee; its §10 wording is normative for anything user-facing about `lock()`.
- [Successor freshness implementation](resources/record-locks.md#successor-freshness-implementation) — The home coordinator admits a successor only once its node has applied every origin-log position inherited from prior clean holders, else the weaker recovery barrier or fail closed.

Expand Down
4 changes: 4 additions & 0 deletions config-root.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -716,6 +716,10 @@
"type": "string",
"description": "Max write queue time before rejecting (e.g. '45s')."
},
"maxLockClockSkew": {
"type": ["string", "number"],
"description": "Maximum clock lead in which a record lock may admit stored state and advance the local transaction-log clock above it. Above the limit, lock() fails with LockUnavailableError until wall time catches up; check for a future context.timestamp or source peer clock skew, or wait. A string accepts ms, s, m, h, or d (e.g. '500ms' or '5s'); a number is seconds. Default: '5s'."
},
"longTransactionReportThreshold": {
"type": ["string", "number"],
"description": "How old a RocksDB transaction handle must be before it is reported in the log (e.g. '5m'). A handle this old can hold write intents that park other writers' commits and pin a read snapshot. Set to 0 to disable the reporting. Default: '5m'"
Expand Down
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@
"test:unit:resources:core:a-q": "mocha \"unitTests/resources/**/[a-qA-Q]*.js\" --exclude \"unitTests/resources/**/*.bench.js\" --exclude \"unitTests/resources/**/*{[Ii]ndex,[Hh]nsw,[Vv]ector}*.js\"",
"test:unit:resources:core:r-z": "mocha \"unitTests/resources/**/*.js\" --exclude \"unitTests/resources/**/*.bench.js\" --exclude \"unitTests/resources/**/*{[Ii]ndex,[Hh]nsw,[Vv]ector}*.js\" --exclude \"unitTests/resources/**/[a-qA-Q]*.js\"",
"test:unit:resources:indexes": "mocha \"unitTests/resources/**/*{[Ii]ndex,[Hh]nsw,[Vv]ector}*.js\" --exclude \"unitTests/resources/**/*.bench.js\"",
"bench": "mocha \"unitTests/resources/caching-rocks-database.bench.js\" \"unitTests/resources/indexRangeSelect.bench.js\" \"unitTests/resources/originClosedFloor.bench.js\" \"unitTests/resources/subscriptionCheckpoint.bench.js\" \"unitTests/resources/liveSubscriptionReauth.bench.js\"",
"bench": "mocha \"unitTests/resources/caching-rocks-database.bench.js\" \"unitTests/resources/indexRangeSelect.bench.js\" \"unitTests/resources/originClosedFloor.bench.js\" \"unitTests/resources/subscriptionCheckpoint.bench.js\" \"unitTests/resources/liveSubscriptionReauth.bench.js\" \"unitTests/resources/recordLockAdmission.bench.js\"",
"bench:record-lock-admission": "mocha unitTests/resources/recordLockAdmission.bench.js",
"test:unit:bin": "mocha \"unitTests/bin/**/*.js\"",
"test:unit:apitests": "node ./dist/bin/harper.js stop && mocha \"unitTests/apiTests/**/*-test.mjs\"",
"test:unit:logging": "mocha \"unitTests/utility/logging/*.js\"",
Expand Down
5 changes: 2 additions & 3 deletions resources/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,9 +217,8 @@ A RocksDB transaction's log key is minted when its native handle is created, or
- **Invariant:** every key that reaches `local` was reserved on its thread before its first staged write (`DatabaseTransaction.save`, and `reserveLocalKey` explicitly on the two derived-index eviction handles); the reservation keeps the thread's published bound at or below the key until the handle commits or is abandoned; a certifier publishes `F ≤ min(a clock sample taken first, every thread's bound)`. `RocksTransactionLogStore.put` throws on an unreserved `local` append, because a logged violation could repair nothing.
- **Shared state:** one `getUserSharedBuffer('origin-closed-floor')` per root store, read with `Atomics` as `BigInt64` bit patterns of float64s (positive doubles order as their bits, and +1 on the bits is `nextafter`). Per thread one bound word. _Before_ minting or adopting a key a thread marks its word pending: the negated lower bound `min(its last settled bound, proposed)`, which holds because admission compares the key against `proposed` only after that store. The certifier, and a progress subscription with no watermark sampling its start at registration (`localStartBound`), raise `proposed` (their clock sample) before reading the bounds — so either the reader saw the pending word or the thread saw this round's sample. A certifier that still sees a pending word after 8 scans does not advance (it retries after a yield); `localStartBound` cannot yield — registration is synchronous, and a worker terminated mid-reservation keeps its pending word until this thread processes `origin-floor-retire` — so it uses the carried magnitude, or the floor published before its sample when that is higher (one read after the scan could pass a key the scan saw commit meanwhile), and never falls back to a clock (harper#3599). A carried bound is as old as its pending reservation, so the start can be early, never late. `proposed` is seeded with a clock sample when the first registry is created, so a carried bound is never `0`; a `0` word is the initial value of a slot being claimed, whose claimer has reserved nothing yet and constrains nothing (a reused slot's stale word in that window only errs low). All floor words are compare-and-swap maxima and persistence is read-compare-write, so two certifiers (a replacement maintenance worker starts before its predecessor retires) can only repeat a round, never regress one.
- **Every key in `local` is unique, because every key is minted.** `reserve` admits only the handle's own native mint, and only when it is at or above `proposed` and beats the ratchet; otherwise the handle takes a fresh key (`freshKey`: the native clock's next value, unique and above every sample so far, else the ratchet's next ulp). A caller's value — a context timestamp, `options.timestamp`, a lock stamp — is never a key: `txnTime` stays the record version and the handle takes a minted key, so such an entry carries both clocks exactly as an applied write does (the record points at its audit head through `additionalAuditRefs`; `_write*` take the log key from the handle). Before this rule an explicit value above the ratchet was admitted as the key without moving the ratchet, so two local transactions with one explicit timestamp shared a key, and a later mint could land on an admitted value; `RocksTransactionLogStore.getSync` stops at the first transaction at a key, so the second was unreachable, and the out-of-order walk and keyed dedup misjudged it (an older write won; a re-delivered additive op applied twice; the `transaction.test.js` resequencing flake — the investigation is [harper#3181](https://github.com/HarperFast/harper/pull/3181), which this rule supersedes). Logs written before this rule may still hold two transactions at one key until `auditRetention` (default 3 days) ages them out; there is no migration, and the keyed lookup keeps returning the first. An unmapped origin (an id with no node name) is filed in `local` too, under a minted key, so its writes carry no origin key in the log: an equal version there — a local write's own, or the same unmapped origin's — is a re-delivery (an origin gives distinct writes distinct versions); unmapped origins are exceptional and do not follow the dual-clock identity. A read handle's minted key holds nothing: it is re-admitted at the first write. A write for a named remote origin (`isLocalOrigin` is `put`'s own resolver; an unnamed non-relayed id resolves to `local`) takes no reservation — the origin's floor governs its log. Crash replay never appends.
- **Durability and restart:** the floor is persisted (`origin-closed-floor`, WAL-durable through `commitAuditMetadata`), then synced to disk (`syncWal`, when the engine offers it — `originFloorsDurable`; the tick skips while a sync is in flight) _before_ it is published; advertised `F` is always the persisted value, at most every `ORIGIN_FLOOR_TICK_MS` (5 s) per database on the maintenance worker. Readers consume only the published value, so after a power loss the restarted origin still seeds its ratchet above every floor anyone saw (harper-pro#922 item 4). At open the record seeds the ratchet, which a native key must beat (one load in the ordinary case). Only once a key has been issued from the ratchet (the clock was behind it) do native keys also raise it by compare-and-swap, so a stale ratchet step never repeats a native key another thread admitted meanwhile; strict mode is never lifted within the process, because any hand-back to load-only admission races a fallback still in flight. A key that cannot beat the ratchet is minted fresh — the native clock's next value when that beats it, else the ratchet's next ulp. A caller's value never touches the ratchet (one far in the future would pin every later key) and is never a key. Behind a recovered floor (a restart after a backward clock step) keys therefore come one ulp at a time above it until the clock passes. The largest key in the log is deliberately _not_ a seed either.
- **Release and liveness:** a reservation ends on the terminal settle of its handle (the success branch after the coordinated-retry check, the terminal-failure branch, `commitTrackedRocksTransaction`'s settle for raw handles) and on every abort path (`abortNativeTransaction`, `doneReadTxn`, the lease-lapsed and no-writes aborts, `abortChainAfterRetries`, the eviction paths). A retry keeps the key and the reservation — the batch it already wrote carries that key; a replay handle past an open iterator takes it over, with its admitted key, and the retained handle holds nothing further. A worker's bounds are retired only from the main thread's `worker.on('exit')`, which fires after rocksdb-js's env cleanup closed the worker's handles with their commits drained (the worker's own exit event fires before that, while a queued commit can still append); main retires them in the stores it has open and broadcasts `origin-floor-retire` so every thread does the same in the stores it has open, since main need not hold them all. A stalled writer holds the floor by design; the lag is visible through `getOriginClosedFloor` (floor, lag, holders).
- **Not covered:** power loss on an engine without `syncWal` (harper-pro then certifies at level 1 only, and no checkpoint is issued); a transaction-log entry below `F` appended but not on disk at power loss is lost at the origin while a receiver may hold it — Harper's commit contract, unchanged; a re-delivery of a local explicit-timestamp write that carries only the version (an own-origin or caching-source replay; replication frames carry the log key and take the keyed dedup) is matched through the received-identity ref the record keeps and the equal-version rule, not by log key: an in-order write rebuilds that ref list, so such a re-delivery that then arrives buried past the reconciliation walk's cap applies a commutative operation twice — bounded only by harper#1148's convergence; an older binary writing below a saved floor; LMDB (no floor, `getOriginClosedFloor` is undefined). Regression coverage: `unitTests/resources/uniqueLocalLogKeys.test.js` (distinct keys for one explicit timestamp, a mint that would have landed on an admitted value, the consumer outcomes); `unitTests/resources/originClosedFloor.test.js` (stalled and killed worker commits, read promotion, explicit timestamps below and above the floor, equal-version local writes, remote-origin routing, the choke point, a concurrent writer against a racing certifier, issuance behind a persisted floor simulated in-process, without a restart or a cleared registry); lock stamps take the same explicit-value path; `recordLock.test.js` pins their version semantics but no stamp below the floor.
- **Durability and restart:** the floor is persisted (`origin-closed-floor`, WAL-durable through `commitAuditMetadata`), then synced to disk (`syncWal`, when the engine offers it — `originFloorsDurable`; the tick skips while a sync is in flight) _before_ it is published; advertised `F` is always the persisted value, at most every `ORIGIN_FLOOR_TICK_MS` (5 s) per database on the maintenance worker. Readers consume only the published value, so after a power loss the restarted origin still seeds its ratchet above every floor anyone saw (harper-pro#922 item 4). At open the record seeds the ratchet, which a native key must beat (one load in the ordinary case). Only once a key has been issued from the ratchet (the clock was behind it) do native keys also raise it by compare-and-swap, so a stale ratchet step never repeats a native key another thread admitted meanwhile; strict mode is never lifted within the process, because any hand-back to load-only admission races a fallback still in flight. A key that cannot beat the ratchet is minted fresh — the native clock's next value when that beats it, else the ratchet's next ulp. A caller's value never touches the ratchet (one far in the future would pin every later key) and is never a key. Behind a recovered floor (a restart after a backward clock step) keys therefore come one ulp at a time above it until the clock passes. The largest key in the log is deliberately _not_ a seed either. A record-lock admission floor within `storage.maxLockClockSkew` is claimed by raising the ratchet just below it and minting a distinct record version above it; the local log key is reserved separately when the write is staged. A key issued from the ratchet is not ordered by the clock, so before it is used a ceiling one second above it is persisted (`local-issuance-ceiling`, `coverLocalIssuance`, flushed before use; restart seeds above that whole reserve), and the first opener seeds issuance above that ceiling (`seedLocalIssuance`); later workers share the immutable initial ceiling in the ratchet buffer and do not consume unused headroom: a restart before the clock passes a claim, or the keys issued after it or after a recovered floor, cannot issue them again. The admission path rejects floors beyond the configured skew cap before claiming a key or persisting its ceiling, and refusal is fenced on the specific record-lock handle. **Release and liveness:** a reservation ends on the terminal settle of its handle (the success branch after the coordinated-retry check, the terminal-failure branch, `commitTrackedRocksTransaction`'s settle for raw handles) and on every abort path (`abortNativeTransaction`, `doneReadTxn`, the lease-lapsed and no-writes aborts, `abortChainAfterRetries`, the eviction paths). A retry keeps the key and the reservation — the batch it already wrote carries that key; a replay handle past an open iterator takes it over, with its admitted key, and the retained handle holds nothing further. A worker's bounds are retired only from the main thread's `worker.on('exit')`, which fires after rocksdb-js's env cleanup closed the worker's handles with their commits drained (the worker's own exit event fires before that, while a queued commit can still append); main retires them in the stores it has open and broadcasts `origin-floor-retire` so every thread does the same in the stores it has open, since main need not hold them all. A stalled writer holds the floor by design; the lag is visible through `getOriginClosedFloor` (floor, lag, holders).
- **Not covered:** power loss on an engine without `syncWal` (harper-pro then certifies at level 1 only, and no checkpoint is issued); a transaction-log entry below `F` appended but not on disk at power loss is lost at the origin while a receiver may hold it — Harper's commit contract, unchanged; a re-delivery of a local explicit-timestamp write that carries only the version (an own-origin or caching-source replay; replication frames carry the log key and take the keyed dedup) is matched through the received-identity ref the record keeps and the equal-version rule, not by log key: an in-order write rebuilds that ref list, so such a re-delivery that then arrives buried past the reconciliation walk's cap applies a commutative operation twice — bounded only by harper#1148's convergence; an older binary writing below a saved floor; LMDB (no floor, `getOriginClosedFloor` is undefined). Regression coverage: `unitTests/resources/uniqueLocalLogKeys.test.js` (distinct keys for one explicit timestamp, a mint that would have landed on an admitted value, the consumer outcomes); `unitTests/resources/originClosedFloor.test.js` (stalled and killed worker commits, read promotion, explicit timestamps below and above the floor, equal-version local writes, remote-origin routing, the choke point, a concurrent writer against a racing certifier, issuance behind a persisted floor simulated in-process, without a restart or a cleared registry); lock stamps use a claimed record version above the admission floor and reserve a distinct local log key separately; `recordLock.test.js` pins their version semantics but no stamp below the floor.

## Historical record reconstruction (`getRecordAtTime`)

Expand Down
Loading
Loading