[kriszyp]

⊙ Problem

Eviction removes the primary entry that anchored retained history. Per-record audit reads then return [], and reinsertion can hide older segments (#2730). Complete transaction-frame reconstruction fixes stale multi-mutation values but repeatedly decodes large frames. Long patch listings also rebuild the same prefixes.

💡 Solution

Recover absent/eligible unlinked history from retained audit entries. Reconstruct complete matching-origin frames and reuse copied entries and ordinary patch images within each history call. Unsafe chains, corruption and budget exhaustion use the plain rebuild. Search-back stops at the lower retained-log/floor boundary. Writes and stored formats stay unchanged. This covers #4063's eligible live-listing goal; that draft is untouched.

⚖️ Alternatives

🤔 Keep retained history despite its scan cost?

Retain it. Per-id scans can decode all retained history synchronously. The owner accepted the measured listing latency and froze optimization. Residual CPU and the reinsertion gap are tracked in #4286 and #4285.

Reader design and native-filtering alternative

🤔 Why keep the reader-local design?

Framing-Verdict: better-alternative-exists — resolved by adopting detached bytes and conservative plain fallback. An observed head change abandons reuse after a confirmed wrong-value regression. Image stamps, refusal caches and LRU eviction remain deferred.

🤔 Native filtering could simplify frame reads

Native record/origin filtering could remove JavaScript scans and copies. It needs a binding/release and does not supply missing predecessor anchors or remove prefix replay. The reader repair delivers measured gains; #4286 tracks remaining work.

🔧 Changes

• Table discovery, carry eligibility and historical replay contexts.
• Complete origin frames and bounded search/origin anchors preserve frame-final values and deletes.
• Shared byte/entry accounting and disposal; changed-head fallback repairs the concurrent coercive-write case. Audit getter forwarding stays reader-only; its opened-store floor accessor avoids a replay import cycle.
• Native memo/oracle regressions, eviction/frame cases, reconstruction rules and manual benchmarks.
• resource map, release fragment.

How retained values and per-call reuse work

historyReader.ts:339

answer = memo.get(key)
if (answer && answer.stamp !== stamp) {
  memo.stopOptimizing()
  return false
}

A new write can change the plain oracle for an older patch. When the existing eligibility check observes a changed head, the call abandons reuse; it does not validate an old carried image against the new head.

Product and architecture tour

Retained values after eviction

Which history survives losing the primary entry?

Before	After
Eviction can make retained per-record history disappear.	Absent records scan retained entries; eligible reinserts bridge older segments.

One bounded reader per call

When can a listing reuse work?

Shared caps on estimated retained bytes/entries: record8MiB/16,384; streaming2MiB/8,192; replay128KiB/256. Frames, pending entries and images share each budget. Exhaustion, unsafe chains or a changed observed head select plain reconstruction; completion/cancellation releases the context.

Full values without persistent caches

Complete matching-origin frames remain the reconstruction unit.

flowchart TD
  A[History entry] --> B{Safe bounded reuse?}
  B -->|yes| C[Copied frame or patch image]
  B -->|no| D[Plain audit reconstruction]
  C --> E[Full value]
  D --> E
  E --> F[Release context when call ends]

Loading

✅ Verification

Performance: Latest single samples: 50,000-row histories 169/155/163 ms; 2,880 live patches 198 ms. Evicted listing: 34.7 s; 1,000-row bulk-patch live/stream/replay: 2.83/6.24/2.90 s. #4286 records costs and the reproducer; accepted medians and main controls are below.

• PR QA: Does retained history remain exact across bounded-reader fallback and worker restart? — queued @ a4b0e86, qa-pr-harper-3504-a4b0e86; later commit 8ee0988 changes documentation only, so the scenario remains applicable.
• Current-head CI: LMDB rerun passes @ 8ee0988; three integration failures match reviewed main. Lint fixed.

Executed checks and scope limits

• Build, lint, formatting and partition pass locally. The corrected base-aware design check passes; its unchanged base note warns three bytes over the ceiling. Targeted suites: 328 RocksDB / 241 LMDB; server integrations: 35.
• Broad resource runs: a–n 3,278 passing / 126 pending; o–z 3,822 / 116 / one clock-ceiling failure. Isolated branch/main controls pass. These runs preceded the final narrow origin fix.
• Latest source history: 32 RocksDB / 23 LMDB (9 engine skips). Seven native regressions have pre-fix failures; three 300-patch bounds fail before the memo. Both engines match the seeded oracle. Remaining coverage: #4287.
• Docs: Storage release fragment included.

Benchmark controls and limits

The scoped source uses RocksDB 2.14.0 and Node 26.2.0. Latest single-sample wall/CPU ms and native reads: first-row169/258/24; middle155/239/24; last163/273/24;2,880 live patches198/278/2,887. Earlier accepted three-sample medians were196/293,202/233,169/212 and182/234 respectively. Same-dependency main3b9072d0f one-sample controls:316/319,1,959/2,339,3,673/4,103 ms with1,259reads each;2,880patches132,537/154,990 ms with24,891,850reads. The historical92.64 ms pre-frame baseline used older dependencies and is not claimed achieved.

Evicted2,880 patches:34,655 ms wall/41,044 CPU,5,765 native reads (one sample). This low read count hides repeated copied-frame restoration. At1,000 rows, live bulk-patch delivery including commit:2,829/3,965 ms,1,000ranges; table history:6,237/8,555 ms,2,001ranges; replay:2,901/3,961 ms,1,002ranges. These are not50,000-row measurements.

Earlier50,000-filler-row fallback probes, one sample each: exact never-cached invalidate109/176 ms wall/CPU (37ranges); missing id132/245 ms (1range); eligible first write102/171 ms (2ranges); non-exact miss120/172 ms (37ranges). Bounding search windows does not bound retained-entry decoding. A floor of1 retains conservative scan behavior. Broader stress/worker/restart coverage is tracked in #4287.

Closing measurements (one sample each,50,000 filler rows): raised-floor nearby read128.840ms wall/239.141CPU versus2.192/8.122 in pre-bound ancestor3d3a165. Both return the correct value; the new prefix scan is tracked by #4293. Clean memo misses disable reuse; image admission can rebuild an already-computed value. Their counters and timings are in #4286.

Documentation-only correction 8ee0988 restores DESIGN.md and resources/DESIGN.md exactly to the reviewed base, preserving the accurate reverse-walk rationale. No runtime/test changes. The base-aware design check and formatting pass; the unchanged base resources note is reported only as a warning. This avoids expanding the history repair into an unrelated shared-note rewrite. The original closing review covers runtime; a subsequent docs-only delta covers this correction. No promotion signal.

The first LMDB attempt exited 139 in native mdb_cursor_close → CursorWrap::close, matching the stack family tracked by #828. The exact-head rerun passed without code changes (completed 2026-11-12 00:46:17 UTC). The preceding PR head and reviewed main also passed that job. The native cause remains unknown; no repair or proven baseline classification is claimed. The unchanged expiry file also passes locally: 10 passing / 4 engine skips. This PR does not change that file or test commands; it adds history-reader coverage.

CI is settled against three integration failures that also fail on reviewed main be77be5: 1/6 Bun, 6/6 Bun and 6/6 uWS. Lint, formatting, Windows unit tests and the LMDB rerun pass on 8ee0988. No further code or review iteration was needed.

All seven remaining closing findings are filed: raised-floor scan #4293; clean miss and cap duplicate work #4286; shared-log-age test fragility #4287; benchmark header and unused limits export (reverse-boundary rationale restored by the documentation correction) #4294. Existing correctness deferrals are #4282, #4283 and #4284. These are tracked limitations, not repaired behavior. The owner's scope cutoff leaves runtime unchanged; the receipt still flags framing-recheck and closing-full obligations. This stays draft without a promotion signal.

Reproduce after building: HISTORY_BENCH_REPETITIONS=1 npx mocha unitTests/resources/historyReader.bench.js. The fixture checks returned values and measures real native calls, wall time and process CPU. Unit shards exclude this manual benchmark.

_{🤖 Generated by OpenAI Codex; posted via @kriszyp.}

Release target: v5.4 — existing milestone retained.
Related PRs: #4063 supersedes (eligible live-listing goal), #3494 overlaps (history filtering), #3578 overlaps (merged reconstruction), #2155 overlaps (expiry flags), #3137 overlaps (audit fan-out), #4235 overlaps (replay handoff), #4270 overlaps (audit floor), #4250 overlaps (TTL flags), #3414 overlaps (reverse history), #3469 overlaps (replacement replay), #3503 overlaps (reload replay), #3356 overlaps (origin checkpoints), 23 others independent

Related-PR discovery was partial.
Complexity: complicated

Deferred findings

• 97353c0b742e5169 major performance — resources/crdt.ts — Exact unlinked-control search-back scans retained history — filed Reduce synchronous audit-history work outside eligible live listings #4286
• 3e9b36fc44ba80ee major performance — resources/crdt.ts — Full-frame read amplification outside live-record listings — filed Reduce synchronous audit-history work outside eligible live listings #4286
• 382e07e5cff92254 major performance — resources/Table.ts — Evicted many-patch histories rebuild quadratically — filed Reduce synchronous audit-history work outside eligible live listings #4286
• c09ed64a6c60d9e5 major performance — resources/Table.ts — Eligible unlinked history requests scan retained logs — filed Reduce synchronous audit-history work outside eligible live listings #4286
• de3a49e291ef1de9 major performance — resources/Table.ts — Never-evicted eligible records may scan retained logs — filed Reduce synchronous audit-history work outside eligible live listings #4286
• 54c06b9b6dd6207f major performance — resources/Table.ts — Missing-id requests scan retained logs per id — filed Reduce synchronous audit-history work outside eligible live listings #4286
• a0f6cfc90d9c1414 major performance — resources/crdt.ts — Nonexact timestamp misses synchronously search earlier history — filed Reduce synchronous audit-history work outside eligible live listings #4286
• 012f52dec67a6e61 major data-integrity — resources/crdt.ts — Pruned predecessor above cutoff reports false absence — filed Historical reconstruction reports false absence at a pruned predecessor #4283
• bf66cc0c57d88414 major correctness — resources/RocksTransactionLogStore.ts — Shared compound-id reference lookups compare object identity — filed Audit reference lookup misses equivalent compound record ids #4282
• f0999a0163b8335a minor correctness — resources/Table.ts — Non-expiring reinsertion on non-evicting tables omits older history — filed Plain reinsertion on a non-evicting table hides earlier retained history #4285
• +6 more in the review receipt

🤔 defer this PR's major performance finding 97353c0b742e5169 (Exact unlinked-control search-back scans retained history) to #4286 instead of fixing it here?

🤔 defer this PR's major performance finding 3e9b36fc44ba80ee (Full-frame read amplification outside live-record listings) to #4286 instead of fixing it here?

🤔 defer this PR's major performance finding 382e07e5cff92254 (Evicted many-patch histories rebuild quadratically) to #4286 instead of fixing it here?

🤔 defer this PR's major performance finding c09ed64a6c60d9e5 (Eligible unlinked history requests scan retained logs) to #4286 instead of fixing it here?

🤔 defer this PR's major performance finding de3a49e291ef1de9 (Never-evicted eligible records may scan retained logs) to #4286 instead of fixing it here?

🤔 defer this PR's major performance finding 54c06b9b6dd6207f (Missing-id requests scan retained logs per id) to #4286 instead of fixing it here?

🤔 defer this PR's major performance finding a0f6cfc90d9c1414 (Nonexact timestamp misses synchronously search earlier history) to #4286 instead of fixing it here?

Origin — the dispatch brief this PR was written from

Record history after eviction, beyond what #3214/#3414 fixed. (D1, #2730) getHistoryOfRecord (resources/Table.ts ~8791-8795) returns [] when primaryStore.getEntry(id) is missing. (D2) After a re-insert, RecordEncoder.ts (~1116) stamps previousVersion from the absent entry, so the audit chain is unlinked; the walk (~8800-8845) only advances through previousVersion < start, so earlier history is dropped; also the loop's count is a const 0, making its count < 1000 bound inert. (D3) crdt.ts (~280, ~318) calls reconstructForward with { txnLogKey: timestamp } and reconstructForward does an exact-key auditStore.get, so any timestamp that is not an exact audit key returns null. Let the per-record walk start without a primary entry and bridge an unlinked boundary using the record's prior audit entries (evict-type entries can serve as a bridge point); in reconstructForward start from the latest audit entry for (table, record) at or before the timestamp; fix the inert count bound.

Acceptance

Dual-engine tests: evicted record's getHistoryOfRecord returns its audited history; history spans an evict + re-insert; time-travel read at a non-exact timestamp reconstructs the right value. Shown failing on origin/main.

_{Dispatch: task harper-record-history-after-eviction · queued by finding-triage/wave-39 · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=codex; ran=cursor-composer,gemini,cursor-grok,claude; late=cursor-grok:adjudicated; adjudicated=domain; blocked=claude(fallback)(out-of-budget); declined=muse; rounds=11; full=6 @ 8ee0988}

_{Review-Attention: deep ~75m (critical: Table.ts, auditStore.ts +1; decisions: documentation-scope, do-less-alternative, perf-cost, deferred-97353c0b, deferred-3e9b36fc, deferred-382e07e5, deferred-c09ed64a, deferred-de3a49e2, deferred-54c06b9b, deferred-a0f6cfc9) @ 8ee0988}

[gemini-code-assist]

Code Review

This pull request enhances historical record reconstruction (getRecordAtTime) and per-record history retrieval (getHistoryOfRecord) to correctly span across record evictions and re-inserts. It introduces log scanning as a fallback when a predecessor link chain is broken due to eviction, optimizes exact-key lookups to avoid scanning, and adds comprehensive unit tests validating these behaviors. There are no review comments, and we have no further feedback to provide.