Skip to content

Omit reload markers from audit history; return an empty history for a record with no audit entry - #3494

Merged
kriszyp merged 4 commits into
mainfrom
fix/history-readers-skip-reload-markers
Nov 4, 2026
Merged

kriszyp merged 4 commits into
mainfrom
fix/history-readers-skip-reload-markers

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 23, 2026 •

Copy link
Copy Markdown
Member
A frantic clerk strings mail across a corkboard while a calm one lifts the blank reload envelope into its own tray

⊙ Problem

read_audit_log and Table.getHistory return the whole-table reload marker that RocksDB base copies write, as a null-id entry that can join a user transaction's group. On LMDB, read_audit_log by hash_value returns HTTP 500 for a record written without a local audit time.

🤔 Is the requirement right as specified?

Yes: history should list record writes only. The task also asked replay and the previousCount backfill to share the skip. #3503 has since merged and makes both refuse or re-snapshot on a marker, so they are out of scope here.

💡 Solution

  • History reads skip reload through one predicate, as they already skip eviction and lock-control entries.
  • getHistoryOfRecord returns [] for a record with no audit position, as it does for an absent one. read_audit_log by hash_value answers { "<id>": [] } instead of 500.

⚖️ Alternatives

  • Filter in auditStore.getRange — checked resume, live fan-out and the derived-index rebuild read reload from the same range.
  • Stop writing the marker into the table's log — subscribers and checked resume read it in log order, and old markers would still need handling.

🔧 Changes

🤔 Should replay's and the backfill's inline evict/lock-control/reload handling call isNonHistoryType?

No. They must see reload to refuse or re-snapshot, so they keep their own checks. The predicate's JSDoc says so.

🤔 Is an empty history the right answer for a record with no audit position?

getHistoryOfRecord now cannot tell "never written" from "written without an audit entry, such as a base copy". The task asked for an empty result. Carrying history across an audit-less base copy is a separate design (harper#2212). Reverting is one line. The !nextVersion check also matches a position of 0; neither engine produces one, and the old code threw on it too.

✅ Verification

  • historyNonRecordEntries.test.js covers each reader, ResourceBridge.readAuditLog grouping and the LMDB hash_value case. On origin/main the reader tests fail; all pass with the change on RocksDB and LMDB.
  • With reloadMarker, subscribeResume, subscribeProgress, subscriptionReplay and transaction: 159 pass on RocksDB and 123 on LMDB. Typecheck, lint and prettier are clean.
  • The read_audit_log tests call ResourceBridge, not an HTTP request. Not run: the full unit and integration suites; CI covers them.
  • check:design-docs --base origin/main reports only the two resources/MAP.md errors main already has; this PR adds none.
  • PR QA: skipped @ 558594d — dispatch unavailable (a one-shot dev agent does not queue fleet tasks)

🤖 Generated by Anthropic Claude Sonnet 5.5 via Claude Code; posted via @kriszyp.

Release target: main — a P3 history-read fix with no reported customer impact; no backport requested
Related PRs: #3503 overlaps, #3504 overlaps, 9 others independent
Complexity: small
Framing-Verdict: chosen-approach-sound

Origin — the dispatch brief this PR was written from

Table.getHistory (~8768), getHistoryOfRecord (~8812) and the subscribe previousCount loop (~7296-7297) in resources/Table.ts filter evict and lock-control audit entries but not reload markers, so getHistory and read_audit_log return type:'reload' entries with id null and merge them into a user transaction group (RocksDB base copies write these markers). Add reload to one shared skip helper used by getHistory, getHistoryOfRecord, the previousCount loop and replay. Also, on LMDB getHistoryOfRecord throws 'The entry does not have a local audit time' (~8799) for an entry written without a local audit time, which becomes an HTTP 500 for read_audit_log by hash_value; return an empty history instead.

Acceptance

Tests: after a reload marker, getHistory/read_audit_log omit it and group user transactions correctly (both engines); LMDB read_audit_log by hash_value on an entry without local audit time returns an empty result, not 500. Shown failing on origin/main.

Dispatch: task harper-audit-history-filter-reload-markers · queued by finding-triage/wave-39 · ran by claude/sonnet/xhigh · worker kzyp-xps-1

Deferred findings

  • 0e66d1bd189fe85e minor correctness — resources/Table.ts — Unchecked startTime prefix replay sends a raw null-id reload with no re-snapshot; back-filled rows never delivered — NOT FILED
  • 1444e0ea9d03865a minor correctness — unitTests/resources/historyNonRecordEntries.test.js — New test pins raw reload delivery on prefix replay; no copyApply back-fill + startTime replay case — NOT FILED

Late Grok findings

5 surviving (highest major) — Grok reviewed d901632
  • major — resources/Table.ts:7343 — Some subscriptions ask Harper to certify delivery progress (reportProgress) but omit databaseGeneration. During the startTime replay, these subscriptions now pass over a reload marker without being refused. Before this change, a prefix replay threw at the marker because its id is null. Now it
  • minor — resources/Table.ts:7356 — An unchecked prefix replay sends a raw reload with a null id and no re-snapshot, so rows the base copy back-filled are never delivered. Deferred; the re-snapshot ordering belongs to the open PR "Refuse or re-snapshot a subscription replay that crosses a table reload marker, instead of losing back-filled rows or throwing on a prefix scope #3503".
  • minor — unitTests/resources/historyNonRecordEntries.test.js:204 — The new test pins raw reload delivery on prefix replay. No test combines a back-fill with a startTime replay. Same deferred issue as above.
  • nit — resources/Table.ts:7341 — The startTime replay filters evict and lock-control entries inline instead of calling isNonHistoryType. The inline filter is deliberate, because this path must not skip reload. Drift risk only. Deferred.
  • nit — resources/auditStore.ts:1301 — The new JSDoc on isNonHistoryType describes a caller that does not use it instead of what the predicate classifies, and repeats resources/DESIGN.md.

Not standing: 3 separate-issue. Grok ran 14m (14m from launch to report); the fold adjudication (cursor-grok@d901632c1f50-e626086e, claude) took 2m.

Late Grok findings

2 surviving (highest nit) — Grok reviewed c68565c
  • nit — resources/Table.ts:10459 — if (!nextVersion) return history; treats a 0 audit position the same as "no position" and returns an empty history. In practice this cannot trigger: on RocksDB txnLogKey comes from entry.version (resources/Table.ts:1858), and on LMDB it comes from localTime. Both are millisecond timestamps
  • nit — resources/Table.ts:8704 — the startTime replay and previousCount backfill paths filter evict and lock-control types inline instead of calling isNonHistoryType. They cannot call it, because they must refuse or re-snapshot on reload rather than skip it (resources/Table.ts:8706-8709, documented at `resources/auditStore.

Not standing: none. Grok ran 9m (9m from launch to report); the fold adjudication (cursor-grok@c68565cef745-5603d9e6, claude) took 1m.

Late Grok findings

1 surviving (highest nit) — Grok reviewed d3d5ca7
  • nit — resources/Table.ts:10459 — if (!nextVersion) return history; treats an audit position of 0 as "no audit head", so a record whose head sat at position 0 would get an empty history. Disputed: audit positions here are timestamp-like, so 0 is not a realistic value (confidence medium; it would be settled by confirming `resolve

Not standing: 1 dropped-factually-wrong. Grok ran 8m (8m from launch to report); the fold adjudication (cursor-grok@d3d5ca7849ff-c385c4b7, claude) took 1m.

Late Grok findings

1 surviving (highest nit) — Grok reviewed 546a34b
  • nit — resources/Table.ts:10917 — getHistoryOfRecord treats an audit position of 0 as "no audit head" and returns an empty history. Before this change it threw. Scope: pre-existing — Convergence-Class: pre-existing

Not standing: none. Grok ran 9m (10m from launch to report); the fold adjudication (cursor-grok@546a34bae2de-05510686, claude) took 1m.

Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=15; full=6 @ 558594d

Review-Attention: study ~11m (critical: Table.ts, auditStore.ts; decisions: empty-history-on-missing-audit-position, reload-hidden-from-read-audit-log) @ 558594d

@kriszyp kriszyp added this to the v5.4 milestone Oct 23, 2026
gemini-code-assist[bot]

This comment was marked as resolved.

@kriszyp
kriszyp force-pushed the fix/history-readers-skip-reload-markers branch from 634daaa to d901632 Compare October 26, 2026 02:40
@kriszyp
kriszyp marked this pull request as ready for review October 27, 2026 15:12
@kriszyp
kriszyp force-pushed the fix/history-readers-skip-reload-markers branch from 87f8b00 to d3d5ca7 Compare October 28, 2026 15:23
kriszyp and others added 4 commits November 2, 2026 10:48
…tory for an unaudited record

getHistory and getHistoryOfRecord excluded eviction and lock-control entries but not the
whole-table reload marker, so read_audit_log returned type:'reload' entries with a null id
and, on RocksDB, grouped one into a user transaction. isNonHistoryType now owns the
exclusion for both readers. The startTime replay and previousCount backfill keep handling
the marker themselves (refuse or re-snapshot).

getHistoryOfRecord threw for an entry with no audit position (an LMDB entry written without
a local audit time), which read_audit_log by hash_value turned into HTTP 500. It now returns
an empty history, as it does for an absent record.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016q5qJ8dVxeE9co9xaEPQtp
Dispatch-Task: pr-maint-9c4f1f0aa5c37bd0cdb8f905b8b9ebea
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016q5qJ8dVxeE9co9xaEPQtp
Dispatch-Task: pr-maint-9c4f1f0aa5c37bd0cdb8f905b8b9ebea
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016q5qJ8dVxeE9co9xaEPQtp
Dispatch-Task: pr-maint-9c4f1f0aa5c37bd0cdb8f905b8b9ebea
…N.md has no byte headroom

resources/DESIGN.md sits at its ceiling on main, so the note moves beside the
existing `getHistory` bullet in audit-retention.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: rebase-merge-kriszyp-harper-3494-5975580c897c
@kriszyp
kriszyp force-pushed the fix/history-readers-skip-reload-markers branch from 5975580 to 558594d Compare November 2, 2026 17:57
@kriszyp
kriszyp merged commit df882a3 into main Nov 4, 2026
73 of 74 checks passed
@kriszyp
kriszyp deleted the fix/history-readers-skip-reload-markers branch November 4, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant