Repository navigation
Description
Summary
Harper core now records every corrupt transaction-log frame it reads past and exposes them as getCorruptFrameReports() (harper#2087, resources/replayLogsGuards.ts). Nothing consumes it, so the only operator-visible signal that a replication stream has lost entries is still a log line.
That is the gap harper#2016 and harper#2063 were filed on: in both incidents cluster_status reported connected: true / replicates: true for days while the stream delivered nothing past a torn frame. #2063 ran 11 days that way. Nothing distinguished it from healthy replication except manually diffing record counts across nodes.
What's needed
Surface the reports in cluster_status (and any component/health payload where a lost-entries condition belongs), so a stream that has amputated acknowledged entries is not reported as healthy.
The core API returns, per break site:
log(scoped<store path>/<log name>),logId,positionmidLog— intact entries followed the break, so entries were lost rather than merely truncatedunreadableBytesstoppedIteration— the most recent encounter ended iteration (torn tail, or the per-iteration resync cap)firstSeen/lastSeen/occurrences
Plus getEvictedCorruptFrameReportCount() for sites dropped past the retention bound.
Two things to work out
The map is per-worker-isolate. A node-wide signal has to aggregate across worker threads — the reader that hit the break is whichever thread happened to drain that log. There is an existing cross-thread status collector (componentStatus.crossThread) that may be the right vehicle, or core may want to grow an aggregating helper; deciding that is part of this work and may mean a small core PR alongside.
What the field should assert. midLog: true means entries were lost from that log — it does not by itself mean the stream is currently stalled, since core now resyncs past the break (harper#2087 + rocksdb-js#750). stoppedIteration: true on a midLog break is the stronger condition: entries after it are unreachable until the log is repaired or ages out. Both are worth reporting, but they warrant different words.
Related
- harper#2016 — Mid-log corrupt transaction-log frame silently truncates replay and replication — acknowledged writes lost — suggested directions 1 and 4 are this issue's operator-facing half
- harper#2063 — Corrupt-log containment in RocksTransactionLogStore.getRange is per-drain, so one bad entry head-of-line-blocks a replication stream forever
- harper#2087 — fix(replay): resync past a mid-log corrupt transaction-log frame and surface it as data loss — provides the API
- rocksdb-js#750 — fix(txnlog): resync past a mid-log corrupt frame instead of ending the log — the reader half
- rocksdb-js#748 — A failed transaction-log append orphans its partial bytes, baking a mid-file framing break into the log — prevention, not recovery
🤖 Filed by Claude Opus 5 on behalf of @kriszyp
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity