Skip to content

cluster_status reports a stream healthy after it has lost transaction-log entries — nothing consumes getCorruptFrameReports() #667

Description

@kriszyp

Summary

Harper core now records every corrupt transaction-log frame it reads past and exposes them as getCorruptFrameReports() (harper#2087, resources/replayLogsGuards.ts). Nothing consumes it, so the only operator-visible signal that a replication stream has lost entries is still a log line.

That is the gap harper#2016 and harper#2063 were filed on: in both incidents cluster_status reported connected: true / replicates: true for days while the stream delivered nothing past a torn frame. #2063 ran 11 days that way. Nothing distinguished it from healthy replication except manually diffing record counts across nodes.

What's needed

Surface the reports in cluster_status (and any component/health payload where a lost-entries condition belongs), so a stream that has amputated acknowledged entries is not reported as healthy.

The core API returns, per break site:

  • log (scoped <store path>/<log name>), logId, position
  • midLog — intact entries followed the break, so entries were lost rather than merely truncated
  • unreadableBytes
  • stoppedIteration — the most recent encounter ended iteration (torn tail, or the per-iteration resync cap)
  • firstSeen / lastSeen / occurrences

Plus getEvictedCorruptFrameReportCount() for sites dropped past the retention bound.

Two things to work out

The map is per-worker-isolate. A node-wide signal has to aggregate across worker threads — the reader that hit the break is whichever thread happened to drain that log. There is an existing cross-thread status collector (componentStatus.crossThread) that may be the right vehicle, or core may want to grow an aggregating helper; deciding that is part of this work and may mean a small core PR alongside.

What the field should assert. midLog: true means entries were lost from that log — it does not by itself mean the stream is currently stalled, since core now resyncs past the break (harper#2087 + rocksdb-js#750). stoppedIteration: true on a midLog break is the stronger condition: entries after it are unreachable until the log is repaired or ages out. Both are worth reporting, but they warrant different words.

Related


🤖 Filed by Claude Opus 5 on behalf of @kriszyp

Activity

  1. added
    bugSomething isn't working
    area:replicationReplication, cluster sync, peer connections
    on Aug 6, 2026
  2. added this to the v5.2 milestone on Aug 7, 2026
  3. added theissue type on Aug 7, 2026
  4. modified the milestones: v5.2, v5.4 on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:replicationReplication, cluster sync, peer connectionsbugSomething isn't working

    Fields

    Priority

    P1

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions