Skip to content

replicatedConfirmation counts peers that do not hold a copy of the record (non-resident or replicateTo-excluded) #1031

Closed

Description

@kriszyp

Problem

replicatedConfirmation: N is meant to tell the writer that N other nodes now hold the write. Today it only tells the writer that N peers have advanced past the write's transaction time for that database. On a table with a record-based residency list, those are different things:

  • A non-resident peer receives only an invalidation entry for the record (an INVALIDATED stub), not the record, yet its per-database applied watermark passes the write's txnTime and it is counted.
  • A peer excluded by replicateTo receives nothing for the record, yet any later write or sequence update for the database advances its watermark past txnTime, and it is counted.

So a write can resolve "confirmed by N" while zero other nodes hold a complete copy. The caller gets no signal; the promise silently means less than it says.

Where it happens: replication/knownNodes.ts — createConfirmationWaiter counts any peer (by identity, harper-pro#633) whose CONFIRMATION_STATUS_POSITION for the database is >= txnTime; the replicationConfirmation(...) registration bounds the count only by the database's recipient set (getReplicationRecipients), not by the record's residency.

Context: documentation#723 was about to document this as the contract. It is on hold until this is fixed.

Required behavior

A confirmation counts only peers that hold a valid (complete) copy of the written record(s). Tables without residency lists (every peer resident) behave exactly as today.

Proposed shape

  • Core computes, per write, the set of resident peers it already knows when it applies residency (record-based residency or setResidencyById), and passes it into the replicationConfirmation callback alongside txnTime and the count.
  • createConfirmationWaiter ignores onConfirm (and the alreadyConfirmedPeers seed) for any peer outside that set.
  • No protocol change: eligibility is decided on the writer, and the existing watermark ack is reused, so mixed-version receivers need nothing new.
  • The writer itself does not count toward N (it stays resident per harper#3128).

Open questions (recommendations from triage)

  1. Transactions spanning records with different residency lists. Recommend: require N confirmations per record from that record's residents (equivalently, a peer counts for a record only if resident for it), rather than one intersected set.
  2. N larger than the number of other resident peers. Recommend: fail fast with a clear error at write time. Silently capping N reproduces this bug; timing out (today's behavior) costs the full confirmation timeout for a request that can never succeed.

Acceptance

  • A test with a record-based residency list where the only non-writer resident is down: a write with replicatedConfirmation: 1 does not confirm from a non-resident or replicateTo-excluded peer, and times out or errors per the ruling above. Fails on current main.
  • Unsharded tables: existing confirmation tests unchanged.
  • A multi-record transaction with differing residency covers question 1's rule.

Activity

  1. added this to the v5.4 milestone on Oct 11, 2026
  2. added theissue type on Oct 11, 2026
  3. self-assigned this
    on Oct 11, 2026
  4. kriszyp commented on Oct 13, 2026

    @kriszyp
    MemberAuthor

    Fixed by #1032 (Count only resident peers toward replicatedConfirmation), merged 2026-10-11. Closing; the merged PR referenced this issue without a closing keyword.

    — Claude Opus 5.5 (triage pass)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:replicationReplication, cluster sync, peer connections

Type

Fields

Priority

P1

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions