From 0ee02e5d11c3192793e5417dbe70ffe9d80c5400 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 08:48:15 -0600 Subject: [PATCH 01/10] Clarify replicatedConfirmation/confirm= any-N semantics in sharding docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit replicatedConfirmation: N (and confirm=N) resolves once any N peers ack the commit (knownNodes.ts createConfirmationWaiter: a Set keyed by peer name, settled at size === confirmationCount) — not N specific peers, so it cannot guarantee a particular node has the write. Point readers at replicateTo with an explicit node list for that instead. Applied to all three Replication Control subsections (REST, Operations API, programmatic) in both reference/replication/sharding.md and the v4 versioned copy. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 6 ++++++ reference_versioned_docs/version-v4/replication/sharding.md | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 41c57635b..20c188e86 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,6 +51,8 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional node before responding +`confirm=N` waits until any N peers acknowledge the commit — not N specific peers — so it cannot guarantee that a particular node has the write, only that N replicas exist somewhere in the cluster. To target specific nodes, use an explicit node list with `X-Replicate-To` instead of (or in addition to) a count. + Specify exact destination nodes by hostname: ```http @@ -78,6 +80,8 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` +`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers — so it cannot guarantee that a particular node has the write. Use `replicateTo` with an explicit node list to target specific nodes. + Or specify explicit nodes: ```jsonc @@ -103,6 +107,8 @@ class MyTable extends tables.MyTable { } ``` +As above, `replicatedConfirmation` waits for any N peers to acknowledge, not N specific ones — set `replicateTo` to an explicit array of node names to target specific nodes. + ## Static Sharding ### Basic Static Shard Configuration diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index 59efbfd80..2f8ffd61a 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -51,6 +51,8 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional node before responding +`confirm=N` waits until any N peers acknowledge the commit — not N specific peers — so it cannot guarantee that a particular node has the write, only that N replicas exist somewhere in the cluster. To target specific nodes, use an explicit node list with `X-Replicate-To` instead of (or in addition to) a count. + Specify exact destination nodes by hostname: ```http @@ -78,6 +80,8 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` +`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers — so it cannot guarantee that a particular node has the write. Use `replicateTo` with an explicit node list to target specific nodes. + Or specify explicit nodes: ```jsonc @@ -103,6 +107,8 @@ class MyTable extends tables.MyTable { } ``` +As above, `replicatedConfirmation` waits for any N peers to acknowledge, not N specific ones — set `replicateTo` to an explicit array of node names to target specific nodes. + ## Static Sharding ### Basic Static Shard Configuration From ba902370b2c051e9cf16deba418cdd3e7f7445f1 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 08:57:42 -0600 Subject: [PATCH 02/10] Fix overclaims in the confirmation-semantics note per review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first pass said confirm=N guarantees "N replicas exist somewhere", and that replicateTo's explicit node list fixes targeting for confirm. Both are false: a peer outside the record's residency still advances the confirmation watermark via an invalidation entry (partial record, no full copy), and the confirmation waiter counts acks from whichever peers this node replicates to regardless of the request's replicateTo list (knownNodes.ts getReplicationRecipients / createConfirmationWaiter). Also drops the implication that a count and a node list combine in one X-Replicate-To value — REST.ts treats "2,node1" as two hostnames, not a count plus a list. Restates the three notes to separate what replicateTo controls (where the record is stored) from what confirm/replicatedConfirmation counts (any N acks, residency-independent), without claiming the combination proves a named node has the write. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 6 +++--- reference_versioned_docs/version-v4/replication/sharding.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 20c188e86..95673b42c 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional node before responding -`confirm=N` waits until any N peers acknowledge the commit — not N specific peers — so it cannot guarantee that a particular node has the write, only that N replicas exist somewhere in the cluster. To target specific nodes, use an explicit node list with `X-Replicate-To` instead of (or in addition to) a count. +`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too (it receives an invalidation entry, not the record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored; it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) Specify exact destination nodes by hostname: @@ -80,7 +80,7 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` -`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers — so it cannot guarantee that a particular node has the write. Use `replicateTo` with an explicit node list to target specific nodes. +`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they only receive an invalidation entry, not the record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored, independently of what `replicatedConfirmation` counts. Or specify explicit nodes: @@ -107,7 +107,7 @@ class MyTable extends tables.MyTable { } ``` -As above, `replicatedConfirmation` waits for any N peers to acknowledge, not N specific ones — set `replicateTo` to an explicit array of node names to target specific nodes. +As above, `replicatedConfirmation` waits for any N peers to acknowledge — not N specific ones, and not necessarily N peers holding the full record. `replicateTo` controls where the record is stored, independently of which peers' acknowledgements count toward `replicatedConfirmation`. ## Static Sharding diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index 2f8ffd61a..fdb7ad267 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional node before responding -`confirm=N` waits until any N peers acknowledge the commit — not N specific peers — so it cannot guarantee that a particular node has the write, only that N replicas exist somewhere in the cluster. To target specific nodes, use an explicit node list with `X-Replicate-To` instead of (or in addition to) a count. +`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too (it receives an invalidation entry, not the record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored; it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) Specify exact destination nodes by hostname: @@ -80,7 +80,7 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` -`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers — so it cannot guarantee that a particular node has the write. Use `replicateTo` with an explicit node list to target specific nodes. +`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they only receive an invalidation entry, not the record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored, independently of what `replicatedConfirmation` counts. Or specify explicit nodes: @@ -107,7 +107,7 @@ class MyTable extends tables.MyTable { } ``` -As above, `replicatedConfirmation` waits for any N peers to acknowledge, not N specific ones — set `replicateTo` to an explicit array of node names to target specific nodes. +As above, `replicatedConfirmation` waits for any N peers to acknowledge — not N specific ones, and not necessarily N peers holding the full record. `replicateTo` controls where the record is stored, independently of which peers' acknowledgements count toward `replicatedConfirmation`. ## Static Sharding From b09196758210ac3421e75ad1f13e4f3cb6f38919 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 09:05:01 -0600 Subject: [PATCH 03/10] Drop the invalidation-entry overclaim for setResidencyById / already-excluded peers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A non-resident peer doesn't always get an invalidation entry: with setResidencyById, or on a put/patch to a record the peer was already excluded from in the previous residency, Harper sends it nothing for that record at all (replicationConnection.ts skip branch, same in v4). Its replication position — and so its confirmation count — still advances via a later sequence update or record, independent of what it actually holds. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 4 ++-- reference_versioned_docs/version-v4/replication/sharding.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 95673b42c..687ef52c2 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional node before responding -`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too (it receives an invalidation entry, not the record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored; it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored; it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) Specify exact destination nodes by hostname: @@ -80,7 +80,7 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` -`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they only receive an invalidation entry, not the record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored, independently of what `replicatedConfirmation` counts. +`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they count once their replication position passes the write, even holding only an invalidation entry or nothing at all for that record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored, independently of what `replicatedConfirmation` counts. Or specify explicit nodes: diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index fdb7ad267..6c871bf94 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional node before responding -`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too (it receives an invalidation entry, not the record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored; it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored; it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) Specify exact destination nodes by hostname: @@ -80,7 +80,7 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` -`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they only receive an invalidation entry, not the record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored, independently of what `replicatedConfirmation` counts. +`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they count once their replication position passes the write, even holding only an invalidation entry or nothing at all for that record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored, independently of what `replicatedConfirmation` counts. Or specify explicit nodes: From 1240f840c04f761694a3f44a6330bd70b8ebb15a Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 09:15:17 -0600 Subject: [PATCH 04/10] Qualify the replicateTo placement claim and nudge skimmers to the caveat "replicateTo controls where the record is stored" was unqualified: a table's setResidency/setResidencyById function takes precedence over replicateTo entirely (Table.ts: setResidencyById short-circuits before context.replicateTo is read; setResidency overwrites the static getResidency method outright). Added the caveat with a pointer to Custom Sharding, in all three sections of both docs. Also softened the two still-unqualified lines directly above/after the any-N explanation (the confirm=1 bullet, and "confirm can be combined with explicit node lists") so a reader skimming past the explanatory paragraph doesn't land on a sentence that reads as a per-node guarantee on its own. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 10 +++++----- .../version-v4/replication/sharding.md | 10 +++++----- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 687ef52c2..fae5a0e13 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -49,9 +49,9 @@ X-Replicate-To: 2;confirm=1 ``` - `2` — replicate to two additional nodes -- `confirm=1` — wait for confirmation from one additional node before responding +- `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored; it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored under the default residency — a table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it — but it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) Specify exact destination nodes by hostname: @@ -60,7 +60,7 @@ PUT /MyTable/3 X-Replicate-To: node1,node2 ``` -The `confirm` parameter can be combined with explicit node lists. +The `confirm` parameter can be combined with explicit node lists, but as above it still only counts acknowledgements from any N peers, not specifically the listed ones. ### Replication Control via Operations API @@ -80,7 +80,7 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` -`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they count once their replication position passes the write, even holding only an invalidation entry or nothing at all for that record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored, independently of what `replicatedConfirmation` counts. +`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they count once their replication position passes the write, even holding only an invalidation entry or nothing at all for that record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored (subject to any `setResidency`/`setResidencyById` function — see Custom Sharding below), independently of what `replicatedConfirmation` counts. Or specify explicit nodes: @@ -107,7 +107,7 @@ class MyTable extends tables.MyTable { } ``` -As above, `replicatedConfirmation` waits for any N peers to acknowledge — not N specific ones, and not necessarily N peers holding the full record. `replicateTo` controls where the record is stored, independently of which peers' acknowledgements count toward `replicatedConfirmation`. +As above, `replicatedConfirmation` waits for any N peers to acknowledge — not N specific ones, and not necessarily N peers holding the full record. `replicateTo` controls where the record is stored (subject to any `setResidency`/`setResidencyById` function), independently of which peers' acknowledgements count toward `replicatedConfirmation`. ## Static Sharding diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index 6c871bf94..6359a33bf 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -49,9 +49,9 @@ X-Replicate-To: 2;confirm=1 ``` - `2` — replicate to two additional nodes -- `confirm=1` — wait for confirmation from one additional node before responding +- `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored; it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored under the default residency — a table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it — but it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) Specify exact destination nodes by hostname: @@ -60,7 +60,7 @@ PUT /MyTable/3 X-Replicate-To: node1,node2 ``` -The `confirm` parameter can be combined with explicit node lists. +The `confirm` parameter can be combined with explicit node lists, but as above it still only counts acknowledgements from any N peers, not specifically the listed ones. ### Replication Control via Operations API @@ -80,7 +80,7 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` -`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they count once their replication position passes the write, even holding only an invalidation entry or nothing at all for that record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored, independently of what `replicatedConfirmation` counts. +`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they count once their replication position passes the write, even holding only an invalidation entry or nothing at all for that record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored (subject to any `setResidency`/`setResidencyById` function — see Custom Sharding below), independently of what `replicatedConfirmation` counts. Or specify explicit nodes: @@ -107,7 +107,7 @@ class MyTable extends tables.MyTable { } ``` -As above, `replicatedConfirmation` waits for any N peers to acknowledge — not N specific ones, and not necessarily N peers holding the full record. `replicateTo` controls where the record is stored, independently of which peers' acknowledgements count toward `replicatedConfirmation`. +As above, `replicatedConfirmation` waits for any N peers to acknowledge — not N specific ones, and not necessarily N peers holding the full record. `replicateTo` controls where the record is stored (subject to any `setResidency`/`setResidencyById` function), independently of which peers' acknowledgements count toward `replicatedConfirmation`. ## Static Sharding From a6b0e7679470be6dcdd695c13ad8ed1f1a8a1503 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 09:28:08 -0600 Subject: [PATCH 05/10] Drop the v4 "distinct" claim and cover count-form X-Replicate-To too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v4's confirmation waiter (knownNodes.ts) is a plain counter with no per-peer identity tracking, unlike v5 Pro's Set-based dedup (knownNodes.ts:894-901) — a peer whose replicated position regresses and re-crosses the write's txnTime can fire onConfirm twice. Dropped "distinct" from the v4 copy only; v5 keeps it since it does dedupe. Also widened the REST section's residency-precedence caveat: it only mentioned "an explicit node list", but setResidencyById short-circuits before context.replicateTo is ever read regardless of whether the request used a count or a node list, so the numeric X-Replicate-To form needed the same caveat (the Operations API/programmatic sections already stated it generically). Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 2 +- reference_versioned_docs/version-v4/replication/sharding.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index fae5a0e13..1f267d786 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored under the default residency — a table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it — but it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` — whether given as a count or an explicit node list — controls where the record is stored under the default residency — a table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it — but it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) Specify exact destination nodes by hostname: diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index 6359a33bf..f3c05f75c 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` with an explicit node list controls where the record is stored under the default residency — a table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it — but it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +`confirm=N` waits until any N peer acknowledgements have come in — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` — whether given as a count or an explicit node list — controls where the record is stored under the default residency — a table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it — but it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) Specify exact destination nodes by hostname: From 797130bd465f30b9dc004ffd398cd7cae44b2f9f Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 09:53:45 -0600 Subject: [PATCH 06/10] Consolidate the confirmation caveat into one place per doc, add failure modes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review rounds flagged the same root cause: repeating the full any-N/placement caveat in the REST, Operations API, and programmatic sections let their wording drift (v5's "distinct" word was missing from two of the three; v4's non-dedup caveat wasn't propagated to the other two after fixing REST). Restructured both docs to state the semantics once, in the REST section, and have Operations API and programmatic point to it — removing the drift risk instead of re-patching each copy. Also documents the confirmation wait's other two exits, which kept surfacing as a gap across rounds: v5 rejects an excessive count with 400 and times out with 504 after 900s; v4 has no timeout and only checks the count against total node count, not actual recipients. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 12 ++++++++---- .../version-v4/replication/sharding.md | 12 ++++++++---- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 1f267d786..32d497049 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,7 +51,11 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` waits until any N distinct peers acknowledge the commit — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` — whether given as a count or an explicit node list — controls where the record is stored under the default residency — a table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it — but it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers. Acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. + +`X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. + +Confirmation can also end three other ways: a count larger than the peers this node actually replicates the database to is rejected outright (HTTP 400); otherwise, the request fails with a 504 if confirmation doesn't complete within the server's timeout (900 seconds by default). Specify exact destination nodes by hostname: @@ -60,7 +64,7 @@ PUT /MyTable/3 X-Replicate-To: node1,node2 ``` -The `confirm` parameter can be combined with explicit node lists, but as above it still only counts acknowledgements from any N peers, not specifically the listed ones. +The `confirm` parameter can be combined with explicit node lists, but as above it still only counts acknowledgements from any N peers, not specifically the listed ones. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) ### Replication Control via Operations API @@ -80,7 +84,7 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` -`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they count once their replication position passes the write, even holding only an invalidation entry or nothing at all for that record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored (subject to any `setResidency`/`setResidencyById` function — see Custom Sharding below), independently of what `replicatedConfirmation` counts. +`replicatedConfirmation` and `replicateTo` carry the same confirmation and placement semantics as `confirm` and `X-Replicate-To` over REST — see [Replication Control via REST Header](#replication-control-via-rest-header) above for exactly what confirmation does, and doesn't, guarantee. Or specify explicit nodes: @@ -107,7 +111,7 @@ class MyTable extends tables.MyTable { } ``` -As above, `replicatedConfirmation` waits for any N peers to acknowledge — not N specific ones, and not necessarily N peers holding the full record. `replicateTo` controls where the record is stored (subject to any `setResidency`/`setResidencyById` function), independently of which peers' acknowledgements count toward `replicatedConfirmation`. +As above, these carry the same semantics as `confirm` and `X-Replicate-To` over REST — see [Replication Control via REST Header](#replication-control-via-rest-header) above. ## Static Sharding diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index f3c05f75c..aee52bdc5 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -51,7 +51,11 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` waits until any N peer acknowledgements have come in — not N specific peers, and a peer outside the record's residency counts too once its replication position passes the write, even though it may hold only an invalidation entry (or nothing at all for that record). So `confirm` cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To` — has the write. `X-Replicate-To` — whether given as a count or an explicit node list — controls where the record is stored under the default residency — a table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it — but it does not change which peers' acknowledgements `confirm` counts. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peer acknowledgements have come in — not N specific peers, and not necessarily N distinct ones: this version doesn't deduplicate by peer identity, so the same peer crossing the write's replicated position twice (for example, after being restored from an earlier backup and replaying) can satisfy more than one of the N on its own. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. + +`X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. + +This version has no timeout on an unmet confirmation count — a request can block indefinitely if a peer never catches up — and only rejects a count larger than the total number of nodes in the cluster, not the peers this node actually replicates the database to. Specify exact destination nodes by hostname: @@ -60,7 +64,7 @@ PUT /MyTable/3 X-Replicate-To: node1,node2 ``` -The `confirm` parameter can be combined with explicit node lists, but as above it still only counts acknowledgements from any N peers, not specifically the listed ones. +The `confirm` parameter can be combined with explicit node lists, but as above it still only counts acknowledgements from any N peers, not specifically the listed ones. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) ### Replication Control via Operations API @@ -80,7 +84,7 @@ Specify `replicateTo` and `replicatedConfirmation` in the operation body: } ``` -`replicatedConfirmation: N` waits until any N peers acknowledge the commit — not N specific peers, including peers outside the record's residency (they count once their replication position passes the write, even holding only an invalidation entry or nothing at all for that record). It does not guarantee that a particular node, even one listed in `replicateTo`, has the write; `replicateTo` controls where the record is stored (subject to any `setResidency`/`setResidencyById` function — see Custom Sharding below), independently of what `replicatedConfirmation` counts. +`replicatedConfirmation` and `replicateTo` carry the same confirmation and placement semantics as `confirm` and `X-Replicate-To` over REST — see [Replication Control via REST Header](#replication-control-via-rest-header) above for exactly what confirmation does, and doesn't, guarantee. Or specify explicit nodes: @@ -107,7 +111,7 @@ class MyTable extends tables.MyTable { } ``` -As above, `replicatedConfirmation` waits for any N peers to acknowledge — not N specific ones, and not necessarily N peers holding the full record. `replicateTo` controls where the record is stored (subject to any `setResidency`/`setResidencyById` function), independently of which peers' acknowledgements count toward `replicatedConfirmation`. +As above, these carry the same semantics as `confirm` and `X-Replicate-To` over REST — see [Replication Control via REST Header](#replication-control-via-rest-header) above. ## Static Sharding From 1634a26faae1472c8283c2754ca337c0d689093a Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 10:12:13 -0600 Subject: [PATCH 07/10] Version-badge the v5 dedup claim, fix post-commit framing and v4 off-by-one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four more verified corrections from review round 6: - Peer-identity dedup landed in harper-pro commit 61a1531b, between v5.2.0 and v5.2.1 (verified via `git merge-base --is-ancestor` across tags) — v5.0.0/v5.1.0/v5.2.0 used a bare counter like v4 still does. Added and a sentence on the earlier behavior. - "Rejected outright (400)" / "fails with a 504" read as if the write itself failed. Both throw from inside the commit's success path (DatabaseTransaction.ts, both versions) — the local write has already landed either way, only the confirmation wait fails. Reworded, and fixed "three other ways" to "two" to match what's actually listed. - v4's rejection check is `confirmationCount > server.nodes.length`, and server.nodes excludes the local node (knownNodes.ts never pushes getThisNodeName()) — "total number of nodes in the cluster" overcounts by one; reworded to "other nodes it knows about". - v4's :67 ("acknowledgements from any N peers") read as N distinct peers, contradicting :54's "not necessarily N distinct ones" two lines up. Reworded to drop the N-peers framing for v4 specifically. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 4 ++-- reference_versioned_docs/version-v4/replication/sharding.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 32d497049..ffe2b119c 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,11 +51,11 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers. Acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. +`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers. As of , acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N; earlier v5 releases counted repeated watermark crossings from one peer the same way v4 still does (see below). A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. `X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. -Confirmation can also end three other ways: a count larger than the peers this node actually replicates the database to is rejected outright (HTTP 400); otherwise, the request fails with a 504 if confirmation doesn't complete within the server's timeout (900 seconds by default). +Confirmation can also end two other ways, both after the write has already committed locally — the record exists either way, only the wait itself fails: a count larger than the peers this node actually replicates the database to is rejected with a 400; otherwise, the request fails with a 504 if confirmation doesn't complete within the server's timeout (900 seconds by default, raised to match a longer configured blob-transfer timeout). Specify exact destination nodes by hostname: diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index aee52bdc5..91355a66c 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -55,7 +55,7 @@ X-Replicate-To: 2;confirm=1 `X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. -This version has no timeout on an unmet confirmation count — a request can block indefinitely if a peer never catches up — and only rejects a count larger than the total number of nodes in the cluster, not the peers this node actually replicates the database to. +This version has no timeout on an unmet confirmation count — a request can block indefinitely if a peer never catches up — and only rejects, after the write has already committed locally, a count larger than the total number of other nodes it knows about, not specifically the peers this node replicates the database to. Specify exact destination nodes by hostname: @@ -64,7 +64,7 @@ PUT /MyTable/3 X-Replicate-To: node1,node2 ``` -The `confirm` parameter can be combined with explicit node lists, but as above it still only counts acknowledgements from any N peers, not specifically the listed ones. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) +The `confirm` parameter can be combined with explicit node lists, but as above it still only counts whichever peer acknowledgements come in, not specifically from the listed nodes. (A bare count and a node list can't be combined in one `X-Replicate-To` value — `X-Replicate-To: 2,node1` parses `2` as a hostname, not a count.) ### Replication Control via Operations API From ea414064fcf9ed41cc92d54e3bba684c9e52f50d Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 10:25:19 -0600 Subject: [PATCH 08/10] Fix VersionBadge rendering, dangling cross-reference, and delete case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified via git history that the whole confirmation rewrite — peer dedup (61a1531b), the 900s timeout (26b7b3b1), and bounding the reject threshold to actual recipients (f41142aa) — landed together, first in v5.2.1 (same commit timestamp, earliest containing tag). Badged both the dedup claim and the failure-mode paragraph instead of just one, since both were previously stated as unconditional v5 behavior. Also: - "As of " rendered literally as "As of Changed in: v5.2.1" since the component already renders its own label — dropped "As of" and used it as a leading fragment instead, matching the convention at clustering.md:144. - "(see below)" pointed at an explanation that only exists on the separate v4 page, not this one — replaced with an inline comparison ("the way v4 still does"). - "the record exists either way" is wrong for a confirmed delete (it commits a tombstone, not a record) — reworded to "the write (or delete)" in both docs. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 4 ++-- reference_versioned_docs/version-v4/replication/sharding.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index ffe2b119c..1240e905c 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,11 +51,11 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers. As of , acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N; earlier v5 releases counted repeated watermark crossings from one peer the same way v4 still does (see below). A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. +`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers. — acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N; earlier releases, like v4 still does, counted repeated watermark crossings from one peer as satisfying more than one of the N on their own. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. `X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. -Confirmation can also end two other ways, both after the write has already committed locally — the record exists either way, only the wait itself fails: a count larger than the peers this node actually replicates the database to is rejected with a 400; otherwise, the request fails with a 504 if confirmation doesn't complete within the server's timeout (900 seconds by default, raised to match a longer configured blob-transfer timeout). +Confirmation can also end two other ways, both after the write (or delete) has already taken effect locally — only the wait itself fails: a count larger than the peers this node actually replicates the database to is rejected with a 400; otherwise, the request fails with a 504 if confirmation doesn't complete within the server's timeout (900 seconds by default, raised to match a longer configured blob-transfer timeout). — earlier releases had no timeout and checked the count against every known node rather than actual recipients, the way v4 still does. Specify exact destination nodes by hostname: diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index 91355a66c..fb64957db 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -55,7 +55,7 @@ X-Replicate-To: 2;confirm=1 `X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. -This version has no timeout on an unmet confirmation count — a request can block indefinitely if a peer never catches up — and only rejects, after the write has already committed locally, a count larger than the total number of other nodes it knows about, not specifically the peers this node replicates the database to. +This version has no timeout on an unmet confirmation count — a request can block indefinitely if a peer never catches up — and only rejects, after the write (or delete) has already committed locally, a count larger than the total number of other nodes it knows about, not specifically the peers this node replicates the database to. Specify exact destination nodes by hostname: From e5d35c3c07a64850f6c255c2cdda7079e39e8848 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 10:43:14 -0600 Subject: [PATCH 09/10] Note the integer requirement, the Pro-only gate, the v4 ack race, and term drift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four more verified items from review round 8: - N must be a positive integer in both versions: the waiter compares with strict equality (confirmed.size === confirmationCount in harper-pro, ++count === confirmationCount in v4/harperdb), so a string or fractional count never matches and the wait never resolves on its own. - Confirmation waiting is Harper Pro only: the OSS core submodule reads `confirmReplication` (DatabaseTransaction.ts/LMDBTransaction.ts) but never calls `replicationConfirmation(...)` to register a handler — only harper-pro's top-level replication/knownNodes.ts does. Without Pro, confirm/replicatedConfirmation is silently a no-op. - v4's wait can also hang on an otherwise-healthy peer: its waiter seeds no already-crossed state when registered (no analog to harper-pro's alreadyConfirmedPeers seed, knownNodes.ts:792-798), so an ack that lands between commit and registration is lost. Broadened "if a peer never catches up" accordingly. - Replaced "watermark crossings" with "replication-position crossings" — the only use of "watermark" on the page, inconsistent with "replication position" used everywhere else. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 2 +- reference_versioned_docs/version-v4/replication/sharding.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 1240e905c..358b4ecbf 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers. — acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N; earlier releases, like v4 still does, counted repeated watermark crossings from one peer as satisfying more than one of the N on their own. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. +`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers, and N must be a positive integer (a non-numeric or fractional value never matches and the wait never resolves on its own). — acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N; earlier releases, like v4 still does, counted repeated replication-position crossings from one peer as satisfying more than one of the N on their own. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. Confirmation waiting requires Harper Pro; without it, `confirm`/`replicatedConfirmation` is accepted but has no effect — the write succeeds immediately with no wait and no error. `X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index fb64957db..6786107c5 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -51,11 +51,11 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peer acknowledgements have come in — not N specific peers, and not necessarily N distinct ones: this version doesn't deduplicate by peer identity, so the same peer crossing the write's replicated position twice (for example, after being restored from an earlier backup and replaying) can satisfy more than one of the N on its own. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. +`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peer acknowledgements have come in — not N specific peers, and not necessarily N distinct ones: this version doesn't deduplicate by peer identity, so the same peer crossing the write's replicated position twice (for example, after being restored from an earlier backup and replaying) can satisfy more than one of the N on its own. N must also be a positive integer — a non-numeric or fractional value never matches and the wait blocks indefinitely even if every peer acknowledges. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. `X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. -This version has no timeout on an unmet confirmation count — a request can block indefinitely if a peer never catches up — and only rejects, after the write (or delete) has already committed locally, a count larger than the total number of other nodes it knows about, not specifically the peers this node replicates the database to. +This version has no timeout on an unmet confirmation count — a request can block indefinitely, even waiting on an otherwise healthy peer, if an acknowledgement that crosses the write's position arrives before the wait is registered — and only rejects, after the write (or delete) has already committed locally, a count larger than the total number of other nodes it knows about, not specifically the peers this node replicates the database to. Specify exact destination nodes by hostname: From 6d513c8b8a9dce830faf7d54c17f49f28b9af9f4 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 9 Oct 2026 10:52:36 -0600 Subject: [PATCH 10/10] Stop asserting what an invalid confirmation count does My own prior claim ("never matches and the wait never resolves") was wrong: the REST parser drops a non-numeric confirm value outright (NaN fails the >= 0 check), the already-confirmed-peers seed can resolve a bogus count immediately if enough peers already crossed before the wait registered, and from v5.2.1 the wait ends in a 504 regardless. Rather than chase the exact behavior of unvalidated input across versions and races, just state the requirement (N must be a positive integer) and say the value isn't validated and an invalid one's behavior is unspecified. Dispatch-Task: documentation-replicated-confirmation-semantics Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YRqjFRTLq5wic3yFKhYajq --- reference/replication/sharding.md | 2 +- reference_versioned_docs/version-v4/replication/sharding.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 358b4ecbf..f2ab2b6ca 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers, and N must be a positive integer (a non-numeric or fractional value never matches and the wait never resolves on its own). — acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N; earlier releases, like v4 still does, counted repeated replication-position crossings from one peer as satisfying more than one of the N on their own. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. Confirmation waiting requires Harper Pro; without it, `confirm`/`replicatedConfirmation` is accepted but has no effect — the write succeeds immediately with no wait and no error. +`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peers acknowledge the commit — not N specific peers. N must be a positive integer; it isn't validated, and a non-integer value's behavior is unspecified. — acknowledgements are deduplicated by peer identity, so the same peer can't satisfy more than one of the N; earlier releases, like v4 still does, counted repeated replication-position crossings from one peer as satisfying more than one of the N on their own. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. Confirmation waiting requires Harper Pro; without it, `confirm`/`replicatedConfirmation` is accepted but has no effect — the write succeeds immediately with no wait and no error. `X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way. diff --git a/reference_versioned_docs/version-v4/replication/sharding.md b/reference_versioned_docs/version-v4/replication/sharding.md index 6786107c5..aad101681 100644 --- a/reference_versioned_docs/version-v4/replication/sharding.md +++ b/reference_versioned_docs/version-v4/replication/sharding.md @@ -51,7 +51,7 @@ X-Replicate-To: 2;confirm=1 - `2` — replicate to two additional nodes - `confirm=1` — wait for confirmation from one additional peer before responding (see below for exactly what this confirms) -`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peer acknowledgements have come in — not N specific peers, and not necessarily N distinct ones: this version doesn't deduplicate by peer identity, so the same peer crossing the write's replicated position twice (for example, after being restored from an earlier backup and replaying) can satisfy more than one of the N on its own. N must also be a positive integer — a non-numeric or fractional value never matches and the wait blocks indefinitely even if every peer acknowledges. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. +`confirm=N` (and the operation-body/programmatic `replicatedConfirmation: N` used below) waits until any N peer acknowledgements have come in — not N specific peers, and not necessarily N distinct ones: this version doesn't deduplicate by peer identity, so the same peer crossing the write's replicated position twice (for example, after being restored from an earlier backup and replaying) can satisfy more than one of the N on its own. N must also be a positive integer; it isn't validated, and a non-integer value's behavior is unspecified. A peer outside the record's residency still counts, once its replication position passes the write — even though it may hold only an invalidation entry, or nothing at all for that record. So confirmation cannot guarantee that N full copies exist, or that any particular node — even one named in `X-Replicate-To`/`replicateTo` — has the write. `X-Replicate-To`/`replicateTo` controls where the record is stored — whether given as a count or an explicit node list — but only under the default residency. A table's `setResidency` or `setResidencyById` function (see Custom Sharding below) takes precedence over it, and placement is independent of what the confirmation count counts either way.