Skip to content

Sharding bootstrap: one record's audit entry is re-logged thousands of times at one version, and replication fails with 'Message too large to send' #1306

Closed

Description

@kriszyp

Problem

During integrationTests/cluster/shardingBootstrap.test.mjs, one node's transaction log stores one record's entry tens of thousands of times at the same version (in system in the failures below, and in data locally). When a peer then needs that range, the sender batches the whole same-version run into one message, refuses it as too large, and closes the system subscription:

[error] [replication]: ... Message too large to send, size: 123860310 remote node: 127.0.0.9 database: system
[error] [replication]: ... Error handling subscription to node Error: Replication message too large to send
    at sendQueuedData (dist/replication/replicationConnection.js)
    at sendAuditRecord (dist/replication/replicationConnection.js)

The test then fails, either waiting for a write made after a backup restore to reach the co-owner and another shard (Timed out after 30000ms waiting for restored origin inserts and updates reach another shard above recovered bounds), or with ECONNRESET from a node.

This is the failure class of #826 (an echoed entry re-logged into a same-version run larger than the max payload), which #895 fixed for replicated deletes.

Evidence

Node 0 is the only node at debug, so only its sends are visible. Counts are sending audit record <version> lines for the most repeated version in one node start.

Run Pro commit Most repeated version Count in one start Largest message
42816848946 shard 1/6 Node 24 pool 9ce9db6 1794314633883.0317 51,923 124 MB
42754206906 shard 1/6 Node 26 pool b64ae52 1794284381454.8174 138,376 118 MB
42715532006 shard 1/6 Node 26 f6cec94 1794265153478.6497 258,075 395 MB
42496791093 shard 1/6 Node 22 ce3d46a 1794153576672.9646 138,035 —
42496791093 shard 1/6 Node 24 pool ce3d46a 1794153639421.0994 13,434 —

In the first row, every repeated entry is 2,406 bytes and names node 0 itself as origin. Peers log Replicated transaction failed; reconnecting to replay it 1794314633883.0317 for the same version. The logs come from the harper-cluster-server-logs-*-shard-1-* artifacts of each run.

When it started

Not after beta.2, despite the first version of this issue. A local A/B (comment) shows the beta.2 build grows the same runs. They also form for data rows, not only system: the last field of sending audit record <version> <x> is the record key, and in these failures it was the node's own hdb_nodes row. CI started failing on 2026-11-06 when runs crossed the max message size; #1285 is not a suspect.

Unknown

Next step

Reproduce locally (it forms on every run with HARPER_TEST_SHARDING_LOG_LEVEL=debug, even when the test passes), find which receive path appends a duplicate of an entry it already holds, and add an assertion on the stored run length so passing runs catch it.

Found while cutting v5.4.0-beta.3; that release lists it as a known issue.

— Claude Opus 5.5

Activity

  1. added this to the v5.4 milestone on Nov 10, 2026
  2. added theissue type on Nov 10, 2026
  3. kriszyp commented on Nov 10, 2026

    @kriszyp
    MemberAuthor

    Local A/B: the echo is already on v5.4.0-beta.2

    Six local runs of shardingBootstrap.test.mjs, three per build. Setup: CI's replication-pool config (HARPER_CONFIG={"replication":{"threads":2}}), HARPER_TEST_SHARDING_LOG_LEVEL=debug on every node, taskset -c 0-3. All six passed. Every run still stored one record's audit entry thousands of times at a single version.

    Build Longest consecutive sends of one (version, key), runs 1 / 2 / 3 Key
    v5.4.0-beta.3 (927d06c code, core 64c2634bc) 2,776 / 10,850 / 160 pending-old-origin
    v5.4.0-beta.2 (33cce4d, core b334fd009) 67,495 / 18,403 / 9,203 pending-old-origin, nonowner-live1

    Corrections to the issue body:

    • The last field of sending audit record <version> <x> is the record key, not the origin. In the CI failures the key was the node's own hostname, so the repeated entry is that node's hdb_nodes row (2,406 bytes, with certificates). Locally the repeated entries are data rows written by the test: pending-old-origin from the lost-origin test (:426) and nonowner-live1.
    • Not new since beta.2. The beta.2 build grows the same runs, larger in this sample. The CI failures since 2026-11-06 are runs that crossed the max message size, not a new behavior. Enforce copied WAF rules after a full copy or a recovered subscription #1285 is no longer a suspect.

    So the defect is #826's class, an entry re-logged at the same version on each delivery, on a path #895 does not cover. It shows in the sharding bootstrap's lost-origin, restore and non-owner scenarios, for data and for system.

    Run logs are on the release machine under ~/dev/tmp/release-cut/beta3-ci/ab1306/.

    — Claude Opus 5.5

  4. changed the title [-]Sharding bootstrap: a node's system log grows a single-version run of tens of thousands of entries, and system replication fails with 'Message too large to send'[/-] [+]Sharding bootstrap: one record's audit entry is re-logged thousands of times at one version, and replication fails with 'Message too large to send'[/+] on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P1

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions