You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Sharding bootstrap: one record's audit entry is re-logged thousands of times at one version, and replication fails with 'Message too large to send' #1306
During integrationTests/cluster/shardingBootstrap.test.mjs, one node's transaction log stores one record's entry tens of thousands of times at the same version (in system in the failures below, and in data locally). When a peer then needs that range, the sender batches the whole same-version run into one message, refuses it as too large, and closes the system subscription:
[error] [replication]: ... Message too large to send, size: 123860310 remote node: 127.0.0.9 database: system
[error] [replication]: ... Error handling subscription to node Error: Replication message too large to send
at sendQueuedData (dist/replication/replicationConnection.js)
at sendAuditRecord (dist/replication/replicationConnection.js)
The test then fails, either waiting for a write made after a backup restore to reach the co-owner and another shard (Timed out after 30000ms waiting for restored origin inserts and updates reach another shard above recovered bounds), or with ECONNRESET from a node.
This is the failure class of #826 (an echoed entry re-logged into a same-version run larger than the max payload), which #895 fixed for replicated deletes.
Evidence
Node 0 is the only node at debug, so only its sends are visible. Counts are sending audit record <version> lines for the most repeated version in one node start.
In the first row, every repeated entry is 2,406 bytes and names node 0 itself as origin. Peers log Replicated transaction failed; reconnecting to replay it 1794314633883.0317 for the same version. The logs come from the harper-cluster-server-logs-*-shard-1-* artifacts of each run.
When it started
Not after beta.2, despite the first version of this issue. A local A/B (comment) shows the beta.2 build grows the same runs. They also form for data rows, not only system: the last field of sending audit record <version> <x> is the record key, and in these failures it was the node's own hdb_nodes row. CI started failing on 2026-11-06 when runs crossed the max message size; #1285 is not a suspect.
Whether it needs a sharded database. So far only this suite shows it.
Next step
Reproduce locally (it forms on every run with HARPER_TEST_SHARDING_LOG_LEVEL=debug, even when the test passes), find which receive path appends a duplicate of an entry it already holds, and add an assertion on the stored run length so passing runs catch it.
Found while cutting v5.4.0-beta.3; that release lists it as a known issue.
Six local runs of shardingBootstrap.test.mjs, three per build. Setup: CI's replication-pool config (HARPER_CONFIG={"replication":{"threads":2}}), HARPER_TEST_SHARDING_LOG_LEVEL=debug on every node, taskset -c 0-3. All six passed. Every run still stored one record's audit entry thousands of times at a single version.
Build
Longest consecutive sends of one (version, key), runs 1 / 2 / 3
The last field of sending audit record <version> <x> is the record key, not the origin. In the CI failures the key was the node's own hostname, so the repeated entry is that node's hdb_nodes row (2,406 bytes, with certificates). Locally the repeated entries are data rows written by the test: pending-old-origin from the lost-origin test (:426) and nonowner-live1.
So the defect is #826's class, an entry re-logged at the same version on each delivery, on a path #895 does not cover. It shows in the sharding bootstrap's lost-origin, restore and non-owner scenarios, for data and for system.
Run logs are on the release machine under ~/dev/tmp/release-cut/beta3-ci/ab1306/.
changed the title [-]Sharding bootstrap: a node's system log grows a single-version run of tens of thousands of entries, and system replication fails with 'Message too large to send'[/-][+]Sharding bootstrap: one record's audit entry is re-logged thousands of times at one version, and replication fails with 'Message too large to send'[/+]on Nov 10, 2026
Activity
kriszyp commentedon Nov 10, 2026
Local A/B: the echo is already on
v5.4.0-beta.2Six local runs of
shardingBootstrap.test.mjs, three per build. Setup: CI's replication-pool config (HARPER_CONFIG={"replication":{"threads":2}}),HARPER_TEST_SHARDING_LOG_LEVEL=debugon every node,taskset -c 0-3. All six passed. Every run still stored one record's audit entry thousands of times at a single version.pending-old-originpending-old-origin,nonowner-live1Corrections to the issue body:
sending audit record <version> <x>is the record key, not the origin. In the CI failures the key was the node's own hostname, so the repeated entry is that node'shdb_nodesrow (2,406 bytes, with certificates). Locally the repeated entries aredatarows written by the test:pending-old-originfrom the lost-origin test (:426) andnonowner-live1.So the defect is #826's class, an entry re-logged at the same version on each delivery, on a path #895 does not cover. It shows in the sharding bootstrap's lost-origin, restore and non-owner scenarios, for
dataand forsystem.Run logs are on the release machine under
~/dev/tmp/release-cut/beta3-ci/ab1306/.— Claude Opus 5.5
[-]Sharding bootstrap: a node's system log grows a single-version run of tens of thousands of entries, and system replication fails with 'Message too large to send'[/-][+]Sharding bootstrap: one record's audit entry is re-logged thousands of times at one version, and replication fails with 'Message too large to send'[/+]