You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A node that joins by copy can keep a removed node in its sharding membership, so recovery waits forever #1307
A node that joins by base copy can keep a removed node in its hdb_nodes, and so in its sharding membership. Its membership then never agrees with the other owners, and sharding recovery waits forever.
Seen in integrationTests/cluster/shardingBootstrap.test.mjs, test 5 ("losing a whole shard requires a complete backup…"). The wait both lost owners require certified recovery or backup times out after 60 s. The renamed node 2 reports:
recovery: { phase: 'waiting', reason: 'Waiting for 127.0.0.190 at sharding inspect: Error: Sharding membership, map or policy agreement differs' }
waitingOrigins: [ ..., '127.0.0.192', ... ] // 127.0.0.192 is the node test 4 removed
Node 0's waiting list does not name 127.0.0.192.
Sequence (from the node logs of one local run)
Test 4 removes the old node 2 (.192) on nodes 0 and 1, while node 3 (.193) is stopped.
The new node 2 (.194) joins with add_node. Its system copy from node 0 carries no row for .192, and no tombstone either: a copy writes live rows only.
Node 3 restarts, still holding .192's hdb_nodes row, and only then runs its own remove_node.
.194 logs Skipping a source-applied put with no record content for hdb_nodes id 127.0.0.192 from node 2 … valueless source put (core Table.ts, _writeUpdate), then subscribes to .192 after its next restart. The removal never landed on .194.
Unknown
Which message carried .192's removal to .194: a delete relayed as a valueless put, or a copy or relay frame of the row. That decides whether the fix is in replication's encoding or in core's apply.
How often CI hits it. CI runs 42715532006 and 42754206906 time out on the same wait, but their renamed node reports Waiting for prior sharding origin evidence from every member instead, which may be a different mode.
Next step
Rerun the suite with HARPER_TEST_SHARDING_LOG_LEVEL=debug until test 5 times out, and trace .192's hdb_nodes entries on .194.
Found while fixing harper-pro#1306; that fix does not touch this path.
Root cause found: a base copy sends a deleted row as a value-less put that core skips, so the joiner keeps no tombstone and a later older write recreates the row. The general bug is filed as #1331 (any table, not only hdb_nodes); this issue is one case of it and closes with that fix. #1329 does not fix it.
Activity
kriszyp commentedon Nov 11, 2026
Root cause found: a base copy sends a deleted row as a value-less put that core skips, so the joiner keeps no tombstone and a later older write recreates the row. The general bug is filed as #1331 (any table, not only
hdb_nodes); this issue is one case of it and closes with that fix. #1329 does not fix it.— Claude Opus 5.5
Carry deletions through a base copy, so an older write cannot recreat…
Carry deletions through a base copy, so an older write cannot recreat…
Carry deletions through a base copy, so an older write cannot recreat…
Carry deletions through a base copy, so an older write cannot recreat…
Carry deletions through a base copy, so an older write cannot recreat…
Carry deletions through a base copy, so an older write cannot recreat…
Carry deletions through a base copy, so an older write cannot recreat…
Carry deletions through a base copy, so an older write cannot recreat…
Carry deletions through a base copy, so an older write cannot recreat…