Skip to content

A node that joins by copy can keep a removed node in its sharding membership, so recovery waits forever #1307

Description

@kriszyp

Problem

A node that joins by base copy can keep a removed node in its hdb_nodes, and so in its sharding membership. Its membership then never agrees with the other owners, and sharding recovery waits forever.

Seen in integrationTests/cluster/shardingBootstrap.test.mjs, test 5 ("losing a whole shard requires a complete backup…"). The wait both lost owners require certified recovery or backup times out after 60 s. The renamed node 2 reports:

recovery: { phase: 'waiting', reason: 'Waiting for 127.0.0.190 at sharding inspect: Error: Sharding membership, map or policy agreement differs' }
waitingOrigins: [ ..., '127.0.0.192', ... ]   // 127.0.0.192 is the node test 4 removed

Node 0's waiting list does not name 127.0.0.192.

Sequence (from the node logs of one local run)

  1. Test 4 removes the old node 2 (.192) on nodes 0 and 1, while node 3 (.193) is stopped.
  2. The new node 2 (.194) joins with add_node. Its system copy from node 0 carries no row for .192, and no tombstone either: a copy writes live rows only.
  3. Node 3 restarts, still holding .192's hdb_nodes row, and only then runs its own remove_node.
  4. .194 logs Skipping a source-applied put with no record content for hdb_nodes id 127.0.0.192 from node 2 … valueless source put (core Table.ts, _writeUpdate), then subscribes to .192 after its next restart. The removal never landed on .194.

Unknown

  • Which message carried .192's removal to .194: a delete relayed as a valueless put, or a copy or relay frame of the row. That decides whether the fix is in replication's encoding or in core's apply.
  • How often CI hits it. CI runs 42715532006 and 42754206906 time out on the same wait, but their renamed node reports Waiting for prior sharding origin evidence from every member instead, which may be a different mode.

Next step

Rerun the suite with HARPER_TEST_SHARDING_LOG_LEVEL=debug until test 5 times out, and trace .192's hdb_nodes entries on .194.

Found while fixing harper-pro#1306; that fix does not touch this path.

— Claude Opus 5.5

Activity

  1. added theissue type on Nov 10, 2026
  2. self-assigned this
    on Nov 10, 2026
  3. kriszyp commented on Nov 11, 2026

    @kriszyp
    MemberAuthor

    Root cause found: a base copy sends a deleted row as a value-less put that core skips, so the joiner keeps no tombstone and a later older write recreates the row. The general bug is filed as #1331 (any table, not only hdb_nodes); this issue is one case of it and closes with that fix. #1329 does not fix it.

    — Claude Opus 5.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

P2

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions