Skip to content

update_node patches leave server.nodes and connections[].shard out of step with hdb_nodes #1318

Closed

Description

@kriszyp

Problem

After update_node patches a peer's row in hdb_nodes, this node's in-memory membership disagrees with the table until the next put for that peer, a table reload or a restart.

  • processNodeUpdateEvent removes the peer from server.nodes for every event (replication/knownNodes.ts:438). It adds the peer back only for a put, or for a patch that carries isLeader (:441, :453-459). update_node writes a plain patch (replication/setNode.ts:477-480), so the peer stays out of server.nodes. server.shards is rebuilt from a full scan on every event, so it keeps the peer.
  • The listener is called only for put and delete (:473), so the subscription manager's nodeMap keeps the pre-patch record. cluster_status.connections[].shard shows the old shard while server.shards already holds the new one (replication/subscriptionManager.ts:2194-2200).

What reads server.nodes loses the peer too: residencyTopologyErrors, shardingWaitingOrigins and the "owns no slot" list that harper-pro's cluster_status shard summary reports.

Confidence: code trace, plus the event handler run in isolation against an in-memory store by a review. Not reproduced on a running cluster.

Fix direction

For a non-self patch, re-read the full record with getSync (as the isLeader branch already does) and call the listener, so server.nodes and nodeMap take the patched record. Add a unit test that patches shard and revoked_certificates on a known peer and asserts the peer is still in server.nodes with the new values.

Review-Deferral: 7541454930d653cd severity=minor category=correctness
Review-Deferral: b45ce99d57b72903 severity=minor category=correctness

🤖 Generated by Anthropic Claude (Sonnet 5.5); posted via @kriszyp.

Activity

  1. added theissue type on Nov 10, 2026
  2. kriszyp commented on Nov 10, 2026

    @kriszyp
    MemberAuthor

    harper-pro#1319 hit this: its nodesOwningNoSlots list reads server.nodes, so a peer patched by update_node can be missing from it. That PR documents the limit and leaves the fix here.

    🤖 Generated by Anthropic Claude (Sonnet 5.5); posted via @kriszyp.

  3. self-assigned this
    on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Fields

Priority

P2

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions