Skip to content

Slot-based declarative residency: a fixed key-to-slot policy and a replicated slot map #980

Description

@kriszyp

Follow-up to #975 (dedicated replication worker pool), under #435. Rewritten 2026-10-15 around slots, by Kris's direction; this replaces the earlier byId: { shards: N } proposal. It also supplies the durable half of #436's residency-vs-routing split (its design item 1). This is the design to implement; the slot migration protocol (move, split, merge) is a separate follow-up, linked below.

Problem

Two defects share one cause: residency is computed by thread-local application code and stored as node names.

  1. Threads without application code cannot compute residency. Table.setResidency(record => …) and Table.setResidencyById(id => …) (core resources/Table.ts) are functions an application installs per thread. Replication pool workers (replication.threads > 0, Run replication on a dedicated worker pool (replication.threads) #975; on by default after Default replication.threads to half the HTTP workers #1052) never load application code. On a pool worker the sender's skip for non-resident peers (replication/replicationConnection.ts, the table.getResidencyById test in the audit-send loop) never applies, so an id-sharded table sends invalidation entries with indexed-field stubs to every non-resident peer, which stores them. The main thread may have the same gap for operations-API writes. verify: an app that calls setResidency, then an upsert operation, then inspect the record's residencyId.
  2. Placement is bound to node identities, and nothing moves data when placement changes. A shard number returned by the function is resolved to node names at write time (residencyFromFunction, core resources/Table.ts), and that node list is interned (residency_by_id) and stored on the record. Records placed by id are never relocated (_recordRelocate returns early when getResidencyById is set). So every membership change strands data:
    • replacing a failed node in a shard leaves records listing the old name;
    • a function that hashes over the node list (rendezvous) moves about 1/n of keys per added node, and those rows stay on the old owner while the new owner has none;
    • a hash % N policy, including this issue's earlier byId: { shards: N }, moves most keys when N changes: a last-digit policy going from 2 to 3 shards moves 5 of 9 digits.

Real usage today has both shapes: cache tables sharded by the last digit of an id in the cache key over replication.shard groups, and schedule rows assigned to a single node by rendezvous hashing over server.nodes.

Proposal: fixed slots, a replicated slot map

Placement takes two levels. The first is fixed forever; only the second changes.

key ──(table policy, fixed)──► slot ──(replicated slot map)──► shard ──(replication.shard config)──► nodes
  • Key → slot is declared table metadata, persisted and replicated with the table the way replicate is (core resources/databases.ts). Every thread that opens the table builds the same function without loading application code. It is part of the storage format: changing it means a full relocation, so it is fixed at table creation.
  • Slot → shard is a small replicated system table (one row per slot range, with a map version). This is the only thing rebalancing changes.
  • Shard → nodes stays the existing replication.shard node config (server.shards, replication/knownNodes.ts).
  • The record stores its slot, not node names. Nodes are resolved from the map when sending, reading or answering a miss. Replacing or adding a node in a shard then changes no records, and moving one slot moves exactly that slot's records.

Schema

# hash slots: slot = fnv1a32(orderedBinary(primaryKey)) % 4096
type Order @table @residency(slots: 4096) {
	id: ID @primaryKey
}

# expression slots, compiled per thread like @computed(from:)
type ItemPage @table @residency(slots: 10, slotFrom: "Number(String(id).match(/(\\d)$/)?.[1] ?? 0)") {
	id: ID @primaryKey
}

# by attribute: the slot is computed from the record, and a change of slot relocates it
type Customer @table @residency(slots: 64, slotFrom: "regionSlot(region)", byAttribute: true) {
	id: ID @primaryKey
	region: String
}
  • slots: S is the fixed slot count, at most 65536 (the existing shard-id limit). The default key → slot function is FNV-1a 32-bit over the ordered-binary key encoding, modulo S, documented as part of the storage format.
  • slotFrom is an expression with id in scope (record too when byAttribute). Like @computed(from:), it is compiled from schema text on each thread that opens the table, so pool workers and the main thread compute the same slot. It must return an integer in [0, S); anything else refuses the write rather than placing it nowhere.
  • Initial assignment is written into the map once, at table creation or first sharding (for example slot k → shard k % shardCount over the shards configured then). It is never recomputed when shards are added; adding a shard is an explicit slot reassignment, done by the follow-up migration protocol.
  • One source per table. A table with @residency refuses setResidency / setResidencyById with a clear error, so a function and the metadata cannot diverge between threads.
  • Per-request targeting stays separate (Replication W7: Robust sharding (residency-vs-routing split) #436 item 1). replicateTo remains a node list for that request and never changes a record's slot.

The two usage shapes above, under this design

  • Last-digit cache keys: slots: 10, slotFrom: <last digit>. Today's (d − 1) % shards becomes the initial map. Going from 2 to 3 shards reassigns a few slots, chosen by the operator, instead of changing the modulus.
  • Rendezvous over nodes: slots: 4096 with the default hash. Replacing a node moves nothing; adding a node reassigns roughly 1/n of the slots through the migration protocol, instead of silently re-owning keys and stranding their rows.

Where the code changes

  • core resources/Table.ts: build the key → slot function from table metadata at table open. Store the slot as the record's residency, either as its own residency kind in residency_by_id (an id naming {slot} instead of a node list) or as a dedicated field; this is open question 2. Resolve slot → nodes through the map wherever a residency list is read today (getResidencyRecord, confirmingResidents, the omit-local-record decision). Refuse the function API on a declarative table.
  • core resources/databases.ts + schema directive parsing: persist and replicate @residency.
  • core: the slot-map system table, its map version, and an in-memory cache every thread reads (small: one row per slot range).
  • harper-pro replication/replicationConnection.ts: the sender's residency test reads slot → nodes from the map, which also replaces the getResidencyById presence test that pool workers fail today. On the wire, send the slot where today the interned node list is sent (RESIDENCY_LIST). That is a protocol change gated by a capability (replication/protocolCapabilities.ts); a peer without the capability keeps receiving the resolved node list.
  • harper-pro replication/replicator.ts: Replicator.load resolves resident nodes for a miss from the slot.

Approaches considered

Axis Candidate Disposition
Different layer Keep residency as application code and load an application-declared residency module on every thread, pool included (replication.residencyModule). Rejected: pool workers exist to run no application code (#975), and it leaves defect 2 untouched; placement stays bound to node names.
Deeper cause Fixed slots with a replicated map, storing the slot (this proposal). Chosen. Fact that beats the alternatives: it is the only option in which membership changes move no records, and it reduces rebalancing to reassigning a slot.
Do less Persist only a "residency by id" mode bit when setResidencyById is called, so the pool's sender skip works; keep node-name storage. Rejected as the design, kept as a possible stopgap: it fixes the pool's sender skip with very little code, but not omission on write, the read path, or defect 2.
Earlier proposal byId: { shards: N } with shard = hash % N, stored as node names. Rejected: changing N moves most keys, and node-name storage strands rows on any membership change.

Open questions

  1. Is the slot map per table, per database, or per slot group shared by several tables? Per database matches replication.databases[].sharded and lets tables that join on the same key co-locate.
  2. Slot storage: a new residency kind inside residency_by_id, or a separate record field? The first keeps the record format; the second removes one lookup on the send path.
  3. Migration of existing sharded tables: an operation that declares @residency on a table already placed by setResidencyById, and how its stored node lists convert to slots. This probably needs a relocation pass.
  4. Schema directive only, or also an operation for tables defined without a schema?
  5. Does the main-thread gap (defect 1, last sentence) exist today? If so it is a bug on its own.

Follow-up

Slot migration (move one slot between shards, split a shard by moving half its slots, merge two shards) is #1053: dual-shard writes during the move, a copy filtered to the slot, a map-version flip, and a local purge that does not replicate as deletes.

🤖 Generated by Anthropic Claude (Opus 5.5); posted via @kriszyp.

Activity

  1. added theissue type on Oct 5, 2026
  2. kriszyp commented on Oct 6, 2026

    @kriszyp
    MemberAuthor

    Same root cause, second instance, found while reviewing #975: an @indexed computed attribute whose resolver the application assigns with setComputedAttribute exists only on threads that ran the application.

    Verified on 2026-10-05 with a 2-node fixture (integrationTests/cluster/fixture-computed-index in the #975 PR):

    write path setComputedAttribute index @computed(from: "…") index
    REST PUT on an HTTP worker indexed indexed
    operations-API upsert (main thread), no replication pool not indexed indexed
    replicated write applied on a replication.threads worker not indexed (now refused, see #975) indexed

    So the main-thread gap exists today regardless of the pool. It answers the verify: above for computed attributes; residency on the main thread is still unverified. The declarative direction this issue proposes for residency covers it too. @computed(from: "…") is already the declarative form, persisted with the schema and resolved on every thread. A setComputedAttribute-only index could become a schema-time error, or a warning naming the threads that cannot maintain it. Core now exposes Table.unresolvedComputedIndexes() for exactly that check.

    🤖 Generated by Claude (Opus 5.5); posted via @kriszyp.

  3. changed the title [-]Declarative residency policy so threads without application code shard correctly[/-] [+]Slot-based declarative residency: a fixed key-to-slot policy and a replicated slot map[/+] on Oct 15, 2026
  4. self-assigned this
    on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

Priority

P2

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions