Repository navigation
Description
Follow-up to #975 (dedicated replication worker pool), under #435. Rewritten 2026-10-15 around slots, by Kris's direction; this replaces the earlier byId: { shards: N } proposal. It also supplies the durable half of #436's residency-vs-routing split (its design item 1). This is the design to implement; the slot migration protocol (move, split, merge) is a separate follow-up, linked below.
Problem
Two defects share one cause: residency is computed by thread-local application code and stored as node names.
- Threads without application code cannot compute residency.
Table.setResidency(record => …)andTable.setResidencyById(id => …)(coreresources/Table.ts) are functions an application installs per thread. Replication pool workers (replication.threads > 0, Run replication on a dedicated worker pool (replication.threads) #975; on by default after Default replication.threads to half the HTTP workers #1052) never load application code. On a pool worker the sender's skip for non-resident peers (replication/replicationConnection.ts, thetable.getResidencyByIdtest in the audit-send loop) never applies, so an id-sharded table sends invalidation entries with indexed-field stubs to every non-resident peer, which stores them. The main thread may have the same gap for operations-API writes.verify:an app that callssetResidency, then anupsertoperation, then inspect the record'sresidencyId. - Placement is bound to node identities, and nothing moves data when placement changes. A shard number returned by the function is resolved to node names at write time (
residencyFromFunction, coreresources/Table.ts), and that node list is interned (residency_by_id) and stored on the record. Records placed by id are never relocated (_recordRelocatereturns early whengetResidencyByIdis set). So every membership change strands data:- replacing a failed node in a shard leaves records listing the old name;
- a function that hashes over the node list (rendezvous) moves about 1/n of keys per added node, and those rows stay on the old owner while the new owner has none;
- a
hash % Npolicy, including this issue's earlierbyId: { shards: N }, moves most keys when N changes: a last-digit policy going from 2 to 3 shards moves 5 of 9 digits.
Real usage today has both shapes: cache tables sharded by the last digit of an id in the cache key over replication.shard groups, and schedule rows assigned to a single node by rendezvous hashing over server.nodes.
Proposal: fixed slots, a replicated slot map
Placement takes two levels. The first is fixed forever; only the second changes.
key ──(table policy, fixed)──► slot ──(replicated slot map)──► shard ──(replication.shard config)──► nodes
- Key → slot is declared table metadata, persisted and replicated with the table the way
replicateis (coreresources/databases.ts). Every thread that opens the table builds the same function without loading application code. It is part of the storage format: changing it means a full relocation, so it is fixed at table creation. - Slot → shard is a small replicated system table (one row per slot range, with a map version). This is the only thing rebalancing changes.
- Shard → nodes stays the existing
replication.shardnode config (server.shards,replication/knownNodes.ts). - The record stores its slot, not node names. Nodes are resolved from the map when sending, reading or answering a miss. Replacing or adding a node in a shard then changes no records, and moving one slot moves exactly that slot's records.
Schema
# hash slots: slot = fnv1a32(orderedBinary(primaryKey)) % 4096
type Order @table @residency(slots: 4096) {
id: ID @primaryKey
}
# expression slots, compiled per thread like @computed(from:)
type ItemPage @table @residency(slots: 10, slotFrom: "Number(String(id).match(/(\\d)$/)?.[1] ?? 0)") {
id: ID @primaryKey
}
# by attribute: the slot is computed from the record, and a change of slot relocates it
type Customer @table @residency(slots: 64, slotFrom: "regionSlot(region)", byAttribute: true) {
id: ID @primaryKey
region: String
}slots: Sis the fixed slot count, at most 65536 (the existing shard-id limit). The default key → slot function is FNV-1a 32-bit over the ordered-binary key encoding, modulo S, documented as part of the storage format.slotFromis an expression withidin scope (recordtoo whenbyAttribute). Like@computed(from:), it is compiled from schema text on each thread that opens the table, so pool workers and the main thread compute the same slot. It must return an integer in[0, S); anything else refuses the write rather than placing it nowhere.- Initial assignment is written into the map once, at table creation or first sharding (for example slot
k→ shardk % shardCountover the shards configured then). It is never recomputed when shards are added; adding a shard is an explicit slot reassignment, done by the follow-up migration protocol. - One source per table. A table with
@residencyrefusessetResidency/setResidencyByIdwith a clear error, so a function and the metadata cannot diverge between threads. - Per-request targeting stays separate (Replication W7: Robust sharding (residency-vs-routing split) #436 item 1).
replicateToremains a node list for that request and never changes a record's slot.
The two usage shapes above, under this design
- Last-digit cache keys:
slots: 10, slotFrom: <last digit>. Today's(d − 1) % shardsbecomes the initial map. Going from 2 to 3 shards reassigns a few slots, chosen by the operator, instead of changing the modulus. - Rendezvous over nodes:
slots: 4096with the default hash. Replacing a node moves nothing; adding a node reassigns roughly 1/n of the slots through the migration protocol, instead of silently re-owning keys and stranding their rows.
Where the code changes
- core
resources/Table.ts: build the key → slot function from table metadata at table open. Store the slot as the record's residency, either as its own residency kind inresidency_by_id(an id naming{slot}instead of a node list) or as a dedicated field; this is open question 2. Resolve slot → nodes through the map wherever a residency list is read today (getResidencyRecord,confirmingResidents, the omit-local-record decision). Refuse the function API on a declarative table. - core
resources/databases.ts+ schema directive parsing: persist and replicate@residency. - core: the slot-map system table, its map version, and an in-memory cache every thread reads (small: one row per slot range).
- harper-pro
replication/replicationConnection.ts: the sender's residency test reads slot → nodes from the map, which also replaces thegetResidencyByIdpresence test that pool workers fail today. On the wire, send the slot where today the interned node list is sent (RESIDENCY_LIST). That is a protocol change gated by a capability (replication/protocolCapabilities.ts); a peer without the capability keeps receiving the resolved node list. - harper-pro
replication/replicator.ts:Replicator.loadresolves resident nodes for a miss from the slot.
Approaches considered
| Axis | Candidate | Disposition |
|---|---|---|
| Different layer | Keep residency as application code and load an application-declared residency module on every thread, pool included (replication.residencyModule). |
Rejected: pool workers exist to run no application code (#975), and it leaves defect 2 untouched; placement stays bound to node names. |
| Deeper cause | Fixed slots with a replicated map, storing the slot (this proposal). | Chosen. Fact that beats the alternatives: it is the only option in which membership changes move no records, and it reduces rebalancing to reassigning a slot. |
| Do less | Persist only a "residency by id" mode bit when setResidencyById is called, so the pool's sender skip works; keep node-name storage. |
Rejected as the design, kept as a possible stopgap: it fixes the pool's sender skip with very little code, but not omission on write, the read path, or defect 2. |
| Earlier proposal | byId: { shards: N } with shard = hash % N, stored as node names. |
Rejected: changing N moves most keys, and node-name storage strands rows on any membership change. |
Open questions
- Is the slot map per table, per database, or per slot group shared by several tables? Per database matches
replication.databases[].shardedand lets tables that join on the same key co-locate. - Slot storage: a new residency kind inside
residency_by_id, or a separate record field? The first keeps the record format; the second removes one lookup on the send path. - Migration of existing sharded tables: an operation that declares
@residencyon a table already placed bysetResidencyById, and how its stored node lists convert to slots. This probably needs a relocation pass. - Schema directive only, or also an operation for tables defined without a schema?
- Does the main-thread gap (defect 1, last sentence) exist today? If so it is a bug on its own.
Follow-up
Slot migration (move one slot between shards, split a shard by moving half its slots, merge two shards) is #1053: dual-shard writes during the move, a copy filtered to the slot, a map-version flip, and a local purge that does not replicate as deletes.
🤖 Generated by Anthropic Claude (Opus 5.5); posted via @kriszyp.
Activity
kriszyp commentedon Oct 6, 2026
Same root cause, second instance, found while reviewing #975: an
@indexedcomputed attribute whose resolver the application assigns withsetComputedAttributeexists only on threads that ran the application.Verified on 2026-10-05 with a 2-node fixture (
integrationTests/cluster/fixture-computed-indexin the #975 PR):setComputedAttributeindex@computed(from: "…")indexPUTon an HTTP workerupsert(main thread), no replication poolreplication.threadsworkerSo the main-thread gap exists today regardless of the pool. It answers the
verify:above for computed attributes; residency on the main thread is still unverified. The declarative direction this issue proposes for residency covers it too.@computed(from: "…")is already the declarative form, persisted with the schema and resolved on every thread. AsetComputedAttribute-only index could become a schema-time error, or a warning naming the threads that cannot maintain it. Core now exposesTable.unresolvedComputedIndexes()for exactly that check.🤖 Generated by Claude (Opus 5.5); posted via @kriszyp.
[-]Declarative residency policy so threads without application code shard correctly[/-][+]Slot-based declarative residency: a fixed key-to-slot policy and a replicated slot map[/+]