Repository navigation
Description
Workstream W7 of #430 · robust sharding (the residency-vs-routing split)
Summary
Sharding is built on "residency" — a node list computed per write and interned to a persistent residencyId. The core flaw is that residency is overloaded: a transient per-request replicateTo is laundered into the same persistent residencyId as a durable shard policy, so request-scoped intent mutates placement and triggers invalidation broadcasts to nodes that were never targets. Compounding gaps make sharding crash-prone. (Most of this is core Table.ts work with replication implications.)
Root cause / current state
replicateTobecomes the residency list and is interned + stamped on the record (Table.tsgetResidency~996–1028). An update withreplicateTo:[node2]makes residency{node1,node2}; node3 (previously resident) gets an invalidate, and areplicateFrom:falseread then returns empty — the record looks deleted (replicateTo update deletes record on non-target node #212). The stub also re-fetches on a later SQL SELECT (replicateTo update leaks to non-target node after SQL SELECT triggers cache load #211).replicateFrom:falseis honored on the main read path (Table.ts~4346) but bypassed on SQL and blob fetch paths (X-Replicate-From: none not respected for blob GETs — still fetches from remote nodes #208/replicateTo update leaks to non-target node after SQL SELECT triggers cache load #211).- The cache-load commit recomputes and re-persists residency (
Table.ts~4896), so a read can mutate placement. - Residency functions run against incomplete stubs (index attrs only) →
Cannot read properties of undefined([sharding] Cannot read properties of undefined (reading 'match') in residency function (verify on current build) #201). Self-exclusion is enforced at the connection boundary (replicationConnection.ts:3426, "Should not connect to self") rather than at routing ([sharding] Error: Connection closed Should not connect to self 1008 (verify on current build) #200). Scatter-gather has no merge-aware limit ([sharding] The 'limit' option doesn't retrieve the expected number of records (verify on current build) #198). Constraint errors on relocate/apply aren't fully caught ([sharding] Harper crash after duplicate value error (verify on current build) #199). A "sharded" table silently degrades to replicate-everywhere ifserver.shardsisn't populated yet (Records written to all nodes despite sharding configuration #257).
Design direction
- First-class distinction between durable residency (shard policy) and per-request targeting, so
replicateTonever re-interns a persistentresidencyIdand never broadcasts invalidations to never-target nodes. - One uniform
replicateFrom/locality choke point across REST/SQL/blob/source-load — not per-path guards. - Read paths are placement-neutral — a non-owning read returns local/empty (or fetches without re-persisting residency) and never rewrites ownership.
- Residency-function contract — invoked only with the fields it declares it needs (or always the full record); config-time validation that a sharded table has a live function and a populated
server.shards. - Merge-aware scatter-gather (limit/offset after the cross-shard merge); exhaustive constraint-error capture on relocate/apply; self-exclusion at residency computation.
Scope
- Separate durable residency from per-request
replicateTo(no persistent re-intern) - Single uniform
replicateFrom/locality choke point (REST/SQL/blob/source-load) - Placement-neutral reads (no ownership rewrite on read/cache-load)
- Residency-function field-presence contract + config-time validation
- Merge-aware limit/offset for scatter-gather
- Exhaustive constraint-error capture; self-exclusion at routing
- Expose shard count (Expose shard count via cluster status API #204) → feeds W8
Retires / advances
- replicateTo update deletes record on non-target node #212 — replicateTo update deletes record on non-target node
- replicateTo update leaks to non-target node after SQL SELECT triggers cache load #211 — replicateTo leaks to non-target after SQL SELECT cache-load
- X-Replicate-From: none not respected for blob GETs — still fetches from remote nodes #208 — X-Replicate-From: none not respected for blob GETs
- [sharding] Cannot read properties of undefined (reading 'match') in residency function (verify on current build) #201 — residency function crash on incomplete stub
- [sharding] Error: Connection closed Should not connect to self 1008 (verify on current build) #200 — "Should not connect to self 1008"
- [sharding] Harper crash after duplicate value error (verify on current build) #199 — crash after duplicate-value error
- [sharding] The 'limit' option doesn't retrieve the expected number of records (verify on current build) #198 —
limitreturns too few across shards - Records written to all nodes despite sharding configuration #257 — records written to all nodes despite sharding config
- Expose shard count via cluster status API #204 — expose shard count via cluster status
Dependencies
Largely independent (core-side). Benefits from W8 for the shard-count exposure.
Effort / risk
L / medium. The residency-vs-routing split is the core refactor.
Acceptance criteria
- A
replicateToupdate neither deletes nor leaks on non-target nodes. X-Replicate-From: noneis honored on blob and SQL paths.- A sharded write lands on exactly the residency nodes; a SELECT on a non-owner doesn't cache.
limitreturns the expected count across shards; sharding misconfig surfaces as an error, not silent full replication.
🤖 Filed by Claude on behalf of Kris.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
69 remaining items