[kriszyp]

⊙ Problem

Source fills and residency stubs in resources/Table.ts left wrong indexes, on both engines:

• A fill indexed before stamping @createdTime/@updatedTime and before deciding residency: first fills indexed no timestamps, revalidations deleted them, id-excluded fills left orphans.
• residencyStub dropped the primary key (index-only selects returned id: undefined) and stored null without a secondary index.
• Writes, refills and evictions over a stub threw when a computed resolver needed a field the stub lacks.

🤔 Is this the requirement?

Yes, as specified: each is a reproducible wrong index or failed write. Put, delete and eviction over a stub are fixed too, since they fail for the same reason as the refill.

💡 Solution

• Fills stamp timestamps before resolving (noCacheStore reads get them), decide residency, then index what they store.
• Residency stubs always carry their key and never run computed resolvers.
• Over a stub, a computed index's old value is a guess; the new value is always written. HNSW removes nodes from its stored state.
• An id-based omission leaves the stored row, indexes and blobs untouched.

Existing data and known limits

• Rows and stubs written by earlier releases keep their shape until rewritten. A rewrite adds missing timestamp entries and keys; an orphaned index entry whose value no row holds stays until the index is rebuilt.
• When an old computed value cannot be resolved over a stub, its committed entry stays (beside the new one, or orphaned on delete). Main threw instead. Tracked in harper#2370.
• An id-based omission keeps a stale local row: harper#3566. A cache invalidation on a table with no secondary index still stores null: harper#3597.

⚖️ Alternatives

Other approaches weighed

• Store computed values in stubs — exact cleanup, but changes the replicated stub format (harper#3363).
• Skip the resolver over every stub — loses exact cleanup when the stub keeps the resolver's inputs.
• Route fills through _writeUpdate — adds a node-identity tie-break a source fill must avoid.

🤔 Raced first fill: should hooks see the stored created time?

When a write lands during a first fill's source call, the stored row keeps that write's created time, as a put would. The caller and @embed/@decide hooks saw the fill's own. Re-running hooks on that race was rejected: no write path shows hooks the stored stamps, and it needs a retry loop around a synchronous commit. Tracked in harper#3567.

🔧 Changes

Files:

• resources/Table.ts — residencyStub(record, id) and its callers (_writeRelocate, _writeUpdate, fill, demoteReplica); fill stamping and commit order; updateIndices with resolveOverStub and the PoolWorkerResolverRefusal it rethrows; the stub flag at delete, evict and the eviction batch; id-omission staging; the created time on cache-invalidation stubs.
• HierarchicalNavigableSmallWorld.ts — index() loads the stored node whenever one exists.
• resources/DESIGN.md, resources/indexes/DESIGN.md, DESIGN.md — fill timing, stub shape, HNSW removal rule.

Product and architecture tour

Product and architecture tour

A source fill stores, then indexes, one decided value

What does a fill write, and in what order?

Before	After
Index the source's answer, then stamp timestamps, then decide residency, then store. The caller and noCacheStore reads saw no stamps.	Stamp before resolving the caller. In the commit, decide residency on a local recordToStore, then index and store exactly that.

The fill's commit

1. Start from the source's answer — updatedRecord is already stamped and is never reassigned, so a retried commit starts over from it.
2. Keep a raced created time — a first fill that replaces a record which landed during the source call copies that record's created time, as a put would.
3. Decide residency — a node outside the residency list stores a keyed stub, or nothing under setResidencyById.
4. Index what is stored — updateIndices sees the stub or record about to be written; an id omission writes no row and touches no index.
5. Store — the row, or for an id omission only the audit record, leaving any stored row and its blobs as they were.

• Raced first fill. The stored row keeps the raced record's created time; the caller and hooks saw the fill's own (harper#3567).
• Id omission over an existing row keeps a stale local row (harper#3566); it no longer drops its index entries or blobs.

A residency stub is a keyed projection

What does a node outside a record's residency store?

Example: a stub for a record with name indexed and a computed upper

The record { id: 'k', name: 'n', body: '…' } is received on a node outside its residency. Main stored { name: 'n' }, so an index-only select returned id: undefined; with no secondary index it stored null, the shape of a delete.

Outcome: The node stores { id: 'k', name: 'n' }, plus the created time when the table has one, and never reads upper.

A residency stub is never null and never reads a computed getter. A table with no secondary index stores { id }.

Computed indexes over a stub

How is a computed index maintained when the existing record is a stub?

The stub lacks most resolver inputs, and the committed computed value is stored nowhere. So the old value is resolved over the stub as a guess: exact when the resolver reads only fields the stub keeps, unknown when it throws, possibly wrong when it falls back. The guess is removed when it differs from the new value. The new value is always written. When the resolver throws, the committed entry stays (harper#2370); main threw and the write failed. A replication worker's refusal to run application resolvers still surfaces.

• HNSW reads its own stored node for an update or removal, so an unknown old vector no longer deletes a node without unlinking its neighbours.
• Callers pass existingIsStub from the entry or staged write that supplied existingRecord; the parameter is required.

Code paths: the fill's commit and updateIndices over a stub

Table.ts:10596

recordToStore = updatedRecord        // or a copy carrying a raced record's created time
residency = getResidency(recordToStore)
if (residency excludes this node)
  recordToStore = getResidencyById ? undefined : residencyStub(recordToStore, id)
if (!omitLocalRecord || recordToStore !== undefined)   // an id omission writes no row
  updateIndices(id, currentRecord, recordToStore, …, omitLocalRecord, existingIsStub)
updateRecord(id, recordToStore, recordToStore === undefined ? { …existingEntry, value: undefined } : existingEntry, …)

Table.ts:9589

 value = resolver ? resolver(record) : record[key]
-existingValue = resolver ? resolver(existingRecord) : existingRecord[key]   // throws over a stub
+existingValue = overStub ? resolveOverStub(resolver, existingRecord) : …   // a throw → undefined
-if (value === existingValue) continue
+if (value === existingValue && !overStub) continue    // a guess proves nothing
 … retained values: not removed, re-added when overStub

✅ Verification

• 19 new unit tests (16 on LMDB; 3 are RocksDB-only) fail on origin/main sources with a forced rebuild, each with its bug signature, and pass here: sourceFillIndexes.test.js (new), residencyReads, writerResidency, residencyReplicas, evictionBatch.
• End to end: two new HTTP tests in cachedComputedAttribute.test.ts (fixture: schema, resources) fail on base (empty timestamp index; the refill logs Error committing cache update TypeError) and pass here.
• Every later guard (guess re-add, HNSW, omission, staging, blobs, created time) fails its test when reverted.
• Full unit and integration gates pass apart from failures that are red on main or pass alone (details below).

Gates and gaps

• test:unit:main 8185 pass, 0 fail.
• test:unit:resources: RocksDB 4926 pass, LMDB 3744 pass; the one failure on each is the WebSocket read-gate test, red on main and fixed there by Give the read-gate WebSocket test's fake socket the off() REST calls on close #3562.
• test:integration:all (3 shards): 9 failures — 4 local Ollama (no models here) and 5 timing tests that pass alone.
• Not covered: a forced second commit of a fill, and a replication worker's refusal over a stub.

PR QA: Do concurrent fills, invalidations and writes across HTTP workers keep a sourced table's stored rows and secondary indexes in agreement? — queued @ 3e477d6, qa-pr-harper-3610-3e477d6

Docs: none needed — bug fixes, no API or configuration change.

Refs harper#3566 harper#3567 harper#2370

_{🤖 Generated by Claude Opus 5.5 (Claude Code dev agent); posted via @kriszyp.}

Release target: main — residency stubs, replicas and preserveComputed exist only on main; v5.3 has only the fill stamping order, a backport only for a customer need
Related PRs: #2155 overlaps (eviction updateIndices calls), #3490 overlaps (fill commit and lock), #3504 overlaps (eviction), #3527 overlaps (residency stub producers), #3529 overlaps (same getFromSource commit block), #3575 overlaps (eviction batcher), #3594 overlaps (@createdtime stamping), 10 others independent
Complexity: complicated

Deferred findings

• 3a73e745182788df minor data-integrity — unitTests/resources/residencyReads.test.js — assert computed 'upper' cleanup on delete/evict of a stub — filed Stale index entries for @indexed @computed attributes survive invalidation #2370
• a11d87cb099c1bb7 minor data-integrity — resources/Table.ts — cache invalidate() on a table with no secondary index stores a null stub, so the refill restamps @createdtime — filed A cache invalidation on a table with no secondary index stores null, so the refill restamps @createdTime #3597
• 01ef86b77acfdd62 nit correctness — unitTests/resources/caching.test.js — assert the resolved fill's createdAt differs from the stored raced createdAt — filed A first source fill raced by a write resolves the caller with its own @createdTime while the row keeps the raced one #3567
• ecba6b0420fcc032 nit maintainability — resources/Table.ts — existingIsStub is a positional boolean each updateIndices caller must remember — filed Replace updateIndices' positional stub booleans with entry-derived state #3598

Late Grok findings

4 surviving (highest major) — Grok reviewed ab773eb

• major — resources/Table.ts:10577 — In a raced first fill, { ...updatedRecord, [created]: racedCreatedTime } copies only the record's own fields. That loses the structPrototype attached at resources/Table.ts:10486-10488, which holds every computed getter (defined at resources/Table.ts:9059-9069). The copy then goes to `getResi
• major — resources/Table.ts:9560 — With existingIsStub, updateIndices skips the old-value resolver for every computed index. A stub keeps the primary key and the non-computed indexed fields (resources/Table.ts:11074-11085), so a computed index derived only from those fields could be resolved, and main did resolve it. When a ref
• minor — resources/Table.ts:10463 — In a raced first fill, the caller receives the fill's own @createdTime (it resolves at :10490, before the commit runs), while the row and index store the raced one (:10574-10577). The DESIGN note at resources/DESIGN.md:310 lists only the model hooks under "Known limit" and still says the cal
• nit — resources/Table.ts:4293 — Added comments that restate code or test titles: resources/Table.ts:4293, integrationTests/resources/cachedComputedAttribute.test.ts:273-274, the first sentence of integrationTests/resources/cached-computed-attribute/schema.graphql:47, and the "stores its own copy" wording at DESIGN.md:37

Not standing: 2 merged. Grok ran 23m (23m from launch to report); the fold adjudication (cursor-grok@ab773eb24e32-4b4cfc49, claude) took 1m.

Origin — the dispatch brief this PR was written from

Two related defects in source fills and residency stubs (resources/Table.ts). (1) residencyStub(record) never copies the primary key, returns null when there are no secondary indexes and no createdTime, and evaluates computed getters (resolvedAttributeNames); callers _writeUpdate, the getFromSource sourceWrite, and demoteReplica (which patches the key by hand). transformEntryForSelect trusts the key on a stub, so an index-only select returns id undefined, and a null stub is stored as value = null. Change to residencyStub(record, id): skip resolvedAttributeNames, always return an object with stub[primaryKey] = id; update the callers and drop the manual patch. (2) getFromSource calls updateIndices before stamping @updatedTime/@createdtime and before the residency decision: a first fill indexes nothing for timestamp indexes, a revalidation deletes the existing entries, an excluded-residency fill leaves orphan index entries, and resolve(resolvedEntry), the noCacheStore return and modelHooksBefore see no timestamps. Stamp both timestamps and decide residency before updateIndices (pass omitLocalRecord), and stamp before resolve/noCacheStore. (3) The refill updateIndices has no preserveComputed, so resolver(stub) throws when a computed resolver reads a non-indexed input: on an INVALIDATED refill skip resolvers on the existing-record side only (stale old computed entries stay with #2370/#3363). Open PR #3529 edits the same sourceWrite.commit block; rebase accordingly.

Acceptance

Tests on rocksdb and lmdb: patch an INVALIDATED stub / REPLICA row then index-only select returns the real id with zero source calls; a table with no secondary indexes stores {id} not null; demote on a @computed(indexed:true) table does not throw; first fill and revalidation keep exactly one correct entry per @createdTime/@updatedTime index; noCacheStore get returns both timestamps; residency-by-id exclusion leaves no index entry; refill of @computed(indexed:true) over a non-indexed input does not throw. Show each new test failing on origin/main sources (forced rebuild) before the fix.

_{Dispatch: task harper-getfromsource-stamp-order-residency-stub · queued by finding-triage/wave-40 · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=14; full=7 @ 6331bd6}

_{Review-Attention: deep ~40m (critical: Table.ts, HierarchicalNavigableSmallWorld.ts; decisions: id-based-omission-keeps-stale-row, stub-computed-old-value-is-a-guess, fill-stamps-before-resolve, no-index-invalidation-null-stub, pool-refusal-only-rethrow, do-less-alternative, release-target-vs-evidence) @ 6331bd6}

[gemini-code-assist]

Code Review

This pull request refactors the handling of residency stubs, source fills, and computed indexes to ensure consistency between stored records and their index entries, particularly during invalidation, refills, and id-based omissions. Key changes include preserving the primary key and creation timestamps on stubs, preventing computed index updates from running resolvers during replica demotions, and ensuring HNSW updates/removals start from the stored node. Feedback on these changes highlights two potential issues: a potential TypeError in residencyStub if the record is null or undefined, and a falsiness bug in HierarchicalNavigableSmallWorld.ts where a nodeId of 0 would be incorrectly treated as falsy.