[kriszyp]

A caching-table query keeps a row the source must reload under negated or grouped key filters, and reloads it before sorting.

⊙ Problem

An invalidated row should reload from the source when queried (#3772). Two cases broke that:

• Negated or grouped primary-key conditions (id=not_in=(b), (id=a|id=c)) dropped the row (#3783).
• Content sorts placed the row by its missing value, so limit/offset cut the wrong page (#3786).

🤔 Is the requirement right?

Yes. The owner accepted reloading every invalidated row in a sorted range.

❓ Your call: Keep that contract, or rank by cached values without reloading?

💡 Solution

• Negations and all-key groups carry a key-only test (keyFilter on negation, on group), read via keyPredicate where rows are kept or dropped. Look hardest at the group: every member needs keyFilter.
• Content sorts reload the row before reading sort keys (prepareForSort, called in sort) and reuse the fresh row.

With c invalidated, GET /T/?id=ne=b&sort(+body)&limit(1) ranks c by its reloaded body.

🤔 Reloads before a sort run one at a time

500 invalidated rows mean 500 serial source calls; parallel loading is tracked in #3908.

🤔 Not covered here

Nested relationship selections and non-leading key conditions (#3907). Index-supplied order, expired rows and stale-while-revalidate rows off the page keep cached values (#3909). Index-covered ['id'] selects are unchanged.

⚖️ Alternatives

Three approaches set aside

• Give negations and groups an idFilter. Relationship id mapping reads idFilter, so children.id=not_in=… would match a dangling foreign key.
• Reload inside getAttributeValue. That skips the materializer's stale-while-revalidate, only-if-cached and resident-load rules, and loads twice.
• Reload stale-while-revalidate rows before sorting (the first plan). Each would start a background load even far off the page; the hook is now asked first.

🔧 Changes

• resources/search.ts: keyFilter on primary-key comparators, negations and all-key groups; keyPredicate, which joinFrom also reads.
• resources/Table.ts:
  • transformToEntries reads keyPredicate.
  • transformEntryForSelect exposes prepareForSort: rows kept by the stale-while-revalidate hook use stale sort values; rows deleted at the source are skipped.
  • transformToOrderedSelect calls it, stops at the first primary-key clause, and moves past an emptied group.
  • getAttributeValue reads a stub's key.
• resources/DESIGN.md: the stub rule.

✅ Verification

• Every row of both issue tables fails on main and passes on RocksDB and LMDB, with one source call per kept stub.
• HTTP: id=ne=b&sort(+body)&limit(1) returns the right row; 3 integration cases fail on main, 8/8 pass.
• Performance: a sorted read with no stubs adds a property read per key check and a flag test per row, no allocation; CPU matches main (numbers below). Stub reloads run serially (see the 🤔 above).
• User-facing docs: none; no API changes.
• PR QA: Do concurrent sorted, paged reads over many invalidated cache rows return each row once, in fresh order, with one source load per row? — queued @ af653b1, qa-pr-harper-3923-af653b1

Commands and results

• invalidatedStubKeyAndSortReads.test.js (36 tests): not_in, not_between, not_starts_with, AND/OR groups, limit/offset/descending, metadata and primary-key sorts, deleted at the source, source rejection, a close during a reload, only-if-cached, stale-while-revalidate, row filters. Against main's Table.ts/search.ts, 16 tests fail per engine, including the negated-join cases added to sortedKeyInvalidationReads.test.js. The sorted-read closure tests in residencyReads.test.js now close during the pre-sort reload, or, with a key-led sort, during materialization.
• pk-filter-invalidated-reads.test.ts: limit(1) and limit(1,2) pages, id=not_in=, a grouped condition.
• test:unit:resources: core:a-n 2,706 and core:o-z 2,961 passing, 0 failing. :indexes: 761 pass and 4 derived-index durability tests fail; those 4 pass alone (160/160) and are green on main CI.
• Integration resources/**: 277/278; the rate-limiter smoke that failed passes alone.
• test:unit:main did not finish: lock-timing tests in components stalled under host load (load average about 26). Left to CI.
• 5,000-row sorted read, no stubs, median per-query CPU over 4 runs: main 19.0–19.8 ms, this branch 18.8–19.7 ms.

_{🤖 Generated by Claude Opus 5.5 (Anthropic); posted via @kriszyp.}

Framing-Verdict: chosen-approach-sound (eb0a6ae932c1)

Release target: v5.4 — follows up #3772 (milestoned v5.4, not on v5.3); a query-correctness fix on main, no backport requested
Related PRs: #2155 overlaps (same joinFrom joined-filter loop), #3772 overlaps (merged; this follows it up), #3927 overlaps (merged; relationship conditions now existential, this rebased onto it), 7 others independent
Complexity: complicated

Deferred findings

• b3de6d8a86f2ce34 major performance — resources/Table.ts — pre-sort stub reloads run serially and scale with the matched set, not the page — filed Sorted reads over a caching table reload invalidated rows one at a time before sorting #3908
• 8497d5e0218cb0f2 major correctness — resources/Table.ts — nested relationship sort never receives prepareForSort — filed Nested relationship selections and non-leading relationship conditions drop or misorder invalidated cache rows #3907
• 62cdde79c28b147d minor correctness — resources/Table.ts — nested relationship predicate drops null invalidated child before predicate; no-predicate path returns deleted child as null — filed Nested relationship selections and non-leading relationship conditions drop or misorder invalidated cache rows #3907
• ea39a25260c849ea minor correctness — resources/Table.ts — null stub under an index-ordered leading sort groups by missing value — filed Index-supplied sort order and expired rows still sort caching-table rows by cached values #3909
• 5c73c13b5de51a26 minor correctness — resources/Table.ts — onlyIfCached scalar computed sort over null stub invokes resolver with null record — filed Index-supplied sort order and expired rows still sort caching-table rows by cached values #3909
• 9349bf2a4a40387e minor correctness — resources/Table.ts — SWR-kept stub sorted outside the page is never revalidated — NOT FILED
• 9f80f430282297a9 minor correctness — resources/Table.ts — expired rows without INVALIDATED/EVICTED flags are sorted by stale value then shown fresh — NOT FILED
• fefbf67b18f168ff minor correctness — resources/search.ts — relationship-group compileConditions combiner omits keyFilter, dropping parents whose only child is a null stub — filed Nested relationship selections and non-leading relationship conditions drop or misorder invalidated cache rows #3907

🤔 defer this PR's major performance finding b3de6d8a86f2ce34 (pre-sort stub reloads run serially and scale with the matched set, not the page) to #3908 instead of fixing it here?

Bundle — this PR covers 1 ref.

Closes #3783

Origin — the dispatch brief this PR was written from

Invalidated cache rows: keep them for negated/grouped key filters and sort them after reload

Fix #3786 and #3783 in one PR. Both follow up merged #3772 in the caching-table INVALIDATED-stub keep/reload path. Owner decision (2026-11-02): a query may call the source to reload an invalidated row it would return; that cost is accepted.

• Invalidated cache rows are left out when a negated or grouped primary-key condition filters the query #3783: a negated primary-key condition (not_equals, not_in, not_between, not_starts_with, …) and an AND/OR group of primary-key conditions carry no idFilter, so transformToEntries (resources/Table.ts, processEntry/applyFilters) drops a null INVALIDATED stub and never reloads it.
• Sorted reads place an invalidated cache row by its missing sort value, before reloading it #3786: a sorted read places a stub by its missing sort value before reloading it, so the reloaded row lands in the wrong position and limit/offset apply to the wrong order.

Order of work:

1. Shared fixture: caching table with no @indexed attribute, rows a/b/c with distinct body, invalidate c (_writeInvalidate), source returns { body: 'fresh ' + id }; RocksDB and LMDB. Reproduce both issues' tables and show them failing on origin/main sources (forced rebuild).
2. Invalidated cache rows are left out when a negated or grouped primary-key condition filters the query #3783: give the negation wrapper in filterByType (resources/search.ts, "negated conditions are filter-only") and AND/OR groups of primary-key conditions in mapConditionsToFilters an idFilter (negated key test, group of key tests), so the stub is kept and reloaded; assert the source is called once per kept stub. A record guard or non-key condition must still drop the stub.
3. Sorted reads place an invalidated cache row by its missing sort value, before reloading it #3786: in the sorted path, reload kept stubs before computing their sort keys (or defer placement until the reload resolves) so a reloaded row sorts by its fresh value and limit/offset apply to the corrected order; cover id in (...) and id ge a, with and without limit: 1. A sorted read with no stubs must cost nothing extra; measure it.
4. Update the stub paragraph in resources/DESIGN.md.

Out of scope: computed-resolver retirement (harper#3782), relationship joins (joinTo cannot apply the key rule), eviction semantics. Coordinate with the in-flight index-covered select work (task harper-covering-index-select-projection, branch perf/covering-index-select), which serves ['id']/['id', attr] selects from the index without loading records: if a covered query reaches an invalidated row, note the interaction in the PR body rather than changing that branch. Release target v5.4 (main).

Acceptance

Unit tests for every row in the #3783 and #3786 issue tables fail on origin/main and pass, on both storage engines; the source is called once per kept stub; an HTTP integration test on a caching table (invalidate a row, then GET /T/?id=ne=b&sort(+body)&limit(1)) returns the reloaded row in its correct position.

_{Dispatch: task harper-invalidated-stub-pk-sort · queued by kris-session · ran by claude/opus/high · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=13; full=7 @ 7056332}

_{Review-Attention: deep ~30m (critical: Table.ts, search.ts; decisions: keyfilter-separate-from-idfilter, reload-before-sort-vs-sort-stale, do-less-alternative, deferred-b3de6d8a) @ 7056332}

[gemini-code-assist]

Code Review

This pull request optimizes and fixes the handling of invalidated or evicted cache rows during primary-key filtered, negated, grouped, and sorted reads. It introduces a prepareForSort mechanism to reload stubs before sorting, uses a unified keyPredicate helper for filtering, and adds comprehensive tests to verify these behaviors. The review feedback recommends a safer check for thenable objects and avoiding variable shadowing in Table.ts.