[kriszyp]

⊙ Problem

Choosing the leading AND condition failed three ways:

• A filter-only condition (IS NULL, LIKE '%x%') estimates Infinity, which made SQL !=/NOT IN and nested FIQL groups fail with 403 under allowFullScan: false.
• in, an empty starts_with and an unindexed range scan every row, yet could lead before an indexed condition.
• A relationship condition inside an OR restricted which related records a row selects (harper#3762).

Changing the lead also changed which related record two conditions on one relationship had to meet, so a finite estimate could widen a collection DELETE.

🤔 Is the requirement right?

Yes: both rules are the owner's decisions.

💡 Solution

• The AND estimate skips non-finite legs; orderConditions leads with the narrowest condition that can drive an index scan.
• Conditions of one AND on a relationship share one set of filters, so they meet one related record whichever leads; an OR condition decides rows only.

⚖️ Alternatives

Rejected approaches

• Infinity estimates for full-scan legs — count: 'estimated' and the lazy index switch read them as counts.
• Rank every leg by "can drive" — reorders the record filters behind the lead.
• Keep rows lead-dependent, as on main — the framing review found the finite estimate could widen a collection DELETE.

🤔 Same-record matching changes a result main pinned

Fix wrong rows, unfiltered selects and ignored sorts in relationship queries pinned that a non-leading AND matches a parent whose children meet the conditions separately. That test now expects no match, as a leading join gave.

🤔 An unindexed foreign key never drives

Its scan reads the local table once per related record, which the estimate does not price, so an indexed condition leads even under allowFullScan: true.

🔧 Changes

• resources/search.ts:
  • conditionDrivesIndex follows searchByIndex's full-scan and missing-index decisions, and relationshipScanRuns extends it hop by hop. Look hardest here: it must stay in step with the executor, and a test checks them leaf by leaf.
  • executeConditions and mapConditionsToFilters take a scope where main threaded inUnion; filterByType registers sibling filters per scope under a symbol key, and the first condition checks the whole set per related record. A to-one relationship shares only its inner scope.
  • Join and selection restrictions moved behind relationshipRestriction, keyed by symbol, so a relationship may be named filters or get.
  • A key condition still rejects a record before any related record loads; an omitted or singleton-null attribute reads as the primary key, and a one-element ['id'] path estimates 1.
• resources/Table.ts: the lead promotion, which keeps a synthetic sort (or a user condition on the sort attribute unless allowFullScan is false) in front; a nested select's conditions follow the same rule.
• resources/DESIGN.md: the estimator, lead, scope and OR rules.

🤔 Async relationship selection still allocates per row

Selecting a to-many relationship whose related records load asynchronously builds a match arrow and a matchingIds closure for each parent row, the same as main. Hoisting the arrow is a separate cleanup.

✅ Verification

• Unit tests fail on the merge base: relationshipConditionScope.test.js (split-child parents in every order, reserved names, two hops, nested selects, collection DELETE, OR branches), estimateFilterOnlySibling.test.js (reducer, SQL mutations, lead selection) and key-only rejection.
• HTTP: a collection DELETE on split-product brands and the SQL conjunction cases with sql.allowFullScan: false.
• Fixtures that wrote a join's filters through filtered.children now go through relationshipRestriction (async reads, key invalidation and the nested-selection cancellation).
• test:unit:typestrip:resources:core and the two touched integration files pass.

Fixes #3725
Closes #3739
Closes #3813

_{🤖 Generated by Claude Opus 5.5 via Claude Code; posted via @kriszyp.}

Release target: v5.4 — fixes a regression from #3601, which is milestoned v5.4
Related PRs: #3601 overlaps (fixes a regression it introduced); #3776 overlaps (merged first); #3775 overlaps (merged first); #3772 overlaps (search.ts and Table.ts); 6 others independent
Complexity: high
Framing-Verdict: chosen-approach-sound

Deferred findings

• db316ad3c3eb6566 major performance — resources/search.ts — Allowed full scans lose an executable relationship lead with an unindexed foreign key — NOT FILED
• 6ce214f314a60636 major correctness — resources/Table.ts — A non-driving sort-aligned lead suppresses sorting or rejects a null-equality query — NOT FILED
• b1c07dbbe9115cd7 minor performance — resources/search.ts — conditionDrivesIndex recomputed recursively per ancestor, O(depth^2) — NOT FILED
• a4bee76312e168cf minor performance — resources/Table.ts — a narrower non-driving condition displaces the synthetic sort, and promotion then skips the sort and promotes another indexed condition — NOT FILED
• 563aa49357eaca44 minor correctness — resources/search.ts — starts_with null/undefined value: TypeError in searchByIndex on lead or lazy index switch; filter path matches 'null' prefix — NOT FILED
• de7af54d4747481c minor performance — resources/search.ts — async to-many selection allocates a match closure and a matchingIds closure per parent row — NOT FILED
• e937c699d7582749 nit test-coverage — unitTests/resources/estimateFilterOnlySibling.test.js — broad-in plus sort case asserts only the plan, not the rows — NOT FILED
• 052305fc5f7fa204 nit reliability — resources/search.ts — Sibling predicates on a leading join rely on lazy iteration of the lead — NOT FILED
• c98b870701ea83f7 nit performance — resources/search.ts — intersectionEstimate multiplies Infinity into the incoming-row hint — filed intersectionEstimate multiplies Infinity into the incoming-row hint used by filter index switching #3726

🤔 defer this PR's major performance finding db316ad3c3eb6566 (Allowed full scans lose an executable relationship lead with an unindexed foreign key) to a separate issue instead of fixing it here?

Origin — the dispatch brief this PR was written from

In estimateCondition (resources/search.ts ~2214-2222), the AND branch multiplies the running estimate by each sub-condition's estimated_count / N without checking that it is finite. Filter-only legs get estimated_count = Infinity (~2283), so any filter-only sibling (IS NULL, LIKE '%x%', LIKE '%x', NOT (...)) turns a finite index-backed estimate into Infinity. Since harper#3601 lowers SQL != and NOT IN into compound AND groups, SQL SELECT/UPDATE/DELETE with != or NOT IN after a filter-only predicate, and nested FIQL AND groups, now fail with 403 under allowFullScan: false, and count: 'estimated' returns null. Skip non-finite sub-estimates in the AND intersection; if every leg is Infinity the group stays Infinity.

Acceptance

Unit test on rocksdb and lmdb covering SQL SELECT/UPDATE/DELETE with != and NOT IN after IS NULL / LIKE '%x%' / LIKE '%x' / NOT(...), a nested FIQL AND group under allowFullScan:false (200, not 403), Table.search({count:'estimated'}) non-null, and orderConditions picking the indexed leg. Show each new test failing on origin/main sources (forced rebuild) before the fix.

_{Dispatch: task harper-estimate-and-infinite-sibling · queued by finding-triage/wave-42 · ran by claude/sonnet/xhigh · worker kzyp-xps-1}

Late Grok findings

2 surviving (highest major) — Grok reviewed 4931740, ruled against e864fea

• major — resources/search.ts:2218 — An AND group that previously estimated Infinity now takes a finite estimate from a leg that cannot drive a scan: an unindexed or empty starts_with, an unindexed range, or an indexed in. The parent AND then sorts that group ahead of an indexed sibling. Example: indexed c = common (50 of 100 r
• nit — resources/search.ts:2216 — Open comment finding about the new line that says what Infinity means in the reducer. It is the only comment the diff adds, and it states a convention that estimateConditionForTable shares across functions, which the code cannot express on its own; author's call

Not standing: 1 merged. Grok ran 15m (16m from launch to report); the fold adjudication (cursor-grok@4931740106a6-37ccf37f, claude) took 1m.

Late Grok findings

4 surviving (highest minor) — Grok reviewed b1cdfad, ruled against 0e74d46

• minor — resources/search.ts:1056 — a starts_with condition whose value is null or undefined throws a TypeError from value.toString() once searchByIndex runs it, either as the lead or through the lazy index switch at resources/search.ts:2092; when it only filters, it matches values beginning with the text "null"/"undefin
• minor — resources/search.ts:71 — when OR branches constrain the same to-many relationship, selection projects related records from the first branch's slot only, so parents matched by a later branch come back with children: []
• minor — resources/search.ts:2221 — conditionDrivesIndex re-walks descendant groups at every ancestor, so planning is O(depth²) in nesting depth
• nit — unitTests/resources/estimateFilterOnlySibling.test.js:317 — comment cleanup is mostly done; the column-narration comment is still there

Not standing: 2 accepted-known. Grok ran 12m (13m from launch to report); the fold adjudication (cursor-grok@b1cdfad2fa38-48603d22, claude) took 1m.

Late Grok findings

2 surviving (highest major) — Grok reviewed 6aa3504, ruled against a36d1b7

• major — resources/search.ts:2300 — Composite (array) primary keys: once an AND group with an infinite-estimate leg gets a finite estimate, the group can lead ahead of id = [...] (estimate 1, search.ts:2329). The id condition then becomes a post-filter. Equality compares with === against a freshly decoded array (search.ts:1938
• major — resources/search.ts:1900 — Selection on an async to-many relationship still allocates a new (value) => value != null && match(value) arrow per parent row, plus the matchingIds closure at :1901. It could be hoisted once per condition. No measurement covers it, so the hot-path floor applies.

Not standing: 1 resolved. Grok ran 18m (18m from launch to report); the fold adjudication (cursor-grok@6aa3504fda13-83a12601, claude) took 1m.

Late Grok findings

1 surviving (highest major) — Grok reviewed cfda0d1

• major — resources/search.ts:1880 — An AND group of two conditions on the same to-many relationship, placed inside an OR, now admits a parent whose related rows meet the conditions separately. When the label condition leads, its join is written to a fresh map (resources/search.ts:161), and the !inUnion gate keeps the tier conditio

Not standing: none. Grok ran 13m (14m from launch to report); the fold adjudication (cursor-grok@cfda0d1eedfc-775b5c75, claude) took 1m.

Late Grok findings

5 surviving (highest major) — Grok reviewed 1f74746

• major — resources/search.ts:1885 — When two conditions on the same to-many relationship are ANDed inside an OR, they no longer have to be met by the same related record. Example: (children.label=sw=match & children.tier=gold) | id=nobody returns parent-3, because one of its children matches the label and a different child is gold
• major — resources/Table.ts:7333 — The exemption that keeps a sort-aligned condition first only covers the synthetic sort pseudo-condition. A user's own condition on the sort attribute (e.g. priority=in=(1) with sort(priority)) cannot drive an index, so :7334 promotes a broader indexed condition ahead of it. Then :7427 sets
• major — resources/search.ts:2356 — A multi-hop relationship path always returns false from conditionDrivesIndex, so when the estimate already sorted a relationship join first, findIndex at resources/Table.ts:7334 promotes any broader local driving condition ahead of it. Example: group='g1' (100 rows) displaces kids.name='a'
• minor — resources/search.ts:2356 — A condition with attribute [] unwraps to undefined, which classifies it as driving the primary store, so it can be promoted to lead. Code-trace only: I did not confirm that searchByIndex then full-scans; reachability depends on a caller sending an empty path.
• nit — resources/search.ts:160 — New comments at resources/search.ts:160, resources/search.ts:1884 and unitTests/resources/relationshipAsyncReads.test.js:418 repeat what resources/DESIGN.md:276 and the test title at :402 already say.

Not standing: none. Grok ran 15m (16m from launch to report); the fold adjudication (cursor-grok@1f74746b6a26-f10ad633, claude) took 1m.

Late Grok findings

3 surviving (highest major) — Grok reviewed eb07c09

• major — resources/search.ts:1989 — With two AND conditions on the same to-one relationship, the id-set shortcut is turned off, so every candidate row loads its related record
• nit — resources/search.ts:1016 — Added comments describe the code next to them
• nit — resources/Table.ts:7389 — The sort-streaming exemption (syntheticOrderCondition / orderAlignedCondition) is not mentioned in resources/DESIGN.md. Grep finds both names only in Table.ts

Not standing: 1 merged. Grok ran 16m (16m from launch to report); the fold adjudication (cursor-grok@eb07c0931360-5eadf0c6, claude) took 1m.

Late Grok findings

1 surviving (highest minor) — Grok reviewed 7005532

• minor — resources/Table.ts:7777 — In a nested select, two AND conditions that go through the same to-many relationship can still be satisfied by different related records. Top-level queries now require both conditions to match the same related record, so the two paths disagree

Not standing: 1 merged. Grok ran 18m (18m from launch to report); the fold adjudication (cursor-grok@700553240f78-8afde2c0, claude) took 1m.

_{Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,cursor-grok; pending=cursor-grok; late=cursor-grok:pending; adjudicated=domain,domain(codex); blocked=claude(fallback)(out-of-budget); declined=muse; rounds=32; full=10 @ 90ebd4e}

_{Review-Attention: deep ~45m (critical: Table.ts, search.ts; decisions: same-record-and-semantics, or-condition-decides-rows-only, promote-driving-lead-unconditionally, and-estimate-ignores-filter-only, pr-scope, do-less-alternative, deferred-db316ad3) @ 90ebd4e}