diff --git a/resources/DESIGN.md b/resources/DESIGN.md index f16e10335d..d667b56102 100644 --- a/resources/DESIGN.md +++ b/resources/DESIGN.md @@ -266,7 +266,7 @@ A contract is `{ path, record?, get/post/put/patch/delete: { query?, body?, resp - The `transactional()` wrapper resolves pending bodies before `allowCreate`/`allowUpdate` and instance dispatch, passing the same concrete object to both. For POST/PUT/PATCH it collects async name/value pairs with `resolveBody.ts`; Node and Web streams pass through unchanged, and generic iterator parts with additional fields are rejected rather than silently reshaped; QUERY resolves before resource loading so its projection participates in `allowRead`. Other methods resolve promises without collecting iterables: CONNECT carries live messages and PUBLISH can carry stream payloads. Authorization is latched before awaiting ordinary write bodies, and record loading can overlap body arrival. Decoder failures remain 400/413 rather than being remapped to authorization failures. Table's direct promised-hook overloads remain supported; restricted arrays retain their concrete-record refusal and explicitly listed readonly fields retain full-PUT backfill. `../unitTests/resources/requestBody.test.js` pins resolution, object identity, authorization timing and live-message preservation. - **Overriding a static entry point takes over its whole contract.** Assigning `static post`/`put`/`patch` on a subclass shadows the `transactional()`-wrapped static, so none of that wrapper runs — including body resolution and the `allowCreate`/`allowUpdate` gate. That is by design (it is how `login.ts` implements a deliberately pre-authentication endpoint), and it means an override owns two obligations the wrapper would otherwise have met: (1) **Resolve or consume `data`:** ordinary REST bodies arrive as a pending promise; multipart forms arrive as `AsyncIterable<{ name, value }>`, without an opt-in flag. A static override can consume parts directly or delegate to `super.post(target, data)` (likewise PUT/PATCH). The inherited wrapper collects fields and repeated names, materializing each streaming file into a replayable Blob before advancing. Multipart filenames are own enumerable properties, and Blob decoding restores that shape; Bun’s inherited name setter otherwise hides the filename from a later encoding. Generic iterator bodies use exact name/value pairs; opaque record iterators belong in a static handler or a record property. `resolveBody.ts` owns named-part aggregation; the multipart iterator's internal `RESOLVE_BODY` hook owns file consumption, avoiding an HTTP parser dependency in Resource. Raw streamed Blobs retain `createBlob(Readable)`'s existing storage-backed read contract: save them before reading, or delegate to materialize the form. Collection buffers up to the existing HTTP body limit; applications needing large streaming uploads should consume in a static override. `defineResource.ts` uses the same resolver before body-schema validation, and `login.ts` resolves streamed credentials explicitly. (2) **The override makes the access-control decision:** no `allow*` predicate runs for it; an override that is not meant to be public must check authorization itself. - **FK-side relationship accessors (`relationship.from`) are a synchronous contract** (v4 parity). The resolvers built in `Table.updatedAttributes` must read through `getSync`/`getEntry` (which block on a RocksDB block-cache miss, like v4's LMDB page faults) — never `get()`, whose `async: true` path returns a Promise on a cache miss and would leak an intermittent value-or-Promise to `record.` in user code. Pinned by the "relationship property access is synchronous" test in `../unitTests/resources/query.test.js`, which stubs `store.get` to throw. Known gap: the `.to` side (`relatedTable.search(...).asArray`) can still yield Promises when the related table is a caching table whose row needs source revalidation ([harper#3588](https://github.com/HarperFast/harper/issues/3588)). **HTTP caching and relationship resolution** (pinned by `@relationship(from:) ETag / conditional GET invalidation` in `../unitTests/resources/restSelectProjectionFixes.test.js`): resolving a non-empty `relationship.from` foreign key marks the active request context uncacheable (`lastModified = UNCACHEABLE_TIMESTAMP`, skipping caching-source contexts) so single-record GETs with `select(...)` or `.property` projection do not emit a parent-only ETag or return a stale 304 after the target row changes, and `relationship.to` projections are likewise uncacheable through `relatedTable.search(...)` (`Table.ts`); **known gap:** `@enumerable` relationships (both `from` and `to`) that are not explicitly projected resolve inside `toJSON` during response serialization, after REST computes ETag/304, so a target row update can still produce a stale parent-version 304. -- **Queries read related records through the async-capable path; only the accessor is synchronous.** Under asynchronous iteration, a query that selects, filters on, joins through, or sorts by a relationship must not call the resolver's synchronous form, which blocks the worker once per related record that misses the in-memory caches; under synchronous iteration it must, because nothing can wait on a Promise there. Each relationship resolver also carries `resolveKeys` (the FK value, or the `to` index scan's ids) and `loadEntries` (the `returnEntry` result read with `async: true`, settling to a Promise when a read goes to storage; once one read does, the rest go at most `RELATED_READS_IN_FLIGHT` at a time, so one wide row cannot queue thousands of background reads). Consumers that map entries to values decide one-or-many by the resolver's `toMany` flag, not `Array.isArray`, because a single related record may have a composite (array) key; `filterMissing` is applied after loading. Pinned by `../unitTests/resources/relationshipAsyncReads.test.js`, which counts the related table's blocking and Promise-returning reads, including a real cold read (flushed SST, cleared record cache) on RocksDB; its filter and sort cases use a related attribute that is deliberately not indexed, because an indexed one makes the planner lead with an index join instead, which for a many-to-one relationship never reads the related record. **Select:** `Table.transformEntryForSelect` takes the related entries from `loadEntries` and projects them through the target table's select transform as before; the row's projection waits on the Promise. A selection with nested `conditions`, `sort`, `offset` or `limit` (`Table._prepareRelationshipSelection`) instead reads the related keys (`resolveKeys`, or a filter's matched ids) and loads each candidate with `async: true` ahead of its yield point, so cancellation still stops reads at the first yield. Loading them through `loadLocalRecord` instead measured about 0.3 µs per warm related row slower (`../unitTests/resources/relationshipWarmReads.bench.js`) and changed analytics and last-modified accounting. **Filter:** the relationship filter from `filterByType` keeps its synchronous form and adds a `maybeAsync` variant built on `loadEntries`; AND/OR groups and multi-hop paths chain through `maybeAsync`, and `transformToEntries` applies filters through it. The synchronous form stays for callers that need a boolean immediately: a custom index's pushed-down `recordFilter` and the synchronous join bodies used under sync iteration. One synchronous read path remains under a `maybeAsync` filter: the leaf comparator's adaptive switch to an index-derived id set (`attributeComparator`) scans the index and, for a `.to` relationship, reads each match with `getSync`. **Join:** a leading relationship condition's `joinTo`/`joinFrom` iterate the related search asynchronously under async iteration (`scanRelated`), reading related records with the query's read transaction; the scan is the only Promise a join returns. It stops, and yields nothing more, once its iterator is returned, as does the sorted post-ordering iterator. Request teardown (`closeOwnedReadIterators()`, the long-transaction monitor) releases the read snapshot through `onDone()` without returning the iterator, so after each awaited read both check `getReadTransactionGuard(readTxn)`, which throws the same `ReadSnapshotExpiredError` (503) or open-too-long error a RocksDB range read over a released snapshot does; `isDone` alone is no signal, because RocksDB handles never set it. LMDB, deprecated, keeps only the `isDone` check. **Sort:** `getAttributeValue` resolves relationship hops with `loadEntries`; the post-ordering iterator awaits the keys before sorting. +- **Queries read related records through the async-capable path; only the accessor is synchronous.** Under asynchronous iteration, a query that selects, filters on, joins through, or sorts by a relationship must not call the resolver's synchronous form, which blocks the worker once per related record that misses the in-memory caches; under synchronous iteration it must, because nothing can wait on a Promise there. Each relationship resolver also carries `resolveKeys` (the FK value, or the `to` index scan's ids) and `loadEntries` (the `returnEntry` result read with `async: true`, settling to a Promise when a read goes to storage; once one read does, the rest go at most `RELATED_READS_IN_FLIGHT` at a time, so one wide row cannot queue thousands of background reads). Consumers that map entries to values decide one-or-many by the resolver's `toMany` flag, not `Array.isArray`, because a single related record may have a composite (array) key; `filterMissing` is applied after loading. Pinned by `../unitTests/resources/relationshipAsyncReads.test.js`, which counts the related table's blocking and Promise-returning reads, including a real cold read (flushed SST, cleared record cache) on RocksDB; its filter and sort cases use a related attribute that is deliberately not indexed, because an indexed one makes the planner lead with an index join instead, which for a many-to-one relationship never reads the related record. **Select:** `Table.transformEntryForSelect` takes the related entries from `loadEntries` and projects them through the target table's select transform as before; the row's projection waits on the Promise. A selection with nested `conditions`, `sort`, `offset` or `limit` (`Table._prepareRelationshipSelection`) instead reads the related keys (`resolveKeys`, a filter's matched ids, or a leading join's keys) and loads each candidate with `async: true`, frozen like a `loadEntries` result, ahead of its yield point, so cancellation still stops reads at the first yield. Its nested predicate (AND/OR groups carry a `maybeAsync` form) and the recheck `transformEntryForSelect` makes after projection both run through `maybeAsync`; a pending recheck settles before the row is projected. Loading them through `loadLocalRecord` instead measured about 0.3 µs per warm related row slower (`../unitTests/resources/relationshipWarmReads.bench.js`) and changed analytics and last-modified accounting. **Filter:** the relationship filter from `filterByType` keeps its synchronous form and adds a `maybeAsync` variant built on `loadEntries`; AND/OR groups and multi-hop paths chain through `maybeAsync`, and `transformToEntries` applies filters through it. The synchronous form stays for callers that need a boolean immediately: a custom index's pushed-down `recordFilter` and the synchronous join bodies used under sync iteration. One synchronous read path remains under a `maybeAsync` filter: the leaf comparator's adaptive switch to an index-derived id set (`attributeComparator`) scans the index and, for a `.to` relationship, reads each match with `getSync`. **Join:** a leading relationship condition's `joinTo`/`joinFrom` iterate the related search asynchronously under async iteration (`scanRelated`), reading related records with the query's read transaction; the scan is the only Promise a join returns. It stops, and yields nothing more, once its iterator is returned, as does the sorted post-ordering iterator. Request teardown (`closeOwnedReadIterators()`, the long-transaction monitor) releases the read snapshot through `onDone()` without returning the iterator, so after each awaited read both check `getReadTransactionGuard(readTxn)`, which throws the same `ReadSnapshotExpiredError` (503) or open-too-long error a RocksDB range read over a released snapshot does; `isDone` alone is no signal, because RocksDB handles never set it. LMDB, deprecated, keeps only the `isDone` check. **Sort:** `getAttributeValue` resolves relationship hops with `loadEntries`; after a to-many hop the rest of the path resolves per related record under one `RELATED_READS_IN_FLIGHT` budget for the whole path (records resolve one at a time while a deeper to-many hop remains), giving an array that compares element by element in relationship order; the post-ordering iterator awaits the keys before sorting. **Select restrictions and the id-filter key space (`filterByType`):** `filtered[name]` holds a leading join's `Map` (its `filters` narrow the scan) or, for AND-scope non-leading conditions on a to-many relationship, one selection object registered when the filters are built: its `fromRecord` keeps the related entries matching every such condition and returns their entry keys, and a multi-hop condition passes it (or the join `Map`) down as the next level's map. Entries are read as own properties of a known shape (`relationshipRestriction`: a `Map`, or a selection with `fromRecord`), because a relationship name can equal an inherited member or the maps' own `filters`/`fromRecord`/`hasMappings` keys; such a name gets no restriction. An OR scope (`inUnion`) still registers lazily, first evaluated condition wins: a known gap, since OR branches should union ([harper#3762](https://github.com/HarperFast/harper/issues/3762), with OR-under-join and to-one→to-many paths). The id-filter shortcut tests `local[from ?? primaryKey]`, so a `to` relationship takes only an id set the adaptive index switch mapped through `to` (`idFilter.mappedTo`); a related primary-key condition, or a deeper hop's propagated set, holds related keys. A missing related record never reaches a predicate. Pinned by `../unitTests/resources/relationshipPrimaryKeyFilters.test.js` and `../unitTests/resources/relationshipQueryContracts.test.js`. - **A combined `@relationship(from, to)` joins `related[to] == local[from]`, never the local primary key**, and only on an array-typed attribute (a scalar one is logged and gets no resolver). Four paths read `from` independently and must stay in step: the array resolver in `Table.updatedAttributes`, the index-driven join in `searchByIndex` (`joinTo` then `joinFrom`), the join-filtered select in `Table.transformEntryForSelect` (`resolver.from`), and a join filter's switch to an id set in `filterByType`. Pinned by the "@relationship with both from and to" block in `../unitTests/resources/query.test.js`, which uses a related record keyed by the local primary key as a decoy. Two more readers (`drivesRebuildingIndex`, the join estimate's `fromIndex` weighting in `estimateCondition`) affect only condition ordering, not join results, so they are not pinned by that block. - When adding a new early-return path inside a commit handler in `_writeUpdate`, follow the blob-cleanup protocol in "Blob orphan cleanup" below. - If you add a new top-level section to `Table.ts`, drop a `// #section: ` marker at its start and add a row to the section map above. diff --git a/resources/Table.ts b/resources/Table.ts index 5684161ff8..8eae8d6164 100644 --- a/resources/Table.ts +++ b/resources/Table.ts @@ -171,6 +171,8 @@ import { executeConditions, resolveComparator, filterByType, + matchesFilters, + relationshipRestriction, fullTextComparatorMode, ALTERNATE_COMPARATOR_NAMES, } from './search.ts'; @@ -7540,8 +7542,7 @@ export function makeTable(options): TableResourceClass { (attribute.limit !== undefined && (!Number.isSafeInteger(limit) || limit < 0)) ) throw new ClientError('Nested limit and offset must be non-negative safe integers'); - const filterMap = filtered?.[attribute.name]; - const childFiltered = Object.assign(Object.create(null), filterMap); + const childFiltered = Object.assign(Object.create(null), relationshipRestriction(filtered, attribute.name)); const rejectUnsupported = (conditions) => { for (const condition of conditions) { if (condition.conditions) rejectUnsupported(condition.conditions); @@ -7572,9 +7573,13 @@ export function makeTable(options): TableResourceClass { if (!predicate) throw new ClientError('This predicate is not supported in a relationship selection'); return predicate; }); - return operator === 'or' + const any = operator === 'or'; + const combined: any = any ? (record, entry) => predicates.some((predicate) => predicate(record, entry)) : (record, entry) => predicates.every((predicate) => predicate(record, entry)); + if (predicates.some((predicate) => predicate.maybeAsync)) + combined.maybeAsync = (record, entry) => matchesFilters(predicates, record, entry, any); + return combined; }; const predicate = conditions.length ? compile(conditions, attribute.operator) : undefined; const plural = Boolean(resolver.to || attributes.find((item) => item.name === attribute.name)?.elements); @@ -7590,6 +7595,8 @@ export function makeTable(options): TableResourceClass { let selectionStopped = false; const relationshipSelection = (record, entry?, isSync = false) => { if (selectionStopped) return undefined; + // per call: an OR-scope condition registers its restriction while the first row is filtered + const filterMap: any = relationshipRestriction(filtered, attribute.name); const readsKeys = !isSync && (filterMap ? !filterMap.hasMappings && resolver.toMany : Boolean(resolver.resolveKeys)); let value; @@ -7634,6 +7641,9 @@ export function makeTable(options): TableResourceClass { }, }; }; + const readOptions = isSync ? { transaction: readTxn } : { transaction: readTxn, async: true }; + const checkActive = getReadTransactionGuard(readTxn); + const matchCandidate = predicate && (isSync ? predicate : (predicate.maybeAsync ?? predicate)); const readCandidate = (candidate) => when(candidate, (candidate) => { if (stopped) return SKIP; @@ -7641,10 +7651,17 @@ export function makeTable(options): TableResourceClass { const entry = typeof candidate === 'object' && candidate.key !== undefined ? candidate - : targetTable.primaryStore.getEntry(candidate, { transaction: readTxn }); - return when(entry, (entry) => - predicate && (!entry?.value || !predicate(entry.value, entry)) ? SKIP : entry - ); + : when(targetTable.primaryStore.getEntry(candidate, readOptions), freezeRelatedEntry); + if (!predicate) return entry; + return when(entry, (entry) => { + if (stopped || !entry?.value) return SKIP; + checkActive?.(); + return when(matchCandidate(entry.value, entry), (matches) => { + if (!matches || stopped) return SKIP; + checkActive?.(); + return entry; + }); + }); }); let examined = 0; candidates = candidates.map((candidate) => { @@ -8015,7 +8032,7 @@ export function makeTable(options): TableResourceClass { for (let i = 0; i < clauseCount; i++) { const clause = clauses[i]; const keys = orderedKeys[i]; - const value = getAttributeValue(entry, clause.attribute, context, clause); + const value = getAttributeValue(entry, clause.attribute, context, clause, stopped); if (value?.then) { const position = keys.push(undefined) - 1; (pending ??= []).push( @@ -8099,7 +8116,8 @@ export function makeTable(options): TableResourceClass { entry, dbOrderedAttribute, context, - (sort as any).dbOrderedSort + (sort as any).dbOrderedSort, + stopped ); if (groupingValue?.then) { groupingValue = await groupingValue; @@ -8335,11 +8353,21 @@ export function makeTable(options): TableResourceClass { } } if (record == null) return canSkip ? SKIP : record; - // Recheck the explicit predicate here because `record` is now the final, materialized - // record — a caching table's source revalidation (above) may have replaced the local copy the - // query filters evaluated. The predicate sees a frozen view so application code can't - // mutate a record the source-revalidation path's deferred commit still needs to write and encode. - if (rowFilter && !rowFilter(frozenRecordView(record))) return canSkip ? SKIP : undefined; + // Source revalidation (above) may have replaced the record the query filters saw. The view is frozen + // because that revalidation's deferred commit still writes and encodes this record. + if (rowFilter) { + const matches = (!this?.isSync && rowFilter.maybeAsync ? rowFilter.maybeAsync : rowFilter)( + frozenRecordView(record) + ); + if (!matches) return canSkip ? SKIP : undefined; + if (typeof matches.then === 'function') + return matches.then((matches) => + matches && !selectionStopped ? selectRecord.call(this, record, entry) : canSkip ? SKIP : undefined + ); + } + return selectRecord.call(this, record, entry); + }; + const selectRecord = function (record, entry) { if (select && !(select[0] === '*' && select.length === 1 && !(select as any).asArray)) { let promises: Promise[]; const selectAttribute = (attribute, callback) => { @@ -8359,7 +8387,7 @@ export function makeTable(options): TableResourceClass { } else callback(selected, attribute_name); return; } - const filterMap = filtered?.[attribute_name]; + const filterMap: any = relationshipRestriction(filtered, attribute_name); if (filterMap) { if (filterMap.hasMappings) { const key = resolver.from ? record[resolver.from] : flattenKey(entry.key); @@ -11572,24 +11600,30 @@ export function makeTable(options): TableResourceClass { if (TableResource.loadAsInstance === false) freezeRecord(entry?.value); return entry; } - function getAttributeValue(entry, attribute_name, context, sort?) { + function getAttributeValue(entry, attribute_name, context, sort?, stopped?: () => boolean) { if (!entry) { return; } const record = (entry.deref ? entry.deref() : entry.value) ?? primaryStore.getEntry(entry.key)?.value; if (typeof attribute_name === 'object') { // attribute_name is an array of attributes, pointing to nested attribute - return getPathValue(record, attribute_name, 0, propertyResolvers, entry, context, sort); + return getPathValue(record, attribute_name, 0, propertyResolvers, entry, context, sort, stopped); } const resolver = propertyResolvers[attribute_name]; if (resolver?.loadEntries) - return getPathValue(record, (resolver.path ??= [attribute_name]), 0, propertyResolvers, entry, context, sort); + return getPathValue( + record, + (resolver.path ??= [attribute_name]), + 0, + propertyResolvers, + entry, + context, + sort, + stopped + ); return resolver ? resolver(record, context, entry, false, sort, true) : record?.[attribute_name]; } - function relatedValue(resolver, loaded) { - return resolver.toMany ? loaded?.map((entry) => entry?.value) : loaded?.value; - } - function getPathValue(value, path, start, resolvers, entry, context, sort) { + function getPathValue(value, path, start, resolvers, entry, context, sort, stopped?: () => boolean) { for (let i = start, l = path.length; i < l; i++) { const attribute = path[i]; const resolver = resolvers?.[attribute]; @@ -11599,10 +11633,13 @@ export function makeTable(options): TableResourceClass { if (loaded?.then) { const nextResolvers = resolvers; return loaded.then((loaded) => - getPathValue(relatedValue(resolver, loaded), path, i + 1, nextResolvers, null, context, sort) + stopped?.() + ? undefined + : relatedPathValue(resolver, loaded, path, i + 1, nextResolvers, context, sort, stopped) ); } - value = relatedValue(resolver, loaded); + if (resolver.toMany) return relatedPathValue(resolver, loaded, path, i + 1, resolvers, context, sort, stopped); + value = loaded?.value; } else { value = resolver && value @@ -11613,6 +11650,50 @@ export function makeTable(options): TableResourceClass { } return value; } + function relatedPathValue(resolver, loaded, path, start, resolvers, context, sort, stopped) { + if (!resolver.toMany) return getPathValue(loaded?.value, path, start, resolvers, null, context, sort, stopped); + if (!loaded) return undefined; + const values = []; + for (let i = 0; i < loaded.length; i++) { + const value = getPathValue(loaded[i]?.value, path, start, resolvers, null, context, sort, stopped); + values.push(value); + if (value?.then) + return remainingRelatedPathValues(loaded, i + 1, values, path, start, resolvers, context, sort, stopped); + } + return values; + } + // one RELATED_READS_IN_FLIGHT budget per path: serial while a deeper to-many hop (with its own window) remains + async function remainingRelatedPathValues(loaded, start, values, path, pathStart, resolvers, context, sort, stopped) { + const checkActive = getReadTransactionGuard(txnForContext(context).getReadTxn()); + const window = reachesToMany(path, pathStart, resolvers) ? 1 : RELATED_READS_IN_FLIGHT; + try { + for (let i = start; i < loaded.length; i++) { + if ((i - start + 1) % window === 0) { + await Promise.all(values.slice(i - window)); + checkActive?.(); + if (stopped?.()) { + Promise.all(values).catch(() => {}); + return; + } + } + values.push(getPathValue(loaded[i]?.value, path, pathStart, resolvers, null, context, sort, stopped)); + } + const resolved = await Promise.all(values); + checkActive?.(); + return resolved; + } catch (error) { + Promise.all(values).catch(() => {}); + throw error; + } + } + function reachesToMany(path, start, resolvers) { + for (let i = start; i < path.length; i++) { + const resolver = resolvers?.[path[i]]; + if (resolver?.toMany) return true; + resolvers = resolver?.definition?.tableClass?.propertyResolvers; + } + return false; + } function transformToEntries(ids, select, context, readTxn, filters?) { // TODO: Test and ensure that we break out of these loops when a connection is lost const filtersLength = filters?.length; diff --git a/resources/search.ts b/resources/search.ts index 2038235979..12901bd73f 100644 --- a/resources/search.ts +++ b/resources/search.ts @@ -75,7 +75,8 @@ export function executeConditions( context, transformToEntries, filtered, - recordAccess? + recordAccess?, + inUnion?: boolean ) { conditions = combineFullTextConditions(conditions, operator, table); const firstSearch = conditions[0]; @@ -114,7 +115,7 @@ export function executeConditions( // during the graph walk instead of post-filtering an under-filled candidate set (#1241). They // also stay in the post-filter chain as a deterministic, harmless re-check. const siblings = conditions.slice(1); - const filters = mapConditionsToFilters(siblings, true, firstSearch.estimated_count); + const filters = mapConditionsToFilters(siblings, true, firstSearch.estimated_count, inUnion); const recordFilters = recordGuards ? filters.concat(recordGuards) : filters; // The sibling conditions a secondary-index range scan answers exactly can seed an allow-set // the index admits from, instead of loading and decoding a record at every visited node. @@ -136,7 +137,7 @@ export function executeConditions( // General path: execute the lead FIRST — a leading join populates `filtered`, which the sibling // filters read as they are built — then apply the sibling filters plus any record guards. const results = executeCondition(firstSearch); - const filters = mapConditionsToFilters(conditions.slice(1), true, firstSearch.estimated_count); + const filters = mapConditionsToFilters(conditions.slice(1), true, firstSearch.estimated_count, inUnion); const recordFilters = recordGuards ? filters.concat(recordGuards) : filters; return recordFilters.length > 0 ? transformToEntries(results, recordFilters) : results; } @@ -150,8 +151,9 @@ export function executeConditions( request, context, transformToEntries, - filtered - // recordAccess intentionally omitted: guards run once, at the top level (see above). + filtered, + undefined, // recordAccess intentionally omitted: guards run once, at the top level (see above). + inUnion || operator === 'or' ); return searchByIndex(condition, txn, condition.descending || request.reverse === true, table, { allowFullScan: request.allowFullScan, @@ -162,13 +164,18 @@ export function executeConditions( resultOffset: request.offset || 0, }); } - function mapConditionsToFilters(conditions, intersection, estimatedIncomingCount) { + function mapConditionsToFilters(conditions, intersection, estimatedIncomingCount, inUnion: boolean) { return conditions .map((condition, index) => { if (condition.conditions) { // this is a group of conditions, we need to combine them const union = condition.operator === 'or'; - const filters = mapConditionsToFilters(condition.conditions, !union, estimatedIncomingCount); + const filters = mapConditionsToFilters( + condition.conditions, + !union, + estimatedIncomingCount, + inUnion || union + ); const groupFilter: any = union ? (record, entry) => filters.some((filter) => filter(record, entry)) : (record, entry) => filters.every((filter) => filter(record, entry)); @@ -177,7 +184,7 @@ export function executeConditions( return groupFilter; } const isPrimaryKey = (condition.attribute || condition[0]) === table.primaryKey; - const filter = filterByType(condition, table, context, filtered, isPrimaryKey, estimatedIncomingCount); + const filter = filterByType(condition, table, context, filtered, isPrimaryKey, estimatedIncomingCount, inUnion); if (intersection && index < conditions.length - 1 && estimatedIncomingCount) { estimatedIncomingCount = intersectionEstimate( table.primaryStore, @@ -1517,6 +1524,22 @@ async function scanRelated(rightIterable, readRecord, joined, add, scan: JoinSca } catch {} } } +class RelationshipSelection { + filters: any[]; + fromRecord: (record: any, sync?: boolean) => any; + constructor(filters: any[], fromRecord: (record: any, sync?: boolean) => any) { + this.filters = filters; + this.fromRecord = fromRecord; + } +} +// keys the join and selection maps keep for themselves, so a relationship of that name cannot be tracked in them +const SELECTION_BOOKKEEPING = new Set(['filters', 'fromRecord', 'hasMappings']); +// A query's `filtered` map holds, per relationship name, a leading join's Map or a selection with `fromRecord`. +// Read it as an own property of that shape: a name can equal an inherited member or the maps' bookkeeping keys. +export function relationshipRestriction(filtered, name: string) { + const restriction = filtered && Object.hasOwn(filtered, name) ? filtered[name] : undefined; + return restriction instanceof Map || typeof restriction?.fromRecord === 'function' ? restriction : undefined; +} function someRelatedMatch(entries, match, start: number) { for (let i = start; i < entries.length; i++) { const value = entries[i]?.value; @@ -1527,7 +1550,7 @@ function someRelatedMatch(entries, match, start: number) { } return false; } -function matchesFilters(filters, record, entry, any: boolean, start = 0) { +export function matchesFilters(filters, record, entry, any: boolean, start = 0) { for (let i = start; i < filters.length; i++) { const filter = filters[i]; const matches = (filter.maybeAsync ?? filter)(record, entry); @@ -1746,7 +1769,15 @@ export function getNestedValue(record: any, path: string | string[]): any { * @param {SearchObject} searchCondition * @returns {({}) => boolean} */ -export function filterByType(searchCondition, Table, context, filtered, isPrimaryKey?, estimatedIncomingCount?) { +export function filterByType( + searchCondition, + Table, + context, + filtered, + isPrimaryKey?, + estimatedIncomingCount?, + inUnion?: boolean +) { const comparator = searchCondition.comparator; const negated = searchCondition.negated; let attribute = searchCondition[0] ?? searchCondition.attribute; @@ -1781,31 +1812,12 @@ export function filterByType(searchCondition, Table, context, filtered, isPrimar } // TODO: If this is a relationship, we can potentially make this more efficient by using the index // and retrieving the set of matching ids first - const filterMap = filtered?.[firstAttributeName]; - const nextFilter = filterByType( - { - attribute: attribute.length > 2 ? attribute.slice(1) : attribute[1], - value, - comparator, - negated, - }, - relatedTable, - context, - filterMap?.[firstAttributeName]?.joined, - attribute[1] === relatedTable.primaryKey, - estimatedIncomingCount - ); - if (!nextFilter) return; - if (filterMap) { - if (!filterMap.filters) filterMap.filters = []; - filterMap.filters.push(nextFilter); - return; - } + const filterMap = relationshipRestriction(filtered, firstAttributeName); + const join: any = filterMap instanceof Map ? filterMap : undefined; const resolver = Table.propertyResolvers?.[firstAttributeName]; - if (resolver?.to) nextFilter.to = resolver.to; - let subIdFilter; + const loadEntries = resolver?.loadEntries; const getSubObject = (record, entry) => { - let subObject, subEntry; + let subObject, subEntry, subEntries; if (resolver) { if (resolver.returnDirect) { // indicates that the resolver will direct return the value instead of an entry @@ -1814,72 +1826,125 @@ export function filterByType(searchCondition, Table, context, filtered, isPrimar } else { subEntry = resolver(record, context, entry, true); if (Array.isArray(subEntry)) { - // if any array, map the values - subObject = subEntry.map((subEntry) => subEntry.value); + subEntries = subEntry.filter((related) => related?.value != null); + subObject = subEntries.map((related) => related.value); subEntry = null; } else { subObject = subEntry?.value; } } } else subObject = record[firstAttributeName]; - return { subObject, subEntry }; + return { subObject, subEntry, subEntries }; + }; + const relatedSelection = (filters: any[]) => + new RelationshipSelection(filters, (record, sync?: boolean) => { + // this is called when selecting the fields to include in results + if (loadEntries && !sync) { + return when(loadEntries(record, context), (loaded) => { + if (!resolver.toMany || !loaded) return loaded?.value; + const matches = mapSettlingOnThrow( + loaded, + (entry) => entry?.value != null && matchesFilters(filters, entry.value, entry, false) + ); + const matchingKeys = (matches) => + loaded + .filter((_entry, i) => matches[i]) + .map((entry) => entry.key ?? entry.value[relatedTable.primaryKey]); + return matches.some((matches) => matches?.then) + ? Promise.all(matches).then(matchingKeys) + : matchingKeys(matches); + }); + } + const { subObject, subEntries } = getSubObject(record, undefined); + if (!Array.isArray(subObject)) return subObject; + const keys = []; + for (let i = 0; i < subObject.length; i++) { + const value = subObject[i]; + const entry = subEntries?.[i]; + if (filters.every((filter) => filter(value, entry))) + keys.push(entry?.key ?? value[relatedTable.primaryKey]); + } + return keys; + }); + // AND scope: registered when built, so no row sees a partial set; OR scope: lazily, first branch wins + const restrictsSelect = filtered && firstAttribute.elements && !SELECTION_BOOKKEEPING.has(firstAttributeName); + let selection; + if (!join && restrictsSelect && !inUnion) { + if (filterMap instanceof RelationshipSelection) selection = filterMap; + else if (!Object.hasOwn(filtered, firstAttributeName)) selection = relatedSelection([]); + } + const nextFilter = filterByType( + { + attribute: attribute.length > 2 ? attribute.slice(1) : attribute[1], + value, + comparator, + negated, + }, + relatedTable, + context, + join ?? selection, + attribute[1] === relatedTable.primaryKey, + estimatedIncomingCount, + inUnion + ); + if (!nextFilter) return; + if (join) { + if (!join.filters) join.filters = []; + join.filters.push(nextFilter); + return; + } + if (selection) { + filtered[firstAttributeName] = selection; + selection.filters.push(nextFilter); + } + if (resolver?.to) nextFilter.to = resolver.to; + let subIdFilter; + // the shortcut tests this record's `from` (or key): a `to` relationship's id filter must be mapped through `to` + const usesIdFilter = () => nextFilter.idFilter && (!resolver.to || nextFilter.idFilter.mappedTo); + const preserveForSelect = () => { + if (inUnion && restrictsSelect && !Object.hasOwn(filtered, firstAttributeName)) + filtered[firstAttributeName] = relatedSelection([nextFilter]); }; const recordFilter = (record, entry) => { - if (resolver) { - if (nextFilter.idFilter) { - // if we are filtering by id, we can use the idFilter to avoid loading the record - if (!subIdFilter) { - if (nextFilter.idFilter.idSet?.size === 1) { - // if there is a single id we are looking for, we can create a new search condition that the - // attribute comparator could eventually use to create a recursive id set - // TODO: Eventually we should be able to handle multiple ids by creating a union - searchCondition = { - attribute: resolver.from ?? Table.primaryKey, // if no from, we use our primary key - value: nextFilter.idFilter.onlyId, - }; - // indicate that we can use an index for this. also we indicate that we allow object matching to allow array ids to directly tested - subIdFilter = attributeComparator(resolver.from ?? Table.primaryKey, nextFilter.idFilter, true, true); - } else - subIdFilter = attributeComparator(resolver.from ?? Table.primaryKey, nextFilter.idFilter, false, true); - } - const matches = subIdFilter(record); - if ((subIdFilter as any).idFilter) (recordFilter as any).idFilter = (subIdFilter as any).idFilter; - return matches; + if (resolver && usesIdFilter()) { + // if we are filtering by id, we can use the idFilter to avoid loading the record + if (!subIdFilter) { + // an array of foreign keys is tested per element; a single key (which may be a composite array) whole + const allowObjectMatching = Boolean(resolver.to) || !resolver.toMany; + if (nextFilter.idFilter.idSet?.size === 1) { + // if there is a single id we are looking for, we can create a new search condition that the + // attribute comparator could eventually use to create a recursive id set + // TODO: Eventually we should be able to handle multiple ids by creating a union + searchCondition = { + attribute: resolver.from ?? Table.primaryKey, // if no from, we use our primary key + value: nextFilter.idFilter.onlyId, + }; + subIdFilter = attributeComparator( + resolver.from ?? Table.primaryKey, + nextFilter.idFilter, + true, + allowObjectMatching + ); + } else + subIdFilter = attributeComparator( + resolver.from ?? Table.primaryKey, + nextFilter.idFilter, + false, + allowObjectMatching + ); } + const matches = subIdFilter(record); + if ((subIdFilter as any).idFilter) (recordFilter as any).idFilter = (subIdFilter as any).idFilter; + if (matches) preserveForSelect(); + return matches; } - const { subObject, subEntry } = getSubObject(record, entry); + const { subObject, subEntry, subEntries } = getSubObject(record, entry); if (!subObject) return false; if (!Array.isArray(subObject)) return nextFilter(subObject, subEntry); preserveForSelect(); - return subObject.some(nextFilter); + return subObject.some((value, i) => nextFilter(value, subEntries?.[i])); }; - const loadEntries = resolver?.loadEntries; const nextMatch = nextFilter.maybeAsync ?? nextFilter; - const preserveForSelect = () => { - if (filtered && !filtered[firstAttributeName]) { - // establish a filtering that can preserve this filter for the select() results of these sub objects - filtered[firstAttributeName] = { - fromRecord(record, sync?: boolean) { - // this is called when selecting the fields to include in results - if (loadEntries && !sync) { - return when(loadEntries(record, context), (loaded) => { - if (!resolver.toMany || !loaded) return loaded?.value; - const values = loaded.map((entry) => entry?.value); - const matches = mapSettlingOnThrow(values, (value) => value != null && nextMatch(value)); - const matchingIds = (matches) => - values.filter((_value, i) => matches[i]).map((value) => value[relatedTable.primaryKey]); - return matches.some((matches) => matches?.then) - ? Promise.all(matches).then(matchingIds) - : matchingIds(matches); - }); - } - const value = getSubObject(record, undefined).subObject; - if (Array.isArray(value)) return value.filter(nextFilter).map((value) => value[relatedTable.primaryKey]); - return value; - }, - }; - } - }; if (loadEntries) { const matchLoaded = (loaded) => { if (!resolver.toMany) return loaded?.value ? nextMatch(loaded.value, loaded) : false; @@ -1888,7 +1953,7 @@ export function filterByType(searchCondition, Table, context, filtered, isPrimar return someRelatedMatch(loaded, nextMatch, 0); }; (recordFilter as any).maybeAsync = (record, entry) => { - if (nextFilter.idFilter) return recordFilter(record, entry); + if (usesIdFilter()) return recordFilter(record, entry); const loaded = loadEntries(record, context); return loaded?.then ? loaded.then(matchLoaded) : matchLoaded(loaded); }; @@ -2077,6 +2142,7 @@ export function filterByType(searchCondition, Table, context, filtered, isPrimar (recordFilter.idFilter as any).idSet = idSet; // the set holds encoded identities; the singleton relationship lookup needs the key itself (recordFilter.idFilter as any).onlyId = idSet.size === 1 ? firstId : undefined; + (recordFilter.idFilter as any).mappedTo = Boolean((recordFilter as any).to); } } return matches; diff --git a/unitTests/resources/relationshipPrimaryKeyFilters.test.js b/unitTests/resources/relationshipPrimaryKeyFilters.test.js new file mode 100644 index 0000000000..e76365734f --- /dev/null +++ b/unitTests/resources/relationshipPrimaryKeyFilters.test.js @@ -0,0 +1,347 @@ +const assert = require('node:assert'); +const { createServer } = require('node:http'); +const { setupTestDBPath } = require('../testUtils.js'); +const { table } = require('#src/resources/databases'); +const { Resources } = require('#src/resources/Resources'); +const { handleApplication } = require('#src/server/REST'); +const { Request } = require('#src/server/serverHelpers/Request'); +const { getWorkerIndex, setMainIsWorker } = require('#src/server/threads/manageThreads'); +require('#src/server/serverHelpers/serverUtilities'); + +const children = [ + { id: 'c1', parentId: 'p1', state: 'open', score: 10 }, + { id: 'c2', parentId: 'p1', state: 'closed', score: 20 }, + { id: 'c3', parentId: 'p1', state: 'open', score: 30 }, + { id: 'c4', parentId: 'p1', state: 'open', score: 40 }, + { id: 'c5', parentId: 'p2', state: 'closed', score: 5 }, + { id: 'c6', parentId: 'p2', state: 'open', score: 50 }, + { id: 'c7', parentId: 'p2', state: 'closed', score: 60 }, +]; + +const parents = [ + { + id: 'p1', + group: 'g1', + childId: 'c1', + childIds: ['c1', 'missing', 'c2', 'c3', 'c4'], + }, + { id: 'p2', group: 'g1', childId: 'missing', childIds: ['c5', 'c6'] }, + { id: 'p3', group: 'g1', childId: 'c5', childIds: [] }, +]; + +describe('relationship primary-key filters and dangling foreign keys', function () { + this.timeout(15000); + let Parent, Child, server, base, previousMainIsWorker, resources; + + before(async function () { + setupTestDBPath(); + previousMainIsWorker = getWorkerIndex() === 0; + setMainIsWorker(true); + + Child = table({ + database: 'relationship_pk_filters', + table: 'Child', + attributes: [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'parentId', type: 'String', indexed: true }, + { name: 'state', type: 'String', indexed: true }, + { name: 'score', type: 'Int', indexed: true }, + { + name: 'siblings', + relationship: { to: 'parentId', from: 'parentId' }, + elements: { type: 'Child', definition: {} }, + }, + ], + }); + Child.attributes.find((attribute) => attribute.name === 'siblings').elements.definition.tableClass = Child; + Child.updatedAttributes(); + + Parent = table({ + database: 'relationship_pk_filters', + table: 'Parent', + attributes: [ + { name: 'id', type: 'ID', isPrimaryKey: true }, + { name: 'group', type: 'String', indexed: true }, + { name: 'childId', type: 'ID' }, + { name: 'childIds', elements: { type: 'ID' } }, + { + name: 'children', + relationship: { to: 'parentId' }, + elements: { type: 'Child', definition: { tableClass: Child } }, + }, + { + name: 'child', + relationship: { from: 'childId' }, + type: 'Child', + definition: { tableClass: Child }, + }, + { + name: 'references', + relationship: { from: 'childIds' }, + elements: { type: 'Child', definition: { tableClass: Child } }, + }, + { + name: 'peers', + relationship: { to: 'group', from: 'group' }, + elements: { type: 'Parent', definition: {} }, + }, + ], + }); + Parent.attributes.find((attribute) => attribute.name === 'peers').elements.definition.tableClass = Parent; + Parent.updatedAttributes(); + + for (const parent of parents) await Parent.put(parent.id, parent); + for (const child of children) await Child.put(child.id, child); + + resources = new Resources(); + resources.set('rel-pk/Parent', Parent); + resources.set('rel-pk/Child', Child); + let rest; + handleApplication({ + resources, + options: { getAll: () => ({ webSocket: false }) }, + routeFor: () => ({ urlPath: '/rel-pk' }), + server: { + http: (handler) => { + rest = handler; + }, + ws: () => [], + }, + }); + server = createServer(async (input, output) => { + try { + const request = new Request(input, output); + request.user = { role: { permission: { super_user: true } } }; + const response = await rest(request, () => ({ + status: 404, + headers: new Map(), + body: 'not found', + })); + output.writeHead(response.status, Object.fromEntries(response.headers)); + if (typeof response.body === 'string' || Buffer.isBuffer(response.body)) output.end(response.body); + else { + for await (const chunk of response.body || []) output.write(chunk); + output.end(); + } + } catch (error) { + output.writeHead(error.statusCode || 500); + output.end(error.message); + } + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + base = `http://127.0.0.1:${server.address().port}`; + }); + + after(async function () { + if (server) await new Promise((resolve) => server.close(resolve)); + if (previousMainIsWorker !== undefined) setMainIsWorker(previousMainIsWorker); + }); + + async function get(path) { + const response = await fetch(base + path); + const text = await response.text(); + assert.strictEqual(response.status, 200, `Expected HTTP 200 for ${path}, got ${response.status}: ${text}`); + return JSON.parse(text); + } + + async function searchParents(queryObject) { + const rows = []; + for await (const row of Parent.search(queryObject)) { + rows.push(JSON.parse(JSON.stringify(row))); + } + return rows; + } + + it('controls: a leading related primary-key condition, a non-leading to-one FK condition, and a nested non-key condition', async function () { + const drivingSingleRest = await get('/rel-pk/Parent/?children.id=c1&select(id,)'); + assert.deepStrictEqual(drivingSingleRest, [{ id: 'p1' }]); + + const drivingGroupRest = await get('/rel-pk/Parent/?children.id=c1&group=g1&select(id,)'); + assert.deepStrictEqual(drivingGroupRest, [{ id: 'p1' }]); + + const drivingEnforcedSearch = await searchParents({ + conditions: [ + { attribute: ['children', 'id'], value: 'c1' }, + { attribute: 'id', value: 'p1' }, + ], + enforceExecutionOrder: true, + select: ['id'], + }); + assert.deepStrictEqual(drivingEnforcedSearch, [{ id: 'p1' }]); + + const fromFkSingularMatch = await get('/rel-pk/Parent/?id=p1&child.id=c1&select(id,)'); + assert.deepStrictEqual(fromFkSingularMatch, [{ id: 'p1' }]); + + const fromFkSingularMiss = await get('/rel-pk/Parent/?id=p1&child.id=c2&select(id,)'); + assert.deepStrictEqual(fromFkSingularMiss, []); + + const nestedNonPk = await get('/rel-pk/Parent/p2?select(id,children[siblings.state=open&select(id,)])'); + assert.deepStrictEqual(nestedNonPk, { + id: 'p2', + children: [{ id: 'c5' }, { id: 'c6' }, { id: 'c7' }], + }); + }); + + it('a non-leading related primary-key condition on a `to` relationship keeps matching parents in any condition order', async function () { + const drivingControl = await searchParents({ + conditions: [ + { attribute: ['children', 'id'], value: 'c1' }, + { attribute: 'id', value: 'p1' }, + ], + enforceExecutionOrder: true, + select: ['id'], + }); + assert.deepStrictEqual(drivingControl, [{ id: 'p1' }]); + + const filterOrderRest = await get('/rel-pk/Parent/?id=p1&children.id=c1&select(id,)'); + const reorderedRest = await get('/rel-pk/Parent/?children.id=c1&id=p1&select(id,)'); + const filterOrderSearch = await searchParents({ + conditions: [ + { attribute: 'id', value: 'p1' }, + { attribute: ['children', 'id'], value: 'c1' }, + ], + select: ['id'], + }); + const groupFilterSearch = await searchParents({ + conditions: [ + { attribute: 'group', value: 'g1' }, + { attribute: ['children', 'id'], value: 'c6' }, + ], + enforceExecutionOrder: true, + select: ['id'], + }); + + assert.deepStrictEqual( + { + filterOrderRest, + reorderedRest, + filterOrderSearch, + groupFilterSearch, + }, + { + filterOrderRest: [{ id: 'p1' }], + reorderedRest: [{ id: 'p1' }], + filterOrderSearch: [{ id: 'p1' }], + groupFilterSearch: [{ id: 'p2' }], + } + ); + }); + + it('a non-leading related primary-key condition on a `to` relationship does not match a parent by its own id', async function () { + assert.strictEqual(Child.primaryStore.getEntry('p1'), undefined); + assert.strictEqual(Child.primaryStore.getEntry('p3'), undefined); + + const drivingControl = await searchParents({ + conditions: [ + { attribute: ['children', 'id'], value: 'p1' }, + { attribute: 'id', value: 'p1' }, + ], + enforceExecutionOrder: true, + select: ['id'], + }); + assert.deepStrictEqual(drivingControl, []); + + const filterOrderRestP1 = await get('/rel-pk/Parent/?id=p1&children.id=p1&select(id,)'); + const filterOrderSearchP1 = await searchParents({ + conditions: [ + { attribute: 'id', value: 'p1' }, + { attribute: ['children', 'id'], value: 'p1' }, + ], + select: ['id'], + }); + const filterOrderRestP3 = await get('/rel-pk/Parent/?id=p3&children.id=p3&select(id,)'); + + assert.deepStrictEqual( + { + filterOrderRestP1, + filterOrderSearchP1, + filterOrderRestP3, + }, + { + filterOrderRestP1: [], + filterOrderSearchP1: [], + filterOrderRestP3: [], + } + ); + }); + + it('a nested selection condition on a related primary key filters by that key, not by the join attribute', async function () { + const nestedRestMatch = await get('/rel-pk/Parent/p2?select(id,children[siblings.id=c6&select(id,)])'); + const nestedSearchMatch = await searchParents({ + conditions: [{ attribute: 'id', value: 'p2' }], + select: [ + 'id', + { + name: 'children', + conditions: [{ attribute: ['siblings', 'id'], value: 'c6' }], + select: ['id'], + }, + ], + }); + const nestedRestFalseMatch = await get('/rel-pk/Parent/p1?select(id,children[siblings.id=p1&select(id,)])'); + + assert.deepStrictEqual( + { + nestedRestMatch, + nestedSearchMatch, + nestedRestFalseMatch, + }, + { + nestedRestMatch: { + id: 'p2', + children: [{ id: 'c5' }, { id: 'c6' }, { id: 'c7' }], + }, + nestedSearchMatch: [{ id: 'p2', children: [{ id: 'c5' }, { id: 'c6' }, { id: 'c7' }] }], + nestedRestFalseMatch: { id: 'p1', children: [] }, + } + ); + }); + + it('an array foreign key with a dangling id filters by element, in conditions and nested selections', async function () { + const nestedPeersResp = await fetch( + `${base}/rel-pk/Parent/p2?select(id,peers[references.state=closed&select(id,)])` + ); + const nestedPeersBody = JSON.parse(await nestedPeersResp.text()); + + let syncSearchOutcome; + try { + syncSearchOutcome = Array.from( + Parent.search({ + conditions: [ + { attribute: 'id', value: 'p1' }, + { attribute: ['references', 'state'], value: 'closed' }, + ], + select: ['id'], + }) + ).map((row) => ({ id: row.id })); + } catch (err) { + syncSearchOutcome = `${err.name}: ${err.message}`; + } + + const fromFkArrayMatchRest = await get('/rel-pk/Parent/?id=p1&references.id=c2&select(id,)'); + const fromFkArrayMissRest = await get('/rel-pk/Parent/?id=p1&references.id=c6&select(id,)'); + const nestedFromFkArrayPkRest = await get('/rel-pk/Parent/p2?select(id,peers[references.id=c2&select(id,)])'); + + assert.deepStrictEqual( + { + nestedPeersStatus: nestedPeersResp.status, + nestedPeersBody, + syncSearchOutcome, + fromFkArrayMatchRest, + fromFkArrayMissRest, + nestedFromFkArrayPkRest, + }, + { + nestedPeersStatus: 200, + nestedPeersBody: { + id: 'p2', + peers: [{ id: 'p1' }, { id: 'p2' }], + }, + syncSearchOutcome: [{ id: 'p1' }], + fromFkArrayMatchRest: [{ id: 'p1' }], + fromFkArrayMissRest: [], + nestedFromFkArrayPkRest: { id: 'p2', peers: [{ id: 'p1' }] }, + } + ); + }); +}); diff --git a/unitTests/resources/relationshipQueryContracts.test.js b/unitTests/resources/relationshipQueryContracts.test.js new file mode 100644 index 0000000000..82bbe9b1ba --- /dev/null +++ b/unitTests/resources/relationshipQueryContracts.test.js @@ -0,0 +1,636 @@ +const assert = require('node:assert'); +const { setImmediate: nextTick } = require('node:timers/promises'); +const { setupTestDBPath } = require('../testUtils'); +const { databases } = require('#src/resources/databases'); +const { loadGQLSchema } = require('#src/resources/graphql'); +const { setMainIsWorker } = require('#src/server/threads/manageThreads'); + +const isLMDB = process.env.HARPER_STORAGE_ENGINE === 'lmdb'; + +function clearValueMap(Table) { + Table.primaryStore.flushSync?.(); + Table.primaryStore.cache?.clear?.(); + Table.primaryStore.valueMap?.clear?.(); +} + +// counts the outermost primary-store reads; an async read is delayed a turn so it behaves like a cold read +function spyPrimaryReads(Table) { + const store = Table.primaryStore; + const originals = new Map(); + let depth = 0; + const stats = { + async: 0, + sync: 0, + inFlight: 0, + maxInFlight: 0, + restore() { + for (const [method, original] of originals) store[method] = original; + }, + }; + const trackPromise = (promise) => { + stats.async++; + stats.inFlight++; + if (stats.inFlight > stats.maxInFlight) stats.maxInFlight = stats.inFlight; + const done = () => { + stats.inFlight--; + }; + promise.then(done, done); + return promise; + }; + const wrap = (method, isAsyncCall) => { + const original = store[method]; + originals.set(method, original); + store[method] = function (id, options) { + const topLevel = depth === 0; + depth++; + try { + const result = original.call(this, id, options); + if (!topLevel) return result; + if (!isAsyncCall(options)) { + stats.sync++; + return result; + } + if (typeof result?.then === 'function') return trackPromise(result); + return trackPromise(nextTick().then(() => result)); + } finally { + depth--; + } + }; + }; + wrap('getEntry', (options) => Boolean(options?.async)); + wrap('get', () => true); + wrap('getSync', () => false); + return stats; +} + +async function collect(iterable) { + const rows = []; + for await (const row of iterable) rows.push(row); + return rows; +} + +describe('Relationship query contracts: async reads, select restrictions and sorting', function () { + let Parent, ColdChild, ColdGrandchild, CompositeChild, Company, Department, Employee; + + before(async function () { + setupTestDBPath(); + setMainIsWorker(true); + await loadGQLSchema(` + type ColdGrandchild @table(database: "relationshipQueryContracts") { + id: ID @primaryKey + code: String @indexed + ownerId: ID @indexed + } + type ColdChild @table(database: "relationshipQueryContracts") { + id: ID @primaryKey + parentId: ID @indexed + name: String @indexed + label: String @indexed + tier: String + score: Int + sortKey: String @indexed + grandchildId: ID @indexed + grandchild: ColdGrandchild @relationship(from: grandchildId) + # named like a key the selection maps keep for themselves + filters: [ColdGrandchild] @relationship(to: ownerId) + } + # no declared primary key: records are stored under a composite key that no record field holds + type CompositeChild @table(database: "relationshipQueryContracts") { + tenantId: String + itemId: String + parentId: ID @indexed + label: String + } + type Parent @table(database: "relationshipQueryContracts") { + id: ID @primaryKey + group: String @indexed + primaryChildId: ID @indexed + primaryChild: ColdChild @relationship(from: primaryChildId) + childIds: [ID] @indexed + linkedChildren: [ColdChild] @relationship(from: childIds) + children: [ColdChild] @relationship(to: parentId) + compositeChildren: [CompositeChild] @relationship(to: parentId) + } + type Company @table(database: "relationshipQueryContracts") { + id: ID @primaryKey + group: String @indexed + departments: [Department] @relationship(to: companyId) + } + type Department @table(database: "relationshipQueryContracts") { + id: ID @primaryKey + companyId: ID @indexed + name: String @indexed + employees: [Employee] @relationship(to: departmentId) + } + type Employee @table(database: "relationshipQueryContracts") { + id: ID @primaryKey + departmentId: ID @indexed + role: String @indexed + active: Boolean @indexed + tag: String + } + `); + const db = databases.relationshipQueryContracts; + for (const Table of Object.values(db)) Table.loadAsInstance = false; + ({ Parent, ColdChild, ColdGrandchild, CompositeChild, Company, Department, Employee } = db); + + await ColdGrandchild.put({ id: 'gc-1', code: 'gc-alpha' }); + await ColdGrandchild.put({ id: 'gc-2', code: 'other-beta' }); + await ColdGrandchild.put({ id: 'gc-flag', code: 'flag-on', ownerId: 'c1' }); + const coldChildren = [ + ['c1', 'parent-1', 'child-1a', 'match-alpha', 'silver', 10, 'z-ignore', 'gc-2'], + ['c2', 'parent-1', 'child-1b', 'match-beta', 'gold', 90, 'a-ignore', 'gc-1'], + ['c3', 'parent-1', 'child-1c', 'other-gamma', 'gold', 80, 'm-ignore', 'gc-2'], + ['c4', 'parent-2', 'child-2a', 'other-delta', 'bronze', 5, 'b-ignore', 'gc-2'], + // different children of parent-3 satisfy different sibling conditions + ['c5', 'parent-3', 'child-3a', 'match-epsilon', 'silver', 1, 'y-ignore', 'gc-2'], + ['c6', 'parent-3', 'child-3b', 'other-zeta', 'gold', 1, 'x-ignore', 'gc-2'], + // OR group: or-a's only child matches the second branch; of or-b's children, c8 matches the first and c9 neither + ['c7', 'or-a', 'child-or-a', 'B', 'gold', 1, 'w-ignore', 'gc-2'], + ['c8', 'or-b', 'child-or-b', 'A', 'silver', 1, 'v-ignore', 'gc-2'], + ['c9', 'or-b', 'child-or-b2', 'Z', 'silver', 1, 'u-ignore', 'gc-2'], + ]; + for (const [id, parentId, name, label, tier, score, sortKey, grandchildId] of coldChildren) + await ColdChild.put({ id, parentId, name, label, tier, score, sortKey, grandchildId }); + await Parent.put({ id: 'parent-1', group: 'g1', primaryChildId: 'c2', childIds: ['c1', 'c2', 'c3'] }); + await Parent.put({ id: 'parent-2', group: 'g1', primaryChildId: 'c1', childIds: ['c4'] }); + await Parent.put({ id: 'parent-3', group: 'g3', childIds: ['c5', 'c6'] }); + await Parent.put({ id: 'or-a', group: 'or-grp', childIds: [] }); + await Parent.put({ id: 'or-b', group: 'or-grp', childIds: [] }); + + await CompositeChild.put(['t1', 'i1'], { tenantId: 't1', itemId: 'i1', parentId: 'parent-1', label: 'keep-me' }); + await CompositeChild.put(['t1', 'i2'], { tenantId: 't1', itemId: 'i2', parentId: 'parent-1', label: 'drop-me' }); + + await Company.put({ id: 'co-split', group: 'g1' }); + await Company.put({ id: 'co-match', group: 'g1' }); + await Department.put({ id: 'dept-split', companyId: 'co-split', name: 'Split Dept' }); + // two to-many hops wide enough that resolving them together would exceed one read window + await Company.put({ id: 'co-wide', group: 'wide-co' }); + for (let d = 0; d < 3; d++) { + await Department.put({ id: `dw-${d}`, companyId: 'co-wide', name: `Wide ${d}` }); + for (let e = 0; e < 12; e++) + await Employee.put({ + id: `ew-${d}-${e}`, + departmentId: `dw-${d}`, + role: 'wide', + active: true, + tag: `w-${d}-${e}`, + }); + } + await Department.put({ id: 'dept-match', companyId: 'co-match', name: 'Match Dept' }); + // co-split has an engineer and an active employee, but no active engineer + await Employee.put({ id: 'emp-split-1', departmentId: 'dept-split', role: 'eng', active: false, tag: 'match-tag' }); + await Employee.put({ + id: 'emp-split-2', + departmentId: 'dept-split', + role: 'sales', + active: true, + tag: 'other-tag', + }); + await Employee.put({ id: 'emp-match-1', departmentId: 'dept-match', role: 'eng', active: true, tag: 'match-tag' }); + await Employee.put({ id: 'emp-match-2', departmentId: 'dept-match', role: 'eng', active: false, tag: 'other-tag' }); + + const sortChildren = [ + ['cs-1', 'p-sort-1', 'charlie'], + ['cs-2', 'p-sort-2', 'alpha'], + ['cs-3', 'p-sort-3', 'bravo'], + ['cs-4a', 'p-sort-4', 'alpha'], + ['cs-4b', 'p-sort-4', 'delta'], + ]; + for (const [id, parentId, sortKey] of sortChildren) + await ColdChild.put({ id, parentId, name: id, label: 'sort-label', tier: 'gold', sortKey, grandchildId: 'gc-2' }); + for (const [id, childIds] of [ + ['p-sort-1', ['cs-1']], + ['p-sort-2', ['cs-2']], + ['p-sort-3', ['cs-3']], + ['p-sort-4', ['cs-4a', 'cs-4b']], + ]) + await Parent.put({ id, group: 'sort-grp', primaryChildId: childIds[0], childIds }); + // more children than are resolved at once (RELATED_READS_IN_FLIGHT), each with its own grandchild + for (let i = 0; i < 40; i++) { + await ColdGrandchild.put({ id: `gc-wide-${i}`, code: `wide-${String(i).padStart(2, '0')}` }); + await ColdChild.put({ id: `cw-${i}`, parentId: 'p-wide', name: `cw-${i}`, grandchildId: `gc-wide-${i}` }); + } + await Parent.put({ id: 'p-wide', group: 'wide-grp', childIds: [] }); + // a grandchild condition led by this four-row group switches to index-derived id sets two hops down + for (let i = 0; i < 4; i++) { + await Parent.put({ id: `idset-${i}`, group: 'idset-grp', childIds: [] }); + for (const suffix of ['a', 'b']) + await ColdChild.put({ + id: `ci-${i}-${suffix}`, + parentId: `idset-${i}`, + name: `ci-${i}-${suffix}`, + grandchildId: (i === 1 && suffix === 'b') || (i === 3 && suffix === 'a') ? 'gc-1' : 'gc-2', + }); + } + // enough rows that a two-row group condition leads without reordering + for (let i = 0; i < 22; i++) await Parent.put({ id: `p-filler-${i}`, group: 'filler-grp', childIds: [] }); + }); + + describe('async reads on cold tables', function () { + it('a non-leading to-many condition reads asynchronously, filters the selected children and freezes them', async function () { + clearValueMap(ColdChild); + const reads = spyPrimaryReads(ColdChild); + try { + const rows = await collect( + Parent.search({ + conditions: [ + { attribute: 'group', value: 'g1' }, + { attribute: ['children', 'label'], comparator: 'starts_with', value: 'match' }, + ], + select: ['id', 'children'], + }) + ); + assert.strictEqual(reads.sync, 0); + assert.ok(reads.async > 0); + assert.deepStrictEqual( + rows.map((row) => row.id), + ['parent-1'] + ); + assert.deepStrictEqual( + rows[0].children.map((child) => child.id), + ['c1', 'c2'] + ); + for (const child of rows[0].children) assert.strictEqual(Object.isFrozen(child), true); + } finally { + reads.restore(); + } + }); + + it('leading joinTo and joinFrom conditions read asynchronously and filter the selected children', async function () { + for (const name of ['children', 'linkedChildren']) { + clearValueMap(ColdChild); + const reads = spyPrimaryReads(ColdChild); + try { + const rows = await collect( + Parent.search({ conditions: [{ attribute: [name, 'name'], value: 'child-1b' }], select: ['id', name] }) + ); + assert.strictEqual(reads.sync, 0, name); + assert.ok(reads.async > 0, name); + assert.strictEqual(rows.length, 1, name); + assert.deepStrictEqual( + rows[0][name].map((child) => child.name), + ['child-1b'] + ); + } finally { + reads.restore(); + } + } + }); + + it('sorting by a to-one relationship attribute reads asynchronously', async function () { + for (const descending of [false, true]) { + clearValueMap(ColdChild); + const reads = spyPrimaryReads(ColdChild); + try { + const rows = await collect( + Parent.search({ + conditions: [{ attribute: 'group', value: 'g1' }], + sort: { attribute: ['primaryChild', 'name'], descending }, + select: ['id'], + }) + ); + assert.strictEqual(reads.sync, 0); + assert.ok(reads.async > 0); + assert.deepStrictEqual( + rows.map((row) => row.id), + descending ? ['parent-1', 'parent-2'] : ['parent-2', 'parent-1'] + ); + } finally { + reads.restore(); + } + } + }); + }); + + describe('select restrictions from non-leading conditions', function () { + it('every sibling condition on a to-many relationship restricts its select()', async function () { + for (const name of ['children', 'linkedChildren']) { + clearValueMap(ColdChild); + const rows = await collect( + Parent.search({ + conditions: [ + { attribute: 'group', value: 'g1' }, + { attribute: [name, 'label'], comparator: 'starts_with', value: 'match' }, + { attribute: [name, 'tier'], value: 'gold' }, + ], + select: ['id', { name, select: ['id', 'label', 'tier'] }], + }) + ); + assert.strictEqual(rows.length, 1, name); + assert.deepStrictEqual(rows[0][name], [{ id: 'c2', label: 'match-beta', tier: 'gold' }], name); + } + }); + + it('sibling conditions met by different children match the parent but select only children meeting both', async function () { + const rows = await collect( + Parent.search({ + conditions: [ + { attribute: 'group', value: 'g3' }, + { attribute: ['children', 'label'], comparator: 'starts_with', value: 'match' }, + { attribute: ['children', 'tier'], value: 'gold' }, + ], + select: ['id', { name: 'children', select: ['id'] }], + }) + ); + assert.deepStrictEqual(rows, [{ id: 'parent-3', children: [] }]); + }); + + it('branches of an OR group never narrow each other, whichever row is evaluated first', async function () { + const rows = await collect( + Parent.search({ + conditions: [ + { attribute: 'group', value: 'or-grp' }, + { + operator: 'or', + conditions: [ + { attribute: ['children', 'label'], value: 'A' }, + { attribute: ['children', 'tier'], value: 'gold' }, + ], + }, + ], + enforceExecutionOrder: true, + select: ['id', { name: 'children', select: ['id'] }], + }) + ); + assert.deepStrictEqual( + rows.map((row) => row.id), + ['or-a', 'or-b'] + ); + assert.deepStrictEqual(rows[1].children, [{ id: 'c8' }]); + }); + + it('a nested selection with a limit sees an OR-scope restriction registered while rows are filtered', async function () { + const rows = await collect( + Parent.search({ + conditions: [ + { attribute: 'group', value: 'or-grp' }, + { + operator: 'or', + conditions: [ + { attribute: ['children', 'label'], value: 'A' }, + { attribute: ['children', 'tier'], value: 'gold' }, + ], + }, + ], + enforceExecutionOrder: true, + select: ['id', { name: 'children', limit: 10, select: ['id'] }], + }) + ); + assert.deepStrictEqual(rows[1], { id: 'or-b', children: [{ id: 'c8' }] }); + }); + + it('a related table without a primary-key field selects matched records by their entry key', async function () { + const unfiltered = await collect( + Parent.search({ + conditions: [{ attribute: 'id', value: 'parent-1' }], + select: ['id', { name: 'compositeChildren', select: ['tenantId', 'itemId', 'label'] }], + }) + ); + assert.deepStrictEqual(unfiltered[0].compositeChildren, [ + { tenantId: 't1', itemId: 'i1', label: 'keep-me' }, + { tenantId: 't1', itemId: 'i2', label: 'drop-me' }, + ]); + const query = { + conditions: [ + { attribute: 'id', value: 'parent-1' }, + { attribute: ['compositeChildren', 'label'], comparator: 'starts_with', value: 'keep' }, + ], + select: ['id', { name: 'compositeChildren', select: ['tenantId', 'itemId', 'label'] }], + }; + const expected = [{ tenantId: 't1', itemId: 'i1', label: 'keep-me' }]; + clearValueMap(CompositeChild); + assert.deepStrictEqual((await collect(Parent.search(query)))[0].compositeChildren, expected, 'cold'); + assert.deepStrictEqual((await collect(Parent.search(query)))[0].compositeChildren, expected, 'warm'); + assert.deepStrictEqual(Array.from(Parent.search(query))[0].compositeChildren, expected, 'sync'); + }); + }); + + describe('multi-hop relationship conditions', function () { + it('a second condition on a leading two-hop path filters the same innermost records', async function () { + clearValueMap(Department); + clearValueMap(Employee); + const rows = await collect( + Company.search({ + conditions: [ + { attribute: ['departments', 'employees', 'role'], value: 'eng' }, + { attribute: ['departments', 'employees', 'active'], value: true }, + ], + select: [ + 'id', + { name: 'departments', select: ['id', { name: 'employees', select: ['id', 'role', 'active'] }] }, + ], + enforceExecutionOrder: true, + }) + ); + assert.deepStrictEqual(rows, [ + { + id: 'co-match', + departments: [{ id: 'dept-match', employees: [{ id: 'emp-match-1', role: 'eng', active: true }] }], + }, + ]); + }); + + it('a non-leading two-hop condition restricts the nested select()', async function () { + clearValueMap(Department); + clearValueMap(Employee); + const rows = await collect( + Company.search({ + conditions: [ + { attribute: 'id', value: 'co-match' }, + { attribute: ['departments', 'employees', 'tag'], comparator: 'starts_with', value: 'match' }, + ], + select: ['id', { name: 'departments', select: ['id', { name: 'employees', select: ['id', 'tag'] }] }], + }) + ); + assert.deepStrictEqual(rows, [ + { id: 'co-match', departments: [{ id: 'dept-match', employees: [{ id: 'emp-match-1', tag: 'match-tag' }] }] }, + ]); + }); + + it('a second hop through a relationship named like selection bookkeeping still filters and projects', async function () { + const rows = await collect( + Parent.search({ + conditions: [ + { attribute: 'group', value: 'g1' }, + { attribute: ['children', 'filters', 'code'], value: 'flag-on' }, + ], + select: ['id', { name: 'children', select: ['id', { name: 'filters', select: ['code'] }] }], + }) + ); + assert.deepStrictEqual(rows, [{ id: 'parent-1', children: [{ id: 'c1', filters: [{ code: 'flag-on' }] }] }]); + }); + + it('an id set of intermediate keys from two hops down is not tested as parent keys', async function () { + // planner-ordered, so the group condition leads with an estimate that enables the adaptive index switch + const query = { + conditions: [ + { attribute: 'group', value: 'idset-grp' }, + { attribute: ['children', 'grandchild', 'code'], value: 'gc-alpha' }, + ], + select: ['id'], + }; + const expected = ['idset-1', 'idset-3']; + assert.deepStrictEqual( + (await collect(Parent.search(query))).map((row) => row.id), + expected + ); + // LMDB cannot iterate an index-led filtered search synchronously + if (!isLMDB) + assert.deepStrictEqual( + Array.from(Parent.search(query)).map((row) => row.id), + expected + ); + }); + }); + + describe('nested relationship selections', function () { + it('a leading join with nested conditions reads candidates asynchronously and freezes them', async function () { + clearValueMap(ColdChild); + const reads = spyPrimaryReads(ColdChild); + try { + const rows = await collect( + Parent.search({ + conditions: [{ attribute: ['children', 'label'], value: 'match-beta' }], + select: ['id', { name: 'children', conditions: [{ attribute: 'tier', value: 'gold' }] }], + }) + ); + assert.deepStrictEqual( + rows.map((row) => [row.id, row.children.map((child) => child.id)]), + [['parent-1', ['c2']]] + ); + assert.strictEqual(reads.sync, 0); + assert.ok(reads.async > 0); + assert.strictEqual(Object.isFrozen(rows[0].children[0]), true); + } finally { + reads.restore(); + } + }); + + it('a nested condition through a further relationship reads it asynchronously', async function () { + clearValueMap(ColdChild); + clearValueMap(ColdGrandchild); + const reads = spyPrimaryReads(ColdGrandchild); + try { + const rows = await collect( + Parent.search({ + conditions: [{ attribute: 'id', value: 'parent-1' }], + select: [ + 'id', + { + name: 'children', + conditions: [{ attribute: ['grandchild', 'code'], comparator: 'starts_with', value: 'gc-' }], + select: ['id'], + }, + ], + }) + ); + assert.deepStrictEqual(rows, [{ id: 'parent-1', children: [{ id: 'c2' }] }]); + assert.strictEqual(reads.sync, 0); + assert.ok(reads.async > 0); + } finally { + reads.restore(); + } + }); + + it('under synchronous iteration a nested condition through a further relationship stays synchronous', function () { + const rows = Array.from( + Parent.search({ + conditions: [{ attribute: 'id', value: 'parent-1' }], + select: [ + 'id', + { + name: 'children', + conditions: [{ attribute: ['grandchild', 'code'], comparator: 'starts_with', value: 'gc-' }], + select: ['id'], + }, + ], + }) + ); + assert.deepStrictEqual(rows, [{ id: 'parent-1', children: [{ id: 'c2' }] }]); + }); + }); + + describe('sorting across a to-many relationship', function () { + it('sorts by the related values in relationship order, compared element by element', async function () { + const sortedIds = async (attribute, descending) => + ( + await collect( + Parent.search({ + conditions: [{ attribute: 'group', value: 'sort-grp' }], + sort: { attribute, descending }, + select: ['id'], + }) + ) + ).map((row) => row.id); + clearValueMap(ColdChild); + const reads = spyPrimaryReads(ColdChild); + try { + assert.deepStrictEqual(await sortedIds(['children', 'sortKey'], false), [ + 'p-sort-2', + 'p-sort-4', + 'p-sort-3', + 'p-sort-1', + ]); + assert.strictEqual(reads.sync, 0); + assert.ok(reads.async > 0); + } finally { + reads.restore(); + } + assert.deepStrictEqual(await sortedIds(['children', 'sortKey'], true), [ + 'p-sort-1', + 'p-sort-3', + 'p-sort-4', + 'p-sort-2', + ]); + assert.deepStrictEqual(await sortedIds(['linkedChildren', 'sortKey'], false), [ + 'p-sort-2', + 'p-sort-4', + 'p-sort-3', + 'p-sort-1', + ]); + }); + + it('a two-hop sort through a wide to-many relationship bounds its in-flight reads', async function () { + clearValueMap(ColdChild); + clearValueMap(ColdGrandchild); + const reads = spyPrimaryReads(ColdGrandchild); + try { + const rows = await collect( + Parent.search({ + conditions: [{ attribute: 'group', value: 'wide-grp' }], + sort: { attribute: ['children', 'grandchild', 'code'] }, + select: ['id'], + }) + ); + assert.deepStrictEqual(rows, [{ id: 'p-wide' }]); + assert.strictEqual(reads.sync, 0); + assert.strictEqual(reads.async, 40); + assert.ok(reads.maxInFlight <= 16, `at most 16 grandchild reads in flight, saw ${reads.maxInFlight}`); + } finally { + reads.restore(); + } + }); + + it('a sort through two to-many hops shares one read window', async function () { + clearValueMap(Department); + clearValueMap(Employee); + const reads = spyPrimaryReads(Employee); + try { + const rows = await collect( + Company.search({ + conditions: [{ attribute: 'group', value: 'wide-co' }], + sort: { attribute: ['departments', 'employees', 'tag'] }, + select: ['id'], + }) + ); + assert.deepStrictEqual(rows, [{ id: 'co-wide' }]); + assert.strictEqual(reads.sync, 0); + assert.strictEqual(reads.async, 36); + assert.ok(reads.maxInFlight <= 16, `at most 16 employee reads in flight, saw ${reads.maxInFlight}`); + } finally { + reads.restore(); + } + }); + }); +});