Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
c288e1b
Read related records asynchronously when a query selects a relationship
kriszyp Oct 26, 2026
d3f0201
Read related records asynchronously when a query filters, joins or so…
kriszyp Oct 26, 2026
007bdd8
Settle relationship projections before the row, and keep warm relatio…
kriszyp Oct 26, 2026
e6ccda9
Describe the async join scan in resources/DESIGN.md
kriszyp Oct 26, 2026
a24401b
Merge remote-tracking branch 'origin/main' into kris/async-relationsh…
kriszyp Oct 26, 2026
4bd6035
Keep non-relationship filters on the synchronous loop, and end stoppe…
kriszyp Oct 26, 2026
2860814
End returned joins and finished-query reads without further work
kriszyp Oct 26, 2026
7f97618
Keep relationship filters synchronous under synchronous iteration
kriszyp Oct 26, 2026
abf8420
Load selected related entries with loadEntries, and keep synchronous …
kriszyp Oct 26, 2026
3957da0
Stop relationship scans on request teardown, and observe pending read…
kriszyp Oct 26, 2026
582a49a
Yield nothing more from relationship joins and sorts once the request…
kriszyp Oct 27, 2026
f4d1e7b
Correct two comments on stopped relationship scans
kriszyp Oct 27, 2026
9c8e300
Fold the relationship query-path note into one paragraph in resources…
kriszyp Oct 27, 2026
e22527c
Merge remote-tracking branch 'origin/main' into kris/async-relationsh…
kriszyp Oct 27, 2026
50d17d3
Let a join scan iterate a related search that returns a plain array
kriszyp Oct 27, 2026
a2a7385
Drop related-read guards that could never see a finished transaction
kriszyp Oct 27, 2026
3647e1c
Merge remote-tracking branch 'origin/main' into kris/async-relationsh…
kriszyp Oct 27, 2026
c15cb8b
Merge remote-tracking branch 'origin/main' into kris/async-relationsh…
kriszyp Oct 27, 2026
df11248
Stop relationship sorts and join scans on a released RocksDB read sna…
kriszyp Oct 27, 2026
fc83dd9
Cap a row's related reads in flight, and close iterators the snapshot…
kriszyp Oct 27, 2026
24c6106
Read a row's first related records together, and close the sort range…
kriszyp Oct 27, 2026
2b715c5
Merge origin/main into async-relationship-select
kriszyp Oct 27, 2026
fd30606
Merge origin/main into async-relationship-select
kriszyp Oct 27, 2026
ebc3fe9
Merge branch 'main' into kris/async-relationship-select
kriszyp Oct 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion resources/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,8 @@ A contract is `{ path, record?, get/post/put/patch/delete: { query?, body?, resp
- New `Resource` subclasses should override **instance** methods (`get`, `put`, ...) for behavior; static methods are the protocol entry points and stay generic.
- The `transactional()` wrapper resolves pending bodies before `allowCreate`/`allowUpdate` and instance dispatch, passing the same concrete object to both. For POST/PUT/PATCH it collects async name/value pairs with `resolveBody.ts`; Node and Web streams pass through unchanged, and generic iterator parts with additional fields are rejected rather than silently reshaped; QUERY resolves before resource loading so its projection participates in `allowRead`. Other methods resolve promises without collecting iterables: CONNECT carries live messages and PUBLISH can carry stream payloads. Authorization is latched before awaiting ordinary write bodies, and record loading can overlap body arrival. Decoder failures remain 400/413 rather than being remapped to authorization failures. Table's direct promised-hook overloads remain supported; restricted arrays retain their concrete-record refusal and explicitly listed readonly fields retain full-PUT backfill. `../unitTests/resources/requestBody.test.js` pins resolution, object identity, authorization timing and live-message preservation.
- **Overriding a static entry point takes over its whole contract.** Assigning `static post`/`put`/`patch` on a subclass shadows the `transactional()`-wrapped static, so none of that wrapper runs β€” including body resolution and the `allowCreate`/`allowUpdate` gate. That is by design (it is how `login.ts` implements a deliberately pre-authentication endpoint), and it means an override owns two obligations the wrapper would otherwise have met: (1) **Resolve or consume `data`:** ordinary REST bodies arrive as a pending promise; multipart forms arrive as `AsyncIterable<{ name, value }>`, without an opt-in flag. A static override can consume parts directly or delegate to `super.post(target, data)` (likewise PUT/PATCH). The inherited wrapper collects fields and repeated names, materializing each streaming file into a replayable Blob before advancing. Multipart filenames are own enumerable properties, and Blob decoding restores that shape; Bun’s inherited name setter otherwise hides the filename from a later encoding. Generic iterator bodies use exact name/value pairs; opaque record iterators belong in a static handler or a record property. `resolveBody.ts` owns named-part aggregation; the multipart iterator's internal `RESOLVE_BODY` hook owns file consumption, avoiding an HTTP parser dependency in Resource. Raw streamed Blobs retain `createBlob(Readable)`'s existing storage-backed read contract: save them before reading, or delegate to materialize the form. Collection buffers up to the existing HTTP body limit; applications needing large streaming uploads should consume in a static override. `defineResource.ts` uses the same resolver before body-schema validation, and `login.ts` resolves streamed credentials explicitly. (2) **The override makes the access-control decision:** no `allow*` predicate runs for it; an override that is not meant to be public must check authorization itself.
- **FK-side relationship accessors (`relationship.from`) are a synchronous contract** (v4 parity). The resolvers built in `Table.updatedAttributes` must read through `getSync`/`getEntry` (which block on a RocksDB block-cache miss, like v4's LMDB page faults) β€” never `get()`, whose `async: true` path returns a Promise on a cache miss and would leak an intermittent value-or-Promise to `record.<relation>` in user code. Pinned by the "relationship property access is synchronous" test in `../unitTests/resources/query.test.js`, which stubs `store.get` to throw (**known gap, not covered by that test:** the `.to` side resolved via `relatedTable.search(...).asArray` is synchronous for a local read, but `asArray` can still hand back a Promise when the search lands on a **caching** table whose row needs source revalidation β€” `transformEntryForSelect` returns `loadingFromSource.then(transform)` for an expired/invalidated entry; same bug class as the `getSync`/`getEntry` FK resolvers, not yet fixed on the `.to` side). **HTTP caching and relationship resolution** (pinned by `@relationship(from:) ETag / conditional GET invalidation` in `../unitTests/resources/restSelectProjectionFixes.test.js`): resolving a non-empty `relationship.from` foreign key marks the active request context uncacheable (`lastModified = UNCACHEABLE_TIMESTAMP`, skipping caching-source contexts) so single-record GETs with `select(...)` or `.property` projection do not emit a parent-only ETag or return a stale 304 after the target row changes, and `relationship.to` projections are likewise uncacheable through `relatedTable.search(...)` (`Table.ts`); **known gap:** `@enumerable` relationships (both `from` and `to`) that are not explicitly projected resolve inside `toJSON` during response serialization, after REST computes ETag/304, so a target row update can still produce a stale parent-version 304.
- **FK-side relationship accessors (`relationship.from`) are a synchronous contract** (v4 parity). The resolvers built in `Table.updatedAttributes` must read through `getSync`/`getEntry` (which block on a RocksDB block-cache miss, like v4's LMDB page faults) β€” never `get()`, whose `async: true` path returns a Promise on a cache miss and would leak an intermittent value-or-Promise to `record.<relation>` in user code. Pinned by the "relationship property access is synchronous" test in `../unitTests/resources/query.test.js`, which stubs `store.get` to throw. Known gap: the `.to` side (`relatedTable.search(...).asArray`) can still yield Promises when the related table is a caching table whose row needs source revalidation ([harper#3588](https://github.com/HarperFast/harper/issues/3588)). **HTTP caching and relationship resolution** (pinned by `@relationship(from:) ETag / conditional GET invalidation` in `../unitTests/resources/restSelectProjectionFixes.test.js`): resolving a non-empty `relationship.from` foreign key marks the active request context uncacheable (`lastModified = UNCACHEABLE_TIMESTAMP`, skipping caching-source contexts) so single-record GETs with `select(...)` or `.property` projection do not emit a parent-only ETag or return a stale 304 after the target row changes, and `relationship.to` projections are likewise uncacheable through `relatedTable.search(...)` (`Table.ts`); **known gap:** `@enumerable` relationships (both `from` and `to`) that are not explicitly projected resolve inside `toJSON` during response serialization, after REST computes ETag/304, so a target row update can still produce a stale parent-version 304.
- **Queries read related records through the async-capable path; only the accessor is synchronous.** Under asynchronous iteration, a query that selects, filters on, joins through, or sorts by a relationship must not call the resolver's synchronous form, which blocks the worker once per related record that misses the in-memory caches; under synchronous iteration it must, because nothing can wait on a Promise there. Each relationship resolver also carries `resolveKeys` (the FK value, or the `to` index scan's ids) and `loadEntries` (the `returnEntry` result read with `async: true`, settling to a Promise when a read goes to storage; once one read does, the rest go at most `RELATED_READS_IN_FLIGHT` at a time, so one wide row cannot queue thousands of background reads). Consumers that map entries to values decide one-or-many by the resolver's `toMany` flag, not `Array.isArray`, because a single related record may have a composite (array) key; `filterMissing` is applied after loading. Pinned by `../unitTests/resources/relationshipAsyncReads.test.js`, which counts the related table's blocking and Promise-returning reads, including a real cold read (flushed SST, cleared record cache) on RocksDB; its filter and sort cases use a related attribute that is deliberately not indexed, because an indexed one makes the planner lead with an index join instead, which for a many-to-one relationship never reads the related record. **Select:** `Table.transformEntryForSelect` takes the related entries from `loadEntries` and projects them through the target table's select transform as before; the row's projection waits on the Promise. A selection with nested `conditions`, `sort`, `offset` or `limit` (`Table._prepareRelationshipSelection`) instead reads the related keys (`resolveKeys`, or a filter's matched ids) and loads each candidate with `async: true` ahead of its yield point, so cancellation still stops reads at the first yield. Loading them through `loadLocalRecord` instead measured about 0.3 Β΅s per warm related row slower (`../unitTests/resources/relationshipWarmReads.bench.js`) and changed analytics and last-modified accounting. **Filter:** the relationship filter from `filterByType` keeps its synchronous form and adds a `maybeAsync` variant built on `loadEntries`; AND/OR groups and multi-hop paths chain through `maybeAsync`, and `transformToEntries` applies filters through it. The synchronous form stays for callers that need a boolean immediately: a custom index's pushed-down `recordFilter` and the synchronous join bodies used under sync iteration. One synchronous read path remains under a `maybeAsync` filter: the leaf comparator's adaptive switch to an index-derived id set (`attributeComparator`) scans the index and, for a `.to` relationship, reads each match with `getSync`. **Join:** a leading relationship condition's `joinTo`/`joinFrom` iterate the related search asynchronously under async iteration (`scanRelated`), reading related records with the query's read transaction; the scan is the only Promise a join returns. It stops, and yields nothing more, once its iterator is returned, as does the sorted post-ordering iterator. Request teardown (`closeOwnedReadIterators()`, the long-transaction monitor) releases the read snapshot through `onDone()` without returning the iterator, so after each awaited read both check `getReadTransactionGuard(readTxn)`, which throws the same `ReadSnapshotExpiredError` (503) or open-too-long error a RocksDB range read over a released snapshot does; `isDone` alone is no signal, because RocksDB handles never set it. LMDB, deprecated, keeps only the `isDone` check. **Sort:** `getAttributeValue` resolves relationship hops with `loadEntries`; the post-ordering iterator awaits the keys before sorting.
- **A combined `@relationship(from, to)` joins `related[to] == local[from]`, never the local primary key**, and only on an array-typed attribute (a scalar one is logged and gets no resolver). Four paths read `from` independently and must stay in step: the array resolver in `Table.updatedAttributes`, the index-driven join in `searchByIndex` (`joinTo` then `joinFrom`), the join-filtered select in `Table.transformEntryForSelect` (`resolver.from`), and a join filter's switch to an id set in `filterByType`. Pinned by the "@relationship with both from and to" block in `../unitTests/resources/query.test.js`, which uses a related record keyed by the local primary key as a decoy. Two more readers (`drivesRebuildingIndex`, the join estimate's `fromIndex` weighting in `estimateCondition`) affect only condition ordering, not join results, so they are not pinned by that block.
- When adding a new early-return path inside a commit handler in `_writeUpdate`, follow the blob-cleanup protocol in "Blob orphan cleanup" below.
- If you add a new top-level section to `Table.ts`, drop a `// #section: <name>` marker at its start and add a row to the section map above.
Expand Down
Loading
Loading