Skip to content

Document that full-text capacity rejections often arrive after the response starts - #4308

Open
kriszyp wants to merge 5 commits into
mainfrom
docs/fulltext-capacity-rejection-status
Open

kriszyp wants to merge 5 commits into
mainfrom
docs/fulltext-capacity-rejection-status

Conversation

@kriszyp

@kriszyp kriszyp commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

The full-text querying page says query-capacity failures return 400 before the response starts. On a REST read Harper has usually sent 200 by then, so a client following the page waits for a status that never comes. Preserve full-text query rejection reasons and document the 64-term limit asks for client guidance built on that 400.

💡 Solution

A new section, How a capacity rejection reaches the client, says which rejections are a 400 and which arrive in the stream, what GET and QUERY return in each format, and what a client should check. Only the should list length is rejected before the query runs; the other limits fail during execution. Five sentences that promised a 400 now point to it. No product behavior changes.

🤔 Does this meet the ask in #3447?

Partly. The page documents the limits and the status a client gets, but the response still carries no reason or code: term, clause and fuzzy-limit overflow share one message, which the page now says. That is product work, so the issue stays open.

⚖️ Alternatives

Sized Blob HTTP responses write 200 before the backing file opens describes the same problem for blob responses and is not covered here.

🤔 Validate the native query before the response starts

Compiling the query against the host policy needs only the query and the policy, so doing it before serialization would make every capacity rejection a plain 400 and drop most of the new table. The task scoped this to docs, so the page describes today's contract. A later fix would make that advice obsolete; revising it is cheap.

🔧 Changes

✅ Verification

  • End-to-end route: live test on a Harper built from this branch with maxQueryTerms: 4 and maxQueryClauses: 8; the scratch probes are not committed. I sent over-limit full-text queries by GET, QUERY, HEAD and the operations API in JSON, NDJSON, CSV, CBOR and MessagePack, and every cell of the new table matches what came back. For example, an NDJSON GET of a 10-branch or group returned 400 while an 8-term matches returned 200 plus a $harperStreamError line; JSON returned a cut-off 200; CSV and MessagePack closed the connection with no status.
  • Also probed: a should list over the limit gives 400 in every format; a counted GET or HEAD (Prefer: count=estimated) gives 400; a plain HEAD gives 200; a resource that collects the iterator gives 400; 17 fuzzy terms and a fuzzy distance above the cap fail in-stream.
  • Gates: prettier --check passes on the three files. The unit and integration suites were not run because no code changed. npm run check:design-docs already fails on main (resources/DESIGN.md is 3 bytes over its ceiling since Clear a removed table TTL everywhere, reject invalid TTLs where they are declared, and scan catalog-loaded tables without a write #4250), and this PR does not touch that file. The docs site build runs in CI.
  • PR QA: skipped @ bd9e718 — docs-only.

🤖 Generated by Anthropic Claude Sonnet 5.5 via Claude Code; posted via @kriszyp.

Release target: main — documents the unreleased 5.4 full-text page

Related PRs: none found
Complexity: easy

Origin — the dispatch brief this PR was written from

Docs correction for harper PR #3847 (merged 2026-11-11). docs/reference/full-text-search/querying.md (~line 296) says query-capacity validation failures return 400 before the response starts, and ~line 139 says a compiled-clause budget overflow returns HTTP 400. On a REST collection read that is not what a client sees for a full-text query the native engine rejects for capacity: server/REST.ts gates the startup status through readStreamStartup only for GET/HEAD within a bounded startup window (~lines 274-276, the contract from PR #2614), and the capacity rejection arrives after the response has started, so the client gets 200 followed by a cut-off JSON body or an in-band NDJSON error record (QUERY has no startup gate at all). Line ~284 of the same page already says a later error terminates a stream that has started. Verify the behavior against the source for GET and QUERY and for each response format, then make the page accurate: state which capacity failures are a 400 (rejected before execution) and which surface in-stream, and what a client should check. Do not change product behavior.

Acceptance

querying.md statements about capacity failures match the source for GET and QUERY across JSON and NDJSON; prettier run on the touched files only; no code change.

Dispatch: task harper-docs-fulltext-capacity-status-streaming · queued by finding-triage/wave-53 · ran by claude/sonnet/xhigh · worker kzyp-xps-1

Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,gemini-carbon; late=gemini-carbon:adjudicated; adjudicated=domain; declined=cursor-grok,muse; rounds=5; full=1 @ bd9e718

Review-Attention: skim ~2m (decisions: requirement-shift, doc-placement) @ bd9e718

kriszyp and others added 4 commits November 12, 2026 08:59
…response starts

The full-text querying page said capacity failures return 400 before the
response starts. Only the `should` list length is checked that early. Term,
compiled-clause and fuzzy-distance overflow are found while the query runs, so
a REST read has usually sent 200 by then: JSON and CBOR are cut off, NDJSON ends
with an error record, CSV and MessagePack can close with no status, and QUERY
never gets a startup window. Add a section saying which rejections are 400, what
each format shows, and what a client should check. Correct the same claim in
query-optimization.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-docs-fulltext-capacity-status-streaming
…r methods

Review found three gaps in the new section. A GET with `Prefer: count=...` and a
bounded limit builds the page before responding, so it returns a plain 400 in
every format. HEAD returns an empty 200 for every format except NDJSON. Methods
other than GET and HEAD have no startup window. Point resources/FULL_TEXT.md at
the published section instead of repeating the old pre-response 400 claim.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-docs-fulltext-capacity-status-streaming
…pacity section

A counted HEAD builds its page like a counted GET, so it returns a plain 400.
The exception holds only within the count-page limits; larger pages stream.
The fixed native fuzzy limits fail during execution the same way as the
configurable ones. Add advice to use NDJSON to tell a rejected query from a
network failure, and not to retry a failed read unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-docs-fulltext-capacity-status-streaming
…vers

The note said "invalid-query rejection", which reads as narrower or broader
than the native E_INVALID_ARGUMENT mapping it describes. Name both cases it
covers: an invalid or over-capacity query.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-docs-fulltext-capacity-status-streaming
@kriszyp kriszyp added this to the v5.4 milestone Nov 12, 2026
… cover offset too

A late review of the previous head found two sentences that were true only in
part. An NDJSON HEAD that misses the startup window also returns an empty 200.
The counted-page exception stops at an offset past the scan window as well as at
a larger page.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-docs-fulltext-capacity-status-streaming

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant