Skip to content

Preserve full-text query rejection reasons and document the 64-term limit #3447

Open

Description

@kriszyp

On Harper 5.3.2 (@harperfast/fulltext 0.5.0), a matches query with 65 distinct analyzed terms fails with ClientError: Full-text query on 'text' is invalid. The equivalent 64-term query succeeds. A query can cross this limit in only 249 characters, making natural-language prompt recall fail despite a 1,000-character application limit.

Minimal repro against a ready full-text index named text:

const q = Array.from({ length: 65 }, (_, i) => `w${i}`).join(' ');
for await (const row of Memory.search({
  conditions: [{ attribute: 'text', comparator: 'matches', value: q }],
  select: ['id'],
  limit: 10,
})) {}

Schema: text: FullText @fullText(fields: [{ name: "title", weight: 2.0 }, { name: "content" }]) on an audited table. An empty ready index is sufficient. Reducing length to 64 succeeds; joining those 65 terms with . instead of spaces also fails. Repeating one word 65 times succeeds, so whitespace-word count alone does not describe the boundary.

A direct native index.search({ text: q, mode: 'any', limit: 10 }) rejects with E_INVALID_ARGUMENT: search text produces more than 64 terms. runtimeInfo().limits reports maxQueryTerms: 64, maxQueryClauses: 256, and maxQueryTextBytes: 65536. matches maps to native any, so this repro does not involve prefix expansion.

Please document the analyzed-term and clause budgets at the Harper query boundary and preserve a safe diagnostic reason/code in the 400 response, or log the native reason without query text. resources/indexes/fullTextQueryIndex.ts:publicSearchError currently maps both E_INVALID_ARGUMENT and E_PREFIX_TOO_BROAD to the same generic 400 and loses their reason. Callers cannot distinguish excess terms from a broad prefix or another invalid argument.

Priority P2: contained shipped-version diagnostic failure; applications can bound automatic queries and degrade hybrid retrieval. Effort Low: expose a vetted reason/code or rate-limited diagnostic and document the existing budget. No release is committed; no customer or security details apply.

🤖 Generated by OpenAI Codex; posted via @kriszyp.

Activity

  1. added theissue type on Oct 22, 2026
  2. kylebernhardy commented on Nov 13, 2026

    @kylebernhardy
    Member

    Status from a review of the open full-text issues: half of this is delivered, and two related findings belong here. I have left the title alone, but it could be narrowed to "stable reason codes for full-text query rejections".

    • Delivered: the 64-term limit is documented and configurable since Adopt fulltext 0.8 with host query policy and fuzzy controls #3824 (resources/FULL_TEXT.md:176). The rejection message now names the limits (resources/indexes/fullTextQueryIndex.ts:1464-1468) and E_PREFIX_TOO_BROAD has its own message (:1462).
    • Remaining: a stable reason or code that separates term-count, clause-count, fuzzy and other E_INVALID_ARGUMENT rejections. The library does not expose a sub-reason today, so this likely needs a small library change or a synchronous precheck in Harper.
    • Finding 1 — ordinary client mistakes share the generic 400. A 2-character matches_prefix (the first keystrokes of any autocomplete), nesting past depth 8, more than 16 fuzzy terms and text over 64 KiB all return "invalid; check its syntax and storage.fullText.query limits". Harper wraps leaves in optional/boost/and, so the user-visible depth limit is below 8. If the library advertised the prefix minimums and depth in runtimeInfo(), Harper could reject these synchronously with a specific message.
    • Finding 2 — on REST the status is usually lost anyway. Everything in FullTextQueryIndex.search runs inside an async execute() consumed on the first next(), after the JSON serializer has already emitted [ (server/serverHelpers/JSONStream.ts:53-88). A native rejection therefore arrives as HTTP 200 with an in-band error. Document that full-text capacity rejections often arrive after the response starts #4308 documents this and Allow bounded first-item admission before committing streamed HTTP responses #2670 is the general fix; a synchronous precheck would sidestep it for the common cases above.

    Both findings are from reading the code, corroborated by integrationTests/server/stream-error-contract.test.ts; I did not run them against a live server.

    Comment generated by kAIle (Claude Fable 5.2)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions