Full-text searches with waitForIndexMilliseconds: 0 intermittently return 503 while a reader refreshes #4224

Problem

A full-text search that passes waitForIndexMilliseconds: 0 can fail with 503 Full-text reader acquisition exceeded the index-wait budget on an index that is ready. It happens whenever the worker's search reader has to be reloaded or reopened when the query arrives. The query is rejected, not slowed down.

Seen on a 4-node v5.4.0-beta.3 cluster running hybrid (HNSW + full-text) search with waitForIndexMilliseconds: 0 and a maxIndexLagMilliseconds tolerance. On every node, including nodes that had not restarted, a polled search returned 503 for about 0.5 s roughly every 30 s. In an 80-query evaluation about 2.5% of searches failed.

A query that leaves out waitForIndexMilliseconds is not affected, even though 0 is the documented default.

Cause

FullTextQueryIndex.#acquireSearchReader (resources/indexes/fullTextQueryIndex.ts) bounds reader acquisition by the freshness option:

const acquisitionTimeout = Math.min(
	MAX_WAIT_FOR_INDEX_MILLISECONDS,
	Math.max(1, condition.waitForIndexMilliseconds ?? MAX_WAIT_FOR_INDEX_MILLISECONDS)
);

• An omitted option gets 30 000 ms. An explicit 0 gets 1 ms.
• A cached reader resolves in microtasks, so it wins the race. A reader that must be reloaded (new publication) or opened (new owner epoch, query-configuration change, cold worker) takes longer than 1 ms, and the query throws.
• The docs already say reader acquisition and freshness waiting "have separate bounds" (docs/reference/configuration/options.md). The code mixes them.

Why every ~30 s with no writes

Measured with the real native addon on a ready index that received no writes:

• The derived-index runner releases ownership after idleGraceMilliseconds (30 s) of idle.
• Harper's periodic audit-metadata commits (persistOriginClosedFloor, raiseAuditFloor in resources/auditStore.ts) emit committed. That wakes the runner, which takes ownership again and mints a new owner epoch: 2→3→4→5 at 33 s, 63 s and 93 s.
• Query reader slots are keyed by owner epoch (#readerFor), so every rotation forces each worker to open a new native reader. A no-wait query that arrives while that reader is opening fails.

Data publications cause the same failure through reader.reload(): 8 of 40 no-wait searches issued right after a write's publication failed.

Expected

waitForIndexMilliseconds bounds only the wait for index coverage (catch-up to a commit). Opening or reloading a reader on a ready index has its own bound. A no-wait query on a ready index never fails because a reader is being refreshed. A positive wait still bounds the coverage wait, and DERIVED_INDEX_LAGGING is still raised when maxIndexLagMilliseconds is exceeded.

Versions

The same expression is in v5.3.0 through v5.3.6 and in the v5.4 prereleases. It was introduced by #2855.

_{🤖 Generated by Claude (Anthropic, Opus); posted via @kriszyp.}