[kriszyp]

⊙ Problem

Since #4248, an index whose database gets no writes for about 30 seconds keeps no owner, so nothing refreshes its coverage clock. A worker that has never searched it has no reader cursor either, so its first no-wait full-text search fails with 503 DERIVED_INDEX_LAGGING although the index is current (QA: 33 of 72 first searches). Its first waiting search instead takes ownership, which reopens every worker's reader.

💡 Solution

When the idle owner releases, it already publishes a mark of the log positions it covered, and the runtime uses that mark to leave the index ownerless. Coverage now accepts the same mark through one shared predicate, quietMarkHolds: a ready index with no new owner, whose logs still match the mark, is current. The full-text query index opts in.

⚖️ Alternatives

🤔 Open the reader before checking coverage

This was the brief's first suggestion. Coverage would then be known only after an await, but searchByIndex reads indexCoverage synchronously for the Harper-Index-Coverage header. A cold waiting search would also still take ownership.

🤔 The quiet period is database-wide

The mark sums every log in the database. A write to any table, or a log rotation or purge, ends it, and a cold search can return 503 until an owner certifies again. Per-table tracking could narrow that later.

🤔 Only full-text opts in

HNSW reads its cursor from its own index store, and that cursor's coverage matches the mark at release, so its path is unchanged.

Other options

• Dormant runners refresh the clock: their wakes come from origin-floor commits about every 5 s, slower than the 3000 ms default tolerance, and only an owner may write the clock.
• A RocksDB copy of the full-text cursor: not atomic with the Tantivy commit payload it describes.
• The backend's getDurableCursor(): a synchronous filesystem inspection on the query path, which the design note rules out.
• Accept the first 503 as retryable: it was 0 of 48 on Keep an idle derived index's owner epoch and full-text readers when its database's log gets no new entries #4248's merge base.

🔧 Changes

• resources/derivedIndexRuntime.ts: quietMarkHolds is the predicate #quietSinceRelease used, now shared. readDerivedIndexCoverage takes an optional quiet identity and certifies through the mark when the clock and cursor cannot. It re-reads the owner epoch after reading positions. ⚠️ Look hardest at that ordering.
• resources/indexes/fullTextQueryIndex.ts: the opt-in is built once per index and passed to every coverage read.
• resources/indexes/fullTextDerivedIndex.ts: FULL_TEXT_QUIET_TOKEN names the backend's constant quietToken().
• resources/indexes/DESIGN.md: the cold-worker sentence names quietMarkHolds; a measurement sentence goes so the note fits its base-aware budget.
• Release note: cold first searches on an idle index no longer return 503, with the write, rotation and purge cases that still can.
• Performance: the bounded fast path is unchanged. A worker with no reader cursor now reads each log's stats, but only when the mark's epoch is current; a worker with a cursor reads them once, as before.

✅ Verification

• derivedIndexIdleEpoch.test.js (real RocksDB store, two runtimes, real full-text backend over an in-memory native lifecycle): first plain and scoring default searches after the clock ages past 3 s return the record with current; strict and waiting cold searches take no ownership; a held write still returns DERIVED_INDEX_LAGGING; negative cases cover another table or token, raised demands, an uncommitted tail, a new log and an epoch minted mid-read. Fails on base: DerivedIndexLagError … last certified 3214 ms ago.
• derivedIndexRuntime.test.js: the runtime contract with a storage-bound token. Fails on base.
• full-text-idle-coverage.test.ts with its fixture (schema, config, runner-lock probe): two HTTP workers, no search before the idle period, then the first plain and scoring QUERY return 200 with current; lag=0; tolerance=3000 and no new owner. Fails on base: the stream aborts with DerivedIndexLagError … last certified 5214 ms ago.
• PR QA: Do cold workers' first full-text searches after an idle release answer current, and never claim current while missing an acknowledged write? — queued @ fc7d79f, qa-pr-harper-4281-fc7d79f
• Docs: release note in this PR; the reference promises nothing about cold workers.

Full gates

• After rebasing onto main a1f2b11: test:unit:resources:core:a-n 3211 passing, 0 failing; the derived-index and full-text unit files 385 passing; full-text-idle-coverage, full-text-search and derived-index-idle-admission 20 pass, 1 skip.
• CI at db54a31: the Next.js adapter, Lint, Format, Docs and Lockfile jobs are green. Every red Integration and Unit job fails the same tests main fails today (getUser 401, SQA-181 legacy 404, mqtt oracle 5, Windows HNSW native-plane lag, branch blob-root Branched databases: blob store sharing, allocator, and GC safety #644, branch removal A fork-removal failure after drop_component answers early is only logged #3463). The shard numbers differ because this PR adds a test file.
• Before the rebase, on e7e9564: test:unit:resources 6772 passing, 110 failing, all main's (Revert rocksdb-js to 2.13.0: the 2.14.0 bump fails unit tests on every leg #4223 cascade, recordLock, rocksOpenConfig), fixed on main by Fix plain-to-HNSW index changes and main's red unit tests under rocksdb-js 2.14 #4260.
• test:unit:main wedged in the components process-group tests after 1093 tests on this machine, so it was run by directory instead: test:unit:server 2401 passing, 3 failing (no 127.0.0.2 alias, install sandbox, process-group harness); test:unit:dataLayer 565 passing, 0 failing.
• Integration: full-text-idle-coverage, full-text-search, full-text-search-docs, derived-index-idle-anchor, derived-index-idle-admission and native-plane-coverage: 32 pass, 1 skip. test:integration:all was not run: this machine has no loopback aliases.
• Lint and typecheck clean. check:design-docs --base origin/main passes for the indexes note.

Regression from #4248. Related: #4231, #4243, #4244.

_{🤖 Generated by Claude Opus 5.5 (Claude Code dev agent); posted via @kriszyp.}

🤖 Generated with Claude Code

Release target: v5.4 — fixes code from #4248, which ships in v5.4

Related PRs: #3846 overlaps (rewrites the same full-text query-plane paragraph of resources/indexes/DESIGN.md), #4248 overlaps (this fixes its cold-worker regression)
Complexity: complicated

Origin — the dispatch brief this PR was written from

Regression from harper PR #4248 (merged 2026-11-11, merge commit 240ac63): after a derived full-text index has been idle long enough for its owner to release (about 30 s with no writes), each cold worker's first default full-text search returns 503 DERIVED_INDEX_LAGGING even though the index is current. Mechanism on origin/main: resources/indexes/fullTextQueryIndex.ts defaults waitForIndexMilliseconds to 0 (~line 251), and on the no-wait path runs coverage = this.#queryCoverage(maxIndexLagMilliseconds) (~line 328) before #search opens a reader. readDerivedIndexCoverage (derivedIndexRuntime.ts ~2754) returns unknown when the shared coverage clock is stale and the worker has no #readerSlot cursor; #quietSinceRelease() (~1134, used ~1101) keeps an idle index from taking a new owner, so nothing refreshes the clock. #queryCoverage then throws and calls #refreshReader(true), which is why the worker's second search succeeds. Fix so a current idle index answers the first search: acquire/refresh the reader before the coverage check on the no-wait path (reuse the reader-acquisition budget added in PR #4231), or otherwise let a worker without a reader cursor establish coverage for a quiet index instead of reporting unknown. Locate code by symbol; line numbers are approximate.

Acceptance

A regression test (unit or integration) that fails on current main: index is current, no writes for longer than the owner-release quiet period, a worker with no reader slot runs a default (waitForIndex 0) full-text search and gets results, not 503; scoring and non-scoring paths both covered. Existing #4248 and #4231 tests stay green. A genuinely lagging index still returns DERIVED_INDEX_LAGGING. Update the relevant DESIGN.md section and add a release note if behavior text changes.

_{Dispatch: task harper-fulltext-idle-coverage-503 · queued by finding-triage/wave-52 · ran by claude/opus/xhigh · worker kris-mac-m3-fleet-18b9c6}

_{Ready-For-Promotion: db54a31 urgency=normal late=folded}

_{Review-Coverage: authored=claude; ran=cursor-composer,codex,cursor-grok,gemini; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=6; full=3 @ db54a31}

_{Review-Attention: deep ~40m (critical: derivedIndexRuntime.ts, fullTextDerivedIndex.ts +1; decisions: query-side-certification-source, constant-quiet-token-on-reader, database-wide-quiet-scope; raised: degraded review) @ db54a31}