Qualify native full-text filtering and catalog-scale operation for v5.4

Fulltext 0.8.0 adoption and phase-two boundary — 30 October 2026

Phase-one implementation: Adopt fulltext 0.8 with host query policy and fuzzy controls. Documentation: Document fulltext 0.8 query policy and fuzzy controls. These are draft changes, not a production rollout decision.

The adoption pins released fulltext 0.8.0, updates ABI 13/query API 7, adds host-controlled reader capacity and per-leaf fuzzy controls, and preserves existing source authorization/version/expiry checks. Defaults remain 64 terms, 256 clauses and maximum fuzzy distance one. Distance two requires explicit operator preview opt-in. All workers inherit one process-captured query policy; apply changes with a full-process restart. Writers keep native defaults and remain independent of query policy. This reader-only decision supersedes the earlier requirement to configure writer handles identically.

Phase two stays open in this issue: define metadata schema/projection/generation identity, metadata-only updates and field permissions; establish full-set authorization, current-source-version, expiry and coverage eligibility; integrate and qualify native sorting, filter pushdown, counts/facets and nested aggregations through Table.search and REST. Post-hit checks cannot repair incorrect native aggregates. Compare existing planner/candidate gating with native pushdown and preserve explicit errors or a correct fallback.

The phase-one focused suite passes 143 cases and the deployed two-worker full-text suite passes 16 cases. The small synthetic default-path comparison is not catalog-scale, concurrent production or sustained-memory qualification. Ordinary table predicates and full-text expression validation also need representative concurrent-load calibration; the added same-engine local guard-cost control is supporting evidence, not a production bound. The existing matched-workload, at-least-100-million-record and agreed query-class p99 gates remain open. Broader local repository gates are non-green and are disclosed in the adoption PR; this update does not waive them or qualify a downgrade procedure. Preserve lifecycle/operator recovery ownership in Implement full-text generation lifecycle, rebuild, eviction, backup, and status.

Existing priority, effort, parent, type and milestone remain unchanged.

Comment generated by kAIle (GPT-6)

Current work — consolidated 29 October 2026

This issue owns Harper consumer qualification and metadata adoption. Lifecycle, rebuild, backup/restore and operator recovery remain in Implement full-text generation lifecycle, rebuild, eviction, backup, and status.

Verified delivery: Fill full-text and autocomplete pages using Harper's existing paging and Document full-text paging and configurable search limits are merged. Each request uses one native reader for search, internal traversal and highlighting; separate pages are independent queries with no retained snapshot continuity guarantee. The superseded cross-request token registry, cursor TTL, remote-owner routing and unpublished-first-token cleanup are not current acceptance requirements. Historical evidence below is retained; unchecked historical lists do not reintroduce the discarded design.

Remaining acceptance

Workstream	Completion criteria
Matched workload and capacity	Compare standalone wrapper, Harper without fulltext and Harper with fulltext on matched hardware/runtime/corpora. Include deep offsets, sparse/no-match filtering, near-limit IDs, cold/warm storage, multiple indexes/workers, concurrent writes and unrelated traffic. Retain the original representative-catalog gate of at least 100 million records and the agreed p99 below 50ms target for accepted query classes; report cold/warm p50/p95/p99 separately. Record throughput, CPU, event-loop delay, memory, I/O, source reads, native batches, rejection rates and package revisions in a durable Harper-owned harness. Wrapper shared-runner results do not prove Harper parity.
Admission, fairness and cancellation	Exercise competing authenticated/anonymous requests across HTTP workers, aggregate native reservations, overload response latency and prompt capacity recovery after abort, serialization failure or native close failure. Measure highlighted-query reader acquisition and source/Blob/highlight waits; exercise publication/local rebuild with the published native addon through real multi-worker REST requests. Preserve current source-version and permission checks without transporting source text through a native-page broker. Measure retained snapshot disk pressure during active requests. Preserve count/byte/deadline bounds; propose quotas or a cache only after measurements justify them. There is no cross-request cursor pool to tune.
Limits, filtering and errors	Calibrate raw-hit/retained-result/request-budget defaults for interactive and bulk use; exercise live settings changes and expensive prefix/fuzzy/phrase queries. Compare existing candidate gating/adaptive batches with native-filter pushdown without duplicating Tantivy scoring. Preserve current permissions/versions/expiry, raw-work accounting and explicit budget failures. Define safe stable error/retry guidance; never silently truncate.
Relevance	Retain the full historical paired-version driver and broader diagnostic corpus; the checked-in 92-case wrapper subset is already delivered. Compare representative customer queries only with approved anonymized data and relevance labels. Exercise short-term stemming, fuzzy exact-match priority, REST permissions, stale/current versions, concurrent writes and larger segment/delete/merge shapes. A shared-runner +8.3% fuzzy-prefix tail is a confidence gap, not a proven regression.
Metadata adoption	After a compatible wrapper release, update ABI/query contracts and tests together; extend schema, projection and generation identity for sort/filter/facet fields, metadata-only writes, arrays/missing/null and field permissions. Integrate bounded metadata ordering and nested aggregations through Table.search/REST. Establish full-set authorization/version/expiry/coverage eligibility before returning counts/facets: post-hit checks cannot repair native aggregates. Reject unsupported predicates or use a defined correct fallback. Qualify RocksDB replay/restart/rebuild/close failures, replicated coverage equivalence, source restore and stale-negative behavior. Measure filter-only text re-analysis, segment/merge churn and write/disk amplification; recommend suitable field volatility/cardinality profiles.
Runtime and rebuild load	Investigate the recorded Bun malformed HTTP and restart abort without weakening readiness assertions. Measure long-scan catch-up, retryable write rejection windows and disk headroom under sustained load; keep multi-node/operator lifecycle acceptance in the linked lifecycle issue.

Configurable query capacity: Harper adoption

Make analyzed query capacity configurable per handle supplies the native policy. This issue owns Harper wiring and qualification after a compatible package release; the feature is not enabled by this checklist.

Verified gap: at Harper main inspected for this update, the dependency is pinned to 0.7.0; the binding expects native ABI 9/query API 4, and its native-open/search-limit configuration has no maxQueryTerms or maxQueryClauses fields. That was the historical gap. Released fulltext 0.8.0 uses native ABI 13/query API 7; the phase-one adoption above updates those contracts together. Reconcile the complete selected release contract during future adoption.

Compatible release: update the dependency/lockfile, binding types, native compatibility/capability validation and matching writer/reader contract tests together. Keep release version selection with release work; this checklist selects no version and changes no consumer pin.
Operator policy: expose and validate host-controlled maxQueryTerms and maxQueryClauses, defaulting to 64 analyzed terms per text leaf and 256 compiled clauses per query. Validate against the selected runtime's supported ceilings (the proposed release supports 1,024/4,096). Requests must not raise host policy. Derive text-tree structure from the configured compiled-clause budget (AND/OR children cost one each; NOT costs two before allocation). Preserve separate depth/byte/filter, phrase/fuzzy construction, queue, response and deadline safeguards; reject overflow explicitly.
Reader-handle ownership: the phase-one adoption applies term/clause policy to reader handles across HTTP/application workers, reacquisition, reopen and restart. Writers retain native defaults because they do not execute queries; invalid unvalidated query policy must not disable indexing. The process captures policy once and workers inherit it: a full Harper process restart applies changes, while worker replacement does not. Query policy stays outside persisted schema/analyzer identity and does not itself require an index rebuild. Qualify migration and production profiles separately.
Harper-owned qualification: exercise Table.search and real multi-worker REST with the published addon. Cover unchanged defaults, raised/lowered boundaries, documents matching only late query terms or later fields, ALL/phrase negative controls, traversal/highlighting, and explicit errors. Verify per-leaf terms plus Boolean/NOT and field-multiplied compiled cost, larger text trees and empty/stopword-only structural boundaries, ordinary/expensive fairness across handles/workers/indexes, concurrent writes, cancellation/deadline recovery and reopen/restart consistency. Retain source-version, expiry and permission checks; never truncate queries or duplicate Tantivy analysis/scoring in Harper.
Rollout and docs: document configuration scope, defaults, restart/reopen behavior, errors and resource tradeoffs. Qualify chosen profiles under this issue's matched-workload gates before enabling higher limits. Record conservative expensive admission for non-ASCII queries on raised-policy handles and the host-derived text-tree bound plus unchanged independent depth, byte, filter and construction limits.

Implementation and Harper qualification remain open. This adds adoption acceptance to the existing issue; it does not approve release, deployment or production enablement.

Comment generated by kAIle (GPT-6)

Native candidate-ID filtering: evaluation and adoption

Not enabled in Harper: native candidateIds submission is absent at the verified revision. Stop requiring unused full-text candidate capabilities removes unused validation; it does not enable this feature. This issue owns the evaluation and any subsequent Harper adoption.

Decision gate: compare existing Harper companion-index gating/adaptive paging, bounded native candidate IDs and indexed native structured filters on matched workloads. Reuse Harper's planner and secondary indexes; keep analysis, matching, scoring, ranking and native collection in fulltext/Tantivy. Enable pushdown only where measurements justify it.
Complete candidates and correct paging: use the existing native record-ID encoding/table identity. Require a complete candidate set or conservative superset for the pushed predicate; never use an approximate index, truncated first page or visible result limit. Define empty/duplicate semantics and account for concurrent mutations, coverage lag and stale entries. With supersets, retain Harper's remaining predicates and bounded traversal until the eligible page is filled, exhaustion established or an explicit budget failure occurs. Skip pushdown or fail when completeness cannot be established.
Harper-owned correctness: retain field/row authorization, current source-version checks, expiry and materialization. A candidate list grants no permissions. Define and test the consistency boundary with one native reader per request; that snapshot does not freeze Harper records or permissions. Cover worker/table isolation and metadata-only updates.
Capabilities and resource policy: validate positive safe-integer maxCandidateIds/maxCandidateBytes when consumed; preserve ABI/API and all other used-limit checks. Unselected paths must work without these optional capabilities. Align binding/request types, writer/reader compatibility and selected-path failure controls. Bound original count, combined UTF-8 bytes, applicable per-ID/packed-frame constraints, enumeration, deduplication, serialization, queueing and cancellation under the request budget. Version 0.7.0 limits are 1,024 IDs/1 MiB, counting duplicates; read the selected runtime rather than hard-code them. Overflow requires a correct existing path or explicit error, never truncation or independent top-k batching.
Equivalent results: compare IDs, scores, UTF-8 ties, offset/limit filling, highlights and total relations against equivalent existing predicates. Preserve ordinary paging between requests. Native counts, facets and metadata ordering require reviewed full-set eligibility; post-hit rechecks cannot repair incorrect native aggregates. Reject unsupported combinations or provide a defined correct fallback.
Executable evidence: test Table.search/REST with the published addon: empty/duplicate/boundary/oversized sets, Unicode/maximum-size IDs, Boolean/prefix/fuzzy queries, sparse/no-match/deep offsets, permissions, stale versions, concurrent writes, cancellation, rebuild/restart and multiple workers/indexes. Negative controls must distinguish missing/invalid capabilities and invalid inputs from successful empty results. Measure cold/warm p50/p95/p99, throughput, CPU/memory/event-loop effects, planning/encoding cost, source reads, native calls and rejections under matched mixed traffic.
Rollout gate: document planner eligibility, fallback/errors, observability and operator enable/disable/recovery. Preserve this issue's original reviewed qualification/rollout gates before feature exposure. Qualify higher candidate, prefix-expansion and payload capacity profiles owns wrapper capacity increases; do not bypass them with Harper-side batching/scoring.

No new tracker, release commitment, dependency bump or production enablement is introduced. Existing fields and historical review markers remain unchanged.

Consolidated trackers and boundaries

Support full-text and autocomplete continuation in Table.search and REST: delivered by ordinary request-scoped paging; retained-token design superseded.
Reuse full-text highlight tracing without an extra continuation reader: the merged query path passes its existing reader to highlighting; remaining highlight/cancellation measurements belong here.
Evaluate fairness for full-text continuation admission: applicable fairness, overload, cleanup and memory/disk measurements move here; token-specific semantics are superseded.
Preserve full-text query rejection reasons and document the 64-term limit remains the concrete diagnostics implementation task.
Remove unused full-text candidate limits from Harper activation requirements remains the bounded compatibility cleanup.
Add global metadata sorting, facets, and filter predicates and Qualify remaining native metadata collector resource tradeoffs retain wrapper implementation/qualification ownership. This consolidation does not approve their merge or Harper adoption.

The original non-token native-filter qualification matrix and rollout gates below remain binding, including working sets larger than RAM, page faults/cache contention, backlog/drain limits, metrics and alerts, and reviewed enable/rollback/incompatible-format rebuild procedures. Metadata schema/query exposure remains gated on reviewed and accepted results; historical package pins are evidence, not current version-selection instructions.

Completed upgrade/readiness CI and bounded safety tests are evidence below, not completion of these workload gates. Original references, review-deferral markers and unclassified failures are preserved in the archive. Existing priority, effort, type, parent, assignment and milestone are unchanged.

Comment generated by kAIle (GPT-6)

Historical qualification evidence and superseded checklists

Current qualification scope

Fill full-text and autocomplete pages using Harper's existing paging removes the separate continuation API. The current implementation uses ordinary limit/offset, one native reader per request and no retained state across requests. Earlier notes below describe intermediate implementations; their continuation registries, IPC, remote-owner timings and 100,000-hit defaults are historical evidence, not acceptance criteria for the current PR.

Current request defaults are 10,000 fetched hits, 10,000 retained offset-plus-limit results and 5,000 ms execution. Replay counts as raw work; raising raw work does not raise retained results. The first batch uses ordinary native search to preserve its queue isolation; fallback traversal uses the same reader. Fulltext 0.6.0 already provides the required traversal API.

Remaining work:

Measure deep offsets, sparse/no-match filters, synchronous source-read event-loop delays, memory and I/O at realistic catalog sizes and cold/warm cache states, under concurrent clients. Per-request budgets do not bound caller concurrency or total process memory.
Qualify fairness and admission across HTTP workers with expensive native queries and ordinary requests competing. Do not assume the native eight-expensive-search permit is a Harper per-client concurrency limit.
Check prompt cancellation during Blob/highlight reads and trace work. The current request checks cancellation between bounded work units; abort during an awaited highlight/source read can continue work until that wait finishes.
Compare candidate-gate and adaptive-batch behavior with existing native filtering, without duplicating Tantivy's query/scoring ownership. Separate future native-filter pushdown from the paging fix.
Review stable error codes/retry guidance for budget and native failures, including E_LOCK_BUSY, before treating the current human-readable messages as a programmatic API.

Current local evidence: 101 focused tests and 14 deployed HTTP tests on Node 24.16 and CI Bun 1.4.3 pass. Populated three-leaf reads plus policy resolution measured 0.638 µs median across 9 × 100,000 calls (0.631–0.667 µs); this microbenchmark does not qualify request latency or concurrent capacity. Full main/resource/integration failures remain disclosed in the draft PR and are not all classified. Do not treat historical measurements below as measurements of this implementation.

Related ownership:

Implement full-text generation lifecycle, rebuild, eviction, backup, and status.
Reduce repeated matching work during deep ranked traversal: native rescoring/collector optimization.
Make query term and clause limits configurable with bounded resource policy: the existing 64-term / 256-clause limits.
Preserve camel-case aggregate values when flattening configuration: shared configuration repair; current paging reads working leaf parameters.
Stop rolling-restart test probes after an assertion failure: independent test cleanup defect.

Comment generated by kAIle (GPT-6)

Metadata sorting, facets and predicates adoption

Track Harper adoption of Add global metadata sorting, facets, and filter predicates in this issue. Wrapper qualification does not enable these capabilities in Harper. Reuse the existing elected derived runtime, cursor publication, coverage and generation lifecycle.

Adopt a published compatible wrapper and update binding contracts, ABI/query validation and tests together. The inspected Harper implementation pins 0.7.0 and expects native ABI 9/query API 4; the wrapper PR advertises native ABI 11/query API 6. Version selection remains release-owned.
Extend schema declarations, stored metadata projection and generation identity for sort/filter/facet fields. Verify metadata-only updates, missing/array transitions and the defined treatment of null through the existing mutation runtime. Check permissions for every newly queryable field.
Integrate ordered multi-field sorting and nested aggregations with Table.search/REST and request budgets. Existing ranked paging uses score/ID order and continuation; metadata requests require a compatible bounded-offset plan. Preserve documented total relations and native terms approximation diagnostics.
Establish aggregate eligibility against the full authorized predicate and existing coverage policy. Current primary-version, expiry and record checks after native hits cannot repair already-collected native counts/facets. Reject or use a defined correct fallback for predicates that cannot cover the full native match set. Do not silently truncate or treat batched candidate IDs as equivalent global collection.
Qualify the actual Harper backend and Table/REST paths for metadata-only writes, permissions, expiry, stale versions, restart/replay, identity rebuild and native close failures. Retain existing elected ownership and fail-closed lifecycle behavior. Source runtime scope remains RocksDB.

Verified against the inspected upstream source: binding compatibility, derived projection, and authoritative hit rechecks. These are adoption acceptance criteria, not newly demonstrated defects in existing ranked search. No wrapper merge, publication, version or Harper dependency pin changes as part of this tracking update. Existing issue fields and ownership are preserved.

Comment generated by kAIle (GPT-6)

Historical scope and measurements from earlier implementations

Outcome

Qualify native Tantivy filtering and catalog-scale full-text operation before Harper exposes filterFields, native structured-filter pushdown, or native candidate-ID filtering. Harper may adopt @harperfast/fulltext 0.5.0 without enabling those capabilities; this issue gates the follow-up feature work.

Release boundary

Harper 5.3 may use Fulltext 0.5.0 for its existing filter-free behavior.
Harper retains its existing adaptive paging and companion secondary-index gates. Those operate in Harper at query time and do not copy structured fields into Tantivy.
Harper does not declare filter metadata in schema, write filter metadata to Tantivy, push structured filters into native searches, or send native candidate IDs until this issue is accepted.
Full-text source fields remain derived from Harper's authoritative records. Tantivy remains a local, rebuildable index.

Qualification matrix

Area	Required scenarios
Mutation correctness	Filter-only updates, text-and-filter updates, null/missing/list transitions, deletes, repeated updates, and values at supported limits
Query correctness	Searches before, during, and after watermark advancement; stale-positive removal; prevention or explicitly accepted handling of stale-negative results
Replication and recovery	Local and replicated writes, replay, new replica build, restart with reusable files, source restore, rebuild, worker handoff/loss, and close/drop
Sustained writes	Realistic pricing, inventory, availability, and category churn across one and multiple full-text indexes
Memory pressure	Warm and cold searches with the index smaller than RAM, larger than RAM, and competing with RocksDB plus the Node.js heap
Storage pressure	Segment creation and merge behavior, deleted-document accumulation, write amplification, temporary disk amplification, rebuild headroom, and shared-device contention
Query load	Representative term frequency, filter selectivity, native candidate-set size, result depth, concurrency, fuzzy/prefix/phrase/highlight usage, and mixed Harper traffic
Failure handling	Native writer/reader failures, stalled open/reload, Blob-source failures, cancellation, shutdown, backlog recovery, and unavailable/corrupt index rebuild

Acceptance criteria

Define and publish the representative catalog: at least 100 million records, field-size distribution, total indexed bytes, update mix, query mix, hardware, and concurrency.
Report cold and warm p50, p95, and p99 separately. Accepted query classes must meet the agreed latency target; the current target is p99 below 50 ms.
Measure source-write throughput, full-text mutation throughput, derived-index lag, backlog drain time, and the point at which indexing no longer keeps pace.
Demonstrate the consistency policy for stale native filter metadata. Authoritative record rechecks must remove stale-positive matches; stale-negative behavior must be prevented by coverage coordination or explicitly accepted and documented.
Measure filter-only update cost, including text re-analysis, segment churn, merge work, and disk amplification.
Measure resident memory, memory-mapped file size, page faults, storage reads, RocksDB cache impact, Node.js memory, and latency when the active working set exceeds RAM.
Exercise multiple indexes and Harper workers. Establish process-wide reader, writer, queue, thread, memory, and native candidate-set limits before enabling the feature.
Verify replica replay and rebuild produce equivalent query results at the same published coverage point.
Define metrics and alerts for readiness, watermark lag, queue depth, rejected mutations, merge pressure, disk headroom, rebuild progress, and native failures.
Document which field volatility and cardinality profiles are suitable for Tantivy filter metadata and which should remain Harper-side filters.
Store comparable benchmark results and workload metadata so changes can be evaluated across Fulltext and Harper releases.
Record the enable, rollback, and incompatible-format rebuild procedure.
Do not expose the Harper schema or query feature until the results and limits are reviewed and accepted.

Deliverables

Reproducible workload and benchmark harness.
Correctness and recovery test matrix with results.
Performance report covering steady state, cold start, memory pressure, rebuild, and sustained mutation load.
Recommended schema constraints, defaults, resource limits, metrics, and rollout plan.
Follow-up implementation issue or pull request only after the release gate is met.

Related work

Transferred comparative qualification

Run the standalone native wrapper, Harper without fulltext, and Harper with a native derived index on matched corpora, query mixes, concurrency, and documented hardware.
Include unrelated-table traffic, sustained mutations, multiple indexes, and multiple Harper workers; report product overhead separately from engine measurements.
Store workload definitions, package/Harper/Tantivy revisions, and comparable results across releases in Harper-owned benchmark history.

Generic native release benchmarks already exist; they do not establish the Harper catalog-scale result or authorize native-filter rollout. Recovery/replica/operational hardening is owned by Implement full-text generation lifecycle, rebuild, eviction, backup, and status.

Ownership reconciliation

Remaining Harper-specific acceptance criteria formerly tracked in fulltext now live in Implement full-text generation lifecycle, rebuild, eviction, backup, and status and Qualify native full-text filtering and catalog-scale operation for v5.4. Wrapper closure records are Deliver the standalone native full-text wrapper, Epic: Qualify, document, package, and release @harperfast/fulltext, Epic: Implement native checkpoint persistence and handle lifecycle, and Implement and qualify native checkpoint publication. Closing those records acknowledges completed wrapper delivery and transfers outstanding product qualification; it does not claim that Harper qualification is complete.

Deferred aggregate admission qualification

Related: Support full-text and autocomplete continuation in Table.search and REST. Its integration must bound its own traversal leases. Broader multi-index capacity qualification remains here.

Configure native process-wide admission before opens and test aggregate reader/writer/thread/queue reservations across Harper workers and indexes. Per-process limits do not imply a cluster-wide quota.
Qualify retained snapshot disk pressure, abandoned cursor cleanup, generation replacement and many concurrent cursors. Configured reservations do not bound total resident memory or retained disk.
Establish explicit overload/expiration outcomes and ordinary-query progress under the chosen host policy; do not infer production latency guarantees from the small native benchmark.

Deep-result and autocomplete completeness qualification depends on Support full-text and autocomplete continuation in Table.search and REST and a compatible published native package. Do not certify those result paths before the integration lands; unrelated lifecycle/coverage work can proceed independently.

Deferred continuation filtering work

The implementation of Support full-text and autocomplete continuation in Table.search and REST uses bounded native batches and current-record filtering. Its continuation path does not yet reuse the ordinary search path's companion-index candidate gate or adaptive batch growth. This is deferred performance work, not evidence that returned rows are incorrect.

Compare ordinary and continued searches with sparse indexed filters at 10,000, 50,000, and 100,000 fetched hits. Record source reads, native batches, worker messages, wall time, event-loop delay, and unrelated-query latency on documented hardware.
Evaluate reuse of Harper's existing candidate gate and adaptive batching before adding another filtering implementation. Preserve immutable token replay, native ordering, current version/permission checks, raw-hit accounting, and explicit budget errors.
Exercise short IDs and IDs near the native limit, warm/cold storage, multiple HTTP workers, and concurrent indexing. A single local traversal time does not establish production capacity.
Keep native ranking and traversal in fulltext/Tantivy. This work does not enable native candidate-ID pushdown or waive the qualification boundary above.

The initial continuation integration now includes the bounded local calibration below. Candidate-gate and batching optimizations, source-read and message accounting, and production catalog qualification remain separate; no release is committed for them.

Deferred limits-resolution work

The pre-PR continuation measurements count limits resolution on the requesting worker, snapshot owner and admission authority. The existing resolver remains the source of validation for mutable configuration; object-identity caching is unsafe because nested settings can change in place.

The initial integration resolves policy once per synchronous command and revalidates later commands. A regression mutates the same settings object to check count-limit shrink and recovery. This adds no cache.

Consider a revision-aware cache only if further measured benefit justifies it. Preserve fresh validation on later requests and worker messages, including in-place settings changes.
Test live changes to count, byte and request-budget settings. A cache or hoist must preserve token invalidation, overload refusal and cleanup recovery.
Keep measurements of a single resolver call separate from end-to-end page latency, native work and IPC. Further caching requires a measured benefit; no new cache or release is committed here.

Raw-hit default calibration

The initial continuation integration retains a configurable 100,000 raw-hit default for ordinary and continued searches. Restoring the former 10,000-hit ceiling would restore the result-depth failure for ordinary filtered queries. This accepts more bounded work to improve completeness; the measurements below show the cost rather than claiming performance parity.

Compare the current default with lower configured budgets under realistic sparse/no-match queries and mixed traffic. Record legitimate queries that cannot fill their visible page, time until an explicit budget error, native work, source reads and unrelated-query latency.
Recommend defaults for interactive and bulk workloads, including expensive prefix/fuzzy/phrase queries and concurrent callers. Keep the configured error boundary explicit; never replace a budget error with silent successful truncation.

This is further qualification and tuning, not a promised default change in the initial integration.

Observed pre-PR calibration, 19 October 2026

The unpublished implementation for Support full-text and autocomplete continuation in Table.search and REST includes a reproducible benchmark and structured results at benchmarks/fulltext-continuation/results-2026-10-19.md and .json. The report will be available with that integration; it is not yet on a published branch.

Observed on Apple M4 Pro, Node 24.16.0, RocksDB and published fulltext 0.6.0 / Tantivy 0.26.2:

All 48 ordinary/initial-page cases and 12 real two-worker resumed-page cases passed their result or explicit-budget-error assertions at 10k, 50k and 100k records.
At 100k, exact/prefix initial continuation with a one-in-500 filter and limit25 had p95 of 57–64 ms. No-match continuation had p95 of 618–664 ms, event-loop p95 of 28–30 ms, and unrelated-table HTTP p95 at or below 1.19 ms.
The identical five-row resumed page at 100k had median worker execution around 11.2 ms locally and 12.5–12.7 ms through the other worker.
Separate profiling counted resolver calls across all participating workers/authority: local resume 25→13, remote resume 49→25, initial sparse page 111→54, and initial no-match page 800→399. Populated resolution itself measured about 515–526 ns/call; the hoist removes modest CPU/allocation work, not native traversal cost.

Timing and V8 coverage run in separate processes. These are warm short-record workloads with 20 timing samples per case and three loaded samples, not a p99 or production-capacity qualification. macOS HTTP reaches one worker; a separate harness uses two actual Harper workers and production continuation messages. Source-read counts, exact wire-message counts, I/O, cold storage, concurrent indexing, large IDs and larger-than-RAM behavior remain unqualified. Keep bounded serialized-response checks while measuring their cost; do not remove enforcement solely to reduce serialization.

Comment generated by kAIle (GPT-6)

Tantivy 0.27 relevance comparison — observed 27 October 2026

Compared published fulltext 0.6.0 / Tantivy 0.26.2 with the local 0.7.0 / Tantivy 0.27.0 candidate from Upgrade fulltext to Tantivy 0.27 with safe index rebuilds. Harper was this main revision; the candidate's production implementation is this qualified revision. Node 24.16.0 / macOS arm64. No dependency pin or ranking defaults changed.

Results

Compared cases	Membership	Ranking
80 synthetic catalog cases: default and equal field weights, stop words disabled, index-time synonyms	Identical in all 80	Identical ordered IDs in all 80
48 diagnostic stemmer cases	Identical in 42; six changed	Identical in 41; seven changed
Same 128 cases through real Harper Table.search and audited RocksDB/native activation	Same outcomes as the direct wrapper comparison, including Harper-side source filtering	All 80 catalog rankings unchanged; the same seven diagnostic changes

Direct-wrapper results were repeated with forward/reverse insertion order in separate processes per version. All 128 within-version results were identical across those orders; 48 pagination comparisons passed. Both versions preserved fixture expectations for phrase gaps, Unicode folding, selected fields, title boosts, empty results and UTF-8 tie order. Four Harper highlight projections were identical. Five catalog hit scores differed by at most 0.00000095367431640625, without changing rank. No unexplained catalog result/ranking regression was observed in this fixture.

Observable diagnostic differences:

ad previously matched the ad, added, and adding word records; now it matches only ad.
add/added/adding/adds now share the same four-record match group. Previously add/adds and ad/added/adding formed separate groups. A single-term phrase query shows the same analyzer change.
Fuzzy added retains the same five records but promotes the new exact-match group, moving ad behind it.

These changes are consistent with the documented stemmer boundary. They are not proof that every changed ranking is more relevant to users.

Scope and reproducibility

Inputs are synthetic, not customer logs: 27 catalog records shaped by Harper full-text usage and query tests, plus 45 single-word diagnostics. Query shapes cover all six modes, Boolean groups, selected fields, availability/price source filtering, arrays, possessives, synonyms and Unicode. Each engine built fresh indexes from identical inputs. Table.search runs used trusted internal calls, not HTTP or caller authorization tests.

Local evidence retained: fixture, native driver, Harper Mocha probe, native/Harper raw result JSON, comparison generator and report. The generator independently checks both layers and regenerates the complete report deterministically. These artifacts are not yet checked into a repository or CI. Fixture SHA-256: 772a931814b25773c377549593da9ea5fc157834c3fc3d9ed20e2592f189c969; native driver SHA-256: 4840764e264982f5d15f89954a4806af9939f281c6662a8c949068ebfb0e4ecc; Harper probe SHA-256: c5425d80924b53939bf24dd4e8b5323949a8dfc5b456d415ddca4e5629687cb8.

Remaining relevance acceptance criteria

Compare representative checked-in Harper query shapes on the synthetic catalog and confirm results through Table.search.
Obtain an anonymized representative customer corpus/query set with relevance judgments. Compare precision/recall and NDCG/MRR only once labels exist; old/new overlap alone is not a relevance-quality score.
Review stemmer changes against domain vocabulary, especially short terms and fuzzy exact-match priority. Check in durable fixtures, the paired driver and approved expected differences as a regression gate.
Extend the passing Table.search comparison to REST, real caller permissions, stale/current source-version races and concurrent source updates. Preserve source projection and authorization in Harper and ranking in Tantivy/fulltext.
Qualify larger corpora and controlled segment/deletion/merge/rebuild shapes, plus the released 0.7.0 artifacts on supported platforms. The local candidate and small fixtures do not establish production relevance or capacity.

Migration and rollout remain owned by Implement full-text generation lifecycle, rebuild, eviction, backup, and status. This evidence narrows the relevance gap; it does not complete this qualification issue or authorize a production pin change. Existing issue priority, effort, type, parent and milestone are preserved.

Comment generated by kAIle (GPT-6)

Durable wrapper examples and regression subset

The reusable wrapper portion of the comparison is now committed in Upgrade fulltext to Tantivy 0.27 with safe index rebuilds: catalog example, shared records, expected rankings, test runner and usage/scope guide.

Preserve 92 query/profile cases: 21 catalog queries under four configurations plus eight focused stemming queries, each in both insertion orders. Explicit ordered IDs, exact counts, versions, UTF-8 ties and selected pagination protect the wrapper contract. Stop-word and title-only synonym cases now distinguish their options.
Run the example through local smoke and packed-consumer discovery. Full local checks and final platform CI pass on the committed revision.
Keep the full historical paired-version driver and broader diagnostic corpus reproducible in a durable harness. The committed 92-case subset is not the earlier complete 128-case wrapper/Harper comparison; no historical binary is needed by ordinary regression tests.
Keep Harper Table.search/source filtering, REST/permissions, current-version races, concurrent updates and lifecycle coverage in Harper. Customer relevance judgments and catalog-scale segment/delete/merge qualification remain open under the existing acceptance criteria.

This completes the reusable wrapper-fixture portion of the earlier durable-fixture item, without completing its paired-driver, broader vocabulary or Harper integration requirements. Existing priority, effort, type, parent and milestone are preserved; Harper's production pin is unchanged.

Comment generated by kAIle (GPT-6)

Fulltext 0.7.0 adoption performance follow-up

The long wrapper upgrade and 0.6.0 baseline already measure the engine swap on a fixed corpus: 100,000 documents, 10,000 mixed queries, concurrency four and identical Node 24.21.0 on Linux x64. Durable indexing was 82,263→85,729 documents/s; mixed QPS 1,336→1,424 and p99 5.576→5.170ms. Fuzzy-prefix p99 was worse, 3.361→3.639ms (+8.3%). These are separate shared runners, so neither a universal improvement nor a reproducible Harper regression is established. Qualify Tantivy 0.27 upgrade and enforce the analyzer rebuild boundary retains the three-pair interpretation and platform/per-mode limits.

Repeat representative Harper autocomplete/prefix/fuzzy-prefix workloads on matched runtime/hardware, including highlighting, authorization/filtering, exact-total and pagination profiles. Report p95/p99, throughput, rejected requests and memory; distinguish confidence limits from reproducible regressions.
Measure consumer costs beyond the wrapper benchmark: source scan, committed-write replay, authoritative hydration, REST admission and ordinary-versus-traversal queue isolation. The wrapper classifier/queue source is unchanged between releases; this is workload qualification rather than a demonstrated classifier defect.
Qualify long rebuilds and catch-up under concurrent writes. The existing lag policy can return retryable write 503s after a long scan; measure the rejection window and catch-up behavior before rollout. The adoption docs now warn operators; no lag-policy bypass or new runtime behavior is introduced.

This preserves the final review's performance-confidence concern as a follow-up, rather than treating one +8.3% tail observation as a proven regression. Existing issue priority, effort, type, parent and milestone remain unchanged.

Comment generated by kAIle (GPT-6)

Adoption safety checks and remaining workload gate — 28 October 2026

Upgrade full-text search to Tantivy 0.27 adds executable released fulltext 0.5.0 → 0.7.0 schema/REST upgrade, managed-worker drain, crash recovery and write-backpressure checks. The fixture version matches published Harper 5.3.4. Lifecycle evidence and scope are recorded in Implement full-text generation lifecycle, rebuild, eviction, backup, and status.

Demonstrate that write-lag rejection is retryable, preserves source/audit data and clears after publication catches up. This is a controlled correctness regression, not a measured rejection-duration or throughput guarantee under sustained load.
Verify deployed two-worker REST search becomes usable with current-analyzer results after old-index rebuild, and recovers by restoring the current engine after old-engine refusal. Streaming response bodies can contain terminal error records under HTTP 200; clients must check them.
Complete the matched Harper workload measurements above: concurrent indexing, hydration, permissions/filtering, highlighting, exact totals, pagination, multiple indexes/workers and unrelated traffic. Wrapper benchmark improvements do not prove Harper parity.
Investigate fuzzy-prefix tails under a controlled representative workload. The existing +8.3% shared-runner observation remains a confidence gap; it is not a demonstrated Harper regression.
Measure long rebuild admission/retry windows, disk headroom, catch-up and recovery under realistic write load. Keep multi-node/operator acceptance in the linked lifecycle issue open.

Claude/Composer reviewed the ordering fix; final direct sibling fixture fixes passed locally and in final-head CI. Final unit CI and integration CI pass on the qualification head. The integration result includes one successful targeted rerun of two unclassified numeric-index failures; the linked lifecycle issue retains the original failure evidence. No merge, deployment or production rollout approval is implied. Existing priority, effort, type, parent and milestone are preserved.

Comment generated by kAIle (GPT-6)

Review-Deferral: a605adc86ce788c0 severity=minor category=performance

This marker tracks the already-open matched-workload/fuzzy-prefix confidence gap above. The +8.3% shared-runner tail observation does not establish a reproducible Harper regression or waive the workload gate.

Bun 1.3.14: full-text upgrade and worker-restart qualification

On 2026-10-28, the local full-text suite ran on macOS arm64 with a Node 24.16.0 test client and Bun 1.3.14 serving Harper: 13 passed, 2 failed. The same two failures reproduce with the original integration test and historical-engine helper from base 04bdbe8a84e4f8f3e20eed3bcec5551f6712e8e3, against the unchanged built production runtime. Those two files were restored byte-for-byte afterward. This establishes that the failures predate the test-hardening follow-up; it does not identify their introducing commit.

The real 0.5-checkpoint upgrade probe receives malformed HTTP chunk framing: HTTPParserError: Invalid character in chunk size, with parser data onnection: close\r\n\r\n. This remains a hard failure; only recognized socket termination/reset errors are retryable. Do not treat malformed JSON, parser errors or request timeouts as readiness.
The request-limit test reports Harper stopped during worker restart: SIGABRT. Its native cause is not yet classified. Do not equate it with the separate uWS shutdown-abort issue without a matching trace.
The same final full-text suite passes 15/15 through Node HTTP and 15/15 through uWS. This does not establish Bun parity or customer-workload parity.

Remaining acceptance: investigate Bun HTTP framing and the restart abort, then pass the real two-worker old-index upgrade/recovery and request-limit restart cases on Bun without relaxing the failure oracle or skipping native coverage. Bun runs the main HTTP port on node:http, not Bun.serve — pick one transport supplies relevant transport context, but is not a proven root cause for either failure. Keep this qualification item open.

npm run build
HARPER_RUNTIME=bun npm run test:integration -- --isolation=none integrationTests/resources/full-text-search.test.ts

Comment generated by kAIle (GPT-6)

Upgrade probe review disposition

Review-Deferral: dc014df5b6e53a17 severity=nit category=reliability

Keep full-text upgrade polling within one readiness deadline, head 4091d374b36488bfb323b82c9a49c156f422ae64, supersedes the earlier disposition of the ten-second request limit. Each request now receives only the remaining time in the existing 30-second overall readiness deadline. The overall deadline is the hang oracle; it does not reset per retry. Expiry remains a hard failure with a readiness-specific assertion message. This addresses the review nit without claiming a Harper runtime defect. The marker above is preserved as review history, rather than an unresolved request for a timeout retry.

A complete HTTP 200 must still contain exactly the unchanged upgrade-ad witness, which predates shutdown and the replacement rebuild. Only an explicit 503 or closed/reset socket is retryable; malformed HTTP/JSON and timeouts never become successful readiness results. Real eleven-second delayed-header/body controls fail with the old ten-second expression and pass with the new remaining-budget expression; an exhausted 500ms budget still fails. Node and uWS deployed full-text suites each passed all 15 cases before the final message-only review fix; the exact final catch assertions and all 15 final Node integration cases pass, and all 72 final-head CI checks are now green.

The schema-level test likewise accepts either an explicit pre-header 503 or a terminated response carrying INDEX_REBUILDING. Its five-second deadline detects the previously reproduced hanging fixture. Generic JSON stream framing is separately exercised in the JSONStream suite. Preserve those boundaries when evolving the tests; do not turn timeouts, malformed HTTP/JSON, or empty completed upgrade results into readiness retries.

Comment generated by kAIle (GPT-6)

Test-hardening CI completion

Harden full-text upgrade crash and readiness tests, exact head 415f7b7925464cda72ffd6fcdc621370cd76200d, now has 72 successful CI checks and no unresolved inline findings. Native full-text passes on Linux, Windows and macOS. The single uWS numeric-index diagnostic rerun passed 584/584; its original failure and unclassified intermittent cause remain recorded in Implement full-text generation lifecycle, rebuild, eviction, backup, and status. This does not supersede the local Bun reproduction above or establish customer workload/capacity parity. Existing rollout gaps and the review-disposition marker remain open.

Comment generated by kAIle (GPT-6)

Readiness follow-up qualification complete

Keep full-text upgrade polling within one readiness deadline, exact head 4091d374b36488bfb323b82c9a49c156f422ae64, has 72 successful CI checks, no pending or failed checks and no unresolved review feedback at qualification. It remains test/documentation work: the remaining overall 30-second timeout, uniform expiry diagnostics and best-effort stream-error qualifier. The accompanying test-import correction uses #src manageThreads to preserve actual source-lane coverage; it does not change the classifier or relax its source expectation. The exact-base main job reproduces the old import-induced partition failure.

Compiled recorder plus runtime-selection suites pass 81; actual source-only recorder passes 59; final Node full-text cases pass 15. Prior uWS full-text cases pass 15 before the diagnostic-only refinement. Delayed-header/body controls prove the old 10-second cutoff failed valid eleven-second completions while the final remaining budget accepts them; expiry still fails. The final catch control covers no headers, 200 and 503.

Broad local outcomes are explicit limits: resources 5,759 passing/232 pending; bounded main 8,386 passing/224 pending/55 failing; integration 2,675 passing/74 failing/206 cancelled/18 skipped. These runs began before the final narrow fixes; focused checks cover the deltas. Main failures include concrete npm policy rejections, path-identity mismatches and process/deployment failures. Not all broad failures are classified, and these gates are not described as passing or universally baseline. Full-text/recorder/runtime-selection targeted checks and final hosted gates pass. The prior-head token-expiry and Windows staging-recovery failures did not recur on final-head CI; their causes remain unclassified, rather than silently waived. No further package, runtime, workload-capacity or production-rollout claim follows from this bounded test follow-up.

Three full outside review passes completed with Claude/Composer coverage 2/2. Final direct requested import/diagnostic fixes were locally verified without an exact-final outside rerun under the narrow final-artifact exception, disclosed in the PR. Machine convergence is not claimed. Broader rollout and customer-workload gaps in this issue stay open.

Comment generated by kAIle (GPT-6)

Facet API ownership

The focused child Expose authorized full-text facets and aggregation results through Table.search and REST owns the opt-in Table.search/REST facet response contract, full-set eligibility requirements, category/range bucket semantics and consumer tests. Coordinate it with the filtering/sorting/count work in this parent; do not duplicate metadata projection or aggregate validation.

This parent retains metadata adoption, native eligibility/coverage decisions, nested-aggregation qualification and catalog-scale/rollout acceptance. The child does not mark any parent acceptance complete, change current priority/effort/type/milestone, or commit its proposed API to this parent's release.

Comment generated by kAIle (GPT-6)

Stored-field ordering: retained memory and cancellation qualification

Allow stored-field sorting and exact counts for full-text search reuses Harper ordering and precomputes one decorated-key matrix for the complete retained set. This avoids repeated resolver/key conversion inside the comparator. The default retained ceiling bounds it; the committed warm single-client measurements do not establish peak-memory behavior for raised ceilings or wide linked sorts. Measure allocation/GC and event-loop delay before recommending larger limits, and compare alternatives only if they preserve Harper value and tie semantics. No memory regression is established by the existing before/after RSS snapshots.

Review-Deferral: 9e9bcae4b1f983c8 severity=minor category=performance

Outer cancellation checks preserve the request error, but candidate-gate fast returns can precede a bounded synchronous work interval, and highlight tracing checks cancellation around the complete helper rather than each native trace batch. Measure abort-to-capacity-recovery latency with real multiple trace batches and concurrent traffic; evaluate activity checks between batches without changing native ownership or suppressing application failures. This is cancellation responsiveness qualification, not an observed incorrect successful response.

Review-Deferral: 16ec1a665216eaf0 severity=minor category=performance

These extend the existing admission/fairness/cancellation and matched-workload acceptance above. Existing priority, effort, type, parent and milestone are unchanged.

Comment generated by kAIle (GPT-6)

Ordering review: capacity contract and cancellation fix

Allow stored-field sorting and exact counts for full-text search preserves the configured raw-hit ceiling, including native prefix replay. A sorted query whose text matches exceed that work budget still returns an explicit error even if companion predicates leave few eligible rows. The error now identifies sorting and suggests narrowing text, removing sorting or raising the work budget. Claude accepted this as the documented contract rather than an incorrect-result defect. Broader native pushdown and workload calibration remain in this issue.

Review-Deferral: 026f85d4beb9bcff severity=minor category=correctness

The cancellation follow-up at 50b34f815 adds activity checks after candidate planning and before source/highlight batches. Real-native negative controls demonstrate the extra candidate and trace calls before the fix. Strengthened tests cover both highlight call sites, multiple leaves within one source batch, and actual reader retirement; omitting the native-call guard or lease release makes the corresponding tests fail. This addresses the earlier 16ec1a665216eaf0 batch-responsiveness finding. Already admitted native calls and Blob hydration within one source batch remain bounded by their existing deadlines; this does not claim immediate cancellation of those operations.

Claude independently checked the candidate-planner algebra: with the current 10,000-hit native window, 4,096-key collection cap and consistent cardinality estimates, the old cap already saturates collection economics and does not change collection or probe selection. Severely underestimated statistics or a smaller future native window are outside that proof. The memory-matrix marker and peak/concurrent workload qualification remain open. Existing priority, effort, type, parent and milestone remain unchanged.

Comment generated by kAIle (GPT-6)

Local qualification evidence — 2026-11-01

Allow stored-field sorting and exact counts for full-text search retains native/Mac evidence, historical HTTP/Linux controls and the matched capacity matrix.

Observed 10M rows/table, NVMe/ext4, eight CPU/4 GiB/high=max/swap zero: feature 36 cells, 585,663 validated reads, 280 read failures, 105 lagging writes; baseline 12 cells, 392,921 reads, 112 terminated reads, 30 lagging writes. All 48 cells persisted; both runs unqualified. Stricter acceptance passes 17 feature/seven baseline cells: validated reads, no failures/unknowns, and nominal-ten-write service >=9.5/s over admission plus drain. All four full-text forms pass read-only at one/eight clients; all full-text write and 32-client cells fail. Baseline reproduces shared failure classes and structured write underdelivery; no shared-cause/statistical A/B claim. No sustained PSI stalls, OOM or watchdog stop; native/harness/executable/cache identities match. All failures and provenance retained. These one-minute cells do not establish soak, leak, cold or above-RAM capacity; evidence does not justify an automatic RAM increase.

Review-Deferral: 510ed61c761edb33 severity=nit category=correctness
Review-Deferral: 5806b74bd5b79d1d severity=nit category=reliability
Review-Deferral: 1f65a5f42734f2ae severity=minor category=reliability
Review-Deferral: ea7c5d20bb39cd11 severity=nit category=reliability
Review-Deferral: 3f6483bd798776f3 severity=nit category=documentation
Review-Deferral: d82bec31df53a7ba severity=nit category=documentation

Caps remain fail-closed; bucket sequence is not coverage. Earlier Linux tmpfs runs do not establish NVMe pressure. The disk-backed 2 GiB control ended under systemd-oomd before its structured timed cell/final verdict; it is partial overload evidence, not a pass. Historical raw numbers, failed setups and inference limits remain in the linked evidence. Above-RAM/cold I/O, sustained memory, fairness and published-wrapper/rebase gates stay open. No release sign-off.

Comment generated by kAIle (GPT-6)

Accepted bounded cancellation residual

Cancellation is checked before each native highlight/source batch. Sequential Blob reads already admitted within that batch can finish before the next check. In-flight native calls finish before the reader lease is released. Keep this bounded residual in the qualification matrix; no immediate new mechanism or cancellation race is proposed.

Review-Deferral: 634218cbceedbfca severity=nit category=reliability

Comment generated by kAIle (GPT-6)

Accepted exact-count drain cost

The Table drain must apply final expiry and row guards after native enumeration. It currently also projects rows that are only being counted; exhausting its separate timer preserves the page with an unknown total, as documented. Track an optimization that keeps final eligibility but avoids unnecessary tail projection. Do not substitute native/retained totals for Harper-eligible totals.

Review-Deferral: 2ca9fdc0d586da65 severity=minor category=correctness

Comment generated by kAIle (GPT-6)

Accepted offset eligibility cost

Highlighted exhaustive preparation checks final source/expiry eligibility for offset rows before selecting the visible page; those rows skip projection and native tracing. The deep-offset observation measures this accepted cost. Any optimization must preserve final eligibility and reader identity. Track it within the existing qualification scope; existing priority/effort/milestone remain unchanged.

Review-Deferral: f7af8528bf2b4ef2 severity=nit category=performance

Comment generated by kAIle (GPT-6)

Optional scoring admission: clause count × field width

Follow-up from Add full-text boosts and optional scoring conditions: qualify the actual Table._prepareQueryConditions path, including repeated field-name arrays, Sets and permission checks. The earlier validator-only benchmark does not cover this path.

Local exploratory measurements used Node 24.16.0 on macOS ARM64, compiled production admission, activated tables with restricted field permissions, fresh requests and five rounds of 5,000 iterations. Median microseconds per request (default fields / explicit fields):

Source fields	1 optional clause	32 optional clauses	127 optional clauses
2	0.59 / 1.07	5.09 / 9.07	17.62 / 27.47
16	1.42 / 3.61	25.02 / 74.46	99.02 / 279.33

These include request construction and assertion overhead. Other tests were running; the explicit 16-field/127-clause samples ranged from 168 to 573 microseconds. A test native binding was used only for table activation. The measurement excludes native query execution, network, allocation profiling, sustained concurrency and service p99. It is not a before/after regression result or evidence that Sets alone account for the measured cost.

Qualification should cover the clause-count × field-width matrix under representative concurrent load, capturing allocation/GC and event-loop delay alongside service latency. If material, evaluate immutable index metadata or query-local reuse; preserve per-clause source authorization, schema invalidation and isolation from caller-owned arrays. Existing issue priority, effort, parent, type and v5.4 milestone remain unchanged; this extends the existing qualification matrix.

Comment generated by kAIle (GPT-6)

Accepted compatibility and timeout contracts to retain in qualification

For Add full-text boosts and optional scoring conditions, legacy ABI 13/query API 7 remains intentionally supported for ordinary queries; scoring fails closed with nonretryable FULL_TEXT_SCORING_UNSUPPORTED (503). There is no activation warning for this supported configuration. This is an accepted compatibility choice, not a defect requiring a warning.

Capability-loading timeout has the stable retryable FULL_TEXT_INITIALIZATION_TIMEOUT code; the pre-existing reader-acquisition timeout remains generic 503. The documentation distinguishes them. Preserve those observable contracts in qualification; any later unification should be an explicit API decision with transport-level tests.

Review-Deferral: b4663a49fc8438d8 severity=none category=maintainability
Review-Deferral: 4283dd7c63dc89f3 severity=none category=correctness

Existing issue fields remain unchanged.

Comment generated by kAIle (GPT-6)

The raw measurements supporting Add full-text and autocomplete snapshot continuation are preserved in an immutable commit archive. The file is 118,079 bytes; its Git blob is b0d739a324634a23263c54ff603b95c98666e21e.

Kris requested removing that raw JSON from the active PR diff. The reproducible harness and readable measurement summary remain available. These measurements precede the review repair restoring a separate 10,000-hit ordinary-search ceiling; they must not be presented as measurements of the repaired default. New measurements will be recorded separately.

The remaining catalog-scale qualification and sparse-filter continuation optimizations remain in this issue's scope.

Comment generated by kAIle (GPT-6)

Review-repair measurements for Add full-text and autocomplete snapshot continuation, 20 October 2026.

Same Apple M4 Pro, Node 24.16.0, fulltext 0.6.0 / Tantivy 0.26.2; 100,000 records and 20 sequential samples per case. Timing ran separately from other tests/builds. Three loaded samples per first-page case are recorded separately.
Ordinary sparse/no-match p95: 21.26–33.49 ms, with an explicit 400 at the restored 10,000-hit ordinary ceiling. No budget error was accepted as a successful empty page. The historical 100,000-hit no-match results were 155–432 ms, depending on the companion index; these are different work ceilings, not an algorithmic speedup.
Continuation still completes sparse/no-match searches: sparse p95 60–69 ms, no-match 650–747 ms. The historical no-match continuation p95 was 618–664 ms. This run does not establish an improvement there; repeated native scans and continuation filtering remain deferred in this qualification issue and Reduce repeated matching work during deep ranked traversal.
Resumed five-row pages p95: local exact 15.60 → 13.50 ms, remote exact 17.96 → 15.18 ms; local prefix 13.66 → 13.11 ms, remote prefix 16.30 → 14.15 ms. All paths returned identical expected rows. These are warm local observations, not production guarantees or an isolated attribution to serialization.

The after measurements used the repair working tree based on f4a8fbe0da2ff237ffce343312c2febce9512c5f; the environment's commit field names that base, not a committed repair. Compiled-file SHA-256 fingerprints below identify the measured source build. The raw records are preserved here rather than added to the PR diff.

Commands (each run separately):

FT_BENCH_SIZES=100000 FT_BENCH_SAMPLES=20 npm run test:integration -- benchmarks/fulltext-continuation.bench.ts
FT_BENCH_SIZES=100000 FT_BENCH_SAMPLES=20 node node_modules/mocha/bin/mocha.js benchmarks/fulltext-continuation-workers.cjs

Measurement records and compiled-build fingerprints

{"environment":{"size":100000,"samples":20,"node":"v24.16.0","reachableHttpWorkers":1,"commit":"f4a8fbe0da2ff237ffce343312c2febce9512c5f","cpu":"Apple M4 Pro","native":{"packageVersion":"0.6.0","tantivyVersion":"0.26.2","nativeAbiVersion":9,"queryApiVersion":4,"queryClassIsolationMinimumSearchThreads":2,"lifecycleApiVersion":1,"mutationBatchApiVersion":5,"storageBackends":["native"],"limits":{"maxCommitPayloadBytes":65536,"maxQueryTextBytes":65536,"maxQueryTerms":64,"maxQueryClauses":256,"maxCandidateIds":1024,"maxCandidateBytes":1048576,"maxRecordIdBytes":4096,"maxRecordVersionBytes":4096,"maxPrefixExpansions":50,"maxFuzzyTerms":16,"maxSearchWindow":10000,"maxSearchPageSize":10000,"maxFilterStringBytes":65530,"maxAutocompleteResults":100,"maxSearchRequestBytes":8388608,"maxSearchResponseBytes":8388608,"maxSearchBudgetMilliseconds":30000,"maxTraceRecords":100,"maxTraceSourceBytes":1048576,"maxTraceSpans":1024,"maxSynonymRules":1024,"maxSynonymReplacements":16,"maxSynonymBytes":1048576}}},"compiledSha256":{"dist/resources/Resource.js":"3687aa377cc5cbc20e11f2db0733b01620d28a2e873dfaba82a8fa6c53247ff1","dist/resources/Table.js":"e1f7a96edf26035a7f7921e8ee2d1954cade6e829b44f5a6914defb8f33cf448","dist/resources/indexes/fullTextQueryIndex.js":"6e85d5ac866248c6ee5ce328377533aad3833f0907e077c1c8c3f99dbe70ef5d","dist/resources/indexes/fullTextQueryProtocol.js":"320b280b5243dc496bc452dcffaa68f6ec2188738e5f70ec28700d5cf4a625be","dist/resources/indexes/fullTextContinuation.js":"8a1f87994a6e1ae35efb3ae1589336b88ce045005dba3b1ba6c1b5533e578c47","dist/resources/indexes/fullTextContinuationOwner.js":"e965278ccd65d2c4366b5b6d299ec190440516adcab6d7b439fee4b0facc8542","dist/resources/indexes/fullTextRuntimeConfiguration.js":"2ed1e247f66c7f75a292dbfb9b44682817a8cd01bb911345e15e79a0284a3f63","dist/resources/indexes/fullTextNativeBinding.js":"d0135c6fb90db50ad6bfa492d4dfd02ad63c787a6e7dbac78555c9467c4efd15"},"http-measurements-after":[{"size":100000,"comparator":"matches","filter":"sparse-unindexed","mode":"ordinary","latencyMilliseconds":{"samples":20,"p50":30.62237500000083,"p95":33.48545800000102,"max":33.88199999999961},"loadedLatencyMilliseconds":{"samples":3,"p50":35.696874999999636,"p95":37.035499999999956,"max":37.035499999999956},"controlLatencyMilliseconds":{"samples":156,"p50":0.6008750000000873,"p95":2.0902079999996204,"max":6.150416000000405},"eventLoopDelayMilliseconds":{"samples":23,"p50":2.670591,"p95":3.002367,"max":4.644863},"peakProcessRssBytes":782352384,"outcomes":[400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400]},{"size":100000,"comparator":"matches","filter":"sparse-unindexed","mode":"continuation","latencyMilliseconds":{"samples":20,"p50":62.649749999998676,"p95":67.05258400000093,"max":68.72379199999887},"loadedLatencyMilliseconds":{"samples":3,"p50":67.94433299999946,"p95":69.78400000000147,"max":69.78400000000147},"controlLatencyMilliseconds":{"samples":352,"p50":0.5542919999988953,"p95":1.0438750000012078,"max":3.3460420000010345},"eventLoopDelayMilliseconds":{"samples":23,"p50":1.896447,"p95":2.160639,"max":2.406399},"peakProcessRssBytes":768131072,"outcomes":["success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success"]},{"size":100000,"comparator":"matches","filter":"none-unindexed","mode":"ordinary","latencyMilliseconds":{"samples":20,"p50":30.513832999999067,"p95":32.10270900000069,"max":39.92250000000058},"loadedLatencyMilliseconds":{"samples":3,"p50":35.09583399999974,"p95":35.64129099999991,"max":35.64129099999991},"controlLatencyMilliseconds":{"samples":212,"p50":0.3875420000003942,"p95":1.5202920000010636,"max":4.594834000001356},"eventLoopDelayMilliseconds":{"samples":23,"p50":2.623487,"p95":2.844671,"max":3.049471},"peakProcessRssBytes":717946880,"outcomes":[400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400]},{"size":100000,"comparator":"matches","filter":"none-unindexed","mode":"continuation","latencyMilliseconds":{"samples":20,"p50":633.6810829999995,"p95":747.4714170000007,"max":775.0284580000007},"loadedLatencyMilliseconds":{"samples":3,"p50":781.4812080000011,"p95":796.8337919999976,"max":796.8337919999976},"controlLatencyMilliseconds":{"samples":2309,"p50":0.7275829999998678,"p95":1.8696249999993597,"max":34.4173750000009},"eventLoopDelayMilliseconds":{"samples":23,"p50":25.149439,"p95":32.161791,"max":32.604159},"peakProcessRssBytes":718864384,"outcomes":["success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success"]},{"size":100000,"comparator":"matches","filter":"sparse-indexed","mode":"ordinary","latencyMilliseconds":{"samples":20,"p50":22.89533400000073,"p95":24.57325000000128,"max":24.95691699999952},"loadedLatencyMilliseconds":{"samples":3,"p50":23.524916999998823,"p95":23.694500000001426,"max":23.694500000001426},"controlLatencyMilliseconds":{"samples":177,"p50":0.45470900000145775,"p95":0.87066699999923,"max":2.404499999996915},"eventLoopDelayMilliseconds":{"samples":23,"p50":1.951743,"p95":2.146303,"max":2.189311},"peakProcessRssBytes":342032384,"outcomes":[400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400]},{"size":100000,"comparator":"matches","filter":"sparse-indexed","mode":"continuation","latencyMilliseconds":{"samples":20,"p50":65.33216700000048,"p95":69.2311660000014,"max":100.63604200000191},"loadedLatencyMilliseconds":{"samples":3,"p50":66.77300000000105,"p95":70.11104200000045,"max":70.11104200000045},"controlLatencyMilliseconds":{"samples":397,"p50":0.4255410000005213,"p95":0.9709169999987353,"max":9.77362500000163},"eventLoopDelayMilliseconds":{"samples":23,"p50":1.747967,"p95":2.156543,"max":23.052287},"peakProcessRssBytes":325599232,"outcomes":["success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success"]},{"size":100000,"comparator":"matches","filter":"none-indexed","mode":"ordinary","latencyMilliseconds":{"samples":20,"p50":20.31987499999741,"p95":21.259750000001077,"max":21.618000000002212},"loadedLatencyMilliseconds":{"samples":3,"p50":22.370333000002574,"p95":22.39179200000217,"max":22.39179200000217},"controlLatencyMilliseconds":{"samples":210,"p50":0.3400839999994787,"p95":0.7154579999987618,"max":6.945541000000958},"eventLoopDelayMilliseconds":{"samples":23,"p50":1.883135,"p95":2.199551,"max":2.316287},"peakProcessRssBytes":307101696,"outcomes":[400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400]},{"size":100000,"comparator":"matches","filter":"none-indexed","mode":"continuation","latencyMilliseconds":{"samples":20,"p50":613.0521660000013,"p95":649.520832999995,"max":661.0045000000027},"loadedLatencyMilliseconds":{"samples":3,"p50":699.2091250000012,"p95":819.6363750000019,"max":819.6363750000019},"controlLatencyMilliseconds":{"samples":2503,"p50":0.5588330000027781,"p95":1.9019999999945867,"max":36.72833299999911},"eventLoopDelayMilliseconds":{"samples":23,"p50":26.148863,"p95":29.638655,"max":30.900223},"peakProcessRssBytes":364363776,"outcomes":["success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success"]},{"size":100000,"comparator":"matches_prefix","filter":"sparse-unindexed","mode":"ordinary","latencyMilliseconds":{"samples":20,"p50":29.207416999997804,"p95":30.78962500000489,"max":67.91412500000297},"loadedLatencyMilliseconds":{"samples":3,"p50":30.863375000000815,"p95":32.72808299999451,"max":32.72808299999451},"controlLatencyMilliseconds":{"samples":216,"p50":0.35779200000251876,"p95":1.4479170000049635,"max":6.688874999999825},"eventLoopDelayMilliseconds":{"samples":23,"p50":2.437119,"p95":2.775039,"max":28.278783},"peakProcessRssBytes":363642880,"outcomes":[400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400]},{"size":100000,"comparator":"matches_prefix","filter":"sparse-unindexed","mode":"continuation","latencyMilliseconds":{"samples":20,"p50":58.7642909999995,"p95":60.02195799999754,"max":60.17033300000185},"loadedLatencyMilliseconds":{"samples":3,"p50":63.53270799999882,"p95":65.26412500000151,"max":65.26412500000151},"controlLatencyMilliseconds":{"samples":479,"p50":0.3284590000039316,"p95":0.776625000005879,"max":9.190290999998979},"eventLoopDelayMilliseconds":{"samples":23,"p50":1.552383,"p95":2.101247,"max":2.177023},"peakProcessRssBytes":382713856,"outcomes":["success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success"]},{"size":100000,"comparator":"matches_prefix","filter":"none-unindexed","mode":"ordinary","latencyMilliseconds":{"samples":20,"p50":29.398667000001296,"p95":32.914834000002884,"max":38.09729199999856},"loadedLatencyMilliseconds":{"samples":3,"p50":34.74604199999885,"p95":35.097874999999476,"max":35.097874999999476},"controlLatencyMilliseconds":{"samples":219,"p50":0.3730829999985872,"p95":1.6170000000056461,"max":7.24033299999428},"eventLoopDelayMilliseconds":{"samples":23,"p50":2.592767,"p95":2.836479,"max":5.926911},"peakProcessRssBytes":395149312,"outcomes":[400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400]},{"size":100000,"comparator":"matches_prefix","filter":"none-unindexed","mode":"continuation","latencyMilliseconds":{"samples":20,"p50":646.5997920000009,"p95":717.7477920000019,"max":834.1469170000055},"loadedLatencyMilliseconds":{"samples":3,"p50":752.898957999998,"p95":792.6262079999942,"max":792.6262079999942},"controlLatencyMilliseconds":{"samples":2512,"p50":0.6232079999972484,"p95":2.050875000000815,"max":37.22162499999831},"eventLoopDelayMilliseconds":{"samples":23,"p50":19.791871,"p95":29.868031,"max":31.686655},"peakProcessRssBytes":373620736,"outcomes":["success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success"]},{"size":100000,"comparator":"matches_prefix","filter":"sparse-indexed","mode":"ordinary","latencyMilliseconds":{"samples":20,"p50":20.834625000003143,"p95":21.558583000005456,"max":22.24358399999619},"loadedLatencyMilliseconds":{"samples":3,"p50":21.793375000008382,"p95":22.24966700001096,"max":22.24966700001096},"controlLatencyMilliseconds":{"samples":207,"p50":0.3658339999965392,"p95":0.7116660000028787,"max":1.314249999995809},"eventLoopDelayMilliseconds":{"samples":23,"p50":2.008063,"p95":2.285567,"max":2.365439},"peakProcessRssBytes":316866560,"outcomes":[400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400]},{"size":100000,"comparator":"matches_prefix","filter":"sparse-indexed","mode":"continuation","latencyMilliseconds":{"samples":20,"p50":63.84145899998839,"p95":65.90029199999117,"max":66.35162499999569},"loadedLatencyMilliseconds":{"samples":3,"p50":67.0638340000005,"p95":70.05449999999837,"max":70.05449999999837},"controlLatencyMilliseconds":{"samples":419,"p50":0.4275000000052387,"p95":0.9225839999999152,"max":9.037209000001894},"eventLoopDelayMilliseconds":{"samples":23,"p50":1.841151,"p95":2.283519,"max":2.531327},"peakProcessRssBytes":326434816,"outcomes":["success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success"]},{"size":100000,"comparator":"matches_prefix","filter":"none-indexed","mode":"ordinary","latencyMilliseconds":{"samples":20,"p50":20.92070899999817,"p95":21.566374999994878,"max":21.58208300000115},"loadedLatencyMilliseconds":{"samples":3,"p50":22.729541000007885,"p95":22.792415999996592,"max":22.792415999996592},"controlLatencyMilliseconds":{"samples":190,"p50":0.3230420000036247,"p95":0.7689580000005662,"max":9.769375000003492},"eventLoopDelayMilliseconds":{"samples":23,"p50":1.874943,"p95":2.168831,"max":2.248703},"peakProcessRssBytes":328155136,"outcomes":[400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400,400]},{"size":100000,"comparator":"matches_prefix","filter":"none-indexed","mode":"continuation","latencyMilliseconds":{"samples":20,"p50":636.5804590000043,"p95":707.7850839999883,"max":719.4046249999956},"loadedLatencyMilliseconds":{"samples":3,"p50":765.27283300001,"p95":819.2727079999895,"max":819.2727079999895},"controlLatencyMilliseconds":{"samples":2791,"p50":0.4592499999998836,"p95":1.6007079999981215,"max":50.27258299999812},"eventLoopDelayMilliseconds":{"samples":23,"p50":25.919487,"p95":31.965183,"max":38.764543},"peakProcessRssBytes":337575936,"outcomes":["success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success","success"]}],"workers-before":[{"size":100000,"comparator":"matches","route":"local","worker":1,"owner":1,"visibleLimit":5,"latencyMilliseconds":{"samples":20,"p50":13.49645800000144,"p95":15.59675000000425,"max":17.377166000005673},"eventLoopDelayMilliseconds":{"samples":20,"p50":1.589247,"p95":2.664447,"max":4.870143}},{"size":100000,"comparator":"matches","route":"remote","worker":2,"owner":1,"visibleLimit":5,"latencyMilliseconds":{"samples":20,"p50":16.529834000000847,"p95":17.963042000003043,"max":18.968290999997407},"eventLoopDelayMilliseconds":{"samples":20,"p50":1.954815,"p95":2.217983,"max":2.332671}},{"size":100000,"comparator":"matches_prefix","route":"local","worker":1,"owner":1,"visibleLimit":5,"latencyMilliseconds":{"samples":20,"p50":13.058499999999185,"p95":13.656749999994645,"max":14.086333000006562},"eventLoopDelayMilliseconds":{"samples":20,"p50":1.477631,"p95":1.773567,"max":2.026495}},{"size":100000,"comparator":"matches_prefix","route":"remote","worker":2,"owner":1,"visibleLimit":5,"latencyMilliseconds":{"samples":20,"p50":15.193874999997206,"p95":16.3023749999993,"max":17.707582999995793},"eventLoopDelayMilliseconds":{"samples":20,"p50":1.681407,"p95":2.012159,"max":2.040831}}],"workers-after-isolated":[{"size":100000,"comparator":"matches","route":"local","worker":1,"owner":1,"visibleLimit":5,"latencyMilliseconds":{"samples":20,"p50":12.199625000001106,"p95":13.502332999996725,"max":13.910666999996465},"eventLoopDelayMilliseconds":{"samples":20,"p50":1.333247,"p95":1.787903,"max":2.379775}},{"size":100000,"comparator":"matches","route":"remote","worker":2,"owner":1,"visibleLimit":5,"latencyMilliseconds":{"samples":20,"p50":13.41779200000019,"p95":15.177999999999884,"max":16.23041699999885},"eventLoopDelayMilliseconds":{"samples":20,"p50":1.672191,"p95":2.226175,"max":3.342335}},{"size":100000,"comparator":"matches_prefix","route":"local","worker":1,"owner":1,"visibleLimit":5,"latencyMilliseconds":{"samples":20,"p50":12.021749999999884,"p95":13.109749999995984,"max":13.376875000001746},"eventLoopDelayMilliseconds":{"samples":20,"p50":1.343487,"p95":1.808383,"max":1.822719}},{"size":100000,"comparator":"matches_prefix","route":"remote","worker":2,"owner":1,"visibleLimit":5,"latencyMilliseconds":{"samples":20,"p50":13.128875000002154,"p95":14.154958999999508,"max":15.782250000003842},"eventLoopDelayMilliseconds":{"samples":20,"p50":1.485823,"p95":2.416639,"max":2.437119}}]}

Comment generated by kAIle (GPT-6)

Final worker measurements for Add full-text and autocomplete snapshot continuation, after the mixed-width response-bound correction.

The source build is a6c42ad8a93af9a484e6266f4b55c03a81e17f33; subsequent changes are test/measurement documentation only. The two-worker harness returned identical expected rows for every exact/prefix and local/remote case over 100,000 records, with 20 samples per case. Owner-worker p95 was 12.13–13.50 ms; cross-worker p95 was 13.70–14.25 ms. This run was separate from test/build workloads, although read-only reviews were active. These warm local observations do not establish production capacity or isolate a single optimization’s benefit.

The earlier first-page measurements in this issue remain observations of the intermediate repair build identified by their fingerprints; they have not been relabeled as measurements of this final source.

Command: FT_BENCH_SIZES=100000 FT_BENCH_SAMPLES=20 node node_modules/mocha/bin/mocha.js benchmarks/fulltext-continuation-workers.cjs

Complete records and compiled-build fingerprints

{
  "sourceHead": "a6c42ad8a93af9a484e6266f4b55c03a81e17f33",
  "node": "v24.16.0",
  "samplesPerCase": 20,
  "records": [
    {
      "size": 100000,
      "comparator": "matches",
      "route": "local",
      "worker": 1,
      "owner": 1,
      "visibleLimit": 5,
      "latencyMilliseconds": {
        "samples": 20,
        "p50": 11.488167000003159,
        "p95": 13.503666999997222,
        "max": 14.376834000002418
      },
      "eventLoopDelayMilliseconds": {
        "samples": 20,
        "p50": 1.318911,
        "p95": 1.747967,
        "max": 2.093055
      }
    },
    {
      "size": 100000,
      "comparator": "matches",
      "route": "remote",
      "worker": 2,
      "owner": 1,
      "visibleLimit": 5,
      "latencyMilliseconds": {
        "samples": 20,
        "p50": 13.094791999996232,
        "p95": 14.251333000000159,
        "max": 14.26337500000227
      },
      "eventLoopDelayMilliseconds": {
        "samples": 20,
        "p50": 1.429503,
        "p95": 1.782783,
        "max": 2.069503
      }
    },
    {
      "size": 100000,
      "comparator": "matches_prefix",
      "route": "local",
      "worker": 1,
      "owner": 1,
      "visibleLimit": 5,
      "latencyMilliseconds": {
        "samples": 20,
        "p50": 11.389125000001513,
        "p95": 12.128959000001487,
        "max": 12.703667000001587
      },
      "eventLoopDelayMilliseconds": {
        "samples": 20,
        "p50": 1.282047,
        "p95": 1.686527,
        "max": 1.922047
      }
    },
    {
      "size": 100000,
      "comparator": "matches_prefix",
      "route": "remote",
      "worker": 2,
      "owner": 1,
      "visibleLimit": 5,
      "latencyMilliseconds": {
        "samples": 20,
        "p50": 12.826334000004863,
        "p95": 13.699124999999185,
        "max": 14.175749999994878
      },
      "eventLoopDelayMilliseconds": {
        "samples": 20,
        "p50": 1.400831,
        "p95": 1.731583,
        "max": 1.880063
      }
    }
  ],
  "compiledSha256": {
    "dist/resources/indexes/fullTextQueryProtocol.js": "8c21413e4d83e421812d7bec16ba4071a8ec41b1f103ef1fff8d47410e2b60fc",
    "dist/resources/indexes/fullTextContinuation.js": "8a1f87994a6e1ae35efb3ae1589336b88ce045005dba3b1ba6c1b5533e578c47",
    "dist/resources/indexes/fullTextContinuationOwner.js": "6d79ee4a7b04b9ea8bdb2dae256aa1241ee62b420f8c10fbc5deef3861446996",
    "dist/resources/indexes/fullTextRuntimeConfiguration.js": "5bdbf525bfabd5e370394d2329f4d55eecbe344b863de6c57a804e7370a534d6",
    "dist/resources/indexes/fullTextNativeBinding.js": "e7411bcf075155d1170b92266317024f85f7b1176c316547b75dee92e1760aa6"
  }
}

Comment generated by kAIle (GPT-6)

Worker measurements after reconciling current main and completing the response-boundary repairs for Add full-text and autocomplete snapshot continuation.

Measured source: 1fce79506556bff71ccc49f5da2c05a7f1a6f383, with main ea43e5677be264f943260eaa337f4b2ab7a0290c merged and its refreshed dependencies installed. The two-worker harness checked identical expected rows across all four exact/prefix and local/remote cases: 100,000 records, five visible rows, 20 samples per case.

Owner-worker p95 was 13.29–14.46 ms; cross-worker p95 was 15.62–16.76 ms. One remote prefix sample reached 38.23 ms. Tests and builds were stopped; read-only review processes remained active. This run does not establish an improvement over the earlier run or production capacity.

The earlier first-page measurements remain observations of their identified intermediate build. Native fulltext remains 0.6.0; no fulltext library changes were made for Kris’s feedback.

Command: FT_BENCH_SIZES=100000 FT_BENCH_SAMPLES=20 node node_modules/mocha/bin/mocha.js benchmarks/fulltext-continuation-workers.cjs

Complete records and compiled-build fingerprints

{
  "sourceHead": "1fce79506556bff71ccc49f5da2c05a7f1a6f383",
  "node": "v24.16.0",
  "fulltext": "0.6.0",
  "extended-iterable": "1.0.5",
  "samplesPerCase": 20,
  "records": [
    {
      "size": 100000,
      "comparator": "matches",
      "route": "local",
      "worker": 1,
      "owner": 1,
      "visibleLimit": 5,
      "latencyMilliseconds": {
        "samples": 20,
        "p50": 13.156457999997656,
        "p95": 14.463916000000609,
        "max": 14.509417000001122
      },
      "eventLoopDelayMilliseconds": {
        "samples": 20,
        "p50": 1.511423,
        "p95": 1.775615,
        "max": 2.211839
      }
    },
    {
      "size": 100000,
      "comparator": "matches",
      "route": "remote",
      "worker": 2,
      "owner": 1,
      "visibleLimit": 5,
      "latencyMilliseconds": {
        "samples": 20,
        "p50": 14.29854200000409,
        "p95": 15.624667000003683,
        "max": 15.715875000001688
      },
      "eventLoopDelayMilliseconds": {
        "samples": 20,
        "p50": 1.627135,
        "p95": 1.863679,
        "max": 1.955839
      }
    },
    {
      "size": 100000,
      "comparator": "matches_prefix",
      "route": "local",
      "worker": 1,
      "owner": 1,
      "visibleLimit": 5,
      "latencyMilliseconds": {
        "samples": 20,
        "p50": 12.487208000005921,
        "p95": 13.293541999999434,
        "max": 13.310749999996915
      },
      "eventLoopDelayMilliseconds": {
        "samples": 20,
        "p50": 1.360895,
        "p95": 2.195455,
        "max": 2.246655
      }
    },
    {
      "size": 100000,
      "comparator": "matches_prefix",
      "route": "remote",
      "worker": 2,
      "owner": 1,
      "visibleLimit": 5,
      "latencyMilliseconds": {
        "samples": 20,
        "p50": 14.083250000003318,
        "p95": 16.763542000000598,
        "max": 38.22812499999418
      },
      "eventLoopDelayMilliseconds": {
        "samples": 20,
        "p50": 1.572863,
        "p95": 2.158591,
        "max": 21.151743
      }
    }
  ],
  "compiledSha256": {
    "dist/resources/indexes/fullTextQueryProtocol.js": "f8aa7a268cc0bbc97b0d1eb71724bb0591a8ac9d44078c0bf995875f59245334",
    "dist/resources/indexes/fullTextContinuation.js": "8a1f87994a6e1ae35efb3ae1589336b88ce045005dba3b1ba6c1b5533e578c47",
    "dist/resources/indexes/fullTextContinuationOwner.js": "6d79ee4a7b04b9ea8bdb2dae256aa1241ee62b420f8c10fbc5deef3861446996",
    "dist/resources/indexes/fullTextRuntimeConfiguration.js": "aca4f7c4e37eebe9ac0cc0ee2a5fdb9cb79aea5a92d41e8ddfb736c5d89972eb",
    "dist/resources/indexes/fullTextNativeBinding.js": "e7411bcf075155d1170b92266317024f85f7b1176c316547b75dee92e1760aa6",
    "dist/server/REST.js": "187e00021bac2a692beb68b38851ead9047f271d1465e075012d5f2af1675e4e",
    "dist/server/threads/socketRouter.js": "13de1895814c35d21ed960fedca4ab35f3ddaf6ea333ae8a3ddac394fec54597"
  }
}

Comment generated by kAIle (GPT-6)

Library implementation is available in draft Expose fuzzy matching controls and align Rust request validation. Future Harper adoption can expose optional fuzzyDistance (0–2, default 1) and fuzzyTranspositions (default true) on fuzzy/fuzzy-prefix searches and Boolean leaves. Existing callers keep their defaults. Facade and addon must upgrade together (ABI 13/query API 7/packed protocol 8); this does not change persisted index identity. No consumer pin or release version is selected here.

Phrase slop is deferred until an official Tantivy release contains the upstream fix; no unreleased native patch is included. Distance-two admission is an estimate rather than a hard memory bound, and broad fuzzy-prefix work remains preview. Failed qualification measurements and resource follow-ups are preserved in Qualify remaining native metadata collector resource tradeoffs.

Adoption refinement: distance 2 is explicitly preview. Keep the default distance 1; qualify the production corpus and concurrency before exposing caller-supplied distance 2. Distance-zero queries now reuse the existing exact-bonus path without fuzzy automata or fuzzy construction charges, preserving the existing eligibility and clause policy. Current-source trace spans can still differ from indexed stem membership; the new runnin/running regression pins that contract.

Harper adoption acceptance

These requirements remain unimplemented and unqualified in Harper:

Leaf-only controls: preserve fuzzyDistance and fuzzyTranspositions through Table.search/REST normalization on supported fuzzy/fuzzy-prefix requests and individual Boolean text leaves. Reject supplied controls on AND/OR/NOT groups and reject a root query combined with simple leaf controls. Never inherit group controls or silently discard them; explicit undefined remains absent.
Distinguishing integration tests: use independently configured fuzzy Boolean leaves with different distances/transposition policies and selected fields. Include nonmatching decoys that would match if either options or field restrictions were ignored. Cover misplaced group controls, root-query/leaf-option conflicts, unchanged defaults, invalid values and recovery after rejection.
Highlight parity: verify UTF-16 highlight spans with the same resolved controls on supported highlighting paths. Preserve existing rejection of unsupported highlighting/query combinations; this checklist does not introduce Boolean tracing support.

Wrapper behavior and regression examples are in the reviewed fuzzy-control implementation. Harper tests must exercise the adopted published package through Harper's own paths.

Comment generated by kAIle (GPT-6)

Review dispositions from Allow stored-field sorting and exact counts for full-text search:

The native fixture’s preexisting any typing remains a maintainability cleanup; this review establishes no new product defect.
The shared HTTP helper rejects client aborts and deadlines, while server-side truncation resolves with aborted: true. This is the accepted current contract; no contract change is planned.

Review-Deferral: 678020879f84af21 severity=nit category=maintainability

Review-Deferral: c82d6feaa1490269 severity=nit category=correctness

Comment generated by kAIle (GPT-6)

Two documented benchmark-tooling limits remain accepted while standardizing Allow stored-field sorting and exact counts for full-text search:

Review-Deferral: 971654163365d792 severity=nit category=security

Failure reports can include local filesystem paths in stack traces. Generated reports stay private and outside committed source; sharing a report still requires reviewing and sanitizing its contents. This tracks a future report-sanitization improvement, without publishing any reports here.

Review-Deferral: 33d88b8514e36740 severity=nit category=reliability

The native cost runner has a fixed five-minute safety timeout, including setup. Large explicit sample/warmup requests can hit that bound and fail without a completed report. The method documents that contract; quick/nightly defaults remain bounded smoke/measurement configurations. A future change may validate the requested work budget or make the cap configurable.

Canonical tracker: Improve full-text benchmark report hygiene and workload limits.

Comment generated by kAIle (GPT-6)

Qualify native full-text filtering and catalog-scale operation for v5.4 #2512

Description

Fulltext 0.8.0 adoption and phase-two boundary — 30 October 2026

Current work — consolidated 29 October 2026

Remaining acceptance

Configurable query capacity: Harper adoption

Native candidate-ID filtering: evaluation and adoption

Consolidated trackers and boundaries

Current qualification scope

Metadata sorting, facets and predicates adoption

Outcome

Release boundary

Qualification matrix

Acceptance criteria

Deliverables

Related work

Transferred comparative qualification

Ownership reconciliation

Deferred aggregate admission qualification

Deferred continuation filtering work

Deferred limits-resolution work

Raw-hit default calibration

Observed pre-PR calibration, 19 October 2026

Tantivy 0.27 relevance comparison — observed 27 October 2026

Results

Scope and reproducibility

Remaining relevance acceptance criteria

Durable wrapper examples and regression subset

Fulltext 0.7.0 adoption performance follow-up

Adoption safety checks and remaining workload gate — 28 October 2026

Bun 1.3.14: full-text upgrade and worker-restart qualification

Upgrade probe review disposition

Test-hardening CI completion

Readiness follow-up qualification complete

Facet API ownership

Stored-field ordering: retained memory and cancellation qualification

Ordering review: capacity contract and cancellation fix

Local qualification evidence — 2026-11-01

Accepted bounded cancellation residual

Accepted exact-count drain cost

Accepted offset eligibility cost

Optional scoring admission: clause count × field width

Accepted compatibility and timeout contracts to retain in qualification

Activity

kylebernhardy commented on Oct 20, 2026

kylebernhardy commented on Oct 20, 2026

kylebernhardy commented on Oct 20, 2026

kylebernhardy commented on Oct 21, 2026

kylebernhardy commented on Oct 30, 2026

Harper adoption acceptance

kylebernhardy commented on Nov 2, 2026

kylebernhardy commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions