[kylebernhardy]

⊙ Problem

Tantivy 0.27 changes English stemming. Existing indexes require rebuilding; Harper records remain authoritative.

🤔 Adopt before workload qualification?

Target v5.4 with executable upgrade checks and explicit rollout gates.

💡 Solution

Pin 0.7.0 for v5.4. Qualify upgrades from the fulltext 0.5.0 engine shipped with Harper 5.3.4 through existing ownership, retirement, scan and replay. Main previously pinned 0.6.0.

⚖️ Alternatives

Reuse existing coordination and native lifecycle APIs. Another rebuild coordinator or file surgery would duplicate ownership. Direct downgrade remains unsupported; recovery restores the current engine with traffic drained.

🔧 Changes

Product and architecture tour

Rebuild before serving the new analyzer

Before	After
Older tokenized checkpoints.	Identity mismatch triggers source rebuild; coverage gates search.

Existing ownership across the upgrade

flowchart LR
    Old[Old checkpoint] --> Refuse[Identity mismatch]
    Refuse --> Drain[Drain readers]
    Drain --> Reset[Native reset]
    Reset --> Replay[Scan and replay]
    Replay --> Ready[Coverage-gated search]

Loading

File	Review focus
.github/workflows/unit-test.yml	Existing native matrix; no added jobs.
dependencies.md	Runtime and historical fixture rationale.
integrationTests/resources/full-text-search.test.ts	Two-worker deployed upgrade and recovery.
package-lock.json	Published current and dev-only historical artifacts.
package.json	Exact current pin; historical alias.
renovate.json	Freeze the migration fixture.
resources/FULL_TEXT.md	Versions, upgrade procedure and recovery limits.
resources/indexes/DESIGN.md	Compatibility ownership and source citations.
unitTests/resources/fixtures/fullTextUpgradeCrashProcess.cjs	Real crash sentinels and completed setup.
unitTests/resources/fixtures/fullTextUpgradeHelpers.cjs	Shared old-engine process isolation.
unitTests/resources/fixtures/fullTextUpgradeProcess.cjs	Real historical seed and refusal.
unitTests/resources/fixtures/fullTextUpgradeWorker.cjs	Managed worker and held native leases.
unitTests/resources/nativeFullTextDerivedIndexLifecycle.test.js	Compatibility, reset, scan and replay.
unitTests/resources/nativeFullTextSchemaUpgrade.test.js	Public admission, source replay and write retry.
unitTests/resources/nativeFullTextUpgradeCrash.test.js	Reset, partial/full publication SIGKILL recovery.
unitTests/resources/nativeFullTextUpgradeWorkers.test.js	Close/reset order across workers; no transient idle assertion.
unitTests/resources/permissions.test.js	Baseline lint fix; unchanged getter behavior.
unitTests/windowsGate.mjs	Include the new native suites.

✅ Verification

• Native 44 passing; built REST/examples 22 passing.
• Broad resources: RocksDB 5,320 passing / 72 pending; LMDB 4,092 / 882. Broad runs preceded final direct fixture fixes, covered by focused tests. Build, lint, formatting, design and partition checks pass.
• Earlier main: 8,307 passing / 221 pending / 46 failing; 40 failure entries reproduce on the exact base, six pass isolated. Earlier integration: 2,519 / 75 failures / 206 cancelled, not all classified. Final-head platform CI passed; uWS numeric-index failures passed a targeted rerun.
• Controlled tests cover public admission, concurrent replay, retryable write rejection, responsive/unresponsive worker leases, failed-close containment, owner loss and process crashes. Streaming REST can return HTTP 200 with a terminal error record; clients must inspect bodies.

Remaining qualification

Implement full-text generation lifecycle, rebuild, eviction, backup, and status retains multi-node, storage durability and downgrade gaps. Qualify native full-text filtering and catalog-scale operation for v5.4 retains matched Harper performance and capacity. Existing long upgrade/baseline measurements cover 100,000 documents and 10,000 mixed queries on matching Node 24.21: indexing 82,263→85,729 documents/s, mixed p99 5.576→5.170ms, fuzzy-prefix p99 3.361→3.639ms (+8.3%). Separate shared runners limit attribution. No Harper parity or reproducible regression claim.

Claude/Composer reviewed the ordering fix. Final direct sibling fixes passed locally and in CI.

PR QA: skipped @ 5006a0f — dispatch unavailable (CLI not installed).

Comment generated by kAIle (GPT-6)

Release target: v5.4 — Kyle's requested adoption release; no v5.3 backport.

Related PRs: #691 overlaps (Document native full-text search, upgrade and recovery guide), #739 overlaps (Document full-text paging and configurable search limits, v5.4 reference and release notes)

The backpressure receipt entry below is resolved in code; its linked issue records passing rejection/recovery checks.

Deferred findings

• a605adc86ce788c0 minor performance — package.json — Fulltext 0.7.0 swap measured; fuzzy-prefix p99 tail regression and catalog-scale qualification owed — filed Qualify native full-text filtering and catalog-scale operation for v5.4 #2512
• b1b0f6aa38552f15 nit maintainability — integrationTests/resources/full-text-search.test.ts — Integration suite imports a unit-test fixture directly — filed Implement full-text generation lifecycle, rebuild, eviction, backup, and status #2510
• fba076e9a15861b3 none test-coverage — resources/FULL_TEXT.md — No test asserts write rejection during the post-rebuild lag window — filed Implement full-text generation lifecycle, rebuild, eviction, backup, and status #2510

_{Review-Coverage: authored=codex; ran=cursor-composer,claude; adjudicated=domain; declined=gemini,cursor-grok,muse; rounds=8; full=5 @ 5006a0f}

_{Review-Attention: read ~7m (sensitive: package-lock.json; decisions: cross-thread-sequence-counters, real-previous-engine-fixture, forced-rebuild-on-upgrade) @ 5006a0f}

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​harperfast/​fulltext@​0.6.0 ⏵ 0.7.0			+1	+1

View full report

[gemini-code-assist]

Code Review

This pull request upgrades the @harperfast/fulltext optional dependency from version 0.6.0 to 0.7.0 (embedding Tantivy 0.27.0) and introduces documentation, tests, and a migration path for upgrading existing indexes. It also adds @harperfast/fulltext-previous as a dev dependency to test index migration from 0.6.0 to 0.7.0 using a child-process fixture. Feedback on the changes highlights an issue in the test fixture where a native reader's close method is called with an unsupported rollback option in the finally block, which should be resolved by closing the reader immediately after opening it.

[kriszyp]

_{🤖 Reviewed with Muse}