Implement full-text generation lifecycle, rebuild, eviction, backup, and status #2510

Outcome

Operate each native Tantivy index as local derived state through Harper's shared derived-index runtime and database lifecycle.

Implemented in Complete native full-text indexing and search #2855

• Use the elected transaction-log runner and resolve current committed primary projections.
• Store the exact Harper cursor and native data revision in the Tantivy commit payload.
• Reuse compatible files and replay after restart; rebuild missing, corrupt, incompatible, or source-incompatible generations before serving queries.
• Build indexes independently on source nodes and replicas from their local authoritative Harper data. Native files and source-node cursors are never replicated.
• Coordinate writer handoff, generation replacement, and reader pause/resume across workers. Pauses are readiness- and epoch-scoped so a replacement owner can recover from a crashed owner.
• Publish readiness and bounded coverage through the existing derived-index mechanisms. Missing or rebuilding indexes fail explicitly.
• Keep native paths internal and retire files through the database/index lifecycle.
• Preserve a committed source transaction when the derived index fails; existing lag and admission behavior remains authoritative.

Remaining operational hardening

• Broader replica bootstrap, restore, backup, and database-incarnation integration tests.
• Disk-headroom, directory ownership/permissions, canonical containment, and encryption prerequisite enforcement beyond the wrapper's safe path contract.
• Operator metrics for queue depth, rebuild progress, lag, unindexable records, and cleanup retries.
• Crash tests covering owner death at each handoff and publication boundary.
• TTL and whole-record cache-eviction integration once those Harper lifecycle hooks are available to the shared derived-index protocol.

Boundary

Harper owns source records, schema identity, replay, election, readiness, authorization, and record materialization. @harperfast/fulltext owns Tantivy files, native writer/reader behavior, mutation validation, publication, and search. There is no RocksDB full-text storage provider.

Transferred integration acceptance criteria

Preserve the still-outstanding product proof from the former fulltext trackers:

• Exercise real committed writes through elected runner, String/array/Blob projection, native mutation, payload-bearing commit, reader reload, and query in reproducible integration fixtures.
• Inject source-commit-before-apply, apply-before-native-commit, native-commit-before-reader-reload, and owner-loss failures; verify IDs and versions through upsert/delete/recreate after recovery.
• Prove worker handoff/quiescence and cross-worker visibility, including publisher death before a refresh notification and recovery of delayed Blob projections without advancing readiness past missing content.
• Verify replicated writes, initial copy/catch-up, source restore, backup/database incarnation changes, eviction, and missing/corrupt/incompatible or source-log-gap rebuild before readiness.
• Automate the product integration slice against an exact published fulltext package pin in Harper CI; distinguish process-crash coverage from untested power-loss/storage durability paths.
• Document Harper deployment, restart/rebuild/restore, disk/merge headroom, directory permissions/encryption, readiness/lag, and recovery procedures in Harper-owned documentation.

Existing focused implementation tests do not satisfy this broader operational matrix by themselves. Product performance qualification remains in Qualify native full-text filtering and catalog-scale operation for v5.4.

Ownership reconciliation

Remaining Harper-specific acceptance criteria formerly tracked in fulltext now live in Implement full-text generation lifecycle, rebuild, eviction, backup, and status and Qualify native full-text filtering and catalog-scale operation for v5.4. Wrapper closure records are Deliver the standalone native full-text wrapper, Epic: Qualify, document, package, and release @harperfast/fulltext, Epic: Implement native checkpoint persistence and handle lifecycle, and Implement and qualify native checkpoint publication. Closing those records acknowledges completed wrapper delivery and transfers outstanding product qualification; it does not claim that Harper qualification is complete.

Observability acceptance criteria

Opt-in telemetry collection: integration contract

Consume Expose fulltext operation telemetry and latency histograms #81 after a compatible wrapper release; update Harper's package/binding/capability contract together. The wrapper PR does not change Harper's dependency or enable collection here.

• Release and ABI compatibility: the current wrapper contract requires native ABI 14 and telemetry API 1; published fulltext 0.8.0 uses ABI 13 and does not provide this contract. Adopt a compatible published facade and matching platform binaries together, updating Harper's exact dependency, lockfile, binding types, runtime capability validation and writer/reader compatibility tests. If the selected release advances the ABI again, validate that release's actual contract rather than hard-coding this proposal's ABI. Preserve other used ABI/query/lifecycle capabilities. Reject mismatched facade/native packages and missing/unsupported telemetry capabilities explicitly; do not silently call internal native exports or bypass version checks. Version selection remains release work.
• Public interface mapping: expose the selected wrapper's public writer/reader telemetry() and configureTelemetry() interfaces through Harper's binding, including unavailable snapshots and both optional flags. Use the wrapper definitions for native admission/queue/execution/outcome metrics, facade batches and preparation rejections, reader observations and final closed-state metrics. Do not sum overlapping native frames/logical batches or legacy status/telemetry durations. Basic accounting is not controlled by Harper's collection setting.
• Default off: when neither an operator nor a registered consumer enables collection, schedule no wrapper telemetry() calls and export no telemetry. Use implied defaults; do not add an explicit default block to shipped configuration. This does not disable basic wrapper operation counters/timing totals or eliminate handle-initialization cost.
• Dynamic caller control: use Harper's existing metrics/configuration lifecycle to start, pause and resume collection on already-open writer/reader handles. No restart, reopen, rebuild or query-policy change is needed. Disabling stops new scheduled calls; an already-running synchronous call completes. Stop collection on consumer removal and shutdown, and prevent stale callbacks after generation replacement or owner handoff.
• Independent expensive options: histogram recording and snapshot-byte observation remain separately opt-in and disabled by default. Collection can run without either option. Stopping polling alone does not turn off an already-enabled recording option; explicitly disable unwanted options through configureTelemetry(). A resumed snapshot retains cumulative counters, so rate calculations use the actual elapsed interval and missing samples stay unavailable rather than zero.
• Ownership and scheduling: stagger collection across bounded eligible handles, reuse existing Harper metrics/export infrastructure, and define elected writer versus worker-local reader aggregation without double counting. Keep source lag, freshness, permissions and end-to-end request attribution Harper-owned; do not reimplement native queues or Tantivy internals.
• Export semantics and failure containment: preserve bigint counter precision and units during serialization; handle native unsigned-64-bit wrap, handle/process replacement and longer collection gaps without negative rates. Convert noncumulative histogram buckets only if the selected exporter requires cumulative ones. Live relaxed snapshots need not conserve across fields; assert conservation only after operations settle. Keep unavailable, stale and pending observations distinct from zero; preserve age/error fields. Catch collector/export failures at Harper's metrics boundary, emit bounded diagnostics and leave search/write/admission policy unchanged. Keep telemetry-only configuration failures distinct from an incompatible native package.
• Reader limits and privacy: generation IDs and opstamps are not source freshness timestamps or cross-node cursors. Snapshot bytes are reader-referenced file sizes, not directory totals or per-index RSS; enabling byte observations schedules no work until a successful publish/reload opportunity, and disabling preserves cached values. Do not invent unsupported automatic-merge or active-cancellation metrics. Bound index/table labels and collector retention; never export query text, record IDs, field values, users, raw cursors, file paths or unbounded error messages as labels. Use the existing protected metrics surface and its access controls rather than adding an unauthenticated endpoint.
• Executable acceptance: test absent/default configuration (zero scheduled calls/exports), enable → disable → re-enable on the same handles, cumulative counters across the gap, independent option toggles, consumer removal/shutdown, unavailable snapshots and owner/generation changes. Verify searches/writes continue while uncollected and that disabled collection creates no recurring telemetry timer/work.
• Compatibility and fault tests: on Harper’s supported Node/platform CI matrix, exercise matching and mismatched facade/addon versions, missing/unsupported telemetry API, binding round trips with values above JavaScript's safe integer range, unavailable/failed snapshots and export failures. Cover per-worker reader and elected-writer aggregation without dropping distinct handles, histogram freeze/resume, byte observation pending/stale/error states, rate baselines after counter reset/replacement, final snapshots after close and bounded collector cleanup. Verify no post-disable scheduled scrape, while allowing a synchronous call already in progress to finish.
• Harper qualification: measure no polling versus the selected staggered collection cadence under matched concurrent search/indexing with multiple handles/workers, including exporter work. The wrapper's same-build local measurement validated 28 million timed queries: median polling throughput cost 0.82%/1.24% at four/sixteen indexes, with a worst paired loss of 5.75%. Basic recording was active in both profiles. These synthetic macOS observations neither choose Harper's interval nor qualify production Linux or sustained memory.

This is tracked future Harper implementation, not delivered runtime wiring. Existing operational, capacity and rollout acceptance remains open.

Comment generated by kAIle (GPT-6)

Extend existing Harper metrics and shared derived-index readiness/lag mechanisms. Consume wrapper telemetry from Expose low-overhead fulltext observability and operation latency metrics; do not reproduce native queues, latency accounting or Tantivy internals in Harper.

• Expose per-node durable indexing lag and search-visible freshness separately. Define each against the authoritative local source-log position and relevant published/reader-visible checkpoint; an opstamp is not a time-based lag metric.
• Distinguish unknown/unavailable freshness from zero lag. Report observation age and avoid comparing unrelated nodes' local cursor values.
• Export bounded readiness states and rebuild/replay progress, duration, last successful publication age and bounded failure reasons. Use an indeterminate progress state when no reliable denominator exists.
• Track lag-based write rejection and search-unavailable events without changing existing admission or readiness policy.
• Track unindexable records, projection retries and generation/file-cleanup retries with bounded reason categories, without exposing record contents or identifiers.
• Attribute end-to-end Table.search/REST timing to native queue wait/execution, candidate processing, authorization/filtering and record hydration where measurable. Count candidates and records hydrated; keep rejected/timed-out requests visible alongside successful ones.
• Use existing process/host monitoring for CPU, RSS, page faults, disk latency and free disk. Do not duplicate exporters or invent precise per-index resident-memory attribution.
• Prevent multi-worker duplication: define elected-owner versus per-reader/per-worker metrics and aggregate counters without double counting. Separate each node's local indexing state from replication health.
• Bound table/index labels and series retention through creation, generation replacement, eviction and deletion. Never label metrics with queries, record IDs, users, cursor values or file paths.
• Test update/delete/recreate, owner handoff, replay/rebuild, publication-versus-reader visibility, failure, cleanup and metric reset semantics. Missing samples must not produce false healthy/zero-lag readings.
• Compare collection stopped/enabled and independent histogram/byte options, including scrape frequency, under concurrent indexing/search and multiple indexes. Basic wrapper accounting remains active in both collection profiles. Measure throughput, p95/p99 latency, CPU and memory overhead before choosing a collection cadence.
• Document operator-facing definitions, limits, troubleshooting examples and the boundary between native metrics and Harper freshness/readiness. Documentation examples must be executable tests where applicable.

Initial priority within this scope: freshness/readiness, queue pressure surfaced from the wrapper, error counters and end-to-end latency breakdown. Detailed segment/merge diagnostics stay wrapper-owned. Dashboard and alert thresholds require qualification evidence; this addition does not claim completed scale qualification or commit a new release.

Comment generated by kAIle (GPT-5)

Deferred indexing-coverage visibility

Follow-up from Support full-text and autocomplete continuation in Table.search and REST: processed coverage and exact native match totals do not prove every source record is represented.

• Connect backend native rejected-upsert accounting to operator-facing health. Source inspection found backend rejection accounting separate from the shared derived-runtime counter; verify the complete path before claiming coverage.
• Test oversized/invalid source replacements that remove old derived content, encoded primary-key overflow, correction/replay/rebuild, restart and owner handoff. Source records remain authoritative.
• Distinguish cumulative rejection events from currently omitted unique records, with explicit reset and recovery semantics. Avoid high-cardinality IDs/content in metrics.

Deep-result and autocomplete completeness qualification depends on Support full-text and autocomplete continuation in Table.search and REST and a compatible published native package. Do not certify those result paths before the integration lands; unrelated lifecycle/coverage work can proceed independently.

Comment generated by kAIle (GPT-6)

Fulltext 0.7.0 adoption gate — verified 27 October 2026

Keep this issue open for Harper adoption of Upgrade fulltext to Tantivy 0.27 with safe index rebuilds. The wrapper PR does not change Harper's 0.6.0 pin. This section records migration evidence and remaining requirements, not production rollout approval.

What already works

Verified by tracing current Harper main and running focused tests on macOS arm64 / Node 24.16:

• Inspection treats incompatible identity as no trusted checkpoint. The shared runtime automatically starts a rebuild when it has no valid durable cursor. Writer-open and query failures also recognize E_IDENTITY_MISMATCH; query failure requests a rebuild and retires readers.
• Rebuild ordering persists condemnation before reset, publishes rebuilding, quiesces the backend, resets under an owner epoch, scans authoritative records and replays subsequent log entries. Readiness returns only after durable coverage. Harper lifecycle hooks drain local and peer query readers before wrapper-owned retirement; failure to prove quiescence does not authorize deleting the directory.
• Existing backend, lifecycle, query, shared-runtime, activation and RocksDB suites: 282 passing, four conditional tests pending. These include simulated failures and native 0.6.0 cases; they are not a complete 0.7.0 rollout matrix.
• A separate real migration probe used published fulltext 0.6.0 / Tantivy 0.26.2 and the locally built 0.7.0 / Tantivy 0.27.0 candidate from the linked wrapper PR. With unchanged logical identity/source generation, inspection and open rejected the old identity. Harper's existing runtime automatically retired/rebuilt, reached ready with a new checkpoint, found the authoritative added/waterproof records with versions, omitted stale indexed content, and left the two-entry source audit log unchanged. One passing probe. Coordination hooks were observed in one process; this did not exercise peer IPC or process-crash recovery. The old seeded payload was deliberately stale, not a valid prior Harper checkpoint.

Changes and qualification required before adoption

• After the 0.7.0 release exists, update Harper's exact optional dependency and lockfile together. Update the published-package test's hardcoded Tantivy 0.26.2 expectation to 0.27.0 and refresh Harper's full-text version/upgrade documentation. Preserve ABI/capability validation; no new rebuild routing is currently indicated by this audit.
• Commit a durable cross-version regression using an actual 0.6.0 index with a valid Harper checkpoint and the same declared generation. Assert mismatch detection, no trust in the prior checkpoint, source scan plus concurrent update/delete replay, new-analyzer search results/versions and no source-log amplification. Cover populated and empty tables.
• Exercise actual multi-worker reader draining during migration, held/in-flight readers, close/coordination/reset failures, owner handoff and restart/crash between condemnation, retirement, scan and publication. Keep queries unavailable while rebuilding; never reset when quiescence is unproven or report ready from a partial rebuild.
• Document and qualify the deployment sequence: drain old workers/readers before activating the new analyzer, reserve disk/time for a full per-node rebuild, and verify catch-up/readiness before restoring full-text traffic. Do not assume old/new workers can safely alternate ownership of the same index during a rolling deployment. Include retryable write-503 windows after long scans, catch-up checks and client retry behavior.
• Qualify a complete operator downgrade route before supporting it. Native reset alone does not prevent another worker or node recreating an index. Direct downgrade is unsupported; the currently documented recovery restores the 0.7-based Harper version with traffic drained.
• Run Harper's applicable full unit/integration and supported-platform packaged-addon gates with the released 0.7.0 artifacts. The local candidate probe and wrapper benchmarks do not replace Harper rollout qualification.

Related qualification: Qualify native full-text filtering and catalog-scale operation for v5.4; wrapper evidence: Qualify Tantivy 0.27 upgrade and enforce the analyzer rebuild boundary. Existing priority, effort, type, parent and milestone remain unchanged; this is additional acceptance scope for the existing lifecycle issue.

Comment generated by kAIle (GPT-6)

v5.4 upgrade-safety qualification — 28 October 2026

Implementation: Upgrade full-text search to Tantivy 0.27. Keep this issue open. These checks establish bounded upgrade behavior, not production rollout approval. Earlier qualification commit is superseded by the final qualification head linked below.

The historical fixture now uses fulltext 0.5.0, shipped in Harper 5.3.4, rather than main's intermediate 0.6.0 pin. It writes real old native files with current Harper source/checkpoint fixtures; it does not boot an older Harper installation. Both use Tantivy 0.26.2. The exact dev-only alias is excluded from Renovate; ordinary consumers install only the current engine. Prior 0.6 evidence above remains historical.

Implemented acceptance checks

• Load a real schema-declared index with a genuine old checkpoint, matching generation and Harper cursor. Exercise production scanning, source resolution and replay. Assert current versions and analyzer results; no native-only ghost, deleted record or source-log amplification.
• Fence public Table.search and REST during rebuild, then return current results after coverage catches up. Ordinary JSON streaming can return HTTP 200 with a terminal error element after headers; buffered count requests return 503. Callers must inspect response bodies as well as status.
• Exercise real native write-lag backpressure: a held publication causes retryable DERIVED_INDEX_LAGGING/503, with no rejected source or audit mutation. After catch-up the same write succeeds. This checks correctness, not a production rejection-duration bound.
• Restart the built Harper distribution with two HTTP workers over an offline-seeded old index; verify readiness, current analyzer results, ghost/deleted absence and subsequent writes. Restore the current engine after old-engine inspection/open/reset refusal and verify search recovery.
• Exercise actual managed-worker ITC pause with both an unresponsive public query and a responsive asynchronous internal candidate-key lease. No reset occurs while drain is unproven. Release permits native close before reset. A failed close is contained by the native lock; the fixture explicitly releases the failed handle before retry, so automatic recovery from a permanently failed close is not claimed.
• Terminate an elected worker after actual reset; verify successor ownership and rebuild. SIGKILL separate processes after reset, partial publication and full scan-boundary publication before ready; verify condemnation, replay/rebuild, final IDs/versions and unchanged authoritative records/audit.
• Document stop-all-old-processes, backup/headroom, per-index readiness and catch-up checks, response-body errors, bounded write retry and restoring the current engine. Direct downgrade and mixed old/new ownership remain unsupported.

Validation and remaining release gate

Observed locally: released-engine native tests 44 passing and built REST/documented examples 22 passing. Broad resources pass: RocksDB 5,320 / 72 pending; LMDB 4,092 / 882 pending. Broad runs preceded the final direct fixture fixes, which have focused native/integration coverage. Build, lint, formatting, partition and design checks pass. Final-head Linux, Windows and macOS native CI and the full unit/integration gates pass; see the final qualification evidence below.

Earlier main results were 8,307 passing, 221 pending and 46 failing: 40 failure entries reproduce on the exact base; six entries pass in isolation. Earlier broad integration had 2,519 passing / 75 failing / 206 cancelled; failures are not all classified; the built full-text suites and actual prior-head Linux/Windows integration CI pass. Do not label every broad local failure baseline.

Still open: a complete older-Harper-installation/database upgrade, multi-node/replica bootstrap and rolling-deployment safety, storage/power-loss durability, a supported operator downgrade route, long concurrent-write rebuild duration, disk/permission/encryption prerequisites, and customer-scale performance. The tests use real native files with existing Harper ownership; no replacement rebuild coordinator was added. Workload qualification stays in Qualify native full-text filtering and catalog-scale operation for v5.4.

Existing priority, effort, type, parent, milestone and assignment are preserved.

Comment generated by kAIle (GPT-6)

Final review tracking

• Historical write-backpressure finding is completed by the schema regression: retryable rejection, no source/audit mutation, catch-up and successful retry. The review receipt retained a resolved severity-none marker; it is not an outstanding rollout gap.

Review-Deferral: fba076e9a15861b3 severity=none category=test-coverage

• If the test layout is reorganized, preserve the shared historical-engine fixture contract between unit and integration suites. The current direct dev-only fixture import is intentional reuse; a one-line re-export adds no current safety benefit. No production API depends on this test layout.

Review-Deferral: b1b0f6aa38552f15 severity=nit category=maintainability

• Current durable CI targets the engine shipped in Harper 5.3.4 (fulltext 0.5.0). Real 0.6 → 0.7 cases passed earlier in this task before retargeting the single alias; that evidence is historical, not retained as a second binary in current CI. If adopting from a released Harper version that ships 0.6, requalify that exact starting installation.

Comment generated by kAIle (GPT-6)

macOS qualification follow-up — 28 October 2026

The failed macOS native job on the earlier qualification head passed 43 native tests and failed the new responsive-worker case. Linux and Windows native 44 passed. The failed assertion incorrectly required the shared retirement fence to become globally idle while the test deliberately held a peer's active query. A different worker can legitimately retain that fence while draining the query; this does not establish a production reset/rebuild defect.

The worker qualification fix removes that invalid pre-release idle condition, aligns fixture budgets and records close invocation, lease release, close completion, reset and native-lock refusal across both worker bindings. Existing native containment and eventual retirement checks remain. No production runtime source changed.

Observed validation: exact five-file native CI slice 44 passing, three fresh responsive-case repeats passed, and an actual early-close negative control failed the safety assertion. The temporarily modified compiled artifact was restored byte-for-byte. Claude and Composer reviewed the ordering fix; final direct sibling-counter adjustments passed the native slice and independent local review, without another outside pass under the narrow final-artifact exception.

Final-head native CI passes on Linux, Windows and macOS. The original macOS failure and its fixture fix remain documented above; broader rollout limits remain open.

Comment generated by kAIle (GPT-6)

Final CI qualification — 28 October 2026

Observed on the final qualification head: Unit Test passes, including all three native platforms; Integration Tests passes after one targeted uWS shard rerun. Package-consumer, downstream Next.js, smoke, lint and formatting checks also pass.

The first uWS shard attempt passed 465 tests and failed two unchanged RocksDB numeric-index type-widening checks: a range query returned 503, and index entries differed after restart. The single diagnostic rerun passed 467, failed zero, with one todo, including both previously failing checks. The fixture uses an ordinary numeric secondary index; it declares no full-text index. The cause remains unclassified: a passing rerun does not establish a baseline defect or prove the failure unrelated to adoption. Original logs are retained. No test or runtime change was made for this rerun.

This settles the bounded PR CI gate, not the broader production rollout gaps above. Keep this issue open.

Comment generated by kAIle (GPT-6)

Harper 5.3.0 / fulltext 0.3.0 qualification gap

Published npm dependency metadata distinguishes the historical starting points: Harper 5.3.0 pins @harperfast/fulltext 0.3.0; Harper 5.3.1–5.3.4 pin 0.5.0. The current adoption regressions use genuine 0.5.0 native files with current Harper source/checkpoint fixtures. They do not qualify a complete Harper 5.3.0 installation/database upgrade or establish 0.3.0 compatibility.

• Qualify the exact published Harper 5.3.0 / fulltext 0.3.0 starting installation, including its supported declarations, source/checkpoint identity and native files, before adding it to the supported upgrade procedure. Verify source-record preservation, rejection or reuse of old derived state as appropriate, rebuild/replay, and query readiness using the actual starting contract.
• Record which earlier beta declarations/index formats are unsupported and distinguish them from any 0.3.0 indexes using current fields. The dependency pin alone does not prove that every such 0.3.0 index is unsafe or incompatible.
• Keep the operator documentation explicit that fulltext 0.3.0 is outside the currently qualified procedure until this evidence exists; do not generalize the 0.5.0 fixture results to every Harper 5.3 release.

Documentation owners: Document native full-text search and Document full-text paging and configurable search limits. This is a remaining qualification gap under the existing lifecycle scope, not a newly demonstrated runtime defect. Existing priority, effort, type, parent, milestone and assignment are preserved.

Comment generated by kAIle (GPT-6)

Follow-up: crash-test observations and upgrade probes

Harden full-text upgrade crash and readiness tests, head 415f7b7925464cda72ffd6fcdc621370cd76200d, addresses the adoption review's Windows ordinary-exit false positive, moves the three upgrade suites into the existing index shard with measured caps, preserves timeout stderr and adapts probes to the current terminated-stream contract. Every completed upgrade page must still contain the unchanged pre-rebuild witness. No Harper runtime, wrapper or Tantivy source changed.

Local full resources: 5,590 passing/220 pending; source-only index shard: 729/7. Final focused schema/JSON stream cases: 21 passing. Deployed full-text cases: 15 passing on Node HTTP and 15 on the verified uWS backend. Two full Claude/Composer review passes completed; the direct final requested test fixes were independently read and locally tested without another outside pass. All 72 final-head CI checks now pass, including native full-text on Linux/Windows/macOS, redistributed shards and platform integration gates; no unresolved review threads remain.

This strengthens the existing bounded checks; it does not close the broader older-installation, multi-node/replica, sustained-write, capacity or supported-downgrade gaps above. Bun's qualification failures and the timeout-oracle review disposition remain in Qualify native full-text filtering and catalog-scale operation for v5.4.

Comment generated by kAIle (GPT-6)

Follow-up CI qualification at 415f7b7925464cda72ffd6fcdc621370cd76200d

Harden full-text upgrade crash and readiness tests completed final-head CI. The original uWS shard passed 582 and failed two unchanged QA-769 RocksDB Int-to-Long numeric-index tests: the range request returned INDEX_REBUILDING/503, then a second restart lacked old-encoded index entries. The single diagnostic rerun passed all 584, including both checks, without code changes. This matches the earlier adoption's intermittent signature, but does not establish the root cause or prove unrelatedness. Both complete logs are retained. No assertions were weakened and no runtime fix is claimed for it.

The bounded full-text test-hardening follow-up is ready for human review; broader operational and workload gaps remain open.

Comment generated by kAIle (GPT-6)

Accepted review decision: internal telemetry consumers

The authorized telemetry contract includes removable internal observation demand independently of persistent analytics. The internal subscription hook intentionally has no production caller yet and is not a public SDK. This is planned use and an accepted tradeoff, not a demonstrated bug. When adding a production caller, require explicit subscribe/unsubscribe ownership, cleanup on lifecycle failures and shutdown, and workload qualification of its cadence and optional recording settings. Internal observation demand must not silently enable persistent analytics.

Review-Deferral: b7fa803b170f80d5 severity=nit category=documentation

Comment generated by kAIle (GPT-6)

Harper telemetry implementation and remaining qualification

Implementation is in Add opt-in native full-text analytics #3946, with Document opt-in full-text analytics and the 0.9 engine #789. The core PR is active and awaiting human review. The original companion documentation PR has merged; review-driven corrections are in Clarify full-text telemetry lifecycle and query-policy guidance. This adopts published fulltext 0.9.0 with ABI 14, query API 8 and telemetry API 1. Collection defaults off, changes dynamically on existing handles, and uses Harper's existing protected analytics retrieval, retention and aggregation. Optional histograms and byte observations remain independently opt-in. Internal consumer demand does not enable persistence.

Local deployed full-text and documentation-example integration suites pass 26 tests after correcting the stale ABI assertion. Earlier finite bounded-lag comparison exercised 8,003,174 correct queries across four 70-second profiles with four indexes: enabled throughput was 0.71% and 2.12% lower in the two pairs. This is collection-only evidence, not a production or sustained-memory bound. Strict-freshness measurements and their larger variation are preserved separately; their cause is not established.

The wrapper's basic counters and timing remain active when Harper collection is off. Previously accepted wrapper qualification is supporting evidence; a Harper 0.8-versus-0.9 comparison has not established its total cost. Qualify representative schema widths, resident-index counts, reader churn and multi-worker workloads before claiming production overhead or memory headroom. Sampling intervals govern periodic observations; successful handle closes add final observations. The worker export buffer is bounded to 128 coalesced rows: active groups retain counters after a dropped row, while overflow can lose a closed group's final totals. Preserve series/gap handling and do not present missing intervals as zero activity.

Review-Deferral: 248af43b4e2d8fc0 severity=minor category=performance

Studio controls and metric presentation are tracked separately in Studio: opt-in full-text analytics controls and metric view #2049. The compatible Harper-pro release is not yet established. This work does not complete the broader generation lifecycle/rebuild/eviction/backup scope of this issue.

Comment generated by kAIle (GPT-6)

Accepted collection cadence and consumer contract

Successful reader/writer closes collect final counters in addition to periodic sampling. This preserves cumulative coverage through reader churn; the configured interval is not a total call/export rate cap. Qualify actual churn before making overhead claims. The removable internal consumer hook is intentionally implemented before its first production caller; it remains internal, requires explicit cleanup ownership and must not silently enable persistence. These choices are accepted rather than unresolved correctness findings. Workers and main enforce policy independently; receivers can discard rows during policy changes, and buffer droppedRows does not count policy filtering.

Review-Deferral: 0c5c09fdd7dfee94 severity=minor category=performance
Review-Deferral: af90f065de585711 severity=minor category=maintainability

Comment generated by kAIle (GPT-6)

CI activation qualification remaining

At runtime head 3b18f148aca0, macOS Node26 native-plane CI passed73 tests and failed two unchanged five-second waits for live configuration to activate collection: managed-worker ordering (nativeFullTextTelemetryWorkers.test.js:99) and Table persistence (nativeFullTextTelemetry.test.js:461). Failed native-plane job. This is separate from the corrected deployed ABI assertion. The same full8-file native CI command passed75 tests locally on Node26; that does not establish why CI failed.

Test-only diagnostic assertions at cafe1f1fa44f retain the deadlines/oracles and report file policy, watched policy, analytics eligibility, watcher arming/polling/read counts and listener count on recurrence. Diagnostic CI run. Historical status: this verification was unsettled at that update; current status is recorded below. No timeout increase, forced watcher refresh, blind rerun or speculative runtime workaround was added.

Comment generated by kAIle (GPT-6)

At the diagnostic head, the macOS native-plane job passed (job), as did Linux. The two activation timeouts did not recur; this is not evidence of a root-cause fix. Keep the original failure and diagnostics for recurrence. Other final-head checks were still running at this update.

Comment generated by kAIle (GPT-6)

Accepted telemetry policy-transition behavior

Workers gate export using their current collection policy; main gates persistence using its current policy when a report arrives. During configuration propagation, main can discard observations, including final close rows, without incrementing the worker buffer's droppedRows. Those rows are not replayed. This is an accepted analytics limitation, not loss of source records or native index data.

Before guaranteeing delivery through configuration changes, decide whether rejection accounting or coordinated policy revisions are needed. Preserve receiver-side disabling until that decision. The implementation and core documentation are in Add opt-in native full-text analytics #3946; the public explanation is in Document opt-in full-text analytics and the 0.9 engine #789.

Review-Deferral: 04874ab2c4f55188 severity=nit category=reliability

Comment generated by kAIle (GPT-6)

Kris review refinements (2026-11-04)

Add opt-in native full-text analytics removes the two explicit historical fulltext aliases while retaining 0.9.0 as the production engine. The framework now uses a local Harper development link with a scoped peer override; the ordinary lock graph no longer contains registry Harper 5.3.1 or its nested fulltext 0.5.0. Genuine 0.5.0/0.7.0/0.8.0 migration tests use separate integrity-locked test installs outside the checkout, provisioned explicitly: unit CI installs all three versions; integration CI installs only 0.5.0. No historical binary or benchmark output is committed.

Graceful HTTP/job shutdown keeps the collector alive through owner close, serializes any active analytics flush, then flushes pending and final observations before worker exit/cleanup acknowledgement. This does not guarantee delivery on forced termination or resolve every broader shutdown case in Graceful shutdown doesn't await scope 'close' cleanup — app teardown races process.exit. Missing/invalid configuration retains the last valid policy and overrides, matching the shared watcher. A valid effective edit can disable collection; environment/in-process overlays still apply. Pending overflow favors replaceable live observations over final rows, ordered by last refresh. Test-only calculation helpers are private.

Verification: 48 strict historical migration tests and the deployed full-text suite (19 tests, including the genuine 0.5 upgrade) passed. The final strict native CI selection passed 86 tests. Six shutdown regressions passed, including actual managed HTTP/job worker persistence, overlapping flush ordering and periodic-failure logging; the final Node26 collector/config/shutdown selection passed 33. Documentation build processed 426 documents. Broader local gates are not green: resources finished 6,460 passing/253 pending/25 failures; integration finished 2,959 passing/68 failures/168 cancelled/20 skipped before the now-corrected missing migration-fixture provisioning; main stalled in processGroupHarness and its owned test process was stopped. Remaining broad failures are not all classified against baseline. These results do not establish production performance, sustained memory or forced-exit delivery guarantees.

Explicitly disabling operator collection clears its pending export window. As with receiver-side policy rejection above, this is deliberate opt-out behavior and is not included in the buffer-overflow droppedRows counter. Re-enabling must use series/coverage semantics rather than treating the gap as zero activity.

Review-Deferral: d7e1c54c6bb70396 severity=none category=reliability

Comment generated by kAIle (GPT-6)

Accepted telemetry listener and shutdown policy

Live opt-in requires configuration and analytics-flush listeners even while collection is disabled. Disabled collection starts no periodic native-sampling timer. This is an accepted implementation cost; it does not establish a zero-cost disabled path or a production overhead bound.

Telemetry remains diagnostic rather than part of job success or storage correctness. A final observation/export failure is warned and graceful HTTP/job cleanup continues. The analytics-owned exit-flush promise reports its own failure; the shutdown owner logs it and continues. Do not interpret a clean worker exit as proof that every final telemetry row persisted, and do not make optional telemetry failures abort otherwise successful jobs.

Review-Deferral: 667f9e0a6b041345 severity=nit category=performance
Review-Deferral: f04d46448e66a0e9 severity=none category=reliability

Comment generated by kAIle (GPT-6)

Development-peer cleanup included in the analytics PR

• Add opt-in native full-text analytics supplies the integration framework with harper: file:. as a development dependency and a framework-scoped $harper override. npm does not treat its root checkout as an installed peer; the override also accepts this prerelease host without bypassing unrelated peers. The lock no longer includes the registry Harper / old fulltext subtree. No old native ABI is coerced onto fulltext 0.9.0.
• Release staging removes development dependencies and the framework-only override, preserving unrelated overrides. Both producers remove the self-link before upload and retain exclusion patterns; existing npm install steps recreate it after download. Build before running the local CLI or integration tests. Cleanup/copy tools must not follow directory links into the checkout.
• Genuine 0.5.0/0.7.0/0.8.0 migration engines remain separately integrity-locked outside the checkout. Removing the development duplicate does not remove that coverage.
• Qualification: framework-scoped resolution and staged-policy regressions failed before the fix; native migration/telemetry selection passed 88 tests and packaging selection passed 63. Initial strict clean installs passed on npm 10/11/12; repeated npm 10 installs preserved a root sentinel and package resolutions. Further repeat and fresh-consumer probes were inconclusive during local slowdown; an unchanged-graph control also exceeded 180 seconds. These results do not qualify install performance. The PR author status records final evidence and broad-gate limits.
• Remove the local self dependency and scoped override together if a released framework later makes its Harper peer optional and its local/explicit binary route is qualified. No framework release or unreleased patch is required for this cleanup.

Comment generated by kAIle (GPT-6)

Current cleanup qualification: deployed full-text 17 passed. Broad resources completed 6,348 passing/243 pending/147 failures; integration 3,077 passing/67 failures/51 cancelled/20 skipped. Main stopped advancing in componentPreparationLock after more than 50 minutes without a summary; only its owned process was stopped. Two EALLOWSCRIPTS-affected integration files passed 35 tests with an empty npm user configuration. Other failures are not all baseline-classified. These results do not establish all-green or install-performance qualification.

Pre-cleanup head 5c568c3 macOS native-plane CI passed 84 and failed four unchanged five-second waits: managed-worker policy ordering, job flush, deferred-scope exit and Table telemetry activation. Linux and Windows native-plane jobs passed. The cause is unproven; no deadline/oracle change or blind rerun is claimed.

Comment generated by kAIle (GPT-6)

Cleanup review reconciliation and remaining qualification

Cleanup and the requested boot/live validation refinement are implemented through b478966830e7. Unknown telemetry options now fail boot validation as they already do in the live resolver. The new regression fails on the preceding validator; all nine configuration tests pass after rebuilding. Final build/lint/format checks pass. This last delta is one production line and eleven test lines. Composer checked it and confirmed alignment with no new blocker; Claude/full review was not rerun. The review tool still records a closing full review as owed.

The latest completed Claude/Composer review did not formally converge. It retains the missing Harper 0.8-versus 0.9 default-path accounting comparison as a major qualification concern, plus enabled-collection storage sizing. Earlier collection-off/on measurements both use 0.9 and do not establish the upgrade cost. Before claiming production overhead or enabling at scale, qualify that comparison, encoded row sizes with/without histograms, index/thread/role cardinality, audit amplification and configured raw/aggregate retention. No storage-size or overhead bound is invented here. Existing internal-consumer and transition-drop limitations above remain accepted and tracked.

The packed-consumer ENOENT review record originated in the macOS temporary-path packaging fixture. Canonicalizing that path makes all 63 packaging tests pass; no continuing release-consumer ENOENT has been demonstrated. Real release-consumer installation was still pending at review because later local full-package probes did not complete. All four consumer-CI lanes subsequently passed at this head, as recorded below. Keep this record as qualification history rather than a claim of a current defect.

Review-Deferral: c986c0e301871907 severity=minor category=reliability

Release staging removes and rejects the one current framework development override. The manifest has no other $ reference today. The checker is deliberately specific; if a future development-only override references another removed dependency, add its release validation or generalize then. This is a future-policy limitation, not a current dangling reference.

Review-Deferral: 10197948fc9e8249 severity=nit category=reliability

Review tooling caveats: the first adjudicator stalled without output; recovery used authorized Claude and Composer. An intermediate planner unexpectedly restored excluded Gemini, which failed authentication and produced no review; that run was stopped and explicit exclusions were applied. The completed full graded trace at 3f0dd0f is preserved, but no exact-final convergence is claimed. The teardown warning at nativeFullTextTelemetry.test.js:42 is a false positive: handles?.reverse() ?? [] already guards absent setup.

Comment generated by kAIle (GPT-6)

Current cleanup-head CI qualification

At b478966830e7, strict install checks on Node 22/24, release-package build, all four consumer-install lanes (npm 10/11/12 on Linux plus npm 12 on Windows), and full-text native planes on Linux/Windows/macOS passed. These actual consumer results complete the remaining packaged-install qualification above for the ENOENT record; there is no demonstrated continuing consumer defect. The archived/downloaded integration build and its npm relink step also completed successfully. Earlier failed macOS waits are preserved; this green occurrence does not establish their root cause.

Two broader jobs failed. Node 24 resource unit tests passed 2,946 with 123 pending and failed one afterEach check in noAbortAfterTxnLog.test.js:80: a deliberately wedged native commit overlapped unhandled 503 rejections from storeRocksDBStatsMetrics through analytics aggregation. uWS integration shard 3 failed sourcedfrom-log-identity.test.ts waiting for three GenericMirror writes and then received HNSW DERIVED_INDEX_LAGGING. Causes and baseline attribution are unproven; no rerun, timeout increase or speculative runtime workaround was made. At this check two other jobs were still active. The PR is not all-green.

Comment generated by kAIle (GPT-6)

Final Kris refinements and nullable-consumer qualification

Add opt-in native full-text analytics at 204dc59383aa passed the exact native CI selection (88 tests) and focused collector/configuration/aggregation/shutdown checks (53 tests). The real LMDB process-exit/reopen and null-metadata regressions fail before their fixes and pass afterward; RocksDB exit/reopen also passes. Main awaits its final raw write and the existing commit barrier, with no added disk-sync guarantee. A synthetic 2.1 MB tracked diff has the expected streamed fingerprint; this is manifest qualification, not a new performance run. Build/lint/design checks and the 426-document companion build pass. Earlier broad-gate failures above remain unclassified in part and are not relabeled green.

Stored telemetry deliberately retains null for unavailable observations and unbounded histogram bounds. describe_metric now omits null-valued attributes from inferred types; get_analytics preserves null. The Grafana datasource's nullable conversion has an explicit nil branch; this source check does not qualify every downstream client or mixed null/numeric frame ordering. Before promising that compatibility, qualify unavailable-to-available fields across windows in actual Grafana frames; do not replace missing observations with invented zeros.

Review-Deferral: ce7cd0c4cd6a8fcb severity=nit category=correctness

Write telemetry through table transactions overlaps raw persistence. Whichever lands second must preserve awaited final commits and rerun the shutdown regressions. Public corrections follow the already-merged Document opt-in full-text analytics and the 0.9 engine in Clarify full-text telemetry lifecycle and query-policy guidance.

Comment generated by kAIle (GPT-6)

The subsequent CI linter passed code lint but failed the merged config/DESIGN.md budget after main grew to within 43 bytes of its limit. The final prose-only trim preserves watcher rationale and links retained telemetry details; design checks pass and the actual merge measures 40,794 bytes under the unchanged 40,960-byte limit. Runtime review/tests remain anchored to the preceding runtime commit. The final trim was second-reader checked without another outside-model pass; CI was pending at that historical update for head 5c568c3. Its completed macOS failure is recorded in the cleanup section above.

Analytics delivery follow-up reconciliation

Aggregate rows are best-effort: the cursor can advance past failed writes, including a logged/dropped 503. A producer-level error handler, as proposed in Add opt-in native full-text analytics, prevents unhandled errors; it does not provide transactional aggregate/cursor persistence. That separate Harper analytics change must persist the cursor with its rows before promising replay or durable delivery.

Review-Deferral: 76eee034ee4212bc severity=minor category=data-integrity

Main-thread receive-time policy filtering can discard a worker row during a policy transition without incrementing droppedRows. This is the same transition-loss policy already tracked above, not a newly demonstrated retention failure. Preserve producer-time policy or add explicit drop accounting before promising transition-complete delivery.

Review-Deferral: ee4ef8e5d12360a6 severity=nit category=reliability

Internal-consumer review record reconciliation

The existing accepted internal telemetry subscription seam has no production caller. It can enable collection, shorten cadence and union optional recording settings; it remains internal diagnostic/test machinery, not a supported public SDK. This record aliases the existing consumer-seam follow-up rather than introducing a new feature request. Before exposing a production consumer, decide ownership and compatibility and qualify its policy interactions.

Review-Deferral: dee1eab820047bf4 severity=nit category=maintainability

Comment generated by kAIle (GPT-6)