Add opt-in native full-text analytics

.github/DESIGN.md

@@ -10,9 +10,9 @@ Index of every design note: [DESIGN.md](../DESIGN.md).
 
 ## The lock file must pass `npm ci` under npm 10 and npm 11 (`workflows/lockfile-npm-compat.yml`)
 
-npm 11 accepts some lock files that npm 10's `npm ci` rejects as out of sync. Renovate and the Sync Lock File workflow (`.node-version`) write the lock with npm 11, but harper supports Node 22, which ships npm 10. harper#3006 auto-merged such a lock: the root `utf-8-validate` 6.x didn't satisfy the `^5.0.2` optional peer of the `ws@7` copies in the react-native subtree, so npm 10 wants a nested 5.x where npm 11 doesn't. harper#3015 removed that root declaration. `lockfile-npm-compat.yml` runs `npm ci --dry-run` on Node 22 and Node 24 for every PR and `main` push, with no path filter so it can be a required check.
+npm 11 accepts some lock files that npm 10's `npm ci` rejects as out of sync. Renovate and the Sync Lock File workflow (`.node-version`) write the lock with npm 11, but harper supports Node 22, which ships npm 10. harper#3006 auto-merged such a lock: the root `utf-8-validate` 6.x didn't satisfy the `^5.0.2` optional peer of the `ws@7` copies in the react-native subtree, so npm 10 wants a nested 5.x where npm 11 doesn't. harper#3015 removed that root declaration. `lockfile-npm-compat.yml` runs `npm ci --strict-peer-deps --dry-run` on Node 22 and Node 24 for every PR and `main` push, with no path filter so it can be a required check.
 
-When the Node 22 leg fails, remove the version conflict at its source. Regenerating the lock with npm 10 doesn't hold: the next npm 11 write prunes the npm-10-only entries again. Release bundling follows production reachability rather than trusting `dev` flags: native-containing trees remain unbundled so the consumer selects its own platform.
+Unsatisfied required peers fail this gate rather than being accepted as install warnings. When the Node 22 leg fails, remove the version conflict at its source. Regenerating the lock with npm 10 doesn't hold: the next npm 11 write prunes the npm-10-only entries again. Release bundling follows production reachability rather than trusting `dev` flags: native-containing trees remain unbundled so the consumer selects its own platform.
 
 ## A release copies to Docker Hub only the image it booted (`workflows/docker-smoke.yml`, `workflows/publish-docker.yaml`)
 
@@ -30,6 +30,8 @@ Branch protection requires only the `Unit Test` job, which needs every other job
 
 Node 26 is the recommended runtime, so every single-version job runs it, and pull requests run unit tests on 24 and 26 (the `lmdb` shard on 26 alone until harper#3349's segfault is fixed) and integration tests on 26 alone. Pushes to `main` run both on 24 and 26. Node 22 runs nightly and on a manual `all` dispatch, as do the native type-strip legs on `22.18.0` and `24.0.0`, the `engines` floors in `package.json`, which are the oldest runtimes that must run the `.ts` sources directly. The nightly schedule runs only on `main`, so a push to a release branch runs that full set: a patch release still ships to Node 22 users. The cherry-pick job dispatches a release branch's unit tests on Node 26 (or 24 when that branch's workflow predates 26) and its integration tests on the Node version whose build artifact its Bun and uWS jobs download (falling back to the unit-test version when all artifact names are matrixed). The integration check names carry the Node version, so moving the PR version means changing the ruleset's required checks in the same release. `.node-version` and `.nvmrc` hold only the major, and every `setup-node` step reading `.node-version` sets `check-latest`, so tooling, release and publish jobs run the newest stable 26.x rather than the runner's cached one or a prerelease.
 
+Integration producers remove `node_modules/harper` before upload and exclude it from artifacts: the globber can still walk an excluded directory link. Each consuming lane's existing npm install recreates the link and bin shims after download.
+
 ### Unit-test fixtures don't spend the `lmdb` shard on one durable commit per row
 
 `test:unit:lmdb` re-runs `unitTests/resources`, `unitTests/apiTests` and `unitTests/bin` on LMDB, where every awaited commit waits on a disk sync. A fixture that seeds or clears one row per transaction costs seconds on the default engine and minutes on LMDB, and a runner with slow fsync multiplies exactly that part. `subscriptionPreviousCountScanBound.test.js` (harper#2826), `subscriptionSuperseded.test.js` (harper#2887) and `calibrationStore.test.js` each pushed the step past its cap this way. Nothing enforces this; the step's `timeout-minutes` is the only detector.

.github/workflows/integration-tests.yml

@@ -112,6 +112,11 @@ jobs:
       - name: Build
         run: npm run build
 
+      - name: Remove local development link before upload
+        timeout-minutes: 1
+        run: |
+          node -e "require('node:fs').rmSync('node_modules/harper', { recursive: true, force: true })"
+
       - name: Upload build artifacts
         uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
         with:
@@ -120,6 +125,7 @@ jobs:
             dist/
             static/
             node_modules/
+            !node_modules/harper
             package.json
           retention-days: 7
 
@@ -194,6 +200,10 @@ jobs:
           fi
           dpkg-query -W openssh-server
 
+      - name: Provision the historical full-text integration engine
+        timeout-minutes: 3
+        run: npm run test:fulltext:install-previous -- "${{ runner.temp }}/fulltext-upgrade" 0.5.0
+
       - name: Run Integration Test Shard ${{ matrix.shard }}
         timeout-minutes: 15
         env:
@@ -205,6 +215,7 @@ jobs:
           # package.json's minor advances.
           HARPER_PREVIOUS_MINOR_PATH: /tmp/harper-prev-minor-52/node_modules/harper:/tmp/harper-prev-minor-53/node_modules/harper
           HARPER_INTEGRATION_TEST_INSTALL_SCRIPT: dist/bin/harper.js
+          HARPER_FULLTEXT_UPGRADE_ROOT: ${{ runner.temp }}/fulltext-upgrade
           # Linux runners have working inotify, so the file-watch-dependent TLS regression
           # tests (cert-key-reload) must run rather than skip — this pipeline owns #2382.
           HARPER_TEST_REQUIRE_FILE_WATCHERS: '1'
@@ -244,6 +255,11 @@ jobs:
       - name: Build
         run: npm run build
 
+      - name: Remove local development link before upload
+        timeout-minutes: 1
+        run: |
+          node -e "require('node:fs').rmSync('node_modules/harper', { recursive: true, force: true })"
+
       - name: Upload build artifacts
         uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
         with:
@@ -252,6 +268,7 @@ jobs:
             dist/
             static/
             node_modules/
+            !node_modules/harper
             package.json
           retention-days: 1
 
@@ -288,12 +305,17 @@ jobs:
         timeout-minutes: 5
         run: npm install --ignore-scripts --prefix "${{ runner.temp }}/harperdb-legacy" harperdb@4
 
+      - name: Provision the historical full-text integration engine
+        timeout-minutes: 3
+        run: npm run test:fulltext:install-previous -- "${{ runner.temp }}/fulltext-upgrade" 0.5.0
+
       - name: Run Integration Test Shard ${{ matrix.shard }}
         timeout-minutes: 20
         env:
           HARPER_INTEGRATION_TEST_LOG_DIR: ${{ runner.temp }}/harper-integration-test-logs
           HARPER_LEGACY_VERSION_PATH: ${{ runner.temp }}/harperdb-legacy/node_modules/harperdb
           HARPER_INTEGRATION_TEST_INSTALL_SCRIPT: dist/bin/harper.js
+          HARPER_FULLTEXT_UPGRADE_ROOT: ${{ runner.temp }}/fulltext-upgrade
         run: npm run test:integration:all -- --shard=${{ matrix.shard }}/6
 
       - name: Upload Harper server logs
@@ -343,12 +365,17 @@ jobs:
         timeout-minutes: 5
         run: npm install --ignore-scripts
 
+      - name: Provision the historical full-text integration engine
+        timeout-minutes: 3
+        run: npm run test:fulltext:install-previous -- "${{ runner.temp }}/fulltext-upgrade" 0.5.0
+
       - name: Run Integration Test Shard ${{ matrix.shard }}
         timeout-minutes: 15
         env:
           HARPER_INTEGRATION_TEST_LOG_DIR: /tmp/harper-integration-test-logs
           HARPER_RUNTIME: bun
           HARPER_INTEGRATION_TEST_INSTALL_SCRIPT: dist/bin/harper.js
+          HARPER_FULLTEXT_UPGRADE_ROOT: ${{ runner.temp }}/fulltext-upgrade
         run: |
           npm run test:integration:all -- --shard=${{ matrix.shard }}/6
 
@@ -403,6 +430,10 @@ jobs:
           mkdir -p /tmp/harper-legacy-51
           npm install --ignore-scripts --prefix /tmp/harper-legacy-51 harper@5.1.22
 
+      - name: Provision the historical full-text integration engine
+        timeout-minutes: 3
+        run: npm run test:fulltext:install-previous -- "${{ runner.temp }}/fulltext-upgrade" 0.5.0
+
       # Serve the plaintext app HTTP port(s) through uWebSockets.js instead of the Node http server
       # (#914). Only non-secure, non-operations TCP ports are affected; secure/replication/ops paths
       # keep running on Node, so this exercises the uWS request/streaming/WS/GraphQL/Fastify-fallback
@@ -415,6 +446,7 @@ jobs:
           HARPER_LEGACY_51_PATH: /tmp/harper-legacy-51/node_modules/harper/dist/bin/harper.js
           HARPER_UWS_HTTP: 1
           HARPER_INTEGRATION_TEST_INSTALL_SCRIPT: dist/bin/harper.js
+          HARPER_FULLTEXT_UPGRADE_ROOT: ${{ runner.temp }}/fulltext-upgrade
         run: |
           npm run test:integration:all -- --shard=${{ matrix.shard }}/6
 

.github/workflows/lockfile-npm-compat.yml

@@ -41,4 +41,4 @@ jobs:
         timeout-minutes: 3
         run: |
           echo "npm $(npm -v)"
-          npm ci --ignore-scripts --dry-run --no-audit --no-fund
+          npm ci --ignore-scripts --strict-peer-deps --dry-run --no-audit --no-fund

.github/workflows/unit-test.yml

@@ -322,16 +322,25 @@ jobs:
       - name: Install dependencies
         run: npm install --ignore-scripts
 
+      - name: Install isolated historical full-text engines
+        timeout-minutes: 3
+        run: npm run test:fulltext:install-previous -- "${{ runner.temp }}/fulltext-upgrade"
+
       - name: Build
         run: npm run build
 
       - name: Verify the published native binding loads
-        run: node --input-type=module --eval "const binding = await import('@harperfast/fulltext/native'); const info = await binding.runtimeInfo(); if (typeof binding.openNativeFullTextIndex !== 'function' || typeof binding.openNativeFullTextReader !== 'function' || typeof binding.NativeFullTextIndex?.prototype.searchPage !== 'function' || info.nativeAbiVersion !== 13 || info.queryApiVersion !== 7) throw new Error('@harperfast/fulltext loaded without the required writer, reader, and native ABI 13 and query API 7 contract');"
+        run: node --input-type=module --eval "const { validateFullTextNativeBinding } = await import('./dist/resources/indexes/fullTextNativeBinding.js'); await validateFullTextNativeBinding(await import('@harperfast/fulltext/native'));"
 
       - name: Native full-text lifecycle test
+        env:
+          HARPER_FULLTEXT_UPGRADE_ROOT: ${{ runner.temp }}/fulltext-upgrade
         run: >-
           npx mocha
-          unitTests/resources/nativeFullTextDerivedIndexLifecycle.test.js
+          unitTests/resources/nativeFullTextDerivedIndexLifecycle.test.js unitTests/resources/nativeFullTextTelemetry.test.js
+          unitTests/resources/nativeFullTextTelemetryClose.test.js
+          unitTests/resources/nativeFullTextTelemetryWorkers.test.js
+          unitTests/resources/nativeFullTextTelemetryShutdown.test.js
           unitTests/resources/nativeFullTextTableSearch.test.js
           unitTests/resources/nativeFullTextIndexSchemaUpgrade.test.js
           unitTests/resources/nativeFullTextIndexUpgradeWorkers.test.js

.gitignore

@@ -52,10 +52,11 @@ harper-*.tgz
 .claude/
 cache/
 
-# YCSB benchmark results (generated)
+# Benchmark results (generated)
 benchmarks/ycsb/results/
 benchmarks/ycsb/bench-out/
 benchmarks/compression/results/
+benchmarks/fulltext-analytics/results/
 
 # export_local integration-test artifacts — these tests pass `path: './'`, so the export lands in
 # the repo root and a `git add -A` after a test run commits it

AGENTS.md

@@ -122,7 +122,7 @@ Use this to land in the right folder before grepping. Every top-level folder is
 ### Non-source
 
 - **`bin/`** — covered above (it's source).
-- **`benchmarks/`** — HNSW vector-search benchmark only (`hnsw-search.js`). Stand-alone; not part of CI.
+- **`benchmarks/`** — Stand-alone HNSW search (`hnsw-search.js`) and paired full-text analytics (`fulltext-analytics.mjs` / `.bench.js`) workloads; not part of CI.
 - **`build-tools/`** — build-pipeline scripts. [build-tools/DESIGN.md](build-tools/DESIGN.md) covers the published artifacts. Tests: `unitTests/build-tools/`; run `npm run test:unit:main` after changes.
 - **`dev/`** — single dev utility (`sync-commits.js`) for cross-repo commit syncing. Not runtime.
 - **`integrationTests/`** — end-to-end tests against a built distribution. Run with `npm run test:integration` / `npm run test:integration:all`. Subdirs mirror source. See `integrationTests/README.md`.

DESIGN.md

@@ -72,12 +72,13 @@ A note lives in the `DESIGN.md` of the directory that owns the code it describes
 - [A filtered vector search may admit from an index key set only if the set holds every predicate match](resources/indexes/DESIGN.md#a-filtered-vector-search-may-admit-from-an-index-key-set-only-if-the-set-holds-every-predicate-match) — An indexed companion condition becomes an HNSW allow-set only when its record predicate rejects every value the index cannot hold.
 - [Filtered full-text search reuses companion secondary indexes before loading source records](resources/indexes/DESIGN.md#filtered-full-text-search-reuses-companion-secondary-indexes-before-loading-source-records) — Selective ranges become bounded key sets; broad indexed equalities become costed point probes before Harper loads authoritative records.
 - [Derived-index runtime: committed-log delivery to native index backends (`resources/derivedIndexRuntime.ts`)](resources/indexes/DESIGN.md#derived-index-runtime-committed-log-delivery-to-native-index-backends-resourcesderivedindexruntimets) — The transaction log is the durable fact; a commit only wakes the lock-elected runner, which resumes every backend from an exact cursor.
-- [Native full-text backend](resources/indexes/DESIGN.md#derived-index-runtime-committed-log-delivery-to-native-index-backends-resourcesderivedindexruntimets) — The native full-text subsection documents Tantivy files as node-local derived state, with one elected writer advancing each index while per-worker readers use publication and reset fencing.
+- [Native full-text backend](resources/indexes/DESIGN.md#derived-index-runtime-committed-log-delivery-to-native-index-backends-resourcesderivedindexruntimets) — One elected writer; node-local Tantivy state and fenced per-worker readers.
 - [Native HNSW plane: a file-primary mmap graph on the derived-index runtime (`resources/indexes/HierarchicalNavigableSmallWorld.ts`, `resources/indexes/hnswDerivedIndex.ts`, `resources/indexes/hnswPlaneBinding.ts`)](resources/indexes/DESIGN.md#native-hnsw-plane-a-file-primary-mmap-graph-on-the-derived-index-runtime-resourcesindexeshierarchicalnavigablesmallworldts-resourcesindexeshnswderivedindexts-resourcesindexeshnswplanebindingts) — The mmap file owned by `@harperfast/hnsw` is the index; it is the default for audited RocksDB tables, and `nativePlane: false` keeps the JS graph.
 - [Scheduler: cluster-once execution without a consensus primitive (`resources/scheduler/`)](resources/scheduler/DESIGN.md#scheduler-cluster-once-execution-without-a-consensus-primitive-resourcesscheduler) — No consensus primitive exists; cluster-once relies on sticky leadership, heartbeat takeover and LWW lease convergence.
 
 ## resources/analytics/
 
+- [Full-text handle ownership](resources/analytics/DESIGN.md#full-text-collection-follows-actual-handle-ownership) — Opt-in; counters survive reader churn.
 - [Analytics aggregation resumes from a raw cursor, which is not a clock (`resources/analytics/write.ts`)](resources/analytics/DESIGN.md#analytics-aggregation-resumes-from-a-raw-cursor-which-is-not-a-clock-resourcesanalyticswritets) — `rawCursor` moves only to a record a cycle actually read and is separate from the cadence marker.
 - [Per-thread rows carry a `thread` attribute under their own metric name (`resources/analytics/write.ts`)](resources/analytics/DESIGN.md#per-thread-rows-carry-a-thread-attribute-under-their-own-metric-name-resourcesanalyticswritets) — `perThread` entries roll up one row per thread; `duration` per thread is `duration-by-thread`, never extra `duration` rows; rows stay flat scalars for the Grafana datasource. · [A reset ends the window that was being flushed too (`resources/analytics/write.ts`)](resources/analytics/DESIGN.md#a-reset-ends-the-window-that-was-being-flushed-too-resourcesanalyticswritets) — `setAnalyticsEnabled()` advances a generation; a flush that sees it change stops delivering.
 - [Aggregation folds each measure over only the samples that carry it (`resources/analytics/write.ts`)](resources/analytics/DESIGN.md#aggregation-folds-each-measure-over-only-the-samples-that-carry-it-resourcesanalyticswritets) — per-measure sample counts, cycle-local, weight each mean; a measure in every sample folds exactly as before.