feat: first-class bm25 ranked-retrieval index

[cpegeric]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #25541

What this PR does / why we need it:

Summary

Add bm25 as a new first-class secondary index type in MatrixOne: a lean,
position-free BM25 ranked-retrieval index for text columns, decoupled from the
classic fulltext index. It answers disjunctive (OR) top-K bag-of-words queries with
Block-Max WAND, has its own query verb BM25(col) AGAINST('query'), and is maintained
incrementally via CDC (async) with scheduled tiered-merge compaction.

Unlike the classic fulltext index (which stores term positions to serve boolean /
phrase / natural-language-with-expansion modes), bm25 is intentionally
position-free. That makes its on-disk index much smaller and its top-K walk much
faster, at the cost of not supporting operator/phrase queries — the two index types are
complementary and can coexist on the same column.

Motivation

• The classic fulltext index pays for term positions on every posting to support
  boolean/phrase modes. A large class of retrieval workloads only need ranked
  bag-of-words top-K (search boxes, RAG retrieval, "most relevant N documents").
• For that workload a position-free BM25 index with Block-Max WAND skips
  non-competitive documents and returns the true top-K without scoring the whole
  postings list. Benchmarks showed ~5× faster query latency than the classic
  fulltext path on the same corpus.

Query surface

-- ranked top-K (BM25 score DESC); LIMIT is pushed into the WAND walk
SELECT id FROM docs WHERE BM25(body) AGAINST('influenza vaccine') LIMIT 10;

-- score in the projection
SELECT id, BM25(body) AGAINST('influenza vaccine') AS score
FROM docs WHERE BM25(body) AGAINST('influenza vaccine') ORDER BY score DESC;

-- filtered retrieval (predicate pushed into the walk as a membership bitset)
SELECT id FROM docs
WHERE BM25(body) AGAINST('influenza vaccine') AND category = 10 LIMIT 10;

BM25() is a distinct verb from MATCH():

• BM25(col) AGAINST('q') → the bm25 index. Mode-free (always ranked bag-of-words).
• MATCH(col) AGAINST('q' [IN … MODE]) → the classic fulltext index (unchanged).

The two are disambiguated by function name alone, so a column can carry both index
types and each verb targets its own — no mode heuristics.

DDL

CREATE INDEX ftx USING bm25 ON docs(body) WITH PARSER gojieba;

-- optional async / scheduled-compaction knobs
CREATE INDEX ftx USING bm25 ON docs(body)
  WITH PARSER gojieba, ASYNC, AUTO_UPDATE, HOUR 3, MAX_INDEX_CAPACITY 1000000;

ALTER TABLE docs ALTER REINDEX ftx BM25 MERGE;    -- incremental tiered merge
ALTER TABLE docs ALTER REINDEX ftx BM25;          -- full rebuild-from-source
DROP INDEX ftx ON docs;

• Single text column (CHAR/VARCHAR/TEXT/JSON/DATALINK).
• WITH PARSER selects the tokenizer: gojieba (jieba dict + per-index overflow
  vocab), ngram, default.
• Follows the vector-index DDL path (USING <algo> → *tree.Index →
  BuildSecondaryIndexDefs), not the fulltext CREATE FULLTEXT INDEX path.

Design

Engine (pkg/bm25/wand)

• Block-Max WAND (position-free, doc-ordered, disjunctive top-K) with a
  membership allow-set consulted during the walk for filtered search.
• Two-tier LSM: tag=0 base sub-indexes (capacity-bounded, recency-ordered) +
  tag=1 CdcTail (delta insert frames + delete frames). Liveness resolved by recency.
• Serialization keys documents by primary key (encodePk/decodePk); a normalizeKey
  canonicalizes string-family pks so they can key the liveness/dedup maps.

Plugin (pkg/bm25/plugin) — via the indexplugin framework

• runtime (catalog hooks): two hidden tables [storage, metadata], AlwaysAsync,
  SyncDescriptor (CDC + idxcron action bm25_reindex, option MERGE),
  ParamsFromTree (parser / async / auto_update / day / hour / second /
  max_index_capacity).
• compile: sync build-from-source or async CDC-arm; HandleReindex (REBUILD/MERGE);
  restore InitSQL; ValidateReindexParams.
• plan: BuildSecondaryIndexDefs (storage + metadata tables, no postings table);
  inert CanApply/ApplyForSort (bm25 uses the BM25() path, not ORDER-BY rewrite).
• idxcron: tail-threshold Updatable; ReindexOption MERGE-vs-REBUILD by dead-doc %.
• iscp: CDC writer = the WAND sql-writer.
• Blank-imported in pkg/indexplugin/all + pkg/indexplugin/iscp.

Hidden tables

• storage — chunked binary (WAND) index blobs (index_id, chunk_id, data, tag).
• metadata — one row per sub-index (index_id, timestamp, checksum, filesize, recency, nrow).
• No postings table — bm25 builds directly from the source rows.

Table-valued functions

• bm25_search(param, cfg{db,index,metadata}, pattern) — tokenize + Block-Max WAND
  top-K, emitting (doc_id, score); honors a runtime-filter membership bitset and
  pushed-down LIMIT.
• bm25_create — build tag=0 base from source.
• bm25_compact — fold tag=1 tail into tag=0 (tiered merge).

Query planning (pkg/sql/plan/apply_indices_bm25.go)

BM25(col) AGAINST('q') binds to the bm25_match function → rewritten to a
bm25_search TVF INNER-JOINed to the source on pk, with a SORT by score DESC and
LIMIT pushdown. When combined with a non-MATCH filter and
fulltext_bloom_filter_pushdown=ON, a pre-filter join builds a membership bitset that
is pushed into the WAND walk so only qualifying documents are scored.

CDC + scheduled compaction

• Post-create DML flows into the tag=1 CdcTail via the ISCP WAND sinker (async).
• idxcron (cluster-singleton) periodically fires an ALTER REINDEX … MERGE to fold
  the tail into the base. Interval knobs: DAY/HOUR/SECOND per-index +
  MO_IDXCRON_TICK_SEC / MO_IDXCRON_CRON / MO_IDXCRON_INTERVAL_SEC envs.

Primary-key support

encodePk covers: int32/64, uint32/64, varchar/char/text/datalink, uuid,
date/datetime/time/timestamp, decimal64/128, and composite pk (packed
CPrimaryKey varchar). Any other single-column pk type
(tinyint/smallint/float/bit/bool/enum) is rejected at CREATE INDEX with a clear
message (rather than silently aborting CDC later).

Lifecycle covered

CREATE (sync + async) → query (BM25(), score, LIMIT pushdown, filtered) → live DML via
CDC → ALTER REINDEX (REBUILD / MERGE) → scheduled idxcron MERGE → CLONE → RESTORE →
DROP.

Testing

• Engine unit (pkg/bm25/wand) — build / search / merge / serialize / tail /
  deletes / compaction (incl. string-pk MERGE regression).
• Plugin unit (pkg/bm25/plugin/{plan,runtime}) — schema build, pk-type
  validation, param extraction, sync descriptor.
• BVT — test/distributed/cases/fulltext/bm25_* (retrieval, datetime, uuid,
  multibase, limit, mode_guard, coexist, pushdown) and
  test/distributed/cases/pessimistic_transaction/bm25/* (basic, async, reindex,
  merge, clone, restore).

Non-goals / known limitations

• Position-free: no boolean / phrase / natural-language-with-expansion. Use a
  classic fulltext MATCH index for those (the two coexist).
• A single query combining BM25() and MATCH() on the same statement is rejected
  (clear error) — run them as separate queries.
• Cross-CN cache freshness after a CDC update is bounded by the shared index-cache TTL
  (same model as the vector indexes).

Acceptance criteria

• CREATE INDEX … USING bm25 … WITH PARSER <tok> builds storage + metadata tables.
• BM25(col) AGAINST('q') returns ranked top-K; LIMIT pushed into the walk.
• Post-create DML converges via CDC; ALTER REINDEX … MERGE folds the tail.
• Filtered retrieval (… AND <filter>) with pushdown returns identical rows to
  the non-pushdown path.
• Coexist: MATCH→classic, BM25→bm25 on the same column.
• Clone / restore reproduce a queryable index; idxcron MERGE fires on schedule.
• Unsupported pk types rejected at CREATE; supported pk types (incl. string /
  uuid / datetime / decimal / composite) round-trip through MERGE.
  eric@mo matrixone % cat ../../../../private/tmp/claude-501/-Users-eric-github-matrixone/3ea221e9-92a9-4e29-a7b4-ae404b34ac35/scratchpad/bm25_index_pr.md

feat: first-class bm25 ranked-retrieval index

What

Adds bm25 — a new first-class, position-free BM25 ranked-retrieval index for
text columns, with its own query verb BM25(col) AGAINST('query'), incremental CDC
maintenance, and scheduled tiered-merge compaction. It is decoupled from the classic
fulltext index and complementary to it (the two can coexist on the same column).

The engine (Block-Max WAND) is ported from the fulltext_wand branch and lifted into a
standalone pkg/bm25/ package; the index is wired through the existing indexplugin
framework (no per-algo switch in the planner/compiler).

Scope: 112 files, ~22k insertions. 18 commits, organized in phases (see below).

Why

The classic fulltext index stores term positions on every posting to serve
boolean/phrase/NL-with-expansion modes. A large class of retrieval workloads (search
boxes, RAG retrieval, "top-N most relevant docs") only need ranked bag-of-words
top-K. A position-free BM25 index with Block-Max WAND skips non-competitive documents
and returns the true top-K without scoring the whole postings list — ~5× faster than
the classic fulltext path on the same corpus in our benchmarks, with a much smaller
on-disk index.

Query surface & DDL

CREATE INDEX ftx USING bm25 ON docs(body) WITH PARSER gojieba;

-- ranked top-K; LIMIT pushed into the WAND walk
SELECT id FROM docs WHERE BM25(body) AGAINST('influenza vaccine') LIMIT 10;

-- score in projection
SELECT id, BM25(body) AGAINST('influenza vaccine') AS score FROM docs
WHERE BM25(body) AGAINST('influenza vaccine') ORDER BY score DESC;

-- filtered retrieval (predicate pushed into the walk as a membership bitset)
SELECT id FROM docs WHERE BM25(body) AGAINST('influenza vaccine') AND category=10 LIMIT 10;

ALTER TABLE docs ALTER REINDEX ftx BM25 MERGE;   -- incremental tiered merge
ALTER TABLE docs ALTER REINDEX ftx BM25;         -- full rebuild-from-source

BM25() is a distinct verb from MATCH() — disambiguated by function name, so a
column can carry both a classic fulltext index and a bm25 index: MATCH(col) targets
the former, BM25(col) the latter, with no mode heuristics.

What's in this PR

Engine — pkg/bm25/wand (Phase 0)

Block-Max WAND top-K, two-tier LSM (tag=0 capacity-bounded base subs + tag=1
CdcTail delta/delete frames), membership-bitset prefilter, pk serialize/deserialize.
Ported byte-for-byte from fulltext_wand, relocated and de-fulltext'd.

Index type identity & grammar (Phase 1)

• catalog.MoIndexBm25Algo (USING bm25), hidden-table type constants.
• Grammar: USING BM25 [WITH PARSER …], REINDEX … BM25 [MERGE], and the distinct
  BM25(col) AGAINST('q') expression production (goyacc regen, 0 conflicts).

The bm25 AlgoPlugin — pkg/bm25/plugin (Phases 1b–4)

runtime (catalog hooks: 2 hidden tables, AlwaysAsync, SyncDescriptor,
ParamsFromTree) · compile (sync/async create, HandleReindex REBUILD/MERGE,
restore, ValidateReindexParams) · plan (BuildSecondaryIndexDefs, inert
vector hooks) · idxcron (tail-threshold Updatable, ReindexOption) · iscp (CDC
writer). Blank-imported in indexplugin/all + indexplugin/iscp.

Hidden tables & TVFs (Phase 2)

Storage (chunked WAND blobs) + metadata (one row per sub-index); no postings table.
bm25_create, bm25_search, bm25_compact TVFs registered in
pkg/sql/colexec/table_function.

Query routing (Phase 3, then decoupled)

BM25() → bm25_match function → bm25_search TVF INNER-JOINed to source on pk,
SORT by score DESC, LIMIT pushdown, optional membership-bitset pre-filter for
… AND <filter>. Lives in apply_indices_bm25.go (self-contained;
apply_indices_fulltext.go untouched), sharing only 3 domain-neutral helpers in
apply_indices_match.go.

Live maintenance (Phase 4)

Post-create DML → tag=1 CdcTail via the ISCP WAND sinker (async). idxcron
cluster-singleton fires scheduled ALTER REINDEX … MERGE. Per-index DAY/HOUR/
SECOND interval + MO_IDXCRON_TICK_SEC bootstrap-tick env (also applied to
ivfflat/cagra/ivfpq for SECOND consistency; hnsw skipped).

Primary-key support

encodePk covers int32/64, uint32/64, varchar/char/text/datalink, uuid,
date/datetime/time/timestamp, decimal64/128, and composite pk (packed varchar).
Unsupported single-column pk types (tinyint/smallint/float/bit/bool/enum) are
rejected at CREATE INDEX, not silently failed at CDC time.

Testing

• Engine unit (pkg/bm25/wand, 9 files) — build/search/merge/serialize/tail/
  deletes/compaction, incl. a string-pk MERGE regression guard.
• Plugin unit (pkg/bm25/plugin/{plan,runtime}) — schema build, pk-type
  validation, param extraction, sync descriptor (0% → ~63%/71% coverage).
• BVT — test/distributed/cases/fulltext/bm25_* (retrieval, datetime, uuid,
  multibase, limit, mode_guard, coexist, pushdown) +
  pessimistic_transaction/bm25/* (basic, async, reindex, merge, clone, restore).
  All pass 100%; pre-existing fulltext_bm25.sql (195 assertions) unchanged.

Self-review (findings addressed)

• [HIGH] string-pk MERGE panic ([]byte used as an un-normalized map key) → fixed
  with normalizeKey + extracted survivingDeletes helper + regression test.
• [MED] unsupported pk types silently aborted CDC → now rejected at CREATE.
• [MED] plugin subpackages had 0% Go coverage → added CPU unit tests.
• Cross-CN cache staleness — confirmed to be the shared vector-index cache model
  (TTL-bounded), not a bm25 regression.

Index-plugin framework compliance

• No new per-algo switch/case MoIndex<X>Algo in pkg/sql/{plan,compile} /
  pkg/catalog — routed via indexplugin.Get.
• No plugin sub-package imports pkg/sql/plan|compile.
• var _ AlgoPlugin / var _ Hooks assertions intact (7 for bm25).
• Blank-imported in indexplugin/all + indexplugin/iscp.

Checklist

• go build / go vet clean on all touched packages.
• Engine + plugin unit tests green.
• BVT suite green (fulltext-dir + pessimistic bm25 cases).
• Self-review gate run; findings fixed or decision-logged.
• Reviewer sign-off.

[cpegeric]

Can you please break up PRs into pieces that only relates to the intended feature? I don't see why we should include tens of thousands of LOC that is totally unrelated to the feature/fix.

We have to wait for the GPU PR to merge before merge this PR.

[aunjgr]

The BM25 feature is mixed with hundreds of unrelated cuVS and vector changes, producing a conflicting diff of roughly 384 files and more than 43k additions. The feature cannot be reviewed or merged safely in this shape. Please split or recreate the BM25 work on current main with only its required dependencies.

[aunjgr]

The BM25 feature is mixed with hundreds of unrelated cuVS and vector changes, producing a conflicting diff of roughly 384 files and more than 43k additions. The feature cannot be reviewed or merged safely in this shape. Please split or recreate the BM25 work on current main with only its required dependencies.