FTS optimize crashes with 'Max offset exceeds length of values' on lancedb >= 0.32

[gloryfromca]

Summary

On lancedb ≥ 0.32, the background FTS index optimization crashes on every cycle with:

Max offset of N exceeds length of values M
  at lance-encoding/.../encodings/logical/list.rs

Because the crash aborts optimize() — which also performs version cleanup — stale index/data files are never reclaimed, so the LanceDB index directory grows unbounded until the disk fills.

Conditions (all three required)

• lancedb >= 0.32 (lance-encoding >= 6.0)
• FTS index built with with_position=True (the previous default)
• an optimize that merges an unindexed tail — i.e. any writes after the index was first built (reproduces with a tail as small as ~100 rows)

Older lancedb (≤ 0.30.x, lance-encoding 4.x) is not affected.

Root cause

Upstream lance regression in the RepDef unravel_offsets path (lance-encoding v4.0.0 OK → v6.0.0 crash). The offsets restored from the RepDef machinery exceed the values buffer, and arrow's ListArray::try_new rejects them. Reported upstream: lance-format/lance#7653.

EverOS's FTS recall is OR-mode BM25 (MatchQuery) and never uses phrase queries, so the token positions that with_position=True stores are never read — the option was pure overhead that happened to trigger the upstream bug.

Fix

Default with_position=False for FTS indexes. This is lossless (no phrase queries anywhere in recall) and avoids building the position posting List that triggers the crash.

[gloryfromca]

Workaround — in-place hotfix for deployments that can't upgrade yet

The proper fix is #336 (upgrade once it lands). This workaround is only for deployments already hit by the crash that can't upgrade right away. Memory data is not touched — LanceDB is rebuildable from the md source of truth.

It (1) flips the config to with_position=False (persistent), (2) drops the crashing FTS indexes, (3) reclaims the bloated on-disk files via the same optimize path EverOS itself uses (no extra dependency).

Order matters: stop the service → run the script → restart. optimize(cleanup_older_than=0) reclaims safely only with no concurrent writer.

1. Stop EverOS (whichever applies): launchctl bootout … / systemctl stop everos / docker stop … / kill the server process.

2. Run this with the same Python that runs EverOS:

import asyncio, datetime as dt
import lancedb
from everos.core.persistence import MemoryRoot

# 1) Persistent: flip with_position -> False in the installed everos source
import everos.core.persistence.lancedb.base as _base
_p = _base.__file__
_s = open(_p, encoding="utf-8").read()
if "with_position=True," in _s:
    open(_p, "w", encoding="utf-8").write(_s.replace("with_position=True,", "with_position=False,"))
    print(f"[1/3] patched {_p} -> with_position=False")
else:
    print("[1/3] already with_position=False, skipping")

# 2) Drop the crashing FTS indexes + reclaim bloated files.
#    Uses AsyncTable.optimize (the same path EverOS uses) -- no pylance needed.
async def _fix():
    db = await lancedb.connect_async(str(MemoryRoot.default().lancedb_dir))
    for name in await db.table_names():
        t = await db.open_table(name)
        for idx in await t.list_indices():
            try:
                await t.drop_index(idx.name)
            except Exception as e:
                print(f"  drop {name}/{idx.name}: {e}")
        try:
            await t.optimize(cleanup_older_than=dt.timedelta(0))
        except Exception as e:
            print(f"  optimize {name}: {e}")
        print(f"[2/3] {name}: dropped + reclaimed")

asyncio.run(_fix())
print("[3/3] done — restart EverOS; indexes rebuild with with_position=False")

3. Restart EverOS. On startup it rebuilds the FTS indexes with with_position=False; optimize stops crashing and disk usage drops back.

Verify: cascade_lancedb_optimize_failed stops recurring, cascade_lancedb_optimized starts appearing, and the .index/lancedb/ directory shrinks.

⚠️ This edits the installed everos source in site-packages, so a pip install --upgrade / reinstall will overwrite it — re-run this, or (preferably) upgrade to the fixed release.