Fix sqlite-vec search performance with proper partition-key migration

[arabold]

Summary

The hotfix in 32050f6 restored stable search by avoiding direct documents_vec.library_id/version_id filters in KNN queries. The previous performance change assumed these columns were sqlite-vec partition keys, but the current schema defines them as regular metadata columns:

CREATE VIRTUAL TABLE documents_vec USING vec0(
  library_id INTEGER NOT NULL,
  version_id INTEGER NOT NULL,
  embedding FLOAT[...]
);

Using direct metadata filters in the KNN query caused search to hang after version resolution in the unified server / MCP search_docs path.

Goal

Restore the intended performance optimization safely by migrating documents_vec to use sqlite-vec partition keys and then re-enable direct partition-filtered KNN search.

Proposed Work

• Add a migration that detects/rebuilds documents_vec as:

CREATE VIRTUAL TABLE documents_vec USING vec0(
  library_id INTEGER partition key,
  version_id INTEGER partition key,
  embedding FLOAT[<configured dimension>]
);

• Backfill vectors from documents.embedding joined through pages and versions.
• Update ensureVectorTable() so new databases and dimension reconciliation create the partition-key schema.
• Restore the direct KNN filter in DocumentStore.findByContent() only after the schema is correct:

WHERE dv.library_id = ?
  AND dv.version_id = ?
  AND dv.embedding MATCH ?
  AND dv.k = ?

• Add tests for old schema detection/rebuild, vector preservation/backfill, and hybrid search using the partition-filtered query.
• Consider migration cost and logging for large databases, since rebuilding documents_vec can be expensive.

Context

Observed hang log stopped after:

🔍 Searching 10four@latest for: agent
🔎 Validating existence of library: 10four
✅ Library '10four' confirmed to exist.
🔍 Finding best version for 10four
ℹ️ Unversioned documents exist for 10four

The next step is DocumentStore.findByContent(), specifically embedding + sqlite-vec KNN search.

Hotfix commit: 32050f6 fix(store): avoid vec metadata filters in search