feat(vector): add opt-in multimodal attachment search

[salmonumbrella]

What changed

• Add an independently enabled visual attachment lane for standalone JPEG, PNG, WebP, and direct-input MP4 attachments, using bounded owning-message context and exact live attachment-occurrence provenance.
• Store generation-scoped visual vectors in SQLite/sqlite-vec and PostgreSQL/pgvector with explicit hosted-processing consent, fenced claims, atomic publication, stale-result suppression, retry, retirement, and crash recovery.
• Add text and image similarity search through the CLI, HTTP API, MCP, and Files workspace, with sender, source, message, date, filename, and MIME filters plus stable pagination.
• Fuse filename and visual results in Files with deterministic reciprocal-rank fusion while leaving ordinary message-search ranking unchanged.
• Keep animated GIF indexing disabled until provider capability is confirmed. PDF extraction, OCR, audio, transcription, transcoding, and generated attachment text remain out of scope.

Why

Message text and attachment metadata cannot find something that is only visible in a screenshot, photo, diagram, or short video. This adds a separate, consent-gated retrieval lane without mixing visual vectors into message embeddings or losing the source message and attachment occurrence behind each result.

The formats supported by this PR are:

• Images: JPEG/JPG, PNG, and WebP
• Video: direct-input MP4

GIF indexing is not enabled in this PR. Animated GIFs remain blocked until the provider capability is authenticated; PDF, OCR, audio, transcription, and transcoding are also out of scope.

Usage

Set VOYAGE_API_KEY, then enable the lane in ~/.msgvault/config.toml:

[vector.multimodal]
enabled = true

Restart the daemon, explicitly consent to hosted attachment processing, then search by text or image:

msgvault multimodal build --yes
msgvault multimodal status
msgvault multimodal search "diagram description"
msgvault multimodal search --image query.png

Closes #609. Refs #612. Built on #589.

Draft boundary: the authenticated Voyage contract and representative-corpus evaluation still need to pass before this is ready to merge.

[roborev-ci]

roborev: Combined Review (01774d9)

High-severity reliability gaps prevent multimodal attachment search from working safely in production.

High

• internal/vector/config.go:370 — Multimodal-only enablement is accepted, but production startup gates vector infrastructure on cfg.Vector.Enabled, and no production path invokes the visual package. With vector.enabled = false and vector.multimodal.enabled = true, the feature silently does nothing. Gate shared infrastructure with AnyLaneEnabled(), wire the visual backend, reconciler, worker, scheduling, post-sync hook, and query path into startup, and add an end-to-end enablement test.

• internal/vector/visual/reconcile.go:145 — Replay advances the journal cursor when claims are created, before work is embedded or published. A later event for the same owner can make the claim’s source fence obsolete after that event has already been acknowledged, leaving the owner stale indefinitely. Advance each page only after durable publication or rejection, or persist a retryable work queue whose source fence can be updated before acknowledgment.

• internal/vector/visual/reconcile.go:91 — Reconciliation retains every eligible attachment’s bytes until the full scan finishes, causing archive-sized memory use and allowing early leases to expire before processing. Additionally, internal/vector/visual/voyage.go:233 enforces request limits only after base64 encoding and JSON marshaling, permitting enormous transient allocations. Process bounded pages immediately, acquire claims near provider execution, and enforce a precomputed encoded-byte budget before allocation.

Medium

• internal/store/dialect_sqlite.go:799 — Invalidation, rejection, and tombstoning clear current_vector_token without retaining it for backend deletion, permanently orphaning vectors. Preserve obsolete tokens until cleanup succeeds or enqueue them transactionally in a deletion outbox.

• internal/vector/visual/reconcile.go:233 — Matching terminal provider failures are retried during every reconciliation, while authorization failures are incorrectly treated as terminal even though corrected credentials can resolve them. Skip genuinely terminal matching outcomes and classify authorization/configuration failures as retryable or run-level errors.

• internal/vector/visual/media.go:151 — GIF inspection calls gif.DecodeAll before checking pixel limits or animation consent, allowing a compressed GIF with numerous or huge frames to exhaust daemon memory. Inspect dimensions and animation structure with a bounded parser before decoding frame pixels.

---

Reviewers: 2 done | Synthesis: codex, 16s | Total: 10m56s

[roborev-ci]

roborev: Combined Review (e7543bd)

The multimodal feature is well-guarded, but high-severity reconciliation durability, performance, and migration gaps must be fixed before merge.

High

• internal/vector/visual/reconcile.go:182 — Full and replay reconciliation can advance the fence or journal cursor while candidates still have active claims. If a claimant crashes before publishing, those candidates are never revisited after lease expiry, permanently leaving the generation incomplete. Do not advance or complete reconciliation while active claims remain; retain the cursor/page and retry after expiry.

• internal/vector/visual/reconcile.go:166, internal/vector/visual/status.go:90 — Every bounded pass restarts at message ID zero, and status calculation reopens every candidate blob. With a two-owner page size, archive indexing causes quadratic media reads. Persist a durable full-reconciliation cursor and calculate coverage from persisted outcomes.

• cmd/msgvault/cmd/serve_vector_init.go:222 — Existing attachments retain the migration default role unknown, which visual candidate selection excludes, while production never invokes the resumable role repair. Existing archives can silently omit historical attachments and activate an empty generation. Run or require role repair before reconciliation, including authoritative source-specific backfills where available.

Medium

• internal/vector/visual/media.go:70 — Missing blobs are retryable only for the package-private ErrContentUnavailable, but production returns wrapped filesystem not-found errors. A missing blob therefore aborts reconciliation/status instead of recording an unavailable outcome. Recognize fs.ErrNotExist or translate storage not-found errors in the production opener.

• api/openapi.yaml:12940 — The visual-search operation declares no request body, although the handler requires JSON text or a multipart image. Define both request content types and schemas, then regenerate clients.

• internal/vector/visual/media.go:301 — MP4 parsing overwrites dimensions for each track, so a trailing audio track can replace valid video dimensions with zeroes and make the file appear malformed. Preserve valid video dimensions or select tracks by handler type, and test a video-plus-audio file with audio last.

• internal/store/visual_reconciliation.go:116 — Generation activation retires the previous generation without scheduling its vector tokens for deletion, leaking vectors and increasing storage and search work after every reconfiguration. Persist cleanup work for retired generations and delete their backend vectors retryably.

---

Reviewers: 2 done | Synthesis: codex, 14s | Total: 17m32s

[roborev-ci]

roborev: Combined Review (d6e8450)

Verdict: Changes requested — two high-severity lifecycle bugs and four medium-severity correctness/security issues remain.

High

• New consumers can remain permanently lagged after journal pruning — internal/store/attachment_changes.go:74
  A new consumer’s baseline uses only retained journal rows. If pruning emptied the journal while the global high-water mark remains nonzero, the consumer starts at sequence 0 and full reconciliation cannot advance its cursor.
  Fix: Derive the baseline from the maximum of the journal and existing consumer cursors/baselines, consistent with AttachmentChangeHighWater, and test registration after pruning.

• Generation rollover leaks consumers and vector tokens — internal/store/visual_reconciliation.go:116
  Activating a generation retires the previous one without unregistering its attachment-change consumer or deleting its tokens. The stale consumer prevents journal pruning, while retired vectors accumulate.
  Fix: Clean up the demoted generation’s tokens and visual/<fingerprint> consumer, including retryable startup cleanup for interrupted rollovers.

Medium

• Legacy attachments remain excluded from visual indexing — internal/store/dialect_sqlite.go:1436, internal/store/attachment_role_repair.go:37
  Migration assigns existing attachments the fail-closed unknown role, but the historical repair routine has no production caller. Existing archives are excluded unless attachments are re-imported.
  Fix: Run resumable MIME repair before visual reconciliation and add authoritative source-specific backfills or refreshes where roles can be recovered.

• Scoped visual search loses relevant results — internal/api/files.go:294
  Search fetches only the global top 100 visual hits, then applies predicates and person/domain scope afterward. Scoped matches below that cutoff disappear and counts are underreported.
  Fix: Push supported hard filters into the visual query and paginate until the filtered candidate pool is complete or exhausted.

• Link-preview media is incorrectly eligible for indexing — internal/beeper/media.go:121
  Although shareMetadata identifies link previews, role assignment marks every non-sticker attachment as standalone, making preview bytes eligible for hosted visual indexing.
  Fix: Pass share classification into role assignment and mark link-preview attachments as preview with importer-semantic provenance.

• Cross-origin GET can trigger unbounded archive-wide blob scans — internal/vector/visual/status.go:90
GET /api/v1/multimodal/status inspects every visual candidate and reads each blob up to 20 MiB. In keyless loopback mode, cross-origin GETs bypass the unsafe-method origin check, operation gate, and rate limiting, allowing repeated concurrent scans that can exhaust I/O, CPU, memory, and request capacity.
  Fix: Prefer stored metadata or a cached, bounded background scan. If synchronous inspection remains, require same-origin access and enforce dedicated concurrency and rate limits for keyless loopback requests.

---

Reviewers: 2 done | Synthesis: codex, 22s | Total: 19m43s

[wesm]

Looking at this

[roborev-ci]

roborev: Combined Review (4c9336e)

Multimodal indexing has two high-severity scope/convergence defects plus several medium lifecycle, coverage, performance, parsing, and API-contract issues.

High

• Account scope ignored for hosted uploads — cmd/msgvault/cmd/serve_vector.go:426
MultimodalConfig.Scope.Accounts is never resolved or propagated, so attachments and context from excluded accounts can be sent to the provider and incur billed processing. Resolve accounts fail-closed, propagate source IDs through reconciliation and candidate filters, and test exclusions.

• Terminal outcomes prevent generation activation — cmd/msgvault/cmd/serve_vector_init.go:248
  Unsupported, malformed, or oversized attachments remain counted as stale, causing builds to loop indefinitely. Exclude durable terminal outcomes from activation-blocking counts or base activation on retryable/journal convergence.

Medium

• Retired generations leak vectors and consumers — internal/store/visual_reconciliation.go:116
  Activation neither deletes old vectors nor unregisters attachment-change consumers, preventing journal pruning. Add crash-safe retired-generation cleanup.

• Production importers bypass multimodal indexing — internal/store/messages.go:4339
  The compatibility writer assigns unknown/legacy_api, while historical role repair has no production caller. Use authoritative roles in importers and run repair during visual initialization/build.

• Status calculation causes quadratic blob reads — internal/vector/visual/status.go:90
  Every bounded build pass triggers a full corpus/blob scan. Derive status from persisted metadata or avoid full recomputation after each pass.

• MP4 audio tracks can erase valid dimensions — internal/vector/visual/media.go:301
  Later tkhd boxes can replace video dimensions with zero. Select the video track via handler metadata or preserve positive dimensions.

• Visual-search OpenAPI operation lacks a request body — internal/api/routes.go:241
  Generated clients issue an empty POST that fails validation. Define JSON and multipart request bodies and regenerate clients.

---

Reviewers: 2 done | Synthesis: codex, 12s | Total: 10m10s

[wesm]

@salmonumbrella — thanks for this. Here's the plan for landing it, so nothing here is a surprise.

What just happened. I squashed the branch to one commit (your authorship preserved) and rebased it onto current main, plus one small lint fix on top. Build, lint, and the vector/store/api/cmd tests pass on the new base.

Ordering with #635. #635 (the continuation of your #616) is landing first. Both PRs add the same attachment-role / occurrence-provenance foundation — attachment_roles.go, attachment_changes.go, attachment_role_repair.go, the change-journal triggers in the dialect files, and the importer touches — but the copy in #635 is newer and carries fixes this branch doesn't have (keyed-occurrence upgrade path, idempotent reconciliation completion, duplicate raw-MIME merge). Once #635 merges, this PR gets rebased onto main and its copy of that foundation is dropped in favor of what's already there.

Moving the reusable pieces into docbank. #635 already imports docbank as a Go library for the provider-facing parts of document processing (format detection, private staging, capability probe/manifest, Mistral transport), while msgvault keeps provenance, consent, orchestration, storage, and search. We're going to do the same split here. The parts of this PR that are pure "bytes + declared type → is this an acceptable provider input" and "talk to the provider" are moving into docbank:

• media sniffing and eligibility (JPEG/PNG/WebP/GIF/MP4 probing, dimension/duration/animation detection, size and pixel caps)
• the Voyage multimodal embedding client (batching, retries, error classification, response validation)
• a capability probe that produces a manifest, so animated GIF and any future format is gated by recorded provider evidence instead of a compile-time constant — the same probe → consent → build flow feat(documents): index attachments with shared Docbank processing #635 uses for Mistral

Everything else stays in msgvault as you wrote it: attachment provenance, generation-scoped claims and atomic publication, reconciliation, the sqlite-vec/pgvector storage, search and Files fusion, consent records, schedules, and the API/CLI/MCP/web surface.

What this means for you. No action needed right now. This PR will get smaller, not different. We'll do the rebase and fix-ups on a kenn-io continuation branch (as with #616 → #635), keeping your original commit and authorship. If you'd rather drive the rebase yourself once #635 and the docbank package are in, just say so and I'll hold off.

[wesm]

Working on docbank#172

[wesm]

Kicked off a refactor of this based on docbank#172

[wesm]

The continuation is up as #650, rebased on main after #635 with your commit and authorship preserved. The attachment-provenance foundation now comes from what #635 landed, and media detection, the Voyage client, and capability gating moved to Docbank's shared packages — the compile-time animated-GIF constant is now recorded probe evidence, with consent bound to the manifest fingerprint. No action needed; follow along on #650 if interested.

[wesm]

@salmonumbrella I need sleep but I will try to wrap up the docbank change and this PR in the morning