Memory embeddings: no periodic backfill — failed one-shot leaves vectors at zero

[alexeyleshchenko]

Problem

The memory embedding pipeline only runs once at startup. If the embedding API is unreachable or misconfigured at boot time, documents are permanently un-embedded until the next restart. There is no retry mechanism.

Additionally, documents created or modified mid-session (new daily memory logs, updated brain files) are never embedded — the freshness check on memory_search triggers FTS reindex but explicitly skips embedding (embedded_at stays NULL).

Current behavior

Phase	When	Embeds?
Startup backfill	Once at boot (cli/ui.rs:323)	Yes — but only for docs missing embedded_at
Freshness check	Every memory_search call	No — FTS only, skips embedding
Config reload	config_manager reload	No — embedding config is cached at startup

Impact

• 589 chunks across 66 documents, only 1 vectorized (from May 2026 with a local model)
• After fixing the api_key misconfig (bug: three API keys in keys.toml silently dropped — never merged into runtime config #1065), the fix has no effect until restart
• If the API call fails for any reason mid-backfill, those docs stay un-embedded forever
• New daily memory/YYYY-MM-DD.md files created during a long-running session never get vectors

Proposed fix

Add a background periodic task (e.g. every 5 minutes) that:

1. Queries docs where embedded_at IS NULL
2. Attempts to embed them with the configured model
3. Logs successes/failures
4. Does NOT crash the agent if embedding fails (best-effort, retry next cycle)

This is especially important because config_manager reload should be able to pick up a newly fixed embedding API key without requiring a restart.

Environment

• OpenCrabs v0.3.80
• Embedding model: text-embedding-v4 (Aliyun DashScope)
• DB: ~/.opencrabs/profiles/ops/memory.db (43 MB, 589 chunks, 1 vectorized)

[adolfousier]

Fixed in ec62af77 on local main (not pushed).

Three defects in the backfill itself, found on the way

The API backfill lived in index.rs rather than in embedding.rs, as the twin of backfill_embeddings. Keeping them apart is exactly what let them drift: #998 and #1001 were both "the placeholder sweep was wired into the local path only", filed weeks apart, because nobody editing the embedder ever saw the API copy. It moved, and two dead things went with it:

• a spawn_blocking that queried get_hashes_needing_embedding, logged the count and dropped it. Its own comment read "actual API calls happen in the next spawn_blocking cycle. For now, just log." It took the store lock for nothing.
• clear_skipped_placeholders() called twice in a row, identical, with two comment blocks crediting #998 and #1001. The second call always returned 0.

Also resolved embedding_api_config() once instead of per document: it re-reads config.toml on every call, and the old loop paid that 65 times.

The sweep

src/memory/backfill_sweep.rs, started at boot after the startup reindex is awaited so the first tick cannot race it. Interval from [memory] backfill_interval_secs, default 300, 0 disables, and vector_enabled = false disables it outright per #1062.

Your config_manager reload requirement needed no plumbing. read_memory_config parses config.toml on every call, so a key fixed mid-session is picked up on the next tick with no cache to invalidate. The interval is resolved per iteration too, so editing it takes effect within one period rather than at the next restart.

Single flight via an AtomicBool, same pattern as freshness::REFRESHING. Not politeness: freshness.rs documents that llama-cpp GGML can segfault under contention, and a tick landing on a running sweep is precisely a second embedder entering alongside the first.

Logged only when there was work. A five-minute timer reporting "0 documents need embeddings" forever is noise that trains people to stop reading the log, which is the same blindness this fixes.

Verification

This is additive, so there is no prior behaviour to revert for a red check. Instead each test was checked against a deliberate mutation to confirm it is not vacuous. Dropping vector_enabled from the interval decision:

test disabled_vectors_disable_the_sweep_regardless_of_interval ... FAILED

The other 5 stayed green on both revisions, which is the point: only the mutated behaviour may move.

Tests: src/tests/memory_backfill_sweep_test.rs, 6 cases. Default 300, explicit interval honoured, 0 disables, vector_enabled = false disables regardless of interval, an embedding section does not change the decision, and a second entry is refused rather than queued.

Stated limit: what is tested is the sweep's decision logic, not the embedding call. That call is network-bound on the API path and loads a GGUF model on the local one, so a test driving it end to end would prove the endpoint is up rather than that the sweep is correct.

Verification: cargo fmt --check exit 0. cargo clippy --locked --lib --bins --tests --all-features -- -D warnings clean. cargo test --all-features --lib 6632 passed, 0 failed, 25 ignored, up 21 from 6611.

Related: #1067 landed in the same commit, because the sweep is what feeds the health entry doctor reads and neither module compiles without the other. Not in the running binary, which was cut from 14a34ff0.