Skip to content

docs: fix drift in Hybrid Search - #321

Draft
jack-arturo wants to merge 1 commit into
mainfrom
docs/audit-core-concepts-hybrid-search-20260821
Draft

docs: fix drift in Hybrid Search#321
jack-arturo wants to merge 1 commit into
mainfrom
docs/audit-core-concepts-hybrid-search-20260821

Conversation

@jack-arturo

Copy link
Copy Markdown
Member

Automated docs-accuracy audit of core-concepts/hybrid-search. First review of this page — no prior review state existed.

The "sum to 1.95, then normalize to [0.0, 1.0]" claim was already proven false on core-concepts/recall-tuning (PR #290). It survives on this page in three more places; _compute_metadata_score() returns the raw weighted sum with no normalization step anywhere in the function.

All ten weight defaults, all six RECALL_* limits, the limit default of 5, and all nine source-note file paths were checked and are correct — the drift is concentrated in how the score is combined and where recency comes from.

claim current state fix evidence
"Raw combined scores are normalized to the range [0.0, 1.0] during final ranking" (repeated in 3 places: formula note, weights-table note, config note) _compute_metadata_score() builds final as a plain weighted sum and returns it — no normalization, clamping, or division follows. With saturated components it exceeds 1.0 Replaced all three with an accurate statement: raw weighted sum, comparable within one result set only. Mermaid "Normalize to 0.0 - 1.0" node removed automem@42ba8b6:automem/utils/scoring.py#L246-L258
Formula reads context_bonus × SEARCH_WEIGHT_RELEVANCE (default: 0.0); weights table labels the row "Context — Context profile scoring bonus" Two distinct terms conflated. SEARCH_WEIGHT_RELEVANCE multiplies relevance_score (consolidation decay relevance). context_bonus is added unweighted and is unaffected by that variable — so the table implied the context bonus is disabled by default when it is not Split into two rows/terms in the formula, weights table, and config table automem@42ba8b6:automem/utils/scoring.py#L253-L257
Recency is "based on the time since last access (or creation if never accessed)"; "Missing last_accessed falls back to timestamp" _compute_recency_score(memory.get("timestamp")) — the string last_accessed does not appear anywhere in scoring.py. Access time never influences ranking Stated that recency decays from timestamp only; documented SEARCH_RECENCY_WINDOW_DAYS (180) and SEARCH_RECENCY_CURVE (linear/exp) as the real controls automem@42ba8b6:automem/utils/scoring.py#L66-L81, automem/config.py#L499-L503
"Missing importance defaults to 0.5"; "Missing confidence defaults to 0.7" Both fall back to 0.0 when absent or non-numeric. (0.5 / 0.7 are write-time defaults on POST /memory, not scoring fallbacks — a memory lacking the field scores zero for that component) Corrected the fallback values and retitled the list to make the write-time/score-time distinction clear automem@42ba8b6:automem/utils/scoring.py#L160-L166
Nine source-note deep links pinned to 0720da2 Stale pin Repinned all nine to 42ba8b6; every path verified to still exist

Verified against: automem@42ba8b61b7d0b24ecaeb7feb4ceef59f09fc7cd0

Questions

  • "10-Component Scoring System" is arguably an 11-term sum. The components dict returns eleven entries (vector, keyword, metadata, relation, tag, importance, confidence, recency, exact, relevance, context). But "10-Component" is used as a site-wide framing — it also appears in reference/api/recall-operations and architecture/overview. Renaming it here alone would make this page inconsistent with two pages outside this audit's scope, so the heading was left as-is. Worth deciding globally.
  • SEARCH_WEIGHT_TEMPORAL (default 0.1) exists in config.py but is absent from this page. It only applies when the recency_bias re-rank runs, which is a separate path from _compute_metadata_score() — so its omission may well be deliberate scoping rather than drift. Not touched.

Unverified

  • The Query Response Times table (20-50ms vector-only, 100-300ms with bridge expansion, etc.) is not derivable from source; no benchmark harness in the repo produces these figures. Left alone.
  • The Benchmark Results table cites LongMemEval 87.00% and LoCoMo 84.74%. Per repo convention those claims live in automem and are sourced from its verification trail, not from this checkout. Left alone.
  • "Optimization tips" claims explicit embedding saves 200-500ms — no measurement in source. Left alone.

Follow-ups

Beyond the 5-fix cap:

  • RECALL_RELEVANCE_GATE is undocumented on this page. When set above 0, results whose topical evidence falls below the gate get importance, confidence, recency, tag, and relevance scaled down linearly by evidence / gate before weighting. This materially changes scoring inside tag-scoped pools and deserves its own subsection.
  • SEARCH_TAG_SCORE_TOKEN_CAP caps the tag-score denominator so long queries aren't penalized — also undocumented.
  • The evidence signal (max(vector, keyword, metadata, exact), deliberately excluding tag overlap as scope-confounded) is a notable design decision with an explanatory comment in source, and has no counterpart in the docs.

Generated by Claude Code

- final_score is a raw weighted sum, not normalized to [0.0, 1.0] (3 places)
- SEARCH_WEIGHT_RELEVANCE weights relevance_score; context_bonus is unweighted
- recency decays from timestamp, never last_accessed; window/curve configurable
- missing importance/confidence score 0.0, not 0.5/0.7
- repin source-note permalinks to automem@42ba8b6

Verified against automem@42ba8b61b7d0b24ecaeb7feb4ceef59f09fc7cd0

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iwvkKvNhquqnG4MQ6EEun
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying automem-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 7237c1a
Status: ✅  Deploy successful!
Preview URL: https://2abbea2f.automem-website.pages.dev
Branch Preview URL: https://docs-audit-core-concepts-hyb-t6y2.automem-website.pages.dev

View logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants