Context. MemGraphRAG's thesis is empirical: GraphRAG trades relevance for recall, validated by per-component ablations on QA benchmarks. stargraph's refs/rag.py is an explicitly-labelled vector+doc POC (one-id-per-line context, DSPy stub) with no recall/relevance evaluation. Without such a harness, none of the construction-quality features (schema filtering, conflict resolution, hub suppression, PPR) can be validated as net-positive — they could silently regress relevance exactly as the paper warns. This gate should land early; it satisfies the CLAUDE.md goal-driven-execution rule.
Proposal. Add an eval harness that, given a labeled QA set (gold passages), runs the retrieval pipeline and reports recall@k + context-relevance (fraction of retrieved context on-topic, approximated via embedding similarity to the gold answer — cheaper + air-gap-friendly vs an LLM judge). Wire as a regression gate: each construction-quality feature must show relevance does not regress while recall holds. Start with one public multi-hop slice (e.g. HotpotQA dev).
Where.
src/stargraph/skills/refs/rag.py (vector+doc POC, no eval; _format_context :134-147; dead graph branch).- New eval module gating schema-filter / PPR / conflict features. NOTE:
bench/ exists but is an agent-authorability tool-call harness (bench/run.py, oracles.py), NOT a RAG recall/relevance harness.
Acceptance criteria.
Caveat. Embedding-similarity relevance is fuzzier than an LLM judge; documented as a cheaper proxy.
Source: arXiv:2606.00610v1 — "MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation". Distilled from arXiv-research/2606.00610v1/analysis.md; file refs verified against current main by the analysis pass.
Context. MemGraphRAG's thesis is empirical: GraphRAG trades relevance for recall, validated by per-component ablations on QA benchmarks. stargraph's
refs/rag.pyis an explicitly-labelled vector+doc POC (one-id-per-line context, DSPy stub) with no recall/relevance evaluation. Without such a harness, none of the construction-quality features (schema filtering, conflict resolution, hub suppression, PPR) can be validated as net-positive — they could silently regress relevance exactly as the paper warns. This gate should land early; it satisfies the CLAUDE.md goal-driven-execution rule.Proposal. Add an eval harness that, given a labeled QA set (gold passages), runs the retrieval pipeline and reports recall@k + context-relevance (fraction of retrieved context on-topic, approximated via embedding similarity to the gold answer — cheaper + air-gap-friendly vs an LLM judge). Wire as a regression gate: each construction-quality feature must show relevance does not regress while recall holds. Start with one public multi-hop slice (e.g. HotpotQA dev).
Where.
src/stargraph/skills/refs/rag.py(vector+doc POC, no eval;_format_context:134-147; dead graph branch).bench/exists but is an agent-authorability tool-call harness (bench/run.py,oracles.py), NOT a RAG recall/relevance harness.Acceptance criteria.
Caveat. Embedding-similarity relevance is fuzzier than an LLM judge; documented as a cheaper proxy.
Source: arXiv:2606.00610v1 — "MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation". Distilled from
arXiv-research/2606.00610v1/analysis.md; file refs verified against currentmainby the analysis pass.