feat(longmem): wire the Go MemPalace server into the LongMemEval harness#260
Conversation
- Factory self-ingests the per-question vault (hindsight pattern), source_file = session stem, per-question wipe via reset_before_ingest - Adapter surfaces the corpus/session id as Entity.id (scorer contract: 'their .id is the session_id'); server's content-hash drawer id moves to properties for provenance - 5q retrieval smoke: 80% R@5 (4/5) — pipeline verified live Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 25 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request integrates the mempalace-server adapter into the evaluation harness, enabling per-question isolation via a full wipe, and updates the adapter's query method to use the source file stem as the entity ID for correct retrieval scoring. The remaining changes primarily involve extensive code formatting across CLI, script, and test files. The review feedback suggests explicitly checking for the required API key environment variable to fail fast, using a platform-independent path parsing approach instead of pathlib.Path for extracting the filename stem, and avoiding runtime modifications to sys.path.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| adapter = MemPalaceServerAdapter( | ||
| api_url=_os.environ.get("MEMPALACE_SERVER_URL", "http://localhost:8000"), | ||
| api_key=_os.environ.get("MEMPALACE_SERVER_API_KEY", "local-dev-key-change-me"), | ||
| reset_before_ingest=True, | ||
| read_only=False, | ||
| ) |
There was a problem hiding this comment.
When fetching credentials from environment variables, explicitly check for their presence and raise a RuntimeError if missing to ensure the application fails fast. This aligns with the repository's general rules.
| adapter = MemPalaceServerAdapter( | |
| api_url=_os.environ.get("MEMPALACE_SERVER_URL", "http://localhost:8000"), | |
| api_key=_os.environ.get("MEMPALACE_SERVER_API_KEY", "local-dev-key-change-me"), | |
| reset_before_ingest=True, | |
| read_only=False, | |
| ) | |
| api_key = _os.environ.get("MEMPALACE_SERVER_API_KEY") | |
| if not api_key: | |
| raise RuntimeError("MEMPALACE_SERVER_API_KEY environment variable is missing") | |
| adapter = MemPalaceServerAdapter( | |
| api_url=_os.environ.get("MEMPALACE_SERVER_URL", "http://localhost:8000"), | |
| api_key=api_key, | |
| reset_before_ingest=True, | |
| read_only=False, | |
| ) |
References
- When fetching credentials from environment variables, explicitly check for their presence and raise a RuntimeError if missing to ensure the application fails fast.
| # server's own drawer id is a content hash and can never match, | ||
| # so surface the source stem as the entity id and keep the | ||
| # drawer hash in properties for provenance. | ||
| entity_id = Path(source_file).stem if source_file else drawer_id |
There was a problem hiding this comment.
Using platform-specific path parsing with pathlib.Path can lead to issues if the client and server run on different operating systems (e.g., Windows vs. Unix) and use different path separators. Extracting the filename stem in a platform-independent manner is more robust.
| entity_id = Path(source_file).stem if source_file else drawer_id | |
| entity_id = source_file.replace("\\\\", "/").split("/")[-1].rsplit(".", 1)[0] if source_file else drawer_id |
| _scripts_dir = str(Path(__file__).resolve().parent.parent / "scripts") | ||
| if _scripts_dir not in _sys.path: | ||
| _sys.path.insert(0, _scripts_dir) |
There was a problem hiding this comment.
Avoid runtime modifications to sys.path for project imports; rely on proper package installation (e.g., editable installs) or PYTHONPATH instead. This aligns with the repository's general rules.
| _scripts_dir = str(Path(__file__).resolve().parent.parent / "scripts") | |
| if _scripts_dir not in _sys.path: | |
| _sys.path.insert(0, _scripts_dir) | |
| # Avoid runtime modifications to sys.path; rely on proper package installation or PYTHONPATH instead. |
References
- Avoid runtime modifications to sys.path for project imports; rely on proper package installation (e.g., editable installs) instead.
Adds a self-ingesting
mempalace-serverfactory tocross_validate_longmemeval(per-question haystack, session-stem source ids, wipe-per-question — mirroring the reference benchmark's methodology) and fixes the adapter'sEntity.idto carry the corpus/session id the session-level R@K scorer matches (drawer hash preserved in properties). Live 5q smoke: 80% R@5. Enables the retrieval-only (no-LLM) independent verification of the Go server's published 0.972.🤖 Generated with Claude Code