From 5be5f501036ba67c5b19926a52f7f10a9f9d4054 Mon Sep 17 00:00:00 2001 From: jp Date: Sat, 4 Jul 2026 08:21:59 -0700 Subject: [PATCH 1/2] feat(site): publish the field catalog + Phase-1 three-way comparison pages The ranked memory-systems catalog (with the engram-2 deaccession card and metric-type badges) and the OMEGA vs mempalace vs Go-server structural comparison, previously served locally. Self-contained pages alongside the field explorer. Co-Authored-By: Claude Fable 5 --- docs/site/catalog.html | 261 +++++++++++++++++++++++++++++++++++++++++ docs/site/phase1.html | 86 ++++++++++++++ 2 files changed, 347 insertions(+) create mode 100644 docs/site/catalog.html create mode 100644 docs/site/phase1.html diff --git a/docs/site/catalog.html b/docs/site/catalog.html new file mode 100644 index 0000000..8b189a0 --- /dev/null +++ b/docs/site/catalog.html @@ -0,0 +1,261 @@ +Field Catalog — AI Memory Systems + + +
+
+
+ Drawer
+ MEM · 2026-07-03 +
+ Field catalog · techempower-org / mempalace +

The Memory Systems Worth Watching

+

Every entry verified against our own research corpus — 411K drawers of production palace, the multipass structural eval, and the benchmark-methodology record. The claims below say which metric they are, because most of this field's score inflation lives in that omission.

+
+ +
+ R@5 · retrieval + QA · end-to-end + self-reported + withdrawn +

Retrieval recall (“was the right document in the top 5?”) runs 20–30 points hotter than end-to-end QA (“did the agent answer correctly?”). Comparing one to the other is the field’s favorite accidental lie.

+
+ +
+

The one that vanished

+
+

engram-2

+

Boris Djordjevic · 199 Biotechnologies (199.bio) · Rust CLI

+ + + + + + +
ClaimedLongMemEval-S R@5 0.990 · R@10 0.998 · MRR 0.946 — config: Gemini Embed 2 + FTS5 + RRF (not the pure-local mode) · LoCoMo-QA 74.5% (strict judge, 200q subset)
Entered orbitPosted by @longevityboris in MemPalace issue #29, Apr 11 — the benchmark-methodology wars
Last seenApr 12, 2026 (dial481 browsing its feat/mlx-reranker branch)
Gone byJun 10, 2026 — 404 recorded in our commit 9781ce5; zero Wayback snapshots; no surviving forks, crate, or package
Survives inOur palace’s verbatim capture of its README claims, and these notes
+

A memory system whose only remaining memory is inside ours. The highest R@5 ever claimed in this field is now unverifiable — which, by the standards everyone here holds everyone else to, means it’s a footnote.

+
+
+ +
+

On our bench right now

+
+
+ Go server · sefodo26 +

Postgres + pgvector + Apache AGE — our production substrate, independently reinvented. Live on katana, smoke-tested, 37 MCP tools. repo · discussion #1856

+
+
+ SME adapter · in flight +

Contract adapter for the Go server under TDD in a worktree; engram (TS) queued next. Then the multipass run: corpora × A/B/C conditions × cohort.

+
+
+ Upstream ask · open +

Our divergence map + upstreaming offer is live as discussion #1918. Awaiting maintainer picks.

+
+
+
+ +
+

Next most interesting · ranked

+
+ +
01 +

OMEGA

+

omega-memory · open-core (Apache 2.0)

+

SQLite + ONNX, zero external deps, 25 MCP tools, AES-256-GCM at rest

+

95.4%QAself

+

The only local-first system claiming to beat the frontier on end-to-end QA. If that number survives a hostile rerun, it’s the story of the year; if not, it’s our best target practice.

+

github.com/omega-memory/core

+
+ +
02 +

Hindsight

+

vectorize.io · Virginia Tech / Washington Post lineage

+

retain / recall / reflect ops; auto-updating “mental models”; Docker

+

91.4%QA

+

The most credible QA number in the field — indie-verified, arXiv-backed, and they run a public benchmark repo that scores other systems too. The adult in the room.

+

their benchmark repo

+
+ +
03 +

agentmemory

+

@rohitg00 · 9.4K★ · 53 MCP tools

+

BM25 + vector + KG triple-stream, RRF fusion, token-budgeted injection

+

95.2%R@5

+

Labels its metrics honestly, reproduced MemPalace’s numbers independently, and wrote the methodology review (#747) the whole field needed. Our most natural benchmarking peer.

+

github.com/rohitg00/agentmemory

+
+ +
04 +

engram-ai

+

@tonitangpotato · Rust core, on crates.io as engramai

+

ACT-R activation, Ebbinghaus forgetting, Hebbian learning; FTS5, no embedder required

+

unbenchmarked

+

The cognitive-science bet: memory strength as an activation model, not a similarity score. Opposite pole to our verbatim thesis — which is exactly why it belongs in the cohort.

+

github.com/tonitangpotato/engram-ai

+
+ +
05 +

engram (Go)

+

Gentleman-Programming

+

Single Go binary; SQLite + FTS5; CLI, HTTP, MCP, and a TUI

+

unbenchmarked

+

Agent-agnostic and deterministic — the closest living relative to what the vanished engram-2 claimed to be. Easiest possible SME stand-up: one binary, no embed server.

+

github.com/Gentleman-Programming/engram

+
+ +
06 +

engram (TS)

+

Boris Djordjevic · 199.bio — engram-2’s author, current artifact

+

TS MCP server; BM25 + LEAF embeddings (Transformers.js) + KG; Ebbinghaus decay

+

unbenchmarked

+

What 199-bio ships now. Benchmarking it live is the closest we can get to testing the vanished 0.990 claim’s lineage. Forked; adapter queued.

+

npm @199-bio/engram

+
+ +
07 +

ai-memory

+

alphaonedev · Rust binary · 43 MCP tools

+

SQLite FTS5, tiered TTL, autonomous curator daemon, Ed25519 attestation

+

97.8%R@5self

+

Cryptographic attestation of memories is a genuinely novel axis — provenance as a first-class feature. The self-reported score needs the usual haircut.

+

github.com/alphaonedev/ai-memory-mcp

+
+ +
08 +

Letta

+

letta-ai (née MemGPT) · ~22.8K★

+

V1 rework (Mar ’26); core/recall/archival tiers; new memory-first coding agent

+

QA-class

+

The incumbent mid-rearchitecture — heartbeats deprecated, MCP moving client-side. Watching which memory ideas survive their own rewrite is free research.

+

letta-code

+
+ +
09 +

engram-mcp

+

@edg-l · Rust

+

Branch-aware session handoffs; local ONNX embeddings; SQLite semantic search

+

unbenchmarked

+

“Branch-aware handoff” is a coding-agent-shaped memory primitive nobody else has — memory keyed to git state, not just time. Small, sharp idea.

+

github.com/edg-l/engram-mcp

+
+ +
10 +

claude-mem

+

thedotmack · 89K★ — the popular vote

+

SQLite + ChromaDB; AI-compressed summaries; explicitly non-verbatim

+

unbenchmarked

+

The anti-thesis at scale: derivative memory with the largest community in the space. Every recall failure it ships is quiet evidence for the verbatim side of the axis.

+

github.com/thedotmack/claude-mem

+
+ +
+
+ + +
diff --git a/docs/site/phase1.html b/docs/site/phase1.html new file mode 100644 index 0000000..e249076 --- /dev/null +++ b/docs/site/phase1.html @@ -0,0 +1,86 @@ +Phase 1 — OMEGA vs mempalace vs Go server + +
+
+ Phase 1 · structural categories · no-LLM · 2026-07-03 +

OMEGA vs mempalace vs the Go server

+

Deterministic SME structural readings — contradiction detection, ingestion integrity, topology, supersession, ontology coherence. Phase 2 (LLM-judged QA, the hostile rerun of OMEGA's 95.4% claim) is gated on API-spend approval. ← Field Catalog · SME site

+
+ +

Column bases — read this first

+
+
OMEGA · measuredomega-memory 1.4.15, isolated store. Good-dog vault (24 notes) ingested; edges are OMEGA's own emergent auto-relate output.
+
mempalace fork · measuredProduction palace via daemon (412K drawers), real AGE knowledge graph, 10K-edge representative sample (full KG ≈ 59K edges). Read-only. Same posture as the 2026-05-31 baselines.
+
Go server · measured (basis-attributed)Good-dog ingested (server wiped first). Adapter now reads the real AGE KG (basis-attributed) — and the KG is empty by design: the Go server does no auto-extraction; its graph fills only via explicit kg_add. The structural cells are honest basis=taxonomy readings of that capability difference.
+
+ +

Readings

+
+ + + + + + + + + +
ReadingOMEGA
good-dog emergent		mempalace
real KG (sample)		Go server
basis=taxonomy (no auto-extraction)
Graph size (entities / edges)18 / 5412,659 / 10,000*3 / 1 †
Cat 4 — canonical collisions0222n/m †
Cat 4 — required-field coverage1.001.001.00 †
Cat 4 — edge-type entropy (bits)1.243.430.0 †
Cat 5 — components / largest1 / 186,072 / 3,900*2 / 2 †
Cat 3 — structural contradiction detection1.00 Δ+1.00 vs flat0 pairs — no contradicts edge type in prod KGn/m †
Cat 3 — vs good-dog ground truth (theme recall)0.50 caught DCM, missed dominancedifferent basisemergent KG empty — no extraction
Cat 6 — supersession completeness0 seeded — corpus text carries none emergentlyno supersedes edge typen/m †
+

* 10K-edge representative sample (CLI default); fragmentation partly reflects sampling. † basis=taxonomy reading — the adapter reads the real KG first (SME #257), which is empty because the server has no auto-extraction; cells reflect that capability difference, not an adapter gap.

+ +

What Phase 1 actually established

+
+

OMEGA is a disciplined structural citizen at toy scale

Single connected component, zero collisions, and its emergent contradicts edges surface all 4 of its own pairs (structural Δ+1.00 over flat) — but only half the ground-truth contradiction themes. The 95.4% QA claim remains untested until Phase 2.

+

New fork finding: 222 canonical collisions in the production KG

The real palace graph carries 222 canonical-key collisions and heavy fragmentation in the sampled read (6,072 components). Some fragmentation is sampling artifact; the collisions are real and actionable — entity-canonicalization follow-up material (composes with the kg_canonical_* write-pass).

+

Measured: the Go server does no automatic knowledge-graph extraction

Verified in source and live (SME #257): add_drawer never populates the AGE graph — only explicit kg_add_entity/kg_add_relation does. So on a raw-text corpus its emergent graph is empty, vs OMEGA's auto-related 18/54 and our fork's write-through LLM triple extraction. A real three-way capability difference — extraction (fork) vs auto-relate (OMEGA) vs storage-only (Go) — now attributable via the recorded snapshot basis. Plus two integrator gotchas: deterministic-ID idempotency and bullet-list splitting.

+

Cat 3/6 zeros on mempalace are basis, not failure

The production KG's edge vocabulary doesn't include contradicts/supersedes — those semantics live in good-dog-style seeded corpora. Matching the 05-30 matrix's honesty rules: those cells are “no such channel,” not “detected nothing.”

+
+ +
+

Provenance. All JSONs in the session scratch (phase1/): omega × 6 (incl. emergent report + ground-truth Cat 3), mempalace real-KG × 5, go-server × 5. Runners: run_omega_good_dog_structural.py (existing) and its new twin run_mempalace_server_good_dog_structural.py (same rows, same scorer). OMEGA 1.4.15 · palace-daemon @ familiar (read-only) · Go server @ localhost:8000 (disposable instance, wiped).

+

Phase 2 (pending approval): Cat 9 + LongMemEval QA with reader gpt-4o-mini / judge gpt-4.1-mini — the actual hostile rerun of OMEGA's 95.4%.

+
+
From bc710ab3b09edb524f62ab6bce4d4846c19cdc71 Mon Sep 17 00:00:00 2001 From: jp Date: Sat, 4 Jul 2026 08:22:37 -0700 Subject: [PATCH 2/2] =?UTF-8?q?fix(site):=20catalog=20bench=20strip=20?= =?UTF-8?q?=E2=86=92=20link=20Phase-1=20results?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/site/catalog.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/site/catalog.html b/docs/site/catalog.html index 8b189a0..2d1996f 100644 --- a/docs/site/catalog.html +++ b/docs/site/catalog.html @@ -147,8 +147,8 @@

On our bench right now

Postgres + pgvector + Apache AGE — our production substrate, independently reinvented. Live on katana, smoke-tested, 37 MCP tools. repo · discussion #1856

- SME adapter · in flight -

Contract adapter for the Go server under TDD in a worktree; engram (TS) queued next. Then the multipass run: corpora × A/B/C conditions × cohort.

+ SME adapters · merged +

Go server + engram (TS) adapters merged (#255–257). Structural three-way complete — Phase 1 results. QA axis next.

Upstream ask · open