Round 2: time-aware retrieval, coverage mode, roll-up cards, grounded honesty#270
Merged
Conversation
…preference, adapter rendering
Context packs now tell the reader which facts are current, superseded, or
time-bounded, derived entirely from values the rows already carry (no extra
store queries):
- compile_context_pack attaches a compact `validity` field to pack memories
with temporal/supersession signal only: valid_from/valid_to (the
far-future unbounded sentinel and NULL are omitted), superseded +
superseded_by_memory_id (own column, migration 20260704_0077, or a
pack-mate's supersedes back-pointer for one-sided patches),
supersedes_memory_id, and corrected_at from the agentic in-place
correction history. Plain memories keep their exact shape.
- Current-version preference (demote-not-drop): the store search stages
already exclude retired rows (status outside active/accepted, closed
validity windows), so supersession pairs only co-occur in a pack via
one-sided pointer state. When that happens the replacement packs directly
above its superseded ancestor while every other item keeps fused (RRF)
order; each move is cycle-guarded and counted in the pack trace as
supersession_reorders. The RRF ranking itself and its trace ranks are
untouched.
- LongMemEval adapter renders the annotation as a compact factual suffix on
the facts-section item lines ("[valid 2023-05-30 -> 2023-08-01]",
"[superseded by a newer entry]", "[updated 2023-08-01; supersedes an
earlier value]"); the official reading templates stay byte-identical and
are now pinned by a test.
- Tests: annotation only-on-signal (schema stability incl. the 9999
sentinel), pair reorder + trace counter + honest fused trace ranks,
one-sided pointer annotation, corrupt-cycle termination, proper-flow
supersede exclusion on real SQLite, KU-shaped leak scenario on real
SQLite and through the real adapter end-to-end, adapter suffix shapes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nswer line
The LongMemEval context packer guaranteed each retrieved source its best
chunk by whole-chunk term overlap, always shown from the chunk head. When
the query's best-matching line sat near the chunk tail (a chess record
matching "27. Kg2 Bd5+"), the answer just past the chunk boundary
("28. Kg3") was excerpted away; enumerated lists were likewise cut mid-run
before the asked-about item.
Now, when a gated query anchor fires, the source's guaranteed excerpt
becomes a window centered on the best-matching line over the session's
chunk stream, and a window edge that cuts an enumerated run extends
through the run. Discipline:
- Triggers are query-surface/content-shape only (weighted query-term line
overlap; line-initial/inline numbering density). No benchmark labels.
- Gated: single-chunk sources, weak matches (weighted score < 3), and
prose whose match already sits in the best chunk take the byte-identical
old path (tested via anchoring-off comparison).
- Round-robin preserved: the anchored window is capped at the baseline
chunk's entry cost, so pass-1 admissions are unchanged; enumeration
extensions spend only the global second-pass pool, and pass 2 skips
chunks a window already shows.
- Deterministic: ties break to the earliest line; expansion prefers the
lighter side, then downward.
- Official reading templates untouched (byte-frozen test added).
Keyless replay on the two verified failures: 1568498a's context now
contains "28. Kg3" (previously cut at move 27); 1903aded keeps
"Transcriptionist". A 16-question before/after sweep shows one gold-answer
gain, zero gold-answer losses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tention
New vnext_grounding module computes a conservative, deterministic,
read-only retrieval statistic: salient query entities (capitalized
spans, quoted titles, domains, honorific surnames, mid-sentence
capitalized tokens -- never acronyms, contractions, or blocklist words)
with ZERO corpus support. Support = entity substrate (names + aliases)
OR a one-row match_any FTS probe over source chunks and memories with
morphological token variants ("Hawaiian" supports "Hawaii"), so every
error leans toward NOT claiming absence.
compile_context_pack attaches pack["grounding"] = {unsupported_entities,
checked} via one clearly-marked block, mirrored into the trace, ONLY
when at least one entity is unsupported; ungated queries take the
byte-identical old path (no key, zero probes -- test-proven). Skipped at
minimal depth. The LongMemEval adapter renders the field as factual
'Note: no stored memories mention "X".' lines inside the retrieval
output (budget-reserved); reading templates stay byte-frozen and are now
pinned by a test.
Offline FTS-only replay over all 181 entity-bearing longmemeval_s
questions: the note fires on 5/15 abstention questions (all truthful,
incl. 2 previously judged wrong) and 2/166 answerable ones (both
lexically accurate). Retrieval-quality eval suite metrics byte-identical
to main; full unit suite (1708) and harness (46) green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes the verified LongMemEval failure shape where a user's stated value
(gold '$50') loses to assistant fare-chatter arithmetic ('¥20,000-30,000
(approximately $180-270)'): user-asserted facts now capture with speaker
provenance and outrank assistant estimates for the same topic slot.
- New alicebot_api.memory_provenance module derives provenance_role from
leading "[USER]:"/"[ASSISTANT]:" transcript tags (content shape only;
no benchmark metadata or question types are ever read) and classifies
user_asserted (first-person pronoun + assertion verb + concrete value:
currency, percent, unit-bearing quantity, or thousands-separated
number) vs assistant_estimate (hedged or ranged figures).
- vnext_capture annotates candidates with role/assertion class, applies
a bounded confidence bias (user-asserted +0.08 capped at 0.95,
assistant estimates -0.06 floored at 0.30 -- bias, not suppression),
stamps provenance_role/assertion_class into memory metadata_json only
when a role is derived, and adds one gated extraction rule
(user_asserted_value) so speaker-tagged first-person user lines with
concrete values become semantic candidates. Untagged content takes the
byte-identical legacy path (tested).
- trusted_fact_promotions orders each pattern group by provenance rank
(user-asserted first, assistant estimates last); provenance-free
groups keep the legacy (memory_key, id) order byte-for-byte (tested).
- Web memory cards (memories list/detail and the vNext candidate list)
show a "You said" / "Assistant suggested" chip when the role is known,
via the shared lib/memory-provenance helper.
Offline verification: 1750 unit + LME harness tests pass; eval suite
(--suite all, sqlite) passes with check statuses byte-identical to main;
FTS-only coverage probe on the 103 currently-wrong questions keeps
any-coverage at 94.2% with one enumeration-question all-coverage flip,
and a 100-question currently-correct sample shows zero coverage flips
(any 98.0 / all 90.0, identical to main).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Date-bearing query text ("in March 2023", "on May 3", "between March and
May 2023", "two months ago", "before today") now parses into one UTC
[start, end) window that joins RRF fusion as ONE MORE ranked list next to
fts/vector/graph — never a hard filter, so a wrong parse cannot evict
lexical, vector, or graph hits.
- vnext_temporal_query.py (new): deterministic parse_temporal_anchor
(explicit dates, month-year, yearless forms resolved against a
caller-supplied reference_time, closed/open ranges, relative phrases;
ambiguous phrases like bare "May" return None) plus
parse_event_datetime for stored/connector date strings. No LLM, no
wall-clock reads, no benchmark metadata.
- search_memories_by_time on both stores: event time is
COALESCE(valid_from, first_seen_at, created_at), window-intersection
match including closed [valid_from, valid_to) overlap, ordered by
proximity to a pivot (window midpoint; open windows pivot on their
closed edge). Same scoping discipline as the sibling search methods.
- Retrieval integration: temporal_anchor stage record in the trace
({source, status, window, parsed_from, candidate_count}; absent when no
anchor parses; honest disabled statuses for minimal depth and stores
without the method), and a temporal_anchor rank-boost list in the fused
sources stage re-ranking already-found candidates by connector-stamped
event dates (source_created_at, then metadata session_date/event_date/
date, then captured_at).
- VNextRetrievalRequest.reference_time: the caller's now for resolving
relative phrases; defaults to the current UTC time.
Tests: table-driven parser suite (69 cases), store method coverage on
both backends (scoping, ordering, boundaries, expiry, pivot), retrieval
fusion proofs (right-dated beats lexically-stronger wrong-dated; no
anchor leaves trace and behavior untouched; a wrong window cannot evict
strong hits; "before today" excludes same-day ingests). Validated against
real docker Postgres (migrated to head) and the LongMemEval coverage
probe: byte-identical per-question coverage on a 140-question slice at
reference=now, and a diagnostic with reference_time=question_date lifts
multi-session all-coverage 85.7%→100% and temporal any-coverage
94.7%→95.5% with zero worsened questions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Aggregation questions (how many / how much total / list every / each of / in what order / which of my ... was the most) lose evidence instances to redundant same-source memories in the pack's fixed slots. This adds a query-surface-gated coverage mode to compile_context_pack: - vnext_coverage_query.py (new): conservative aggregation-intent detector (dormant = None; query text only, never benchmark labels), clause decomposition, fair clause-row interleaving, and a generic post-fusion instance-diversity pass (near-verbatim text Jaccard and/or provenance group key) that only ever reorders and reselects fused candidates. - vnext_retrieval.py: marked coverage blocks — deepen the memory pool x3 under intent, demote same-source memory re-statements (group-key only) behind distinct-source instances over the whole deepened pool, backfill clause sub-retrieval rows behind fused winners (never score-fused), near-verbatim source demotion, honest trace stage (absent when dormant), provenance winners captured pre-diversity so the source stage stays decoupled. Selected sessions are a superset of the ungated pack's. Measured (free FTS-only coverage probe, paired on identical stores): - 55 judged-wrong multi-session questions: all-evidence coverage 58.2% -> 67.3% (+9.1pt; second ingest seed 60.0% -> 69.1%, also +9.1pt), 5 wins / 0 losses, evidence-session hits 123 -> 132. - All 172 detector-fired questions: all-coverage 79.7% -> 83.1%, any-coverage 98.8% -> 99.4%, 6 wins / 0 losses, 0 questions lost hits. - Naive design variants measured and rejected: RRF-fused clause lists (-3.7pt), source-pool deepening (-1.9pt), memory text-similarity demotion (evidence losses via provenance coupling). Guards: dormant path proven byte-identical (uuid-pinned A/B test plus hard-disabled control); eval suite passes with retrieval_quality metrics unchanged; 1734 unit tests and the LongMemEval harness tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New eval/longmemeval/verification.py: a separate post-generation chat call checks the answer's concrete claims against the retrieved context (never the gold answer or any benchmark label) and, only when a LOAD-BEARING claim is unsupported, replaces the hypothesis with the dataset-style abstention phrasing. Off by default; enabling it stamps verify_grounding plus the verifier model into the config fingerprint, every gated row records the original hypothesis and the full verdict, and any verifier error or unparseable reply fails open with the original answer byte-identical. The official reading and judge templates are untouched and now byte-frozen by sha256 tests; mock-client tests cover pass-through, gating, fail-open, fingerprint disclosure, and verifier-payload hygiene. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Roll-up cards pre-aggregate same-topic memory instances ("hours played -
5 instances in total: ...") so aggregation-phrased recall hits ONE
canonical card instead of needing every instance to win a pack slot.
- vnext_rollups.py (new): deterministic same-entity (vnext_entities
substrate, alias-folded via one bulk find_entities_by_names call) and
lexical-topic grouping - no LLM required. Optional model refinement of
the card summary rides the existing routing seam with a grounding
check and deterministic refusal fallback; no API key needed anywhere.
- Cards are status="candidate" proposals carrying the standard
metadata_json.consolidation block, so the existing review surface and
accept_consolidation_candidate handle them unchanged: acceptance
promotes the card to a first-class FTS/vector-indexed memory and
supersedes NOTHING - members stay individually recallable. A new
member in an accepted topic produces a REVISION proposal whose
acceptance retires only the previous card, never the members.
- The pass rides the scheduled memory_consolidation workflow (never the
commit path); a SQL-trace test proves the commit statement sequence
is identical with and without roll-up state present.
- Near-duplicate clusters are excluded from grouping so the dedup/merge
pipeline keeps owning them; a Jaccard guard covers provider-less runs.
- No migration: cards reuse the memories table (candidate rows + JSON
value/metadata) and the existing memory_consolidation artifact type.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Strict lexical search cannot bridge category-phrased questions to instance-phrased memories: "charity event fundraising total" shares no token with a memory that only says "Bike-a-Thon raised $5,000". This gives every memory a capped set of derived retrieval keys and folds them into full-text search on both backends, weighted so derived keys make rows FINDABLE without outranking direct text matches. - vnext_fact_keys.py (new): derive_fact_keys with two tiers -- (a) deterministic entity/attribute recombination (generic hypernym lexicon, currency/percent/unit phrasings, shallow value attribute pairs, linked-entity names/aliases/type words, human-authored memory_key path words) and (b) an optional OpenAI-compatible chat endpoint behind the ALICE_FACT_KEYS_BASE_URL/_MODEL/_API_KEY env seam, mirroring vnext_embeddings (unconfigured => tier (a) only, no network anywhere; provider failures degrade to tier (a) and log memory.fact_keys_failed). Keys are sanitized indexable text only, capped at 8 keys x 80 chars, novelty-filtered against the memory's own indexed text. - Migration 20260707_0082 + sqlite_schema: nullable memories.fact_keys (NULL = never derived, '' = derived, nothing to add). Postgres search_tsv is dropped and re-added with a setweight 'D' term (ts_rank default weight 0.1 vs title's 1.0) and the GIN index is rebuilt; ADD COLUMN computes it for existing rows. SQLite memories_fts gains a fact_keys column at bm25 weight 0.1, and bootstrap now drops-and-recreates an FTS table (plus its sync triggers) whose column set predates the current DDL, so pre-existing database files upgrade in place with the one-shot rebuild. - Stores: update_memory_fact_keys / list_memories_missing_fact_keys on both backends, placed beside the embedding indexing surface; search method signatures unchanged. True redaction now clears fact_keys alongside the embedding -- derived keys echo what the memory said. - Write path: one marked integration line in the memory-commit post-create hook, pinned to the deterministic tier so commits never make a synchronous model call. Measured +~0.12ms per commit, flat across 0/1k/5k-row tables (no O(N) work). Capture/promotion sites are owned by a parallel workstream; until wired, those rows are covered by the backfill. - Backfill: scripts/backfill_memory_fact_keys.py (documented one-shot; sqlite:/// and postgresql:// URLs; --deterministic-only flag). Verified: full unit suite 1761 passed; full live-Postgres integration suite passed including a real 0081->0082->0081 migration round trip with pre-existing rows; LongMemEval harness tests pass; retrieval quality eval suite (ALICEBOT_EVAL_DATABASE_URL=sqlite:///:memory:, --suite all) has statuses, failing-case sets, and non-latency metrics byte-identical to main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # apps/api/src/alicebot_api/vnext_retrieval.py # tests/unit/test_vnext_retrieval.py
# Conflicts: # eval/longmemeval/test_harness.py
# Conflicts: # eval/longmemeval/adapter.py # eval/longmemeval/test_harness.py
- adapter passes the question's own date as the temporal-anchor reference clock (official benchmark input the reading template already presents), unlocking relative-phrase anchors - fact keys attach at both promotion moments (product review-accept in mcp_tools, harness candidate promotion), deterministic tier only so neither path ever makes a synchronous model call - tests/unit/test_vnext_retrieval.py rebuilt via three-way merge + anchored splicing after interleaved-insertion conflicts tore two functions across hunk boundaries Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hygiene, capture cleanup, pack provenance
Fix 1 (eval/longmemeval/adapter.py): query-anchored excerpts now require
enumeration shape for EVERY anchor move (cross-chunk prose anchors were
displacing the head-biased best chunk that carried the answer), and the
window grew a symmetric UPWARD enumeration extension so a list cut at its
top edge recovers items 1..k-1 into the pass-2 pool.
Fix 2 (vnext_coverage_query.py): coverage-mode memory diversity key
refined from bare source_id to (source_id, source_chunk_id) with a
source-id fallback -- different turns of one session are distinct facts,
not restatements.
Fix 3 (vnext_fact_keys.py): _value_attribute_keys skips provenance and
identifier attributes (source_id/source_chunk_id, *_id/*_ids names) and
uuid/hex-shaped string values, so captured rows stop indexing
"source chunk id <uuid>" garbage into FTS (was 897/897 memories).
Fix 4 (vnext_capture.py): deleted the dead user-asserted/assistant-
estimate confidence deltas (pack ranking never reads confidence; the
promotion-rank ordering stays), and user-asserted-value promotions now
dedupe against existing canonical memory texts store-wide instead of
batch-locally.
Fix 5 (adapter.py RetrievalOutcome): checkpoint rows persist pack
provenance -- retrieved source session ids, selected memory ids, and the
sha256 of the rendered context block (ids + hash, never the text) -- so
future paired flips are offline-attributable. Config fingerprint
unchanged.
Offline verification: loss replays recover 2c63a862 (2/15 turn),
3e321797 (tomato list items via upward extension), 0a34ad58 (Suica
ticket-gate head chunk), 89941a93 partial ("hybrid bike" row back);
16-question gain-guard sweep shows zero evidence regressions; fresh
ingests carry zero uuid fact-key echoes across 10 spot stores; official
reading/judge templates byte-identical; full unit suite (1950), harness
(80), and eval battery all pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine features from the round-2 roadmap, built in parallel worktrees, integrated with cross-branch wiring, and measured through the full ladder (free coverage probe → targeted paired subset → full 172-question paired slice). No new benchmark claim: the published 79.4% stands; the paired slice measures this branch at parity (+1 net, p=1.0) with the movement where the features aim (temporal +2, multi-session +1, abstention +1). These ship for product value.
Features
(source_id, source_chunk_id).20260707_0082) — derived category/attribute keys at low FTS weight, both backends; identifier-shaped attributes excluded.Found-and-fixed along the way
Adversarial loss-forensics on the first slice run attributed every regression to a mechanism and produced surgical fixes, all verified: an excerpt-anchoring gate hole, a too-coarse diversity key, UUID pollution in derived keys (897/897 memories), and dead confidence-delta config. Stage-B paired verification: 6 of 7 convicted questions flipped back correct, zero gain regressions, all guards held.
Verification
1950 unit tests, 80 harness tests, 388 integration tests on fresh migrated Postgres (non-superuser role), six-suite eval battery, honesty audit (no benchmark-label code paths; official templates sha-pinned byte-identical). Paid measurement: 52-question stage-B (+6 net vs baseline on fix targets) and full 172 paired slice (parity, guards held).
Upgrade Overview
Protected Areas
Compatibility Impact
Migration
20260707_0082adds nullablememories.fact_keysand extends both FTS indexes (Postgres tsvector rebuild with weight-D term; SQLite FTS5 column with stale-shape rebuild at bootstrap).mcp_toolsreview-accept now attaches deterministic fact keys at promotion (never a model call).trusted_fact_promotions/capture record speaker provenance; promotion ordering prefers user-asserted values on ties — no suppression of assistant content (tested). Retrieval gains gated stages (temporal anchor, coverage mode, grounding note) that are dormant unless their query-surface trigger fires; dormant paths byte-identical (tested). Pack items may carry a new optionalvalidityfield.Migration / Rollout
Postgres:
alembic upgrade head(0081 → 0082; tsvector rebuild may pause on very large memories tables). SQLite: automatic at next open. Optional backfill CLI derives keys for pre-existing rows. Normal code rollout otherwise.Operator Action
None required. Optional: run the fact-keys backfill for older stores; set
ALICE_FACT_KEYS_*env to enable the model tier at backfill time.Validation
Full suites as above; offline coverage probe on all 500 benchmark questions (evidence coverage 84.6% → 85.4% overall, multi-session 74.4% → 76.7%, temporal 76.7% → 79.0%); paired slice guards: single-session-user and assistant at 100%, abstention +1.
Rollback
alembic downgrade -1removes the column/index cleanly; SQLite FTS rebuilds idempotently. Reverting the merge restores prior retrieval exactly (all new stages are additive and gated); no data loss — fact keys and provenance metadata are derived state.🤖 Generated with Claude Code