Round 3: deterministic retrieval ordering, roll-up acceptance harness, grounding productization#271
Merged
Conversation
…am, docs
Three pieces, all read-only over stores and dormant unless triggered:
1. vnext_grounding salience extension (round-3 loss forensics):
attribute-qualified lowercase compounds are now salient -- a measured
quantity ("my 30-gallon tank", physical units only, never time
units) or a possessive noun-noun compound with a curated head
("my snake plant", "my soccer team"). Three stacked conservatism
gates (qualifier shape, curated head lexicon, modifier hygiene) are
table-tested against generic-noun false positives. Probe hygiene:
pure-number tokens no longer fabricate corpus support when word
tokens exist ("30" on a receipt is not a mention of the 30-gallon
tank); all-number names keep their digits. 500-question offline
sweep: salience fires on 2 more questions (eeda8a6d_abs,
gpt4_8e165409); note fire-rate unchanged at 5/172 probed stores
(2.9%), all truthful.
2. NEW vnext_answer_verification: the product answer-grounding seam
adapting the harness verification concept. verify_answer_grounding(
answer_text, pack, chat_config) -> fail-open, self-disclosing
verdict (template sha + provider/model in every record); optional
apply_answer_grounding_gate withholds only on clean load-bearing
failures. Wired NOWHERE by default -- a test asserts no other
alicebot_api module imports it, and the byte-frozen harness
verification.py is untouched (sha-pin harness tests pass).
3. alice_context_pack now forwards pack["grounding"] (the compact view
silently dropped it, so the statistic never reached tool callers);
absent key stays absent, ungated responses byte-identical. Grounding
section added to docs/alpha/mcp-tools.md describing exactly what
ships.
Verification: 1999 unit tests pass; eval battery --suite all passes
with case-level results identical to main (13 unenforced paraphrase
misses, offline no-embeddings condition); harness suite 79 passed
(template sha pins green); SQLite commit path flat across preload
sizes (0.66ms at 0 and 5000 rows).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rds through the real acceptance path The benchmark harness never accepted consolidation roll-up proposals, so the round-2 cards (vnext_rollups) were invisible to every run. New post-promotion ingest step, off by default and always in the config fingerprint: - adapter.QuestionRun._consolidate_and_accept_rollups: runs the roll-up portion of the scheduled consolidation workflow (the exact VNextRollupService.propose_rollups invocation generate_memory_consolidation makes; the wrapper itself needs the artifact surface SQLiteVNextStore lacks) and review-accepts every card via VNextMemoryCommitService.accept_consolidation_candidate with identity=None - the same service call behind POST /v0/vnext/memories/accept-consolidation, the CLI, and the MCP review tool. Models the product's human review workflow the same way the harness already mirrors review-accept promotion for capture candidates. No store-level shortcuts. - Promotion stamps metadata_json.session_date (only when the step is on) so roll-up instances carry their session's date instead of the replay wall clock - the per-instance date hook vnext_rollups._member_date documents. Off flag, the promotion patch and the ingest record are byte-identical to before (no new keys). - runner: --accept-rollups flag, RunnerConfig field, fingerprint field (feeds the digest), per-store proposed/accepted counts in each checkpoint row's ingest.rollups block. - tests: real-path acceptance evidence (acceptance metadata, promoted revision, agent.memory_consolidation_accepted event), members stay active, card wins a pack slot on an aggregation fixture, off-flag byte-identity, cross-ingest determinism + idempotency, fingerprint disclosure, dry-run end-to-end with and without the flag. Keyless measurement on 15 aggregation-shaped multi-session failures from the 79.4% run: cards fire everywhere (8 proposed = 8 accepted per store, <1s) but grouping quality kills the value - only 3/15 questions get any accepted card into a max_items=16 pack, only 3/15 best cards reach even 2 content-term overlap with the question, and evidence-session coverage is unchanged (any 100%, all 33.3% both arms). Default stays off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…scade Equal-score ordering ties (RRF fusion, graph-stage timestamps, temporal source boosts, equal-score FTS/vector stage runs) fell through to the row id -- a uuid re-rolled per ingest -- so identical-config runs re-rolled every near-equal coin flip. Ties now resolve through a content-stable cascade (older content-stamped event/session date, longer content, the content text, capture fingerprint) with the id kept as the final total-order key; disclosed in the pack trace as fusion.tie_break=content_stable_v1. Measured keyless at the published run shape (max_items=16, 24k budget): two-seed re-ingest of 40 questions diverged on 7/40 before (1 session-set change) and 0/40 after; a synthetic exact-tie corpus flipped 4/8 pairs before, 0/8 after; the 172 kept run-C stores replay byte-stable across repeats, with one-in/one-out tie resolutions on 38 packs (3 session-set, 4 memory-set). Full unit suite, harness tests, and the live eval battery (all six suites) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three items from the round-3 queue, each validated offline before this PR. No benchmark claim changes.
Deterministic retrieval ordering (the churn killer)
Every equal-score ordering decision — RRF fused ties, graph/temporal stage ties, FTS/vector stage runs — now resolves through a content-stable cascade (event date → content length → text → capture fingerprint), with row ids demoted to the final total-order guarantee. Measured: two-seed re-ingest pack divergence went 7/40 → 0/40; synthetic exact-tie corpora went 4/8 uuid coin-flips → 0/8. Disclosed in traces as
fusion.tie_break: content_stable_v1(MCP debug trace included). This protects paired benchmark gains from re-rolling and makes recall reproducible for users.Roll-up acceptance harness step (honest negative result)
--accept-rollups(default off, config-fingerprint-stamped) review-accepts consolidation roll-up proposals through the real acceptance path (accept_consolidation_candidate), modeling the product's human review workflow. The required offline measurement killed the +5–10 point estimate: cards fire but grouping quality is too noisy (junk-entity labels dominate; only 3/15 target questions get a card into a pack slot; coverage probe unchanged). The flag ships OFF; roll-up grouping quality (entity blocklists, group-utility gates) is queued as product work — it also affects what the review console shows users.Grounding productization
vnext_answer_verificationlibrary seam adapts the harness grounding-gate concept for product integrators (fail-open, self-disclosing verdicts); nothing wired by default.Verification
2099 unit+harness tests green, six-suite eval battery passes, official templates sha-pinned byte-identical, honesty audit clean (no benchmark-label code paths).
Upgrade Overview
Protected Areas
Compatibility Impact
mcp_toolsdebug traces gainfusion.tie_break. Equal-score retrieval ties now order by content-stable keys instead of ids: rankings can shift within exact ties only (a resolution, not a regression); dormant/ungated paths otherwise byte-identical (tested). No schema changes; no API surface changes; new library module is opt-in.Migration / Rollout
Normal code rollout; no migrations.
Operator Action
None.
Validation
Suites above; two-seed determinism measurements; 500-question grounding sweep; off-flag byte-identity tests for the harness step.
Rollback
Revert the merge commit; all changes are code-only and additive.
🤖 Generated with Claude Code