Roll-up grouping quality: cards that read like topics#272
Merged
Conversation
Round-3 measurement showed VNextRollupService proposals were junk on
conversational stores: pronoun-contraction labels (I'm, I've), assistant
boilerplate (AI), store-generic lexical anchors (many, need, use), and
broken title subspans (Us Part II), with the max_rollups cap binding on
the first-formed groups instead of the best.
Grouping quality overhaul, all deterministic and store-local:
- structural label hygiene: pronoun contractions, single letters, and
function-word-only labels never head a card; "yes/yeah/okay/ok/ai"
join the conversational-plumbing token blocklist
- frequency-derived generic anchors: a stem dispersed across >=30% of
the store's distinct sessions is plumbing, not a topic (measured per
store; disabled below 30 rows / 10 sessions so small stores keep full
grouping power) — no hardcoded English topic list
- broken-subspan repair: entity labels truncated at a lowercase title
connector are expanded to the dominant full span in the member texts
("Us Part II" -> "The Last of Us Part II"), grouping key unchanged
- group-utility gate: proposals need >=3 members, an aggregation signal
(>=2 label-proximate distinct values — amounts/dates sharing a
sentence with a label stem — or >=3 distinct sessions), and a label
that is coherent (every content word in >=50% of members) and
store-specific; failing groups are dropped without claiming members,
disclosed in one aggregate skip line and a quality_gate metadata block
- ranking: surviving groups are proposed best-first by distinct values
x distinct sessions x label specificity, so max_rollups keeps the
strongest topics
- card text: titles carry the dominant value unit ("hours played — 5
instances, amounts in hours"); the card head carries runner-up label
surface forms ("plant (also: plants)") for exact-token retrieval
On the 15 aggregation-shaped LongMemEval stores (fresh keyless ingest,
accept-rollups arm): junk-label rate 34.2% -> 0%, questions with a
question-topical card 0 -> 6/15, slot-winning cards now exactly topical
with 4/5-4/4-4/4 evidence-session coverage (before: junk labels, 0 full
coverage); coverage probe unchanged (any 100%, all 33.3%); rollup pass
0.6s -> 0.8s per ~800-row store. Commit path stays O(1) and review-gate
semantics are untouched (both guarded by existing tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two user-visible residues the adversarial review flagged on the roll-up
grouping overhaul, both fixed structurally in vnext_rollups.py:
Card labels: a label head must be a content-bearing token. The junk
check now rejects (a) closed-class heads via a curated adverb/
preposition/conjunction list ("even", "still", "right", "towards",
"typically") regardless of the group's value signal, (b) light-verb
heads ("add", "used", "incorporate") without a noun in the label --
amounts do not rescue these because light verbs attract incidental
quantities, and (c) any other bare verb head (regular -ed morphology
plus curated transaction irregulars like "bought") unless the group
aggregates >= 2 distinct label-proximate amounts: "bought $120/$450"
aggregates purchases, while a bare verb with only session spread is
plumbing. Closed-class words are also excluded from topic-label
construction ("hiked trail", never "along hiked"), and "day" joins the
today/yesterday/time blocklist class.
Instance lines: instance labels now get the same hygiene as card
labels -- broken subspans are repaired against the member's own text
("Us Part II" -> "The Last of Us Part II"), pronoun-contraction and
closed-class fragments ("I'm", "Since I'm ...") are filtered, and a
member whose labels all filter out keeps its value+date line under a
neutral dominant-noun label. The subspan walk no longer treats pronoun
contractions as capitalized title words ("I'm thinking" is a sentence
subject, not a truncated title).
Measured on the 15 aggregation stores (fresh keyless ingest, same
probes as the branch baseline): junk-by-human-judgment cards drop from
12/120 to 0/120, instance-label defects from 362/2853 to 0/2901, with
topical cards unchanged at 6/15 questions and pack-slot wins improving
from 3/15 to 4/15 (full-evidence slot coverage 2 -> 3).
Tests: table-driven head-rule cases (closed-class rejection, light-verb
rejection, bare-verb with/without values), end-to-end stores for both
sides of the value rule, and instance-line repair/filter tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Round 4 of the memory-quality campaign: the consolidation roll-up engine now proposes cards a human recognizes as topics — this is what the review console shows users, so it's product UX first and benchmark substrate second.
What changed (vnext_rollups.py only, plus tests)
Measured (15 aggregation-heavy stores + 5 product-shaped fixtures, all offline)
Verification
2050 unit tests, 85 harness tests, six-suite eval battery pass. No migrations, no protected paths, no API surface changes — proposals remain review-gated candidates.
Known residuals (documented, next rounds)
Cards still lose pack slots to their own members on some aggregation queries (retrieval-fusion ranking, not grouping), and ~8/15 benchmark aggregation questions have no shared lexical anchor at all (needs embedding-based grouping — only meaningful in embeddings-on deployments).
🤖 Generated with Claude Code