diff --git a/CHANGELOG.md b/CHANGELOG.md index 5830442..ab38875 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## Unreleased +- Roll-up proposal quality overhaul: structural label hygiene (pronoun/contraction/closed-class/light-verb heads never title cards), store-measured generic-anchor detection (frequency-derived per store, no hardcoded topic list), broken-subspan label repair ("Us Part II" → "The Last of Us Part II") applied to card titles and instance lines, a group-utility gate (groups must aggregate distinct values or sessions with a coherent, specific label — failures are dropped, not proposed), utility-ranked proposals under the cap, and topic-shaped card titles with dominant value units. Measured on aggregation-heavy stores: junk-label rate 34% → 0%, instance-line defects 13% → 0%, with review proposals now reading as human-recognizable topics. + - Deterministic retrieval ordering: every equal-score tie (RRF fusion, graph and temporal stages, FTS/vector stage runs) now resolves through a content-stable cascade (event date, content length, text, capture fingerprint) instead of falling through to row ids — re-ingesting the same content yields byte-identical packs (two-seed divergence 7/40 → 0/40); disclosed as `fusion.tie_break: content_stable_v1` in retrieval traces. - Benchmark harness gains a disclosed `--accept-rollups` step (default off, fingerprint-stamped) that review-accepts consolidation roll-up proposals through the real acceptance path, modeling the product's human review workflow; offline measurement found current roll-up grouping quality too noisy to help the benchmark, so the flag stays off and grouping quality is queued as product work. - Grounding for products: the context-pack entity note now recognizes quantity-qualified and possessive compound entities ("30-gallon tank", "my snake plant") with conservative false-positive guards, and a new answer-verification library seam (`vnext_answer_verification`) lets integrators opt into post-generation grounding checks; nothing changes by default. diff --git a/apps/api/src/alicebot_api/vnext_rollups.py b/apps/api/src/alicebot_api/vnext_rollups.py index 7683aa5..8c11a47 100644 --- a/apps/api/src/alicebot_api/vnext_rollups.py +++ b/apps/api/src/alicebot_api/vnext_rollups.py @@ -52,6 +52,44 @@ Jaccard) are left to the near-duplicate dedup/merge pipeline — a roll-up aggregates DISTINCT instances of one topic, it does not dedupe copies. +Label hygiene and the group-utility gate +---------------------------------------- +A group only becomes a proposal if a human would recognize its card as a +topic that aggregates something. Everything below is deterministic and +store-local (measured from the grouped rows, never from a benchmark +label or an ambient corpus): + +- structural label hygiene: pronoun/contraction labels ("I'm", "I've"), + single letters, and labels made only of function words never label a + card; a label's head token must be content-bearing — closed-class + heads (adverbs/prepositions/conjunctions: "even", "still", "right", + "towards") and light-verb heads without a noun ("add", "used", + "incorporate") never head a card, and any other bare verb head needs + a value-based aggregation signal (>= 2 distinct label-proximate + amounts: "bought $120/$450" aggregates purchases; a bare verb with + only session spread is plumbing). Entity labels that are broken + subspans of a longer title ("Us Part II" inside "The Last of Us Part + II") are repaired by expanding them to the dominant full span found + in the member texts. Instance labels inside the card body get the + same subspan repair and pronoun/fragment filtering, falling back to + a neutral dominant-noun label so the value+date line is never lost. +- frequency-derived generic anchors: a stem that appears in a large + fraction of the store's *sessions* is conversational plumbing ("need", + "great", "help"), not a topic, no matter the language. The threshold + is measured per store (session dispersion), so nothing here hardcodes + an English vocabulary beyond the small structural lists above, and + small stores (below the stats floor) skip the frequency test entirely. +- group-utility gate: a proposal needs >= 3 members AND an aggregation + signal (>= 2 distinct instance values — amounts or in-text dates — or + >= 3 distinct sessions), a label whose content words appear in a + majority of member texts, and a label that is specific for the store. + Groups failing the gate are DROPPED (they do not claim members and do + not consume ``max_rollups`` slots), with one aggregate skip line + documenting the counts per reason. +- ranking: surviving groups are proposed best-first by aggregation + utility (distinct values x distinct sessions x label specificity), so + the ``max_rollups`` cap keeps the strongest topics, not the first. + The optional model seam (``generation_mode="model_backed"`` through the existing routing seam in ``vnext_model_intelligence``) only refines the card's one-line summary; the deterministic instance list is always @@ -114,20 +152,157 @@ # consolidation-merge grounding guard). ROLLUP_SUMMARY_GROUNDING_MIN_OVERLAP = 0.5 +# -- group-utility gate thresholds --------------------------------------------- + +# A roll-up aggregates; below three instances there is nothing to +# aggregate, whatever ``min_members`` is configured to. +MIN_AGGREGATION_MEMBERS = 3 + +# Aggregation signal: the group carries at least this many distinct +# instance values (currency/unit amounts or in-text dates) ... +MIN_DISTINCT_INSTANCE_VALUES = 2 +# ... OR spans at least this many distinct sessions/days. +MIN_DISTINCT_SESSIONS = 3 + +# Every content word of a card's label must appear in at least this +# fraction of the member texts — the label has to describe the group. +LABEL_COHERENCE_MIN_FRACTION = 0.5 + +# Frequency-derived generic-anchor detection (store-local, no hardcoded +# vocabulary): a stem that appears in at least this fraction of the +# store's distinct sessions is conversational plumbing, not a topic. +# Calibrated on real conversational stores, where plumbing stems +# ("need", "great", "help", "i'm") disperse across 32%-84% of sessions +# while genuine topics ("workshops", "games", "plants") stay under ~30% +# even when their raw memory counts are higher. +GENERIC_ANCHOR_SESSION_DISPERSION = 0.30 + +# Dispersion statistics are meaningless on tiny corpora: below these +# floors no stem is ever frequency-blocked (structural hygiene still +# applies). Small personal stores therefore keep full grouping power. +GENERIC_ANCHOR_MIN_ROWS = 30 +GENERIC_ANCHOR_MIN_SESSIONS = 10 + # Topic tokens that are conversational plumbing rather than topics; the -# speaker tags cover LongMemEval-style "[USER]: ..." transcripts. +# speaker tags cover LongMemEval-style "[USER]: ..." transcripts, and +# "ai" covers assistant self-reference ("As an AI ...") — in a store +# built from assistant chats that token labels boilerplate, not a topic +# (specific AI subjects still surface through their own entity/anchor +# labels: product names, "machine learning", model names, ...). _TOPIC_TOKEN_BLOCKLIST = frozenset( { "user", "assistant", "alice", "memory", "memories", "session", "chat", "remember", "today", "yesterday", "tomorrow", "really", "recently", "think", "thanks", "thank", "please", "would", "like", "want", "wanted", "going", "got", "get", "also", "one", "two", - "new", "time", "lot", "bit", + "new", "time", "day", "lot", "bit", "yes", "yeah", "okay", "ok", "ai", } ) +# Closed-class contraction heads: a label token like "i'm"/"you'd" whose +# head is one of these is a pronoun contraction, never a topic. Structural +# (grammar, not vocabulary), so it applies at any store size. +_PRONOUN_CONTRACTION_HEADS = frozenset( + {"i", "you", "he", "she", "it", "we", "they", "that", "there", "this", + "who", "what", "let", "here"} +) + +# Closed-class label heads: adverbs, prepositions, conjunctions, and +# discourse particles that slip past ``FTS_QUERY_STOPWORDS`` (which only +# covers query plumbing). Function words are a finite class — unlike +# topic vocabulary this list can be curated once — and none of them can +# name what a card aggregates ("Roll-up: even — 15 instances"), however +# strong the group's value signal is. Structural, so it applies at any +# store size. +_CLOSED_CLASS_LABEL_HEADS = frozenset( + { + # adverbs / discourse particles + "even", "still", "right", "already", "almost", "always", "anyway", + "aside", "away", "back", "certainly", "currently", "definitely", + "especially", "eventually", "finally", "furthermore", "generally", + "however", "indeed", "instead", "later", "maybe", "meanwhile", + "moreover", "never", "often", "otherwise", "perhaps", "probably", + "quite", "rather", "somehow", "sometimes", "somewhat", "soon", + "specifically", "together", "typically", "usually", + # prepositions not already in the query stopword list + "across", "along", "among", "amongst", "around", "behind", "beneath", + "beside", "besides", "beyond", "despite", "except", "inside", "near", + "onto", "outside", "past", "since", "throughout", "till", "toward", + "towards", "underneath", "until", "unto", "upon", "via", "within", + "without", + # conjunctions + "although", "though", "unless", "whereas", "whether", "yet", + } +) + +# Light / grammaticalized verb forms: verbs so bleached ("add", "used", +# "incorporate", "got", "made") that a card headed by one aggregates +# nothing a human would recognize, EVEN when its instances carry amounts +# ("Roll-up: add — 16 instances, amounts in minutes" was cooking-step +# plumbing, not an aggregation topic). Like the closed-class list this is +# a small curated set of function-word-adjacent forms, not an open topic +# vocabulary. A light-verb head is only acceptable when a noun in the +# label carries the topic ("add" alone never; "decided meridian" fine). +_LIGHT_VERB_FORMS = frozenset( + { + "add", "adds", "use", "uses", "used", "incorporate", "incorporates", + "get", "gets", "gotten", "make", "makes", "made", "take", "takes", + "took", "taken", "put", "puts", "give", "gives", "gave", "given", + "go", "goes", "went", "gone", "come", "comes", "came", "keep", + "keeps", "kept", "bring", "brings", "brought", "need", "needs", + "know", "knows", "knew", "known", "say", "says", "said", "tell", + "tells", "told", "see", "sees", "seen", "find", "finds", + "think", "thinks", "thought", + } +) + +# Transaction verbs whose bare form only means something as an +# aggregation over amounts: "bought $120 / $450" aggregates purchases, +# but "bought" with nothing to sum is plumbing. Irregular pasts listed +# explicitly because the ``-ed`` morphology test below cannot see them. +# Deliberately NOT a general verb list: contentful activity verbs +# ("flew", "hiked" via -ed) name a topic by themselves and are handled +# by the same value test only when morphology already marks them. +_TRANSACTION_VERB_FORMS = frozenset( + {"bought", "sold", "paid", "spent"} +) + + +def _is_verb_form(token: str) -> bool: + """Verb-shaped label token, detected without NLP: curated light-verb + and transaction-verb forms plus the regular past/participle ``-ed`` + morphology (``-eed`` nouns like "speed"/"seed" excluded). ``-ing`` + forms are deliberately NOT verb-shaped: as label tokens gerunds act + as nouns ("reading", "playing").""" + if token in _LIGHT_VERB_FORMS or token in _TRANSACTION_VERB_FORMS: + return True + return len(token) >= 4 and token.endswith("ed") and not token.endswith("eed") + +# Lowercase connectors that may sit INSIDE a capitalized title span +# ("The Last of Us Part II", "Lord of the Rings"); used when repairing +# entity labels that extraction truncated at such a connector. +_TITLE_CONNECTOR_TOKENS = frozenset( + {"of", "the", "and", "for", "de", "la", "le", "du", "da", "di", "del", + "van", "von", "&"} +) + +# Month names count as in-text date values for the aggregation-signal +# test; only capitalized surfaces are counted so modal "may" stays out. +_MONTH_TOKENS = frozenset( + {"january", "february", "march", "april", "may", "june", "july", + "august", "september", "october", "november", "december"} +) + _TOKEN_RE = re.compile(r"[a-z0-9][a-z0-9'-]*") _SURFACE_TOKEN_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9'-]*") +# Word tokens with positions, for the label-repair span walk. +_WORD_WITH_POS_RE = re.compile(r"[A-Za-z0-9][\w'’&-]*") +# Numeric calendar dates ("2023-05-10", "05/10", "5/10/2023"). +_DATE_MENTION_RE = re.compile( + r"(? JsonObject: return { @@ -235,6 +416,7 @@ def to_metadata(self) -> JsonObject: "groups": list(self.groups), "proposals": list(self.proposals), "skipped": list(self.skipped), + "quality_gate": dict(self.quality_gate), } def markdown_lines(self) -> list[str]: @@ -309,20 +491,40 @@ def _topic_tokens(text: str) -> set[str]: return tokens -def _surface_form(members: tuple[JsonObject, ...], anchor: str) -> str: - """Most frequent original surface form whose light stem equals - ``anchor`` (ties break alphabetically) — labels show real words.""" +def _surface_counts(members: tuple[JsonObject, ...], anchor: str) -> Counter[str]: counts: Counter[str] = Counter() for member in members: for raw in _SURFACE_TOKEN_RE.findall(_member_text(member)): lowered = raw.casefold() if _light_stem(lowered) == anchor: counts[lowered] += 1 + return counts + + +def _surface_form(members: tuple[JsonObject, ...], anchor: str) -> str: + """Most frequent original surface form whose light stem equals + ``anchor`` (ties break alphabetically) — labels show real words.""" + counts = _surface_counts(members, anchor) if not counts: return anchor return min(counts, key=lambda token: (-counts[token], token)) +def _surface_variants(members: tuple[JsonObject, ...], anchors: list[str], label: str) -> tuple[str, ...]: + """Runner-up surface forms of the label's stems ("plants" next to + "plant"), so the card also carries the inflection the members — and an + aggregation query — actually use. At most one variant per stem.""" + label_tokens = set(_TOKEN_RE.findall(label.casefold())) + variants: list[str] = [] + for anchor in anchors: + counts = _surface_counts(members, anchor) + for surface in sorted(counts, key=lambda token: (-counts[token], token)): + if surface not in label_tokens and surface not in variants: + variants.append(surface) + break + return tuple(variants) + + def _mean_pairwise_jaccard(token_sets: list[set[str]]) -> float: pairs: list[float] = [] for index, left in enumerate(token_sets): @@ -395,9 +597,424 @@ def _strip_speaker_tag(text: str) -> str: return _SPEAKER_TAG_RE.sub("", text, count=1).strip() +# -- label hygiene & the group-utility gate -------------------------------------- + + +def _stats_tokens(text: str) -> set[str]: + """Stemmed content tokens for corpus statistics and coherence checks. + + Wider than ``_topic_tokens`` (length >= 2, so acronym entity labels + like "AI" resolve to a measured stem) but the same stopword filter, + so plumbing stems and topic stems live in one vocabulary.""" + tokens: set[str] = set() + for raw in _TOKEN_RE.findall(text.casefold()): + if len(raw) < 2 or not any(char.isalpha() for char in raw): + continue + if raw in FTS_QUERY_STOPWORDS or raw in _TOPIC_TOKEN_BLOCKLIST: + continue + tokens.add(_light_stem(raw)) + return tokens + + +def _member_session_key(row: JsonObject) -> str | None: + """Best-effort per-member occasion key: source/session provenance when + the store carries it, else the member's content/event date. Only + DISTINCTNESS matters (the gate and the dispersion statistics count + occasions); the key never reaches a card.""" + metadata = row.get("metadata_json") + if isinstance(metadata, dict): + for key in ("source_id", "session_id", "session_date"): + value = metadata.get(key) + if isinstance(value, str) and value.strip(): + return f"{key}:{value.strip()}" + member_date = _member_date(row) + return f"date:{member_date}" if member_date else None + + +def _member_instance_values(text: str) -> frozenset[str]: + """Distinct aggregation values carried by one member text: unit/currency + amounts plus in-text dates (numeric or capitalized month names).""" + values: set[str] = set() + for amount in _member_amounts(text, limit=8): + values.add(" ".join(amount.casefold().split())) + for match in _DATE_MENTION_RE.finditer(text): + values.add(match.group(1)) + for token in _SURFACE_TOKEN_RE.findall(text): + if token[0].isupper() and token.casefold() in _MONTH_TOKENS: + values.add(token.casefold()) + return frozenset(values) + + +@dataclass(frozen=True, slots=True) +class _RowProfile: + stems: frozenset[str] + session_key: str | None + # (sentence stems, sentence values, sentence amounts) per sentence, so + # a group can count the values that sit in the SAME sentence as one of + # its label stems — a card about hours played aggregates the amounts + # attached to playing, not every number a long member text happens to + # mention elsewhere. Amounts (currency/unit quantities) are carried + # separately from the wider value set (which adds in-text dates) + # because the bare-verb label rule keys on amounts specifically. + sentences: tuple[tuple[frozenset[str], frozenset[str], frozenset[str]], ...] + + +def _row_profiles(rows: list[JsonObject]) -> dict[str, _RowProfile]: + profiles: dict[str, _RowProfile] = {} + for row in rows: + text = _strip_speaker_tag(_member_text(row)) + sentences: list[tuple[frozenset[str], frozenset[str], frozenset[str]]] = [] + for sentence in _SENTENCE_SPLIT_RE.split(text): + if not sentence.strip(): + continue + values = _member_instance_values(sentence) + if values: + amounts = frozenset( + " ".join(amount.casefold().split()) + for amount in _member_amounts(sentence, limit=8) + ) + sentences.append((frozenset(_stats_tokens(sentence)), values, amounts)) + profiles[str(row.get("id"))] = _RowProfile( + stems=frozenset(_stats_tokens(text)), + session_key=_member_session_key(row), + sentences=tuple(sentences), + ) + return profiles + + +class _CorpusStats: + """Store-local session-dispersion statistics for anchor stems. + + ``dispersion(stem)`` is the fraction of the store's distinct sessions + whose memories contain the stem. Conversational plumbing ("need", + "great", "i'm") disperses across most sessions; genuine topics + concentrate in a few. Below the row/session floors the statistics are + disabled and nothing is frequency-blocked.""" + + __slots__ = ("enabled", "session_count", "_stem_session_counts") + + def __init__(self, profiles: dict[str, _RowProfile]) -> None: + stem_sessions: dict[str, set[str]] = {} + sessions: set[str] = set() + for profile in profiles.values(): + if profile.session_key is None: + continue + sessions.add(profile.session_key) + for stem in profile.stems: + stem_sessions.setdefault(stem, set()).add(profile.session_key) + self.session_count = len(sessions) + self.enabled = ( + len(profiles) >= GENERIC_ANCHOR_MIN_ROWS + and self.session_count >= GENERIC_ANCHOR_MIN_SESSIONS + ) + self._stem_session_counts = ( + {stem: len(keys) for stem, keys in stem_sessions.items()} if self.enabled else {} + ) + + def dispersion(self, stem: str) -> float: + if not self.enabled or self.session_count == 0: + return 0.0 + return self._stem_session_counts.get(stem, 0) / self.session_count + + def is_generic(self, stem: str) -> bool: + return self.dispersion(stem) >= GENERIC_ANCHOR_SESSION_DISPERSION + + +def _label_content_tokens(label: str) -> list[str]: + """Casefolded content tokens of a label: pronoun contractions, single + letters, stopwords, and bare numbers do not count as content.""" + content: list[str] = [] + for token in _TOKEN_RE.findall(label.replace("’", "'").casefold()): + head = token.split("'", 1)[0] + if "'" in token and head in _PRONOUN_CONTRACTION_HEADS: + continue + if token in _PRONOUN_CONTRACTION_HEADS: + continue + if len(token) < 2 or not any(char.isalpha() for char in token): + continue + if token in FTS_QUERY_STOPWORDS or token in _TOPIC_TOKEN_BLOCKLIST: + continue + content.append(token) + return content + + +def _label_junk_reason( + label: str, + stats: _CorpusStats, + *, + label_amount_count: int | None = None, +) -> str | None: + """Why the label can never head a card, or None if it is presentable. + + The head (first content token) must be content-bearing: + + - a closed-class head (adverb/preposition/conjunction: "even", + "still", "right", "towards") never labels a card, whatever the + group's value signal; + - a light-verb head ("add", "used", "incorporate") needs a noun in + the label to carry the topic — amounts do not rescue it, because + light verbs attract incidental quantities ("add ... 10 minutes"); + - any other verb-shaped head (regular ``-ed`` forms, transaction + verbs like "bought") without an accompanying noun is acceptable + only when the group aggregates values: at least + ``MIN_DISTINCT_INSTANCE_VALUES`` distinct label-proximate amounts + ("bought $120/$450" aggregates purchases; a bare verb with only + session-dispersion signal is plumbing). ``label_amount_count`` is + that measured count; ``None`` (label-only callers with no group in + hand) skips the value-dependent rule. + """ + content = _label_content_tokens(label) + if not content: + return "label_without_content_words" + head = content[0] + if head in _CLOSED_CLASS_LABEL_HEADS: + return "label_head_closed_class" + if _is_verb_form(head): + has_noun = any( + token not in _CLOSED_CLASS_LABEL_HEADS and not _is_verb_form(token) + for token in content[1:] + ) + if not has_noun: + if head in _LIGHT_VERB_FORMS: + return "label_head_light_verb" + if ( + label_amount_count is not None + and label_amount_count < MIN_DISTINCT_INSTANCE_VALUES + ): + return "label_bare_verb_without_values" + stems = [_light_stem(token) for token in content] + if stats.enabled and min(stats.dispersion(stem) for stem in stems) >= GENERIC_ANCHOR_SESSION_DISPERSION: + return "label_generic_for_store" + return None + + +def _label_specificity(label: str, stats: _CorpusStats) -> float: + """1 - session dispersion of the label's most specific content stem + (1.0 when statistics are disabled or the label has no measured stem).""" + stems = [_light_stem(token) for token in _label_content_tokens(label)] + if not stems: + return 0.0 + return 1.0 - min(stats.dispersion(stem) for stem in stems) + + +def _natural_surface_order(members: tuple[JsonObject, ...], surfaces: list[str]) -> list[str]: + """Order a two-word topic label the way the member texts say it + ("credit card", not "card credit"); ties keep the support order.""" + if len(surfaces) != 2: + return surfaces + first, second = surfaces + forward = backward = 0 + for member in members[:JACCARD_SAMPLE_MEMBERS]: + text = " ".join(_member_text(member).casefold().split()) + forward += text.count(f"{first} {second}") + backward += text.count(f"{second} {first}") + if backward > forward: + return [second, first] + return surfaces + + +def _label_coherence(label: str, member_profiles: list[_RowProfile]) -> float: + """Minimum member-coverage across the label's content stems: every + content word of the label must describe most of the group.""" + stems = [_light_stem(token) for token in _label_content_tokens(label)] + if not stems or not member_profiles: + return 0.0 + coverage = [] + for stem in stems: + hits = sum(1 for profile in member_profiles if stem in profile.stems) + coverage.append(hits / len(member_profiles)) + return min(coverage) + + +def _expanded_entity_label(label: str, members: tuple[JsonObject, ...]) -> str: + """Repair entity labels that are broken subspans of a longer title. + + Extraction can truncate "The Last of Us Part II" to "Us Part II" (the + lowercase connector splits the capitalized span). Walking each label + occurrence leftward through capitalized words and title connectors in + the member texts recovers the full span; when a strict majority of + occurrences sit inside such a longer span, the label becomes the most + common expansion. Deterministic; bounded by the same member sample the + duplicate-group guard uses.""" + label_tokens = label.split() + if not label_tokens: + return label + expansions: Counter[str] = Counter() + occurrences = 0 + for member in members[:JACCARD_SAMPLE_MEMBERS]: + text = _strip_speaker_tag(_member_text(member)) + tokens = [(match.group(), match.start(), match.end()) for match in _WORD_WITH_POS_RE.finditer(text)] + for index in range(len(tokens) - len(label_tokens) + 1): + window = tokens[index : index + len(label_tokens)] + if [token[0] for token in window] != label_tokens: + continue + if any( + text[window[position][2] : window[position + 1][1]].strip() + for position in range(len(window) - 1) + ): + continue # punctuation inside the window: not this span + occurrences += 1 + start = index + + def _title_word(word: str) -> bool: + # A capitalized word can extend the span, but pronoun + # contractions ("I'm thinking ...") are sentence subjects, + # not truncated title material: compare the head before + # the apostrophe, not just the whole token. + head = word.casefold().replace("’", "'").split("'", 1)[0] + return word[0].isupper() and head not in _PRONOUN_CONTRACTION_HEADS + + while start > 0: + previous = tokens[start - 1] + if text[previous[2] : tokens[start][1]].strip(): + break # sentence punctuation between words: stop + word = previous[0] + if _title_word(word): + start -= 1 + continue + if word.casefold() in _TITLE_CONNECTOR_TOKENS and start - 1 > 0: + before = tokens[start - 2] + if not text[before[2] : previous[1]].strip() and _title_word(before[0]): + start -= 2 + continue + break + if start < index: + expansions[" ".join(token[0] for token in tokens[start : index + len(label_tokens)])] += 1 + if occurrences and sum(expansions.values()) * 2 > occurrences: + return min(expansions, key=lambda name: (-expansions[name], name)) + return label + + +@dataclass(frozen=True, slots=True) +class _GroupUtility: + distinct_values: int + # Label-proximate currency/unit amounts only (a subset of + # ``distinct_values``, which also counts in-text dates); the + # bare-verb label rule keys on this. + distinct_amounts: int + distinct_sessions: int + label_specificity: float + label_coherence: float + score: float + + def to_record(self) -> JsonObject: + return { + "distinct_values": self.distinct_values, + "distinct_amounts": self.distinct_amounts, + "distinct_sessions": self.distinct_sessions, + "label_specificity": round(self.label_specificity, 4), + "label_coherence": round(self.label_coherence, 4), + "score": round(self.score, 4), + } + + +def _group_utility( + label: str, + members: tuple[JsonObject, ...] | list[JsonObject], + profiles: dict[str, _RowProfile], + stats: _CorpusStats, +) -> tuple[_GroupUtility, str | None]: + """(utility, failure_reason). The gate a group must pass to become a + proposal: an aggregation signal (label-proximate distinct values or + distinct sessions) plus a coherent, store-specific label. + + Values only count when they share a sentence with one of the label's + content stems: a card about hours played aggregates the amounts + attached to playing, not every number its member texts mention.""" + label_stems = frozenset(_light_stem(token) for token in _label_content_tokens(label)) + member_profiles = [ + profiles[str(member.get("id"))] for member in members if str(member.get("id")) in profiles + ] + values: set[str] = set() + amounts: set[str] = set() + sessions: set[str] = set() + for profile in member_profiles: + for sentence_stems, sentence_values, sentence_amounts in profile.sentences: + if label_stems & sentence_stems: + values.update(sentence_values) + amounts.update(sentence_amounts) + if profile.session_key is not None: + sessions.add(profile.session_key) + specificity = _label_specificity(label, stats) + coherence = _label_coherence(label, member_profiles) + utility = _GroupUtility( + distinct_values=len(values), + distinct_amounts=len(amounts), + distinct_sessions=len(sessions), + label_specificity=specificity, + label_coherence=coherence, + score=max(1, len(values)) * max(1, len(sessions)) * specificity, + ) + if len(members) < MIN_AGGREGATION_MEMBERS: + return utility, "below_min_aggregation_members" + if ( + utility.distinct_values < MIN_DISTINCT_INSTANCE_VALUES + and utility.distinct_sessions < MIN_DISTINCT_SESSIONS + ): + return utility, "no_aggregation_signal" + if coherence < LABEL_COHERENCE_MIN_FRACTION: + return utility, "label_not_coherent_with_members" + return utility, None + + +def _instance_label_junk_reason(label: str) -> str | None: + """Structural-only junk test for instance labels (no store statistics): + labels with no content words ("I'm", "I've"), closed-class heads, and + mid-sentence fragment openers — a label whose first token is a pronoun + contraction or closed-class word ("Since I'm ...", "I'm logging ...") + is a truncation artifact, not a name. The same filters card labels + get, minus the group-dependent rules.""" + content = _label_content_tokens(label) + if not content: + return "label_without_content_words" + raw_tokens = _TOKEN_RE.findall(label.replace("’", "'").casefold()) + first = raw_tokens[0] if raw_tokens else "" + if first in _PRONOUN_CONTRACTION_HEADS or first.split("'", 1)[0] in _PRONOUN_CONTRACTION_HEADS: + return "label_pronoun_fragment" + if first in _CLOSED_CLASS_LABEL_HEADS or content[0] in _CLOSED_CLASS_LABEL_HEADS: + return "label_head_closed_class" + return None + + +def _dominant_noun_label(text: str) -> str | None: + """Neutral display label for an instance whose extracted labels all + filtered out: the text's most frequent content-bearing, non-verb + surface token (ties break alphabetically on the casefolded token; + the first-seen surface form is displayed). Deterministic.""" + counts: Counter[str] = Counter() + surfaces: dict[str, str] = {} + for raw in _SURFACE_TOKEN_RE.findall(text): + token = raw.casefold().replace("’", "'") + head = token.split("'", 1)[0] + if "'" in token and head in _PRONOUN_CONTRACTION_HEADS: + continue + if token in _PRONOUN_CONTRACTION_HEADS: + continue + if len(token) < 3 or not any(char.isalpha() for char in token): + continue + if token in FTS_QUERY_STOPWORDS or token in _TOPIC_TOKEN_BLOCKLIST: + continue + if token in _CLOSED_CLASS_LABEL_HEADS or _is_verb_form(token): + continue + counts[token] += 1 + surfaces.setdefault(token, raw) + if not counts: + return None + best = min(counts, key=lambda token: (-counts[token], token)) + return surfaces[best] + + def _instance_label(row: JsonObject, *, exclude_normalized: str | None = None) -> str: - """Entity surface when extraction finds one, else the member title, - else the truncated text — short, deterministic, display-safe. + """Entity surface when extraction finds a presentable one, else the + member title, else a neutral noun label, else the truncated text — + short, deterministic, display-safe. + + Instance labels get the same hygiene the card label gets: broken + subspans are repaired against the member's own text ("Us Part II" -> + "The Last of Us Part II") and pronoun-contraction / closed-class-head + fragments ("I'm", "Since I'm") are filtered; a member whose labels all + filter out keeps its value+date line under a neutral label derived + from its dominant noun token. ``exclude_normalized`` skips the group's own entity so an entity-group card labels each instance by its distinguishing content, not by the @@ -408,12 +1025,21 @@ def _instance_label(row: JsonObject, *, exclude_normalized: str | None = None) - for candidate in extract_entity_candidates(text) if candidate.normalized != exclude_normalized ] - if candidates: - best = max(candidates, key=lambda candidate: (candidate.confidence, candidate.occurrences)) - return best.name + for candidate in sorted( + candidates, + key=lambda candidate: (-candidate.confidence, -candidate.occurrences, candidate.name), + ): + label = _expanded_entity_label(candidate.name, (row,)) + if _instance_label_junk_reason(label) is None: + return label title = row.get("title") if isinstance(title, str) and title.strip(): - return _strip_speaker_tag(" ".join(title.split()))[:80] + cleaned = _strip_speaker_tag(" ".join(title.split()))[:80] + if cleaned and _instance_label_junk_reason(cleaned) is None: + return cleaned + neutral = _dominant_noun_label(text) + if neutral is not None: + return neutral return text[:80] @@ -432,6 +1058,34 @@ def _instance_record(row: JsonObject, *, exclude_normalized: str | None = None) return record +def _amount_unit(amount: str) -> str | None: + """Unit carried by one amount surface: leading currency symbol or the + trailing unit word ("30 hours" -> "hours", "$40" -> "$").""" + amount = amount.strip() + if amount[:1] in "$€£": + return amount[:1] + match = re.search(r"([a-z%]+)\s*$", amount.casefold()) + return match.group(1) if match else None + + +def _dominant_amount_unit(instances: list[JsonObject]) -> str | None: + """Most common unit across the instances' amounts, when at least two + instances carry it (ties break alphabetically) — a card that mostly + lists hours is titled as an hours aggregation.""" + counts: Counter[str] = Counter() + for instance in instances: + amounts = instance.get("amounts") + if not isinstance(amounts, list): + continue + units = {unit for amount in amounts if (unit := _amount_unit(str(amount))) is not None} + for unit in units: + counts[unit] += 1 + if not counts: + return None + unit = min(counts, key=lambda name: (-counts[name], name)) + return unit if counts[unit] >= 2 else None + + def _render_instance(instance: JsonObject) -> str: details: list[str] = [] amounts = instance.get("amounts") @@ -685,12 +1339,30 @@ def _group_members( *, options: RollupOptions, exclude_member_id_sets: list[set[str]], - ) -> tuple[list[_RollupGroup], list[str]]: + ) -> tuple[list[_RollupGroup], list[str], JsonObject]: skipped: list[str] = [] claimed: set[str] = set() groups: list[_RollupGroup] = [] - - def _admit(key: str, kind: str, label: str, members: list[JsonObject]) -> None: + dropped: Counter[str] = Counter() + dropped_examples: list[str] = [] + + profiles = _row_profiles(rows) + stats = _CorpusStats(profiles) + + def _drop(key: str, reason: str) -> None: + # Gate-dropped groups do NOT claim members and are reported as + # one aggregate skip line (not one line per junk anchor). + dropped[reason] += 1 + if len(dropped_examples) < 6: + dropped_examples.append(f"{key} ({reason})") + + def _admit( + key: str, + kind: str, + label: str, + members: list[JsonObject], + label_variants: tuple[str, ...] = (), + ) -> None: if len(members) > MAX_ROLLUP_GROUP_MEMBERS: skipped.append(f"group_too_large: {key} (members={len(members)})") return @@ -710,9 +1382,30 @@ def _admit(key: str, kind: str, label: str, members: list[JsonObject]) -> None: f"(mean_jaccard={jaccard:.2f}, members={len(members)})" ) return + # Label hygiene, then the group-utility gate; failing groups are + # dropped and their members stay available to later groups. The + # utility is measured first because the bare-verb label rule + # needs the group's label-proximate amount count. + utility, gate_reason = _group_utility(label, members, profiles, stats) + junk_reason = _label_junk_reason( + label, stats, label_amount_count=utility.distinct_amounts + ) + if junk_reason is not None: + _drop(key, junk_reason) + return + if gate_reason is not None: + _drop(key, gate_reason) + return claimed.update(member_ids) groups.append( - _RollupGroup(rollup_key=key, group_kind=kind, label=label, members=_sorted_members(members)) + _RollupGroup( + rollup_key=key, + group_kind=kind, + label=label, + members=_sorted_members(members), + utility=utility, + label_variants=label_variants, + ) ) # One extraction pass per row; both grouping passes reuse it. @@ -737,7 +1430,10 @@ def _admit(key: str, kind: str, label: str, members: list[JsonObject]) -> None: members = [row for row in entity_members[key] if str(row.get("id")) not in claimed] if len(members) < options.min_members: continue - _admit(f"entity:{key}", "entity", entity_display[key], members) + # Repair labels extraction truncated at a lowercase title + # connector ("Us Part II" -> "The Last of Us Part II"). + label = _expanded_entity_label(entity_display[key], tuple(members)) + _admit(f"entity:{key}", "entity", label, members) # Pass 2 — lexical-topic anchors over members no entity group claimed. remaining = [row for row in rows if str(row.get("id")) not in claimed] @@ -746,18 +1442,52 @@ def _admit(key: str, kind: str, label: str, members: list[JsonObject]) -> None: for token in sorted(_topic_tokens(_member_text(row))): anchor_members.setdefault(token, []).append(row) for anchor in sorted(anchor_members, key=lambda token: (-len(anchor_members[token]), token)): + if stats.is_generic(anchor): + # Frequency-derived: an anchor dispersed across most of the + # store's sessions is plumbing, not a topic. Checked before + # claiming so real topics keep these members. + if len(anchor_members[anchor]) >= options.min_members: + _drop(f"topic:{anchor}", "anchor_generic_for_store") + continue members = [row for row in anchor_members[anchor] if str(row.get("id")) not in claimed] if len(members) < options.min_members: continue # Label: up to two topic stems shared by EVERY member (surface # forms), so the card carries the words an aggregation query # would use ("hours played"), not just the single anchor. - shared = set.intersection(*(_topic_tokens(_member_text(member)) for member in members)) + # Closed-class words never carry a topic, so they are not + # label material ("hiked trail", never "along hiked"). + shared = { + stem + for stem in set.intersection(*(_topic_tokens(_member_text(member)) for member in members)) + if not stats.is_generic(stem) and stem not in _CLOSED_CLASS_LABEL_HEADS + } ordered = sorted(shared, key=lambda token: (-len(anchor_members.get(token, ())), token)) or [anchor] - label = " ".join(_surface_form(tuple(members), stem) for stem in ordered[:2]) - _admit(f"topic:{anchor}", "topic", label, members) - - return groups, skipped + surfaces = [_surface_form(tuple(members), stem) for stem in ordered[:2]] + label = " ".join(_natural_surface_order(tuple(members), surfaces)) + variants = _surface_variants(tuple(members), ordered[:2], label) + _admit(f"topic:{anchor}", "topic", label, members, label_variants=variants) + + # Rank by aggregation utility so max_rollups keeps the best groups, + # not the first-admitted ones. Deterministic tie-breaks. + groups.sort(key=lambda group: (-group.utility.score, -len(group.members), group.rollup_key)) + + gate_record: JsonObject = { + "anchor_stats_enabled": stats.enabled, + "session_count": stats.session_count, + "generic_anchor_session_dispersion": GENERIC_ANCHOR_SESSION_DISPERSION, + "min_aggregation_members": MIN_AGGREGATION_MEMBERS, + "dropped_group_count": sum(dropped.values()), + "dropped_by_reason": {reason: dropped[reason] for reason in sorted(dropped)}, + } + if dropped: + reasons = ", ".join(f"{reason}={count}" for reason, count in sorted(dropped.items())) + examples = "; ".join(dropped_examples) + skipped.append( + f"quality_gate_dropped: {sum(dropped.values())} group(s) not proposed " + f"({reasons}); e.g. {examples}" + ) + return groups, skipped, gate_record # -- accepted / pending state ------------------------------------------------- @@ -793,10 +1523,21 @@ def _render_card( model_summary: str | None, ) -> tuple[str, str, str]: """(title, canonical_text, summary) — deterministic; the optional - model summary is appended as a clearly-labelled extra sentence.""" + model summary is appended as a clearly-labelled extra sentence. + + The title reads like a topic: label plus the dominant value unit + carried by the instances ("hours played — 5 instances, amounts in + hours"), derived deterministically from the instance amounts.""" rendered = "; ".join(_render_instance(instance) for instance in instances) - title = f"Roll-up: {group.label} ({len(instances)} instances in total)" - canonical_text = f"{group.label} — {len(instances)} instances in total: {rendered}." + unit = _dominant_amount_unit(instances) + if unit is not None: + title = f"Roll-up: {group.label} — {len(instances)} instances, amounts in {unit}" + else: + title = f"Roll-up: {group.label} ({len(instances)} instances in total)" + head = group.label + if group.label_variants: + head = f"{group.label} (also: {', '.join(group.label_variants)})" + canonical_text = f"{head} — {len(instances)} instances in total: {rendered}." if model_summary: canonical_text = f"{canonical_text} Summary: {model_summary}" summary = canonical_text[:280] @@ -939,10 +1680,11 @@ def propose_rollups( outcome.skipped.append("fewer_memories_than_min_members") return outcome - groups, group_skips = self._group_members( + groups, group_skips, gate_record = self._group_members( rows, options=options, exclude_member_id_sets=exclude_member_id_sets or [] ) outcome.skipped.extend(group_skips) + outcome.quality_gate = gate_record if len(groups) > options.max_rollups: outcome.skipped.append( f"rollup_bound: {len(groups) - options.max_rollups} groups beyond " @@ -961,6 +1703,7 @@ def propose_rollups( "label": group.label, "member_ids": member_ids, "rollup_digest": rollup_digest, + "aggregation": group.utility.to_record(), } accepted_card = accepted.get(group.rollup_key) @@ -1064,6 +1807,13 @@ def propose_rollups( "DEFAULT_MAX_ROLLUPS", "DEFAULT_ROLLUP_MIN_MEMBERS", "DUPLICATE_GROUP_JACCARD_THRESHOLD", + "GENERIC_ANCHOR_MIN_ROWS", + "GENERIC_ANCHOR_MIN_SESSIONS", + "GENERIC_ANCHOR_SESSION_DISPERSION", + "LABEL_COHERENCE_MIN_FRACTION", + "MIN_AGGREGATION_MEMBERS", + "MIN_DISTINCT_INSTANCE_VALUES", + "MIN_DISTINCT_SESSIONS", "ROLLUP_CANDIDATE_KIND", "ROLLUP_PROPOSAL_KIND", "RollupOptions", diff --git a/tests/unit/test_vnext_consolidation.py b/tests/unit/test_vnext_consolidation.py index 5da68fd..8e890a9 100644 --- a/tests/unit/test_vnext_consolidation.py +++ b/tests/unit/test_vnext_consolidation.py @@ -542,6 +542,48 @@ def test_rollup_pass_skips_groups_covered_by_near_duplicate_clusters() -> None: assert "## Roll-up Proposals" in artifact["content_markdown"] +def test_rollup_quality_gate_drops_junk_groups_and_is_disclosed() -> None: + """Pronoun-contraction groups never reach the review console through the + scheduled consolidation workflow; the artifact discloses the gate.""" + store = FakeConsolidationStore() + specs = ( + ("I played The Last of Us Part II for 30 hours", "2023-05-10"), + ("I played Assassin's Creed Odyssey for 70 hours", "2023-05-18"), + ("I played Hollow Knight for 25 hours", "2023-06-02"), + ("I played Stardew Valley for 85 hours", "2023-06-20"), + ("I'm excited about the violet harbor", None), + ("I'm tired after the granite summit", None), + ("I'm curious about the copper lantern", None), + ) + for index, (text, session_date) in enumerate(specs): + metadata: JsonObject = {"session_date": session_date} if session_date else {} + store.create_memory( + { + "memory_key": f"memory.gate-{index}", + "value": {"text": text}, + "status": "active", + "memory_type": "episode", + "title": text[:80], + "canonical_text": text, + "summary": text[:80], + "domain": "personal", + "sensitivity": "internal", + "metadata_json": metadata, + } + ) + artifact = VNextConsolidationService(store, embedding_provider=None).generate_memory_consolidation( + MemoryConsolidationRequest() + ) + + rollups = artifact["metadata_json"]["rollups"] + assert rollups["enabled"] is True + labels = [proposal["label"].casefold() for proposal in rollups["proposals"]] + assert labels and all(not label.startswith("i'") for label in labels) + assert rollups["quality_gate"]["dropped_group_count"] >= 1 + assert "label_without_content_words" in rollups["quality_gate"]["dropped_by_reason"] + assert any(reason.startswith("quality_gate_dropped") for reason in rollups["skipped"]) + + def test_propose_rollups_false_discloses_disabled_state() -> None: store = FakeConsolidationStore() mapping: dict[str, list[float]] = {} diff --git a/tests/unit/test_vnext_rollups.py b/tests/unit/test_vnext_rollups.py index eeff860..8c8dbf3 100644 --- a/tests/unit/test_vnext_rollups.py +++ b/tests/unit/test_vnext_rollups.py @@ -23,6 +23,10 @@ RollupOptions, VNextRollupService, VNextRollupValidationError, + _CorpusStats, + _instance_label, + _instance_record, + _label_junk_reason, ) @@ -320,6 +324,410 @@ def test_max_rollups_bound_is_reported() -> None: assert any("rollup_bound" in reason for reason in outcome.skipped) +# -- label hygiene & group-utility gate ------------------------------------------------ + + +def _seed_texts(store, specs: tuple[tuple[str, str | None], ...], *, prefix: str = "memory.gate") -> None: + for index, (text, session_date) in enumerate(specs): + metadata: JsonObject = {} + if session_date is not None: + metadata["session_date"] = session_date + store.create_memory( + { + "memory_key": f"{prefix}-{index}", + "value": {"text": text}, + "status": "active", + "memory_type": "episode", + "title": text[:80], + "canonical_text": text, + "summary": text[:80], + "domain": "personal", + "sensitivity": "internal", + "metadata_json": metadata, + } + ) + + +JUNK_TRIO = ( + ("I'm excited about the violet harbor", None), + ("I'm tired after the granite summit", None), + ("I'm curious about the copper lantern", None), +) + + +def test_pronoun_contraction_labels_never_head_cards() -> None: + """Members sharing only \"I'm\" produce no card: the label has no + content words, so the group is dropped instead of proposed.""" + store = FakeRollupStore() + _seed_texts(store, JUNK_TRIO) + outcome = VNextRollupService(store).propose_rollups() + assert outcome.proposals == [] + assert _rollup_candidates(store) == [] + gate_lines = [reason for reason in outcome.skipped if reason.startswith("quality_gate_dropped")] + assert gate_lines and "label_without_content_words" in gate_lines[0] + assert outcome.quality_gate["dropped_by_reason"]["label_without_content_words"] >= 1 + + +def test_frequency_generic_anchor_is_not_a_topic() -> None: + """Store-measured generic anchors: a stem dispersed across most + sessions ('need') cannot anchor a topic, while a concentrated topic + ('hiked') still can — no hardcoded vocabulary involved.""" + store = FakeRollupStore() + filler = tuple( + (f"need marker{index:02d}a marker{index:02d}b", f"2023-03-{(index % 12) + 1:02d}") + for index in range(36) + ) + hikes = tuple( + (f"Hiked {distance} km along the {name} trail", f"2023-04-{day:02d}") + for distance, name, day in ( + (5, "juniper", 2), + (8, "basalt", 9), + (11, "willow", 16), + (13, "mesa", 23), + ) + ) + _seed_texts(store, filler + hikes) + outcome = VNextRollupService(store).propose_rollups() + + assert outcome.quality_gate["anchor_stats_enabled"] is True + assert len(outcome.proposals) == 1 + proposal = outcome.proposals[0] + assert "hiked" in proposal["label"].casefold() + assert proposal["aggregation"]["distinct_values"] >= 4 + assert proposal["aggregation"]["distinct_sessions"] >= 4 + # The plumbing anchor was dropped BEFORE claiming members, in the + # aggregate skip line, not proposed and not one-line-per-anchor. + assert outcome.quality_gate["dropped_by_reason"]["anchor_generic_for_store"] >= 1 + assert not any("need" == proposal["label"].casefold() for proposal in outcome.proposals) + + +def test_broken_subspan_entity_label_is_repaired() -> None: + """Extraction truncates 'The Last of Us Part II' to 'Us Part II' (the + lowercase connector splits the span); the card label is repaired to the + dominant full span while the grouping key stays stable.""" + store = FakeRollupStore() + _seed_texts( + store, + ( + ("I finished The Last of Us Part II after 30 hours", "2023-05-10"), + ("Replaying The Last of Us Part II took 12 hours", "2023-05-18"), + ("The Last of Us Part II photo mode ate 3 hours", "2023-06-02"), + ("Speedran The Last of Us Part II in 9 hours", "2023-06-11"), + ), + ) + outcome = VNextRollupService(store).propose_rollups() + assert len(outcome.proposals) == 1 + proposal = outcome.proposals[0] + assert proposal["rollup_key"] == "entity:us part ii" # grouping unchanged + assert proposal["label"] == "The Last of Us Part II" # display repaired + card = _rollup_candidates(store)[0] + assert "The Last of Us Part II" in card["title"] + + +def test_groups_without_aggregation_signal_are_dropped() -> None: + """Three same-entity mentions with no amounts, dates, or session spread + aggregate nothing; the group is dropped, not proposed.""" + store = FakeRollupStore() + _seed_texts( + store, + ( + ("Quartz Peak has a nice lodge", None), + ("The lodge at Quartz Peak seems busy", None), + ("Thinking about the Quartz Peak lodge", None), + ), + ) + outcome = VNextRollupService(store).propose_rollups() + assert outcome.proposals == [] + assert _rollup_candidates(store) == [] + assert outcome.quality_gate["dropped_by_reason"]["no_aggregation_signal"] >= 1 + + +def test_ranking_prefers_higher_aggregation_utility() -> None: + """max_rollups keeps the strongest aggregation, not the first-formed + group: five games across five sessions beat three ferry receipts in one.""" + store = FakeRollupStore() + _seed_game_memories(store) + _seed_texts( + store, + ( + ("Paid $40 for the ferry pass", None), + ("Paid $25 for the ferry pass upgrade", None), + ("Paid $40 ferry pass renewal fee", None), + ), + ) + outcome = VNextRollupService(store).propose_rollups( + options=RollupOptions.from_metadata({"rollup_options": {"max_rollups": 1}}) + ) + assert len(outcome.proposals) == 1 + label = outcome.proposals[0]["label"].casefold() + assert "played" in label or "hours" in label + assert any("rollup_bound" in reason for reason in outcome.skipped) + + +def test_title_reads_like_a_topic_with_dominant_unit() -> None: + store = FakeRollupStore() + _seed_game_memories(store) + outcome = VNextRollupService(store).propose_rollups() + card = _rollup_candidates(store)[0] + assert "amounts in hours" in card["title"] + aggregation = outcome.proposals[0]["aggregation"] + assert aggregation["distinct_values"] == 5 + assert aggregation["distinct_sessions"] == 5 + assert aggregation["score"] > 0 + assert aggregation["label_coherence"] == 1.0 + + +# -- label heads: closed-class words and bare verbs -------------------------------- + + +LABEL_HEAD_CASES = ( + # Closed-class heads are junk no matter how strong the value signal: + # the verifier's residue cards ("even — 15 instances, amounts in $") + # all carried amounts. + ("even", 5, "label_head_closed_class"), + ("still", 5, "label_head_closed_class"), + ("right", 5, "label_head_closed_class"), + ("towards", 5, "label_head_closed_class"), + ("typically", 5, "label_head_closed_class"), + # Light-verb heads are junk without a noun, amounts or not: light + # verbs attract incidental quantities ("add ... for 10 minutes"). + ("add", 5, "label_head_light_verb"), + ("used", 5, "label_head_light_verb"), + ("incorporate", 5, "label_head_light_verb"), + # Other bare verb heads are acceptable exactly when the group + # aggregates values ("bought $120/$450" aggregates purchases; a bare + # verb with only session spread is plumbing). + ("bought", 3, None), + ("bought", 0, "label_bare_verb_without_values"), + ("hiked", 2, None), + ("hiked", 0, "label_bare_verb_without_values"), + # A verb the deterministic machinery cannot mark (irregular past, + # not curated) stays content-bearing. + ("flew", 0, None), + # A noun anywhere in the label carries the topic. + ("miles ran", 0, None), + ("decided meridian", 0, None), + ("finished reading", 0, None), # gerunds act as nouns + ("hours played", 0, None), + # Plain noun labels are never head-junk. + ("workshops", 0, None), + ("model kits", 0, None), +) + + +@pytest.mark.parametrize( + "label,amounts,expected", LABEL_HEAD_CASES, ids=[f"{c[0]}-{c[1]}" for c in LABEL_HEAD_CASES] +) +def test_label_head_junk_rules(label, amounts, expected) -> None: + """Table-driven head hygiene: a card label's head token must be + content-bearing (no closed-class words, no light verbs without a + noun, no bare verbs without a value-based aggregation signal).""" + stats = _CorpusStats({}) # below the stats floor: structural rules only + assert _label_junk_reason(label, stats, label_amount_count=amounts) == expected + + +def test_closed_class_head_never_labels_card_even_with_amounts() -> None: + """Members sharing only a preposition ('towards') with dollar amounts + in every text: real aggregation signal, but a closed-class head can + never name a topic — the group is dropped, not proposed.""" + store = FakeRollupStore() + _seed_texts( + store, + ( + ("Saved $40 towards a harbor kayak", "2026-03-02"), + ("Chipped $25 towards the granite lodge", "2026-03-09"), + ("Banked $60 towards the copper lantern", "2026-03-16"), + ), + ) + outcome = VNextRollupService(store).propose_rollups() + assert outcome.proposals == [] + assert _rollup_candidates(store) == [] + assert outcome.quality_gate["dropped_by_reason"]["label_head_closed_class"] >= 1 + + +def test_light_verb_head_never_labels_card_even_with_amounts() -> None: + """Members sharing only a light verb ('used') with amounts: light + verbs attract incidental quantities, so the value signal does not + rescue the head — the group is dropped.""" + store = FakeRollupStore() + _seed_texts( + store, + ( + ("Used the harbor pass, $12 fare", "2026-03-02"), + ("Used the granite entrance, $15 fee", "2026-03-09"), + ("Used the copper gate, $18 toll", "2026-03-16"), + ), + ) + outcome = VNextRollupService(store).propose_rollups() + assert outcome.proposals == [] + assert _rollup_candidates(store) == [] + assert outcome.quality_gate["dropped_by_reason"]["label_head_light_verb"] >= 1 + + +def test_bare_transaction_verb_needs_value_signal() -> None: + """Both sides of the bare-verb rule on real stores: 'bought' with + only session spread is plumbing (dropped); 'bought' aggregating + distinct prices is a purchases card (proposed).""" + plain = FakeRollupStore() + _seed_texts( + plain, + ( + ("Bought a harbor kayak pass", "2026-03-02"), + ("Bought the granite lodge voucher", "2026-03-09"), + ("Bought a copper lantern kit", "2026-03-16"), + ), + ) + without_values = VNextRollupService(plain).propose_rollups() + assert without_values.proposals == [] + assert ( + without_values.quality_gate["dropped_by_reason"]["label_bare_verb_without_values"] >= 1 + ) + + priced = FakeRollupStore() + _seed_texts( + priced, + ( + ("Bought a harbor kayak pass for $40", "2026-03-02"), + ("Bought the granite lodge voucher for $25", "2026-03-09"), + ("Bought a copper lantern kit for $60", "2026-03-16"), + ), + ) + with_values = VNextRollupService(priced).propose_rollups() + assert len(with_values.proposals) == 1 + proposal = with_values.proposals[0] + assert "bought" in proposal["label"].casefold() + assert proposal["aggregation"]["distinct_amounts"] >= 2 + + +# -- instance-line hygiene --------------------------------------------------------- + + +def test_instance_labels_get_subspan_repair() -> None: + """The card-label subspan repair also applies inside instance lines: + 'Us Part II' renders as 'The Last of Us Part II' next to its value + and date.""" + store = FakeRollupStore() + _seed_game_memories(store) + VNextRollupService(store).propose_rollups() + card = _rollup_candidates(store)[0] + labels = [instance["label"] for instance in card["value"]["rollup"]["instances"]] + assert "The Last of Us Part II" in labels + assert "Us Part II" not in labels + assert "The Last of Us Part II (30 hours; 2023-05-10" in card["canonical_text"] + + +def test_fragment_instance_label_filters_to_neutral_noun() -> None: + """An instance whose extracted label is a pronoun/closed-class + fragment ('Since I'm') keeps its value+date line under a neutral + content-bearing label instead.""" + row: JsonObject = { + "id": "frag-1", + "title": "Since I'm in Denver, the pottery workshop cost $40", + "canonical_text": "Since I'm in Denver, the pottery workshop cost $40", + "metadata_json": {"session_date": "2026-03-02"}, + } + label = _instance_label(row) + assert label + lowered = label.casefold() + assert not lowered.startswith("since") + assert "i'm" not in lowered + record = _instance_record(row) + assert record["label"] == label + assert record["amounts"] == ["$40"] # the value line survives the filter + assert record["date"] == "2026-03-02" + + +def test_pronoun_contraction_instance_label_is_filtered() -> None: + """A bare pronoun-contraction candidate ('I'm') never labels an + instance line.""" + row: JsonObject = { + "id": "frag-2", + "title": "I'm logging 3 hours at the granite summit", + "canonical_text": "I'm logging 3 hours at the granite summit", + "metadata_json": {}, + } + label = _instance_label(row) + assert label + assert not label.casefold().startswith(("i'", "i’")) + + +PRODUCT_STORE_FIXTURES = ( + ( + "workouts", + ( + ("Ran 3 miles around Lakeview loop", "2026-03-02"), + ("Ran 5 miles along the river path", "2026-03-09"), + ("Ran 4 miles at the track", "2026-03-16"), + ("Ran 6 miles on the canyon trail", "2026-03-23"), + ) + + JUNK_TRIO, + ("mile", "ran"), + ), + ( + "project_decisions", + ( + ("Decided to adopt the Meridian rollout plan for launch", "2026-01-05"), + ("Decided to delay the Meridian rollout by two weeks", "2026-01-12"), + ("Decided the Meridian rollout needs a canary stage", "2026-01-19"), + ), + ("meridian", "rollout", "decided"), + ), + ( + "shopping", + ( + ("Bought a Herman Miller chair for $450", "2026-02-01"), + ("Bought new running shoes for $120", "2026-02-08"), + ("Bought a standing desk mat for $60", "2026-02-15"), + ), + ("bought",), + ), + ( + "reading_list", + ( + ("Finished reading Project Hail Mary in March", "2026-03-05"), + ("Finished reading The Martian in April", "2026-04-02"), + ("Finished reading Recursion in May", "2026-05-07"), + ), + ("reading", "finished"), + ), + ( + "travel", + ( + ("Flew to Lisbon for the spring conference", "2026-04-10"), + ("Flew to Oslo for the design retreat", "2026-05-20"), + ("Flew to Kyoto for the autumn workshop", "2026-10-15"), + ), + ("flew",), + ), +) + + +@pytest.mark.parametrize("name,specs,expected_label_words", PRODUCT_STORE_FIXTURES, ids=[f[0] for f in PRODUCT_STORE_FIXTURES]) +def test_product_shaped_stores_produce_topical_cards(name, specs, expected_label_words) -> None: + """Non-benchmark product stores: every proposed card passes the utility + gate and its label reads like a topic (contains the fixture's topical + words, never a pronoun contraction or function-word label).""" + store = FakeRollupStore() + _seed_texts(store, specs, prefix=f"memory.{name}") + outcome = VNextRollupService(store).propose_rollups() + + assert outcome.proposals, f"{name}: expected at least one topical card" + labels = [proposal["label"] for proposal in outcome.proposals] + assert any( + any(word in label.casefold() for word in expected_label_words) for label in labels + ), f"{name}: no topical label in {labels}" + for proposal in outcome.proposals: + label = proposal["label"].casefold() + # Structural hygiene: no pronoun-contraction or single-letter labels. + assert not label.startswith(("i'", "you'", "it'", "that'", "there'")) + assert len(label) >= 2 + aggregation = proposal["aggregation"] + assert ( + aggregation["distinct_values"] >= 2 or aggregation["distinct_sessions"] >= 3 + ), f"{name}: proposal without aggregation signal: {proposal}" + assert aggregation["label_coherence"] >= 0.5 + + # -- model refinement (existing provider seam, deterministic fallback) ---------------