Bug: unmatched entity is diluted into unrelated generic query results

[1wgrumph]

Migrated from private source issue alphazede/bran-dev#18

Summary

BRAN query can dilute an unmatched, high-specificity entity with generic request terms and return unrelated locators without a miss or low-confidence warning.

This creates a grounding hazard: an agent can ask for one named component, receive unrelated files selected from generic terms such as tests or documentation, and mistake command success for evidence coverage.

Environment

• Repository queried: alphazede/developers
• Native policy: present
• bran check <repo> bran-strict: selected profile passed
• Pinned runtime: bran 0.1.0
• Source commit: 39abfdcc823e0a1754c56ceb4a28df9faae375d8
• Runtime SHA-256 matched the release pin

Reproduction

Exact entity controls:

bran query <repo> "grok-safe"
bran query <repo> "grok_safe_collector.py"

Both return:

{
  "status": "ok",
  "selected_locators": [],
  "source_rankings": [],
  "warnings": [],
  "failures": []
}

Expanded natural-language request:

bran query <repo> "Where are the Grok Safe wrapper, runner, completion semantics, transcript handling, and tests documented and implemented?"

This returns four az-sports-dev locators unrelated to Grok Safe. The selections are attributed to metadata_seed; rankings are driven by generic path matches such as test/public content. The response still has:

{
  "status": "ok",
  "warnings": [],
  "failures": []
}

The named entity is absent from every selected locator.

Expected behavior

When a query contains a high-specificity anchor that has no match, BRAN should not silently satisfy the request using only generic residual terms.

One safe result would be:

• empty selections plus a typed semantic miss;
• a warning naming only the unmatched query token/entity, not repository content;
• explicit query-term coverage showing that the named anchor was not grounded; or
• low-confidence partial results clearly marked as not answering the named component.

Command execution success may remain status:"ok", but the response needs a machine-readable semantic outcome such as query_outcome:"miss" or coverage:"partial_unanchored".

Acceptance criteria

• Add a fixture with one unmatched unique entity plus generic words that match many indexed paths.
• Do not return those generic matches as fully grounded results.
• Add machine-readable matched/unmatched anchor or term coverage.
• Distinguish exact miss, grounded partial match, and unanchored generic match.
• Preserve deterministic ranking and bounded output.
• Do not leak repository content in warnings.
• Confirm ordinary multi-term queries with a grounded entity still return ranked evidence.
• Document how callers should interpret empty selections versus partial/unanchored selections.

[1wgrumph]

Comment by @1wgrumph on original issue:

Fixed in bfe9eb9, merged to main as 037e415.

Unmatched high-specificity entities now return no rankings instead of diluted generic matches. Verified on merged main:

bran query . "zzq-nonexistent-collector"
  -> ranked: 0 | warnings: [unmatched_query_terms: zzq-nonexistent-collector]

bran query . "Where are the zzq-nonexistent-collector wrapper, runner,
              semantics, transcript handling, and tests ...?"
  -> ranked: 0 | warnings: [unmatched_query_terms: zzq-nonexistent-collector]

Previously these returned 4 and 32 unrelated locators with status: ok.

The sub-token defect is also fixed: the warning now names the full entity rather than only the zzq fragment, so a partial token match no longer counts as an entity match.

Precision regression from the earlier body-ranking change is resolved (32 -> 10 results on a normal body query). Positive controls confirm no over-suppression: valid_sha256 -> crates/bran-core/src/agent/runtime.rs rank 1 (exact:body); okf-v0.2 compatibility profile -> crates/bran-core/src/profile.rs rank 1.

#19 regression held: bran query . "compatible superset not a fork" still ranks docs/integrations/proposals/okf-layered-profile-separation.md first.

Gates: 44 bran-cli + 288 bran-core tests pass; ./tools/ci/check.sh --fast PASS.

[1wgrumph]

Comment by @1wgrumph on original issue:

Reopening. I closed this too confidently.

The reported repro is genuinely fixed. Both original cases used compound identifiers (grok-safe, grok_safe_collector.py), and those now return zero rankings with an unmatched_query_terms warning.

But the general dilution behaviour survives for a bare single-word entity inside a natural-language query. Measured on this repository at 0a1de91:

Query	ranked	warning
zephyrite	0	zephyrite
zephyrite-collector	0	zephyrite-collector
zephyrite_collector	0	zephyrite_collector
where is the zephyrite collector configured and tested	32	zephyrite
where is the zephyrite-collector configured and tested	0	zephyrite-collector

Row 4 is the defect this issue describes, still present. The unmatched term is reported, but 32 unrelated documents are still returned under status: ok, selected on the generic words around it. A caller that does not read warnings sees a successful, plausible-looking result set about a component that does not exist.

So suppression currently keys on the term being a compound identifier (hyphen or underscore), treated as one high-specificity unit. A lone word is not distinguished from ordinary query vocabulary.

What is still needed

A principled rule for when a single unmatched token is specific enough that its absence should suppress the whole result. Candidates worth evaluating: corpus rarity (a term absent from the bundle and unlike ordinary English), casing or morphology signals, or requiring at least one matched high-specificity term before returning anything at all.

Acceptance

• A bare unmatched high-specificity word inside a longer query suppresses results or is explicitly marked low-confidence, not merely warned about.
• Ordinary vocabulary in a query does not trigger false suppression; natural-language queries that should match still do.
• Regression tests cover the bare-word case in a sentence, not only the compound-identifier case.

[1wgrumph]

Comment by @1wgrumph on original issue:

Work in progress on branch fix/18-bare-word-dilution (c31d2a2, not merged).

The rule implemented

An unmatched bare term counts as high-specificity when it either mixes letters and digits, or is at least 8 characters, is not in a small guard list, and has no stem relation to the corpus's own vocabulary. The corpus-derived stem signal is primary; the guard list is a measured residual.

Two alternatives were rejected with evidence: a naive stem rule false-suppressed 14 common words and was polluted by binary-asset tokens, and requiring at least one matched high-specificity term broke release manifest signature at every threshold tried.

Verified working

query	ranked	note
zephyrite-collector	0	compound, alone
zephyrite_collector	0	underscore, alone
where is the zephyrite-collector configured and tested	0	compound in a sentence
where is the zephyrite collector configured and tested	0	the bare-word case this issue is about
where is frontmatter validated	32	unchanged
valid_sha256	5	unchanged
compatible superset not a fork	11, correct file rank 1	#19 preserved
release manifest signature	32	unchanged

Rare-but-real terms are unaffected: sigstore keyless signing 15, obsidian export 32, crossterm 3, okf_status frontmatter 32.

Why it is not merged

False suppression on ordinary English. A legitimate query is destroyed by a common word that happens to be absent from the corpus:

"unfortunately what breaks the gate"  ->  ranked=0, unmatched_query_terms: unfortunately

unfortunately is 13 characters, ordinary English, and absent here, so it clears the length threshold and fails the stem check. The 3-word guard list cannot scale to cover this class; the next such word is one bug report away.

Short distinctive terms still dilute. The 8-character threshold means genuinely specific short words behave as before:

"explain the parlay permutation algorithm"  ->  ranked=29, unmatched_query_terms: parlay
"what is the Kubernetes deployment strategy"  ->  ranked=0,  unmatched_query_terms: kubernetes

Both absent from this repository; only the 10-character one suppresses. The threshold is doing the work that specificity should be doing.

What a correct rule needs

Length is a proxy for specificity and a poor one in both directions. A better signal would separate "rare in general language" from "rare in this corpus", since unfortunately is common in English and absent here, while parlay is uncommon in English and also absent. The current rule cannot tell those apart.

[1wgrumph]

Comment by @1wgrumph on original issue:

Status update — implementation is integrated and pushed to bran-dev main at 3feeba3ab30f0504a601ffbb732f98801e25c5e8.

• Query results now expose machine-readable grounded, miss, and partial_unanchored outcomes plus matched/unmatched term coverage.
• The reopened bare-word case is covered without length, dictionary, or morphology suppression heuristics; ordinary unmatched vocabulary keeps useful rankings but cannot look fully grounded.
• The integrated fast/full gates pass, including the BRAN: agents list hard-errors with missing_provider_descriptor #18 regressions and the later DOCX adapter: preserve Word structure, review state, and citation anchors #21/PPTX adapter: preserve slide structure, notes, media, and reading order #22 query changes.
• Public snapshot commit 35f577845a428e0788a1b60b58aebd939bf96220 is in draft PR Export bran-dev 3feeba3 public snapshot #11.

Keeping this issue open: PR #11 is still draft, required fast/CodeQL checks are in progress, owner review and separate merge authorization are pending, and the blocking public-main drift guard cannot pass until that PR is merged.

[1wgrumph]

Comment by @1wgrumph on original issue:

Merge closeout update — the guarded PR #11 merge was attempted against exact head 35f577845a428e0788a1b60b58aebd939bf96220 and GitHub rejected merge method merge with HTTP 405.

Verified repository settings:

• allow_merge_commit=false
• allow_rebase_merge=true
• allow_squash_merge=false

BRAN's export contract requires a merge commit so the reviewed export commit remains an approved parent; rebase or squash is not an allowed fallback. PR #11 has been returned to draft with this blocker recorded. All hosted checks remain green and no PR content changed.

Keeping this issue open. The next owner decision is whether to authorize enabling merge commits on alphazede/bran (temporarily or permanently). After that, PR #11 can be merged, the public-main drift guard rerun, and eligible issues closed. Tagging, releasing, and package publication remain out of scope.

[1wgrumph]

Comment by @1wgrumph on original issue:

Closing as completed.

Final receipts:

• Canonical source: alphazede/bran-dev main at 3feeba3ab30f0504a601ffbb732f98801e25c5e8.
• Public export: 35f577845a428e0788a1b60b58aebd939bf96220.
• Public merge: Export bran-dev 3feeba3 public snapshot #11, merge commit d05c5f9f0eb24f32956dd49c88cf2f4e200d1245.
• The merge commit preserves the reviewed export as its second parent.
• Query outcomes distinguish grounded, miss, and partial_unanchored with matched/unmatched term coverage; the reopened bare-word case and ordinary-vocabulary control are covered.
• Hosted fast/CodeQL, local full, public bran-strict, and the exact public drift guard pass.

No tag, release, or package publication was performed.

[1wgrumph]

Closed in original repository (bran-dev#18).