[PROBLEM] Rank repos by a copy-count buried in prose, joined on names that do not match

[Suraj-gameramp]

Business question (exact ask)

Which repository of a given language was copied the most, where the copy count is written inside each repository's free-text description and has to be joined to a second source that keys on a differently formatted name? (Note: this is a reproducible finding from running agents on a public benchmark, not a proprietary production incident. It stands in for a common cross-source reconciliation problem, so I am submitting it as a representative case.)

Category (select all that apply)

Integrating Unstructured Data, Implicit Relationship Discovery

Time window & calendars

Not time-bounded. Static snapshot.

Data sources & backends (no data; just names)

• repo_metadata (SQLite): one row per repo, with a free-text repo_data_description that embeds phrases like "seen N times" or "appearing N times", plus a language_description field.
• A second source keyed on a repo name in a different format (org/name versus orgname/name).
• Backing: the github_repos domain of the DataAgentBench public benchmark.

Entities & identifiers

• Repository. The identifier shows up as org/name in one source and as a normalized or differently cased string in the other.
• Source of truth for the count is the number written in the prose description.

Join logic & business rules

• Pull the integer copy count out of the free text (several phrasings, strip thousands separators).
• Filter to the requested language, which itself is sometimes stated in prose rather than a clean field.
• Normalize repo names across the two sources before joining (case, separators, org-prefix format).

Expected output shape (columns, types, grain)

grain: single record (or a short ranked list)
columns:
  - repo_name: string
  - copy_count: integer

Tools/approach attempted

Claude Opus 4.7 and Claude Fable 5, in a spec-then-execute pipeline with a regex parsing helper and a fuzzy name matcher.

Failure mode (be specific)

Two failures stack, and both are defensible but wrong:

1. Parsing the count out of prose. There are competing phrasings ("seen N times", "appearing N times", "copied N times") and more than one number in the description, so the agent picks a plausible parse that is not the intended one.
2. The fuzzy name join. The two sources key on names in different formats, and the agent's normalization lands on the wrong match. The scorer flagged it directly: "No fuzzy match found for 'swiftandroid/swift' within 3-character distance." The wrong repo was chosen because of the name-format mismatch.

Both are genuinely the agent's fault, not benchmark defects, and in our runs this was the hardest agent-side surface: pulling a value out of text and then resolving entities across sources when there is no clean join key. This query scored 0 out of 5 across all trials.

Scale (approximate)

≤100k rows

Policy/constraints (optional)

No response

Contribution agreement

• I confirm this submission contains no proprietary data or secrets
• I agree this problem can be used in the public benchmark