[PROBLEM] Which policy did this deal violate? Agent stops at the first plausible match

[Suraj-gameramp]

Business question (exact ask)

Given a sales quote or opportunity, identify which internal policy (knowledge-base article) it violates. (Note: this is a reproducible finding from running an agent on a public benchmark, not a proprietary production incident. The pattern is common in real compliance work, so I am submitting it as a representative case.)

Category (select all that apply)

Governance & Compliance, Single-Source SQL

Time window & calendars

Point-in-time on the current CRM snapshot. No fiscal calendar dependence.

Data sources & backends (no data; just names)

• A Salesforce-style CRM with objects like Quote, Opportunity, Account, and Knowledge_Article (a policy library of a few dozen articles).
• Backing: PostgreSQL (the crmarenapro domain of the DataAgentBench public benchmark).

Entities & identifiers

• Quote (quote_id) and policy article (article_id).
• Source of truth is the policy library. The answer is one specific article_id.

Join logic & business rules

• Check the quote against every policy article's conditions, not just the first one that looks relevant.
• The correct answer is the article whose conditions the quote actually breaches (for example a volume-based-discount policy), which may not be the first article whose topic looks like a match.

Expected output shape (columns, types, grain)

grain: single record
columns:
  - article_id: string (the violated policy)

Tools/approach attempted

Claude Opus 4.7 as the executor, with Python and SQL tools, iteration cap set to both 40 and 60, across 13 independent trials.

Failure mode (be specific)

The agent finds one plausible supporting article (a mandatory-bundles policy), calls it the answer, and stops without checking the rest of the library. One of the articles it skipped (volume-based-discounts) is the actual violation. 12 of 13 independent runs failed the same way and returned the same wrong article.
Two things surprised us:

1. Raising the iteration cap from 40 to 60 did not help. The model decides it is done around iteration 8 to 17 and never uses the extra budget. This is early stopping by choice, not a resource limit.
2. The one run that got it right only did so because PostgreSQL happened to be in a broken state. The repeated connection failures forced about 14 retry iterations, and during that flailing the agent fell back to enumerating every policy article, which is how it finally found the right one.

The takeaway is that giving the agent more steps does not fix early stopping. Being exhaustive has to be an explicit obligation ("list all candidates before you pick one"), and current agents do not impose that on themselves.

Scale (approximate)

≤100k rows

Policy/constraints (optional)

The task is policy-grounded by nature. The answer is a specific governance clause, and a confident wrong citation is worse than saying "not sure."

Contribution agreement

• I confirm this submission contains no proprietary data or secrets
• I agree this problem can be used in the public benchmark