[PROBLEM] Average processing fee when a null rule field means "applies to all"

[Suraj-gameramp]

Business question (exact ask)

What is the average fee a card scheme would charge for a credit transaction of 10 EUR? Fees come from a rule table where each rule can constrain card_scheme, is_credit, MCC, account_type, and so on. (Note: this is a reproducible finding from running agents on a public benchmark, not a proprietary production incident. It maps to a very common enterprise pattern, so I am submitting it as a representative case.)

Category (select all that apply)

Single-Source SQL, Business Term Disambiguation

Time window & calendars

Not time-bounded. A static rule table plus one month of transactions. No fiscal calendar or timezone dependence.

Data sources & backends (no data; just names)

• Rule/reference table "fees" (JSON, roughly 1,000 rules), with fields like card_scheme, is_credit, mcc, account_type, fixed_amount, rate.
• Transactions table "payments" (CSV, roughly 138k rows).
• A documentation file (manual.md) that states the null convention in prose.
• Backing: local files loaded into pandas/DuckDB (the DABStep public benchmark).

Entities & identifiers

• Fee rule (no stable primary key; matched by attribute predicates).
• Transaction (psp_reference).
• Source of truth for the null convention is manual.md line 95: "If a field is set to null it means that it applies to all possible values of that field."

Join logic & business rules

• A rule applies to a transaction when every non-null field in the rule matches the transaction, and every null field acts as a wildcard that matches anything.
• So for "credit transactions" the applicable rules are the ones where is_credit is True OR is_credit is null. The null rules apply to credit too.
• Take the mean fee over the applicable rules.

Expected output shape (columns, types, grain)

grain: single scalar
columns:
  - avg_fee_eur: number (6 decimal places)

Tools/approach attempted

Kimi K2.6, Claude Sonnet, Claude Opus, and GPT-5, each as the executor in a ReAct-style Python-tool agent, and also inside a spec-then-execute pipeline.

Failure mode (be specific)

Every model filters with strict equality (is_credit == True) and quietly drops the rules where is_credit is null, even though the doc it just read says null means "applies to all values." The interesting part is that this is not a reading problem. Models that state the convention correctly in their own words still write the wrong filter when they get to code. We verified it by reproducing both readings on the real data:

• Task 1275 (SwiftCharge, credit, 10 EUR): is_credit==True gives 0.122408, which is the model's answer; True OR null gives 0.120609, which matches gold exactly to 6 decimals.
• Task 1278 (NexPay, credit, 50 EUR): is_credit==True gives 0.353053; True OR null gives 0.352294, which matches gold.

This one bug accounted for about 27 of 93 committed-answer failures for one model. Using a stronger model did not fix it (Sonnet to Opus actually made it a little worse), and neither did adding an explicit instruction to honor the documented null semantics.

Scale (approximate)

100k–10M rows

Policy/constraints (optional)

No response

Contribution agreement

• I confirm this submission contains no proprietary data or secrets
• I agree this problem can be used in the public benchmark