Skip to content

business_letters scorer crashes on non-conforming model responses #111

Description

@MHindermann

Summary

Scoring business_letters for command-a-vision-07-2025 (T1142) crashes with AttributeError: 'NoneType' object has no attribute 'send_date' when the model's response doesn't conform to the required structured output.

Repro

Run T1142 (scripts/run_benchmarks_command-a-vision-07-2025.py). Cohere returns the four fields at top level plus an extra content key instead of the expected metadata-wrapped structure.

Root cause (benchmarks/business_letters/benchmark.py)

  1. Cohere structured-output validation fails (metadata Field required) → falls back to JSON mode → answer.parsed = {letter_title, send_date, sender_persons, receiver_persons,
    content} (no metadata wrapper).
  2. score_request_answer: data["metadata"] → KeyError → fallback _initialize_letter(raw_letter=data).
  3. _initialize_letter calls Letter(**raw_letter); Letter has no content field → TypeError → returns None.
  4. _score_send_date(predicted_letter=None) → None.send_date → crash.

Expected behavior

A model that fails to return the required structured output should be scored as a failure for that item (0 TP, all ground-truth as FN), not crash the run, and not be salvaged by stripping non-conforming fields, which would credit a model for ignoring the schema.

Proposed fix

  • score_request_answer: when response_letter is None (parsing failed for any reason), return the complete-failure score (0 TP, all FN)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions