Skip to content

ITF: parse the CI dataset's "(line N)" traces - #216

Merged
tarekziade merged 1 commit into
mainfrom
itf/parse-line-prefixed-traces
Oct 11, 2026
Merged

tarekziade merged 1 commit into
mainfrom
itf/parse-line-prefixed-traces

Conversation

@tarekziade

Copy link
Copy Markdown
Collaborator

Follow-up to #215, which only patched guidance routing. This fixes the parser itself.

The bug. Every trace in the CI dataset's model_results.json reads (line 130) AssertionError: … (all 376 integration failures on 2026-10-10). terminal_exception() only matched pytest E Exc: lines or a bare Exc: line, so it returned (None, "") for every trace. As a result:

  • every group's terminal_exc was "other", so crash groups were never split by exception type and the AssertionError routing never fired;
  • classify() fell back to symptom regexes, and 86 of 222 AssertionErrors came out other.

The fix. _BARE_EXC_PAT accepts the (line N) prefix. routing_exc() (#215) is now redundant and is removed: instruction_addendum and dispatch_category read terminal_exc again.

What it changes, measured on the 2026-10-11 window (370 persistent failures) with the real clustering, the real GitHub PR ledger and 30 days of attribution:

before after
groups 110 106
output_mismatch / other 48 / 50 70 / 22
cuda_runtime 1 3
fingerprints unchanged – 42 (68 change)
open serge PRs matched to a group 27 27
rejections attached 19 19 (16 after the markers below)
groups deferred for ≥2 rejections minimax_m3_vl, deepseek_v32, moshi, qwen3_vl_moe same

Open PRs survive the re-key because of #214's test-overlap match. For closed PRs, FINGERPRINT_REKEY_DATE = "2026-10-12": a closed ITF PR opened before that date also matches by the tests it lists. Without that, plbart (#48605), blt (#48534) and big_bird (#48437, the <unk> hack, along with its maintainer feedback) would lose their history.

The 17 stale and duplicate serge PRs closed by hand on 2026-10-11 now carry <!-- serge-closed:stale|superseded --> in their bodies, so the bridge doesn't count them as rejections. The two moshi expectation rewrites stay as real rejections.

Merging is shipping (the nightly installs @main). Tests: test_ci_dataset_line_prefix, test_line_prefixed_assertion_is_an_output_mismatch, a pick_targets-built routing test, and test_pre_rekey_closed_pr_keeps_its_rejection_by_test. Full suite: 1142 passed. prompt_ab.py's ministral3-line-prefixed-assertion fixture now has the shape the triage builds.

🤖 Generated with Claude Code

terminal_exception() never matched "(line 130)  AssertionError: ...", so every
group's terminal exception was "other" and assertions were classified by symptom
text. Fixing it reclassifies 22 groups other -> output_mismatch and changes most
fingerprints; closed PRs opened before 2026-10-12 now also match by test so their
rejections carry over. routing_exc is no longer needed.
@tarekziade
tarekziade merged commit 7534a9c into main Oct 11, 2026
@tarekziade
tarekziade deleted the itf/parse-line-prefixed-traces branch October 11, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant