Repository navigation
ITF: parse the CI dataset's "(line N)" traces - #216
Merged
Merged
Conversation
terminal_exception() never matched "(line 130) AssertionError: ...", so every group's terminal exception was "other" and assertions were classified by symptom text. Fixing it reclassifies 22 groups other -> output_mismatch and changes most fingerprints; closed PRs opened before 2026-10-12 now also match by test so their rejections carry over. routing_exc is no longer needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #215, which only patched guidance routing. This fixes the parser itself.
The bug. Every trace in the CI dataset's
model_results.jsonreads(line 130) AssertionError: …(all 376 integration failures on 2026-10-10).terminal_exception()only matched pytestE Exc:lines or a bareExc:line, so it returned(None, "")for every trace. As a result:terminal_excwas"other", so crash groups were never split by exception type and theAssertionErrorrouting never fired;classify()fell back to symptom regexes, and 86 of 222 AssertionErrors came outother.The fix.
_BARE_EXC_PATaccepts the(line N)prefix.routing_exc()(#215) is now redundant and is removed:instruction_addendumanddispatch_categoryreadterminal_excagain.What it changes, measured on the 2026-10-11 window (370 persistent failures) with the real clustering, the real GitHub PR ledger and 30 days of attribution:
output_mismatch/othercuda_runtimeOpen PRs survive the re-key because of #214's test-overlap match. For closed PRs,
FINGERPRINT_REKEY_DATE = "2026-10-12": a closed ITF PR opened before that date also matches by the tests it lists. Without that, plbart (#48605), blt (#48534) and big_bird (#48437, the<unk>hack, along with its maintainer feedback) would lose their history.The 17 stale and duplicate serge PRs closed by hand on 2026-10-11 now carry
<!-- serge-closed:stale|superseded -->in their bodies, so the bridge doesn't count them as rejections. The two moshi expectation rewrites stay as real rejections.Merging is shipping (the nightly installs
@main). Tests:test_ci_dataset_line_prefix,test_line_prefixed_assertion_is_an_output_mismatch, a pick_targets-built routing test, andtest_pre_rekey_closed_pr_keeps_its_rejection_by_test. Full suite: 1142 passed.prompt_ab.py'sministral3-line-prefixed-assertionfixture now has the shape the triage builds.🤖 Generated with Claude Code