Send back expectation changes that are not on the CI runner's key

[tarekziade]

serge's expectation updates mostly don't have the shape maintainers accept. Of the serge PRs opened in Sept–Oct 2026, the 3 value updates that touched only the CI runner's ("cuda", 8) entry all merged cleanly (huggingface/transformers#48515, #49068, #49076). The one that overwrote ("cuda", None) merged with a wrong value (#48580). The open queue overwrote (None, None) (#47616, #49480), rewrote XPU values with no XPU evidence (#48955), loosened tolerances (#49143) and edited plain literals in place (#47840). The prompt already asked for a device key. This PR enforces it.

reviewbot/expectation_keys.py parses each changed test file before and after the patch (ast), maps every Expectations({...}) block key → value, and reports:

• a change to, or removal of, any key other than the runner's ("cuda", 8) / ("cuda", (8, 6)) (A10G, compute capability 8.6). Demotion is allowed: an existing value moved unchanged to (None, None), which is what #48198 did;
• an added key other than the runner's, or a (None, None) value that isn't an existing one;
• a runner value moving by more than 5% of the block's largest magnitude. Accepted drift was 0.4% (#48515) and 3.6% (#49068, bf16 near zero); the pvt groups that should have been refused moved 18% and 45%;
• a loosened atol/rtol, in any patch;
• in an expectation-only patch, an expected value or assertion changed outside any Expectations block. Test-code fixes are left alone (#47697 renamed a config key).

Wiring. _validate_patch runs it after the patch applies, before the brevity pass and the normalizer. A violation resets the worktree and comes back as correction feedback (rule text included), using the same TASK_NORMALIZE_MAX_RETRIES budget. If a violation survives the budget, the PR still opens, with a "⚠️ Expectation change not in the maintainer shape" section listing the problems. TASK_EXPECTATION_KEY_CHECK=0 turns the check off.

Checked against real PRs. Every serge PR since 2026-07-20, comparing each test file at its base and head commits:

• all 4 PRs I'd call auto-merge-eligible pass (#48838, #48882, #48945, #49040)
• 7 of 8 red-flag PRs are flagged (#47698 changes the assertion's structure, not a key)
• all 3 "right change, wrong key" PRs are flagged, with a fix the model can apply
• the rule is stricter than maintainers on 5 merged PRs (plain-literal text edits, broad keys). serge can always meet it by converting to Expectations with the runner key.

The prompt side (the guidance now names the runner key, and a routing bug kept it from reaching most assertion groups) is in huggingface/transformers-ci#215.

Tests: tests/test_expectation_keys.py (17 cases cut down from the PRs above) and ExpectationShapeValidationTests in tests/test_tasks.py. Full suite: 1390 passed.

🤖 Generated with Claude Code