feat(data): task_data validation gating: auto mode, misplaced-field check, repo drift test (#2827 cherry-pick into r0.6.0) - #2854
Open
adil-a wants to merge 1 commit into
Open
Conversation
Contributor
Author
|
/ok to test 3444562 |
…heck, repo drift test (#2827) ## What this does This PR turns the schema validation from #2800 into a gate. **Auto validation mode.** `task_data_validation` gains an `auto` setting. Auto is now the default. Auto resolves to `error` in `example_validation` mode and to `warn` in `train_preparation` mode. Committed example data is known clean, so a bad row should fail the PR gate. User training data must not crash long collate jobs, so it warns instead. Before: the default was `warn` everywhere. A bad committed example row only printed a warning. After: `gym dataset collate +mode=example_validation` fails on a bad row with a per-row error report. Training collate still warns and completes. **Misplaced-field check.** Some schema fields carry a `legacy_location: verifier_metadata` marker. The marker means the server reads that field only from inside `verifier_metadata`. If a row puts the field at the top level instead, the server never sees it. The validator now counts these rows and reports them as `misplaced_keys`. The instruction_following schema drops its markers because that server accepts both placements. **Schema presence test.** Every resources server with an `app.py` must ship a `task_data.py`. New servers can no longer skip the schema. **Repo drift test.** A new unit test walks every committed dataset in the repo and validates each row against its server's schema. If a server's wire model changes and the schema does not, this test fails. This is the drift gate requested in the #2795 review. The presence test already paid for itself. terminal_bench_2_1 landed on main after the schema sweep and had no schema. The test caught it, and this PR adds its schema. **Review fixes.** Three additions from the first review round. Undeclared row keys now fail the gate instead of only being reported; no committed row carries one, and the stricter gate immediately caught two missing fields in a new schema. The drift test now also validates datasets owned by self-contained agents (agents with no resources server), using the same schema fallback collate uses; schemas for the nine such agents with committed data are included. `normalize_task_fields` now unpacks a migrated `task_data` container the same way it unpacks `verifier_metadata`, so nested fields cannot dodge validation. ## Verification - All 302 tests in `tests/unit_tests/test_task_data.py` pass. - Live smoke: a bad row (`expected_city` as an int) fails `gym dataset collate` in `example_validation` mode with a per-row pydantic report. The same row only warns in `train_preparation` mode and the run completes. - Golden routing gate: base capture vs branch capture, strict compare, 0 differences across 233 closures. - ruff and pre-commit are clean. --------- Signed-off-by: Adil Asif <adasif@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit b369322)
ananthsub
force-pushed
the
cherry-pick-2827-r0.6.0
branch
from
August 28, 2026 21:44
3444562 to
27f790a
Compare
Contributor
Author
|
/ok to test 27f790a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #2827 (merge commit b369322) into r0.6.0.
Clean pick: #2827 was built on the exact commit the release branch was cut from, so the commit applied without conflicts. All 330 task_data and collate tests pass on this branch.
See #2827 for the full description and verification.