Validate eval fixture destinations

[AbhitejJohn]

Summary

Require every mapping-style environment.files entry to declare a non-empty string dest. Add regression cases for a missing field and an explicitly empty value, and document the fail-closed materialization rule.

Related issue

Related to #1207 evaluation run 37693909743.

Why

The failure is harness/spec-load. Vally rejected the eval before execution with environment-file-missing-field, but check_eval_quality.py skipped destination validation when dest was absent or empty because the check was conditional on a truthy value.

Impact

Malformed fixture mappings now fail the repository quality gate before evaluation dispatch. Existing valid mappings keep the same containment and materialization behavior.

Validation

• python -m py_compile eng\eval-quality\check_eval_quality.py eng\eval-quality\selftest_eval_quality.py — passed.

• Targeted self-test invocation below — passed both fixture mapping requires destination and fixture mapping rejects empty destination.

  @'
  from pathlib import Path
  source = Path(r'eng\eval-quality\selftest_eval_quality.py').read_text(encoding='utf-8')
  namespace = {'__name__': 'eval_quality_targeted_selftest'}
  exec(source.split('print("Eval quality gate')[0], namespace)
  checks = [
      namespace['failing_output_case']('fixture mapping requires destination', namespace['missing_fixture_destination'], 'requires a non-empty string dest'),
      namespace['failing_output_case']('fixture mapping rejects empty destination', namespace['empty_fixture_destination'], 'requires a non-empty string dest'),
  ]
  raise SystemExit(0 if all(checks) else 1)
  '@ | python -

• python eng\eval-quality\check_eval_quality.py — passed; enforced 1 changed eval suite and reported No errors.

• python eng\eval-quality\check_eval_quality.py --all — scanned all 101 eval specs; failed on 544 pre-existing legacy errors, with no dest errors outside the regression cases.

• python eng\eval-quality\selftest_eval_quality.py — the first 6 cases passed, then the existing symlink test stopped on this Windows host with WinError 1314 because symlink privilege is unavailable.

Checklist

• I searched existing issues and pull requests to avoid duplicates.
• I kept this pull request focused and avoided unrelated refactors.
• I added or updated tests, evals, or documentation when changing skill or agent behavior.
• I updated CODEOWNERS when adding or moving owned content. (Not applicable.)
• I updated all marketplace manifests when plugin metadata changed. (Not applicable.)
• I updated eng/known-domains.txt for any new external domains referenced by skill content. (Not applicable.)

Evaluation changes only

For every eval-related change, including a new eval:

• Scenarios are necessary, distinct, and use natural prompts. (No scenarios changed.)
• Graders cover the full result, accept the golden result, and reject a realistic mutation. (Not applicable to gate self-tests.)
• No-op, dormancy, and statistical power are covered where needed. (Not applicable.)
• I ran the applicable production evaluation path and recorded the result above. (No production eval content changed.)
• If this fixes a failed eval, I classified the failure before editing skill content.
• If this broadly changes routing or behavior, I checked separate model-family evidence. (Not a routing or model behavior change.)

[Copilot]

🟡 Changes recommended

The empty-string validation branch is not exercised because the test value parses as YAML null.

1 open finding

• Use a quoted empty string to test empty destination handling · New

What changed in this PR

Strengthens the eval quality gate by rejecting fixture mappings without valid destinations.

Changes:

• Validates environment.files[].dest as a non-empty string.
• Adds regression self-tests.
• Documents fail-closed fixture materialization.

File	Description
eng/​eval-quality/​check_eval_quality.py	Adds destination validation.
eng/​eval-quality/​selftest_eval_quality.py	Adds missing and empty destination cases.
eng/​eval-quality/​README.md	Documents destination requirements.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

👋 @AbhitejJohn — this PR has 1 unresolved review thread(s),merge conflict. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[Copilot]

🟡 Changes recommended

Root-level environment.files mappings remain unvalidated, and the README contains conflicting requirements.

2 open findings

• Validate suite-level environment fixtures · New
• Align mapping destination documentation · New

1 resolved since last review

• Use a quoted empty string to test empty destination handling

🧠 Review effort: Balanced

[Copilot]

🟢 Approval recommended

The implementation, regression coverage, and documentation consistently address the reported validation gap.

0 open findings

2 resolved since last review

• Validate suite-level environment fixtures
• Align mapping destination documentation

🧠 Review effort: Balanced

[github-actions]

✅ Evaluation passed for 3626166. cc @AbhitejJohn @JanKrivanek — please review.