Preserve Copilot SDK runtime dotfiles in validator packages

[AbhitejJohn]

Summary

Preserve required Copilot SDK runtime dotfiles when NuGet packs the skill-validator tool.

The validator checks for SDK dependency updates passed, but RID-specific packaging failed on Windows and macOS because NuGet excluded plugins/computer-use/.release-target by default:

• PR #1271 check run: NU5119 ... .release-target was not added to the package ... use -NoDefaultExcludes.
• PR #1202 check run: the same NU5119 failure on Windows x64, Windows ARM64, and macOS ARM64.

This adds the NuGet-supported <NoDefaultExcludes>true</NoDefaultExcludes> property only to eng/skill-validator/src/SkillValidator.csproj. It does not suppress warnings, remove the runtime marker, or weaken validation.

Copilot prepared this draft on Abhitej's behalf.

Related issue

N/A. The direct incidents are PR #1271 and PR #1202.

Root cause and history

GitHub.Copilot.SDK 1.0.16 bundles Copilot CLI 1.0.90, whose published runtime contains plugins/computer-use/.release-target. NuGet's default pack exclusions reject files that start with ., and this repository treats NU5119 as an error. The project had no NoDefaultExcludes setting on main, so package creation failed after build and tests had passed.

Repository history contains a prior local-only commit, 9975eea5c5 (Preserve Copilot SDK runtime dotfiles in tool packages), with the same minimal property. It was not an ancestor of main and was not present in an open pull request. The current patch restores that proven intent on a new branch.

Validation

• dotnet pack eng\skill-validator\src\SkillValidator.csproj --use-current-runtime -p:PublishAot=false /bl:artifacts\logs\pack-rid-sdk-1.0.16-runtime-dotfiles-noaot.binlog with the repository-pinned SDK 11.0.100-preview.3.26207.106 and a temporary local GitHub.Copilot.SDK 1.0.16 dependency: passed.
• NuGet archive inspection: passed; exactly one entry matched tools/net10.0/win-x64/runtimes/win-x64/native/plugins/computer-use/.release-target.
• dotnet test eng\skill-validator\SkillValidator.slnx: passed, 962/962 tests.
• git diff --check: passed.
• Full local NativeAOT pack: not completed because this machine does not have the Visual Studio C++ linker workload. The command passed the original NU5119 point and then stopped at Platform linker not found. GitHub Actions remains the cross-platform NativeAOT validation source.

Risk and limits

• Risk: Low. NuGet can now include dotfiles already present in the explicit package source set. This is required for the Copilot CLI runtime marker.
• Security: The change does not add secrets, permissions, domains, or external services. It does not suppress package analysis.
• Limit: The local end-to-end pack used PublishAot=false because the C++ linker workload is absent. The package exclusion behavior and resulting archive were validated directly.

Rollback

Remove <NoDefaultExcludes>true</NoDefaultExcludes> or revert commit a25501053. This returns the previous package behavior and will also restore the known NU5119 failure when the runtime contains .release-target.

Checklist

• I searched existing issues and pull requests to avoid duplicates.
• I kept this pull request focused and avoided unrelated refactors.
• I added or updated tests, evals, or documentation when changing skill or agent behavior. (Existing packaging and test coverage validate this project-only build fix.)
• I updated CODEOWNERS when adding or moving owned content. (No owned content moved or added.)
• I updated all marketplace manifests when plugin metadata changed. (No plugin metadata changed.)
• I updated eng/known-domains.txt for any new external domains referenced by skill content. (No skill content or domains changed.)

Evaluation changes only

Not applicable; this change does not modify an evaluation.

• Scenarios are necessary, distinct, and use natural prompts.
• Graders cover the full result, accept the golden result, and reject a realistic mutation.
• No-op, dormancy, and statistical power are covered where needed.
• I ran the applicable production evaluation path and recorded the result above.
• If this fixes a failed eval, I classified the failure before editing skill content.
• If this broadly changes routing or behavior, I checked separate model-family evidence.

[Copilot]

🟢 Approval recommended

The focused NuGet setting directly addresses the verified cross-platform packaging failure without weakening validation.

0 open findings

What changed in this PR

Ensures NuGet preserves required Copilot SDK runtime dotfiles when packaging the skill-validator.

Changes:

• Enables NoDefaultExcludes for the tool package.
• Documents the required .release-target runtime marker.

File	Description
eng/​skill-validator/​src/​SkillValidator.csproj	Preserves runtime dotfiles during NuGet packing.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

❌ Evaluation did not complete (upstream job failed or was skipped). View workflow run

[github-actions]

❌ Evaluation did not complete (upstream job failed or was skipped). View workflow run

[Copilot]

🟢 Approval recommended

The focused packaging change directly addresses the documented cross-platform NU5119 failures.

0 open findings

🧠 Review effort: Balanced

[github-actions]

📊 Skill, Agent, and Workflow Evaluation Results

Workflow results measure offline prompt decisions and proposed outputs, not live Actions jobs or publication.

22 model/target results across 11 targets and 2 models — ✅ 4 improved, ➖ 12 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only), 🔻 6 objective regressions.

Measurement identity: evaluated commit e4fb06acaed78e14efb45c9f1d3236c9378cb80f; 2 judge models.

Measurement health: 22 expected / 22 observed / 22 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: enabled; 6 results show a proven objective regression.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
build-failure-analysis	claude-sonnet-5	🔻 Objective regression	n=10; 2W/4T/4L; d=6; p=0.344; net -20.0%	—	—	Inspect objective completion losses and fix them before merge.
build-failure-analysis	gpt-5.6-luna	🔻 Objective regression	n=10; 4W/5T/1L; d=5; p=0.188; net +30.0%	—	—	Inspect objective completion losses and fix them before merge.
msbuild-quality-review	claude-sonnet-5	➖ Improvement signal, unproven	n=10; 5W/4T/1L; d=6; p=0.109; net +40.0%	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
msbuild-quality-review	gpt-5.6-luna	🔻 Objective regression	n=10; 3W/6T/1L; d=4; p=0.312; net +20.0%	—	—	Inspect objective completion losses and fix them before merge.
technology-selection	claude-sonnet-5	✅ Improved	n=6; 5W/1T/0L; d=5; p=0.031; net +83.3%	🟡 0.49	Activation: isolated 6/6; plugin 5/6	Fix activation gaps; Review overfit evidence.
technology-selection	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%	🟡 0.41	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-authoring	claude-sonnet-5	➖ Improvement signal, unproven	n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%	✅ 0.13	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
template-authoring	gpt-5.6-luna	➖ Improvement signal, unproven	n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%	🟡 0.38	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
template-comparison	claude-sonnet-5	✅ Improved	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%	🟡 0.35	—	Review overfit evidence.
template-comparison	gpt-5.6-luna	➖ Mixed evidence	n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%	✅ 0.20	—	Compare winning and losing scenarios to isolate where the target helps versus hurts.
template-discovery	claude-sonnet-5	➖ Improvement signal, unproven	n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%	🟡 0.33	Activation: isolated 7/7; plugin 6/7	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
template-discovery	gpt-5.6-luna	➖ Improvement signal, unproven	n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%	🟡 0.30	Activation: isolated 7/7; plugin 6/7	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
template-instantiation	claude-sonnet-5	➖ Improvement signal, tie-limited	n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%	🟡 0.28	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-instantiation	gpt-5.6-luna	➖ Mixed evidence	n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%	✅ 0.16	—	Compare winning and losing scenarios to isolate where the target helps versus hurts.
template-smart-defaults	claude-sonnet-5	✅ Improved	n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%	🟡 0.48	Activation: isolated 7/7; plugin 6/7	Fix activation gaps; Review overfit evidence.
template-smart-defaults	gpt-5.6-luna	✅ Improved	n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%	🟡 0.32	Activation: isolated 7/7; plugin 5/7	Fix activation gaps; Review overfit evidence.
template-validation	claude-sonnet-5	➖ Improvement signal, tie-limited	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%	🟡 0.44	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
template-validation	gpt-5.6-luna	➖ Mixed evidence	n=7; 2W/3T/2L; d=4; p=0.687; net +0.0%	🟡 0.39	—	Compare winning and losing scenarios to isolate where the target helps versus hurts.
test-failure-analysis	claude-sonnet-5	🔻 Objective regression	n=12; 7W/1T/4L; d=11; p=0.274; net +25.0%	—	—	Inspect objective completion losses and fix them before merge.
test-failure-analysis	gpt-5.6-luna	🔻 Objective regression	n=12; 9W/3T/0L; d=9; p=0.002; net +75.0%	—	—	Inspect objective completion losses and fix them before merge.
unskip-closed-tests	claude-sonnet-5	➖ Improvement signal, unproven	n=11; 5W/4T/2L; d=7; p=0.227; net +27.3%	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
unskip-closed-tests	gpt-5.6-luna	🔻 Objective regression	n=11; 3W/4T/4L; d=7; p=0.500; net -9.1%	—	—	Inspect objective completion losses and fix them before merge.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

🔻 Objective regression — build-failure-analysis (claude-sonnet-5)

Why: Net win -20.0% (2W/4T/4L over 10 preference-eligible stimulus vote(s), sign test p=0.344), mean preference -20.0% across 10 paired run(s) — no improvement — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 2W/4T/4L; d=6; p=0.344; net -20.0%

Repeated-run reliability (not used by the gate): 10 paired runs (2W/4T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Bound a restore diagnosis to the searched feed	Eligible	-100.0%	-100.0%	0/0/1
▼ Group a removed API cascade in the second build leg	Eligible	-100.0%	-40.0%	0/0/1
▼ Report the absence of all binary logs without a diagnosis	Eligible	-100.0%	-100.0%	0/0/1
= Resolve a disposal warning promoted to an error	Eligible	+0.0%	+0.0%	0/1/0
= Retain useful findings while naming a missing module	Eligible	+0.0%	+0.0%	0/1/0
= Suppress obsolete analysis after a head update	Eligible	+0.0%	+0.0%	0/1/0
▼ Suppress obsolete analysis after the merge revision moves	Eligible	-100.0%	-40.0%	0/0/1
= Withhold conclusions when the failed leg has no log	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Bound a restore diagnosis to the searched feed: A reaches the same supported root-cause conclusion while accurately expressing the crucial upstream and feed-version uncertainty. B's inline review suggestion is useful, but its unsupported claim about the mirror's maximum version and weaker handling of upstream verification m...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — build-failure-analysis (gpt-5.6-luna)

Why: Net win +30.0% (4W/5T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +6.0% across 10 paired run(s) — not credible (sign test p=0.188 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 4W/5T/1L; d=5; p=0.188; net +30.0%

Repeated-run reliability (not used by the gate): 10 paired runs (4W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Preserve a silent target-process failure	Eligible	+0.0%	+0.0%	0/1/0
= Recognize a failed pipeline with a clean build	Eligible	+0.0%	+0.0%	0/1/0
▼ Report the absence of all binary logs without a diagnosis	Eligible	-100.0%	-100.0%	0/0/1
= Resolve a disposal warning promoted to an error	Eligible	+0.0%	+0.0%	0/1/0
= Suppress obsolete analysis after the merge revision moves	Eligible	+0.0%	+0.0%	0/1/0
= Withhold conclusions when the failed leg has no log	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Preserve a silent target-process failure: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — msbuild-quality-review (gpt-5.6-luna)

Why: Net win +20.0% (3W/6T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +14.0% across 10 paired run(s) — not credible — 6 of 10 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the agent is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=10; 3W/6T/1L; d=4; p=0.312; net +20.0%

Repeated-run reliability (not used by the gate): 10 paired runs (3W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Accept a valid required import in the packed layout	Eligible	+0.0%	+0.0%	0/1/0
= Avoid findings when the changed file cannot be read	Eligible	+0.0%	+0.0%	0/1/0
= Catch a property evaluated before its discriminator exists	Eligible	+0.0%	+0.0%	0/1/0
= Filter intentionally broken fixture changes	Eligible	+0.0%	+0.0%	0/1/0
= Limit findings to the changed in-scope policy	Eligible	+0.0%	+0.0%	0/1/0
= Preserve explicit FSharp compilation order	Eligible	+0.0%	+0.0%	0/1/0
▼ Preserve the caller extension chain	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Accept a valid required import in the packed layout: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — test-failure-analysis (claude-sonnet-5)

Why: Net win +25.0% (7W/1T/4L over 12 preference-eligible stimulus vote(s), sign test p=0.274), mean preference +15.0% across 12 paired run(s) — not credible (sign test p=0.274 > 0.05) — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=12; 7W/1T/4L; d=11; p=0.274; net +25.0%

Repeated-run reliability (not used by the gate): 12 paired runs (7W/1T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Apply both duration thresholds and baseline requirements	Eligible	-100.0%	-40.0%	0/0/1
▼ Close the preliminary lifecycle without claiming passing tests	Eligible	-100.0%	-100.0%	0/0/1
▼ Distinguish a process crash from an assertion failure	Eligible	-100.0%	-40.0%	0/0/1
▼ Group matching terminal failures across modules	Eligible	-100.0%	-40.0%	0/0/1
= Ignore contributor-authored lifecycle impersonation	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Apply both duration thresholds and baseline requirements: The substantive evidence analysis is equivalent, but A is safer and more schema-aligned: it uses the likely permitted finding classification "failure" and avoids B's unsupported characterization of the tested SHA as a merge commit. B's "regression" classification may violate t...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — test-failure-analysis (gpt-5.6-luna)

Why: Net win +75.0% (9W/3T/0L over 12 preference-eligible stimulus vote(s), sign test p=0.002), mean preference +65.0% across 12 paired run(s) — credibly better — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=12; 9W/3T/0L; d=9; p=0.002; net +75.0%

Repeated-run reliability (not used by the gate): 12 paired runs (9W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid a visible response for an empty complete bundle	Eligible	+0.0%	+0.0%	0/1/0
= Group matching terminal failures across modules	Eligible	+0.0%	+0.0%	0/1/0
= Ignore contributor-authored lifecycle impersonation	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid a visible response for an empty complete bundle: Position-swap inconsistent (forward: skill, reverse: baseline). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔻 Objective regression — unskip-closed-tests (gpt-5.6-luna)

Why: Net win -9.1% (3W/4T/4L over 11 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +1.8% across 11 paired run(s) — no improvement — native evaluator reported an objective task-completion regression

Next action: Inspect objective completion losses and fix them before merge.

State: VALID_REGRESSION (native_completion_regression)

Gate evidence: n=11; 3W/4T/4L; d=7; p=0.500; net -9.1%

Repeated-run reliability (not used by the gate): 11 paired runs (3W/4T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Defer a completed issue unrelated to the ignored test	Eligible	+0.0%	+0.0%	0/1/0
▼ Defer an incomplete partial class with nested tests	Eligible	-100.0%	-40.0%	0/0/1
= Do not treat a zero-result verification exit as authorization	Eligible	+0.0%	+0.0%	0/1/0
▼ Preserve distinct candidates across modules with identical attributes	Eligible	-100.0%	-40.0%	0/0/1
= Reject a legacy manifest without source-bound identities	Eligible	+0.0%	+0.0%	0/1/0
= Reject a manifest for an obsolete source commit	Eligible	+0.0%	+0.0%	0/1/0
▼ Select a class site with complete direct-test enumeration	Eligible	-100.0%	-40.0%	0/0/1
▼ Select a method whose tracking pull request is merged	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Defer a completed issue unrelated to the ignored test: Position-swap inconsistent (forward: tie, reverse: skill). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — msbuild-quality-review (claude-sonnet-5)

Why: Net win +40.0% (5W/4T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +28.0% across 10 paired run(s) — not credible (sign test p=0.109 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=10; 5W/4T/1L; d=6; p=0.109; net +40.0%

Repeated-run reliability (not used by the gate): 10 paired runs (5W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Avoid duplicating default source items	Eligible	-100.0%	-40.0%	0/0/1
= Avoid findings when the changed file cannot be read	Eligible	+0.0%	+0.0%	0/1/0
= Catch a property evaluated before its discriminator exists	Eligible	+0.0%	+0.0%	0/1/0
= Identify a generated-file collision across projects and frameworks	Eligible	+0.0%	+0.0%	0/1/0
= Preserve explicit FSharp compilation order	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Avoid duplicating default source items: Both produce the correct review decision and core remediation. A is more technically exact, more focused on the minimal fix, and provides a more complete actionable finding with location and replacement suggestion.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — technology-selection (gpt-5.6-luna)

Why: Net win +66.7% (4W/2T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +46.7% across 6 paired run(s) — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 6 paired runs (4W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= ML.NET classification on tabular data	Eligible	+0.0%	+0.0%	0/1/0
= Select ML.NET for tabular churn prediction	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• ML.NET classification on tabular data: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — template-authoring (claude-sonnet-5)

Why: Net win +71.4% (6W/0T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +28.6% across 7 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%

Overfit: Low (score 0.13)

Repeated-run reliability (not used by the gate): 7 paired runs (6W/0T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Create template from existing project	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Create template from existing project: Both produce and validate a functional reusable xUnit template that preserves the source project and has the required name. A is better overall because its final response clearly tells the team how to consume it; B adds useful parameterization and documentation but its optiona...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — template-authoring (gpt-5.6-luna)

Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +22.9% across 7 paired run(s) — not credible (sign test p=0.109 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%

Overfit: Moderate (score 0.38)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Add a choice parameter with conditional project content	Eligible	-100.0%	-40.0%	0/0/1
= Preserve central package management conventions	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add a choice parameter with conditional project content: Response A provides a more complete and professionally polished template parameter definition. The critical difference is that Response A includes displayName fields for each choice (showing users friendly names like 'SQL Server' and 'PostgreSQL'), while Response B omits these...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Mixed evidence — template-comparison (gpt-5.6-luna)

Why: Net win +0.0% (3W/1T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.656), mean preference +0.0% across 7 paired run(s) — no improvement

Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%

Overfit: Low (score 0.20)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/1T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Choose MVC or Razor Pages for an admin portal	Eligible	-100.0%	-40.0%	0/0/1
▼ Choose a Blazor or Razor Pages template	Eligible	-100.0%	-40.0%	0/0/1
= Compare REST and gRPC service templates	Eligible	+0.0%	+0.0%	0/1/0
▼ Compare webapi vs webapp side by side	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Choose MVC or Razor Pages for an admin portal: Response A provides a more persuasive and architecturally-grounded recommendation tailored to the user's stated scenario (internal admin portal that may grow with shared controller logic). It explains specifically why MVC's patterns (base controllers, action filters, cross-cut...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — template-discovery (claude-sonnet-5)

Why: Net win +71.4% (6W/0T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +45.7% across 7 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.33)

Repeated-run reliability (not used by the gate): 7 paired runs (6W/0T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Inspect web API template parameters	Eligible	-100.0%	-40.0%	0/0/1
▲ Preview project creation with dry run	Eligible	+100.0%	+100.0%	1/0/0

Illustrative judge evidence:

• Inspect web API template parameters: A is a solid, complete inspection summary and supplies a valid example command. B is otherwise largely accurate, but its final example directly contradicts its own stated allowed auth values, a material flaw in a request that specifically asked for an example command.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — template-discovery (gpt-5.6-luna)

Why: Net win +42.9% (4W/2T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +25.7% across 7 paired run(s) — not credible (sign test p=0.188 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.30)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Find installed test project template choices	Eligible	+0.0%	+0.0%	0/1/0
▼ Inspect web API template parameters	Eligible	-100.0%	-40.0%	0/0/1
▲ Preview project creation with dry run	Eligible	+100.0%	+100.0%	1/0/0
= Search NuGet for specialized template	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Find installed test project template choices: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — template-instantiation (claude-sonnet-5)

Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +17.1% across 7 paired run(s) — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%

Overfit: Moderate (score 0.28)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Compose an application and test solution	Eligible	+0.0%	+0.0%	0/1/0
= Create a controller-based web API	Eligible	+0.0%	+0.0%	0/1/0
= Create a worker service	Eligible	+0.0%	+0.0%	0/1/0
= Create and build a class library	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Compose an application and test solution: Both agents fully completed the requested scaffolding, corrected the SDK's default .slnx output to the required .sln format, added the projects and reference, and successfully built the solution. B is a little more explicit in its final wording, but there is no meaningful diff...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Mixed evidence — template-instantiation (gpt-5.6-luna)

Why: Net win +0.0% (1W/5T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 7 paired run(s) — no improvement

Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%

Overfit: Low (score 0.16)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Adapt a generated test project to central package management	Eligible	-100.0%	-40.0%	0/0/1
= Compose an application and test solution	Eligible	+0.0%	+0.0%	0/1/0
= Create a controller-based web API	Eligible	+0.0%	+0.0%	0/1/0
= Create a worker service	Eligible	+0.0%	+0.0%	0/1/0
= Create and build a class library	Eligible	+0.0%	+0.0%	0/1/0
= Match a neighboring project's target framework	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Adapt a generated test project to central package management: Both responses successfully complete the task with working builds. However, Response A provides superior transparency and verification: it explicitly confirms via regex search that no PackageReferences retain inline versions (TURN 15), and clearly displays the final Directory....

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — template-validation (claude-sonnet-5)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +20.0% across 7 paired run(s) — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%

Overfit: Moderate (score 0.44)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Check every short name in an alias array	Eligible	+0.0%	+0.0%	0/1/0
= Find incomplete symbol definitions	Eligible	+0.0%	+0.0%	0/1/0
▼ Report a malformed template with its parse location	Eligible	-100.0%	-40.0%	0/0/1
= Validate discoverability metadata severity	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Check every short name in an alias array: Position-swap inconsistent (forward: A, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Mixed evidence — template-validation (gpt-5.6-luna)

Why: Net win +0.0% (2W/3T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.687), mean preference +0.0% across 7 paired run(s) — no improvement

Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 2W/3T/2L; d=4; p=0.687; net +0.0%

Overfit: Moderate (score 0.39)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Check every short name in an alias array	Eligible	+0.0%	+0.0%	0/1/0
= Find incomplete symbol definitions	Eligible	+0.0%	+0.0%	0/1/0
▼ Report a malformed template with its parse location	Eligible	-100.0%	-100.0%	0/0/1
= Validate correct template and suggest improvements	Eligible	+0.0%	+0.0%	0/1/0
▼ Validate discoverability metadata severity	Eligible	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

• Check every short name in an alias array: Both responses correctly answer the task: not every alias is safe to publish, specifically identifying search as conflicting with built-in dotnet new search, and confirming acme-service is safe. They perform identically on all four rubric criteria. Response A is more con...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — unskip-closed-tests (claude-sonnet-5)

Why: Net win +27.3% (5W/4T/2L over 11 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +10.9% across 11 paired run(s) — not credible (sign test p=0.227 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 5W/4T/2L; d=7; p=0.227; net +27.3%

Repeated-run reliability (not used by the gate): 11 paired runs (5W/4T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Defer a completed issue unrelated to the ignored test	Eligible	-100.0%	-40.0%	0/0/1
▼ Defer an incomplete partial class with nested tests	Eligible	-100.0%	-40.0%	0/0/1
= Defer open and not-planned tracking items	Eligible	+0.0%	+0.0%	0/1/0
= Reject a legacy manifest without source-bound identities	Eligible	+0.0%	+0.0%	0/1/0
= Reject a manifest for an obsolete source commit	Eligible	+0.0%	+0.0%	0/1/0
= Select a class site with complete direct-test enumeration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Defer a completed issue unrelated to the ignored test: Both outputs are substantively safe and correctly avoid an unskip. A is modestly stronger because its comment action preserves the precise evidence-backed mismatch in a structured finding and gives a concrete, non-speculative next step; B's noop is safe but less informative an...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — technology-selection (claude-sonnet-5)

Why: Net win +83.3% (5W/1T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.031), mean preference +63.3% across 6 paired run(s) — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=6; 5W/1T/0L; d=5; p=0.031; net +83.3%

Warnings: Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.49)

Repeated-run reliability (not used by the gate): 6 paired runs (5W/1T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Agentic workflow with guardrails	Eligible	+100.0%	+100.0%	1/0/0
= Select ML.NET for tabular churn prediction	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Select ML.NET for tabular churn prediction: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — template-comparison (claude-sonnet-5)

Why: Net win +85.7% (6W/1T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +42.9% across 7 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 7 paired runs (6W/1T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Compare webapi vs webapp side by side	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Compare webapi vs webapp side by side: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — template-smart-defaults (claude-sonnet-5)

Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +74.3% across 7 paired run(s) — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.48)

Repeated-run reliability (not used by the gate): 7 paired runs (7W/0T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Controllers exclude the minimal-API flag	Eligible	+100.0%	+100.0%	1/0/0

✅ Improved — template-smart-defaults (gpt-5.6-luna)

Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +74.3% across 7 paired run(s) — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%

Warnings: Activation: isolated 7/7; plugin 5/7

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 7 paired runs (7W/0T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Auth implies HTTPS stays enabled	Eligible	+100.0%	+100.0%	1/0/0
▲ Controllers exclude the minimal-API flag	Eligible	+100.0%	+100.0%	1/0/0

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1279 in dotnet/skills, download eval artifacts with gh run download 37980478823 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/e4fb06acaed78e14efb45c9f1d3236c9378cb80f/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

✅ Evaluation passed for e4fb06a. cc @AbhitejJohn @JanKrivanek — please review.