Add frontend-to-backend deployment readiness contract across plan/integrate/deploy agents

.github/instructions/copilot-on-rails-docs.instructions.md

@@ -0,0 +1,73 @@
+---
+description: 'Keep the Create New Project with Copilot (Copilot on Rails) user guide and support runbook in sync whenever the feature changes.'
+applyTo: "src/webviews/copilotOnRails/**, src/commands/copilotOnRails/**, src/chat/tools/copilotOnRails/**, src/utils/copilotOnRails/**, src/tree/project/**, resources/agents/**"
+---
+
+# Keep the Copilot on Rails docs in sync
+
+You are editing the **Create New Project with Copilot** feature (codename *Copilot on Rails*, command prefix
+`copilotOnRails.`). Its end-user guide and support/triage runbook lives at
+[docs/copilot-create-project.md](../../docs/copilot-create-project.md).
+
+**Rule:** any change to this feature's user-visible behavior, surfaces, or support flow must be reflected in
+that document **in the same change**. Treat the doc as part of the feature — a change that alters behavior
+without updating it is incomplete.
+
+## When a change requires a doc update
+
+Update the matching section of `docs/copilot-create-project.md` when you:
+
+| Change | Section(s) to update |
+| --- | --- |
+| Add / rename / remove a `copilotOnRails.*` command (TS handler **or** `package.json` / `package.nls.json`) | UI surfaces reference (Part 3), Commands appendix, and the relevant stage |
+| Add / rename / remove an MCP tool (`src/chat/tools/copilotOnRails/**`) | The MCP tools table and the pipeline diagram |
+| Add / change / remove a webview or its behavior (`src/webviews/copilotOnRails/**`) | UI surfaces table, the stage that uses it, and its screenshot |
+| Add / change / remove an agent or a hand-off (`resources/agents/**`) | The agents table, the Mermaid pipeline diagram, and the affected stage |
+| Change a `.azure/*` artifact, the `.github/agents` download behavior, or a `workspaceState` key | Files & state |
+| Change what diagnostics capture, or the Report Issue / Inspect Diagnostics behavior | Support & triage runbook, including the "What the diagnostics contain (privacy)" section |
+| Change the launch / resume / empty-folder / autopilot flow | Launching, Resuming a session, and Autopilot mode |
+
+## New or changed UI — flag screenshots to re-capture
+
+Screenshots are captured by hand and stored separately, so the agent can't re-shoot them. When your change
+touches the UI, **tell the developer which images to refresh** and why:
+
+- **Altered an existing screen** (relabeled or moved control, restyled view, new or removed field, changed
+  copy, different states): its screenshot is now **stale even though the placeholder already exists**. Name
+  the affected file(s) and say in one line what changed.
+- **Added a brand-new screen or state**: add the matching `📷` placeholder (the blockquote plus its centered
+  `<p align="center"><img …></p>` reference — images in this doc are centered, not raw `![]()`) and a
+  screenshot references section, then tell the dev it needs a first capture.
+- **Removed a screen**: delete its placeholder and checklist entry, and note the removal.
+- Never delete an existing placeholder just because its PNG is still missing — the images are captured
+  separately from the prose.
+- When unsure whether a visual change is significant, flag it anyway.
+
+Use this map from source area to the screenshot(s) it backs:
+
+| You changed… | Screenshot(s) to re-capture |
+| --- | --- |
+| `src/tree/project/**`, the `azureProject` view / welcome content | `01-launch-azure-project-view.png`, `12-azure-project-progress-tree.png` |
+| The launch / empty-folder / resume flow (`createProjectWithCopilot.ts`, `resume*`) | `02-empty-folder-prompt.png`, `11-resume-prompt.png` |
+| `CreateProjectView` (prompt + model picker) | `03-create-project-prompt.png` |
+| `RequirementsView` | `04-requirements-view.png` |
+| `ScaffoldPlanView` / plan preview (incl. UI preview cards) | `05-plan-preview.png` |
+| `FrontendPreviewView` (Approve UI) | `06-frontend-preview-approve-ui.png` |
+| `ScaffoldNextStepsView` | `07-scaffold-next-steps.png` |
+| `LocalPlanView` (debug plan) | `08-debug-plan-view.png` |
+| `LocalDevNextStepsView` | `09-debug-next-steps.png` |
+| `DeploymentPlanView` | `10-deployment-plan-view.png` |
+| `DeployResultView` | `15-deployment-results-view.png` |
+| `reportIssue` (issue template) | `13-report-issue-github.png` |
+| `inspectDiagnostics` (JSON payload) | `14-inspect-diagnostics-json.png` |
+
+## Before you finish
+
+- **Report screenshots to the developer:** in your summary, list every image your change makes stale (by
+  filename, with a one-line reason) plus any placeholders you added or removed, so they can capture or
+  refresh them. If your change touched no UI, say so.
+- Re-read the affected sections and confirm every command id, MCP tool name, agent name, file path, and
+  view→command mapping still matches the code you changed.
+- Keep the reference tables and the Mermaid pipeline diagram accurate.
+- If nothing user-visible changed (a pure internal refactor), no doc update is needed — note that briefly
+  instead of editing the doc.

.github/workflows/agent-contracts.yml

@@ -0,0 +1,110 @@
+name: Agent Contracts
+
+# The half of the Vally eval suite that needs no Copilot credentials.
+#
+# Running the agent itself now happens on MSBench (see msbench-evals.yml), but
+# the checks below still earn their place in PR CI: they are fast, they need no
+# token, and they guard the graders and agent assets that the MSBench run
+# depends on. A broken grader or a drifted instruction file would otherwise only
+# surface as a confusing eval failure much later.
+#
+# Deliberately absent: anything that drives a live agent. That now happens on
+# MSBench (see msbench-evals.yml), and the files that did it headlessly —
+# run-eval.cjs, check-copilot-auth.ts, check-gate-tools.ts and the SDK executor —
+# have been deleted rather than left unreferenced in the tree.
+on:
+  pull_request:
+    paths:
+      - 'evals/**'
+      - 'resources/agents/**'
+      - '.vally.yaml'
+      - '.github/workflows/agent-contracts.yml'
+  workflow_dispatch:
+
+env:
+  NODE_VERSION: '22'
+
+jobs:
+  contracts:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+
+      - name: Install eval dependencies
+        run: npm ci
+        working-directory: evals
+
+      - name: Add Vally CLI to PATH
+        run: echo "$GITHUB_WORKSPACE/evals/node_modules/.bin" >> "$GITHUB_PATH"
+
+      # A rule removed from the shipped agent should not linger in the eval's
+      # copy of it, or the eval grades a prompt we no longer ship.
+      - name: Check agent instruction drift
+        run: npm run drift
+        working-directory: evals
+
+      # Graders run straight off TypeScript source, so a type error is a broken
+      # grader — catch it before it costs a full eval run.
+      - name: Type-check evals
+        run: npm run typecheck
+        working-directory: evals
+
+      # The grader import scanner decides what gets staged into the container, and a
+      # naive one already failed the build on a prose sentence. These cases are the
+      # ones it must get right — including that a genuine bare import STILL throws,
+      # since the guard's entire value is its ability to fail.
+      - name: Self-test the import scanner
+        run: npm run imports:self-test
+        working-directory: evals
+
+      # Prove the graders still detect the regressions they claim to detect.
+      - name: Certify graders
+        run: npm run certify
+        working-directory: evals
+
+      # Same idea one layer up: prove the stack schema still rejects the sixteen
+      # broken stack files it claims to reject. A schema that quietly accepts
+      # everything would let a stack requiring a binary the container does not
+      # have reach a paid run, which is the cost this check exists to avoid.
+      - name: Check stack schema
+        run: npm run stacks:check
+        working-directory: evals
+
+      # A SQL assertion has no grader filename, so its `comment` is the only
+      # stable identity it carries into stored run results. Rewording one forks
+      # that gate into a second identity with no history — silently, since the
+      # assertion still runs and still passes. The liveness sentinel reached ten
+      # different wordings across eleven stimuli before anyone noticed, and no
+      # identity scheme can repair that retroactively. Comments are the one place
+      # paraphrasing is normally harmless, so nothing but a check will stop it.
+      - name: Check shared assertion comments are canonical
+        run: npm run gates
+        working-directory: evals
+
+      # Run through the evals package so the spec is actually linted: a bare
+      # `vally lint` at the repo root discovers no skills and silently passes.
+      - name: Lint eval specs
+        run: npm run lint
+        working-directory: evals
+
+      # `run.sh` promises a clean machine can execute it, and the MSBench eval job
+      # depends on that: it installs nothing before running `run.sh --skip-build`.
+      # A script it invokes that statically imports a package therefore fails on
+      # the one path where failure costs money. That rule was written down, in
+      # prose, and then broken by the next change to the same file family — so it
+      # is a mechanism now rather than a hope.
+      #
+      # Deliberately last: this step moves `evals/node_modules` aside and restores
+      # it in a `finally`. Running it after everything else means that even a
+      # catastrophic failure to restore cannot make an unrelated step fail with a
+      # confusing error.
+      - name: Check run.sh works on a clean machine
+        run: npm run clean-machine:check
+        working-directory: evals

.github/workflows/msbench-evals.yml

@@ -0,0 +1,150 @@
+name: MSBench Evals
+
+# Runs the project-plan eval on MSBench against a real VSIX build of this
+# extension, rather than against agent instructions in isolation.
+#
+# Split into two jobs on purpose:
+#
+#   build - packages the VSIX and asserts what ended up inside it, using the same
+#           guards run.sh applies locally. The shared build template
+#           (microsoft/vscode-azuretools jobs.yml) already compiles and packages,
+#           but it never inspects the archive, so a .vscodeignore rule that drops
+#           resources/agents/ passes every other check in the repo and then shows
+#           up as an agent that mysteriously ignores its instructions. Needs no
+#           credentials, so it catches that on a PR instead of in an MSBench run.
+#   eval  - submits to MSBench. Needs Azure auth, so it is manual-only; see
+#           evals/msbench/README.md ("Running in CI") for the one-time setup.
+#
+# Unlike the credential-free gates in agent-contracts.yml, this cannot ride on
+# GITHUB_TOKEN: MSBench runs on CES, which authenticates callers by Entra client id.
+on:
+  pull_request:
+    paths:
+      - 'evals/msbench/**'
+      - '.github/workflows/msbench-evals.yml'
+      # The build job's lasting value is asserting what ends up *inside* the
+      # VSIX, which the shared build template does not check. Both inputs to
+      # that live outside evals/, so they have to trigger it themselves.
+      - '.vscodeignore'
+      - 'resources/agents/**'
+  workflow_dispatch:
+    inputs:
+      benchmark:
+        description: 'Benchmark instance to borrow for its container image'
+        required: false
+        default: 'vscbench.say_hello'
+      stimulus:
+        description: 'Stimulus to submit. The default is the cheapest one whose answer we already know.'
+        required: false
+        default: 'scaffold-unapproved-plan'
+
+env:
+  NODE_VERSION: '22'
+  PYTHON_VERSION: '3.12'
+
+jobs:
+  # Everything that can be verified without credentials.
+  build:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+
+      # `stage-graders.ts` copies an allowlist of dependency-free packages out of
+      # evals/node_modules into the staged tree, so a grader's bare import (today
+      # `jsonc-parser`, for the JSON-with-comments in launch.json) resolves inside
+      # the container, which has no install step. It hard-errors rather than
+      # staging a partial tree, so without this the build job fails before the
+      # VSIX assertions it exists to run.
+      - name: Install eval dependencies
+        run: npm ci
+        working-directory: evals
+
+      - name: Build and verify the VSIX
+        run: ./evals/msbench/run.sh --build-only
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: msbench-vsix
+          path: evals/msbench/assets/extensions/*.vsix
+          retention-days: 7
+
+  eval:
+    # Manual only: the Azure identity has to be allowlisted by the MSBench team
+    # before this can pass, so running it on PRs would only ever be red.
+    if: github.event_name == 'workflow_dispatch'
+    needs: build
+    runs-on: ubuntu-latest
+    # A cold run is ~15 min; the timeout is generous so a slow queue does not
+    # look like a product failure.
+    timeout-minutes: 60
+    permissions:
+      contents: read
+      id-token: write # Fetch an OIDC token for azure/login.
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - uses: actions/download-artifact@v4
+        with:
+          name: msbench-vsix
+          path: evals/msbench/assets/extensions
+
+      # Needed here too, not just in `build`: `--skip-build` skips the VSIX, but
+      # graders are staged on every invocation because they are read straight off
+      # the working tree, and staging them needs evals/node_modules for the
+      # allowlisted packages.
+      - name: Install eval dependencies
+        run: npm ci
+        working-directory: evals
+
+      # run.sh mints the MSBench feed token with `az account get-access-token`,
+      # so it only needs an already-authenticated az. A self-hosted runner that
+      # is already signed in can skip this step entirely.
+      - name: Azure login
+        uses: azure/login@v2
+        with:
+          client-id: ${{ secrets.MSBENCH_AZURE_CLIENT_ID }}
+          tenant-id: ${{ secrets.MSBENCH_AZURE_TENANT_ID }}
+          subscription-id: ${{ secrets.MSBENCH_AZURE_SUBSCRIPTION_ID }}
+
+      # STIMULUS is read from the environment by run.sh, the same way BENCHMARK
+      # is, so `--stimulus` on the command line still wins for a local run.
+      #
+      # It is set explicitly rather than left to run.sh's default, which is
+      # `photo-app-requirements` — a full planning run whose result would then
+      # have to be interpreted. The first CI runs are testing the *pipeline*, so
+      # they use a stimulus whose answer is already known locally (6/6 green):
+      # a red then means CI is broken, which is the only question being asked.
+      # A first run against an unknown-answer stimulus cannot separate "CI is
+      # misconfigured" from "the product changed".
+      - name: Run the MSBench eval
+        run: ./evals/msbench/run.sh --skip-build --output "$RUNNER_TEMP/report.json" --data_dir "$RUNNER_TEMP/msbench-data"
+        env:
+          BENCHMARK: ${{ inputs.benchmark }}
+          STIMULUS: ${{ inputs.stimulus }}
+
+      # Keep the report and per-instance data so a failure can be diagnosed from
+      # the transcript and patch rather than by re-running.
+      - name: Upload run artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: msbench-results
+          path: |
+            ${{ runner.temp }}/report.json
+            ${{ runner.temp }}/msbench-data/**
+          if-no-files-found: ignore

.gitignore

@@ -67,3 +67,42 @@ testWorkspace
 test-results.xml
 dist
 stats.json
+results/
+
+# Generated eval skills (built from resources/agents/*.agent.md at run time)
+evals/.generated/
+
+# Grader sources staged into the MSBench agent assets by evals/msbench/run.sh.
+# Generated from the working tree on every run; a checked-in copy would drift.
+evals/msbench/assets/graders/
+
+# Built from evals/msbench/config/{base,stimuli/*}.yaml by run.sh. run-agent.sh
+# only ever reads this one filename, so selecting a stimulus means writing it.
+evals/msbench/assets/user-overrides.yaml
+
+# The resolved stack, projected as JSON for the container to read (the graders
+# run from staged source with no node_modules, so they cannot parse the YAML).
+# Written by build-config.ts on every --stack build, alongside the graders rather
+# than inside assets/graders/, which stage-graders.ts wipes.
+evals/msbench/assets/stack.json
+
+# Starting workspace materialised by evals/msbench/stage-workspace.ts from the
+# stimulus's `# seed:` directive. Derived from a checked-in fixture on every run,
+# and cleared between stimuli — a checked-in copy would be a third source of
+# truth for a document that already has two.
+evals/msbench/assets/workspace/
+
+# The known-debuggable project the breakpoint stimulus runs against, copied from
+# evals/grader-certification/reference-node-fullstack by run.sh on every run.
+# Same reason as the graders above: a second checked-in copy would drift from the
+# fixture that evals/debug-probe certifies against, and then a green run here and
+# a green certification would stop meaning the same thing.
+evals/msbench/assets/fixtures/
+
+# Mutual-exclusion lock taken by run.sh while it owns assets/.
+evals/msbench/assets/.run.lock/
+
+# Extraction cache for evals/msbench/regrade.ts, keyed by run id. Downloaded
+# from stored MSBench results and reused across iterations so re-grading stays a
+# local operation; safe to delete at any time.
+evals/msbench/.regrade/

.vally.yaml

@@ -0,0 +1,20 @@
+# Vally project configuration
+#
+# `paths.results` is an *output* directory, created on demand and gitignored, so
+# it is correctly absent from a clean checkout. Every other path here is an input
+# and must exist — a path that points at nothing lints clean and silently
+# contributes no evals, which is worse than a missing-file error.
+#
+# No `environments.mcpServers` entry. The workflow-tools stand-in existed for the
+# headless SDK runner, which has been deleted — the agent now runs on MSBench
+# against the extension's real in-process MCP server. The `workflow-tools-*` tool
+# names in the specs below are the names Vally matches against a trajectory, and
+# do not require a server to be declared here.
+paths:
+  evals: [evals]
+  results: results
+
+suites:
+  project-plan:
+    description: azure-project-plan agent contracts
+    evals: ["evals/project-plan/**"]