Add an evaluation suite for the Workshop agent

.github/workflows/workshop-evals.yml

@@ -0,0 +1,154 @@
+name: Workshop evals
+
+# Live agent evals: every run boots a workerd Workshop and drives a real model, so this is kept out
+# of regular CI. It runs on demand, and on a pull request only when someone adds the `run-evals`
+# label. Add a schedule after results have durable ingestion; expiring workflow artifacts do not
+# support trend analysis or regression alerts across runs.
+on:
+  workflow_dispatch:
+    inputs:
+      trials:
+        description: Repetitions per task and model
+        default: "10"
+      models:
+        description: Comma-separated agent models, or blank for the defaults
+        default: ""
+  pull_request:
+    types: [labeled, synchronize]
+
+permissions:
+  contents: read
+
+env:
+  NODE_VERSION: "24.19.0"
+  # Manual runs want statistics; a labelled pull request wants a quick read.
+  WORKSHOP_EVAL_TRIALS: ${{ inputs.trials || (github.event_name == 'pull_request' && '3' || '10') }}
+  WORKSHOP_EVAL_MODELS: ${{ inputs.models }}
+
+# One eval run at a time. Each is long and spends real inference; overlapping runs would multiply
+# that spend and queue 8 more runners behind the ones already going. Log attribution is safe either
+# way, since each trial filters on its own fresh workspace.
+concurrency:
+  group: workshop-evals-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  evals:
+    name: ${{ matrix.expectation }} (shard ${{ matrix.shard }}/6)
+    # A fork's pull request cannot reach the secrets, and would spend this repository's inference
+    # budget if it could. Same guard as bonk-pr.yml.
+    if: >-
+      github.event_name != 'pull_request' ||
+      (github.event.pull_request.head.repo.id == github.event.pull_request.base.repo.id &&
+       contains(github.event.pull_request.labels.*.name, 'run-evals'))
+    runs-on: ubuntu-latest
+    timeout-minutes: 240
+    strategy:
+      fail-fast: false
+      # Workers AI rate-limits per account, and every shard runs an agent that makes many model
+      # calls per turn. Running all twelve jobs at once earns a wall of 429s, which the scorer
+      # correctly leaves unscored -- so the run completes having measured almost nothing. Cap the
+      # concurrency instead; raise it only against a measured limit for the account in use.
+      max-parallel: 4
+      matrix:
+        expectation: [required, frontier]
+        # A trial builds a whole application, so it costs minutes, not seconds: the tasks in this
+        # suite run 3-7 minutes each. Six shards keeps a ten-trial run inside the job timeout.
+        shard: [1, 2, 3, 4, 5, 6]
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      # Ahead of setup-node, which shells out to `pnpm store path` to find the directory it caches.
+      - name: Enable Corepack
+        run: corepack enable
+
+      - name: Set up Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: pnpm
+
+      - name: Install dependencies
+        run: pnpm install --frozen-lockfile
+
+      # The harness boots the real workshop-backend Worker, whose `src/generated/` codegen is
+      # gitignored. Without this the Worker fails to bundle and every trial dies at startup.
+      - name: Build
+        run: pnpm build
+
+      - name: Run evals
+        working-directory: packages/workshop-evals
+        env:
+          CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CF_AI_GATEWAY_ACCOUNT_ID }}
+          CLOUDFLARE_API_TOKEN: ${{ secrets.CF_AI_GATEWAY_TOKEN }}
+          WORKSHOP_EVAL_GATEWAY_ID: ${{ secrets.CF_AI_GATEWAY_NAME }}
+          WORKSHOP_EVAL_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
+          WORKSHOP_EVAL_SHARD: ${{ matrix.shard }}/6
+        run: pnpm run eval:${{ matrix.expectation }}
+
+      - name: Upload results
+        # Frontier tasks are expected to fail and required tasks are worth diagnosing when they do,
+        # so the report is always kept.
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: eval-results-${{ matrix.expectation }}-${{ matrix.shard }}
+          path: packages/workshop-evals/.wrangler/evals/
+          retention-days: 30
+
+  summary:
+    name: Summary
+    needs: evals
+    if: always() && needs.evals.result != 'skipped'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Enable Corepack
+        run: corepack enable
+
+      - name: Set up Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: pnpm
+
+      - name: Install dependencies
+        run: pnpm install --frozen-lockfile
+
+      - name: Download shard results
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+        with:
+          path: shards
+          pattern: eval-results-*
+
+      # Each shard reduced separately, because the reducer refuses a duplicated run ID and merging
+      # raw reports would hide a genuine collision rather than report it.
+      - name: Summarize
+        working-directory: packages/workshop-evals
+        run: |
+          set -euo pipefail
+          shopt -s nullglob
+          reports=("$GITHUB_WORKSPACE"/shards/*/*.json)
+          if [[ ${#reports[@]} -eq 0 ]]; then
+            echo "No eval reports were produced." | tee -a "$GITHUB_STEP_SUMMARY"
+            exit 1
+          fi
+          for report in "${reports[@]}"; do
+            name="$(basename "$(dirname "$report")")"
+            pnpm run eval:summary "$report" "$name"
+            cat ".wrangler/evals/$name.md" >> "$GITHUB_STEP_SUMMARY"
+          done
+
+      - name: Upload summaries
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: eval-summaries
+          path: packages/workshop-evals/.wrangler/evals/*.json
+          retention-days: 30

AGENTS.md

@@ -61,6 +61,7 @@ Release pipeline (`scripts/release/`) — how customer instances get deployed:
 To test changes:
 - Run `pnpm build` to type-check, or `vp run -F <package> build` for one package — most packages declare `build` as a task rather than a script, and `pnpm --filter` cannot see a task. It is a type check and codegen pass, not a compile: every package but `typed-storage` is `noEmit`, because nothing imports the others' `dist` — wrangler and vite bundle from source. `typed-storage` emits because its `exports` resolves to `dist/index.js`. A re-run with nothing changed replays from the task cache — see below.
 - Run `pnpm test` to run unit tests, though as of this writing most packages don't have tests yet. It runs the root's own `node --test 'scripts/**/*.test.ts'` suite and then the per-package tests.
+- Four suites cover more than one module at a time; [`docs/integration-testing.md`](docs/integration-testing.md) compares them and says which to use. `workshop-backend/__integration__` runs in-process under `@cloudflare/vitest-pool-workers`, so it is where `cloudflare:test` (`abortAllDurableObjects()`, `runInDurableObject()`) reaches Durable Object internals. `packages/integration-tests` boots real Workers out-of-process, reaches only the public Cap'n Web API, and executes Gadget code through a Worker Loader; it owns the harness, RPC client, and `AgentSession`. A consumer repo's per-vendor suite reuses that toolkit against one real gatekeeper. `packages/workshop-evals` adds a live model, so it is the only suite that needs inference and the only one outside `pnpm test`.
 - The cached per-package test run is a Vite+ `test` task in each package's `vite.config.ts` rather than a `test` script, so its `input` can exclude the scratch paths vitest writes and reads back (`scripts/vitest-task-vite-config.ts`, shared by all of them). Gatekeepers with a configurator UI re-export `withTests` from `gatekeeper-configurator-vite-config.ts` to get both tasks at once; the ones with no test files re-export its default instead, because `vitest run` exits 1 when it finds none.
 - Two ways to run one package's tests: `pnpm --filter <package> test:run` goes straight to vitest, `vp run -F <package> test` goes through the cache. The cached path replays instantly when the package is untouched, but its fingerprinting and archiving lose to plain vitest on a package you just edited — by more than the whole suite costs on a small one. Use `test:run` while iterating and `pnpm test` to verify. The direct script is `test:run` rather than `test` because a task may not share a name with a script.
 - `pnpm build` and `pnpm clean` are `vp run -r <task>`, not `pnpm run --recursive <task>`. Vite+ runs the same per-package scripts and tasks, in dependency order, but caches each one against its inputs, so an unchanged package replays its previous output instead of re-running. Commands joined with `&&` — or given as an array in a task — are cached as separate entries, so a package whose codegen is fresh can still re-run its `tsc`. `vp run --last-details` explains every hit and miss, which is the thing to read when a build is slower than expected. Don't reintroduce a root script that calls `pnpm run --recursive`: `vp run -r` selects the root package too, and would run it as a task and rebuild the whole workspace a second time.
@@ -138,11 +139,7 @@ IMPORTANT: Frontend error reporting is a separate, opt-in path:
   send bounded reports with `postMessage`; the host accepts them only from the known frame window
   with origin `null`, adds host-owned surface/vendor context, and performs the same-origin POST.
   Do not add direct cross-origin reporting from a gatekeeper Worker domain.
-- Frontend reports never convey authority. `reportedUserId` is supplied by the client and unverified
-  — the name records that it is a report, not a finding — so it is a diagnostic label only and must
-  never be read to make a decision. `pageLocation` is origin and pathname only, rebuilt by
-  `normalizePageLocation` rather than trusted from producers, because a share link's fragment is a
-  bearer capability and an `href` also retains credentials; non-`http(s)` URLs are dropped entirely.
-- Install automatic capture only in trusted first-party surfaces, never gadget/user-authored code.
+- Frontend reports and frame metadata are diagnostic only and never convey identity or authority.
+  Install automatic capture only in trusted first-party surfaces, never gadget/user-authored code.
   Exception messages and stacks reach the external Reporter, so never intentionally put secrets,
   prompts, tokens, headers, or request/response bodies in thrown errors or report metadata.

docs/ai-gateway-billing.md

@@ -13,9 +13,10 @@ turn, the overseer calls `checkUsageAndBalance`:
 - **Connected, balance ≥ `$2`** → allowed, routed through the user's own account so usage bills
   their Cloudflare credits — even while free-tier allowance remains. The platform is never charged
   for funded users, and their daily free-tier counter is left untouched.
-- **Otherwise, within the free tier** → allowed, served via the platform's configured AI Gateway
-  (all providers, Workers AI included). This includes connected users whose balance is below `$2`
-  (incl. $0).
+- **Otherwise, within the free tier** → allowed, served via the platform's configured AI Gateway.
+  Workers AI uses the same Gateway ID unless `CF_AI_GATEWAY_WAI_DIRECT=true` sends it straight to
+  the Workers AI REST endpoint or `CF_AI_GATEWAY_WAI` selects another Gateway. This includes
+  connected users whose balance is below `$2` (incl. $0).
 - **Free tier exhausted, no Cloudflare account connected** → blocked, with a prompt to connect.
 - **Free tier exhausted, connected but balance below `$2`** → blocked, with a prompt to add credits.
 
@@ -53,24 +54,19 @@ CLOUDFLARE_OAUTH_CLIENT_SECRET=...
 CF_AI_GATEWAY=your-gateway
 CF_AI_GATEWAY_PROVIDERS=anthropic,openai,google
 
-# Required whenever CF_AI_GATEWAY is set:
+# Required whenever CF_AI_GATEWAY is set (all inference goes over HTTPS with tokens):
 CF_AI_GATEWAY_ACCOUNT_ID=...
-# Required unless the WORKERS_AI binding carries gateway traffic; always required for the
-# google provider:
 CF_AI_GATEWAY_API_TOKEN=...
+
+# To send Workers AI straight to its REST endpoint (no gateway, no cost logs):
+CF_AI_GATEWAY_WAI_DIRECT=true
 ```
 
-Gateway mode always requires `CF_AI_GATEWAY_ACCOUNT_ID` plus a transport: the `WORKERS_AI`
-binding when present (binding requests are pre-authenticated, and cost-log reads work through
-the binding too), or otherwise an API token with AI Gateway Run and Read permissions — Read
-access lets Gadgets retrieve each log's cost for user-visible accounting. The binding transport
-only works when the Gateway lives in the Worker's own account, which the Worker can't verify at
-runtime — a deployment whose Gateway is in a different account must set
-`CF_AI_GATEWAY_USE_BINDING=false` to opt out and use the token transport. That is a flag rather
-than an unbinding because `WORKERS_AI` also backs the webFetch tool's document-to-Markdown
-conversion (and is hardcoded for every released backend), so removing it would break that instead
-of just moving gateway traffic. The token stays required for the `google` provider even when the
-binding transport applies. Every provider, Workers AI included, routes through the same Gateway.
+Gateway mode always requires `CF_AI_GATEWAY_ACCOUNT_ID` and an API token with AI Gateway Run and
+Read permissions; Read access lets Gadgets retrieve each log's cost for user-visible accounting.
+Workers AI uses `CF_AI_GATEWAY` as its Gateway ID by default; set `CF_AI_GATEWAY_WAI` to select
+another Gateway, or `CF_AI_GATEWAY_WAI_DIRECT=true` to call the Workers AI REST endpoint directly
+(same credentials, no gateway cost logs).
 
 The Cloudflare dashboard OAuth endpoints and scopes are **hardcoded** in the Cloudflare gatekeeper
 (`packages/gatekeeper-cloudflare/src/oauth.ts`):

docs/integration-testing.md

@@ -2,24 +2,50 @@
 
 This is how the integration test suites work, and why they are shaped the way they are.
 
-There are two kinds of suite:
-
-| | this repo's `packages/integration-tests` | a consumer repo's per-vendor suite |
-|---|---|---|
-| Runs | `pnpm test` (part of CI's normal test job) | its own CI step |
-| Gatekeeper | a fixture Worker whose verification outcome the tests set | a real vendor gatekeeper, unmodified |
-| Covers | the overseer's observer logic | a genuinely expired credential, end to end |
-| Owns | the harness, interceptor, and RPC client | that vendor's handlers and token minting |
+## Which suite to add to
+
+Four suites exercise more than one module at a time. Each reaches a different part of the system, so
+the choice decides what a test can assert.
+
+| Suite | Runtime | Reaches | Use it for |
+|---|---|---|---|
+| `workshop-backend/__integration__` | **in-process** workerd (`@cloudflare/vitest-pool-workers`) | Worker and Durable Object internals, via `cloudflare:test` | the backend's own resilience: DO resets, native-RPC error shapes, session recovery |
+| `packages/integration-tests` | **out-of-process** workerd (`wrangler`'s `createTestHarness()`) | only the public Cap'n Web API, as a browser would | the overseer's observer logic, gatekeeper flows, Gadget behaviour end to end |
+| a consumer repo's per-vendor suite | out-of-process, same toolkit | one real vendor gatekeeper, unmodified | a genuinely expired credential, end to end |
+| `packages/workshop-evals` | out-of-process **plus a live model** | the production agent, then the Gadget it built | whether the agent delivers a working application |
+
+The dividing line is the first column, and it decides what is available to you:
+
+- **In-process only.** `cloudflare:test` — `abortAllDurableObjects()`, `runInDurableObject(stub, fn)`,
+  and therefore `state.abort(reason)` — plus direct access to `exports.SomeDurableObject`. This is the
+  surgical way to reset a specific Durable Object, and the only way to assert on a *native* RPC
+  boundary. Fake timers also work here.
+- **Out-of-process only.** Real WebSocket transport, several Workers bound to each other, service
+  bindings, and a Worker Loader actually executing Gadget code. The code under test is in another
+  process, so none of the `cloudflare:test` helpers exist and nothing can be reached except through
+  the API a client has.
+
+Two consequences follow:
+
+- **Each runtime restarts a server differently.** In-process, abort the object directly.
+  Out-of-process, call `AgentSession.restartGadgets()`, which applies an empty code update — the same
+  thing the platform does on every code change. It confirms the restart by checking that outstanding
+  stubs became invalid.
+- **A live model belongs only in `workshop-evals`.** `pnpm test` stays deterministic and free, so the
+  eval suite runs on its own schedule. A test that needs no model belongs in one of the first two
+  suites, where it runs on every commit.
 
 A consumer repo is one that vendors this repo as a `public/` submodule and consumes the toolkit as a
 workspace dependency (`public/packages/integration-tests` in its `pnpm-workspace.yaml`).
 
-No such suite lives in this repo, and nothing here depends on one existing. The second column is
-described anyway because it is what the toolkit is parameterised *for*: the harness takes a list of
-gatekeepers and the interceptor takes pluggable handler modules precisely so a suite can be added
-outside this repo without forking either. Where this doc describes a per-vendor suite, take it as the
-worked example of that shape — one gatekeeper run unmodified against its vendor's mocked endpoints —
-rather than as something you will find here.
+No such suite lives in this repo, and nothing here depends on one existing. It is described anyway
+because it is what the toolkit is parameterised *for*: the harness takes a list of gatekeepers and the
+interceptor takes pluggable handler modules precisely so a suite can be added outside this repo
+without forking either. Where this doc describes a per-vendor suite, take it as the worked example of
+that shape — one gatekeeper run unmodified against its vendor's mocked endpoints — rather than as
+something you will find here.
+
+The rest of this document is about `packages/integration-tests` and the toolkit it owns.
 
 ## What these tests are