feat(ci): run tests and type checks on pull requests

[mattrossman]

Motivation

We have unit / e2e tests across apps/* and packages/* but don't run them or type checks in CI, so they quietly regress (e.g. #351 broke a Codex parser test that #366 added the same day).

Changes

• Code Quality gets a typecheck job next to format
• A new Tests workflow runs test, test-platform-lite (which is slow enough for its own parallel job), and test-docker. The Docker tests take ~2.5 min, so they only run when a PR touches relevant files (skipped example).
• Refresh results commits drop [skip ci] (added in ci: skip CI on the generated eval results commit #332) and push with our GH App token instead, so required checks run on them without approval. A run-evals-changed refresh's results commit ran CI as the app with no approval prompt and didn't start another refresh.
• Upgraded to Vitest v5 so we get per-package labels on the job summary (compare before and after).

Tests	Code Quality

Example with tests failing (run):

I added OPENAI_API_KEY to Dependabot secrets so Tests runs on Dependabot PRs. Fork PRs don't get secrets as is the case on supabase/mcp.

Follow-ups

• After merge, require format, typecheck, test, test-platform-lite, and test-docker on the main: review required ruleset.
• AI-934: replace the framework smoke tests with per-scorer Vitest files.
• AI-1295: cut platform-lite setup time with PGlite snapshots.

Closes AI-1121

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Oct 8, 2026 1:06pm UTC

[barryroodt]

Looks good to merge from my side! Nice touch giving the bot push its own concurrency group so it can't cancel the run that pushed it. One small thing:

pnpm check now fails with a missing script. The rename to test left a few docs pointing at the old command, and the README still says it "Runs typechecks plus local smoke tests" when pnpm test doesn't typecheck:

• README.md#L154
• README.md#L171-L174
• CONTRIBUTING.md#L99
• flakiness.md#L160

[mattrossman]

@barryroodt just fixed up those docs to use the new pnpm test and pnpm typecheck scripts 👍