Skip to content

feat(evals): branching entitlement regression evals - #375

Draft
claude[bot] wants to merge 22 commits into
mainfrom
feat/branching-entitlement-regression-evals
Draft

feat(evals): branching entitlement regression evals#375
claude[bot] wants to merge 22 commits into
mainfrom
feat/branching-entitlement-regression-evals

Conversation

@claude

@claude claude Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Requested by Pedro Rodrigues · Slack thread

Before: About 21% of MCP create_branch calls fail, mostly free-plan orgs getting PaymentRequiredException ("Branching is supported only on the Pro plan or above"), often after get_cost already quoted them a branch price. Nothing in this repo covered branching. platform-lite had no branch routes and a hardcoded free org.

After: Three regression evals (tools mode, interface: mcp) for the agreed fix (Slack, AI-1294):

  • deploy-database-002-branch-free-plan: free org. The agent checks get_organization, never calls get_cost/confirm_cost/create_branch, has zero tool errors, and tells the user a paid plan is needed.
  • deploy-database-003-branch-paid-plan-twice: Pro org. First branch goes get_cost → confirm_cost → create_branch, both branches exist, get_organization is called at most once, zero tool errors.
  • deploy-database-004-branch-free-plan-project-scoped: free org, project-scoped MCP. At most one create_branch, no confirm_cost, zero tool errors, and the agent explains the plan.

A "tool error" means isError: true or a thrown error. The server's planned non-error "requires Pro" response passes.

These evals are expected to fail until the MCP server change ships and MCP_SERVER_VERSION is bumped.

How:

  • platform-lite: the org name and plan can be seeded. GET /v1/organizations no longer returns the plan (it didn't upstream either). There's a minimal GET /v1/organizations/{slug}/entitlements that only covers branching, and GET/POST /v1/projects/{ref}/branches, which returns the platform's 402 for free orgs. A branch is a fresh database with the parent's migrations replayed. Projects can also be seeded with migration history.
  • Harness: adds remote/organization.json and remote/migrations/ seeds, plus two frontmatter keys for tools evals: mcpFeatures: [branching] (opt-in, so other evals' tool surface doesn't change) and projectScoped: true (--project-ref).
  • Fallbacks: the harness is single-turn, so 002 drops the "ok I upgraded, try again" turn and 003 asks for both branches in one prompt. Each eval README explains this.
  • Known limit: a local ai-sdk run showed agents stopping to ask the user to OK the branch price before confirm_cost. In 003 nobody answers, so that run fails because the harness is single-turn, not because the agent is wrong. A real follow-up turn needs multi-turn support across all agent harnesses, which should be its own PR.

Tests: pnpm typecheck, platform-lite tests (111 pass, new branching.test.ts), core tests, framework tests, the new test:regression-lib scorer tests (17), pnpm eval:dry, pnpm format:check.

AI disclosure: Written end to end by Claude Code (AI). Pedro Rodrigues is the accountable owner and should review before merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01L6acVm9fA1bXNQrxtPMcLt

Generated by Claude Code

@claude
feat(evals): add branching entitlement regression evals
c92dd35 
Three tools-mode regression evals for MCP branching on free vs paid plans,
from the create_branch PaymentRequiredException failures (AI-1294):

- deploy-database-002: free plan, agent should check the plan with
  get_organization and stop before get_cost/confirm_cost/create_branch.
- deploy-database-003: Pro plan, two branches in one prompt via the cost
  flow, without re-checking the plan.
- deploy-database-004: free plan, project-scoped MCP; one create_branch
  attempt should get a non-error "requires a paid plan" result.

Expected to fail until the MCP server change ships and MCP_SERVER_VERSION
is bumped.

Infra:
- platform-lite: seedable org plan, plan-free org list, entitlements
  endpoint (branching only), branch list/create with the platform's 402
  for free orgs, branches replay migration history, seedable migrations.
- harness: remote/organization.json and remote/migrations/ seeds, and
  projectScoped / mcpFeatures frontmatter for tools evals.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L6acVm9fA1bXNQrxtPMcLt
@claude claude Bot added the run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes label Oct 7, 2026
@vercel

vercel Bot commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
evals Ignored Ignored Preview Oct 8, 2026 1:25pm UTC

Request Review

@claude

claude Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

CI results from run 37674475604: codex-gpt-6-luna and codex-gpt-6-luna-no-skills, 3 runs each. Pulled from the raw-results artifact.

Eval Passed Failure type
002 free plan 0/6 Agent behavior, expected until the server ships. In 5 of 6 runs it called get_cost before checking the plan, and in 5 of 6 create_branch returned the 402 as a tool error (Branching is supported only on the Pro plan or above). In 4 of 6 it never called get_organization. The paid-plan explanation judge passed in 5 of 6.
003 paid plan, two branches 4/6 Harness limitation. Both failures stopped after get_cost to ask the user to approve the price, and a single-turn eval has nobody to answer. Passing runs went get_cost → confirm_cost → create_branch with 0 get_organization calls and no errors.
004 free plan, project-scoped 0/6 Server gap, expected. No run called create_branch, because its schema requires confirm_cost_id and nothing in project-scoped mode can produce one. Agents blamed the missing confirmation step and not the plan. No tool errors, no confirm_cost.

What this means for the server change: in project-scoped mode, create_branch has to check entitlement before it requires a cost confirmation (or make confirm_cost_id optional there). Otherwise agents won't call it and won't get the "requires Pro" result.

The results refresh only added the 18 new rows. No existing results changed.

Generated by Claude Code

@Rodriguespn
fix(evals): require a create_branch attempt in 004, classify 003 appr…
796786c 
…oval stop as harness limit
@Rodriguespn
chore(platform-lite): sync advertised OpenAPI entries for org, entitl…
3f45850 
…ements and branch routes with upstream
@Rodriguespn
refactor: simplify branching eval helpers and platform-lite seeding
9a4c182 
- platform-lite: one migration history array (with SQL) instead of parallel arrays; Branch/Organization types from the generated Management API schema; organization.json loading lives with the other seed loaders
- core/sandbox: MCP feature lists typed as McpFeature
- regression lib: drop unused is_error/snake-case handling, single-pass cost-flow check, colocate 003's branch listing and ignore the default branch
@Rodriguespn Rodriguespn mentioned this pull request Oct 8, 2026
Draft
Rodriguespn and others added 4 commits October 8, 2026 10:53
@Rodriguespn
refactor(core): move generic MCP tool-call checks into the framework
3ca0ede 
isToolError, checkNoMcpToolErrors and checkMcpCallCount now live in @supabase-evals/core (mcp-tool-checks.ts) so any tools eval can use them. The branching-specific cost-flow check stays in evals/regression/lib/branching.ts.
@Rodriguespn
feat(evals): cost-consent prompt line, upgrade-link judge, TEMP MCP p…
6382546 
…review pin

- 002-004 PROMPT.md: 'Whatever it costs is fine, no need to check with me.' replaces the approval turn the single-turn harness can't give
- 002/004 judge now requires linking the org billing page (https://supabase.com/dashboard/org/<slug>/billing) to upgrade
- TEMP(AI-1292): MCP_SERVER_VERSION points at the supabase/mcp#463 pkg.pr.new preview; revert before merge
@Rodriguespn
refactor(evals): shared MCP tool-call checks live in evals/lib
ea8f4c2 
checkMcpCallCount, checkNoMcpToolErrors and isToolError move from @supabase-evals/core to evals/lib/ so every suite can reuse them; discovery skips evals/lib; test:regression-lib becomes test:evals-lib; CONTRIBUTING points scorer authors at the shared checks.
@github-actions
chore: refresh eval results [skip ci]
9948c53
Rodriguespn and others added 2 commits October 8, 2026 11:34
@Rodriguespn
fix(evals): scope shared MCP checks to the Supabase MCP server
899f903 
Codex records its own list_mcp_resources helper as an MCP call; a failed helper call (wrong server name) false-failed 004 in CI run 37763240750.
@github-actions
chore: refresh eval results [skip ci]
2e5cc72
Rodriguespn and others added 6 commits October 8, 2026 11:42
@Rodriguespn
feat(evals): 002 expects one availability attempt and an upgrade offer
725c432 
Branching availability isn't only the plan, so 002 no longer requires get_organization first. It allows at most one get_cost/create_branch call, no confirm_cost and no tool errors. The shared judge (002/004) requires saying branching is unavailable, asking about upgrading, and linking the org billing page. checkMcpCallCount accepts several tool names counted together.
@Rodriguespn
docs(evals): 003 approval stops after get_cost are a harness limit
6e69094
@Rodriguespn
docs(evals): get_cost -> confirm_cost is the pre-2026-07-28 cost flow
bd9e67b 
Clients with form elicitation (MCP revision 2026-07-28) confirm the branch cost inside create_branch; 003 and its cost-flow check cover the legacy flow.
@Rodriguespn
feat(evals): branching evals follow the platform availability design
26a63a8 
- platform-lite: GET /v1/projects/{ref}/branching -> { available } (same rule as branch creation's 402); drop the org entitlements route and emulation that only the old design used
- 002: no get_organization requirement; <=1 get_cost/create_branch, no confirm_cost, no tool errors, upgrade-offer judge
- 003: drop the get_organization at-most-once check
- 004: exactly one create_branch that returns the non-error unavailable result, no tool errors, same judge
- READMEs describe the platform endpoint and the non-error result
@Rodriguespn
feat(evals): judge only requires saying the plan doesn't include bran…
6aa2881 
…ching

Per the user: the reply needn't link the billing page or ask a question; mentioning the upgrade is optional.
@github-actions
chore: refresh eval results [skip ci]
9d8b65a
Rodriguespn and others added 2 commits October 8, 2026 13:18
@Rodriguespn
test(platform-lite): /branching answers for a branch's own project ref
29bfc1c 
The MCP server resolves branch-id tools to the branch ref and checks availability there.
@github-actions
chore: refresh eval results [skip ci]
358afa9
Rodriguespn and others added 2 commits October 8, 2026 13:32
@Rodriguespn
feat(evals): 002/004 pass on one branching-tool unavailable result
3063de3 
Every branching tool now fails fast with the same non-error result, so 002 and 004 require exactly one branching-tool call (create/list/delete/merge/reset/rebase_branch or get_cost) to return it, no confirm_cost and no tool errors; create_branch isn't required. 003 unchanged.
@github-actions
chore: refresh eval results [skip ci]
b491008
@Rodriguespn
feat(evals): 002/004 allow parallel first branching calls, fail on re…
000e909 
…tries

Pass when at least one branching-tool call returns the non-error unavailable result and no branching call starts after the first such result came back. Parallel first calls count as one attempt. CI run 5 rescored offline: 002 6/6, 004 6/6.

This branch was successfully deployed

1 active (outdated) deployment
Preview — b491008e Deployed Oct 8, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@claude @Rodriguespn