Skip to content

feat(evals): accept any npm version spec for cliVersion - #383

Open
Coly010 wants to merge 2 commits into
mainfrom
feat/cli-version-any-dist-tag
Open

Coly010 wants to merge 2 commits into
mainfrom
feat/cli-version-any-dist-tag

Conversation

@Coly010

@Coly010 Coly010 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #381 (review).

Fixes AI-1299

Current Behavior

localStackRuntime({ cliVersion }) accepts an exact version, or one of a closed set of channels ('stable' | 'beta' | 'next'). Each channel is hard-wired to an npm dist-tag, a SUPABASE_CLI_{STABLE,BETA,NEXT}_VERSION env var and an eval-refresh.yml dispatch input, so adding a channel needs framework changes. When npm points beta at a draft release, the resolver walks back to an older beta. One failed resolution fails the whole Vercel run.

Expected Behavior

  • cliVersion takes anything npm view supabase@<spec> resolves:
    • Exact version: 2.120.0, or v2.120.0 with the v stripped.
    • Dist-tag: latest, beta, next, …
    • Semver range: ^2.120.0. As with npm-pick-manifest, a range resolves to latest when that satisfies it, and to the highest satisfying non-prerelease version otherwise.
  • Experiments that used stable now use latest, with no alias. The experiment name codex-gpt-6-luna-cli-stable is unchanged, so the dashboards, which key on that name, are unaffected. The marker's channel field now holds the spec, e.g. latest.
  • The beta walk-back is removed. The .deb asset check stays, so a release with a missing asset fails with a clear error.
  • The runner resolves each spec once per run with Promise.allSettled. A spec that fails to resolve fails only the pairs that use it, through the usual SANDBOX FAILED path. Every other pair runs with the pins that did resolve, forwarded in a single SUPABASE_CLI_VERSION_PINS JSON env var.
  • The cli_*_version dispatch inputs are removed from eval-refresh.yml. To reproduce a run, pin an exact cliVersion in the experiment.
  • Braintrust row metadata now records the resolved version as cli_version.
  • The dist-tags document and the packument are each fetched once per process, and spec lookups use Object.hasOwn.
  • README and CONTRIBUTING are updated.

🤖 Generated with Claude Code

…ion workflow inputs

localStackRuntime({ cliVersion }) now takes an exact version or any npm
dist-tag of supabase instead of a closed stable/beta/next list. Run-level
pins move to a single SUPABASE_CLI_DIST_TAG_PINS JSON env var, and the
cli_*_version dispatch inputs are removed from eval-refresh.yml.
@Coly010
Coly010 requested a review from a team as a code owner October 8, 2026 16:01
@Coly010
Coly010 requested a review from mattrossman October 8, 2026 16:01
@vercel

vercel Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
evals Ignored Ignored Preview Oct 9, 2026 8:42am UTC

Request Review

Resolve exact versions, dist-tags and semver ranges the way
npm view supabase@<spec> does, drop the beta walk-back, fail only the
pairs whose spec cannot be resolved, and record the resolved version as
cli_version in Braintrust metadata.

Refs AI-1299
@Coly010 Coly010 changed the title feat(evals): accept any npm dist-tag for cliVersion and drop CLI version workflow inputs feat(evals): accept any npm version spec for cliVersion Oct 9, 2026
@Coly010 Coly010 self-assigned this Oct 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant