Skip to content

chore(release): cut 0.15.0 "evals regression gate" (R7) - #206

Merged
TinDang97 merged 1 commit into
mainfrom
chore/cut-0.15.0-evals
Aug 14, 2026
Merged

chore(release): cut 0.15.0 "evals regression gate" (R7)#206
TinDang97 merged 1 commit into
mainfrom
chore/cut-0.15.0-evals

Conversation

@TinDang97

Copy link
Copy Markdown
Collaborator

chore(release): cut 0.15.0 "evals regression gate" (R7)

Closes and archives R7 milestone evals-regression-gate (7/7 exit criteria) and cuts the release.

R7 is the first product-feature milestone since R6's substrate work: a tenant runs a named eval set against a candidate model and gets a scored pass/fail verdict against a pinned baseline, so a model swap is proven safe BEFORE it ships — without touching the hot proxy path, auto-promoting on a verdict, or scoring with an LLM judge.

Shipped across the milestone (all merged, all gated PASS)

Task PR What
eval-set-store #201 /v1/evals/sets + cases; ZDR disposition frozen first (atomic re-check, slow-double verified)
deterministic-scorers #202 exact · contains · regex · json-schema; re-run stable
eval-run-executor #203 governed set replay — billed traffic, per-tenant breaker
baseline-and-verdict #204 pin baseline, exact integer verdict, explicit no_baseline
evals-console #205 verdict-first, read-focused, reuses the /v1 verdict core (byte-identical)

Release mechanics

  • Version bumped across all 8 sites per the RELEASES.md checklist: pyproject + __version__ fallback, Chart.yaml version (0.2.10.2.2) + appVersion, values.yaml gateway+dashboard tags, values-prod.yaml both -prod overrides.
  • Dashboard package.json stays independently versioned (0.1.0) — a gateway release does not ship a phantom UI release.
  • Guards green locally: tests/release_provenance (served /openapi.json == pyproject == fallback; RELEASES newest ≥ pyproject) + production digest-pin / multi-arch image guards — 14 passed.

Waivers

Two carry forward from 0.14.0, both still in window: suite-stability + pgvector-deploy-runbook (RISK-ACCEPTED, owner Tin Dang, expire 2026-09-30). ci-restoration stays DISCHARGED. No new waivers.

⚠ Human-run after merge (tagging PUBLISHES images via publish-images.yml)

  1. git tag -a v0.15.0 on the merge commit, and push
  2. verify the publish-images run is green (reads back both manifests) — do not deploy a tag whose publish job is not green
  3. staging first, following the pgvector runbook (dump/restore path)

⚠ OPEN — carried to R8 soc2-groundwork, NOT waived

The required_approving_review_count: 1 gate was satisfied on all five R7 PRs by an operator-directed self-approval via a second account (pilotspacex-byte), disclosed on each PR. That evidences a green required check on the merged artifact; it does not evidence independent four-eyes review. Every R7 merge still owes a genuine second human with write access before the SOC 2 audit.

Close and archive R7 milestone `evals-regression-gate` (7/7 exit criteria) and
cut the release. R7 is the first product-feature milestone since R6's substrate
work: a tenant runs a named eval set against a candidate model and gets a scored
pass/fail verdict against a pinned baseline, so a model swap is proven safe
BEFORE it ships — without touching the hot proxy path, auto-promoting on a
verdict, or scoring with an LLM judge.

Shipped across the milestone (all merged, all gated PASS):
  - eval-set-store (#201)      — /v1/evals/sets + cases; ZDR disposition frozen first
  - deterministic-scorers (#202) — exact · contains · regex · json-schema, re-run stable
  - eval-run-executor (#203)   — governed set replay: billed traffic, per-tenant breaker
  - baseline-and-verdict (#204) — pin baseline, exact integer verdict, no_baseline state
  - evals-console (#205)       — verdict-first, read-focused, reuses the /v1 verdict core

Version bumped across all eight sites per the RELEASES.md checklist: pyproject +
`__version__` fallback, Chart.yaml version (0.2.1->0.2.2) + appVersion, values.yaml
gateway+dashboard tags, values-prod.yaml both `-prod` overrides. Dashboard
package.json stays independently versioned (0.1.0) — a gateway release does not
ship a phantom UI release. Guards green: tests/release_provenance (served
/openapi.json == pyproject == fallback, RELEASES newest >= pyproject) and the
production digest-pin / multi-arch image guards (14 passed).

Waivers: 0.14.0's suite-stability and pgvector-deploy-runbook carry forward
unchanged (both RISK-ACCEPTED, owner Tin Dang, expire 2026-09-30); ci-restoration
stays DISCHARGED. No new waivers.

REMAINING IS HUMAN-RUN (tagging PUBLISHES images via publish-images.yml):
  1. git tag -a v0.15.0 on the merge commit, and push
  2. verify the publish-images run is green (reads back the manifest for both
     images; do not deploy a tag whose publish job is not green)
  3. staging first, following the pgvector runbook (dump/restore path)

OPEN, carried to R8 `soc2-groundwork` and NOT waived: the
`required_approving_review_count: 1` gate was satisfied on all five R7 PRs by an
operator-directed self-approval via a second account (`pilotspacex-byte`),
disclosed on each PR. That evidences a green required check on the merged
artifact; it does NOT evidence independent four-eyes review. Every R7 merge still
owes a genuine second human with write access before the SOC 2 audit.

author: Tin Dang

@pilotspacex-byte pilotspacex-byte left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operator-directed approval via the pilotspacex-byte account to satisfy the required-review gate on a solo-maintainer repo. DISCLOSED and NOT independent four-eyes review: this is the same operator who authored the change, using a second GitHub account. It evidences a green required check on the merged artifact, not a genuine second reviewer. This R7 release (and PRs #201-#205) still owes a real second human with write access before the SOC 2 audit — tracked in R8 soc2-groundwork.

@TinDang97
TinDang97 merged commit c9d4320 into main Aug 14, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants