feat(catalog): reconcile gpt-5.6-sol/terra/luna onto v0.75 (v0.76-v0.78) - #836
feat(catalog): reconcile gpt-5.6-sol/terra/luna onto v0.75 (v0.76-v0.78)#836aminsamir45 wants to merge 1 commit into
Conversation
…-v0.78 gpt-5.6-sol/terra/luna were catalog-added in #721 (2026-07-15) but their roster-onboarding overlay (branch add-gpt56-v075-overlay, commit 4ae87e1, "v0.75 = v0.74 + gpt-5.6-*") was never merged. A different v0.75 forked from an earlier parent (v0.73 + claude-opus-5, #833) landed first and won the version name, orphaning the gpt-5.6 overlay. Sol/Terra/Luna never shipped to deployed_models as a result — the Weave dashboard genuinely never had them. Reconciles by appending all three onto the CURRENT (opus-5) v0.75 via append_quality_column.py, one version per model: v0.76 = v0.75 + gpt-5.6-sol (TierHigh, $5/$30) v0.77 = v0.76 + gpt-5.6-terra (TierHigh, $2.5/$15) v0.78 = v0.77 + gpt-5.6-luna (TierMid, $1/$6) Two fixes vs. a naive append: 1. Parity-gate gotcha: this lineage's rankings.json was baked with a UNIFORM alpha=0.96 blend (per build_v075_opus5_swap.py, proven against v0.73), but append_quality_column.py's load_version_knobs() reads the per-cluster alpha-floor VECTOR from metadata.yaml's training.default_routing_knobs -- a separate request-time dial default. Forcing uniform alpha=0.96 reproduces v0.73/v0.75's own rankings.json bit-for-bit (Delta=0.0); the vector reading fails parity by 0.159. 2. Full-corpus fit vs. impute-only: fitting each model's column against only its own Tier-3 shards starves the >=4-incumbent minimum and silently imputes every cluster from gpt-5.5. Assembled the combined Tier-3 Easy corpus from all sibling onboarding runs already on GCS (fable-5, sonnet-5, kimi-k2.7, minimax-m3, qwen3.7-plus, glm-5.2, plus the three gpt-5.6 runs) so coding clusters 5/8/13/14 get real fitted cells (5-7 incumbent points each) instead of pure imputation. centroids.bin byte-identical at every step (no re-cluster); parity gate passes (Delta=0.0) against each immediate parent. $0 spend -- all solve data already existed on GCS from the original onboarding probes (2026-07-15/16). go build/vet/test all green. Candidate only: no live bake-off run, no latest bump. Full writeup: router-internal/docs/eval/RESULTS_gpt56_v076_v078_reconcile_opus5_fork_20260724.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
🧭 Cluster routing review
Cluster routing report —
|
| register | Opus % | premium % | top models |
|---|---|---|---|
| conversational | 13% | 23% | deepseek-v4-pro:13 gpt-5.6-luna:7 claude-opus-5:4 gemini-3.1-pro-preview:2 |
| trivial_nl | 3% | 37% | deepseek-v4-pro:13 gemini-3.1-pro-preview:10 gpt-5.6-luna:5 claude-opus-5:1 |
| knowledge_qa | 73% | 80% | claude-opus-5:22 deepseek-v4-pro:2 gemini-3.1-pro-preview:2 gpt-5.6-luna:2 |
| easy_code | 40% | 57% | claude-opus-5:12 deepseek-v4-pro:6 gpt-5.6-luna:6 gpt-5.5:3 |
| hard_code | 97% | 100% | claude-opus-5:29 gpt-5.5:1 |
| agentic_tool | 90% | 90% | claude-opus-5:27 gpt-5.6-luna:3 |
Change vs deployed (v0.75)
| register | Opus % | premium % |
|---|---|---|
| conversational | 13% → 13% (+0) | 43% → 23% (-20) |
| trivial_nl | 3% → 3% (+0) | 53% → 37% (-17) |
| knowledge_qa | 73% → 73% (+0) | 87% → 80% (-7) |
| easy_code | 40% → 40% (+0) | 80% → 57% (-23) |
| hard_code | 97% → 97% (+0) | 100% → 100% (+0) |
| agentic_tool | 90% → 90% (+0) | 100% → 90% (-10) |
What each cluster represents (probe register mix by nearest centroid)
| cluster | probes | dominant | register breakdown |
|---|---|---|---|
| 0 | 36 | agentic_tool | agentic_tool:12 knowledge_qa:9 hard_code:8 conversational:4 easy_code:2 trivial_nl:1 |
| 1 | 8 | easy_code | easy_code:3 agentic_tool:2 conversational:1 knowledge_qa:1 trivial_nl:1 |
| 2 | 2 | conversational | conversational:2 |
| 3 | 1 | knowledge_qa | knowledge_qa:1 |
| 4 | 3 | conversational | conversational:1 easy_code:1 trivial_nl:1 |
| 5 | 14 | agentic_tool | agentic_tool:5 conversational:2 easy_code:2 hard_code:2 knowledge_qa:2 trivial_nl:1 |
| 6 | 1 | easy_code | easy_code:1 |
| 7 | 6 | agentic_tool | agentic_tool:2 easy_code:2 hard_code:1 knowledge_qa:1 |
| 8 | 9 | knowledge_qa | knowledge_qa:4 agentic_tool:2 hard_code:2 easy_code:1 |
| 9 | 10 | conversational | conversational:4 trivial_nl:4 easy_code:1 knowledge_qa:1 |
| 10 | 24 | conversational | conversational:12 trivial_nl:7 easy_code:3 knowledge_qa:2 |
| 11 | 4 | trivial_nl | trivial_nl:2 easy_code:1 knowledge_qa:1 |
| 12 | 13 | easy_code | easy_code:9 knowledge_qa:2 agentic_tool:1 trivial_nl:1 |
| 13 | 25 | hard_code | hard_code:17 agentic_tool:6 easy_code:1 knowledge_qa:1 |
| 14 | 7 | knowledge_qa | knowledge_qa:4 easy_code:2 trivial_nl:1 |
| 15 | 17 | trivial_nl | trivial_nl:11 conversational:4 easy_code:1 knowledge_qa:1 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit feffe34. Configure here.
| qwen/qwen3.7-plus: 0.0004 | ||
| claude-sonnet-5: 0.003 | ||
| claude-fable-5: 0.01 | ||
| claude-opus-5: 0.005 |
There was a problem hiding this comment.
Incomplete metadata roster overlay
Medium Severity
deployed_models and cost_per_1k_input_usd still mirror the v0.75 18-model roster and omit gpt-5.6-sol / terra / luna, while model_registry, quality_means, model_features, model_axes, and rankings already include them. CheapestModel reads costs from this map and silently skips unlisted models, and the changelog claims a 21-model roster.
Additional Locations (2)
Triggered by learned rule: Cluster artifact YAML/JSON keys must match Go struct tags in artifacts.go
Reviewed by Cursor Bugbot for commit feffe34. Configure here.
| "qwen/qwen3.7-plus": 0.03878787878787882, | ||
| "xiaomi/mimo-v2.5-pro": 0.6127412731317272, | ||
| "z-ai/glm-5.2": 0.03474747474747478 | ||
| }, |
There was a problem hiding this comment.
Imputed conversational argmax flip
High Severity
Unmeasured clusters impute quality from gpt-5.5, so cheaper gpt-5.6-terra/luna outscore it on conversational clusters 2/9/10 where v0.75 had gpt-5.5 as top-1. Runtime scoring argmaxes baked rankings.json, so promotion would shift that traffic onto models with no measured signal there, contradicting the stated non-winning onboarding pattern.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit feffe34. Configure here.
 T-Rex LogsWhat T-Rex did
Reviews (1): Last reviewed commit: "feat(catalog): reconcile gpt-5.6-sol/ter..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| deployed_models: | ||
| - claude-haiku-4-5 | ||
| - claude-opus-5 | ||
| - deepseek/deepseek-v4-flash | ||
| - deepseek/deepseek-v4-pro | ||
| - gemini-3.1-flash-lite-preview | ||
| - gemini-3.1-pro-preview | ||
| - gemini-3.5-flash | ||
| - gpt-5.4-mini | ||
| - gpt-5.5 | ||
| - qwen/qwen3-coder-next | ||
| - qwen/qwen3-next-80b-a3b-instruct | ||
| - xiaomi/mimo-v2.5-pro | ||
| - z-ai/glm-5.2 | ||
| - minimax/minimax-m3 | ||
| - moonshotai/kimi-k2.7 | ||
| - qwen/qwen3.7-plus | ||
| - claude-sonnet-5 | ||
| - claude-fable-5 |
There was a problem hiding this comment.
metadata.yaml still lists only the v0.75 18-model roster, so gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna are absent from both deployed_models and cost_per_1k_input_usd even though model_registry.json, quality_means.json, and rankings.json include them. This defeats the PR's dashboard/provenance goal and makes metadata-driven helpers like CheapestModel skip these models. The same stale roster pattern appears in v0.76 and v0.77.
|
Closing: wrong surface. Prod and staging run gpt-5.6 sol/terra/luna have in fact been routable in prod since 2026-07-21: they are already arms in the pinned HMM bandit state ( Not lost work — two findings from this build are being carried into the docs/skills so the next person doesn't repeat it:
If we ever need gpt-5.6 on the |
2 participants
Summary
gpt-5.6-sol/terra/lunawere catalog-added in #721 (2026-07-15), but their roster-onboarding overlay (add-gpt56-v075-overlay, commit4ae87e1, "v0.75 = v0.74 + gpt-5.6-*") was never merged. A differentv0.75—v0.75 = v0.73 + claude-opus-5(#833) — forked from an earlier parent and landed first, winning the version name and orphaning the gpt-5.6 branch. Net effect: Sol/Terra/Luna never shipped todeployed_models, so the Weave dashboard never showed them despite the catalog/pricing/force-model work being live for over a week.This PR re-appends all three onto the current (post-opus-5)
v0.75as three sequential column-append versions:v0.76= v0.75 +gpt-5.6-sol(TierHigh, $5/$30)v0.77= v0.76 +gpt-5.6-terra(TierHigh, $2.5/$15)v0.78= v0.77 +gpt-5.6-luna(TierMid, $1/$6)$0 spend — reuses Tier-3 SWE-bench Easy shards already sitting on GCS from the original onboarding probes (
add-gpt-5.6-{sol,terra,luna}, 582 shards each, 2026-07-15/16). No new inference.Two gotchas fixed along the way
append_quality_column.py'sload_version_knobs()reads the per-cluster alpha-floor vector frommetadata.yaml'straining.default_routing_knobs— but this lineage'srankings.jsonwas actually baked with a uniform alpha=0.96 blend (perbuild_v075_opus5_swap.py, proven against v0.73). The vector reading fails the tool's own parity gate (Δ=0.159); forcing uniform alpha=0.96 reproduces v0.73/v0.75 bit-for-bit (Δ=0.0). This affects every version in the v0.68/v0.70–v0.75 lineage, not just this PR.Full writeup:
router-internal/docs/eval/RESULTS_gpt56_v076_v078_reconcile_opus5_fork_20260724.md(companion doc lives in the WorkWeave monorepo, not this repo).Test plan
go build ./...— cleango vet ./...— cleango test ./...— all 58 packages passrankings.json) at each of the three append stepsoffline_screen.py --versions v0.75,v0.76,v0.77,v0.78— no coding-cluster argmax regression; new models enter as viable non-winning targets at current evidence level (same pattern as prior additive onboardings)ROUTER_MODEL_LIFECYCLE.md,artifacts/latestmust not move without one — this PR does not touchartifacts/latestand should not be promoted without a follow-up bake-off PR.🤖 Generated with Claude Code