Yl231 sync leaderboard 2026 05#40
Merged
Merged
Conversation
- leaderboard.json: add Weave Router (74.61) and Auto Router (70.05) entries; fix NotDiamond Arena 63.0 -> 57.29 (per RouterArena PR #34). - category_scores.json: add per-domain breakdowns for r2-router, auto_router, and weave-router. Computed locally from RouterArena full-split predictions joined with dataset Domain/Category/Difficulty metadata; robustness from the per-router robustness predictions. - routers.json: add Weave Router and Auto Router metadata. Drop the orphan universal_router entry (had no corresponding leaderboard row). - routers.yaml: append R2-Router (was only in routers.json), Weave Router, and Auto Router. - flip_labels/: add per-query flip labels for r2-router, auto_router, weave-router, and vllm-sr. Computed as model-selection equality between full and robustness predictions, universal-name normalized. Sqwish Router (RouterArena PR #94) is excluded; will be added in a follow-up once that submission's CI re-run completes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- leaderboard.json: add Sqwish Router (RouterArena PR #94) at rank 1. Metrics from the /evaluate CI run on 2026-05-20: Arena 75.27, Accuracy 76.40, Cost/1K $0.18, Robustness 100.00, Opt 14.92/17.17/100.00. - category_scores.json: add per-domain entry for sqwish-router. Also refresh entries for r2-router, auto_router, weave-router so that difficulty-bucket cells with no samples default to 0 (matching the existing TypeScript schema) instead of null. The TS build was rejecting the prior null cells against CompareSubcategory.metrics. - routers.json + routers.yaml: add Sqwish Router metadata (closed-source, affiliation Sqwish, 5-model pool from the PR config). - flip_labels/flip_labels_sqwish-router.json: new (all-stable, since Sqwish picked the same model on every robustness pair). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
RouterArena PR #97 (merged into main 2026-05-19) re-applied the PR #95 optimality fix and re-ran the /evaluate workflow for the four routers with for_optimality entries. The earlier sync used the pre-PR-#97 buggy values; this commit updates them to match origin/main README. - Weave Router: OptSel 1.38 -> 1.27, OptCost 12.27 -> 12.10, OptAcc 100.00 -> 89.56 - R2-Router: OptSel 32.38 -> 24.51, OptCost 74.16 -> 48.70, OptAcc 100.00 -> 99.85 - Auto Router: OptSel 43.77 -> 37.58, OptCost 42.29 -> 40.02, OptAcc 100.00 -> 86.04 - vLLM-SR: OptSel 94.10 -> 84.66, OptCost 90.12 -> 90.71, OptAcc 100.00 -> 89.24 Sqwish Router was evaluated by CI against PR #94's pre-#97 worktree, so its bot-reported numbers used the buggy formula. Recomputing locally with the merged main run.py on the same prediction file: - Sqwish Router: OptSel 14.92 -> 7.41, OptCost 17.17 -> 25.10, OptAcc 100.00 -> 90.47 Other rows (paper-era + Chayan) are unaffected since PR #97 only re-ran workflow-evaluated routers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
yl231
force-pushed
the
yl231-sync-leaderboard-2026-05
branch
from
May 20, 2026 04:02
6cb30a9 to
c16a1cb
Compare
…ment arrays) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.