Upload robustness score details#34
Merged
Merged
Conversation
yl231
added a commit
that referenced
this pull request
May 20, 2026
- leaderboard.json: add Weave Router (74.61) and Auto Router (70.05) entries; fix NotDiamond Arena 63.0 -> 57.29 (per RouterArena PR #34). - category_scores.json: add per-domain breakdowns for r2-router, auto_router, and weave-router. Computed locally from RouterArena full-split predictions joined with dataset Domain/Category/Difficulty metadata; robustness from the per-router robustness predictions. - routers.json: add Weave Router and Auto Router metadata. Drop the orphan universal_router entry (had no corresponding leaderboard row). - routers.yaml: append R2-Router (was only in routers.json), Weave Router, and Auto Router. - flip_labels/: add per-query flip labels for r2-router, auto_router, weave-router, and vllm-sr. Computed as model-selection equality between full and robustness predictions, universal-name normalized. Sqwish Router (RouterArena PR #94) is excluded; will be added in a follow-up once that submission's CI re-run completes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
yl231
added a commit
that referenced
this pull request
May 20, 2026
* Sync leaderboard with RouterArena README (Weave, Auto, R2 entries) - leaderboard.json: add Weave Router (74.61) and Auto Router (70.05) entries; fix NotDiamond Arena 63.0 -> 57.29 (per RouterArena PR #34). - category_scores.json: add per-domain breakdowns for r2-router, auto_router, and weave-router. Computed locally from RouterArena full-split predictions joined with dataset Domain/Category/Difficulty metadata; robustness from the per-router robustness predictions. - routers.json: add Weave Router and Auto Router metadata. Drop the orphan universal_router entry (had no corresponding leaderboard row). - routers.yaml: append R2-Router (was only in routers.json), Weave Router, and Auto Router. - flip_labels/: add per-query flip labels for r2-router, auto_router, weave-router, and vllm-sr. Computed as model-selection equality between full and robustness predictions, universal-name normalized. Sqwish Router (RouterArena PR #94) is excluded; will be added in a follow-up once that submission's CI re-run completes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add Sqwish Router entry; default null per-domain cells to 0 - leaderboard.json: add Sqwish Router (RouterArena PR #94) at rank 1. Metrics from the /evaluate CI run on 2026-05-20: Arena 75.27, Accuracy 76.40, Cost/1K $0.18, Robustness 100.00, Opt 14.92/17.17/100.00. - category_scores.json: add per-domain entry for sqwish-router. Also refresh entries for r2-router, auto_router, weave-router so that difficulty-bucket cells with no samples default to 0 (matching the existing TypeScript schema) instead of null. The TS build was rejecting the prior null cells against CompareSubcategory.metrics. - routers.json + routers.yaml: add Sqwish Router metadata (closed-source, affiliation Sqwish, 5-model pool from the PR config). - flip_labels/flip_labels_sqwish-router.json: new (all-stable, since Sqwish picked the same model on every robustness pair). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Apply PR #97 optimality fix to all workflow-evaluated routers RouterArena PR #97 (merged into main 2026-05-19) re-applied the PR #95 optimality fix and re-ran the /evaluate workflow for the four routers with for_optimality entries. The earlier sync used the pre-PR-#97 buggy values; this commit updates them to match origin/main README. - Weave Router: OptSel 1.38 -> 1.27, OptCost 12.27 -> 12.10, OptAcc 100.00 -> 89.56 - R2-Router: OptSel 32.38 -> 24.51, OptCost 74.16 -> 48.70, OptAcc 100.00 -> 99.85 - Auto Router: OptSel 43.77 -> 37.58, OptCost 42.29 -> 40.02, OptAcc 100.00 -> 86.04 - vLLM-SR: OptSel 94.10 -> 84.66, OptCost 90.12 -> 90.71, OptAcc 100.00 -> 89.24 Sqwish Router was evaluated by CI against PR #94's pre-#97 worktree, so its bot-reported numbers used the buggy formula. Recomputing locally with the merged main run.py on the same prediction file: - Sqwish Router: OptSel 14.92 -> 7.41, OptCost 17.17 -> 25.10, OptAcc 100.00 -> 90.47 Other rows (paper-era + Chayan) are unaffected since PR #97 only re-ran workflow-evaluated routers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Format data files with prettier (trailing newlines, inline single-element arrays) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: yl231 <yl231@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a new category under src/data, called
flip_labels, each json inside the category has two fields,global indexas the key,flip: 1 for model selection fliped, 0 for model selection not fliped