Skip to content

Add OrcaRouter-Adaptive submission to leaderboard - #43

Closed
ZhenghuaBao wants to merge 2 commits into
RouteWorks:mainfrom
ZhenghuaBao:add-orcarouter-adaptive
Closed

Add OrcaRouter-Adaptive submission to leaderboard#43
ZhenghuaBao wants to merge 2 commits into
RouteWorks:mainfrom
ZhenghuaBao:add-orcarouter-adaptive

Conversation

@ZhenghuaBao

@ZhenghuaBao ZhenghuaBao commented May 29, 2026

Copy link
Copy Markdown

Summary

Adds the OrcaRouter-Adaptive entry to the leaderboard data, mirroring the upstream RouterArena submission merged in PR #104 (predictions), PR #108 (README rank), and PR #110 (website link).

Submission

  • Router: OrcaRouter-Adaptive
  • Approach: LinUCB contextual bandit with embedding features
  • Model pool (10): Anthropic claude-haiku/sonnet-4, Google gemini-2.5-flash/-lite, OpenAI gpt-4o-mini/gpt-5-mini, DeepSeek chat/reasoner, Alibaba qwen3-235b/30b
  • Website: https://www.orcarouter.ai/
  • Type: closed-source (paper + GitHub may be added in a follow-up)

Bot-reported metrics

Metric Value
Arena Score 72.08
Accuracy 75.54
Cost / 1K Queries $1.00
Robustness 22.62
Optimal Selection
Optimal Cost
Optimal Accuracy
Latency

The three optimality columns and latency are dashed — the submitted prediction file did not include for_optimality=True entries, same convention as GPT-5 and Azure-Model-Router. May be filled in a follow-up submission.

Files touched

  • src/data/routerMetrics/leaderboard.json — new row, sorted by Arena Score (between Sqwish 75.27 and R2-Router 71.6)
  • src/data/routers.json — new metadata entry
  • src/data/routers.yaml — new metadata entry

No category_scores.json change — routerData.ts auto-synthesizes the entry from the leaderboard row.

Test plan

  • python -c "json.load(...)" on both edited JSON files — parses clean

ZhenghuaBao and others added 2 commits May 29, 2026 10:45
Mirrors the upstream RouterArena submission merged in PR #104 (predictions),
#108 (README), and #110 (link). LinUCB contextual bandit over a 10-model
pool; Arena 72.08, Accuracy 75.54, Cost/1K $1.00, Robustness 22.62. The
three optimality columns and latency are nulled — the submitted prediction
file did not include for_optimality entries, same convention as GPT-5 and
Azure-Model-Router.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ZhenghuaBao

Copy link
Copy Markdown
Author

Hi @yl231,

Thank you for maintaining RouterArena. I would like to request that this PR be reviewed and merged so that our work can be displayed on the official RouterArena website.

Thanks for your time and consideration.

@yl231

yl231 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Closing as superseded — OrcaRouter-Adaptive has been live on the leaderboard for several weeks via the automated RouterArena → website sync. This standalone PR is no longer needed.

@yl231 yl231 closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants