Skip to content

Add Lynkr router submission#167

Open
vishalveerareddy123 wants to merge 3 commits into
RouteWorks:mainfrom
vishalveerareddy123:add-lynkr
Open

Add Lynkr router submission#167
vishalveerareddy123 wants to merge 3 commits into
RouteWorks:mainfrom
vishalveerareddy123:add-lynkr

Conversation

@vishalveerareddy123

Copy link
Copy Markdown

Router: Lynkr

Lynkr is an open-source (Apache-2.0), self-hosted LLM gateway that routes requests across local and cloud models. Routing here uses its embedding-anchor tier classifier (no LLM classifier call — routing decisions are local and deterministic).

Files

  • router_inference/config/lynkr.json — router configuration
  • router_inference/predictions/lynkr.json — full-split predictions with generated_result populated
  • router_inference/predictions/lynkr-robustness.json — robustness predictions
  • router_inference/router/lynkr_router.py + registration in __init__.py — adapter for reproducibility
  • Additive entries in model_cost/model_cost.json (z-ai/glm-4.7 pricing) and universal_model_names.py for the pool models

Local evaluation results (for reference; please re-run officially)

Arena 67.65 · Accuracy 68.41% · $0.29/1K queries · Robustness 92.38 · Opt.Sel 11.13 / Opt.Cost 17.00 / Opt.Acc 85.37

Disclosures

  • Optimality metrics were computed on 629/809 of the optimality queries locally — our OpenRouter credits ran out on the GLM tail. The prediction files are complete; your official evaluation should produce full coverage.
  • Routing is heavily concentrated on gpt-oss-120b (~96% of queries) — a consequence of the tier boundaries, not tuning on RouterArena data.
  • Per the evaluation-only policy: no Lynkr component was trained, fitted, or tuned on RouterArena data or labels. The anchor classifier was built entirely from Lynkr's own telemetry before this evaluation; RouterArena was run once as a held-out test.

Happy to adjust anything to match submission conventions.

Lynkr is an open-source (Apache-2.0), self-hosted LLM gateway that routes
requests via embedding-anchor tiering. Pool: openai/gpt-oss-120b,
qwen3-235b-a22b-2507, z-ai/glm-4.7 (all via OpenRouter).

Includes router config, full-split predictions with generated_result,
robustness predictions, the router adapter class, and additive entries to
model_cost/universal_model_names for the pool models.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vishalveerareddy123

Copy link
Copy Markdown
Author

/evaluate

…t router

13 entries where the model responded but no answer text was extractable now
carry generated_answer="" and success=false with an explanatory error field
(previously null answer with success=true). Validator passes locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vishalveerareddy123

Copy link
Copy Markdown
Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: lynkr
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.6761
Accuracy 68.35%
Total Cost $2.424762
Avg Cost per Query $0.000289
Avg Cost per 1K Queries $0.2887
Number of Queries 8400
Abnormal Entries 6
Robustness Score 0.9238

⚠️ 6 of 8400 queries (0.1%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.1116
Opt.Cost (Cost Efficiency) 0.1700
Opt.Acc (Accuracy vs Optimal) 0.8565

Evaluation completed by RouterArena automated workflow

All previously-failed entries (credit exhaustion during original run) now have
real generated results — 176/176 successful. Validator passes locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vishalveerareddy123

Copy link
Copy Markdown
Author

/evaluate

1 similar comment
@vishalveerareddy123

Copy link
Copy Markdown
Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: lynkr
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.6765
Accuracy 68.41%
Total Cost $2.456236
Avg Cost per Query $0.000292
Avg Cost per 1K Queries $0.2924
Number of Queries 8400
Abnormal Entries 0
Robustness Score 0.9238

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.1097
Opt.Cost (Cost Efficiency) 0.1608
Opt.Acc (Accuracy vs Optimal) 0.8448

Evaluation completed by RouterArena automated workflow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant