Proposal to add "Weighted Fallback" to Load-Balancing Strategys

[weselben]

Current Behavior

The Load-Balancing Strategy dropdown in the UI (Source → Targets configuration) currently offers only two options:

1. Round-robin — rotates across all targets, honoring weights
2. Lowest cost — picks the cheapest target per request

There is no option for a prioritized fallback strategy, where requests always go to the highest-priority (highest-weight) target, and only cascade to lower-priority targets when the primary is unavailable (e.g., circuit breaker open, rate-limited, or returning errors).

Desired Behavior

Add a third option: Weighted Fallback — or alternatively a Fallback strategy toggle.

Behavior:

• Targets are ordered by their assigned weight (highest first)
• Every request always tries the highest-weighted target first
• Only if that target fails or is in circuit breaker / unavailable state, the request falls back to the next target in priority order
• Continues cascading down the list until a healthy target responds or all targets are exhausted

Use Case

In setups where:

• One target is significantly more capable/reliable (e.g., kimicode/bge_m3_embed with weight 10)
• A secondary target exists as a cheap backup (e.g., local ollama/qwen3-embedding:0.6b with weight 1)

…the user wants predictable prioritization — always use the best model unless it's down, rather than distributing traffic round-robin across tiers of very different quality.

Proposed UI Change

In the Load-Balancing Strategy dropdown, add:

Weighted fallback — tries highest-weight target first; falls back on circuit breaker

Optional Enhancement

Consider clarifying the weight semantics per strategy in the UI:

• Round-robin: weight = traffic distribution ratio
• Weighted fallback: weight = priority order (highest = primary)

---

Technical Context & Impacted Areas

To introduce this strategy, the codebase needs to register the new identifier and adjust target routing:

• Backend Strategy Registry & Validation:

  • internal/virtualmodels/types.go and internal/virtualmodels/validation.go define and validate acceptable load-balancing strategy values. A new strategy value (e.g., weighted_fallback or fallback) needs to be supported here.

• Load-Balancing Logic:

  • internal/virtualmodels/balancer.go houses target selection (balancedResolution). While it already filters out saturated targets using targetsWithCapacity, the routing logic needs to support a mode that selects the highest-weighted target from the remaining active pool (rather than cycling round-robin or picking by token cost). It should also gracefully fallback if all targets are saturated.

• Admin UI & Templates:

  • internal/admin/dashboard/templates/page-models.html manages the strategy dropdown.
  • internal/admin/dashboard/static/js/modules/virtual-models.js handles label formatting and form visibility (such as ensuring weight inputs are shown when the fallback strategy is selected).

• Tests and Documentation:

  • internal/virtualmodels/balancer_test.go and docs/features/virtual-models.mdx will need updates to cover and document the priority fallback behavior.

[coderabbitai]

🔗 Related PRs

#208 - feat(fallback): enable fallback by default and align async chart states [merged]
#236 - docs(features): add failover page [merged]
#444 - feat: add manual failover management [merged]
#451 - refactor(failover): unify feature naming on "failover" [merged]
#492 - fix(ratelimit): honest 429 when every virtual-model target is saturated [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[SantiagoDePolonia]

AFAIK
It's already there - those arrows icon button on the screenshot. The feature is called failover and it's a separated thing than load balancing. Maybe it should be the same thing?

Definitely let me know if it's what you mean.

The models order is the priority (you called it weight).

Documentation page: https://gomodel.enterpilot.io/docs/features/failover

[weselben]

You're absolutely right, failover already exists as a separate feature, and the model order serves as the priority. Consolidating failover into the load-balancing strategies would make a lot of sense from a UX standpoint.

My take on why merging them would be valuable:

The failover setting is available for "real" models, but when you're configuring a virtual model, it's simply not there, the load-balancing dropdown only offers Round-robin and Lowest cost. There's no way to set up a prioritized fallback chain per virtual model through the UI.

My suggestion: Add "Failover" as a third load-balancing strategy option for virtual models. This would let users configure a virtual model to always try e.g. Kimi first, then cascade to GLM or MiniMax if the primary is unavailable, all managed granularly per virtual model without needing separate failover configuration.

From my HA/networking background, failover is typically considered a load-balancing strategy anyway, so merging them feels conceptually clean and logical, rather than having two separate systems.

TL;DR: Failover exists for real models, but virtual models lack it entirely in their load-balancing strategies. Adding it there would solve the gap.