Skip to content

feat(routing): weighted policy for load balancing across providers - #553

Merged
njbrake merged 7 commits into
mainfrom
feat/weighted-routing
Aug 11, 2026
Merged

feat(routing): weighted policy for load balancing across providers#553
njbrake merged 7 commits into
mainfrom
feat/weighted-routing

Conversation

@njbrake

@njbrake njbrake commented Aug 11, 2026

Copy link
Copy Markdown
Member

Description

Note: this description was drafted by Claude Opus 5 via back-and-forth with @njbrake. The reasoning and decisions are his; the prose is Claude's.

Continues #545 by @pss-julien, whose three commits are preserved here with their authorship. That PR's head branch was the fork's main, which has since been reset, and GitHub does not honor "allow edits by maintainers" for a fork owned by an organization, so the branch moved here to carry the review fixes.

What it adds: a select entry can name the weighted router, which draws one candidate per request in proportion to declared weights, so traffic splits across providers behind one policy name. The design rationale is recorded in #545 and in docs/routing.md.

On top of @pss-julien's work, one review commit:

  • declared_shares reported an even split for an all-zero pool while the draw kept declared order, so explain, router_weights, the CLI, and confidence all printed a split the runtime does not serve.
  • /v1/routing/status and preferences/rank treated a weighted policy as a learned one. The second is the sharper edge: a workspace whose only router policy was weighted could no longer seed examples for a learned policy it was preparing.
  • "Retryable failure" is stale phrasing since fix(routing): fall back on every provider failure #546. Restored in the new prose, the CLI output, and the test comments.
  • The allow-list explain transcript omitted the on_failure drop, which the same filter removes.
  • Rebased onto main, dashboard bundle rebuilt, OpenAPI regenerated.

PR Type

  • New Feature
  • Bug Fix
  • Refactor
  • Documentation
  • Infrastructure / CI

Relevant issues

Continues #545. No issue filed.

Checklist

  • I understand the code I am submitting.
  • I have added or updated tests that cover my change (tests/unit, tests/integration).
  • I ran the Definition of Done checks locally (make lint, make typecheck, make test). Unit and web suites plus openapi-check and postman-check ran locally; the integration suite needs PostgreSQL and ran on CI.
  • Documentation was updated where necessary.
  • If the API contract changed, I regenerated the OpenAPI spec (uv run python scripts/generate_openapi.py).

AI Usage

  • No AI was used.
  • AI was used for drafting/refactoring.
  • This is fully AI-generated.

AI Model/Tool used:

Claude Opus 5 via Claude Code, for the review fixes and this description. The three underlying commits were contributed by @pss-julien, who reported them as fully AI generated with the same tool in #545.

Any additional AI details you'd like to share:

Two places worth a human eye. The declared_shares change reverses a behavior @pss-julien deliberately pinned in a test, so it is a decision to confirm rather than a bug fix everyone agrees on. The routing memory change narrows what POST /v1/routing/preferences/rank accepts as a teachable score key, which is a behavior change for any workspace running a weighted policy.

  • I am an AI Agent filling out this form (check box if true)

🤖 Generated with Claude Code

Summary

  • Added weighted routing across providers with configurable traffic shares.
  • Added zero-weight candidates as failover targets.
  • Updated routing explanations, CLI output, status endpoints, confidence reporting, and OpenAPI types.
  • Added dashboard support for creating, editing, validating, and explaining weighted policies.
  • Updated routing documentation and fallback terminology.
  • Kept weighted routing independent of pricing and routing-memory data.
  • Added comprehensive unit and integration test coverage.
  • Updated the dashboard bundle.

@njbrake
njbrake deployed to integration-tests August 11, 2026 13:52 — with GitHub Actions Active
@github-actions github-actions Bot added the missing-template PR is missing required template sections label Aug 11, 2026
@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@njbrake, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 12 seconds

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1a3c1144-1d74-4de7-a3e9-2dc39864bc96

📥 Commits

Reviewing files that changed from the base of the PR and between 6879b14 and 31a161c.

📒 Files selected for processing (1)
  • web/src/pages/RoutingPage.test.tsx

Walkthrough

This PR adds weighted provider routing with validated traffic shares, weighted failover, caller-specific explanations, API and CLI metadata, dashboard configuration flows, documentation, generated dashboard assets, and unit and integration coverage.

Changes

Weighted routing core

Layer / File(s) Summary
Policy validation and weighted backend
src/gateway/models/routing.py, src/gateway/services/routing/...
Policies now validate weighted candidate maps. The backend normalizes shares, orders candidates without replacement, preserves zero-weight failover candidates, and bypasses pricing and learning-specific checks.
Execution and policy explanation
src/gateway/services/routing/decide.py, src/gateway/api/routes/routing.py, src/gateway/cli.py, src/gateway/services/pricing_init_service.py, src/gateway/api/routes/routing_memory.py
Routing passes weights into backend decisions. Explanations and CLI output show filtered, normalized shares. API responses expose router_weights.
Dashboard weighted policy workflow
web/src/api/types.ts, web/src/pages/RoutingPage.tsx, web/src/pages/RoutingPage.test.tsx
The dashboard supports weighted policy creation, editing, validation, serialization, normalized share display, weighted labels, and backend-specific guidance.
Dashboard bundle delivery
src/gateway/static/dashboard/index.html, src/gateway/static/dashboard/assets/*
The dashboard loads the new generated entry bundle and updated hashed module dependencies.
Routing validation and integration tests
tests/unit/test_weighted_router.py, tests/integration/test_routing_weighted.py
Tests cover weighted selection, validation, filtering, failover, opt-out behavior, explanations, pricing independence, usage attribution, and dynamic pricing metadata.
Routing documentation and configuration
config.example.yml, docs/*.md, src/gateway/AGENTS.md
Documentation describes weighted routing, candidate shares, fallback behavior, filtering, observability, candidate limits, dashboard behavior, API responses, and failure semantics.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

  • mozilla-ai/otari#188: Provides the learned-routing backend infrastructure extended by this weighted backend.
  • mozilla-ai/otari#492: Shares the named routing-policy framework and routing model/API surfaces extended here.
  • mozilla-ai/otari#545: Contains the same weighted-routing implementation and related backend, dashboard, documentation, and test changes.

Suggested labels: area/backend, area/dashboard

Suggested reviewers: khaledosman, tbille, dpoulopoulos

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 3.51% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title uses a valid scoped Conventional Commit prefix, clearly describes weighted routing, uses 66 characters, and remains within the length limit.
Description check ✅ Passed The description follows the template, explains the feature and rationale, identifies related work, records testing, and completes the checklist and AI usage details.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/weighted-routing
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch feat/weighted-routing

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot removed the missing-template PR is missing required template sections label Aug 11, 2026
@khaledosman
khaledosman requested a lite review from Copilot August 11, 2026 14:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new weighted router backend to the gateway’s routing system so a single policy can load-balance traffic across multiple provider:model candidates according to operator-declared shares, with supporting updates across API surfaces (explain + OpenAPI), CLI output, dashboard UI, and tests.

Changes:

  • Introduces weighted routing backend with schema validation, deterministic “explain” rendering, and request-path integration.
  • Updates dashboard routing editor/table to display and author weighted splits; adjusts routing-memory surfaces to apply only to learned (knn) policies.
  • Adds unit + integration coverage and updates docs/OpenAPI + rebuilt dashboard bundle artifacts.

Reviewed changes

Copilot reviewed 34 out of 37 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
web/src/pages/RoutingPage.tsx Dashboard routing UI: weighted split authoring, labeling, summaries, Examples gating
web/src/pages/RoutingPage.test.tsx UI tests for weighted routing creation/editing/validation and labeling behavior
web/src/api/types.ts Adds weights to policy select entries and router_weights to explain response typing
tests/unit/test_weighted_router.py Unit tests for weighted schema rules, draw properties, and explain behavior
tests/integration/test_routing_weighted.py End-to-end integration tests for weighted routing behavior and management surfaces
src/gateway/services/routing/weighted.py Implements weighted backend (draw, share normalization, explain ordering)
src/gateway/services/routing/decide.py Passes weights into routing context; adds explain_router_ordering helper; log level control
src/gateway/services/routing/backends.py Registers weighted backend; adds backend capability helpers (learns/pricing/weighted)
src/gateway/services/routing/init.py Re-exports weighted backend constant and backend capability helpers
src/gateway/models/routing.py Schema support for weights, validation rules, and WEIGHTED_BACKEND constant
src/gateway/api/routes/routing.py explain now includes weighted ordering and returns router_weights; pricing validation scoped to learned backend
src/gateway/api/routes/routing_memory.py Treats only learned backends as routing-memory consumers (excludes weighted)
src/gateway/cli.py CLI routing explain renders weighted shares and uses explain-time weighted ordering
src/gateway/services/pricing_init_service.py Startup pricing warnings scoped to backends that actually require pricing
docs/routing.md Documents weighted routing semantics, constraints, and how to inspect behavior
docs/dashboard.md Documents new weighted split control and behavior on the dashboard
docs/api-reference.md Notes router_weights in /v1/routing/policies/explain for weighted policies
docs/public/openapi.json Regenerated OpenAPI schema including router_weights and updated descriptions
src/gateway/AGENTS.md Updates internal routing architecture notes to include weighted backend
src/gateway/static/dashboard/index.html Updated dashboard bundle entrypoint hash
src/gateway/static/dashboard/assets/RoutingPage-Dqh6Cebs.js New compiled dashboard chunk containing weighted routing UI changes
src/gateway/static/dashboard/assets/RoutingPage-B0PbG8Zg.js Removed old compiled routing page chunk (replaced by new hash)
src/gateway/static/dashboard/assets/UsersPage-BUrrwJKv.js Rebuilt dashboard asset (hash/import updates)
src/gateway/static/dashboard/assets/ToolsGuardrailsPage-mzjN9zP-.js Rebuilt dashboard asset (hash/import updates)
src/gateway/static/dashboard/assets/TablePagination-CdsMK2Es.js Rebuilt dashboard asset (hash/import updates)
src/gateway/static/dashboard/assets/OverviewPage-BGv4NjCr.js Rebuilt dashboard asset (hash/import updates)
src/gateway/static/dashboard/assets/ModelScopeControl-BR6e2y7T.js Rebuilt dashboard asset (hash/import updates)
src/gateway/static/dashboard/assets/KeysPage-B_esvNYQ.js Rebuilt dashboard asset (hash/import updates)
src/gateway/static/dashboard/assets/ConfirmDialog-CP3FsMvH.js Rebuilt dashboard asset (hash/import updates)
Files not reviewed (1)
  • src/gateway/static/dashboard/assets/RoutingPage-Dqh6Cebs.js: Generated file
Suppressed comments (1)

web/src/pages/RoutingPage.tsx:153

  • routerBackendOf returns the raw router string, but backend resolution is case/whitespace-insensitive. Downstream UI logic (labels, summaries, Examples button gating) can mis-handle valid backends if the config/API stored a different casing or included whitespace.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread web/src/pages/RoutingPage.tsx
Comment thread web/src/pages/RoutingPage.tsx
@njbrake
njbrake deployed to integration-tests August 11, 2026 14:21 — with GitHub Actions Active

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@config.example.yml`:
- Line 142: Update the `on_failure` comment in the configuration example to
specify that it applies to an eligible provider failure occurring before any
response bytes are sent; preserve the existing description that it controls what
is tried next and in what order.

In `@docs/routing.md`:
- Around line 158-160: Qualify zero-share wording in both user-facing
descriptions: in docs/routing.md lines 158-160, change “serving none of the
traffic” to “receiving none of the weighted traffic”; in docs/dashboard.md lines
308-314, change “takes no traffic” to “takes no weighted traffic.”

In `@src/gateway/static/dashboard/assets/index-D0MBX8Ui.js`:
- Around line 1-2: Regenerate the dashboard entrypoint and all dependent chunks
from one shared source revision so their export bindings are consistent. At
src/gateway/static/dashboard/assets/index-D0MBX8Ui.js lines 1-2, rebuild the
shared runtime and commit every regenerated hashed asset; at
src/gateway/static/dashboard/assets/SettingsPage-BtxuYOZY.js line 1, restore the
y binding and align loading, pricing, and select imports; at
src/gateway/static/dashboard/assets/ToolsGuardrailsPage-mzjN9zP-.js line 1,
replace the undeclared O() reference and correct tool query and mutation
imports; at src/gateway/static/dashboard/assets/RoutingPage-Dqh6Cebs.js line 1,
bind $e to the tool-settings query rather than the pricing-delete mutation.
Smoke-test /settings, /tools, and /routing.

In `@src/gateway/static/dashboard/assets/RoutingPage-Dqh6Cebs.js`:
- Line 1: Update Ve() to use the tool-settings query export rather than the
pricing-delete mutation currently bound to $e. Rebuild the chunk against the
current shared export map or import the correct tool-settings hook, preserving
Ve()’s guardrails_url lookup and configured state behavior.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 114eee14-d94f-4c8b-b678-f817ed403ace

📥 Commits

Reviewing files that changed from the base of the PR and between 1ab8a4b and 49d5306.

⛔ Files ignored due to path filters (1)
  • docs/public/openapi.json is excluded by !docs/public/openapi.json
📒 Files selected for processing (38)
  • config.example.yml
  • docs/api-reference.md
  • docs/dashboard.md
  • docs/index.md
  • docs/routing.md
  • src/gateway/AGENTS.md
  • src/gateway/api/routes/routing.py
  • src/gateway/api/routes/routing_memory.py
  • src/gateway/cli.py
  • src/gateway/models/routing.py
  • src/gateway/services/pricing_init_service.py
  • src/gateway/services/routing/__init__.py
  • src/gateway/services/routing/backends.py
  • src/gateway/services/routing/decide.py
  • src/gateway/services/routing/weighted.py
  • src/gateway/static/dashboard/assets/ActivityPage-C9_HiWjD.js
  • src/gateway/static/dashboard/assets/BudgetsPage-BSua8jzY.js
  • src/gateway/static/dashboard/assets/ConfirmDialog-CP3FsMvH.js
  • src/gateway/static/dashboard/assets/DocsPage-CwwWDa-K.js
  • src/gateway/static/dashboard/assets/KeysPage-B_esvNYQ.js
  • src/gateway/static/dashboard/assets/ModelScopeControl-BR6e2y7T.js
  • src/gateway/static/dashboard/assets/ModelsPage-NwBGkmaA.js
  • src/gateway/static/dashboard/assets/OverviewPage-BGv4NjCr.js
  • src/gateway/static/dashboard/assets/ProvidersPage-DOyggcy8.js
  • src/gateway/static/dashboard/assets/RoutingPage-B0PbG8Zg.js
  • src/gateway/static/dashboard/assets/RoutingPage-Dqh6Cebs.js
  • src/gateway/static/dashboard/assets/SettingsPage-BtxuYOZY.js
  • src/gateway/static/dashboard/assets/TablePagination-CdsMK2Es.js
  • src/gateway/static/dashboard/assets/ToolsGuardrailsPage-mzjN9zP-.js
  • src/gateway/static/dashboard/assets/UsagePage-MIcf1ELP.js
  • src/gateway/static/dashboard/assets/UsersPage-BUrrwJKv.js
  • src/gateway/static/dashboard/assets/index-D0MBX8Ui.js
  • src/gateway/static/dashboard/index.html
  • tests/integration/test_routing_weighted.py
  • tests/unit/test_weighted_router.py
  • web/src/api/types.ts
  • web/src/pages/RoutingPage.test.tsx
  • web/src/pages/RoutingPage.tsx
💤 Files with no reviewable changes (1)
  • src/gateway/static/dashboard/assets/RoutingPage-B0PbG8Zg.js

Comment thread config.example.yml Outdated
Comment thread docs/routing.md
Comment thread src/gateway/static/dashboard/assets/index-D0MBX8Ui.js Outdated
Comment thread src/gateway/static/dashboard/assets/RoutingPage-Dqh6Cebs.js Outdated

@khaledosman khaledosman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gates pass on 49d5306: make lint (architecture + Ruff), make typecheck, openapi-check, postman-check, tests/unit/test_weighted_router.py, and web RoutingPage.test.tsx. The committed dashboard bundle carries the new RoutingPage/DocsPage output and is referenced from index-D0MBX8Ui.js.

Two inline: one confirmed regression for the noop placeholder backend, one cosmetic note on the share input.


🤖 Review by Claude Opus 5 via Claude Code

Comment thread src/gateway/services/routing/backends.py Outdated
Comment thread web/src/pages/RoutingPage.tsx Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The job was not started because the account is locked due to a billing issue.

@njbrake

njbrake commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

Note: this comment was drafted by Claude Opus 5 via back-and-forth with @njbrake. The reasoning and decisions are his; the prose is Claude's.

All eight threads addressed in 6879b14: six fixes, two disagreements. Resolving them all.

@khaledosman on noop. Fixed, and thank you for the before/after table, it made the regression unarguable. backend_learns is now backend_pool_is_teachable, false only for the weighted backend, so noop and a name from a newer build keep exactly the behavior they had on main. Test added covering both the status listing and the rank typo guard.

@khaledosman on the share field, and Copilot on non-finite input. Fixed together. The field now holds the typed text and parses once, so 7.5 survives the keystroke after the decimal point, a cleared field stays cleared instead of snapping to 0, and Infinity or a negative is refused in the form where the API refuses it rather than sailing through to a 400.

Copilot on backend-name normalization. Fixed. normalizedBackend trims and lowercases before comparing, so a policy spelled " Weighted ", which the gateway resolves happily, is editable and tagged Weighted instead of read-only and "Routed". Test added.

CodeRabbit on the zero-share wording. Taken, in both guides: a zero-weight candidate takes no weighted traffic, and still serves opt-out callers when it is also the default target.

CodeRabbit on on_failure in config.example.yml. Taken, reworded to say failover stops once part of a response has reached the caller.

CodeRabbit on the dashboard chunks (Critical and Major). Not reproducible; resolving as false positives. RoutingPage-Dqh6Cebs.js imports H as $e, and index export H is the tool-settings query (queryFn:()=>i("/v1/tool-settings")), which is what Ve() wants. Export $ is indeed the pricing-delete mutation, but nothing binds $e to it. O and y are both present in their own chunks' import lists. The bundle is the output of a single vite build, and npm --prefix web run build reproduces the committed tree byte for byte, so its cross-chunk bindings cannot be inconsistent by construction.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/gateway/static/dashboard/assets/index-DB9mxJVs.js`:
- Line 2: Update the session helpers at and around at(), ot(), and the
AuthProvider callbacks in ut() to serialize authentication requests with an
in-flight guard, ensuring logout completes before clearing state or allowing a
new login. Apply the existing Ee timeout through the same timeout mechanism used
by i() to both POST and DELETE session requests, and await the logout request
before showing the login screen.

In `@web/src/pages/RoutingPage.test.tsx`:
- Around line 550-596: The existing tests only verify display and editability,
not successful serialization. Extend the fractional-share test to open the
editor, save, and assert the POST payload preserves 7.5; extend the loosely
spelled backend test to edit and save, then assert the emitted weighted
specification uses the normalized backend path and expected weights.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: c5cc25cc-5b34-4ffe-ae0f-be1a4af150a4

📥 Commits

Reviewing files that changed from the base of the PR and between 49d5306 and 6879b14.

📒 Files selected for processing (26)
  • config.example.yml
  • docs/dashboard.md
  • docs/routing.md
  • src/gateway/api/routes/routing_memory.py
  • src/gateway/services/routing/__init__.py
  • src/gateway/services/routing/backends.py
  • src/gateway/static/dashboard/assets/ActivityPage-D6-YEyjz.js
  • src/gateway/static/dashboard/assets/BudgetsPage-DF21khAQ.js
  • src/gateway/static/dashboard/assets/ConfirmDialog-Bk4-ghkz.js
  • src/gateway/static/dashboard/assets/DocsPage-BIu60Z5q.js
  • src/gateway/static/dashboard/assets/KeysPage-CVbrPDbf.js
  • src/gateway/static/dashboard/assets/ModelScopeControl-DWhpSerS.js
  • src/gateway/static/dashboard/assets/ModelsPage-47hJWPEK.js
  • src/gateway/static/dashboard/assets/OverviewPage-kp3aG0sw.js
  • src/gateway/static/dashboard/assets/ProvidersPage-B5b0u3cx.js
  • src/gateway/static/dashboard/assets/RoutingPage-C5-iZGTv.js
  • src/gateway/static/dashboard/assets/SettingsPage-DUpymCp3.js
  • src/gateway/static/dashboard/assets/TablePagination-bQsnzK0q.js
  • src/gateway/static/dashboard/assets/ToolsGuardrailsPage-B0vzdnDJ.js
  • src/gateway/static/dashboard/assets/UsagePage-CWRc02TG.js
  • src/gateway/static/dashboard/assets/UsersPage-DykLjCx3.js
  • src/gateway/static/dashboard/assets/index-DB9mxJVs.js
  • src/gateway/static/dashboard/index.html
  • tests/unit/test_weighted_router.py
  • web/src/pages/RoutingPage.test.tsx
  • web/src/pages/RoutingPage.tsx
🚧 Files skipped from review as they are similar to previous changes (6)
  • config.example.yml
  • src/gateway/api/routes/routing_memory.py
  • docs/routing.md
  • docs/dashboard.md
  • src/gateway/services/routing/init.py
  • web/src/pages/RoutingPage.tsx

Comment thread src/gateway/static/dashboard/assets/index-DB9mxJVs.js Outdated
Comment thread web/src/pages/RoutingPage.test.tsx
@pss-julien

Copy link
Copy Markdown
Contributor

@njbrake apologies for closing the other PR, I ran into rebase/merge conflicts because other commits were touching the static asset files. I decided to start over and first remove the static assets from the Git repository in favor of compiling them at build time.

I'd suggest merging #556 before this one. I've already prepared a branch that will merge cleanly once #556 lands: https://github.com/pocketsizesun/otari/tree/feat/weighted-load-balancing

@njbrake

njbrake commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

@pss-julien Great, thank you! Yes no worries about this. I will take the PRs from here and get them over the finish line. Thanks for the great ideas and solid contributions. All the PRs have your cmmits in them so you'll get contribution credit for them 🙏

pss-julien and others added 6 commits August 11, 2026 16:33
A `select` entry can now name the `weighted` router, which draws one
candidate per request in proportion to declared weights, so traffic
splits across providers behind one policy name.

    select:
      - router: weighted
        candidates: [openai:gpt-5, anthropic:claude-sonnet-4-5]
        weights: {openai:gpt-5: 70, anthropic:claude-sonnet-4-5: 30}
      - default: openai:gpt-5

Design choices worth recording:

* Weights are normalized rather than percentages, so 7:3 and 70:30 are
  one split. A candidate left out of `weights` gets zero, which drains a
  provider without deleting it: it takes no traffic and stays at the tail
  of the plan as a failover target. An all-zero split is refused, since
  it could never select anything.
* Each request is an independent draw and nothing is remembered, so the
  split is as correct behind many replicas as behind one and none of the
  stickiness machinery in docs/routing-scaling.md is needed. The cost is
  that the ratio converges statistically.
* The draw continues without replacement, so the whole ordering is the
  plan: a retryable failure lands on another weighted provider before
  `on_failure` is reached, and a failing provider sheds its share with no
  health tracking.
* No pricing is required, unlike the kNN router which scores cost. The
  pricing gate is now scoped by `backend_requires_pricing`, so a weighted
  policy is not refused for missing rows it never reads.
* A weighted decision is not logged per request (it would be the log at
  load balancer volume); the usage row carries `router:weighted` and the
  draw is available at debug.
* `explain` and the CLI render the split itself, normalized over the
  candidates the caller may use, because a weighted decision needs no
  request state. The declared pool is still handed to the compiler so a
  filtered candidate keeps appearing in `dropped` with its reason.
* `Otari-Router: off` keeps its meaning and serves the default target,
  which pins a caller to one provider during an incident.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… literals

`_rank` takes `dict[str, float]`, and a dict bound to a variable before the
call is inferred as `dict[str, int]`, which is invariant and so not accepted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A pool with a repeated model was authorable in the policy form: the save
validation only checked that each row was non-empty. On a weighted policy the
duplicate then collapsed in the weight map, so the two rows the operator saw
became one key and the split submitted was not the split shown; the API refuses
a repeated candidate on any router entry, so the form was authoring a 400.
Uniqueness is now checked over the trimmed, non-empty rows, which leaves a pair
of still-empty rows reading as unfinished rather than as a duplicate.

Also name an unknown router backend neutrally. The chip and the row summary
called every non-weighted backend "Learned", which is true of knn and would be a
guess about the first backend added after that line was written; such a policy
is already shown read-only, so "Routed" is what the table can honestly say.

Adds a unit test for a drained pool of two, which pins a deliberate divergence:
declared_shares reports an even split because it has no order to express, while
weighted_ordering keeps declared order so the last resort stays predictable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five review items on the weighted router.

`declared_shares` returned an even split for an all-zero pool while
`weighted_ordering` kept declared order, so `explain`, `router_weights`,
the CLI and `confidence` all reported 50/50 for a plan that sent every
request to the first candidate. Reachable when a caller's allow-list
filters out every positively weighted candidate. The reported split now
follows the head that actually serves.

The routing memory surfaces treated a weighted policy as a learned one,
because both name a router. `GET /v1/routing/status` listed it under a
warmth report it never reads, and worse, `POST /v1/routing/preferences/rank`
let its pool decide which score keys a user may teach, so a workspace whose
only router policy was weighted could no longer seed examples for the
learned policy it was preparing. Both now gate on `backend_learns`.

"Retryable failure" is stale phrasing: #546 made the walker fall back on
every pre-response failure and rewrote the docs accordingly. Restored in
the new prose, the CLI output and the test comments.

The allow-list `explain` transcript omitted the `on_failure` candidate,
which is filtered by the same rule, so the example showed one drop where
the command prints two.

Also rebased onto main, rebuilt the dashboard bundle, regenerated the
OpenAPI spec, and named `weighted.py` in the gateway's backend map.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review findings on #553.

When a caller's allow-list filtered out every candidate in a weighted
split, `explain_router_ordering` returned nothing, so the compiler fell
through to the failure chain and the split disappeared from the plan and
from `dropped` alike. The command then printed the learned router's
decline text, which talks about cold pools and low confidence on a policy
that has neither. It now returns the declared ordering with empty shares,
so every filtered candidate is named with its reason, and the CLI keys its
closing note on the backend rather than on whether any share survived.

The pricing claim was too broad: the weighted router reads no price, but
`require_pricing` is on by default, so a metered caller drawn onto an
unpriced candidate still gets a 402. Narrowed in the docs and in the
module docstring, which the same page already contradicted.

Also: the docs said the dashboard renders the explain endpoint's
`router_weights`, which it does not (the page computes the declared split
client-side); the candidate cap rule still said "for a learned policy";
`docs/index.md` did not mention the weighted split; and
`config.example.yml` still said "retryable failure", stale since #546.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… share field

Review findings from @khaledosman and Copilot on #553.

`backend_learns` narrowed the routing memory surfaces to the kNN backend,
which also dropped `noop`. That backend exists to hold a policy's shape
while its pool is being taught, so its candidates are the very ones an
operator is seeding: `GET /v1/routing/status` stopped reporting the pool's
warmth and `POST /v1/routing/preferences/rank` started refusing the
examples meant for it, which is the failure this change was made to
prevent, landing on the workflow the backend exists for. Replaced with
`backend_pool_is_teachable`, false only for the weighted backend, so
`noop` and a name from a newer build keep the behavior they had.

The dashboard compared a backend name without normalizing it, so a policy
spelled `" Weighted "`, which the gateway resolves happily, was shown
read-only and labelled wrong.

The share field parsed on every keystroke, so a decimal point was
swallowed as it was typed, a cleared field snapped to 0, and "Infinity"
passed the form's check to be refused by the API. The field now holds what
was typed and parses once, with non-finite and negative values refused
where the API refuses them.

Also qualified the zero-share wording in both user guides: a zero-weight
candidate takes no *weighted* traffic, and still serves opt-out callers
when it is the default target. Same for the `on_failure` line in
config.example.yml, which did not say failover stops once a response has
started.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@njbrake
njbrake force-pushed the feat/weighted-routing branch from 6879b14 to dd1fd57 Compare August 11, 2026 16:36
@njbrake
njbrake deployed to integration-tests August 11, 2026 16:36 — with GitHub Actions Active
…t the form

Both new cases stopped at what the form displayed. The fractional-share test
now saves and asserts the POST body carries 7.5, and the loosely spelled
backend test now saves and asserts the emitted entry, which also pins that the
spelling is normalized on the way out. Confirmed both fail against the
pre-fix behavior.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@njbrake
njbrake deployed to integration-tests August 11, 2026 17:00 — with GitHub Actions Active
@njbrake

njbrake commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

Note: this comment was drafted by Claude Opus 5 via back-and-forth with @njbrake. The reasoning and decisions are his; the prose is Claude's.

Two more threads from the latest CodeRabbit pass, both now closed out in 31a161c.

Edge-case save paths (RoutingPage.test.tsx). Fair, taken. Both cases stopped at what the form displayed. The fractional test now saves and asserts the POST body carries 7.5, and the loosely spelled backend test now saves and asserts the emitted entry, which additionally pins that the spelling is normalized on the way out. I checked both against the pre-fix behavior: with normalizedBackend returning the raw name and the save path truncating, exactly these two fail, so they are regression tests rather than restatements.

Session request serialization. Out of scope here, so resolving without a change. The logout race is real (AuthContext.logout fires void deleteSession() and does not await it, so a fast re-login can post before the delete lands), but web/src/api/client.ts is byte-identical to main and untouched by this PR, which adds a routing backend. The anchor is also a dashboard bundle chunk, and #556 stopped tracking the built bundle, so that file no longer exists in the repo. Worth its own issue against the auth path.

@njbrake
njbrake merged commit 709c794 into main Aug 11, 2026
13 checks passed
@njbrake
njbrake deleted the feat/weighted-routing branch August 11, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants