What problem does this solve?
Choosing which LLM to use in production is typically based on benchmarks or intuition, not on data from your own traffic. There is currently no way to run a controlled experiment in Routerly that routes a fraction of real requests to a candidate model and compares its outcomes (latency, cost, quality) against the control — without modifying client code.
Proposed solution
Add an experiment routing policy that splits traffic between two or more models according to configured weights, tracks results per arm, and reports aggregate metrics.
Config:
{
"routing": {
"policy": "experiment",
"experiment": {
"id": "gpt4o-vs-sonnet-june",
"arms": [
{ "modelId": "gpt-4o", "weight": 80 },
{ "modelId": "claude-sonnet-4-5", "weight": 20 }
],
"trackMetrics": ["latency", "cost", "error_rate", "output_tokens"]
}
}
}Behaviour:
- Each incoming request is deterministically assigned to an arm based on a hash of a stable identifier (session ID, user ID, or request ID)
- The assignment is recorded on the usage record (
experimentId, arm)
- If the assigned model fails, fallback behaviour follows the project's normal fallback config (not the experiment arm)
Dashboard:
- Experiments page: list active and completed experiments
- Experiment detail: per-arm metrics table — request count, p50/p95 latency, avg cost/request, error rate, avg output tokens
- Traffic allocation gauge: live split vs configured split (detects if one arm is error-ing out)
- Stop/pause/resume controls
Alternatives you've considered
Running experiments at the application layer by manually alternating model names in client code. Works but requires code changes, bypasses Routerly's budget/auth logic, and produces no automatic comparison data.
Who would benefit from this?
Any team that wants to evaluate a new model or provider in production before committing to it. Particularly useful when migrating between model generations or evaluating cost/quality tradeoffs.
Additional context
TensorZero and Portkey both offer model A/B testing as a gateway-level feature. The key value is that experiments are configured in the gateway, not in application code, making them available to any caller without code changes.
What problem does this solve?
Choosing which LLM to use in production is typically based on benchmarks or intuition, not on data from your own traffic. There is currently no way to run a controlled experiment in Routerly that routes a fraction of real requests to a candidate model and compares its outcomes (latency, cost, quality) against the control — without modifying client code.
Proposed solution
Add an
experimentrouting policy that splits traffic between two or more models according to configured weights, tracks results per arm, and reports aggregate metrics.Config:
{ "routing": { "policy": "experiment", "experiment": { "id": "gpt4o-vs-sonnet-june", "arms": [ { "modelId": "gpt-4o", "weight": 80 }, { "modelId": "claude-sonnet-4-5", "weight": 20 } ], "trackMetrics": ["latency", "cost", "error_rate", "output_tokens"] } } }Behaviour:
experimentId,arm)Dashboard:
Alternatives you've considered
Running experiments at the application layer by manually alternating model names in client code. Works but requires code changes, bypasses Routerly's budget/auth logic, and produces no automatic comparison data.
Who would benefit from this?
Any team that wants to evaluate a new model or provider in production before committing to it. Particularly useful when migrating between model generations or evaluating cost/quality tradeoffs.
Additional context
TensorZero and Portkey both offer model A/B testing as a gateway-level feature. The key value is that experiments are configured in the gateway, not in application code, making them available to any caller without code changes.