feat: add model routing and composer compatibility#3
Open
kksp993 wants to merge 2 commits into
Open
Conversation
Allow operators to restrict selected models to explicit credential allowlists. This keeps subscription-gated model traffic on compatible credentials while preserving the existing load balancer for unmatched models.
Persist a stable Grok agent identifier and route composer responses requests through a CLI-compatible chat completions adapter. This matches the official client wire shape for composer models while keeping the public Responses API behavior intact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hi, thank you for maintaining this project. While using grokbuild-proxy with multiple credentials, I ran into two practical compatibility gaps that seem useful to support in the project itself:
grok-composer-2.5-fastappears to be more reliable when requests match the official Grok CLI chat-completions wire shape.This PR adds two opt-in building blocks to improve those cases while preserving existing behavior by default.
Current behavior / limitation
1. Credentials cannot be restricted per model
At the moment, model requests are selected through the normal credential load balancer. That works well when all credentials have the same capabilities, but it can be problematic when different credentials have different model access.
For example, if only a subset of credentials can access a subscription-gated model, requests for that model may be routed to an incompatible credential and fail, even though a compatible credential exists in the pool.
2. Composer model compatibility
For
grok-composer-2.5-fast, upstream behavior appears to differ depending on the request shape. The official Grok CLI sends composer generation requests through/chat/completionswith Grok CLI-style headers and a stable agent identifier.This PR keeps the public proxy API behavior intact, but adapts composer requests internally to better match that official client shape.
What this PR changes
Commit 1: model-based credential routing
Adds optional config-driven model routing:
Behavior:
This avoids sending subscription-gated model traffic to incompatible credentials.
Commit 2: Grok CLI-compatible composer adapter
Adds:
x-grok-agent-idsupport./chat/completions,This keeps downstream compatibility while matching the upstream request shape that composer models expect.
Backward compatibility
This should be backward compatible:
model_routingis disabled by default./v1/responsescallers can continue using the same public API.Test plan
Added/updated tests for:
Tested locally with:
go test ./cmd/grokbuild-proxy ./internal/config ./internal/proxy ./internal/upstreamNotes
I tried to keep this PR configuration-driven and avoid introducing credential schema changes. The goal is to give operators a safe way to separate credentials by model capability without changing existing import flows or default runtime behavior.