Problem
Newer models across providers no longer accept arbitrary sampling parameters, but the clients forward them unconditionally, causing hard 400s:
- Anthropic — claude-opus-4-7, claude-opus-4-8, claude-sonnet-5 reject temperature entirely:
400 invalid_request_error: temperature is deprecated for this model.
- OpenAI — gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.1, gpt-5.2, gpt-5.5, o3 only accept the default temperature=1; any other value 400s.
On the Anthropic structured-output path the failure also cascades: it falls back to text mode and exhausts the retry loop before erroring out.
Current workaround
Each caller patches params per-model before calling the client (benchmark_base.py):
if self.model in ["gpt-5.5-2026-04-23","gpt-5","gpt-5-mini","gpt-5-nano","gpt-5.1-2025-11-13","gpt-5.2","o3"]:
self.temperature = 1
if self.model in ["claude-opus-4-7", "claude-opus-4-8", "claude-sonnet-5"]:
self.temperature = None
This list must be edited for every new model and is duplicated by every consumer of the library — it belongs in the client.
Proposed fix
Centralize sampling-param compatibility in the client layer. Two complementary approaches:
- Per-model capability metadata — declare which models drop or pin sampling params, and normalize (temperature/top_p/top_k) accordingly before the request.
- Catch-and-retry — on the specific 400 (temperature is deprecated / unsupported value), strip or reset the offending param and retry once.
Prefer (2) as the general safety net (no per-model list to maintain) with (1) for known cases to avoid the wasted round-trip. Apply consistently across claude_client.py and openai_client.py (and any
other backends that forward sampling params).
Affected
- ai_client/claude_client.py — _do_prompt (optional_params handling)
- ai_client/openai_client.py — equivalent sampling-param handling
Problem
Newer models across providers no longer accept arbitrary sampling parameters, but the clients forward them unconditionally, causing hard 400s:
400 invalid_request_error:
temperatureis deprecated for this model.On the Anthropic structured-output path the failure also cascades: it falls back to text mode and exhausts the retry loop before erroring out.
Current workaround
Each caller patches params per-model before calling the client (benchmark_base.py):
if self.model in ["gpt-5.5-2026-04-23","gpt-5","gpt-5-mini","gpt-5-nano","gpt-5.1-2025-11-13","gpt-5.2","o3"]:
self.temperature = 1
if self.model in ["claude-opus-4-7", "claude-opus-4-8", "claude-sonnet-5"]:
self.temperature = None
This list must be edited for every new model and is duplicated by every consumer of the library — it belongs in the client.
Proposed fix
Centralize sampling-param compatibility in the client layer. Two complementary approaches:
Prefer (2) as the general safety net (no per-model list to maintain) with (1) for known cases to avoid the wasted round-trip. Apply consistently across claude_client.py and openai_client.py (and any
other backends that forward sampling params).
Affected