Skip to content

Client should handle models that reject or restrict sampling params (temperature) #9

Description

@MHindermann

Problem

Newer models across providers no longer accept arbitrary sampling parameters, but the clients forward them unconditionally, causing hard 400s:

  • Anthropic — claude-opus-4-7, claude-opus-4-8, claude-sonnet-5 reject temperature entirely:
    400 invalid_request_error: temperature is deprecated for this model.
  • OpenAI — gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.1, gpt-5.2, gpt-5.5, o3 only accept the default temperature=1; any other value 400s.

On the Anthropic structured-output path the failure also cascades: it falls back to text mode and exhausts the retry loop before erroring out.

Current workaround

Each caller patches params per-model before calling the client (benchmark_base.py):

if self.model in ["gpt-5.5-2026-04-23","gpt-5","gpt-5-mini","gpt-5-nano","gpt-5.1-2025-11-13","gpt-5.2","o3"]:
self.temperature = 1
if self.model in ["claude-opus-4-7", "claude-opus-4-8", "claude-sonnet-5"]:
self.temperature = None

This list must be edited for every new model and is duplicated by every consumer of the library — it belongs in the client.

Proposed fix

Centralize sampling-param compatibility in the client layer. Two complementary approaches:

  1. Per-model capability metadata — declare which models drop or pin sampling params, and normalize (temperature/top_p/top_k) accordingly before the request.
  2. Catch-and-retry — on the specific 400 (temperature is deprecated / unsupported value), strip or reset the offending param and retry once.

Prefer (2) as the general safety net (no per-model list to maintain) with (1) for known cases to avoid the wasted round-trip. Apply consistently across claude_client.py and openai_client.py (and any
other backends that forward sampling params).

Affected

  • ai_client/claude_client.py — _do_prompt (optional_params handling)
  • ai_client/openai_client.py — equivalent sampling-param handling

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions