Skip to content

fix: stop response_format/stop leaking into the ranked-choice judge call - #163

Open
shrdgn wants to merge 1 commit into
mainfrom
claude/ranked-response-format-leak
Open

fix: stop response_format/stop leaking into the ranked-choice judge call#163
shrdgn wants to merge 1 commit into
mainfrom
claude/ranked-response-format-leak

Conversation

@shrdgn

@shrdgn shrdgn commented Aug 28, 2026

Copy link
Copy Markdown
Owner

What & why

ranked.pick_best's internal classifier call (asked to reply with "ONLY the number" of the best panel answer, max_tokens=8) copies the client's request body and only strips tool-calling fields via strip_tool_fields.

That's inconsistent with synthesize.py's judge call, where response_format/stop are intentionally forwarded because the judge's own generated text is the visible answer (documented in README.md and DESIGN.md: "max_tokens, stop, response_format apply to the judge (visible output)"). In ranked.py, though, the classifier's own text is never returned to the client — only the chosen panel response (responses[index].content) is.

If a client sets response_format: {"type": "json_object"} or a stop sequence on the request, it now leaks into this internal ranking call. That can break _parse_choice (e.g. the classifier tries to emit JSON instead of a bare number) or make the upstream reject the call outright — silently defaulting the ranked pick to index 0 instead of the actual best answer.

Fix: pop response_format and stop from the copied body before sending the ranking call, alongside the existing strip_tool_fields.

How it was tested

$ uv run pytest tests/test_ranked.py -q
19 passed in 0.48s
$ uv run ruff check openfusion/ranked.py tests/test_ranked.py
All checks passed!
$ uv run mypy openfusion/ranked.py
Success: no issues found in 1 source file
  • ruff check . passes
  • pytest -q passes (no live network)
  • New behavior has a test (test_pick_best_strips_response_format_and_stop, asserts the upstream mock never receives either field)
  • Docs updated if config / request surface / defaults changed — n/a, no surface change, this restores documented behavior
  • No secrets, prompts, or response bodies added to logs or metrics
  • Quality/cost claims backed by a reproducible bench/run.py number (if applicable) — n/a, correctness fix

Notes for reviewers

Small, isolated change — two lines in ranked.py plus a regression test. Found via an automated repo-review pass cross-checking ranked.py against synthesize.py's (correct, documented) handling of the same fields.


Generated by Claude Code

ranked.pick_best's internal classifier call ("reply with ONLY the number")
copies the client's request body and only strips tool-calling fields.
Unlike synthesize.py's judge call — where response_format/stop legitimately
apply because the judge's own text IS the visible answer — this call's text
is never returned to the client; only the chosen panel response is. A
client-set response_format or stop sequence can break _parse_choice or make
the upstream reject the call, silently defaulting the ranked pick to index 0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants