fix: stop response_format/stop leaking into the ranked-choice judge call - #163
Open
shrdgn wants to merge 1 commit into
Open
fix: stop response_format/stop leaking into the ranked-choice judge call#163shrdgn wants to merge 1 commit into
shrdgn wants to merge 1 commit into
Conversation
ranked.pick_best's internal classifier call ("reply with ONLY the number")
copies the client's request body and only strips tool-calling fields.
Unlike synthesize.py's judge call — where response_format/stop legitimately
apply because the judge's own text IS the visible answer — this call's text
is never returned to the client; only the chosen panel response is. A
client-set response_format or stop sequence can break _parse_choice or make
the upstream reject the call, silently defaulting the ranked pick to index 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
ranked.pick_best's internal classifier call (asked to reply with "ONLY the number" of the best panel answer,max_tokens=8) copies the client's request body and only strips tool-calling fields viastrip_tool_fields.That's inconsistent with
synthesize.py's judge call, whereresponse_format/stopare intentionally forwarded because the judge's own generated text is the visible answer (documented in README.md and DESIGN.md: "max_tokens,stop,response_formatapply to the judge (visible output)"). Inranked.py, though, the classifier's own text is never returned to the client — only the chosen panel response (responses[index].content) is.If a client sets
response_format: {"type": "json_object"}or astopsequence on the request, it now leaks into this internal ranking call. That can break_parse_choice(e.g. the classifier tries to emit JSON instead of a bare number) or make the upstream reject the call outright — silently defaulting the ranked pick to index 0 instead of the actual best answer.Fix: pop
response_formatandstopfrom the copied body before sending the ranking call, alongside the existingstrip_tool_fields.How it was tested
ruff check .passespytest -qpasses (no live network)test_pick_best_strips_response_format_and_stop, asserts the upstream mock never receives either field)bench/run.pynumber (if applicable) — n/a, correctness fixNotes for reviewers
Small, isolated change — two lines in
ranked.pyplus a regression test. Found via an automated repo-review pass cross-checkingranked.pyagainstsynthesize.py's (correct, documented) handling of the same fields.Generated by Claude Code