fix(openai): accumulate streamed refusal and reasoning_content - #3797
Open
CodeWithMoin wants to merge 1 commit into
Open
CodeWithMoin wants to merge 1 commit into
CodeWithMoin wants to merge 1 commit into
Conversation
Only content, function_call and tool_calls were declared as accumulated fields on a streamed chat completion message. refusal and reasoning_content fell through to the new-entry/replacement path, so each chunk overwrote the last and output.value kept only the final fragment. Declare both as string accumulators, like content. Fixes Arize-ai#3785
Contributor
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
Author
|
recheck |
Author
|
I have read the CLA Document and I hereby sign the CLA |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3785
What changed
On a streamed chat completion,
refusalandreasoning_contentare now accumulated across chunks, same ascontent. One change in_response_accumulator.py: both are declared as_StringAccumulator()in themessageschema.Why
Only
content,function_callandtool_callswere declared there. Any other string field on the delta went through the new-entry path on the first chunk and the replacement path on every chunk after, sooutput.valuekept only the last fragment.refusalis a first-party delta field, andreasoning_contentis what DeepSeek, vLLM and Qwen3 stream through abase_urloverride. So a streamed trace showed a plausible but wrong value, and didn't match the same call withstream=False.I kept it to these two fields rather than accumulating every undeclared string. Scalar fields like
rolecan repeat across chunks and would get concatenated.Trace output
output.valuemessage for the same four streamed chunks (tworeasoning_content, twocontent):Tests
New
tests/openinference/instrumentation/openai/test_response_accumulator.py:content,refusalandreasoning_contentstreamed over three chunks each come out whole, androlestays"assistant"On
maintherefusalandreasoning_contentcases fail andcontentpasses. With the fix all 4 pass.The one failure is
test_tool_calls. It fails the same way onmainin a full run and passes alone, so it looks like test-order state and isn't related. mypy is clean on the changed files.