Conversation
`_Stream` finished the span only when iteration raised StopIteration / StopAsyncIteration, or when an error propagated out of `__next__`. A caller that leaves the `with` block early, calls `close()` / `aclose()`, or drops the stream object never reaches those paths, so the span stayed open, was never exported, and the whole LLM call disappeared from the trace. End the tracing from `__exit__`, `__aexit__`, `close`, `aclose` and `__del__`, leaving the status UNSET so a truncated stream stays distinguishable from a completed one -- the same shape `openinference-instrumentation-ollama` already uses for its streams. `_WithSpan.finish_tracing` returns early once the span is finished, so these calls are no-ops after a normal completion.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happens
A streamed chat completion that the caller stops consuming early is missing from the trace
entirely.
_Streamends the LLM span only when iteration raisesStopIteration/StopAsyncIteration, or when an error propagates out of__next__. The ordinary ways to stopearly — leaving the
withblock, callingclose()/aclose(), or dropping the object — handcontrol back without ever finishing tracing. An unended span is never exported, so a call that
really happened, really cost tokens and may have already streamed a partial answer to the user
leaves no record.
langchain_openaiis a live consumer of the first path: it wraps streaming inwithblocks andbreaks out early.
Exported spans for that snippet
CONTRIBUTINGasks for trace output when a change affects emitted spans. This is reproducedoffline against a stubbed HTTP backend, so the evidence below is the exported span rendered from an
InMemorySpanExporterrather than a Phoenix screenshot; one script, same environment, both trees:Root cause
In
python/instrumentation/openinference-instrumentation-openai/src/openinference/instrumentation/openai/_stream.py,__exit__and__aexit__call the wrapped teardown and return.close()is not overridden, soObjectProxyforwards it straight to the SDK object, and on openai >= 3.xAsyncStream.aclose()calls the wrapped
close(), which bypasses the wrapper too. None of those paths reaches_finish_tracing.Change
Finish the tracing on every teardown path, leaving the status
UNSETso a truncated stream staysdistinguishable from a completed one (
OK) or a failed one (ERROR, already recorded by__next__):__exit__/__aexit__:try/finally, so the span still ends if the wrapped teardown raises.close():Stream.close()is synchronous whileAsyncStream.close()is a coroutine; theawaited case finishes once the wrapped close completes.
aclose(): covers the SDK method that bypassesclose()on the proxy.__del__: last resort for a stream dropped without any explicit teardown; it suppressesexceptions so finalization can never raise into user code.
close/aclose/__del__use thegetattr(...) + callable()guard and afinally-basedfinish taken from
openinference-instrumentation-ollama/src/openinference/instrumentation/ollama/_stream.py,which already ends abandoned streams with status
UNSETand has a committed test for it(
test_chat_stream_abandoned_before_iteration). For contrast:togetherandcoherefinish on__exit__withOK, andanthropicandmistralaihave no teardown handling. This PR followsthe
ollamaconvention, becauseUNSETkeeps "the caller stopped early" distinguishable from"the answer arrived".
_WithSpan.finish_tracingin_with_span.pyreturns as soon as its_is_finishedflag is set, sothe added calls are no-ops after a normal completion: no span is ended twice, and a completed
stream keeps its
OKstatus. The last test in the new file pins that.Tests
tests/openinference/instrumentation/openai/test_stream_lifecycle.pycovers the five paths plus acontrol asserting that a fully consumed stream still yields exactly one
OKspan. No network and nocredentials: the SSE body is served through
respx, which the package'stest-requirements.txtalready pins.
a719562e)mainpytest tests/openinference/instrumentation/openai/test_stream_lifecycle.pywithopenai==2.8.0(theci-openaifactor)openai(3.16.2, theopenai-latestfactor)pytest tests(whole package, pinned deps)ruff format --diff ./ruff check --no-fix .(0.9.2, the pinned version)mypy .(1.11.2, strict, as tox runs it)The
aclosecase is skipped underopenai==2.8.0because that release has noAsyncStream.aclose; it runs onopenai-latest.Each hunk carries its weight. Reverting one at a time (latest
openai, same file) fails exactly thetests that claim it:
__exit__back to a plain delegatetest_leaving_context_without_exhausting_ends_one_span[False],[True]__aexit__back to a plain delegatetest_leaving_context_without_exhausting_ends_one_span[True],test_closing_without_exhausting_ends_one_span[True]close()removedtest_closing_without_exhausting_ends_one_span[False],[True],test_aclosing_without_exhausting_ends_one_span,test_stream_never_iterated_ends_one_spanaclose()removedtest_aclosing_without_exhausting_ends_one_span,test_stream_never_iterated_ends_one_span__del__removedtest_stream_never_iterated_ends_one_spanEvery test holds a reference to the stream until after its own assertions, so a span can only be
finished by the path the test exercises and never by a later
__del__during collection garbage.Environment: macOS arm64, Python 3.11.15; dependencies installed from the package's own
test-requirements.txtfor the pinned column anduv pip install -U openaifor the latest column.A whole-package failure that predates this change
test_tool_calls.py::test_tool_callsfails in a whole-package run on unmodifiedmainwith thesame command and dependency set (
1 failed, 484 passed, measured with no new test file present inthe tree), passes when the file is run alone, and shows the same
assert 2 == 1attest_tool_calls.py:107on this branch. It looks order-dependent through the session-scopedexporter. I did not investigate it and this PR does not change it; the numbers above are reported
as measured so the two are not confused.
Scope
Closes #3790. That issue asked two questions. On status: this uses
UNSET, reservingERRORforexceptions raised through iteration. On the
__del__fallback: included, because theollamainstrumentor already ships the same fallback in
mainwith a committed test, so this makes the twopackages behave alike rather than adding a new idea — if you would rather keep this PR to the
explicit teardown paths, say so and I will drop that method and its test.
#3784 is the same class of defect in the cohere instrumentor and is not addressed here.