Repository navigation
fix(chat): close the source stream when an adapter stops reading early - #1172
Open
abhiramshaji-cyber wants to merge 1 commit into
Open
abhiramshaji-cyber wants to merge 1 commit into
abhiramshaji-cyber wants to merge 1 commit into
Conversation
The wrappers that handleStream passes to adapter.stream() and to the post and edit fallback were hand built iterators with next() but no return(). Leaving the loop early never reached the finally in takeUntilAborted, so the source generator stayed open. Both wrappers are now async generators, which forward return() and throw() to the source. Fixes vercel#1170 Signed-off-by: Abhiram Shaji <write4abhiram@gmail.com>
Contributor
|
@abhiramshaji-cyber is attempting to deploy a commit to the Vercel Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When an adapter stops reading a streamed reply early, the model stream behind
thread.post()now gets closed instead of running on with nobody reading it.Fixes #1170. Hi @r13i, hope you're doing well! Thank you so much for this report. The repro was spot on and made it easy to confirm, and your suggested
return()fix pointed straight at the problem. I went with a slightly different shape (below), so please tell me if you'd rather see your version.What this code is for
handleStream()inpackages/chat/src/thread.tsruns when someone passes an async iterable tothread.post(), for example an AI SDKtextStream. It feeds the stream throughfromFullStream,withoutReasoningandtakeUntilAborted, then hands it to the adapter'sstream(), or tofallbackStream()(post, then edit) when the adapter has none.Before this PR
An SMS adapter's
stream()reads the reply and throws"too long"once it passes the length limit.thread.post(model()).handleStream()passes the adapterwrappedStream, a plain object whose iterator hasnext()and nothing else.for awaitthrows, so JavaScript looks forreturn()on that iterator to close it.return(), so nothing is passed on totakeUntilAborted.takeUntilAbortedstays paused at itsyield, and itsfinally, the code that callsiterator.return()on the source, never runs.model()stays open, and in the repro from the issuecleanedUpstaysfalse.textOnlyStreamhas the same gap, so a failedpostMessageinsidefallbackStream()leaves the source open too.After this PR
Same adapter, same
thread.post(model()).handleStream()passes the adaptertapText(textStream, ...), an async generator.for awaitthrows, and JavaScript callsreturn()on thetapTextgenerator.for awaitinsidetapText, which callsreturn()ontakeUntilAborted.takeUntilAbortedruns itsfinallyand callsiterator.return()onmodel().cleanedUpistrue.tapTextpasses every chunk through and fillsaccumulatedthe same way the old wrapper did.The diagram
flowchart LR classDef box fill:#eef1ff,stroke:#4c5bd4,color:#111827 classDef bad fill:#ffd6d6,stroke:#cc0000,color:#111827,stroke-width:3px classDef good fill:#c8f0c8,stroke:#008800,color:#111827,stroke-width:3px subgraph BEFORE["BEFORE: cleanedUp stays false"] direction TB B1["adapter stream() leaves for await early"] --> B2{"wrappedStream has only next()"} B2 -->|"return() not found"| B3["takeUntilAborted stays paused at yield"] B3 --> B4["model() stays open"] end subgraph AFTER["AFTER: cleanedUp is true"] direction TB A1["adapter stream() leaves for await early"] --> A2{"THE CHANGE: tapText async generator"} A2 -->|"return() passed on"| A3["takeUntilAborted finally runs"] A3 --> A4["model() gets return()"] A2 -->|"adapter reads to the end"| A5["accumulated text posted as before"] end B1 ~~~ A1 class B1 box class A1,A5 box class B2,B3,B4 bad class A2,A3,A4 good linkStyle 1 stroke:#cc0000,stroke-width:5px linkStyle 4 stroke:#008800,stroke-width:5px style BEFORE fill:#fff5f5,stroke:#cc0000,color:#111827 style AFTER fill:#f3fff3,stroke:#008800,color:#111827thread.posttohandleStreamand everything it calls, Functions view in ArchitectThe change
Both hand built iterators are replaced by small async generators next to
withoutReasoning:tapText()for the adapter path andtextOnly()for the fallback, sharing achunkText()that returns the text of a string ormarkdown_textchunk. An async generator'sfor awaitcallsreturn()on its source by itself when the loop exits early, the same waywithoutReasoningalready does, so thefinallyintakeUntilAbortednow runs on every early exit. That is the whole fix; nothing else inhandleStream()moved.What changes for a user
break, a thrown error, or a failed fallback post now closes the source stream, so the model request stops.accumulatedtext, same skipping oftask_update,plan_updateandreasoningin the fallback.takeUntilAborted, and a source whose iterator has noreturn()still stops cleanly.null, the fallback now gets a closed stream instead of the rest. No adapter in this repo does that (Slack and Teams returnnullbefore reading), and the chunks it read were already lost before this change.One question, if you have a moment: an adapter that throws before reading a single chunk still leaves the source open, because
takeUntilAbortednever started and has nofinallyto run yet. Would you likehandleStream()to close the raw stream in that case too, or is reading at least once the case you hit in practice? Any feedback or tips are very welcome!Test plan
New
source stream cleanuptests inpackages/chat/src/thread.test.tscover a throw mid loop, an earlybreak(checking the posted text too), a failed fallback post, an abort, and a source iterator with noreturn(). The first three fail onmainand pass here.Checklist
git commit -s)pnpm validatepasses