Fix multi-byte UTF-8 corruption in streaming Ask responses - #228
Open
surgaev wants to merge 1 commit into
Open
Conversation
_processStream() had two related bugs in how it consumed the SSE
response stream:
1. decoder.decode(value) was called without { stream: true }. When a
multi-byte UTF-8 character (e.g. Cyrillic, or any non-ASCII text)
happened to be split across two network chunks, TextDecoder would
mangle or drop the incomplete sequence instead of buffering it,
since it wasn't told the stream was still ongoing.
2. Each chunk was decoded and split on '\n' independently
(chunk.split('\n')), with no buffering across reader.read() calls.
If a single `data: {...}` SSE line was split across two chunks,
the JSON.parse in the following try/catch would throw and the
whole token for that line was silently discarded — the catch
block was empty, so this failure mode left no trace in the logs.
Both are now fixed: decode with { stream: true } so partial UTF-8
sequences are held over instead of corrupted, and lineBuffer carries
over any trailing incomplete line to be prefixed onto the next chunk
before splitting/parsing. Reproducible by asking a question in a
non-ASCII language (e.g. Russian) and getting back responses with
missing letters or whole word fragments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
askService.js's_processStreamreads the streaming response body chunk-by-chunk and callschunk.split('\n')directly on each decoded chunk to find SSE lines.Two related bugs:
TextDecoder.decode()was called without{ stream: true }, so a multi-byte UTF-8 character (e.g. Cyrillic, CJK, emoji) split across two network chunk boundaries gets decoded incorrectly — each half is decoded independently instead of being buffered until a full character is available, producing garbled/replacement characters (�) in streamed output.data: ...line isn't guaranteed to arrive in one chunk. Splitting each chunk on\nindependently (instead of maintaining a persistent line buffer across chunks) can silently drop or merge lines whenever a line boundary doesn't line up with a chunk boundary.Users streaming Ask responses in any non-ASCII-heavy language (Russian, Chinese, Japanese, etc.) intermittently see corrupted characters mid-response, and occasionally a line gets dropped entirely.
Fix
lineBufferthat accumulates decoded text across chunks and only processes complete lines (splitting on the last\n), carrying any trailing partial line over to the next chunk.decoder.decode(value)todecoder.decode(value, { stream: true }), which tellsTextDecoderto hold back a trailing incomplete multi-byte sequence and prepend it to the nextdecode()call instead of emitting a corrupted character immediately.Testing
Verified locally with a Russian-language conversation (
OPENAI_TRANSCRIBE_LANG=ru/ Russian prompts) streamed through Ask: previously-frequent�corruption in streamed responses no longer occurs, and long responses no longer show dropped/merged lines.