Skip to content

fix(openai): prime Images streams before upstream first byte#2349

Open
jiwangyihao wants to merge 1 commit into
Wei-Shaw:mainfrom
jiwangyihao:fix/openai-images-stream-first-byte-upstream
Open

fix(openai): prime Images streams before upstream first byte#2349
jiwangyihao wants to merge 1 commit into
Wei-Shaw:mainfrom
jiwangyihao:fix/openai-images-stream-first-byte-upstream

Conversation

@jiwangyihao
Copy link
Copy Markdown

Summary

  • Prime OpenAI Images stream=true responses after local validation/routing and before waiting for the upstream response header.
  • Send an initial SSE comment and pre-upstream keepalive comments using gateway.image_stream_keepalive_interval.
  • Stop the primer before service-level response writes, and use SSE error events for post-primer upstream errors/fallbacks.

Motivation

Images stream requests can spend a long time waiting for the upstream response header before any downstream byte is written. Reverse proxies/CDNs with first-byte or idle timeouts may close the client connection before the existing stream keepalive loop starts.

This change moves the downstream SSE opening point earlier for Images stream=true requests without changing non-streaming behavior.

Behavior

  • stream=false is unchanged.
  • stream=true Images requests receive SSE headers and an initial : comment before upstream headers arrive.
  • Once the SSE response has been opened, later upstream HTTP errors or non-SSE fallbacks are written as SSE event: error events instead of JSON responses.
  • The primer is stopped before service forwarding writes to the same response writer.

Tests

  • C:/Users/34404/Documents/GitHub/workbench/toolchains/go/bin/go.exe test -tags unit ./internal/handler ./internal/service -run 'TestOpenAIImagesStreamPrimer|TestHandleOpenAIImagesStreamPrimerForwardError|TestShouldWriteOpenAIImagesStreamPrimerForwardErrorIgnoresPrimerComments|TestOpenAIGatewayServiceForwardImages_(APIKeyStream|OAuthStreaming|APIKeyStreamJSONAfterPrimer|APIKeyStreamHTTPErrorAfterPrimer|OAuthStreamHTTPErrorAfterPrimer)' -count=1 -timeout=120s
  • C:/Users/34404/Documents/GitHub/workbench/toolchains/go/bin/go.exe test -tags unit ./internal/handler ./internal/service -count=1 -timeout=180s
  • git diff --check origin/main...HEAD

Refs #2031

Prime OpenAI Images stream responses after local validation and routing but before waiting for upstream response headers.

Use SSE comments for the primer and convert post-primer upstream errors/fallbacks to SSE error events.

Refs Wei-Shaw#2031
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant