fix(vision): forward MCP tool result images to vision-capable models#79
Merged
Conversation
Images nested inside LanguageModelToolResultPart.content (e.g. screenshots
returned by chrome-devtools-mcp, playwright-mcp) were silently dropped by
convertMessage(): partToText() returned "" for any image DataPart via its
catch-all fallback. The result was an empty tool message sent to the model,
so vision-capable models like Kimi K2.7 Code honestly reported they could
not see the image — even though the same model worked fine when the image
was pasted directly into chat.
Root cause: partToText() handled TextPart / ToolCallPart / internal DataPart
/ string, but image DataParts nested inside tool results fell through to the
catch-all return "". Top-level image attachments worked because they hit a
separate handler in convertMessage() that explicitly checked
part.mimeType.startsWith("image/").
Fix:
- convertMessage(): walk part.content and partition each item into either
text (via partToText) or an OpenAiContentPart image part (via
dataPartToBase64). Emit a multimodal OpenAiContentPart[] on the tool
message when any image is present; otherwise emit a plain string so
text-only tool results stay byte-identical to the previous behavior.
- AnthropicToolResultBlock.content type widened from string to
string | AnthropicContentBlock[]. New anthropicToolResultContent() returns
the string form when there are no images and the array form (text + image
blocks) only when an image is present.
- responsesInputItemsFromMessage() tool branch: new responsesToolOutput()
helper that degrades images to a placeholder note (Responses API
function_call_output.output is string-only).
- googleContentsFromMessages() tool branch: new
googleFunctionResponseContent() that emits parts:[{text},{inlineData}]
when an image is present.
- New MAX_TOOL_RESULT_IMAGE_BYTES = 1_000_000 constant. Oversized images
(>1 MB raw bytes) are replaced with an actionable placeholder note so a
single full-page MCP screenshot can't push the request payload past the
upstream limit. Manual testing revealed agent screenshot loops producing
4.6 MB payloads that triggered upstream 400 "Upstream request failed".
Drive-by log noise reduction (pre-existing, surfaced during testing):
- provideLanguageModelChatInformation: replaced per-model log spam (22 lines
per call x ~3 calls/sec) with one summary line per invocation.
- fetchModels: replaced vscode.window.showWarningMessage on transient
upstream 400/503 (common from OpenCode's shared gateway, especially on
auto-registered AGENT_* variants the user may not actively use) with an
Output-channel log. Bundled fallbackModels snapshot keeps the picker
functional.
Verification:
- tsc -p ./ exit 0 (TypeScript strict, no errors)
- node --test out/test/**/*.test.js — 107/107 pass, 0 regression
- Manual test: chrome-devtools-mcp + OpenCode Go model successfully read
and described the returned screenshot
Docs:
- docs/issues/34-20260720-mcp-tool-result-image-dropped.md (root-cause
investigation + timeline + follow-up bugs uncovered during testing)
- docs/features/12-20260720-mcp-tool-result-image-support.md (architecture,
per-transport behavior, code locations, limitations)
- CHANGELOG.md [Unreleased] section
- docs/devlog.md session entry
Fixes #77
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes #77.
MCP tool result images (e.g.
chrome-devtools-mcpscreenshots) were silently dropped during serialization — vision-capable models like Kimi K2.7 honestly reported "I cannot see the image". Pasted image attachments worked because they hit a different code path; only images nested insideLanguageModelToolResultPart.contentwere lost.Why
partToText()handled TextPart / ToolCallPart / internal DataPart / string, but image DataParts nested inside tool results fell through to the catch-allreturn "". The model received an empty tool message.How
convertMessage()now walkspart.contentand emits a multimodalOpenAiContentPart[]on the tool message when any image is present. Text-only tool results stay byte-identical to before.tool_result.content: AnthropicContentBlock[]), Google Gemini (functionResponse.parts: [{inlineData}]), and Responses API (degrades to placeholder note — API is string-only).MAX_TOOL_RESULT_IMAGE_BYTES = 1_000_000size guard after manual testing revealed agent screenshot loops producing 4.6 MB payloads that triggered upstream 400.Drive-by fixes
provideLanguageModelChatInformation: replaced per-model log spam (22 lines × ~3 calls/sec) with one summary line per invocation.fetchModels: replaced modalshowWarningMessageon transient upstream 400/503 with Output-channel log. The bundledfallbackModelssnapshot keeps the picker functional, so the popup was just noise — especially on auto-registeredopencodezen-agentthat the user may not actively use.Verification
tsc -p ./cleannode --test out/test/**/*.test.js— 107/107 pass, no regressionchrome-devtools-mcp+ OpenCode Go model: model read and described the screenshotDocs
docs/issues/34-20260720-mcp-tool-result-image-dropped.md— root-cause investigationdocs/features/12-20260720-mcp-tool-result-image-support.md— architecture & per-transport behaviorCHANGELOG.md [Unreleased]Notes for review
gpt-5-codextoday; placeholder note is intentional.