Conversation
added 2 commits
September 13, 2026 10:48
Give the model eyes and hands on macOS via five built-in tools, gated to darwin like the speech tools: - computer_screenshot: capture the full screen or a region via screencapture and return it as image content (createToolCallResponse) plus a text note; the temp capture file is always cleaned up. - computer_click: left/right/double clicks at coordinates through CoreGraphics events posted from osascript's JXA ObjC bridge (a move settles hover state first; double clicks set ClickState 2). - computer_scroll: scroll-wheel events in any direction, vertical and horizontal axes. - computer_type: keystroke for ASCII text (with escaping); non-ASCII text goes through a clipboard paste and says so in the result. - computer_key: named special keys mapped to macOS key codes, with cmd/ctrl/alt/shift/fn modifier combinations. The OS bridge (run/readFile/removeFile/temporaryPath) is injected, so tests drive every tool and error path without touching the real screen or clipboard. macOS prompts for Screen Recording and Accessibility permissions on first use; the tool descriptions say so. Icons are registered in the playground's built-in tool icon map.
Four fixes from a self-review against real hardware:
- Screenshots now report the pixel-to-point scale in their text note
(e.g. "Image: 3420x2214 px (2x points) ... divide image pixel
coordinates by 2") by parsing the PNG IHDR for pixel size and probing
the main display's point size through CoreGraphics (cached). Region
captures derive the scale from their own requested width, so they work
even when the screen probe fails. Without this, clicks aimed at
coordinates read off a Retina capture landed at 2x the intended spot.
- Removed the "fn" modifier: System Events rejects {fn down} at
AppleScript compile time (verified with osacompile), so any fn combo
errored.
- Click and region coordinates accept fractional points now -- models
dividing pixel coordinates naturally produce x.5 values.
- Fixed the screen-size probe to destructure execFileAsync's result;
calling .trim() on the {stdout, stderr} object threw and silently
degraded every capture to the no-scale note (caught by a live
end-to-end run, not by the fake-backed unit tests).
Test fakes now emit a real PNG header and a configurable screen size,
covering the scale note, the degraded notes, fractional coordinates,
and the fn rejection.
Contributor
Author
|
Self-reviewed the original submission against real hardware and fixed four real issues (05fe02a):
All 21 computer tests pass (fakes now emit a real PNG header and a configurable screen size), full typecheck/lint green, and both capture modes were re-verified live on Retina. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a computer use tool set so the model can see the screen and act on it, as five new built-in tools (gated to darwin exactly like the speech tools):
computer_screenshotscreencapture(temp file read + always cleaned up)computer_clickosascriptJXA ObjC bridge (move settles hover first; double click setsClickState2)computer_scrollCGEventCreateScrollWheelEventcomputer_typekeystrokefor ASCII; non-ASCII goes through a clipboard paste and the result discloses thatcomputer_keycmd+shift+t,ctrl+up, ...)key code/keystroke ... using {...}with a macOS key-code tableZero new dependencies — everything shells out to tools macOS ships with. The image path uses the existing
createToolCallResponse/ImageContentcontract, so the screenshot is delivered to the model as a real image block.Permission & safety notes
run/readFile/removeFile/temporaryPath) is injected, so the whole suite runs without touching the real screen, clipboard, or keyboard. The JXA CoreGraphics scripts andscreencapturewere verified live on macOS.i18n boundary compliance (per AGENTS.md)
Tool names, descriptions, and execution-path result strings are model-facing and stay canonical English; no user-facing UI copy is added, so no message-tree keys are needed. Icons for the new tools are registered in the playground icon map.
Test plan
computer.test.ts(25 tests) driving every tool and error path through fake OS bridges: script construction for clicks/scrolls/keys, AppleScript escaping, ASCII vs clipboard typing, region validation, temp-file cleanup on failurebuilt-in-tools-module.test.ts(darwin branch)bun run typecheck+ ESLint clean; fullbun testgreen except the two pre-existing langgraph-template failures that reproduce on cleanmain