Skip to content

feat(tools): add macOS computer-use built-in tools - #170

Open
tokg-venv wants to merge 2 commits into
deer-flow:mainfrom
tokg-venv:feat/computer-use
Open

tokg-venv wants to merge 2 commits into
deer-flow:mainfrom
tokg-venv:feat/computer-use

Conversation

@tokg-venv

Copy link
Copy Markdown
Contributor

Summary

Adds a computer use tool set so the model can see the screen and act on it, as five new built-in tools (gated to darwin exactly like the speech tools):

Tool What it does macOS mechanism
computer_screenshot Full-screen or region capture returned as image content plus a text note screencapture (temp file read + always cleaned up)
computer_click Left / right / double click at coordinates CoreGraphics events posted via osascript JXA ObjC bridge (move settles hover first; double click sets ClickState 2)
computer_scroll Scroll-wheel events in all four directions, 1-50 lines CGEventCreateScrollWheelEvent
computer_type Type into the focused control System Events keystroke for ASCII; non-ASCII goes through a clipboard paste and the result discloses that
computer_key Named special keys + modifier combos (cmd+shift+t, ctrl+up, ...) key code / keystroke ... using {...} with a macOS key-code table

Zero new dependencies — everything shells out to tools macOS ships with. The image path uses the existing createToolCallResponse/ImageContent contract, so the screenshot is delivered to the model as a real image block.

Permission & safety notes

  • macOS prompts for Screen Recording (screenshots) and Accessibility (clicks/typing/scroll) on first use; the tool descriptions tell the model about this.
  • The tools are opt-in per thread like every built-in (the Add built-in tools dialog lists them automatically via the existing registry).
  • The OS bridge (run/readFile/removeFile/temporaryPath) is injected, so the whole suite runs without touching the real screen, clipboard, or keyboard. The JXA CoreGraphics scripts and screencapture were verified live on macOS.

i18n boundary compliance (per AGENTS.md)

Tool names, descriptions, and execution-path result strings are model-facing and stay canonical English; no user-facing UI copy is added, so no message-tree keys are needed. Icons for the new tools are registered in the playground icon map.

Test plan

  • New computer.test.ts (25 tests) driving every tool and error path through fake OS bridges: script construction for clicks/scrolls/keys, AppleScript escaping, ASCII vs clipboard typing, region validation, temp-file cleanup on failure
  • Module registration order updated in built-in-tools-module.test.ts (darwin branch)
  • Full bun run typecheck + ESLint clean; full bun test green except the two pre-existing langgraph-template failures that reproduce on clean main

zcai7675-bot added 2 commits September 13, 2026 10:48
Give the model eyes and hands on macOS via five built-in tools, gated
to darwin like the speech tools:

- computer_screenshot: capture the full screen or a region via
  screencapture and return it as image content (createToolCallResponse)
  plus a text note; the temp capture file is always cleaned up.
- computer_click: left/right/double clicks at coordinates through
  CoreGraphics events posted from osascript's JXA ObjC bridge (a move
  settles hover state first; double clicks set ClickState 2).
- computer_scroll: scroll-wheel events in any direction, vertical and
  horizontal axes.
- computer_type: keystroke for ASCII text (with escaping); non-ASCII
  text goes through a clipboard paste and says so in the result.
- computer_key: named special keys mapped to macOS key codes, with
  cmd/ctrl/alt/shift/fn modifier combinations.

The OS bridge (run/readFile/removeFile/temporaryPath) is injected, so
tests drive every tool and error path without touching the real screen
or clipboard. macOS prompts for Screen Recording and Accessibility
permissions on first use; the tool descriptions say so. Icons are
registered in the playground's built-in tool icon map.
Four fixes from a self-review against real hardware:

- Screenshots now report the pixel-to-point scale in their text note
  (e.g. "Image: 3420x2214 px (2x points) ... divide image pixel
  coordinates by 2") by parsing the PNG IHDR for pixel size and probing
  the main display's point size through CoreGraphics (cached). Region
  captures derive the scale from their own requested width, so they work
  even when the screen probe fails. Without this, clicks aimed at
  coordinates read off a Retina capture landed at 2x the intended spot.
- Removed the "fn" modifier: System Events rejects {fn down} at
  AppleScript compile time (verified with osacompile), so any fn combo
  errored.
- Click and region coordinates accept fractional points now -- models
  dividing pixel coordinates naturally produce x.5 values.
- Fixed the screen-size probe to destructure execFileAsync's result;
  calling .trim() on the {stdout, stderr} object threw and silently
  degraded every capture to the no-scale note (caught by a live
  end-to-end run, not by the fake-backed unit tests).

Test fakes now emit a real PNG header and a configurable screen size,
covering the scale note, the degraded notes, fractional coordinates,
and the fn rejection.
@tokg-venv

Copy link
Copy Markdown
Contributor Author

Self-reviewed the original submission against real hardware and fixed four real issues (05fe02a):

  1. Retina coordinate drift (the big one)screencapture returns 2x pixel images while clicks take points, and nothing told the model how to convert; on this machine a full-screen capture is 3420x2214 px over a 1710x1107 pt screen, so clicks aimed at image coordinates landed at 2x the intended spot. Screenshots now append a scale note to the text block ("Image: 3420x2214 px (2x points) … divide image pixel coordinates by 2"), derived by parsing the PNG IHDR and probing the main display via CoreGraphics (cached; region captures derive the scale from their own requested width so they degrade gracefully if the probe fails).
  2. fn modifier always failed — System Events rejects using {fn down} at AppleScript compile time (verified via osacompile), so any fn+X combo errored; removed from the modifier table.
  3. Fractional coordinates rejected — models dividing pixel coordinates naturally produce 855.5; the integer-only validation now accepts fractional points (negatives still rejected).
  4. Screen probe returned null silently — the default screenPoints implementation called .trim() on execFileAsync's {stdout, stderr} object; the TypeError was swallowed by the fallback and every capture degraded to the no-scale note. Found by a live end-to-end run (the fake-backed unit tests can't catch it), fixed by destructuring, and the parser is now extracted for coverage.

All 21 computer tests pass (fakes now emit a real PNG header and a configurable screen size), full typecheck/lint green, and both capture modes were re-verified live on Retina.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant