[iamnbutler]

Lets an agent read much more of a window: its full text, one deep subtree's elements, and text in windows that only have pixels.

Tools

• desktop_read (new, read-only) returns the Accessibility text of one exact window, up to 96 KB.
  • The read uses a 4,000-element, 64-level, 1,000-children budget, limited to about 18 s.
  • Text comes back in tree order (pre-order, which is DOM order for web content), one line per visual line. Inline links and wrapped text are joined on their line, and table cells are separated by tabs.
  • Content scrolled out of view is included when the app exposes it.
  • truncated and truncation name what cut the text short: text_limit or the traversal's ax_truncated_depth|count|children|deadline or ax_incomplete_read.
  • Optional scope (snapshot + element from an earlier accessibility desktop_inspect of the same window) reads only that element's subtree, such as an AXWebArea.
  • Secure text field values are skipped. It publishes no snapshot, sends no input and never activates anything.
  • On failure it returns a read_error with guidance that points canvases, games and remote desktops to pixels inspection with OCR.
• desktop_inspect scope: lists only one element's subtree, in tree order, each element with attributes.depth.
  • Accessibility mode reads the window with the same larger budget and saves the snapshot as usual. The result's text bound rises from 32 KB to 64 KB.
  • The scope element is found again by role, depth, bounds, label, title and identifier, plus its ordinal among identical matches. It must also be in the same process generation. A moved, changed or ambiguous element is refused, and the unused new snapshot is cleaned up.
  • Snapshot authority is unchanged. The new snapshot stores everything it traversed, the listed IDs belong to it, and actions resolve against it exactly as before. Without scope, inspection behaves as it does today: budget 15/200/100, same ordering.
• ocr: true on desktop_inspect (either mode) and desktop_screen runs VNRecognizeTextRequest at the accurate level, with language correction, in the client on the PNG it just captured.
  • recognized_text.lines lists text, confidence and box as top-left-normalized fractions of the screenshot. A box's center is a valid pointer point for that inspection's snapshot.
  • desktop_screen adds bounds in global desktop points.
  • Limits: 300 lines, 16 KB of text, and 8 s (shortened so the call stays inside the 30 s client deadline). Lines are ordered by row, then left to right.
  • It is labelled as recognized from pixels and not ground truth. A failure or timeout becomes recognized_text.error and never fails the observation.

Bridge

• I checked what the Bridge already offers. desktopObservation with a larger AXTraversalBudget and saveSnapshot: false already gives a fresh, exact-window, read-only detached traversal, so desktop_read uses only operations that are already allowed. No runtime allowlist changes were needed.
• inspectAccessibilityTree was not used, because with web focus it is a mutation. Peekaboo's own accessibilityAndOCR was not used either, because pixel snapshots require detection none.
• One problem was that DetectedElement is flat and has no hierarchy, and elements.all is grouped by type. peekaboo-element-depth.patch (last in the stack, after peekaboo-element-action.patch) adds a one-line depth attribute in DetachedAXObservationWorker. Element IDs are already pre-order (elem_N), so depth recovers each subtree. Traversal, budgets, IDs and authority are unchanged. It is documented in native/patches/README.md.

Verified

• npm run build:js && npm run check passes.
• node dist/build.js --out build --scratch native/.build passes. I also reset the scratch Peekaboo checkout to the pinned revision and rebuilt, so the full stack, including the new patch, applies cleanly.
• OCR (ocr.swift, compiled standalone with a test harness, not committed):
  • On a rendered 800×400 PNG it read "Quarterly report", "Revenue grew 12 percent", "Cancel" and "Save" with correct top-left boxes (for example 0.049, 0.122 against 40/800 and 0.11 expected). The screen bounds mapping is correct.
  • On a 4×110-line 1440×1700 image it truncated at 300 lines (137 omitted, 41 KB pretty-printed), taking 1.3 s.
  • A 1 ms timeout, and a missing file, each return error.
  • Warm recognition took 0.15 to 1.3 s. The very first run on this Mac took about 33 s while Vision prepared its model. The timeout error therefore says to retry once.
• Text assembly (text.swift, standalone harness):
  • Inline links and wrapped text join on their line ("…and then see docs. Then more.").
  • A heading and its static text are not duplicated, and a button's label is used only when it has no text descendants.
  • Field values are read, and the secure field is skipped. Table cells are joined by tabs, and offscreen Unicode text is kept.
  • The byte limit truncates on character boundaries and reports the omitted pieces. Reading a subtree slice works.
• TypeScript client, run against a fake native client:
  • The argv shapes are correct (inspect … accessibility ocr, … scope <snap> <elem>, read <pid> <win> [<snap> <elem>], screen <path> [display] ocr).
  • A malformed scope, or a scope in pixels mode, is refused before dispatch.
  • A native read failure becomes a structured read_error. Recognized text is carried through for inspect and screen.
• Native argv count checks were run against the built client.
• Peekaboo swift tests were not run. The patched line is in DetachedAXObservationWorker.process, which needs live AX, and the existing worker tests cover only static helpers.

Pending: screen locked

The Mac stayed locked for the whole session, so nothing ran live. With a signed dev build (open -n --env DESKTOP_TOOLS_STATE=/tmp/readdev "<built app>", then DESKTOP_TOOLS_STATE=/tmp/readdev node dist/mcp.js):

• Safari, a long article (for example a Wikipedia page): desktop_read returns the whole article text in order, including below the fold, and truncated is false or names text_limit. Note the time taken.
• Safari: desktop_inspect, then desktop_read with scope on the AXWebArea element. Check that the result has no toolbar or tab text.
• Safari: desktop_inspect with scope on the web area. It should list links deep in the page with attributes.depth. Then desktop_click one listed link by ID with the new snapshot, and check that it navigates.
• Scope refusal: scroll the page, then reuse the old scope. Expect the "not found unchanged" refusal and no input.
• Mail message list, and a Finder folder with hundreds of files in list view: desktop_read lists the rows. A scoped inspect of the list or outline shows rows past the first 200 elements.
• System Settings, a large pane (for example Privacy & Security): desktop_read text is complete.
• A canvas or Electron/Qt window with little AX: desktop_inspect mode: "pixels", ocr: true returns lines. A desktop_click at a line's box center hits that text.
• desktop_screen ocr: true: line bounds match the windows' positions on screen.
• TextEdit with a long document: desktop_read returns the AXTextArea value once, not duplicated.
• Timing: a read of a huge page stays under 25 s and reports ax_truncated_deadline instead of failing.

Caveats

• The scope is found again by matching and does not root the native traversal. A subtree late in a very large window can still be cut short by the 4,000-element or time budget, and the result reports this through truncation or detection_metadata.
• "Reading order" is tree order. For OCR it is row order, so multi-column layouts interleave by row.
• OCR runs on the 1× logical capture, so very small text can be misread.
• docs/tools.md now says 39 tools, counting the tools merged since; main still documented 35.