From 3836d591c0efa98cb1115e8dcf7710b39d2d5e8f Mon Sep 17 00:00:00 2001 From: iamnbutler Date: Tue, 3 Nov 2026 08:28:48 -0500 Subject: [PATCH] Docs: catch up with 0.20.0 and the native2-only build - tools.md: background keys making the window key (key_window, list_focused, KEY_WINDOW_UNAVAILABLE, KEY_NO_FOCUSED_VIEW), web terminals (web_terminal), the WKWebView tree wait; 48 tools; drop the Peekaboo-era capture gate and apps budget, WINDOW_MINIMIZED, DESKTOP_EXPECT_OPS and client-side reads; add open_save_panel_service to the availability reasons. - app.md: per-agent cursors and colors; Always Allow Agent-Supplied Passwords. - embed.md: the password autofill and confirmation C ABI. - recipes.md, node.md, live-checks.md: field names that changed, password fill, and #5's remaining check. Co-Authored-By: Claude --- docs/app.md | 4 +++- docs/embed.md | 3 +++ docs/live-checks.md | 2 +- docs/node.md | 2 +- docs/recipes.md | 6 +++--- docs/tools.md | 21 +++++++++++++-------- 6 files changed, 24 insertions(+), 14 deletions(-) diff --git a/docs/app.md b/docs/app.md index 0c5e2c4c..7ca36768 100644 --- a/docs/app.md +++ b/docs/app.md @@ -1,6 +1,6 @@ # Desktop Tools app -`Desktop Tools.app` (`dev.githubnext.desktop-tools`) is a menu bar host for agents that don't embed the library, such as Claude Code or Codex through the [MCP server](mcp.md). It holds its own Accessibility, Screen Recording and Keyboard input grants, requested from its menu while any is missing, and serves `~/.local/state/desktop-tools/desktop.sock`. It links `desktop-client` there to its signed client. Only one copy runs for each state folder. See [Keeping the socket served](#keeping-the-socket-served). **Show agent cursor**, on by default, draws the agent's own pointer for each click, scroll, drag, hover and press. **Highlight agent's window**, also on by default, outlines the window the agent is observing or acting on and labels it with the agent and app, such as "Claude is using TextEdit". See [input](tools.md#input). +`Desktop Tools.app` (`dev.githubnext.desktop-tools`) is a menu bar host for agents that don't embed the library, such as Claude Code or Codex through the [MCP server](mcp.md). It holds its own Accessibility, Screen Recording and Keyboard input grants, requested from its menu while any is missing, and serves `~/.local/state/desktop-tools/desktop.sock`. It links `desktop-client` there to its signed client. Only one copy runs for each state folder. See [Keeping the socket served](#keeping-the-socket-served). **Show agent cursor**, on by default, draws each agent's own pointer for each click, scroll, drag, hover and press, so several agents' cursors move independently. Each agent has its own color: orange for Claude, blue for Codex, and for any other agent a color picked from its name, the same every time. **Highlight agent's window**, also on by default, outlines the window the agent is observing or acting on and labels it with the agent and app, such as "Claude is using TextEdit". See [input](tools.md#input). ## Keeping the socket served @@ -64,6 +64,8 @@ Every desktop change is logged, and so are screen captures and clipboard reads. The menu shows a **Grant** item only for a permission the app is missing. **Pause All Input** sits just above Quit. +**Always Allow Agent-Supplied Passwords**, off by default, lets `desktop_fill_password` fill a password an agent passes in without asking you each time. Keychain items still need macOS's own approval. + **Pause All Input** stops agents from changing anything until you turn it off again. While paused, the app refuses every click, key, typing, scroll, drag, menu command, app or window change, dialog action and clipboard write before it is sent. The refusal tells the agent to ask you to resume. Agents can still observe: listings, the accessibility tree, screen captures and clipboard reads keep working. So does turning on a Chromium or Electron app's web accessibility before a read, since it only lets the app's web content be read and sends no input (see [tools](tools.md#input)). The menu bar icon becomes a pause sign while input is paused. Only this menu item changes it; no client can pause or resume, and agent input can't reach the menu. Pausing isn't saved, so the app starts resumed. ## Build diff --git a/docs/embed.md b/docs/embed.md index 4f1acfc2..09f7a8ac 100644 --- a/docs/embed.md +++ b/docs/embed.md @@ -19,6 +19,9 @@ | `char *desktop_tools_activity(void)` | JSON array of the last 20 operations, newest first: `{ time, agent, pid, operation, app?, window?, outcome, count }`; free with `desktop_tools_free`. With a caller handler, `agent` and `pid` name the verified caller | | `void desktop_tools_set_log(const char *path)` | Append each operation listed by `desktop_tools_activity` to `path`, one line each, created readable only by the user; past 1 MiB it moves to `.1`. Pass NULL to stop | | `void desktop_tools_log(const char *message)` | Append one line of the app's own to that log, such as when the user pauses input | +| `void desktop_tools_set_password_autofill(bool)` | Fill passwords that agents supply without asking each time; Keychain items still need macOS's approval | +| `void desktop_tools_set_confirmation_handlers(ask, withdraw)` | Ask the user, on the main thread, before filling an agent-supplied password. `ask` gets an ID and JSON `{ key, title, message, allow, deny }`; `withdraw` gets the ID of a question nobody answered in time. Without them, agent-supplied passwords are refused unless always allowed | +| `void desktop_tools_answer_confirmation(uint64_t id, bool allowed)` | Answer one question from `ask`; an ID already answered or timed out is ignored | | `void desktop_tools_free(void *)` | Release a status or activity string | macOS grants Accessibility, Screen Recording, and event synthesis to the embedding app. Each embedding app needs its own grants on each machine. Clients must be signed by the host's team with the exact allowlisted identifier. In turn, a client signed with a team sends nothing, not even its hello, to a socket served by a process that isn't the same user's and signed by that team, and fails with `HOST_UNTRUSTED`; this covers `desktop-tools-client` and the app's `desktop-tools-mcp`. Both checks read the signature of the exact process at the other end of the socket, by its audit token. An unsigned build, such as `swift build` makes for development, skips them, as the host checks its client only when it is signed itself. The agent cursor and window outline draw click-through panels from the GUI process, so turn them on only from an app with a running AppKit main loop. The outline's label names the agent from `DESKTOP_TOOLS_AGENT` in the requesting client's environment, which `createDesktop` sets from its `agent` option; without it the label names the program that runs the client, as approvals do. Claude is orange, Codex blue, and any other agent gets a color fixed by its name; each agent has its own cursor and outline. For agents outside an embedding app, use the [Desktop Tools app](app.md). diff --git a/docs/live-checks.md b/docs/live-checks.md index 094cd81d..db744eda 100644 --- a/docs/live-checks.md +++ b/docs/live-checks.md @@ -45,6 +45,6 @@ These stay manual, because they are destructive, change system settings, need a - **#86 and #92 in other apps:** Numbers, Excel, Mail and Chrome, and a long Finder folder of 300 or more files. - **#88 in other apps:** Chrome, read-only native text views and Preview PDFs. - **#107 web accessibility in Electron apps:** inspect a Slack, VS Code or Discord window and check that `web_accessibility` is `enabled`, with `web_accessibility_via` `manual`, once per launch and the web content's fields are listed, that the app's window still moves and resizes normally, that the log lists **Turn on web accessibility** once, and that it still runs while input is paused. -- **#5 background paste into web content (spike, before any patch):** with a WebKit app such as Safari in the background and a textarea focused, answer: (1) is Edit > Paste `AXEnabled`, with and without the host's background key-window preparation; (2) does an Accessibility press of Edit > Paste paste into the focused textarea; (3) must the textarea get `AXFocused` first? +- **#5 in Safari:** checked only in a `WKWebView` test app. With Safari in the background and a page's text field focused, check that the first `desktop_inspect` lists the field and that `desktop_key` Command-V pastes into it. - **#27 Codex:** adding the server to Codex's configuration. - Anything that asks for a password or an administrator's approval. diff --git a/docs/node.md b/docs/node.md index 72337b32..a7f9315e 100644 --- a/docs/node.md +++ b/docs/node.md @@ -17,7 +17,7 @@ Each result has: - `text`: bounded JSON written for a model. - `data`: the same final object, parsed. -- `image`: a base64 JPEG or PNG, when the result has one. +- `image`: `{ data, mimeType }`, base64 JPEG or PNG bytes, when the result has one. - `outcome`, for actions: `completed`, `refused`, or `unknown`. Each request goes whole to the host, through its `desktop-tools-client`, as one tool call, so the host checks the arguments and shapes the result. One client process serves every call (`desktop-tools-client --serve`) and is replaced when it exits. Results take the compact shape described in [Result shape](tools.md#result-shape). Pass `verbose: true` in the config, or set `DESKTOP_TOOLS_VERBOSE=1`, for native results whole, with their `target_receipt`. Check the inventory completeness fields before relying on a listing. Image bytes appear only in `image`. diff --git a/docs/recipes.md b/docs/recipes.md index ee986af0..e67e2dc1 100644 --- a/docs/recipes.md +++ b/docs/recipes.md @@ -20,7 +20,7 @@ If the app has **File > Export as PDF…**, as Preview, Pages, Safari and Notes 3. inspect the window that holds the panel. Find the **PDF** menu button at the bottom left *(unverified: role `AXMenuButton`, title `PDF`)*. 4. `desktop_click` the PDF button. The fresh observation lists the open menu's items first. 5. `desktop_element_action` with `action: "AXPick"` on **Save as PDF** (or `desktop_click` it). A Save panel opens. -6. `desktop_dialog` with the app target. Expect `is_file_panel` true. Use its resolved window target for the next calls. +6. `desktop_dialog` with the app target. Expect `is_file_dialog` true. Use its resolved window target for the next calls. 7. `desktop_dialog_answer` with `path: "/Users/me/Desktop/out.pdf"`. Expect `completed`: the panel reads back the folder and name. 8. `desktop_dialog_answer` with `button: "Save"`. @@ -78,7 +78,7 @@ An app's **File > Open…** (⌘O), or its "add a project" or "choose folder" bu 5. `desktop_click` the family row, then the typeface row, each from the latest observation. 6. `desktop_type` the size field with `text: "18"`, then `desktop_key` `enter`. -**Verify:** inspect the panel and check that the family and typeface rows read as selected (`isSelected`) and that the size field shows the value. The text itself carries no readable font through `desktop_read`. +**Verify:** inspect the panel and check that the family and typeface rows show `selected: true` and that the size field shows the value. The text itself carries no readable font through `desktop_read`. **Goes wrong:** @@ -234,7 +234,7 @@ There is no pinch or rotate gesture. Use the app's own command: it acts on an ex - `desktop_type` on a web field that isn't focused is refused before any input, with a hint to click it first. Clicking is never done implicitly. - Typed web text is at most 1,000 UTF-16 code units with no newlines or tabs. - An `unknown` result means the readback didn't match. Autocomplete or input masks can reformat the text, such as phone numbers. Inspect before retrying. -- Password fields are refused, because password entry is not supported. +- Password fields refuse `desktop_type` with `SECURE_FIELD_REQUIRES_FILL`; fill them with `desktop_fill_password`, preferably from a Keychain item. - An autocomplete popup can take focus after typing. Press `escape` with `desktop_key` before the next click if it covers the next field. ## Pick from a native select in a browser diff --git a/docs/tools.md b/docs/tools.md index 3af18d97..29e13841 100644 --- a/docs/tools.md +++ b/docs/tools.md @@ -1,6 +1,6 @@ # Tools -The [MCP server](mcp.md) and `desktopTools()` expose 47 tools, one per protocol operation. Each tool's description in `src/tools.ts` says, in a few sentences, what it does, whether it is foreground or background input, when its result is `completed` rather than `unknown`, and its main refusal and recovery. Rules shared by every tool, such as single-use snapshots and never replaying input, are in the MCP server's instructions. This page holds the details. The client validates every argument, so the schemas name fields and keep only the limits that help choose values. +The [MCP server](mcp.md) and `desktopTools()` expose 48 tools, one per protocol operation. Each tool's description in `src/tools.ts` says, in a few sentences, what it does, whether it is foreground or background input, when its result is `completed` rather than `unknown`, and its main refusal and recovery. Rules shared by every tool, such as single-use snapshots and never replaying input, are in the MCP server's instructions. This page holds the details. The client validates every argument, so the schemas name fields and keep only the limits that help choose values. [Recipes](recipes.md) give tool sequences, derived from the code, for tasks that take several calls, such as printing to PDF, installing from a DMG or filling a web form. @@ -32,11 +32,10 @@ Each `desktop_windows` row has an `availability` object with a `state` (`availab | `off_screen` | The window's bounds are on no display | `desktop_move` | | `not_on_screen` | macOS reports it off screen for no reason the listing shows | `desktop_focus` | | `accessibility_unavailable` | It's on screen, but Accessibility can't see it (`limited`) | Wait briefly if it just opened, or inspect with `mode: "pixels"` | +| `open_save_panel_service` | The window belongs to the Open and Save panel service, not the app | Act on the app's window with the same title and bounds | When `desktop_inspect`, `desktop_read` or an action is refused, the result's `target_availability` carries the same classification, read from a fresh listing after the failure. A window that looks available but failed with an incomplete Accessibility tree, as while a web view first loads, is `sparse_accessibility`. `desktop_inspect` and `desktop_read` already wait a second and read once more after an incomplete Accessibility tree, within their 30-second budget; when that also fails, the error says `retried: true`. A page still loading may fill in after a few more seconds; otherwise use pixels mode and OCR. A window missing from the listing is `window_gone`. The classification names the likely cause; it can't prove it. -Captures go through ScreenCaptureKit, which every Peekaboo process for the user shares one call at a time. When an earlier capture timed out and hasn't returned, which usually means another capture on the Mac is busy, a new capture waits up to 2 seconds for it. If it still hasn't returned, the capture is refused with a message saying the problem is temporary; wait a few seconds and inspect again. There is no fallback capture, because every other capture path would also use ScreenCaptureKit while the abandoned call still runs. `desktop_apps` has 2.5 seconds to read the app inventory and is refused, not cut short, when the Mac is too busy to finish in time. - `desktop_wait` replaces polling `desktop_inspect` or `desktop_windows` after slow work such as a launch, a page load, an export or a dialog opening. Each call waits for one condition (`until`): - `window_appears`: an app, by `bundle_id` or `pid`, has a window that is on screen, minimized, or on another Space or in a hidden app (a window macOS keeps listed but ordered out on the current Space, such as the Spotlight panel, doesn't count), optionally with an exact `title` or a `title_contains` substring. `window_gone`: the window `pid` and `window` is no longer listed, or its app quit. @@ -54,7 +53,7 @@ The wait polls with backoff, from 200 ms up to 2 seconds between polls, for `tim - If it doesn't hold in time, the outcome stays `unknown`, with `expectation.last_observation`. - **Completed or refused outcome.** The result comes back as it is, with no polling. -The input is sent once and never repeated. Another action opts in by adding its op to `DESKTOP_EXPECT_OPS` and `expect` to its schema. +The input is sent once and never repeated. Another action opts in by adding its tool to `expectTools` in `native2/Sources/DesktopHost/Tools.swift` and `expect` to its schema in `src/tools.ts`. ## Result shape @@ -99,7 +98,7 @@ Window operations other than focus use background Accessibility and don't activa Both `desktop_spaces` and `desktop_screen` describe each display: its `desktop_screen` index, name, whether it is the main display (index 0, which holds the menu bar and the coordinate origin), `bounds` and `visible_bounds` in global desktop points with a top-left origin, as window bounds use, `scale` (pixels per point), `pixel_width` and `pixel_height`, and `rotation` in degrees. `visible_bounds` is what is left after the menu bar and Dock, the area `desktop_maximize` fills; a display left of or above the main one has negative coordinates. `desktop_spaces` reads this without a capture, so prefer it when only the arrangement is needed. -`desktop_apps` also returns `system`, the desktop-wide state worth checking before acting. `focus` names the frontmost app, its frontmost window and its focused element by role, title, identifier and bounds, which is where keyboard input and `desktop_system_key` would land; the element's value is never included, so field contents stay behind `desktop_read`. `appearance` gives dark or light mode, automatic switching, the accent color and the accessibility display settings that change how apps draw. `session` says whether the screen is locked, the screensaver runs, every display sleeps, or secure keyboard entry is on, which refuses typed text. `input` gives the keyboard layout, which `desktop_key` uses to find character keys, and the pointer's position and display. `menu_bar_autohides` and `dock_autohides` explain `visible_bounds`. Everything but the focused element comes from the client process without Accessibility; that one read goes through the host with its own 300 ms limit, after the inventory, so a slow app leaves `focus.element` null with a warning instead of delaying the call. Focus modes such as Do Not Disturb have no public read and are not reported. +`desktop_apps` also returns `system`, the desktop-wide state worth checking before acting. `focus` names the frontmost app, its frontmost window and its focused element by role, title, identifier and bounds, which is where keyboard input and `desktop_system_key` would land; the element's value is never included, so field contents stay behind `desktop_read`. `appearance` gives dark or light mode, automatic switching, the accent color and the accessibility display settings that change how apps draw. `session` says whether the screen is locked, the screensaver runs, every display sleeps, or secure keyboard entry is on, which refuses typed text. `input` gives the keyboard layout, which `desktop_key` uses to find character keys, and the pointer's position and display. `menu_bar_autohides` and `dock_autohides` explain `visible_bounds`. Everything but the focused element is read without Accessibility; that one read has its own 300 ms limit, after the inventory, so a slow app leaves `focus.element` null with a warning instead of delaying the call. Focus modes such as Do Not Disturb have no public read and are not reported. `desktop_switch_space` is foreground input. It presses Move left a space or Move right a space (Control-Left and Control-Right by default) once per step, reading the Spaces back after each, and reports `completed` only when WindowServer shows the target Space as current. Otherwise the result is `unknown` and names the Space that is current now. macOS applies those shortcuts to the display that holds the pointer or the key window, so with separate Spaces per display the switch is refused before any key unless both are on the target Space's display. `focus` in `desktop_spaces` shows where they are; use `desktop_hover` to move the pointer and `desktop_focus` to bring a window there forward. It is also refused when either shortcut is turned off or rebound to something other than an arrow key with modifiers in Keyboard Shortcuts > Mission Control. @@ -146,6 +145,8 @@ When the host turns on the window highlight, the window an inspection or action `desktop_key` goes to the control the snapshot reports as focused. When there is none, as in Finder lists, outlines, tables, canvases and many web areas, it goes to whatever has keyboard focus in that window; the result's `receiver` is `observed_control` or `window_focus`. Either way the key is sent in the background, and is refused unless the app's focused element is still inside the window, and the window is still the app's key window, before each event. +Lists, tables and web views take some keys, such as arrows, only in the key window. So when the app isn't active, the host first makes the window key without bringing the app forward, and the result's `key_window` is `made_key` or `already_key`. In Chrome, which has no standard title bar, it does this through the blank part of the tab strip. When the window can't be made key, the key is still sent but the result is `unknown` with `KEY_WINDOW_UNAVAILABLE` and `key_window: "not_key"`, naming the cause. When only the window itself has keyboard focus, as after a background click selects a row in a SwiftUI `List`, the host first focuses the window's one list or table that has a selected row and reports `list_focused`. If no view has focus, the result is `unknown` with `KEY_NO_FOCUSED_VIEW`. In an app built on `WKWebView`, which reports no focused element, keys go to the page field that reports itself focused. ([#226](https://github.com/githubnext/desktop-tools/issues/226), [#5](https://github.com/githubnext/desktop-tools/issues/5)) + A `desktop_click` with `modifiers` (any of `command`, `option` and `shift`), such as Command-click or Shift-click to extend a selection in Finder or Mail, is foreground input even on an element, because an Accessibility press ignores modifier keys. The host activates the app, moves the real pointer to the point (an element's observed center), holds the keys with real key events around a real click, then restores the pointer and the previously focused app and window; the result reports `cursor_restoration` and `focus_restoration`. Nothing reads back what the click selected, so its result is `unknown`. A Control-click is a right click: use `kind: "right"` without modifiers. Modifiers work with single, double and triple clicks. `foreground: true` sends a plain single, double or triple click the same way, without keys, for apps that act only on real user input: a background single click is an Accessibility press or selection, so an app whose handler checks that `NSApp.currentEvent` is a mouse or key event, as many do to tell user clicks from programmatic changes, ignores it. A background click that selects a row says so in its `note`. ([#225](https://github.com/githubnext/desktop-tools/issues/225)) `desktop_key` takes `keys`, a sequence of up to 16 chords each with its own modifiers, or `repeat` with one `key` to press it up to 32 times, all against one snapshot. Every key of a sequence goes to the window's keyboard focus, since focus may move as it runs (as Tab does), and before each one the host checks that the app's focused element is still in this window and that it is still the app's key window. The sequence stops at the first key that is refused or fails, with `KEY_SEQUENCE_STOPPED`; `keys_sent` says how many went, and a sequence that stopped early is `unknown`. A key that opens another window, such as Command-N, moves focus out of this one, so the next key stops the sequence; if the host can't tell whether that key was posted, the message says it may have been sent. @@ -156,6 +157,8 @@ A `desktop_click` with `modifiers` (any of `command`, `option` and `shift`), suc Chromium and Electron apps, such as Chrome, Edge, Slack, VS Code and Discord, build the Accessibility tree of their web content only after an assistive app asks for it. Until then a web view lists few or no elements. So before `desktop_inspect`, `desktop_read` or a `desktop_wait` element or text poll reads an app whose bundle ships an Electron, Chromium Embedded or Chromium framework, the host sets `AXManualAccessibility` on that app. It also does so after a read that finds a web area with at most one element under it, and then reads the window once more. It waits half a second after turning it on, so the page can build its tree, and when that read still finds no web content it waits another second and reads once more. The host sets it at most once per launch of the app. +An app built on `WKWebView` builds its page's tree only when it's first asked for, so the first read finds an empty group where the page will be. When `desktop_inspect` finds such a group, it reads again every 250 ms, for up to 2 seconds, until the web area appears. ([#5](https://github.com/githubnext/desktop-tools/issues/5)) + Web accessibility makes the app keep a tree for every page, which can slow a heavy page down. So the host turns it back off, setting the attribute it set to false, once no read of that app has relied on it for 10 minutes, as long as a snapshot lives, so a snapshot's web elements stay readable while it does. Pass `web_accessibility: false` to turn it off at once; the result then reports `restored`. The host turns off only what it turned on: when it found web accessibility already on, as VoiceOver or another assistive app leaves it, it never turns it off. Chrome 155 and later reject `AXManualAccessibility`. For an app that ships Chromium, the host then sets `AXEnhancedUserInterface`, as VoiceOver does, and counts it on only when it reads back on. That attribute can change how some apps animate and place their windows, so it is never set for other apps. In Chrome, window moves and resizes still read back exactly with it on. The result reports `web_accessibility`, and `web_accessibility_via` (`manual` or `enhanced_ui`) when this read turned it on: @@ -192,6 +195,8 @@ With `ocr: true`, `desktop_inspect` (in either mode) and `desktop_screen` also r `desktop_insert` pastes into the focused control the snapshot reports, and restores the clipboard once it reads the edit back. When the snapshot reports no focused control, as in terminals, canvases, custom views and many web editors, it types the text into whatever has keyboard focus in that window instead, with the same check `desktop_key` makes before each character: the app's focused element must still be inside the window, and the window must be the app's key window. Pass `route: "type"` to type into an observed control whose text can't be read, such as a terminal's text area, or `route: "paste"` to refuse typing. Typing doesn't touch the clipboard, takes at most 1,000 UTF-16 code units, and refuses newlines, tabs and other control characters, because typing them presses Return or Tab; press those with `desktop_key` where you mean to. It is refused while secure keyboard entry is on, as for a password field or Terminal's Secure Keyboard Entry. A typed insertion is `unknown` unless the host reads back an observed control's new value, so inspect the window before acting again. +A web terminal's input, the hidden text area of xterm.js as in Google Cloud Shell, is always typed into, and the result carries `web_terminal: true`. That input hands each character to the terminal and clears itself, so nothing can be read back. Typed text shows once the terminal's shell echoes it, which can be late over a slow connection. Check for it with `desktop_inspect` and `ocr: true`, then press Return with `desktop_key`. Don't type it again because it hasn't shown yet. `route: "paste"` there is refused with `PASTE_UNVERIFIABLE_RECEIVER`, because a paste could never be confirmed, would stay on the clipboard, and could run later. ([#229](https://github.com/githubnext/desktop-tools/issues/229)) + `desktop_type` sets a native field's value through Accessibility. Browsers accept that write for a field in a web page without firing the page's `input` and `change` events, so React and similar forms keep their old state. The host checks each text field's ancestry before writing, and a field inside a web area, or one whose ancestry it can't read, is never written that way. Instead, when that field is the snapshot's keyboard focus, `desktop_type` selects its text and types the replacement with key events sent to the app in the background, then reads the field back: the result is `completed` when the text matches and `unknown` otherwise. A web field without focus is refused before any input, with a hint to click it and inspect again; a click is a page action, so it is never done implicitly. Typed text is limited as for typed insertion, 1000 UTF-16 code units with no control characters. A secure text field, such as a password field, reports the role `AXTextField` with `subrole: "AXSecureTextField"`. Its value is never read: Accessibility gives only one mask character per typed character, and even that is left out of inspections, reads and waits. `desktop_type`, `desktop_set_value` and `desktop_select` refuse a secure field before any input, with `SECURE_FIELD_REQUIRES_FILL`. While a secure field has keyboard focus, `desktop_key` refuses keys that type characters and Command-V, and `desktop_insert` refuses every route, with the same code and nothing sent. Return, Tab, Escape, arrows, Delete and other shortcuts still work. @@ -216,7 +221,7 @@ The host replaces the field's contents with key events. The fill is completed wh `desktop_input_source` lists the enabled keyboard input sources that can be selected, as the input menu shows them. Each source has an `id`, `name`, `kind` (`keyboard_layout`, `input_mode` for one mode of an input method such as Hiragana, or `input_method`), its first 8 `languages` and whether it is selected. The result also gives `current` and the `keyboard_layout` that `desktop_key` maps characters through. -With `select` set to a listed `id`, the host selects that source with Text Input Sources (`TISSelectInputSource`), with no keys pressed, and the client then reads the current source back for up to a second: +With `select` set to a listed `id`, the host selects that source with Text Input Sources (`TISSelectInputSource`), with no keys pressed, and then reads the current source back for up to a second: - **`completed`** means the source reads back as selected, with `before` and `current`. - **Already selected:** the result is `completed` with `changed: false`, and nothing is changed. - **Not enabled:** the request is refused. @@ -225,11 +230,11 @@ With `select` set to a listed `id`, the host selects that source with Text Input - **macOS refuses the selection:** if the source still reads back unchanged, the result is `refused`. - **No readback:** the result is `unknown`. -Listing runs in the client process and needs no permission. Selecting goes through the host like other input, so the app's pause and activity list cover it; it needs no Accessibility either. The switch applies to every app until it is changed again. An input method's new mode may reach the frontmost app only after its next focus change. +Listing needs no permission. Selecting goes through the host like other input, so the app's pause and activity list cover it; it needs no Accessibility either. The switch applies to every app until it is changed again. An input method's new mode may reach the frontmost app only after its next focus change. `desktop_notifications` inspects Notification Center's windows on screen: its banners and alerts, and its panel when open (click the clock with `desktop_menubar_click` to open it). Each notification has an `id`, its app, title, subtitle, body and time where they can be read, its raw `texts` and `description`, and its `actions`: `open`, plus the custom actions Notification Center lists for it by exact name, such as Close, Show, Options or the posting app's own. `desktop_notification_action` takes that `id` and one of those actions. `open` is the notification's default click (on a stack, it expands the group); `close` runs Close; any other name runs that one listed custom action through Accessibility. Only Notification Center's elements list or run custom actions, at most 8 per element, each one line of at most 64 characters, and a name listed twice is never offered. A notification that has gone or changed is refused before input. Afterwards the notifications are read again: the result is `completed` only when the notification is gone, and `unknown` otherwise. System alerts such as permission prompts are dialogs, so `desktop_notifications` lists them under `system_alerts` for `desktop_dialog` and `desktop_dialog_answer` and never acts on them. This is built against the Notification Center structure of macOS 11 to 15. -An action refused because its window was minimized after inspection reports `WINDOW_MINIMIZED`, with the low-level reason as its `cause`. Restore the window with `desktop_restore`, then inspect it again. +An action refused because its window was minimized after inspection carries `target_availability` with `reason: "minimized"`. Restore the window with `desktop_restore`, then inspect it again. ## Details by tool