Skip to content

feat: drag between windows and apps - #52

Merged
iamnbutler merged 2 commits into
mainfrom
feat/drag-across
Oct 17, 2026
Merged

iamnbutler merged 2 commits into
mainfrom
feat/drag-across

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

Adds drags into another window or app: desktop_drag gains an optional to_snapshot.

Behavior

  • No to_snapshot: unchanged exact-window background drag.
  • With to_snapshot: from is a point in snapshot and to is a point in the other window's to_snapshot. Exact-window delivery can't leave its window, so this uses the approved foreground path:
    1. Resolve both snapshots: ownership, exact window, and pixel-backed coordinate authority. Map both points.
    2. Check with CGWindowList that each window is frontmost at its point and still at its snapshot bounds, skipping the host's overlays (shared exposes helper from feat: hover and press-and-hold #45). If not, refuse before any input.
    3. Take both snapshot leases.
    4. moveMouseWithOutcome to the start point (200 ms), then the global dragWithOutcome (.linear; default 800 ms; steps = duration/16, clamped to 10…120; left or right button).
    5. Finish both leases with fresh observation required.
    6. Always report unknown with DESKTOP_ACTION_UNVERIFIED. The message says the real pointer moved and the source app may have come forward, and asks the agent to inspect both windows. The result includes destination_receipt and pointer_moved.
  • Errors after dispatch are unknown, and errors before it are refused. If the second lease fails, the first is released.
  • The runtime's allowedOperations gains .drag.

Patch

peekaboo-drag-settle.patch (end of the stack): GestureService.playDragPath posted mouse-down and the first drag event back to back, then released immediately on the last point, so AppKit drag sessions often never start or the destination never accepts the drop.

  • It now waits 150 ms after the press.
  • It rests 150 ms over the end point, posting 3 drag events there, before releasing.
  • The release still runs in defer, so cancellation releases the button.

Only the global drag uses this path; the exact-window drag doesn't.

Overlay

Upstream already sends showMouseMovement for the hover and showSwipeGesture for the drag, so the agent cursor glides to the start and draws the trail.

Tests

  • npm run check
  • Signed native build on the full stack.
  • Live, dev app: to_snapshot equal to snapshot is refused, as are identical points without to_snapshot (TypeScript). Unknown snapshots are refused natively on the cross-window path before input.
  • No new Swift unit test: the patch only changes real-event timing, which unit tests can't observe.

Pending: screen locked

The screen is locked (IOConsoleLocked), so desktop_inspect can't capture. Run these with the dev app:

  1. File from Finder into TextEdit: open a Finder window with a .txt file and a TextEdit document side by side. Inspect both, then desktop_drag from the file icon to the TextEdit text area with to_snapshot. Expect unknown and pointer_moved: true, with the file's name or contents inserted in TextEdit.
  2. Text between apps: select a word in TextEdit and drag it into a Notes or Safari text field. Expect it copied there.
  3. Window to window in the same app: drag a Finder file between two Finder windows. Expect it moved.
  4. Covered end point: a window covering to → refused, and the pointer doesn't move.
  5. Moved window: move a window after inspecting → refused.
  6. Settle timing: check that the exact-window desktop_drag within one window still behaves as before (it doesn't use the patched path).
  7. Overlay: the agent cursor goes to the start, then draws the trail to the end, and the window highlight follows the windows.
  8. No stuck button: after each drag, a normal click works.

@iamnbutler
iamnbutler merged commit 6f2391b into main Oct 17, 2026
2 checks passed
@iamnbutler
iamnbutler deleted the feat/drag-across branch October 17, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant