Skip to content

Fix Computer Use permissions, passive screenshots and Windows compatibility - #469

Merged
Wang-Daoji merged 2 commits into
MemTensor:computer_usefrom
memory-lee:codex/fix-computer-use-onboarding-and-windows
Sep 18, 2026
Merged

Wang-Daoji merged 2 commits into
MemTensor:computer_usefrom
memory-lee:codex/fix-computer-use-onboarding-and-windows

Conversation

@memory-lee

@memory-lee memory-lee commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix the remaining Computer Use onboarding and screenshot issues, and prevent unsupported Computer History entry points on Windows. This branch starts from computer_use and includes only this round of fixes; it does not bring in the unrelated release-branch changes.

  • Let native onboarding own the first permission flow, removing duplicate Settings/Accessibility prompts while keeping application operations blocked until authorization.
  • Match the running helper to its compiled build identity. Install the development helper at a stable location with a separate development bundle ID, preserve unchanged signed bundles, and keep its visible name Open Computer Use.
  • Reconnect the native proxy after the helper's Quit & Reopen. Never replay an action whose result was lost, and require a new user message after authorization. Memmy itself stays open when only its helper restarts.
  • Add get_screen_state for passive display observation and guide the agent to use it for screen-reading requests, so it does not open Finder/Recents as a substitute.
  • Make Computer History macOS-only across navigation, saved/deep-linked page selections, model synchronization, built-in skills, tools and HTTP routes. Windows Computer Use remains available.
  • Mark the default account agent_chat model as TEXT_IMAGE, passing screenshots directly while preserving the existing fallback for explicit image-input rejection.

The patched macOS runtime is built during development and DMG packaging from a pinned MIT-licensed source snapshot. Source preparation is repeatable and preserves compilation caches without retaining old patch-added files. Native CI builds and tests that runtime. Windows/Linux Computer Use packaging remains unchanged.

Validation

Validated on the clean computer_use-based PR worktree:

  • Agent: 17 test files, 340 tests passed, covering permissions, MCP, helper resolution, History platform gates, skills and image/fallback paths.
  • Frontend: 9 test files, 90 tests passed, including History platform navigation and existing macOS flows.
  • Packaging/contracts: 2 test files, 11 tests passed.
  • Native source preparation and signed development-installer fixtures: 9 tests passed.
  • Swift: 4 build-identity and 8 passive-observation fixture checks passed; both arm64 and x64 release builds and ad-hoc signature verification passed.
  • The compiled Swift proxy passed the reconnect/no-replay test against a temporary local socket.
  • Frontend, Agent and Electron TypeScript checks passed; shell syntax and git diff --check passed.

The user previously confirmed the repaired development authorization flow and a subsequent screen-reading request. Automated native tests use fixtures; they do not establish signed-installer acceptance on another machine. Windows desktop acceptance and a measured live foreground/Finder-window invariance check remain unverified.

Scope

The screenshot/Base64 token-estimation display issue is intentionally deferred. No global token-counting change, release-version change or unrelated release-branch work is included.

Test organization

  • Move the model-sync test to App/frontend/desktop/src/app/tests/.
  • Move the native Swift regression sources to tests/open-computer-use/, update source preparation and CI path filters, and document the new location.
  • Revalidated the relocated frontend tests (7), source-preparation tests (2), frontend type checking, and Swift identity/observation fixtures (4 + 8). No application behavior changes in this follow-up.

@Wang-Daoji
Wang-Daoji merged commit db8c946 into MemTensor:computer_use Sep 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants