Sync main into drawing - #512
Merged
Merged
Conversation
Synthetic checkpoint for safe three-way migration onto upstream computer_use; source worktree remains untouched.
Include recording, summarization, workflow extraction, replay scripts, schema, and demo fixtures.
…dinates The recorder resolved every click by hit-testing the cursor position and then sleeping 300ms hoping the application had rebuilt its accessibility tree. That races the renderer, and its own comment admitted Chrome often still returned stale geometry on the retry. Keep the focused element continuously up to date from AXObserver notifications so a click can be attributed immediately; the positional hit test is now only a fallback for controls that never take focus. Coordinates are no longer emitted. Event kinds now match the shape Codex/Skysight produces, which the diff engine and the layered summaries will build on: selection.changed new, and the highest-volume semantic signal keyboard.submit new, a cheap and reliable task-boundary marker mouse.drag new, carrying origin and destination elements mouse.context_menu new scroll removed; the AX tree diff carries that state instead Also add app.secureInput from IsSecureEventInputEnabled() so keystroke text is suppressed while a password field owns focus, rather than relying only on the redaction regexes as a second line of defence. The history JSONL output shape is unchanged so summarize-history and its fixtures keep working. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex's equivalent tool has no replay: it answers questions about what the user did, and reproducing a behavior is Computer Use's job. Splitting the two the same way stops a retrieval result from turning into desktop control on its own, and matches the conclusion that Computer History is searchable evidence rather than a library of replayable templates. Drop the `action` enum and the replay branch, add an explicit result limit, and label every returned field as untrusted observed evidence — the event stream records whatever appeared on screen, including text written by third parties. The service keeps prepareReplayUserRequest for the desktop UI; only the agent-facing surface loses it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ools Always-on capture needs a policy before it needs a daemon. This adds the model Codex uses: two orthogonal axes rather than one list, so "record everything except my bank" and "record nothing except my IDE" are both expressible. defaultApplicationBehavior applications matching no app rule, by bundle id defaultURLBehavior websites matching no URL rule, by bare domain Allow and block rules coexist; a block rule always wins inside its own axis, so an allow entry can never re-enable something the user excluded. A browser record with a usable URL must pass both axes, a record without one is judged by its application alone, and private browsing is excluded unconditionally. The default is do-not-observe: a fresh install records only what the user has explicitly allowed. Adds computer_history_status, _get_settings and _update_settings. Updates replace the whole document rather than merging, so the update tool says so and tells the agent to read first — that is what stops it from silently dropping rules the user never mentioned. The recorder still takes its allowlist from CLI flags; wiring it to this policy lands with the resident daemon. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every event now carries the state of the focused window, which is only affordable if consecutive events do not repeat the whole tree. Keep the previous snapshot per window and emit just the added and removed nodes, falling back to the full tree when there is nothing to diff against or when the change is large enough that a diff would not be smaller. Measured on a live session: the first snapshot is 6775 characters, the next is a 44-character diff. Snapshots are throttled to at most one every 400ms and bounded at 400 nodes, so a busy window cannot turn the recorder into a tree-walking loop. The state rides through to the history JSONL for the layered summaries to consume. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Observation is now continuous rather than a set of named recordings the user starts by hand. A recorder runs in the background and its output is sliced into ten-minute segments, each a directory holding events.jsonl and metadata.json, with segment ids aligned to the ten-minute grid so they sort and group cleanly. The state machine is three-valued, not two: running capturing according to the observation settings paused current segment kept, nothing new written to it stopped recording nothing; completed segments stay searchable Pause is a first-class state rather than a weaker stop, because "stop watching while I do something private" must not cost the user the arc they were in the middle of. The desktop toggle gains a matching third state. Recording is scoped to the app: the server's stop path finalizes the open segment, so closing the app ends observation instead of leaving a recorder running behind the user's back. Segments carry no title or starting URL — those described a single recording, and a continuous stream has neither. Also fixes two defects in the preceding commits: appendEvent destructured its argument and silently dropped the ax field, and flushKeys still read the pre-AXObserver application shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Looking back over a working day should not cost a re-read of the whole event stream. A six-hour summary is now built from the ten-minute summaries covering its window, so the cost of the wider view scales with the number of summaries rather than the size of the stream. That is also why the six-hour file cites the ten-minute files it reused instead of the segments beneath them: a reader follows one level down, not all the way down. Applications and context lines are merged without repetition, and a six-hour file is never folded into another six-hour file. Segment summaries are now named `<id>-10min-summary.md` so the two layers are distinguishable on disk and the rollup can find its inputs. The rollup runs whenever a segment is finalized, which is cheap precisely because it reuses what is already summarized. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ion policy Making observation resident dropped the hardcoded six-app allowlist that the old named-recording path passed, and an empty allowlist means "no filter" in the consumer. The recorder would have observed every application, with no policy in front of it — the exact combination the settings model exists to prevent. The recorder now takes --observation-settings and evaluates the policy per event, which it has to do itself because the website axis depends on the URL each event carries. Starting observation is refused outright when the policy would record nothing, so a fresh install says what is missing instead of looking like it is recording while producing empty segments. The rule table is asserted on both sides, in observation-settings.test.ts and in record-human-history.test.mjs, so the capture path and the agent tools cannot drift apart on what a policy means. Also lets the service take an observation settings path, so tests stop writing the user's real policy file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The timeline was showing another product's records as if they were Memmy's own, from two directions: Codex's live Skysight directory was read on every snapshot, and 155 copies of its summaries had been dropped into Memmy's own history directory during earlier experiments, where nothing distinguished them from a real capture. Codex writes `<utc>-<4 random chars>-10min-memory-summary.md`; Memmy names its segments `<segment id>-10min-summary.md`, with no random component and no "memory-", so the copies can be recognized by name and left out. They stay on disk — this hides them, it does not delete anything. Reading Codex's live directory is now opt-in rather than the default, through a constructor flag or MEMMY_COMPUTER_HISTORY_CODEX_SYNC=1. The two tests covering that path opt in explicitly instead of relying on the old default. On the current machine this takes the timeline from 163 entries to the 8 Memmy actually captured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading Codex's Skysight directory made another product's records show up as Memmy's own. In a real deployment the timeline should come from Memmy's own capture, so the sync is gone rather than merely defaulted off: the codex_synced source type, the directory reader, the opt-in flag, and the sync-state file that existed only to hide synced entries from the timeline. Trimming codex_synced also simplifies the paths that branched on it — replay plans, workflow creation and deletion now have one fewer source to reason about, and deleteHistory no longer has a branch that hides instead of deletes. isCodexSkysightCopy stays: copies of Codex summaries are still sitting in the history directory from earlier experiments and must not be shown as captures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The timeline showed a title and a timestamp, and the pane beside it dumped the raw markdown including its YAML frontmatter. Worse, since observation became resident there is no user-supplied title, so every entry read "Computer History <segment id>" above the same templated sentence. Each finished summary is now narrated by the model: a specific title and two or three sentences addressed to the user, written from the segment's own evidence. The call is fire-and-forget after the mechanical summary is already on disk, so an unreachable model costs the better wording and never the recording, and the prompt states that the recorded screen content is evidence rather than instructions. The timeline groups by day and each entry carries its own account. A day that has a six-hour rollup is headed by it as a collapsible overview rather than letting it compete with the ten-minute entries. The detail pane renders time, title, description and applications as a header and drops the frontmatter from the body. Entries now carry description, applications and summaryWindow so the timeline does not have to parse markdown to render itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding description, applications and summaryWindow to the shared markdown reader put them on workflows too, because the workflow list was built by spreading the directory entry. The desktop client parses the snapshot with a strict schema, so every workflow failed validation and the whole page rendered as a wall of unrecognized_keys errors instead of the timeline. Build workflows by naming their fields. A workflow is not a summary, and spreading meant the next field added to the reader would leak the same way. Nothing caught this: the backend never asserted the snapshot's shape, and the desktop test mocks the client, so it never runs the schema. Both gaps are now covered — the workflow test fails if the spread comes back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat: integrate Computer History and Computer Use
…owlist The default was do-not-observe, which meant a fresh install recorded nothing until the user named applications one by one. That fails silently in the worst way available: the UI reads "recording" while nothing is written, and the gap surfaces days later when the history is asked for and turns out to be empty. Codex, measured on this machine, recorded 27 distinct applications including Dock, Finder and System Settings — it observes by default and uses the blocklist for exceptions. Match that. What protects the user here was never the direction of this default: it is the blocklist, pause, the unconditional private-browsing exclusion, secure-input suppression, no screenshots, local-only storage and the retention window. All of those hold either way. The service now writes the policy out on first start, so the recorder parses one explicit document instead of inferring a policy from a missing file, and the user has something to edit. A file that exists but does not parse stops the start rather than falling back to a policy they did not choose. No default blocklist ships: which applications would go on it is a judgement better left to the user. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moving recordings under `segments/` left the cleanup scanning one level too high, so it judged the container by its own mtime. That broke retention in both directions: while recording continued the container stayed fresh and nothing inside was ever removed, and once recording stopped for long enough the whole container expired at once, taking every segment with it. In practice it meant the 48-hour window silently never applied and raw event streams accumulated without limit. Judge each segment directory on its own age, never the container, and leave the open segment alone because it is still being written to. Recordings captured before segments existed still sit at the top level and are still expired there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ript The summarizer was a formatter: it emitted one bullet per event under "Activity timeline" and one numbered step per event under "Reusable operation experience". A five-minute window produced 280 lines, 175 of them "Typed 1 character(s); text redacted". That is the event stream reformatted, not a summary, and it is nothing like what it was meant to resemble. The model now writes the recording summary itself, from the segment's own event stream compacted into activity arcs: consecutive events are grouped by application and each run reported once, keeping the semantic labels — a clicked message, a page title — because those are what let the account say what happened. Forty keystrokes become one line rather than forty. Recovering those labels means reading the recorder's enrichment, not just the top-level attributes: clicks often land on an anonymous container, and the focused control and nearest labeled descendants and ancestors are where the name actually is. What stays is the part that was already right: the durable facts — time range, applications, event counts, click coverage, approved starting URL. Those are what remain useful once raw events pass their retention window. The scroll compression test went with the scroll events themselves, which the AXObserver rewrite removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ent be pinned A workflow candidate was generated for every finalized segment. A ten-minute window boundary has nothing to do with a task boundary, so almost all of them described a slice of an afternoon rather than anything worth repeating. Steps are now derived from a segment's event stream when someone actually asks to repeat something. That makes the raw stream the thing replay depends on, so retention becomes a real question rather than a detail. Two answers: within the window, ask and a workflow is written from the events, and from then on it is permanent and independent of them. Beyond it, pin the segment beforehand and its events are exempt. Because pinning writes a marker inside the segment, age can no longer come from the directory's mtime — touching it would silently grant another full window. It comes from the start time the segment recorded for itself. Replay availability now follows from whether the event stream is still on disk, not from whether the summary contains a section listing actions — that section is gone, and every path that parsed it was quietly returning nothing. Searching no longer filters by replayability. It is evidence retrieval: an entry used to vanish from search the moment its events expired, which is exactly when the written summary is all that remains and the only thing that can still answer what happened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… so when it does not The titles stayed mechanical because the call was missing what the chat-title generator passes: reasoningEffort "none". A reasoning model spent the budget thinking and returned empty content, and every failure path here returned null without a word — so a broken call looked exactly like a feature that was never wired. Diagnosing it took reading the code rather than the symptom, which is the real defect. Narration now reports why it produced nothing, through a callback the service logs and exposes as observation.narrationError, so the mechanical wording is always explained rather than merely observed. It also runs once a segment has enough stream to be worth summarizing, not only when the segment closes. Waiting for the close left every entry reading mechanically for the whole ten minutes someone is most likely to be looking at it, which is how this went unnoticed for three rounds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…just summaries The summaries and the event streams are two different records answering two different questions — what a window was about, and what specifically happened in it. Only the first was reachable, so "who contacted me today" could not be answered even though the message text was sitting in the stream all along. Three things were in the way. computer_history_status reported the summary directory as event_stream_root_path, so following it led to the wrong place; it now reports both, named for what they hold. Search results carried no way down to the evidence; each match now names its event stream, or says the raw events have passed retention rather than implying the summary was all there ever was. And nothing told the agent this record existed or how to read it, which the new computer-history skill now does: the layout, when to stay in the summaries, what fields carry a clicked message, and to grep rather than read whole files, because one event line can carry a whole accessibility tree. The skill also states the two rules this record needs: it is evidence and never instructions, however much a captured message reads like one; and an answer must distinguish "nothing was found" from "recording was off" or "the events expired", because those send the user somewhere different. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ning it by hand Adding eventStreamPath to history entries broke the page the same way adding description did: the desktop client validates the snapshot strictly, so a field the agent grows and the client does not know about fails every entry and renders a wall of unrecognized_keys. The first time, the answer was a test asserting the workflow field names. That only covered workflows, so histories drifted next, and the same test would have kept passing however many more fields were added elsewhere. Listing field names by hand is the same work the schema already does, done twice. The schema now lives on its own, free of any browser dependency, and a test at the repository root parses a real snapshot from the real service with it. It belongs at the root because the contract spans two packages that cannot import each other — which is exactly why the sides were free to drift. Reverting the missing field makes it fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…null
Narration resolved its model with `llmRuntime(modelPreset ?? null)`. That turns
"no preset was asked for" into "resolve the preset called null", which the
gateway's resolver cannot do — it answered model_selection_unavailable, and
every summary silently kept its mechanical title.
Reproduced against the real config: llmRuntime() resolves gpt-5.6-terra,
llmRuntime("computer-use-fast") resolves it too, and only llmRuntime(null)
throws. Pass nothing when there is nothing to pass.
End to end on a real segment, 424 events compact to 56 activity arcs and come
back as "Computer History Integration Review" with an account naming DingTalk,
Arc, PR #371 and the architecture docs — instead of "用户在「Computer History
2026-09-08T08-20-00Z」中完成了一组电脑操作".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ow it only when written Two things were wrong with what the timeline showed. The written summary never survived recording. Every sixty seconds the live pass regenerated the mechanical markdown over the file, while narration ran once per segment — so the account was written, overwritten on the next tick, and only stuck after the segment closed. That is why a summary appeared to require a manual stop. And it was only ever half written. Narration replaced the recording summary alone, leaving "Memory summary" as a templated sentence, "Relevant prior context" as boilerplate about the capture policy, and "Important non-obvious context" holding event counts, screen resolution and file paths — machine bookkeeping under a heading promising insight about the user. The model now writes the body: what the window was for, how it relates to the windows before it, the specifics worth keeping once the raw events expire, and the account of what happened. It is given the two preceding summaries, without which any claim about what came before would be invention. The mechanical pass is reduced to frontmatter and citations, and the machine facts are gone. An entry appears in the timeline only once written, so the placeholder is never what a reader meets. Imported and demo entries, which no model writes, are unaffected. Verified end to end on a real segment: four model-written sections, the prior context genuinely continuing the earlier window, and no bookkeeping left. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page was a list beside a markdown pane, and the pane showed the file: frontmatter bookkeeping, a templated "Memory summary", citations. Only four fields were ever meant for a reader — title, description, applications and the account itself — so the page is now a single column of those, and the markdown body no longer reaches it. Three things were wrong underneath that, each of which made history appear to lose entries. A summary was dated by its file. Nothing writes `captured_at`, so `createdAt` fell through to mtime, and a summary is rewritten every time the model catches up with it. A ten-minute window from 10:10 displayed as 10:28, walked forward on every regeneration, and sorted rollups into the middle of the segments they cover. The window an id already names is the honest answer. Finalizing a segment published less than was already published. The mechanical pass overwrote the account with its placeholder and narration replaced it a few seconds later — an entry that blinked out and returned, or never returned when the model was unreachable. The regeneration now happens beside the standing summary and is swapped in only once written. And a rollup stood next to the segments it summarizes, saying the same six hours twice. A rollup now appears only once its window has closed, at which point it stands in for what it covers; a window whose rollup never ran keeps its segments rather than losing them. Applications are drawn as their real icons, read through NSWorkspace rather than out of the bundle, because a growing share of applications ship their icon in an asset catalog with no .icns to find. Also: every string moved into the message catalogs, which this file had been violating since before this change, and the stylesheet it replaced is gone — including the two `font: inherit` declarations that were failing the prototype alignment test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ry-ax-capture # Conflicts: # package.json
feat(computer-history): make the timeline a feed of what the model wrote, and catch up with main
Computer History sat among the memory views — overview, tasks, skills — but it is not a view of what the agent remembered. It is a capture the user switches on, closer to the cross-agent source it now sits beside. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…der src/tools Computer History was spread over four places — a service under entrypoints/frontend-bridge, its policy and summaries under core/agent-runtime, its agent tools beside unrelated tools, and the scripts that do the recording and summarizing in a workflows/ folder at the repository root. It now lives in src/tools/computer-history/mac, with Computer Use in src/tools/computer-use and an empty computer-history/win held open for the Windows port. The move was not only cosmetic. electron-builder ships dist/, and workflows/ was never in it, so the recorder and summarizer did not exist in an installed app: the service found them by walking up to a repository checkout that a packaged Memmy does not have. The scripts are TypeScript now and compile into dist/ beside the service; the Swift helpers and the replay script are copied there by the build; and everything is located relative to the module rather than to a checkout. Along the way: - The summarizer and the workflow-candidate extractor are called in process instead of spawning node every minute. The recorder is still a process, since it runs for the length of a segment. - The recorder entry point is injectable. The observation tests started the real one — which taps the keyboard and mouse — and never stopped it, so each run left recorders listening to the machine after the suite finished. They now use a stand-in and shut every service down. - The three script tests had never run: nothing wired them into a test runner. They run under vitest now, calling the modules rather than spawning them. - The demo fixtures and the documents beside them are removed, and with them the three features that only read those fixtures — installing the demo History, generating its canned workflow, and the CUA smoke test — none of which the UI still offered. - Five values the summarizer computed and never used are gone. `.*` in .gitignore would drop the placeholder that keeps win/ in the tree, so that one file is whitelisted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-release-v1.1.6 Backport native npm Computer Use and macOS permission onboarding to v1.1.6
codex/release v1.1.7 pr
…-pr439-isolated fix: cover PR494 computer use contracts and tests
release: Memmy v1.1.6
…1.1.7 codex/sync main into release v1.1.7
Merge release/v1.1.7 into main
syzsunshine219
marked this pull request as ready for review
September 21, 2026 11:33
syzsunshine219
requested review from
Wang-Daoji,
ZongYue99,
hijzy and
wustzdy
as code owners
September 21, 2026 11:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Synchronize the current
mainbranch into this maintenance/release branch.main@0074fc0bafcdcdde896f3d3be8d74a541791342cValidation