A macOS app that records your screen while you use your apps. You mark the moments that need fixing, and afterwards you get a "control center" page with the video, the marks, a timeline of what was on screen, and your narration.
./scripts/build-app.shThe build script builds, signs, installs the app to /Applications, and launches it, so you can also launch it from Spotlight or Raycast. If the app is running, the script quits it (finishing any recording first) and opens the new build. Set INSTALL_DIR= (empty) to skip installing and launching, or point it somewhere else such as ~/Applications.
If the installed Swift Command Line Tools have a PackageDescription linker mismatch, the script automatically compiles their public manifest interface into .build/manifest-api and uses it for the build. No temporary scripts or system toolchain changes are needed.
| Hotkey | Action |
|---|---|
| ⌃⌥R | Start / stop recording |
| ⌃⌥M | Mark this moment (context saved now, a frame is taken from the video later) |
| ⌃⌥S | Region screenshot: drag a rectangle (click = full screen, Esc = cancel) plus context |
The app has one home, the Sessions window, and everything else mirrors it:
- Sessions window: opens at launch and whenever you open the app again. Press Record at the bottom of the sidebar (or ⌃⌥R anywhere). The window steps aside so you're back in the app you were using.
- Recording controls: a slim floating bar that appears only while recording, with the timer, a live microphone level, the mark count and Mark / Region / Stop. If the mic is on but no audio is arriving, the meter says No audio. It's left out of the video. Drag it anywhere. Hover and click × (or right-click → Hide Controls) if it's in the way; Recording → Show Recording Controls brings it back. Hover the mark count for the latest mark.
- Menu bar: a red dot with
3:42 ⚑5(elapsed time and marks) while recording, plus the same actions and your five most recent sessions.
While a recording runs, it sits at the top of the sidebar and shows live in the window: the clock and each mark as you make it. Stop always brings the Sessions window forward on the new session. It shows processing progress (transcribing, pulling frames) and turns into the full session when it's done. If processing fails, the window shows the end of the log and a Try Again button.
Use the Mic switch on the sidebar's record card (or Recording → Record Microphone) to turn narration on or off. Turn it off for silent sessions: you still get video, screenshots and context.
The app has a Dock icon and menus while the Sessions window is open or a recording is running; otherwise it lives in the menu bar. Quitting mid-recording finishes the video first.
- Sidebar: sessions by day, with thumbnail, length, mark count, a waveform when the session has narration, and the apps you used. Right-click a session to process it again. Drop a session folder (or a folder of sessions) on the window, or use File → Open Session Folder…, to add one from elsewhere.
- Rename (right-click, or Return on the selected session) gives a session a name. It's kept in
session-info.json, so the folder keeps its timestamp; clear the name to go back to the date. - Move to Trash (right-click, or ⌘⌫) works on one session or several: ⌘- or ⇧-click to select more. Trashed sessions can be put back from the Trash. The session being recorded can't be trashed.
- Rename (right-click, or Return on the selected session) gives a session a name. It's kept in
- Stage: the recording, with floating playback controls.
- Timeline: active app, page events and narration on one time axis, with the marks pinned on the time ruler. Click or drag to seek; hover for a preview of the frame at that moment plus the app, window, narration and any mark or page event there.
- Now strip: the app, window and URL at the playhead, and the sentence being spoken.
- Inspector: three tabs, each with one job.
- Marks: what you flagged. Click a mark to open it in place: title, actions, note, the shot (with the full frame beside a region shot), what was said, the page's problems and the element under the pointer. Copy as Markdown is per mark and for the whole session (toolbar).
- Transcript: what was said. Each mark's number sits beside the sentence it landed in; click it to open the mark.
- Events: what happened. App and window switches and page events (clicks, navigation, console, requests) in one log, with problems in red and identical events in a row folded into one with a ×N count. The filter menu narrows it to one kind, such as Problems.
- Find (⌘F) filters all three tabs at once and highlights the matches. Each tab's count shows how many it has. Esc clears it.
| Key | Action |
|---|---|
| Space | Play / pause |
| J / K | Next / previous mark |
| ⌘F | Find in marks, transcript and events |
| → / ← | Skip 5 s |
| M | Add a mark at the playhead (on the canvas: at the moment under the pointer) |
| S | Capture a region of the current frame (drag; click for the whole frame; Esc cancels); on the canvas, drag over an item |
| C | Switch between the video and the canvas (once a canvas is built) |
| I / O | Start or end a selection at the playhead |
| ⇧ drag | Select a range on the timeline |
| ⌫ | Cut the selected range |
| ⌘Z / ⇧⌘Z | Undo / redo a cut |
Select a range with Shift-drag on the timeline (or I and O at the playhead) and press ⌫ or Cut. Cut anywhere, as often as you like. The video, marks, narration and every lane close up around the cut, and a scissors marker shows where it was. Click a marker to see what was removed and restore that cut. ⌘Z/⇧⌘Z undo and redo, and Session → Restore Original Recording puts everything back. Cuts never touch recording.mp4; they're stored in edits.json (see Edited sessions below).
Press E (or the pencil in the playback controls) to draw on the video. The video pauses and a toolbar shows the tools and colors, with RegionShot's keys: S select, R rectangle, A arrow, T text, H highlight (translucent, with its own color, yellow by default) and D freehand (lightly smoothed). Click with the text tool to type a note; Return commits it, ⇧Return adds a line break and Esc cancels. Double-click text to edit it, and drag its side handles to change the wrap width without changing the font size. Arrows have a handle at each end. Drag a drawing to move it, drag its handles to resize it, and press ⌫ to delete it. Changing the color recolors the selected drawing. ⌘Z undoes drawings and cuts alike. E or Esc leaves draw mode.
Each drawing shows for 3 seconds from where you drew it. The Drawn lane on the timeline has a bar for each one: drag the bar to move it in time, or drag an end to show it for longer. Drawings are stored in annotations.json at their time in the original recording, so cuts and joins never move them off their moment. Positions are fractions of the video frame, so they stay put at any window size.
Select two or more sessions in the sidebar (⌘-click or ⇧-click) and choose Join into One Session (also in the right-click menu). A new session plays them end to end, oldest first, with one timeline: marks are numbered in play order and a purple label shows where each source starts. Cuts already made in a source carry over, and you can cut the joined session like any other. The sources aren't changed or copied: the joined folder's edits.json points at them ("source": "../2026-10-13_00-57-10"), so keep them next to it. A note on a mark from a source is saved in that source, and marks added while reviewing the joined session are saved in the joined session. If a source is moved or deleted, the joined session says which one is missing. Sources of different sizes are scaled to fit one frame.
Click a mark's picture (or its Full screen inset, or View Image in a mark's menu) to open it large over the session, inside the window. Nothing opens in Quick Look or Preview.
| Input | Action |
|---|---|
| Pinch, mouse wheel, ⌘+ / ⌘− | Zoom toward the pointer |
| Two-finger scroll, drag, middle-drag, Space-drag | Pan |
| ⌘0 / ⌘1, double-click | Fit / actual pixels (double-click toggles) |
| ← / → | Previous / next image in the session |
| Esc | Close |
Annotating. The toolbar has RegionShot's tools, one key each, drawn straight onto the image:
| Key | Tool |
|---|---|
| V | Select: click a shape to select it, drag to move, drag a handle to resize, Delete removes it, arrow keys nudge it (⇧ for 10 px). Double-click text to edit it. |
| R / A / H | Rectangle, arrow, translucent highlight. Drag to draw (⇧ for squares and 45° arrows). Each tool also grabs and edits shapes of its own kind. |
| T | Text: click to type, Return to finish, ⇧Return for a new line, Esc to cancel. Drag a selected note's side handles to change where it wraps. |
| P | Freehand, lightly smoothed. |
| C | Crop: drag the area to keep, Return to apply. Reset keeps the whole image. |
Saving. Annotations save when you leave an image (← →, Esc, closing the window or switching sessions), with no dialog. The original capture is never changed: the drawing goes to annotations/<dir>-<name>.json and a flattened copy to annotations/<dir>-<name>.png, and the session is re-processed so the mark's Markdown, session.md, Copy as Markdown, the browser viewer and Export to Product Grid all use the marked-up image. session.json keeps the captures in sourceImage and sourceFrame. Annotated pictures show a pencil badge, and reopening one brings the shapes back to edit. Revert to Original clears everything (undoable until you leave the image); deleting the annotations/ folder and re-processing does the same for a whole session.
One color applies to strokes and text backgrounds, and highlight remembers its own (yellow by default). Picking a color while a shape is selected recolors it. ⌘Z and ⇧⌘Z undo and redo everything, crop included. Space-drag pans with any tool.
You can mark things you notice while reviewing, not just while recording. M (or the flag in the playback controls) adds a mark at the playhead. S (or the viewfinder) pauses and lets you drag a box on the video; the region is cut from the frame at full resolution. Every mark has a Note field, and a note becomes the mark's title.
Review marks carry what the recording knows at that moment: the app, window and URL from the activity log, the narration around it, and page events from the 30 s before. There's no live Chrome snapshot (element under the pointer, saved page). They're stored in marks.jsonl with "source": "review" and the session is re-processed, so Markdown, Copy as Markdown and export treat them like any other mark. Review marks can be deleted; marks made while recording can't.
The window updates on its own when a session is re-processed.
Build Canvas (toolbar) lays the recording out in space so you can find what went past faster than by scrubbing. Drag a box around the page content once (click for the whole frame), and the stage switches to Canvas. It reads left to right in time order:
-
A page you scrolled becomes one tall column, joined from the frames that showed it. A fixed header or footer appears once, and a sticky sidebar inside the box is cropped off.
-
Everything else is a card: one frame for each moment the content settled into a new state, plus a card now and then while it keeps moving without joining (animation, video).
-
A join is made only when the frames verifiably line up. A jump past the overlap, or rows that repeat so that several distances fit, ends the column, and the next item is labelled Not joined.
-
Hover: the now strip and the transcript show what was said when that part was on screen (in a column, the frame that supplied the row under the pointer).
-
Click: the stage goes back to the video at that exact frame.
-
M on the canvas adds a mark at the moment under the pointer. S (or the viewfinder in the canvas controls) lets you drag over an item to capture that region as a shot mark. The shot comes from the one frame that supplied the middle of the selection, clipped to what that frame showed, so a mark never mixes moments. These are ordinary review marks, with notes, Markdown and export.
-
Right-click an item for Copy Image and Export PNG…. Exported files are named with the session and the time span; a column joins frames from across that span.
-
Session menu: Rebuild Canvas (same content box), Choose Content Area and Rebuild…, Delete Canvas.
Building runs only when you ask, takes a few seconds for a few minutes of video, and never runs during recording or re-processing. It writes canvas.json and canvas/ into the session folder; delete them (or use Delete Canvas) and the session is exactly as before. If the recording changes, the canvas says so and offers Rebuild.
Export to Product Grid (toolbar) sends a session's marks to the grid stored in a project folder: <folder>/pgm/data, the folder pgm viewer uses by default, so the grid lives next to the code it describes. Exports only ever use pgm --local; nothing goes to a remote server.
- Pick a recent project folder (or Add Folder…), a project in its grid (or New Project…) and optionally a top-level node to put the session under.
- The session becomes one node (summary and narration) with one child per mark: context is the mark's Markdown, images are its screenshot and full frame.
- Exporting again updates the same nodes: their context and the images this app added are replaced, and edits made in the grid (title, status, other images) are kept. Nodes deleted in the grid are recreated. The mapping is kept in
<session>/exports/pgm.json. - Open Grid runs
pgm vieweron the folder.
Requires the pgm CLI (npm install -g @byronwallrus/product-grid-cli). viewer.html is still written to every session folder as a portable copy (Session → Open in Browser).
Push a session to your own Cloudflare Worker to watch it from any browser and let agents read it, without the recording Mac. Nothing uploads until you push.
- Cloud Settings… (⌘,): the cloud's address and Sign In. Sign-in opens the browser: enter the emailed code, then Allow. There are no tokens to paste, and the sign-in renews while you use it.
- Push to Cloud (toolbar, a session's menu, or right-click in the sidebar) uploads the session as you see it: cuts and joins flattened into one video, drawings at their moments, marks, transcript, logs, and saved pages up to a size budget (half the video, at least 10 MB).
audio.wavnever uploads. Only changed files go up, so Push Again is quick. - The toolbar and the sidebar row say when the cloud copy is out of date because you cut or drew after pushing.
- Open Cloud Copy, Copy Cloud Link and Open Cloud Library open the session or the list of every pushed session. The cloud viewer's ‹ All sessions goes back to the list, and the list shows Signed-in apps with Sign out for each.
- From another Mac: sessions you pushed from your other Macs appear in the sidebar among your sessions by date, ghosted, with the Mac they came from. Download brings one into your library as an ordinary session (drawings included, still editable), and it opens when done. If you push it again after another Mac pushed a newer copy, the app asks before replacing it.
- Agents:
claude mcp add --transport http sessions <cloud-url>/mcp, then sign in the same way. They can list sessions, read marks as Markdown with their screenshots, and get the frame at any moment.
The cloud copy is read-only; each Mac's folder is the source of truth for what it pushes. A push records <session>/exports/cloud.json. Format, API and sign-in: docs/cloud-sessions.md. Running and deploying the Worker: cloud/README.md.
To work on the window without touching an installed copy of the app, run the debug build on a folder of sessions: swift run ContextRecorder --library tmp/ (add --mark 2 to open a mark, --chrome-setup to open the Chrome extension sheet, --demo recording|processing|failed to fake those states on the newest session, or --export <project-folder> to open the export sheet; --export-run also starts the export). This skips recording, hotkeys and permission prompts. Set CR_APPEARANCE=dark to check dark mode.
First run asks for Screen Recording, Microphone and (the first time you mark in Chrome) Automation → Chrome permissions. The build is signed with a stable local identity, so macOS keeps these permissions across rebuilds.
extension/ is an MV3 extension. It sends page context to the app over ws://127.0.0.1:47823. Your web apps don't need any changes.
Chrome doesn't let another app install an extension that isn't in the Web Store, so you load it once yourself. The app guides you: click the Chrome badge on the record card (or Help → Set Up Chrome Extension…, also in the menu-bar menu while it isn't connected).
- Click Copy Folder Path and Open Chrome Extensions. The app copies the extension to
~/Library/Application Support/Context Recorder/Chrome Extension, puts that path on the clipboard and openschrome://extensions. - Turn on Developer mode, click Load unpacked, press ⌘⇧G, paste and press Return, then click Select. You can also drag the folder icon from the sheet onto the page.
- The sheet shows Connected once the extension finds the app. It retries every 30 s, so the app and Chrome can start in either order. Tabs that were already open are picked up without reloading.
Each launch refreshes that folder with the app's copy of the extension. When an older version connects, the app tells it to reload itself, so updates need no clicks.
What it records:
- As it happens (
browser.jsonl): clicks (selector, text, React/Vue component chain), SPA navigation,console.error/console.warn, uncaught exceptions and unhandled promise rejections, and failed or slowfetch/XHR requests (status, timing and the first 800 characters of an error response body). - On each mark when Chrome is in front (
mark.browser):- the element under the mouse and the focused element: selector, text, component chain, the nearest named component's props (shallow), and the source location when the page is a dev build (see below)
- the selected text, the viewport and window position, and the page's last 40 events
- the saved page (
pages/<id>.mhtml): the DOM, styles and images as they were at the mark. Open it in Chrome and inspect it with DevTools. - for region screenshots: the elements inside the region (
browser.region), each with selector, text and source
Source locations come from whatever the dev build exposes, in this order:
- React ≤ 18:
_debugSource, the exact JSX file and line. - Svelte:
__svelte_meta. - Vue 3: the component's
__file. - Source attributes added by build plugins:
data-inspector-relative-path/data-inspector-line(code-inspector),data-source-loc,data-locatorjs. Use one of these for Solid. - React 19: the
_debugStackframe. The file is right, but the line refers to the transformed output.
Production builds have none of these, so you get selectors and readable component names only.
The region is mapped to page coordinates using the pointer position, which the app and the page both know at the moment of the keypress. If the pointer wasn't over the page, it falls back to the window geometry. Browser zoom is taken into account.
Input values are never recorded, only which field changed. The app accepts connections only from chrome-extension:// origins, so web pages can't connect to it.
~/Movies/ContextRecorder/<timestamp>/
| File | What it holds |
|---|---|
recording.mp4 |
Main display at 15 fps, ≤1920 px wide, plus mic audio when enabled |
marks.jsonl |
One line per mark: time offset, kind, context (app, window, Chrome URL and title, mouse position) |
activity.jsonl |
A new line whenever the front app, window title or Chrome tab changes |
browser.jsonl |
Page events from the Chrome extension |
pages/*.mhtml |
Saved page for each mark made in Chrome |
screenshots/ |
Region screenshots |
frames/ |
Full-screen video frame for every mark |
transcript.json |
Parakeet output, with timestamps for every sentence and token |
marks/NNN.md, session.md |
Markdown for each mark, with the narration window (20 s before and 10 s after the mark; ⚑ shows where the mark landed) |
session.json |
Everything above in one file, for the viewer and exporters |
edits.json |
Only in edited sessions: the kept ranges of the recording, in play order (see below) |
viewer.html |
Portable web copy of the session. Opens straight from disk, no server needed |
canvas.json, canvas/ |
The canvas, when built: items left to right, each tied to the video frame it came from |
A session with an edits.json plays only its kept ranges:
{"segments": [{"source": ".", "start": 10, "end": 60}, {"source": ".", "start": 100, "end": 200}]}Processing writes session.json, the Markdown and the export data in edited time: marks, activity, page events and the transcript move with the video, and anything inside a removed range drops out (cut marks are kept under hiddenMarks). Each mark keeps sourceOffsetSeconds, and Markdown video references point at the original recording and time. recording.mp4 and the raw .jsonl files are never changed; marks added while reviewing an edited session are stored at their original time. Delete edits.json and re-process to get the session back as recorded.
How the time mapping is split:
process_session.pyis the only place that converts context (marks, transcript, activity, page events, narration) to edited time. It writes the resolved table tosession.json.segments: each segment with itseditedStartand source video.- Swift (
EditMapinLibrary/Edits.swift) uses that table for two things only: it builds the player'sAVMutableComposition, and it maps the playhead back to(source, original time)when you add a mark, a shot or a drawing. A session with one full-length segment plays fromAVPlayer(url:)as before. - Frames are cut by ffmpeg from the source recording at the original time, so the
frames/cache stays valid through any cut. - A joined session records
sourceSessionon each mark and points at the sources' images through relative paths; it copies nothing. Sources of different sizes play in one video track with a fit-to-frame video composition. - Every cut re-runs the whole script. With the transcript and frames cached, that takes well under a second.
The time mapping is tested by python3 -m unittest scripts/test_process_session.py and, in the app, swift run ContextRecorder --library <session> --self-test edits; both read the cases in scripts/fixtures/edit-cases.json.
To re-run processing, for example with a different narration window:
python3 scripts/process_session.py ~/Movies/ContextRecorder/<session> --before 30 --after 5Delete transcript.json first if you want the audio transcribed again.
Sources/ContextRecorder/: Swift app.Recorder(ScreenCaptureKit and AVAssetWriter),HotKeys(Carbon),ContextProbe(front app, window, Chrome tab),ActivityLog,Session.Sources/ContextRecorder/Library/: the Sessions window (SwiftUI, macOS 26).SessionLibraryscans the folders,SessionModelowns one open session and its player,SessionDatadecodessession.json,MarkTextbuilds titles and the Markdown.EditsandTimelineEditinghold cuts and joins, andAnnotationsthe drawings over the video.Sources/CanvasKit/: video → canvas analysis.FrameSourcedecodes with exact presentation times (AVFoundation, so they match the player),Segmentersplits it into runs,JoinerandRowProfileEstimatorfind and verify scrolling, andCanvasAnalyzercomposes the columns and writescanvas.json.Sources/CanvasTool/:canvas-tool, the fast loop for canvas work.swift run -c release canvas-tool checkgenerates synthetic scrolling clips with known answers (a barcode below the content encodes each frame number), analyzes them and checks the result, including that every seek shows the right frame.canvas-tool analyze <session> --box x,y,w,hbuilds a session's canvas from the command line.Sources/ContextRecorder/Cloud/: the cloud copy (CloudBundle), push (CloudClient), sign-in (CloudAuth), the push and settings UI (CloudViews), download from another Mac (CloudPull, the sidebar's ghosted cloud-only rows inCloudLibraryViews), and--cloud-bundle/--push/--cloud-list/--pull/--cloud-snapshot(CloudCLI).--self-test cloud-bundlechecks every exported frame against the edit map;--self-test cloud-clientruns push, download and sign-in against a fake Worker.cloud/: the Cloudflare Worker (TypeScript): storage, viewer, sign-in and MCP. See cloud/README.md.scripts/process_session.py: post-processing (ffmpeg, parakeet-mlx). Standard library only.scripts/viewer.html: viewer template. The session data gets embedded in it, or, served from the cloud, it fetchessession.jsonbeside it.site/: the home page at byronwall.github.io/context-recorder. It is static HTML, CSS and one script (player.js, the demo player);.github/workflows/pages.ymldeploys it on each push tomainthat changes it.scripts/site-demo/make_session.pybuilds the staged demo session and copies it tosite/demo/, where the player and the browser viewer read it.scripts/site-demo/shoot.shcaptures the Sessions window on it forsite/img/.scripts/make-icon.swift: draws the app icon and writesResources/AppIcon.icns(swift scripts/make-icon.swift).
Requires macOS 26, the Command Line Tools (Xcode isn't needed), ffmpeg, and parakeet-mlx on the login-shell PATH.
- rrweb DOM recording, to replay the DOM over time alongside the video.
- Session editing, after cuts, joins and drawings: suggested cuts (silence, idle time, "keep marks ± N s"), video drawings in mark frames, a blur tool, MP4 export with video drawings burned in, joined sessions in the local
viewer.html, and reordering or picking ranges across sources. - Video to canvas: built (see Canvas above).
- Cloud sessions: built (see Cloud above). Share links and frames from Media Transformations are next.