A foldable simulator's screen stream follows the panel in use, across folds, without a reconnect #1531

Request: none

Problem

On a foldable iOS simulator (iPhone Duo), the screen stream always shows the cover panel. Unfolded, the cover is painted black, so every viewer of the lease's stream sees black while screenshots of the panel in use show the screen. Taps through a lease's input miss on a Duo in both poses: folded they never land, and unfolded they land turned a quarter turn. Reported in #1457, which stays open until this feature ships. Proof of concept: #1525.

Who it is for

Anyone who watches or drives a leased iPhone Duo simulator: an operator on the console's Watch page, and a lease holder's own app that reads GET /v1/leases/{id}/stream and sends taps over WS /v1/leases/{id}/input, for example to test an app on both panels of a foldable.

Outcome

The picture is always the panel in use, upright, in both poses. When the device folds or unfolds, the open stream switches to the other panel within about 5 seconds, with no reconnect. A tap at a point of the picture lands on that point of the panel in use, folded or unfolded, and keeps doing so after a fold with the input socket left open. Every other device streams and takes taps as today.

How it works for the user

sequenceDiagram
  participant U as Viewer (console or app)
  participant W as Worker
  participant D as iPhone Duo
  U->>W: GET /v1/leases/{id}/stream
  W->>D: start the capture: pick the lit panel
  W-->>U: headers (size of that panel), meta, frames
  U->>W: WS /v1/leases/{id}/input
  W-->>U: screen config (that panel's size)
  Note over D: the device unfolds
  W-->>U: meta (inner panel's size), keyframe, frames
  W-->>U: screen config (inner panel's size)
  U->>W: tap (0.8, 0.3)
  W->>D: lands at (0.8, 0.3) of the inner panel
  Note over W: the log line never comes (another iOS)
  W-->>U: stays on the panel picked at open, frames go on

Before, unfolded, through the stream (from daemon.log):

Screen stream opened {"codec":"h264","width":494,"height":720}   (cover; the picture is black)

After, a stream held open across unfold, then fold:

Screen stream opened {"codec":"h264","width":494,"height":720}
[capture] picture size changed to 720x506      (inner, upright)
[capture] picture size changed to 494x720      (cover)

Examples

Which panel a stream starts on, and what it does next:

Where a tap at fraction (u, v) of the picture lands:

What could go wrong

• A new iOS release changes SpringBoard's log wording: folds are no longer followed and the picture stays on the panel it opened on. Likely over time; the log says so.
• A stream opened unfolded on an all-black screen shows black for a few seconds before it switches. Rare.
• An app viewer that reads the picture size once draws a stretched picture after a fold until it reconnects. Only apps written against today's single size.
• A physical iPhone that rotates mid-stream now changes size too; nobody has run that on hardware.

Words used

• panel: one of a foldable's two screens. The Duo's cover is the outside one (1398×2034), the inner one opens like a book (2007×2853, mounted sideways).
• cover, inner: SpringBoard's own names for the two panels.
• panel in use: the panel SpringBoard shows the picture on; the other is painted black.
• lit: a panel whose picture is not all black.
• meta frame: the first frame of a stream body, saying the codec and the picture's size; after this feature it may come again when the size changes.
• screen config: the input socket's message (0x82) that tells a client the screen's size.
• SpringBoard's log: the simulator's own system log; its DisplayContentMode lines name the panel in use after each fold, e.g. .inner:pri .cover:aux.

Non-goals

• Two viewers of one device still run two captures (One capture per lease, shared by every viewer of its screen #602).
• A rotated simulator keeps today's picture and taps.
• Physical foldables, and devices with more than two panels (they stay on the first panel and log one line).
• The half-open pose: only open and closed are followed.
• Multi-touch, scroll and edge gestures on the inner panel are sent through the same mapping but are not checked by this feature.

Completion conditions

• On a leased Duo opened unfolded, the console's Watch page and GET /v1/leases/{id}/stream show the inner panel upright from the first lit frame.
• With the stream held open, unfolding then folding the Duo switches the picture each time within 5 s of the fold command, in one response, with no reconnect; each part decodes.
• Over WS /v1/leases/{id}/input, taps at (0.2, 0.1), (0.8, 0.3) and (0.5, 0.9) land at those points of the panel in use, folded and unfolded, with the socket left open across the fold; the screen config after the fold names the new panel's size.
• An iPhone 16 lease streams and takes taps as before.
• After a SIGKILL of a Duo's capture or input helper, no log stream process for that simulator is left after 5 s.
• The website's watching-the-screen page and the stream's OpenAPI header descriptions say the size may change mid-stream.

Open questions

Decisions

• ADR 0075 — A foldable simulator streams the panel in use, and a stream may change size while it runs (docs(adr): propose 0075, a foldable simulator streams the panel in use (#1531) #1532): picks the panel, makes the log a hint, allows a mid-stream size change for every source, and kills the helpers with the capture.
• ADR 0049 — A lease holder watches and drives its device's screen: input speaks serve-sim's messages, positions are fractions of the screen, and a later screen config goes to every socket.
• ADR 0031 — A physical iPhone streams through CoreDevice: the screen-source seam this feature changes the size rule of (narrowed by 0075).
• Decided by the maintainer: the panel is picked by which one is lit when the stream opens, then followed through SpringBoard's log while it runs.
• Decided by the maintainer: a size change mid-stream is announced by a new meta frame; the size headers describe the picture at open.
• Decided by the maintainer: a physical iPhone that rotates mid-stream is rescaled the same way; listed as untested on hardware.
• Decided by the maintainer: taps follow the panel in use.
• Decided in the session: a new ADR, not an edit of 0031, because 0031 is about the physical iPhone's source.
• Decided in the session: a log line that never comes keeps the panel picked at open and logs one line, because the log text is undocumented and must never fail a stream.
• Decided in the session: the log child of a capture or input helper ends within 5 s of the helper's end, SIGKILL included, because a leaked log stream runs until the simulator shuts down.
• Decided in the session: the Swift package gets a test target, run by swift test on a Mac and listed in toolchain.md, first for the log-line parser and the tap mapping, because those break on a new iOS release and no lane runs Swift today.
• Decided in the session: the capture and the input helper share one panel follower in simlock-screen-capture, because they must agree on the panel and the input helper cannot read which panel is lit from the host.
• Decided in the session: both panels dark (or both lit) at open starts on the cover and takes the boot log's last move when it is read, because a black app must not pick the wrong panel for good.
• Decided in the session: a tap on a two-panel device goes to the panel's own digitizer target (screenID | 0x40000000: cover 0x40000001, inner 0x40000003), and the inner panel's fraction is sent as (v, 1 − u); a one-panel device keeps target 0x32. Measured on an iOS 27.1 Duo, because 0x32 misses the cover.
• Decided in the session: the screen config stays orientation: "portrait" and gives the upright size of the panel in use (cover 1398×2034, inner 2853×2007), because a client maps fractions on the picture it sees.
• Decided in the session: each simctl spawn … log runs under a /bin/sh watchdog whose stdin is a pipe only its helper holds; when the pipe closes (the helper ends any way, SIGKILL included) the watchdog kills simctl, which ends the log inside the simulator. Run on an iOS 26.5 simulator: after a SIGKILL of the parent, simctl and the in-simulator log were gone within 3 s.
• Decided in the session: which panel is lit is read by xcrun simctl io <udid> screenshot --display=<screen id> - of both panels at once, cut at 3 s, because the input helper captures no framebuffer and the capture's FrameCapture hands over one panel only; a failed or late read counts as dark.
• Decided in the session: of two panels the smaller is the cover; a panel's screen id is the one simctl io <udid> enumerate lists under the panel's deviceName (Duo: primary = 1, primary-1 = 3), because that id is also its touch target.
• Decided in the session: the inner panel's picture is turned clockwise a quarter (undoing nativeRotation 270), so it matches simctl io screenshot of that panel, the frame the taps were measured in.
• Decided in the session: the slow lane folds with agent-device fold open|closed (agent-device 0.21.35 or later, found through SIMLOCK_E2E_AGENT_DEVICE as e2e/slow-agent-device.test.ts:52 does, per agent-device help foldable) and sees taps through a small committed fixture app; the test is skipped with a reason when agent-device, the iPhone Duo device type or an iOS 27.1+ runtime is missing, because the repo has no fold of its own and no iOS slow test reads a tap today.
• Decided in the session: the switch shows within 5 s of the fold (decided by the maintainer 2026-11-12: the lane measured 3.7–4.0 s on this host); a few black frames during the fold are fine, because the PoC measured 1.2–1.5 s.

Behaviours

Stored state

Interfaces

Existing code

Commit 84c85b1.

Searches: metaEmitted (1), session.scaler (2), expectedScreenSizes (5), preferredScreenSize (3), nativeScreenSizes (3), touchTarget (3), readConfig (2), onConfig in src/ non-test (4), simlock-stream-width (1), Rotation in the website docs (1).

Tasks

flowchart LR
  A["#1534 stream follows the panel"] --> B["#1535 taps follow the panel"]

• A foldable simulator's stream shows the panel in use and switches on a fold #1534 A foldable simulator's stream shows the panel in use and switches on a fold — a viewer of a leased Duo sees the panel in use, upright, and the switch on a fold in the open stream.
  Risk: an unfolded Duo still streams black, or a log stream is left running inside the simulator.
• Taps on a foldable simulator land on the panel in use #1535 Taps on a foldable simulator land on the panel in use — a lease holder's taps land where they are sent, folded or unfolded, with the socket left open.
  Risk: taps on a Duo miss or land turned a quarter.

Completion conditions:

• Inner panel upright from the first lit frame: A foldable simulator's stream shows the panel in use and switches on a fold #1534.
• The switch within 5 s, in one response: A foldable simulator's stream shows the panel in use and switches on a fold #1534.
• Taps at (0.2, 0.1), (0.8, 0.3), (0.5, 0.9) land, and the screen config after a fold: Taps on a foldable simulator land on the panel in use #1535.
• An iPhone 16 streams and takes taps as before: A foldable simulator's stream shows the panel in use and switches on a fold #1534 (stream), Taps on a foldable simulator land on the panel in use #1535 (taps).
• No log stream left 5 s after a SIGKILL: A foldable simulator's stream shows the panel in use and switches on a fold #1534 (the capture), Taps on a foldable simulator land on the panel in use #1535 (the input helper).
• The website and OpenAPI descriptions: A foldable simulator's stream shows the panel in use and switches on a fold #1534 (stream), Taps on a foldable simulator land on the panel in use #1535 (input section).

Open work in the same area:

• One capture per lease, shared by every viewer of its screen #602 (feature:spec, one capture per lease): changes how many captures run, not what one capture shows; no Depends on, and A foldable simulator's stream shows the panel in use and switches on a fold #1534 leaves its Known gap as it is.
• iOS simulator input on Xcode 27: hardware keys, app switcher, Lock and rotation #1024 (feature:spec, iOS input on Xcode 27): changes keys, the app switcher and Lock in the same input helper (InputMode.swift, HIDInjector.swift); no task of it is ready, so no Depends on; whichever lands second rebases.
• PoC: an iPhone Duo's stream follows the panel in use (#1457) #1525 (draft PR, the proof of concept): input to A foldable simulator's stream shows the panel in use and switches on a fold #1534; closed once A foldable simulator's stream shows the panel in use and switches on a fold #1534 and Taps on a foldable simulator land on the panel in use #1535 land.
• feat(console): watch running device screens side by side #1256 (PR, the console's screens wall): reads the same stream; it takes a later meta frame as the Watch page does (ui/src/data/stream.ts), so no Depends on.

Written by an agent.