Skip to content

One capture per lease, shared by every viewer of its screen #602

Open

Description

@V3RON

Request: none

Problem

Every viewer of a device's screen starts its own capture on the worker (#593). Two people watching one device cost two capture programs on iOS (a second encoder) or two scrcpy servers on an Android emulator. Through a gateway each viewer also holds its own relay channel. A worker link carries at most 64 at once and the 65th waits, so many viewers crowd out agent-device calls to that worker.

A viewer that stops reading also slows its own capture to its pace, because nothing is ever dropped.

Who it is for

Operators and agents who watch the same device at once: a team following one agent's run, or a dashboard beside a person's console tab.

Outcome

  • Any number of viewers of one device share one capture on its worker, whichever path they asked by (the device path or a lease path).
  • A viewer that joins late sees a picture within 5 seconds, on a still screen too, on every platform: iOS simulator, iPhone, Android emulator and Android phone.
  • A slow viewer misses frames instead of slowing the others.
  • The capture stops when the last viewer leaves, or when the device stops or its lease ends, as today.
  • A gateway sends one relayed stream per device (or per lease) to a worker, however many viewers it has.

How it works for the user

sequenceDiagram
  participant A as Viewer A
  participant B as Viewer B (joins later)
  participant G as Gateway
  participant W as Worker
  participant D as Device capture
  A->>G: GET /v1/stream/workers/w1/devices/d1
  G->>W: relay /stream/devices/d1 (the only one)
  W->>D: open the one capture
  D-->>W: meta, parameter sets, key frame, deltas...
  W-->>G: frames
  G-->>A: frames
  B->>G: GET the same device
  Note over G: no second relay
  G-->>B: headers (current size), meta, parameter sets
  D-->>W: key frame (at most 5 s later)
  G-->>B: key frame, then live frames
  A-xG: A closes its tab
  Note over G,W: B still watching, nothing stops
  B-xG: B closes its tab
  G-xW: relay ends, capture stops
Loading

Failure paths:

  • The capture cannot start: every viewer that was waiting for it gets the same refusal (for example 503 STREAM_UNAVAILABLE). The next viewer tries a fresh capture.
  • The capture ends or fails mid-stream: every viewer's stream ends the same way. The console reopens a stream that ended with no reason, as today.
  • A lease ends: every viewer gets the one last frame that says why, then the end.
  • A viewer stops reading (a background tab, a slow link): it skips frames up to the next key frame. The others see no change.

What a person sees, before and after, with two console tabs on one leased simulator:

before: ps -ax | grep -c '[s]imlock-screen-capture'   → 2
after:  ps -ax | grep -c '[s]imlock-screen-capture'   → 1

The second tab, opened on a still screen: today it shows a picture at once (its own capture). After this, it shows "starting" for at most 5 seconds, then the picture.

Examples

What a viewer that joins a running capture gets first (iOS or Android, H.264):

The shared capture has sent so far Today (own capture) After
meta, parameter sets, key frame, 40 deltas its own meta, parameter sets, key frame meta, parameter sets, then nothing until the next key frame (≤ 5 s), then key frame and live
meta 494×720, …, meta 720×506 (a fold), parameter sets, key frame, deltas its own capture at the current size headers 720×506, meta 720×506, its parameter sets, the next key frame
nothing yet (still starting) its own capture starting waits for the same start as the first viewer, gets everything from meta on
an MJPEG capture (no H.264) its own capture meta, then the next picture: every MJPEG picture is whole
a lease end's last frame — the last frame, then the end

What a viewer whose queue is full gets:

Queued for it, then After
deltas past the bound, a new delta arrives queued deltas dropped, new deltas skipped until the next key frame
full, then a meta or parameter sets arrive kept: they are never dropped
full, then the lease ends the last frame is kept, then the end

What could go wrong

  • A viewer that joins on a still screen sees "starting" for up to 5 seconds instead of a picture at once. Likely, and by design.
  • On Android a still screen restarts scrcpy's encoder every 5 seconds. If a device's encoder handles that badly, the picture could flicker or stall. Unlikely: the emulator check showed a clean new key frame within 0.1 s.
  • A picture slightly softer while moving, because key frames come more often at the same bitrate. Mild.
  • A bug in sharing would hit every viewer of a device at once, not only one.

Words used

  • capture: the program on the worker that films one device's screen: simlock-screen-capture (iOS simulator), simlock-device-capture (iPhone) or a scrcpy server (Android).
  • key frame: a whole picture a decoder can start from, e.g. tag 0x02. Every other picture (a delta, tag 0x03) only says what changed since the frame before.
  • parameter sets: the decoder's setup (the description frame, tag 0x01), sent once per encoder. A viewer needs it before its first key frame.
  • meta frame: the picture's codec and size (tag 0x05); a new one comes when the size changes.
  • reset video: scrcpy's control message 17, which restarts its encoder; a new session, parameter sets and a key frame follow within about 0.1 s (measured on an API 35 emulator).
  • viewer queue: the frames read from the capture that one viewer has not taken yet.

Non-goals

  • No viewer limit per device.
  • No replay of the frames since the last key frame: a joiner waits for the next one.
  • No on-demand key frame when a viewer joins; the 5-second bound is the only promise.
  • No sharing between a device-path relay and a lease-path relay on a gateway: they are two relays to the worker, which still runs one capture.
  • No change to the console, the HTTP API shape or the frame envelope.

Completion conditions

  • Two viewers of one iOS simulator, one opened by its device path and one by its lease path, run one simlock-screen-capture process on the worker.
  • Two viewers of one Android emulator run one scrcpy video server on it.
  • A viewer that opens the stream of a still iOS simulator, a still Android emulator, while another viewer watches it, gets a key frame within 5 seconds.
  • A viewer that stops reading for 30 seconds does not delay another viewer's frames, and gets a key frame once it reads again.
  • Through a gateway, three viewers of one device hold one relay channel to the worker.
  • The capture stops once the last viewer leaves, and when the lease ends every viewer gets the lease-end frame.

Open questions

none

Decisions

  • ADR 0079 (PR docs(adr): propose 0079, one capture per device is shared and every capture sends a key frame every 5 s (#602) #1593) — One capture per device, shared on the worker and the gateway, with a key frame at least every 5 s: what is shared, what the gateway reads, the key-frame promise, what a slow viewer gets. Narrows ADR 0045 §5 and the stream rule that a gateway relays the body unread.
  • ADR 0045 — A device's screen streams to operators: §5 already keys the shared capture by device; §4's ends stay.
  • Maintainer: sharing on the worker and on the gateway.
  • Maintainer: no limit on viewers per device.
  • Maintainer: a late joiner waits for the next key frame, and every capture sends one at least every 5 s, still screens included.
  • Decided in the session: a slow viewer drops frames up to its next key frame and is never cut off, because its queue is bounded (B7) and costs little.
  • Decided in the session: the capture stops as soon as the last viewer leaves, as today, because the issue asks for it and the console reconnects within seconds anyway.
  • Decided in the session: the Android driver sends reset video after 5 s with no key frame, and sets scrcpy's i-frame-interval to 2, because the emulator check showed scrcpy's interval counts frames at a nominal 60 fps and sends nothing on a still screen (0 frames in 12 s; no key frame in 16 s of scrolling at the default).
  • Decided in the session: iOS forces a key frame when 5 s of wall clock have passed since the last one, in the shared encode loop, because the encoder's own interval counts frames (fps × 5), which is about 30 s on a still simulator, and its timestamps are a frame counter, so a duration limit would count frames too.
  • Decided in the session: a gateway shares a relay per worker and what the path names (a device, or the worker's lease id), because a gateway does not know a lease's device before the worker answers; the worker still runs one capture for both.
  • Decided in the session: a shared relay is started with the gateway's own signal and the first viewer's tenant, because the first viewer leaving must not end it, and the worker's relay door serves any tenant as an operator.
  • Decided in the session: a joiner's stream starts at a key frame, and the server skips the deltas before it, so the console and other viewers need no change.
  • Decided in the session: a viewer queue holds at most 2 MiB (about 10 s at today's 1500 kbps), because that rides out a short stall without a drop and bounds a stuck viewer's memory.

Behaviours

Id Behaviour (invoker, action, what it sees) Seam Cases Task
B1 An iOS capture (simulator or iPhone) given frames writes a key frame at least every 5 s of frame time, still screen included; the first frame is still a key frame. encodeLoop in vendor/screen-capture/Sources/SimlockCapture/Lifecycle.swift, through EncodeLoopTests.swift with fake frames idle-floor frames (5/s) for 12 s: key frames at 0 s, ≤ 5 s, ≤ 10 s; 30 frames/s for 12 s: the same; an encoder-chosen key frame at 3 s moves the next forced one to ≤ 8 s; a size change still gets meta, parameter sets and a key frame (existing test) T1
B2 A viewer of a still iOS simulator gets a key frame at least every 5 s. slow lane, one simulator still home screen for 15 s: ≥ 3 key frames, no gap over 5.5 s T1
B3 An Android stream starts its scrcpy server with the control channel on, power_on=false, clipboard_autosync=false and video_codec_options=i-frame-interval:int=2; the control socket is read and discarded. src/drivers/android/screen/session.ts, through session.test.ts with a fake target the server arguments (I4); a phone keeps stay_awake=true, an emulator has none; the control socket's bytes are discarded; the control socket ending before close errors the body with the server's last output, as the video socket's end does T2
B4 An Android stream with no key frame for 5 s sends reset video on the control socket; the new session's meta, parameter sets and key frame go out as a rotation's do today. session.ts through session.test.ts with a fake clock and target still screen: one reset at 5 s, another at 10 s if none came; a key frame at 4 s: no reset at 5 s, next due at 9 s; a reset whose write fails errors the body; the session after a reset writes meta, description, key frame (existing envelope rule) T2
B5 A viewer of a still Android emulator gets a key frame at least every 5 s. slow lane, one emulator still home screen for 15 s: ≥ 3 key frames, no gap over 5.5 s T2
B6 Two or more viewers of one device on a worker, by the device path, a lease path or both, share one capture: one openScreen call; each gets 200 with the current meta in its headers, and a body that starts with the current meta frame, the current parameter sets, then frames from the next key frame. createWorkerStream (src/stream/create.ts) through create.test.ts with a fake ScreenOpener two device-path viewers; a device-path and a lease-path viewer; a second viewer while the first's open is still pending (one open, both answered from its meta); a joiner after a size change (headers and first meta give the new size); a joiner before any parameter sets; an MJPEG capture (no wait for a key frame); a viewer refused by the device checks (agent on a device path, unknown device, not running) joins nothing T3
B7 A viewer that stops reading on a worker never delays the capture or another viewer: past 2 MiB queued it drops its queued frames and skips new ones up to the next key frame; meta, parameter sets and the lease-end frame are never dropped. create.test.ts with a fake capture one viewer stalled while another reads 2,000 frames; the stalled one reads again and gets meta and parameter sets it missed, then a key frame; a full queue when the lease ends still ends with the lease-end frame; alive frames are dropped with deltas T3
B8 On a worker the capture is closed once, when its last viewer leaves; a viewer that leaves while others watch closes nothing. create.test.ts two viewers, the first leaves (no close), the second leaves (one close); a viewer with expiresAt reaches it (its stream ends, the capture goes on for the other); every viewer leaves during the open (the open's signal aborts, what it attached is closed once); one leaves during the open and another stays (the open goes on) T3
B9 A lease end or a device stop ends every viewer of the device the way it ends one today: each gets the lease-end frame when there is a reason, then the end; the reclaim waits for the one close. create.test.ts lease end with three viewers (three bodies end with one ended frame each, one close); device.shutdown with two viewers (no frame); device.absence-detected on a leased device (device-lost frame once); a viewer that asks after the stop gets a fresh capture or DEVICE_NOT_RUNNING as today T3
B10 A capture that fails to start refuses every viewer waiting for it with the same refusal; one that fails mid-stream errors every viewer's body; the next viewer starts a fresh capture. create.test.ts start fails with ScreenUnavailableError (two viewers, 503 STREAM_UNAVAILABLE each, same message); start exceeds the start bound; mid-stream read error; silence bound passes (every body errors); the device stops during a shared start (DEVICE_STOPPED to each); after a failure a new viewer gets a new openScreen T3
B11 Two or more viewers of one device through a gateway share one relay to the worker per worker and target (a device, or the worker's lease id); a device-path and a lease-path viewer use one relay each. createGatewayStream (src/stream/create.ts) through gateway-link.test.ts with a fake FleetPort three device-path viewers, one relay call; the lease's owner and an operator on one lease path, one call; a device-path and a lease-path viewer, two calls; an agent on a device path and an unknown worker are refused before joining (no call) T4
B12 On a gateway a joiner, a slow viewer and the last viewer leaving behave as on a worker (B6–B8): the joiner gets headers from the relay's current meta, then meta, parameter sets and the next key frame; a slow viewer drops up to its next key frame; the relay's body is cancelled once, when its last viewer leaves; the first viewer leaving does not end it. gateway-link.test.ts joiner after the relay's first key frame; joiner after a later meta frame; a stalled viewer while another reads; first viewer leaves, the second still gets frames; last leaves, the relay's request is aborted once T4
B13 A shared relay that is refused or fails refuses or ends every viewer waiting on it the same way; a body that breaks the frame envelope errors every viewer and ends the relay. gateway-link.test.ts worker refusal 409 DEVICE_NOT_RUNNING to two viewers; WORKER_UNREACHABLE to two viewers; a relayed frame over MAX_FRAME_BYTES; a relayed body whose first frame is not a meta frame; the relay body ending (the lease-end frame then the end, to each) T4

Stored state

Id What and where Created, changed, deleted by Listed or swept by A restart mid-step leaves Schema step Task Used by
S1 A shared capture per device, in the worker's memory (src/stream/captures.ts): the capture slot, the broadcast (I1) and its viewers created by the first viewer that passes the device checks; a viewer joins or leaves; deleted when the last leaves, the capture ends or fails, or the slot is stopped the lease end and the device events (stopOnDevice), as today nothing: the process is gone, the capture program ends when its stdin or socket closes, as today none T3 T4
S2 A shared relay per worker and target, in the gateway's memory (src/stream/gateway-link.ts): the relayed response, its broadcast (I1), its abort controller created by the first viewer that passes the gateway's checks; deleted when the last viewer leaves, or the relay ends or fails — nothing: the relay channel closes with the process, and the worker's capture ends with it, as today none T4 —

Interfaces

Id What Shape and bounds Errors and limits Where Task Used by
I1 The broadcast: one frame source shared by viewers. createBroadcast(source: CapturedStream, options: { onIdle(): Promise<void> }): Broadcast; Broadcast.join(signal: AbortSignal): CapturedStream (its meta the current one, its close leaves); Broadcast.viewers: number reads the source as fast as it delivers, whatever the viewers do; keeps the last meta frame and the last parameter sets; a joiner's body is the last meta frame, the last parameter sets (H.264 only), then frames from the next key frame (MJPEG: the next frame); a viewer queue is bounded at MAX_VIEWER_QUEUE_BYTES = 2 MiB, past which it drops queued 0x02/0x03/0x04/0x06 frames and skips new ones up to the next key frame, keeping 0x01, 0x05, 0x07; onIdle runs once, when the last viewer leaves; the source's end ends every body, its error errors every body with that error a frame the splitter (I2) refuses errors every body with its message and runs onIdle src/stream/broadcast.ts (new) T3 T4
I2 The frame splitter: FrameSplitter.push(chunk: Uint8Array): Frame[], Frame = { tag: number; bytes: Uint8Array } (the whole frame, header included) reads only the 4-byte length and the tag byte; a frame is at most MAX_FRAME_BYTES (8 MiB) over the bound: The screen stream sent a frame of <n> bytes, over the 8388608 a frame may be; a length of 0: The screen stream's output is not a frame envelope src/stream/framing.ts T3 T4
I3 iOS key-frame bound: the encode loop forces a key frame when KEYFRAME_MAX_GAP_NANOS = 5 s of frame arrival time (frame.atNanos) have passed since the last key frame it wrote, whoever chose it the first frame stays forced; MaxKeyFrameInterval stays fps × 5 — vendor/screen-capture/Sources/SimlockCapture/Lifecycle.swift T1 —
I4 Android stream server arguments: control=false becomes control=true, plus power_on=false, clipboard_autosync=false, video_codec_options=i-frame-interval:int=2; the second socket of the scid is the control socket, connected after the video socket scrcpy v5.0.1 Options.java (control, power_on, clipboard_autosync, video_codec_options) the control socket's connect shares the start budget (START_DEADLINE_MS) src/drivers/android/screen/session.ts, scrcpy-link.ts T2 —
I5 Reset video: one byte 0x11 (scrcpy TYPE_RESET_VIDEO = 17) on the control socket, sent when KEYFRAME_OVERDUE_MS = 5,000 ms pass on the clock since the last key frame (or the session's start) the timer restarts on each key frame and after each reset a failed write errors the body with The screen server's control socket failed: <reason> and closes the feed src/drivers/android/screen/session.ts T2 —

Existing code

Commit 5e639b4.

Searches: beginOnDevice|stopOnDevice (4 files), openScreen (6), readScreenFeed (3), fleet.relay|relayPath (2), control=false|control=true (2: screen session.ts:256, input device-input.ts:180), MaxKeyFrameInterval|forceKeyframe (2 Swift files), awaitingKeyframe (1, ui/src/data/stream.ts), Each viewer is|#602|Two viewers in docs (5).

Id Code (path:line) Today With this feature Handled by
X1 src/stream/worker-link.ts:57-96 serve every request registers a slot and calls openScreen, and its viewer's abort releases its slot the first viewer opens; later viewers join the device's broadcast; a viewer's abort leaves it B6, B8, T3
X2 src/stream/captures.ts:47 running: Map<string, Set<Slot>> a set of slots per device, one per viewer one slot per device, holding the shared capture (S1); stopOnDevice still stops it and waits for its close S1, B9, T3
X3 src/stream/feed.ts:27-65 readScreenFeed one reader per capture, pull-based, with the start and silence bounds and the lease-end last frame unchanged; its body is the broadcast's source, read by the broadcast alone I1, B10, T3
X4 src/stream/service.ts:123-156 endingAt wraps a viewer's body with its expiresAt deadline unchanged: it wraps a viewer's own body from join, so one viewer's deadline ends only that viewer B8, T3
X5 src/stream/gateway-link.ts:53-62 serveDevice one fleet.relay per viewer, with the viewer's request.signal and tenant one relay per worker and target, with the gateway's own signal and the first viewer's tenant; viewers join its broadcast S2, B11, T4
X6 src/stream/gateway-link.ts:151-160 fromResponse the body is relayed unread the body goes through the frame splitter (I2), which reads lengths and tags, never a payload I2, B13, T4
X7 src/drivers/android/screen/session.ts:251-268 control=false, no codec options I4's arguments; a control socket; the reset timer (I5) B3, B4, T2
X8 src/drivers/android/screen/session.ts:376 started on the first key frame, ALIVE_AFTER_MS 5 s alive frames after 5 s of quiet unchanged; the reset timer runs beside it and a reset's key frame resets the alive timer as any frame does B4, T2
X9 src/drivers/android/screen/envelope.ts:30-73 a scrcpy session event writes a new meta frame, then the description before the next picture unchanged: a reset's new session goes through it B4, T2
X10 vendor/screen-capture/Sources/SimlockCapture/Lifecycle.swift:223 forceKeyframe = !started, H264Encoder.swift:212 MaxKeyFrameInterval fps * 5, H264Encoder.swift:96-97 timestamps from a frame counter a key frame forced only first; the encoder's own interval counts frames I3: also forced after 5 s of arrival time B1, T1
X11 vendor/screen-capture/Sources/simlock-device-capture/DeviceScreenSource.swift:648-655 keepAlive a still iPhone's last picture is fed every 200 ms unchanged: it is what lets I3 fire on a still iPhone B1, T1
X12 ui/src/data/stream.ts:100-104, 184-190 the decoder starts with awaitingKeyframe = false, needs a description first unchanged: a joiner's body (I1) starts with a description and its first picture is a key frame B6
X13 src/drivers/android/input/device-input.ts:180 the input's own control-only server per lease, its own scid unchanged: the stream's control socket is another scid's B3
X14 src/stream/AGENTS.md (Gateway: "relays the body unread"; Known gaps: "Each viewer is its own capture", "A frame skipped by a slow viewer is not dropped"), src/drivers/android/screen/AGENTS.md (rules "changes no device setting", "scrcpy sends nothing for a still screen"; gaps "Key frames come at scrcpy's default interval", "Two viewers of one lease"), src/drivers/ios/AGENTS.md ("Two viewers ask for two streams (#602)") describe one capture per viewer the gaps go, the gateway rule says it reads lengths and tags only, the Android rules name the control socket and the reset T1, T2, T3, T4
X15 website/src/content/docs/docs/using-devices/watching-the-screen.mdx:78 "Each viewer is a separate capture." says two tabs capture twice viewers share one capture; a joiner can wait up to 5 s for its picture; a slow viewer skips frames T3
X16 docs/internal/adr/0045-a-devices-screen-streams-to-operators-not-a-leases.md §5 "One capture per viewer, as today", left to #602 narrowed by ADR 0079 ADR 0079
X17 src/link/index.ts:32,125-139 OPEN_LIMIT = 64, the 65th openChannel waits each viewer through a gateway holds one one per shared relay (S2) B11, T4

Tasks

To be written after approval.

Written by an agent.

Activity

  1. added
    feature:specSpec in progress, business and technical together.
    on Oct 15, 2026
  2. V3RON commented on Nov 16, 2026

    @V3RON
    ContributorAuthor

    Design posted: the business sections were rewritten from the spec session. Decisions, Behaviours B1–B13, Stored state S1–S2, Interfaces I1–I5 and Existing code X1–X17 were added. ADR 0079 is proposed in #1593.

    Written by an agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature:specSpec in progress, business and technical together.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions