diff --git a/docs/internal/adr/0023-agent-device-is-served-by-an-optional-edge-module.md b/docs/internal/adr/0023-agent-device-is-served-by-an-optional-edge-module.md index d4a319e79..0accb0b5a 100644 --- a/docs/internal/adr/0023-agent-device-is-served-by-an-optional-edge-module.md +++ b/docs/internal/adr/0023-agent-device-is-served-by-an-optional-edge-module.md @@ -1,6 +1,6 @@ # 0023. agent-device is served by an optional edge module, and the simlock lease is the only clock -- **Status:** Accepted; §1 narrowed by 0026 for gateway mode; §3 narrowed by 0030 (no reap while turned off); §3–4 superseded in part by [ADR 0043](0043-physical-devices-reach-agent-device-through-a-second-daemon.md); §1, §3 and §4 narrowed by [ADR 0046](0046-a-standalone-worker-serves-agent-device-by-default-with-a-private-profile.md) for a standalone worker; §2 and §4 narrowed by [ADR 0070](0070-test-runners-drive-a-lease-through-the-agent-device-front.md) +- **Status:** Accepted; §1 narrowed by 0026 for gateway mode; §3 narrowed by 0030 (no reap while turned off); §3–4 superseded in part by [ADR 0043](0043-physical-devices-reach-agent-device-through-a-second-daemon.md); §1, §3 and §4 narrowed by [ADR 0046](0046-a-standalone-worker-serves-agent-device-by-default-with-a-private-profile.md) for a standalone worker; §2 and §4 narrowed by [ADR 0070](0070-test-runners-drive-a-lease-through-the-agent-device-front.md); §3 superseded in part by [ADR 0071](0071-a-restart-re-allocates-a-live-agent-device-lease.md) - **Date:** 2026-10-12 - **Issue:** [#466](https://github.com/callstackincubator/simlock/issues/466) - **Supersedes:** nothing. Builds on [ADR 0020](0020-a-requester-may-choose-its-lease-id.md) diff --git a/docs/internal/adr/0070-test-runners-drive-a-lease-through-the-agent-device-front.md b/docs/internal/adr/0070-test-runners-drive-a-lease-through-the-agent-device-front.md index 6777a46d4..79b65fce7 100644 --- a/docs/internal/adr/0070-test-runners-drive-a-lease-through-the-agent-device-front.md +++ b/docs/internal/adr/0070-test-runners-drive-a-lease-through-the-agent-device-front.md @@ -1,6 +1,6 @@ # 0070. Test runners drive a lease through the agent-device front: boot is answered, a close may keep the lease, and each lease is its own tenant -- **Status:** Accepted +- **Status:** Accepted; §2 narrowed by [ADR 0071](0071-a-restart-re-allocates-a-live-agent-device-lease.md) - **Date:** 2026-11-07 - **Issue:** [#1419](https://github.com/callstackincubator/simlock/issues/1419) (request [#1292](https://github.com/callstackincubator/simlock/issues/1292)) - **Narrows:** [ADR 0023](0023-agent-device-is-served-by-an-optional-edge-module.md) diff --git a/docs/internal/adr/0071-a-restart-re-allocates-a-live-agent-device-lease.md b/docs/internal/adr/0071-a-restart-re-allocates-a-live-agent-device-lease.md new file mode 100644 index 000000000..10ca62f35 --- /dev/null +++ b/docs/internal/adr/0071-a-restart-re-allocates-a-live-agent-device-lease.md @@ -0,0 +1,165 @@ +# 0071. A restart re-allocates a live agent-device lease on its device + +- **Status:** Accepted — not yet implemented +- **Date:** 2026-11-09 +- **Issue:** [#1461](https://github.com/callstackincubator/simlock/issues/1461) (bug [#1103](https://github.com/callstackincubator/simlock/issues/1103)) +- **Supersedes in part:** [ADR 0023](0023-agent-device-is-served-by-an-optional-edge-module.md) + §3 ("A restart loses its agent-device leases; the front answers their + next heartbeat as ended by simlock and releases the simlock lease") and + the Consequences line "the leases survive and end on their next + heartbeat or at their TTL". +- **Narrows:** [ADR 0070](0070-test-runners-drive-a-lease-through-the-agent-device-front.md) + §2: a lease re-allocated from an entry rebuilt after a worker restart + does not carry `retainOnClose`; one re-allocated from a surviving entry + does. + +## Context + +agent-device keeps its leases in memory. When its daemon restarts, alone or +with the worker, it forgets them. simlock keeps the simlock lease of a +device that is still running (ADR 0019), so `simlock status` shows the +lease. The agent's next heartbeat gets `UNAUTHORIZED` / `LEASE_NOT_FOUND` +from agent-device, and the front releases the simlock lease +(`src/agent-device/front/heartbeat.ts:154`). The device is wiped, and the +agent is told `lease ended by simlock; run disconnect, then open`. + +The agent-device CLI heartbeats before each command and sends its lease +scope with it: `runId`, `clientId`, `leaseProvider`, `leaseBackend` and +`deviceKey` (agent-device v0.21.35, +`src/cli/commands/connection-runtime.ts:892-941`). agent-device accepts an +allocate with a lease id the caller chooses, unless that daemon already +issued the id (`LEASE_ID_IN_USE`, +`packages/daemon/src/resources/lease-registry.ts:81-98`). The issued ids +live in memory, so a restarted daemon accepts it. So the +front has what it needs to allocate the same lease again, as long as it +knows the device. + +```mermaid +sequenceDiagram + participant CLI as agent-device CLI + participant F as simlock front + participant AD as agent-device daemon (restarted) + participant L as simlock leasing + CLI->>F: heartbeat (lease id, scope) + F->>AD: heartbeat + AD-->>F: UNAUTHORIZED LEASE_NOT_FOUND + Note over F: today + F->>L: lease.release + F-->>CLI: lease ended by simlock +``` + +## Decision + +### 1. `LEASE_NOT_FOUND` on a lease simlock still holds re-allocates it + +When agent-device answers a heartbeat with `LEASE_NOT_FOUND`, the front +asks simlock whether the caller still holds the lease: leasing's +`lease.list` names it, on a worker or a gateway. If it does, and agent-device did not end the lease itself (`endedLeases`), the +front: + +1. sends `agent_device.lease.allocate` with the same lease id, the lease's + device address, the scope the heartbeat carried, and the `ttlMs` ceiling + a first allocate gets; +2. sends the heartbeat again and answers with its reply. + +The re-allocate keeps the heartbeat's `deviceKey`, which a first allocate +drops: agent-device refuses a later heartbeat whose key the lease lacks. +It also keeps `retainOnClose` (ADR 0070 §2) when the front's map entry +stores it: the entry records it when the lease is allocated, so a +re-allocate from a surviving entry carries it. An entry rebuilt after a +standalone worker restart has none (§4). + +The agent keeps the same simlock lease, the same device and the same app +data. Its agent-device session does not survive (simlock empties the +daemon's sessions folder at start), so the agent may need `open` again; +`open` works on the same device. +Commands, recordings and session state that the restart interrupted are not +recovered. + +```mermaid +sequenceDiagram + participant CLI as agent-device CLI + participant F as simlock front + participant AD as agent-device daemon (restarted) + participant L as simlock leasing + CLI->>F: heartbeat (lease id, scope) + F->>AD: heartbeat + AD-->>F: LEASE_NOT_FOUND + F->>L: still held by this caller? + L-->>F: yes + F->>AD: allocate (same id, same device, same scope) + AD-->>F: ok + F->>AD: heartbeat + AD-->>F: ok + F-->>CLI: ok +``` + +Only one re-allocate runs for a lease at a time. A heartbeat that arrives +while one runs waits for it. + +### 2. What stays final + +- `LEASE_EXPIRED`, `LEASE_REVOKED`, and a lease agent-device ended itself + (`endedLeases`): as today, the simlock lease is released. +- A lease simlock no longer holds, released or expired: as today, the + answer is ended by simlock. agent-device reports its own expired lease as + `LEASE_NOT_FOUND` too. Since simlock's lease is the only clock, the "still + held" check keeps that case final. +- Nothing revives a lease that has ended. + +### 3. A failed re-allocate keeps the simlock lease + +If agent-device refuses the allocate (for example `LEASE_ID_IN_USE` on a +daemon that did not restart) or does not answer, the front answers with a retryable error and does not release the +simlock lease. The next heartbeat tries again. While agent-device is +still starting, the heartbeat itself gets `…: starting`, as today, and no +re-allocate is sent. The lease ends only by +release or at its TTL, as any simlock lease does. + +On a gateway, a lease the worker's list leaves out for a moment while the +worker starts (the gateway's index still holds it) also gets the +retryable answer, not a release. + +### 4. Where the front finds the device + +- **agent-device restarted alone:** the front's map still has the entry. +- **Gateway:** the gateway's map has the entry, or rebuilds it from the + worker's lease list after a gateway restart, as today. A worker restart, + a gateway restart, or both in either order are covered. +- **Standalone worker restarted:** its map is empty. The worker's front + rebuilds entries the way the gateway does (`adoptable`): from the + registry snapshot the module already reads, its leases (one whose + `requesterId` is its own id and whose `ownerId` is the caller) and their + devices (address, platform, model, physical). Nothing new is stored. + +A phone lease's re-allocate names the phone's serial, so it goes to the +phone daemon as a first allocate does. + +## Consequences + +- A worker restart, or a crash like #1409, no longer wipes every + agent-device lease on the worker. Agents see retryable errors for the + restart (measured by the reporter at about 7 s on Android and 28 s on iOS), + then `open` again. +- `retainOnClose` (ADR 0070 §2) is stored on the front's map entry when + the lease is allocated, so an agent-device restart alone keeps it. It is + in neither leasing's record nor the heartbeat, so an entry rebuilt from + leasing (a standalone worker restart, or a gateway restart followed by + a worker restart) does not carry it, and a close after that ends the + lease. A worker restart already breaks + that test run. Tracked as its own issue. +- A lease whose re-allocate keeps failing holds its device until its TTL, + not until the next heartbeat. +- The edge tests that pin today's release after a restart change. + +## Alternatives considered + +- **Persist leases inside agent-device.** It removes the second allocate, + but it changes another tool. Rejected by the maintainer. +- **Save front entries in `state.db`.** It keeps `retainOnClose`, but it + adds a table and a migration, and it copies facts leasing already holds. + Rejected: rebuilding from leasing covers every other field. +- **Re-allocate on every command, not only the heartbeat.** The CLI sends a + heartbeat before each command anyway, so this only adds paths. +- **Also re-allocate on `LEASE_EXPIRED` and `LEASE_REVOKED`.** Rejected: + those are ends a person or the clock chose. diff --git a/docs/internal/adr/README.md b/docs/internal/adr/README.md index b60b6c38f..3ed0fed7e 100644 --- a/docs/internal/adr/README.md +++ b/docs/internal/adr/README.md @@ -61,7 +61,7 @@ the status is stale. | [0020](0020-a-requester-may-choose-its-lease-id.md) | A requester may choose its lease id, and the gateway passes it through bare | Accepted | | [0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md) | A gateway dispatch is a probe, and the gateway records every fleet request's outcome | Accepted — partly implemented: §5 (usage figures) is not built | | [0022](0022-a-physical-device-is-a-second-device-type-the-core-knows.md) | A physical device is a second device type the core knows | Proposed; amends 0001 §3 and §5, 0015 §3 and 0019 §1–2 for physical devices | -| [0023](0023-agent-device-is-served-by-an-optional-edge-module.md) | agent-device is served by an optional edge module, and the simlock lease is the only clock | Accepted; §1 narrowed by 0026 for gateway mode; §3 narrowed by 0030 (no reap while turned off); §3–4 superseded in part by 0043; §1, §3–4 narrowed by 0046; §2 and §4 narrowed by 0070 | +| [0023](0023-agent-device-is-served-by-an-optional-edge-module.md) | agent-device is served by an optional edge module, and the simlock lease is the only clock | Accepted; §1 narrowed by 0026 for gateway mode; §3 narrowed by 0030 (no reap while turned off); §3–4 superseded in part by 0043; §1, §3–4 narrowed by 0046; §2 and §4 narrowed by 0070; §3 superseded in part by 0071 | | [0024](0024-leasing-calls-lease-end-participants-before-reclaim.md) | Leasing calls lease-end participants before it reclaims a device | Accepted | | [0025](0025-simlock-ships-android-profiles-and-a-model-pairs-only-with-images-that-fit.md) | Simlock ships Android device profiles, and a model pairs only with images that fit it | Accepted | | [0026](0026-agent-device-through-a-gateway-relays-over-the-uplink.md) | agent-device through a gateway: workers advertise capabilities, and the host link relays over the uplink | Accepted; §2 narrowed by 0028 for relayed bodies; §3's in-process link narrowed by 0043; §1 narrowed by 0067 (a core relay target) | @@ -105,4 +105,5 @@ the status is stale. | [0067](0067-a-lease-has-a-file-folder-on-its-worker-that-http-callers-fill-and-read.md) | A lease has a file folder on its worker that HTTP callers fill and read | Accepted; narrows 0005 Non-goals (file transfer) and §19b′, 0026 §1 (a core relay target) and 0028 §4 (a file PUT's deadline) | | [0068](0068-a-worker-pauses-warming-and-new-devices-under-host-memory-pressure.md) | A worker pauses warming and new devices under host memory pressure | Accepted; narrows 0029, adds a rank stage to 0009 | | [0069](0069-a-learned-kind-keeps-one-ready-copy-while-its-copies-are-leased.md) | A learned kind keeps one ready copy while its copies are leased | Accepted; narrows 0038 §2 and §3 | -| [0070](0070-test-runners-drive-a-lease-through-the-agent-device-front.md) | Test runners drive a lease through the agent-device front: boot is answered, a close may keep the lease, and each lease is its own tenant | Accepted; narrows 0023 §2 and §4 | +| [0070](0070-test-runners-drive-a-lease-through-the-agent-device-front.md) | Test runners drive a lease through the agent-device front: boot is answered, a close may keep the lease, and each lease is its own tenant | Accepted; narrows 0023 §2 and §4; §2 narrowed by 0071 | +| [0071](0071-a-restart-re-allocates-a-live-agent-device-lease.md) | A restart re-allocates a live agent-device lease on its device | Accepted — not yet implemented; supersedes 0023 §3 in part; narrows 0070 §2 |