NVIDIA
diff --git a/‎Cargo.lock‎
Lines changed: 1 addition & 0 deletions b/‎Cargo.lock‎
Lines changed: 1 addition & 0 deletions
diff --git a/‎architecture/gateway.md‎
Lines changed: 76 additions & 0 deletions b/‎architecture/gateway.md‎
Lines changed: 76 additions & 0 deletions
diff --git a/‎crates/openshell-cli/src/run.rs‎
Lines changed: 3 additions & 0 deletions b/‎crates/openshell-cli/src/run.rs‎
Lines changed: 3 additions & 0 deletions
diff --git a/‎crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs‎
Lines changed: 13 additions & 0 deletions b/‎crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs‎
Lines changed: 13 additions & 0 deletions
diff --git a/‎crates/openshell-sdk/Cargo.toml‎
Lines changed: 1 addition & 0 deletions b/‎crates/openshell-sdk/Cargo.toml‎
Lines changed: 1 addition & 0 deletions
@@ -440,6 +440,82 @@ Domain objects use shared metadata: stable server-generated IDs, human-readable
 names, creation timestamps, and labels. Crate-level details live in
 `crates/openshell-core/README.md`.
 
+### Watch streams
+
+`WatchSandbox` merges three per-sandbox sources into one client stream: status
+snapshots, server/sandbox logs, and platform events. Logs and platform events
+are resumable; a shared per-sandbox allocator stamps each with a `cursor`.
+Cursor-ordered delivery is guaranteed for the replay phase: on resume the
+buffered events from both sources are sorted before emission. Live events are
+monotonic within each source, but the two sources are read independently, so a
+client should order across sources by `cursor` rather than by arrival. Status
+snapshots and warnings are re-read on demand and carry an empty cursor.
+
+#### Cursor spaces
+
+A sandbox's cursors live in a **cursor space**: a `{epoch, seq}` pair, where the
+epoch is a UUID minted on the first publish and `seq` counts from 1. The epoch
+is dropped by `TracingLogBus::remove`, so a teardown — or a gateway restart —
+retires the space, and the next publish mints a new one. A sequence number alone
+cannot distinguish a caught-up client from one holding a cursor out of a space
+that no longer exists, because the replacement space reuses the same numbers;
+the epoch answers *which counter issued this*, which is the question resume
+validation actually has to ask. Behind multiple replicas the same rule makes a
+reconnect to a different replica fail loudly rather than return the wrong
+events.
+
+On the wire a cursor is an opaque, fixed-width token. Clients may only compare
+two cursors from one stream and keep the greater; the encoding zero-pads `seq`
+so that byte-wise comparison matches sequence order, which is what lets every
+SDK track a high-water mark without parsing. A stream only ever observes one
+epoch — a reset closes both resumable broadcast receivers, ending the stream
+rather than switching spaces mid-flight — so that comparison is always well
+defined where clients are allowed to use it. The gateway does not rely on it:
+server-side ordering runs on the raw `u64` seq carried alongside each event in
+`CursoredEvent`, never on the token.
+
+The gateway holds a bounded in-memory tail per sandbox. Loss is reported with
+two distinct, documented behaviors:
+
+- **Recoverable lag** — a broadcast receiver falls behind and the server skips
+  ahead. The stream emits a `SandboxStreamWarning` event and continues. Since
+  cursors are opaque, the warning is the client's only signal.
+- **Unrecoverable gap** — the server sends a snapshot, then terminates with
+  `OUT_OF_RANGE`. Three cases reach it: the tail was trimmed past the requested
+  cursor, the cursor's epoch does not match the sandbox's current space, or no
+  space exists because nothing has been published since teardown. The status
+  tells the client to restart with an empty cursor; retrying the same token
+  fails identically. A token the gateway could not have issued is rejected
+  earlier, as `INVALID_ARGUMENT` on the call itself.
+
+Both resumable sources draw from one cursor space, so the server merges them by
+seq before emitting rather than draining each in turn: on resume it replays only
+events after the client's cursor, and without one it replays each bus's retained
+tail. Either way the batch leaves in ascending cursor order. The two tails are
+bounded independently (`log_tail_lines` and `event_tail`), so merging orders
+whatever each bus kept; it does not align their depths.
+
+The broadcast receivers are subscribed before replay, so an event buffered during
+initialization could appear in both replay and the live receiver; the producer
+tracks the highest replayed seq and suppresses live events at or below it, so
+each event is delivered once. That mark is per source. The two tails are read at
+different instants and bounded independently, so one shared mark would let the
+deeper source censor the shallower one — with `event_tail` unset the mark rises
+to the newest buffered log while no platform event is replayed at all, and
+platform events published during initialization are discarded as duplicates of a
+replay that never ran. Subscribing never mints a cursor space, so a
+resume against a torn-down sandbox cannot create the space its stale cursor is
+then checked against. Clients track the highest observed `cursor` and pass it as
+`resume_after_cursor` on reconnect.
+
+The epoch is validated twice on resume: once before reading the tails and again
+once both are in hand, before anything is emitted. The check and each read take
+their locks separately, so a teardown plus a republish can retire the validated
+space and install a replacement in between; the reads would then apply the old
+space's seq to the replacement's buffers, and a trimmed-range check that only
+compares numbers would report no gap while skipping the replacement's lower
+events. The second look ends the stream with `OUT_OF_RANGE` instead.
+
 ## Persistence
 
 The gateway persistence layer is a protobuf object store. Domain services store
 
@@ -809,6 +809,7 @@ pub async fn sandbox_create(
             since_time: None,
             log_sources: vec!["gateway".to_string()],
             log_min_level: String::new(),
+            resume_after_cursor: String::new(),
         })
         .await
         .into_diagnostic()?
@@ -3613,6 +3614,7 @@ async fn wait_for_lifecycle_phase(
             since_time: None,
             log_sources: Vec::new(),
             log_min_level: String::new(),
+            resume_after_cursor: String::new(),
         })
         .await
         .into_diagnostic()?
@@ -5847,6 +5849,7 @@ pub async fn sandbox_logs(
                     .into_diagnostic()?,
                 log_sources: source_filter,
                 log_min_level: level.to_uppercase(),
+                resume_after_cursor: String::new(),
             })
             .await
             .into_diagnostic()?
 
@@ -743,6 +743,7 @@ impl OpenShell for TestOpenShell {
             let _ = tx
                 .send(Ok(SandboxStreamEvent {
                     payload: Some(sandbox_stream_event::Payload::Sandbox(provisioning)),
+                    cursor: String::new(),
                 }))
                 .await;
             if terminal_after_provisional_container_exit
@@ -753,6 +754,7 @@ impl OpenShell for TestOpenShell {
                         payload: Some(sandbox_stream_event::Payload::Sandbox(
                             provisional_container_exit,
                         )),
+                        cursor: String::new(),
                     }))
                     .await;
                 provisional_container_exit_sent.notify_waiters();
@@ -764,6 +766,7 @@ impl OpenShell for TestOpenShell {
                 let _ = tx
                     .send(Ok(SandboxStreamEvent {
                         payload: Some(sandbox_stream_event::Payload::Sandbox(completed)),
+                        cursor: String::new(),
                     }))
                     .await;
                 return;
@@ -777,11 +780,13 @@ impl OpenShell for TestOpenShell {
                             message: "Started VM launcher".to_string(),
                             ..PlatformEvent::default()
                         })),
+                        cursor: String::new(),
                     }))
                     .await;
                 let _ = tx
                     .send(Ok(SandboxStreamEvent {
                         payload: Some(sandbox_stream_event::Payload::Sandbox(error)),
+                        cursor: String::new(),
                     }))
                     .await;
                 tokio::time::sleep(Duration::from_secs(5)).await;
@@ -801,12 +806,14 @@ impl OpenShell for TestOpenShell {
                                 source: "gateway".to_string(),
                                 fields: HashMap::new(),
                             })),
+                            cursor: String::new(),
                         }))
                         .await;
                 }
                 let _ = tx
                     .send(Ok(SandboxStreamEvent {
                         payload: Some(sandbox_stream_event::Payload::Sandbox(ready)),
+                        cursor: String::new(),
                     }))
                     .await;
                 return;
@@ -815,6 +822,7 @@ impl OpenShell for TestOpenShell {
                 let _ = tx
                     .send(Ok(SandboxStreamEvent {
                         payload: Some(sandbox_stream_event::Payload::Sandbox(completed)),
+                        cursor: String::new(),
                     }))
                     .await;
                 return;
@@ -829,6 +837,7 @@ impl OpenShell for TestOpenShell {
                             message: "Preparing rootfs".to_string(),
                             ..PlatformEvent::default()
                         })),
+                        cursor: String::new(),
                     }))
                     .await;
                 tokio::time::sleep(Duration::from_millis(600)).await;
@@ -840,12 +849,14 @@ impl OpenShell for TestOpenShell {
                             message: "Formatting root disk".to_string(),
                             ..PlatformEvent::default()
                         })),
+                        cursor: String::new(),
                     }))
                     .await;
                 tokio::time::sleep(Duration::from_millis(600)).await;
                 let _ = tx
                     .send(Ok(SandboxStreamEvent {
                         payload: Some(sandbox_stream_event::Payload::Sandbox(ready)),
+                        cursor: String::new(),
                     }))
                     .await;
                 return;
@@ -857,11 +868,13 @@ impl OpenShell for TestOpenShell {
                         message: "Sandbox scheduled".to_string(),
                         ..PlatformEvent::default()
                     })),
+                    cursor: String::new(),
                 }))
                 .await;
             let _ = tx
                 .send(Ok(SandboxStreamEvent {
                     payload: Some(sandbox_stream_event::Payload::Sandbox(ready)),
+                    cursor: String::new(),
                 }))
                 .await;
         });
 
@@ -28,6 +28,7 @@ tokio-tungstenite = { workspace = true }
 tonic = { workspace = true, features = ["tls-native-roots"] }
 tower = { workspace = true }
 tracing = { workspace = true }
+async-stream = "0.3.6"
 
 [dev-dependencies]
 serde_json = { workspace = true }