chore: merge main into sandbox image update

# Conflicts:
#	crates/openshell-sandbox-backend/src/boundary_protocol.rs
#	e2e/rust/tests/host_gateway_alias.rs

Author: drew

.agents/skills/helm-dev-environment/SKILL.md

@@ -113,10 +113,31 @@ trace-context annotation. The same command exposes OTLP/gRPC on
 `gateway:vm` tasks detect the collector listener at startup and enable trace
 export only while it is reachable.
 
-**HA test deploy** (two gateway replicas + external PostgreSQL Secret): uncomment
-`#- ci/values-high-availability.yaml` in `deploy/helm/openshell/skaffold.yaml`,
-create the Secret named `openshell-ha-pg` with a `uri` key, then run
-`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.
+The Skaffold profile for HA reverse-proxy development is available from
+`deploy/helm/openshell/`:
+
+```bash
+# Two gateway replicas + external PostgreSQL Secret + Envoy Gateway + Gateway API route.
+KUBECONFIG=../../../kubeconfig skaffold run -p high-availability
+```
+
+The `high-availability` profile expects a Secret named `openshell-ha-pg` in the `openshell`
+namespace with a `uri` key. For local manual testing, either create your own
+PostgreSQL Secret or use the e2e PostgreSQL fixture manifest in
+`e2e/kubernetes/postgres-fixture.yaml`.
+
+For the `high-availability` profile, return to the repository root and apply the
+GatewayClass and BackendTrafficPolicy manifest after Skaffold has installed
+Envoy Gateway:
+
+```bash
+KUBECONFIG=kubeconfig mise run helm:gateway:apply
+```
+
+The BackendTrafficPolicy disables Envoy request and stream-duration timeouts for
+OpenShell's `GRPCRoute`. Keep that policy in `deploy/kube/manifests/envoy-gateway-openshell.yaml`,
+not in the Helm chart; it is required for long-lived gRPC create/watch/exec/relay
+streams during gateway rollouts and scale events.
 
 ### TLS behaviour
 
@@ -187,23 +208,85 @@ but will point to a deleted cluster — safe to ignore or clean up manually.
 
 ## Optional Add-ons
 
-Each add-on requires uncommenting the corresponding `valuesFiles` entry in
-`deploy/helm/openshell/skaffold.yaml` before running `helm:skaffold:dev` or `helm:skaffold:run`.
+Some add-ons can be enabled by uncommenting values in `skaffold.yaml`, but prefer
+the dedicated Skaffold profiles when they exist. Profiles avoid leaving local
+manual edits in the worktree.
 
 ### Envoy Gateway (Gateway API / GRPCRoute)
 
-Envoy Gateway is already installed by Skaffold (the `envoy-gateway` Helm release in
-`skaffold.yaml`). To activate routing:
+Use the `high-availability` Skaffold profile for HA reverse-proxy testing. The
+profile intentionally includes Envoy Gateway so multi-replica behavior is
+exercised through the same Gateway API path used by reverse-proxy deployments:
 
-1. Uncomment `#- values-gateway.yaml` in `skaffold.yaml`
-2. Redeploy: `mise run helm:skaffold:run`
-3. Apply the GatewayClass: `mise run helm:gateway:apply`
-4. Access: `http://127.0.0.1:8080`
+```bash
+cd deploy/helm/openshell
+KUBECONFIG=../../../kubeconfig skaffold run -p high-availability
+cd ../../..
+KUBECONFIG=kubeconfig mise run helm:gateway:apply
+```
+
+`values-gateway.yaml` creates a `Gateway` (listener on port 80, class `eg`) and
+`GRPCRoute` in the `openshell` namespace. The `high-availability` profile
+installs the Envoy Gateway Helm chart and layers both
+`values-high-availability.yaml` and `values-gateway.yaml` onto the OpenShell
+release.
+
+`deploy/kube/manifests/envoy-gateway-openshell.yaml` creates:
+
+- `GatewayClass/eg`
+- `BackendTrafficPolicy/openshell-grpc-timeouts`
+
+The Envoy Gateway proxy Service is usually exposed through the k3d load balancer
+at `http://127.0.0.1:8080`. If the cluster was created with a different
+`HELM_K3S_LB_HOST_PORT`, use that host port instead.
+
+For manual tests against an existing cluster, prefer forwarding the Envoy proxy
+Service rather than `svc/openshell`. That keeps client traffic on the same path
+as a real reverse proxy while gateway pods rotate behind it:
+
+```bash
+KUBECONFIG=kubeconfig kubectl get svc -A \
+  -l gateway.envoyproxy.io/owning-gateway-name=openshell
+KUBECONFIG=kubeconfig kubectl -n <envoy-service-namespace> port-forward \
+  svc/<envoy-service-name> 8080:80
+openshell gateway add http://127.0.0.1:8080 --name openshell --local
+```
+
+When running e2e tests manually through Envoy, register gateway metadata (as
+above) instead of relying only on `OPENSHELL_GATEWAY_ENDPOINT`; some tests call
+`openshell gateway info` and expect metadata for the active gateway.
+
+### Kubernetes E2E Notes
+
+Use `mise run e2e:kubernetes` for the standard Helm-backed Kubernetes suite.
+The kube e2e wrapper creates only one port-forward, to `svc/openshell`; it no
+longer forwards the unauthenticated health listener or runs a `/readyz` e2e
+target. `/readyz` remains covered by server unit/integration tests.
+
+Use `mise run e2e:kubernetes:ha-rebalancing` for full-suite HA coverage. The
+task creates an external PostgreSQL fixture, installs Envoy Gateway, applies
+`deploy/kube/manifests/envoy-gateway-openshell.yaml`, enables the chart
+`GRPCRoute`, and runs the full Kubernetes e2e suite, including
+`kubernetes_ha_rebalancing`. That coverage validates sandbox create/watch and
+exec through the Envoy proxy while gateway replicas scale up, scale down, and
+rotate. It also keeps a long-running sandbox alive and runs upload/download
+operations while gateway pods roll, so file sync exercises the same relay retry
+path as interactive sessions.
+
+If you reuse an existing Skaffold cluster for the full kube suite, make sure the
+chart has `server.hostGatewayIP` set so sandbox pods can resolve
+`host.openshell.internal` back to the test host. The e2e wrapper detects this on
+chart installs; manual reuse may require:
+
+```bash
+HOST_GATEWAY_IP="${OPENSHELL_E2E_HOST_GATEWAY_IP:?set host gateway IP}"
+KUBECONFIG=kubeconfig helm upgrade openshell deploy/helm/openshell \
+  --namespace openshell --reuse-values \
+  --set "server.hostGatewayIP=${HOST_GATEWAY_IP}" \
+  --wait --timeout 5m
+```
 
-`values-gateway.yaml` creates a `Gateway` (listener on port 80, class `eg`) and a
-`GRPCRoute` in the `openshell` namespace. Envoy Gateway provisions a LoadBalancer
-service for the proxy; klipper-lb binds it to hostPort 80, reachable via the
-`8080:80` load balancer port mapping.
+Use the IP that pods in that cluster use to reach listeners on the test host.
 
 ### BackendTLSPolicy (end-to-end TLS)
 
@@ -356,6 +439,6 @@ for dependencies still declared in `Chart.yaml`.
 | `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
 | `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
 | `deploy/helm/openshell/ci/values-credential-driver-vault.yaml` | Vault credential-driver validation overlay with HTTPS and private-CA trust |
-| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass for Envoy Gateway (`mise run helm:gateway:apply`) |
+| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass and BackendTrafficPolicy for Envoy Gateway (`mise run helm:gateway:apply`) |
 | `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status |
 | `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy, realm import, and development TLS trust anchor |

.github/workflows/branch-e2e.yml

@@ -426,6 +426,9 @@ jobs:
       job-name: Kubernetes HA E2E (Rust smoke)
       extra-helm-values: deploy/helm/openshell/ci/values-high-availability.yaml
       external-postgres-secret: openshell-ha-pg
+      test-name: kubernetes_ha_rebalancing
+      kubernetes-features: e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-ha
+      use-envoy-gateway: true
       conformance-artifact-prefix: openshell-conformance
 
   kubernetes-credential-drivers-e2e:

.github/workflows/e2e-kubernetes-test.yml

@@ -52,6 +52,11 @@ on:
         required: false
         type: string
         default: ""
+      use-envoy-gateway:
+        description: "Route the test through Envoy Gateway instead of a direct gateway Service port-forward"
+        required: false
+        type: boolean
+        default: false
       mise-version:
         description: "mise version to install on the bare Kubernetes e2e runner"
         required: false
@@ -140,6 +145,7 @@ jobs:
           OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET: ${{ inputs.external-postgres-secret }}
           OPENSHELL_E2E_KUBE_TEST: ${{ inputs.test-name }}
           OPENSHELL_E2E_KUBERNETES_FEATURES: ${{ inputs.kubernetes-features }}
+          OPENSHELL_E2E_KUBE_USE_ENVOY: ${{ inputs.use-envoy-gateway && '1' || '0' }}
           IMAGE_TAG: ${{ inputs.image-tag }}
           OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell
           E2E_TASK: ${{ inputs.e2e-task }}

Cargo.lock

@@ -98,15 +98,14 @@ shortly after carrying a stale `Provisioning` or `Unknown` backend phase. The
 composition rule treats a connected session as the stronger signal and keeps `Ready`
 in that case, preventing a lagging snapshot from undoing the session-driven promotion.
 
-**Known HA limitation:** Supervisor sessions are process-local while the public
-sandbox phase is shared. A replica that reconciles a driver snapshot without owning
-the active supervisor session can demote the shared phase to `Provisioning`. The
-session-owning replica may not receive another connection event to restore `Ready`,
-so a usable sandbox can remain unavailable through the public phase gate. Reliable
-HA readiness requires persisted or leased supervisor presence plus routing to the
-session-owning replica. That work is deferred to GitHub issue #1868. Until then,
-deployments that require reliable readiness composition must run a single gateway
-replica.
+**HA session composition:** Live relay handles remain process-local, while the
+session-owning gateway publishes a short-lived owner record in shared PostgreSQL.
+Driver reconciliation treats a fresh local or remote owner as connected, so a
+non-owner replica cannot demote the shared sandbox phase merely because it lacks the
+in-memory stream. Session-bound requests are forwarded to the owning gateway; a
+supervisor reconnect publishes a higher connection epoch before stale-session cleanup
+can demote readiness. Multi-replica deployments therefore require shared PostgreSQL
+and the gateway peer Service configured by the Helm chart.
 
 **Extension point:** Driver-reported readiness is a capability, not an
 operator-configurable hook. A driver may enable it only when it owns workload
@@ -119,9 +118,12 @@ The capability RPC reports driver identity, version, and the default sandbox
 image used by the gateway. GPU availability stays driver-local and is validated
 when a sandbox create request asks for GPU resources.
 
-The gateway records driver identity and version from the startup capability
-response. Elevated gateway info reports that initialized driver snapshot instead
-of re-querying drivers on each request.
+The gateway sends its common extension peer metadata with the startup capability
+request. The driver validates that metadata before responding, and the gateway
+rejects a driver whose protocol major or capability requirements are
+incompatible. It records the negotiated protocol, implementation identity and
+version, capability sets, and typed resource support once. Elevated gateway info
+reports that immutable snapshot instead of re-querying drivers on each request.
 
 ## Compiled Driver Selection
 

architecture/compute-runtimes.md

@@ -354,6 +354,70 @@ authenticated sandbox ID with any sandbox ID or name resolved from the request.
 Supervisor control and relay streams require a matching sandbox principal before
 the gateway registers the session or bridges relay bytes.
 
+## HA Supervisor Ownership
+
+In multi-replica Kubernetes deployments, every gateway pod can accept client
+RPCs, but a sandbox supervisor maintains one active stream to one gateway
+replica at a time. The connected replica publishes a short-lived supervisor
+owner record in the shared Postgres object store with its replica id, peer DNS
+endpoint, supervisor instance id, and connection epoch. Ownership does not move
+because another gateway receives a client request. It changes only when the
+supervisor opens a new control stream, usually after the previous owner pod is
+terminated or the stream breaks. A reconnect from the same supervisor instance
+with a newer epoch can supersede the previous owner before the TTL expires, and
+heartbeats from the active connection renew that current owner record.
+Cleanup from an older connection checks the shared owner record before and
+after changing sandbox readiness. It cannot demote a sandbox after a newer
+replica has published replacement ownership.
+
+Session-bound operations such as exec, TCP forwarding, file sync, and sandbox
+service routing first check the local session registry. If the supervisor is
+owned by another gateway replica, the serving gateway opens an internal
+`PeerRelay` stream to that owner and asks it to open the supervisor relay. This
+keeps client traffic working when a Kubernetes Service routes the client to a
+non-owner gateway pod. If a peer owner is stale or unreachable during a rollout,
+the serving gateway retries ownership lookup until the normal relay wait
+deadline. Each retry re-reads the owner record, so a supervisor reconnect or
+heartbeat can surface a new owner; if no fresh reachable owner appears before
+the deadline, the client operation fails rather than electing an owner itself.
+Provider-readiness reports, endpoint-status reports, and provider-status reads
+also follow the durable owner record through unary peer RPCs. The owning replica
+validates the current supervisor session and keeps the in-memory evidence; a
+non-owner never accepts evidence from a stale local session or projects a
+remote session as disconnected.
+
+Nothing redistributes established sessions, so after a rolling restart the last
+surviving replica holds most sessions and a new replica serves none until
+sandboxes reconnect. That skew decays only as sandboxes churn. Client traffic
+stays correct throughout because a non-owner relays to the owner.
+
+File upload and download use tar-over-SSH through the same relay path. A gateway
+pod termination drops the active SSH proxy byte stream, so the CLI retries the
+whole sync operation with a fresh SSH session instead of attempting mid-stream
+resume.
+
+Gateway peer RPCs authenticate with Kubernetes ServiceAccount identity rather
+than a shared secret. Helm mounts a projected, pod-bound token with audience
+`openshell-gateway-peer`; the receiving gateway validates it through
+TokenReview, checks the live pod UID and chart selector labels, and authorizes
+only the internal peer RPC methods. When gateway TLS is enabled, peer clients
+also trust the chart CA, present the chart-generated client certificate for
+mTLS, and verify the stable gateway Service DNS name even when connecting to a
+Deployment pod IP.
+
+`WatchSandbox` uses the local update bus for same-replica writes. On
+multi-replica backends one shared poller per gateway observes resource-version
+changes made by other replicas and feeds that bus for all local watchers,
+avoiding a database poll per client stream. SQLite deployments do not run the
+poller because they are single-replica and the local bus already sees every
+write.
+
+Mutations whose invariants span sandbox, provider-profile, policy, or provider
+records take a process-local mutex and a shared PostgreSQL advisory lock. The
+database session remains dedicated to the request and closes when the guard is
+dropped, which releases the lock on normal completion, cancellation, or error.
+SQLite deployments use only the local mutex because they are single-replica.
+
 ## API Surface
 
 The gateway API is organized around platform objects and operational streams:
@@ -753,7 +817,7 @@ migrations backfill existing rows with version 1.
 Provider profile imports, updates, and deletes hold the sandbox synchronization
 guard while checking attached-sandbox dynamic token grant ambiguity or in-use
 state and writing the profile record. Sandbox creation with initial providers and
-sandbox provider attach/detach use the same guard, so one gateway process cannot
+sandbox provider attach/detach use the same guard, so gateway replicas cannot
 interleave a profile mutation with a sandbox provider-set mutation that would
 leave an ambiguous final dynamic-token state or a deleted custom profile that is
 still referenced by a sandbox.
@@ -782,6 +846,15 @@ resolution and again by the sandbox placeholder resolver. This keeps expired
 credentials from resolving even when a running sandbox still has retained
 placeholder generations from an earlier provider credential snapshot.
 
+All gateway-owned extension registries negotiate the same peer metadata envelope
+before accepting work. Compute drivers, credential drivers, gateway interceptors,
+and supervisor middleware retain their typed family manifests. Both the gateway
+and extension run the shared validator against the startup exchange, enforcing
+protocol-major compatibility and mutual required-capability sets before either
+peer accepts the other. The gateway aggregates immutable, non-secret startup
+snapshots for the protected gateway-info API; it does not publish transport,
+authentication, or backend configuration.
+
 Static credential delivery is capability-negotiated and endpoint-bound. The
 gateway classifies each returned environment entry as either a credential or
 non-secret provider configuration and associates every credential key with the
@@ -810,7 +883,13 @@ Provider receipts, installation status, and common operations represent absolute
 
 Provider installation reports belong to the existing `ConnectSupervisor` session. Each report names that session, has an increasing sequence, and expires unless the supervisor reports again. Reconnection or disconnect invalidates prior observations; stored change records survive a gateway restart, but runtime evidence does not. Replaying an identical report cannot extend its lifetime.
 
-Reports and status also compare the supervisor instance with the sandbox's persisted current instance. A different supervisor becoming current invalidates an older connection, including one retained by another gateway replica. Observations stay local to the gateway holding the supervisor session; a status request reaching a replica without that session returns pending. Multi-replica deployments therefore retain the existing supervisor-session routing requirement.
+Reports and status also compare the supervisor instance with the sandbox's
+persisted current instance. A different supervisor becoming current invalidates
+an older connection, including one retained by another gateway replica.
+Observations stay local to the gateway holding the supervisor session. A status
+request or report reaching another replica follows the shared owner record to
+that gateway, which remains the sole authority for accepting and projecting the
+session's evidence.
 
 The supervisor reports success only after it installs the matching credentials, activates the effective policy, and receives an acknowledgment from the authenticated workload boundary that it installed the environment for future processes. Environment synchronization shares the process-launch lock, and its acknowledgment identifies the exact publication, including retries at the same provider revision. Failed policy installation cannot reuse evidence for a different installed policy. Ready and revoked statuses also recheck the requested sandbox, provider, attachment and configuration identities; revision fingerprints are compared only for equality. Revocation applies to future credential resolution and future processes. Requests already forwarded upstream can still finish.