Skip to content

Commit 96c08f1

Browse files
authored
refactor(isolation)!: make confirmation backend-neutral (#3366)
* refactor(isolation): make confirmation backend-neutral Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): validate confirmation evidence at host boundary Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): keep fence evidence driver-owned Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(isolation)!: validate explicit fence projections Require each compute driver to map its native evidence to individual outer-fence guarantees, and reject incomplete projections before a boundary becomes ready. Exercise the assembled remote confirmation path for invalid audit, property, generation, and digest evidence. BREAKING CHANGE: BoundaryConfig and SandboxRuntimeDescriptor use outer_fence projections rather than the earlier driver_fence representation. State written by earlier builds cannot be decoded; operators must stop and recreate affected sandboxes after upgrading. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): clarify outer fence ownership Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(isolation): update confirmation test fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>
1 parent 0a770d9 commit 96c08f1

20 files changed

Lines changed: 1058 additions & 394 deletions

File tree

Cargo.lock

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

architecture/sandbox.md

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -63,20 +63,52 @@ replacement from granting authority.
6363
## Startup Flow
6464

6565
1. The driver resolves the immutable workload identity, installs the outer
66-
network fence, and starts `openshell-sandbox` with one-use bootstrap state.
66+
network fence, validates its native evidence, and starts `openshell-sandbox`
67+
with one-use bootstrap state. Docker inspects container networking,
68+
Kubernetes verifies its NetworkPolicy, and VM drivers inspect the guest
69+
device model; those native schemas remain in their driver crates.
6770
2. The sandbox consumes and unlinks bootstrap material, proves the admitted
6871
runtime posture, and listens on the protected driver channel. It does not
6972
run untrusted code yet.
7073
3. `openshell-supervisor` loads policy and runtime settings from the gateway,
7174
attaches to the sandbox, and verifies the driver's generation and evidence.
7275
4. The sandbox installs its seccomp notification broker and Landlock baseline,
73-
then reports measured confirmation. The supervisor must accept that evidence
74-
before it sends the launch permit.
76+
validates its mechanism-specific audit evidence, and reports backend-neutral
77+
enforcement properties. The supervisor must accept those properties and
78+
their immutable session and resource binding before it sends the launch
79+
permit. Other isolation backends may establish the same properties with
80+
different mechanisms and retain their detailed evidence in backend-owned
81+
audit data.
7582
5. The sandbox starts the canonical process through its single workload
7683
launcher. The supervisor starts SSH and registers its gateway session.
7784
6. Exec, signaling, PTY, DNS, TCP, and loopback-forwarding operations cross the
7885
authenticated channel for the lifetime of the sandbox generation.
7986

87+
The shared isolation contract receives only normalized outer-fence guarantees:
88+
egress is default-deny, there is no unmanaged egress path, the evidence is bound
89+
to the sandbox generation, revocation has been verified, and controller loss
90+
fails closed. A digest commits those guarantees to the native
91+
evidence without teaching the shared contract about container networks,
92+
Kubernetes objects, VM devices, or accelerator resources.
93+
94+
The component that owns the outer fence also validates its native evidence and
95+
makes that projection explicitly. In the current Docker, Podman, Kubernetes,
96+
and VM placements, that component is the compute driver. A delegated isolation
97+
backend may own the fence and make the same projection instead. Non-empty native
98+
evidence alone does not establish a guarantee:
99+
100+
| Current enforcement owner | Native evidence | Guarantees projected by the owner |
101+
|---|---|---|
102+
| Docker | Pinned container ID, `network_mode=none`, and no unexpected network attachments | No workload route establishes default-deny, revocation, and controller-loss behavior; the attachment inspection establishes that no unmanaged route exists. |
103+
| Podman | Pinned container ID, `--network=none`, and no unexpected network attachments | The same container-network facts establish the same four guarantees. |
104+
| Kubernetes | NetworkPolicy UID and resource version, ingress and egress isolation, and zero workload egress rules | The persisted, selecting policy establishes default-deny and continued denial after revocation or controller loss; zero egress rules establish that no unmanaged route is permitted. |
105+
| VM | Generation and zero guest network devices | The absent NIC establishes all four guarantees; approved traffic uses the separate supervisor-owned channel. |
106+
107+
The shared contract checks that all four guarantees are present, that the
108+
projection names the admitted generation, and that its evidence digest matches
109+
the value passed to the workload-side runtime. It does not infer guarantees or
110+
interpret the native fields.
111+
80112
When the admitted main process exits, its status and retained terminal output
81113
remain available. The confirmed sandbox and supervisor-owned access plane continue
82114
to serve policy-authorized exec and loopback forwarding until explicit stop or
@@ -100,7 +132,7 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
100132
| Filesystem policy | Landlock restricts the paths the agent can read or write. |
101133
| Process policy | Sandbox and children run as one immutable non-root identity with zero capabilities. |
102134
| Seccomp notification | Virtualizes supported INET sockets and sends DNS/TCP decisions to the supervisor without nftables or proxy environment variables. |
103-
| Driver outer fence | Docker `network_mode=none`, a NIC-less VM, or Kubernetes NetworkPolicy prevents any missed or unsupported kernel path from escaping. |
135+
| Outer network fence | The component that owns network enforcement prevents any missed or unsupported kernel path from escaping. Current examples are Docker `network_mode=none`, a NIC-less VM, and Kubernetes NetworkPolicy. |
104136
| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. |
105137

106138
The supervisor may enrich baseline filesystem allowances for runtime-required
@@ -201,7 +233,7 @@ cannot transfer that approval to another socket.
201233

202234
The outer fence remains mandatory. If notification handling misses a syscall,
203235
loses the supervisor, exceeds a bound, or encounters an unsupported socket
204-
type, the request fails and the driver-owned fence still blocks direct egress.
236+
type, the request fails and the outer fence still blocks direct egress.
205237

206238
CONNECT and absolute-form forward HTTP are explicit-proxy adapters over the same
207239
egress pipeline. Each adapter normalizes its request into an egress intent, and

crates/openshell-driver-docker/src/isolation.rs

Lines changed: 83 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,52 @@ use std::collections::{BTreeMap, HashMap};
1111
use std::net::IpAddr;
1212
use std::path::PathBuf;
1313

14-
use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity};
14+
use openshell_isolation_interface::contract::{
15+
BackendError, OuterFenceGuarantee, OuterFenceGuarantees, ResolvedWorkloadIdentity,
16+
};
1517
use openshell_sandbox_backend::GPU_RESOURCE_CLAIM;
1618
use openshell_sandbox_backend::boundary_protocol::{
1719
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor,
1820
SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
1921
};
22+
use serde::Serialize;
23+
24+
#[derive(Serialize)]
25+
struct DockerOuterFenceEvidence<'a> {
26+
container_id: &'a str,
27+
network_mode: &'static str,
28+
unexpected_networks: &'a [String],
29+
}
30+
31+
impl DockerOuterFenceEvidence<'_> {
32+
fn project(&self, generation: &str) -> Result<OuterFenceGuarantees, BackendError> {
33+
if self.container_id.is_empty() {
34+
return Err(BackendError::Descriptor(
35+
"Docker outer fence evidence is incomplete".to_string(),
36+
));
37+
}
38+
let mut established = Vec::new();
39+
if self.network_mode == "none" {
40+
// With no container network namespace attachment, workload egress
41+
// remains denied both after revocation and if the supervisor exits.
42+
established.extend([
43+
OuterFenceGuarantee::DefaultDenyEgress,
44+
OuterFenceGuarantee::RevocationVerified,
45+
OuterFenceGuarantee::ControllerLossFailsClosed,
46+
]);
47+
}
48+
if self.unexpected_networks.is_empty() {
49+
established.push(OuterFenceGuarantee::NoUnmanagedEgressPath);
50+
}
51+
let encoded = serde_json::to_vec(self).map_err(|error| {
52+
BackendError::Descriptor(format!("encode Docker outer fence evidence: {error}"))
53+
})?;
54+
let projection =
55+
OuterFenceGuarantees::from_enforcement_evidence(generation, established, &encoded)?;
56+
projection.validate(generation)?;
57+
Ok(projection)
58+
}
59+
}
2060

2161
/// Driver-owned inputs that bind one Docker container to one boundary.
2262
pub struct DockerBoundarySpec {
@@ -48,21 +88,22 @@ pub struct DockerBoundaryProvisioning {
4888
impl DockerBoundarySpec {
4989
/// Produce both sides of the common protocol from the same immutable
5090
/// Docker coordinates so attach cannot bind a different container.
51-
#[must_use]
52-
pub fn provision(self) -> DockerBoundaryProvisioning {
91+
pub fn provision(self) -> Result<DockerBoundaryProvisioning, BackendError> {
5392
let mut resource_claims = BTreeMap::from([
5493
("docker.container_id".to_string(), self.container_id),
5594
("docker.image_identity".to_string(), self.image_identity),
5695
]);
5796
if self.gpu_requested {
5897
resource_claims.insert(GPU_RESOURCE_CLAIM.to_string(), "true".to_string());
5998
}
60-
let driver_fence = DriverFenceEvidence::Docker {
61-
container_id: resource_claims["docker.container_id"].clone(),
62-
network_mode: "none".to_string(),
63-
unexpected_networks: Vec::new(),
64-
};
65-
DockerBoundaryProvisioning {
99+
let unexpected_networks = Vec::new();
100+
let outer_fence = DockerOuterFenceEvidence {
101+
container_id: &resource_claims["docker.container_id"],
102+
network_mode: "none",
103+
unexpected_networks: &unexpected_networks,
104+
}
105+
.project(&self.generation)?;
106+
Ok(DockerBoundaryProvisioning {
66107
boundary_config: BoundaryConfig {
67108
boundary_id: self.boundary_id.clone(),
68109
generation: self.generation.clone(),
@@ -78,7 +119,7 @@ impl DockerBoundarySpec {
78119
resource_claims: resource_claims.clone(),
79120
resource_claim_files: BTreeMap::new(),
80121
workload_identity: self.workload_identity.clone(),
81-
driver_fence: driver_fence.clone(),
122+
outer_fence: outer_fence.clone(),
82123
child_env: self.child_env,
83124
},
84125
runtime_descriptor: SandboxRuntimeDescriptor {
@@ -92,16 +133,40 @@ impl DockerBoundarySpec {
92133
tls: self.supervisor_tls,
93134
host_gateway_ip: self.host_gateway_ip,
94135
resource_claims,
95-
driver_fence,
136+
outer_fence,
96137
},
97-
}
138+
})
98139
}
99140
}
100141

101142
#[cfg(test)]
102143
mod tests {
103144
use super::*;
104145

146+
#[test]
147+
fn outer_fence_projection_rejects_each_missing_native_fact() {
148+
let unexpected_networks = vec!["bridge".to_string()];
149+
for evidence in [
150+
DockerOuterFenceEvidence {
151+
container_id: "",
152+
network_mode: "none",
153+
unexpected_networks: &[],
154+
},
155+
DockerOuterFenceEvidence {
156+
container_id: "container",
157+
network_mode: "bridge",
158+
unexpected_networks: &[],
159+
},
160+
DockerOuterFenceEvidence {
161+
container_id: "container",
162+
network_mode: "none",
163+
unexpected_networks: &unexpected_networks,
164+
},
165+
] {
166+
assert!(evidence.project("generation-1").is_err());
167+
}
168+
}
169+
105170
#[test]
106171
fn provisioning_binds_container_and_image_claims() {
107172
let session_id = openshell_core::SandboxSessionId::new();
@@ -143,7 +208,8 @@ mod tests {
143208
.unwrap(),
144209
child_env: HashMap::new(),
145210
}
146-
.provision();
211+
.provision()
212+
.unwrap();
147213

148214
assert_eq!(
149215
provisioned.boundary_config.resource_claims,
@@ -158,14 +224,14 @@ mod tests {
158224
"true"
159225
);
160226
assert_eq!(
161-
provisioned.boundary_config.driver_fence,
162-
provisioned.runtime_descriptor.driver_fence
227+
provisioned.boundary_config.outer_fence,
228+
provisioned.runtime_descriptor.outer_fence
163229
);
164230
assert!(
165231
provisioned
166232
.runtime_descriptor
167-
.driver_fence
168-
.validate()
233+
.outer_fence
234+
.validate("generation-1")
169235
.is_ok()
170236
);
171237
}

crates/openshell-driver-docker/src/lib.rs

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4361,7 +4361,8 @@ async fn prepare_docker_boundary_files(
43614361
workload_identity: workload_identity.clone(),
43624362
child_env: docker_child_environment(sandbox),
43634363
}
4364-
.provision();
4364+
.provision()
4365+
.map_err(|error| Status::failed_precondition(error.to_string()))?;
43654366
let boundary_config = provisioning
43664367
.boundary_config
43674368
.encode()

crates/openshell-driver-kubernetes/src/driver.rs

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2380,7 +2380,8 @@ impl KubernetesComputeDriver {
23802380
workload_identity,
23812381
child_env,
23822382
}
2383-
.provision();
2383+
.provision()
2384+
.map_err(|error| KubernetesDriverError::Message(error.to_string()))?;
23842385
let descriptor = provisioned
23852386
.runtime_descriptor
23862387
.backend_descriptor()
@@ -2618,7 +2619,8 @@ impl KubernetesComputeDriver {
26182619
workload_identity,
26192620
child_env,
26202621
}
2621-
.provision();
2622+
.provision()
2623+
.map_err(|error| KubernetesDriverError::Message(error.to_string()))?;
26222624
let descriptor = provisioned
26232625
.runtime_descriptor
26242626
.backend_descriptor()

0 commit comments

Comments
 (0)