Skip to content

feat(inspect-ai): add OCI container execution mode and Compose parser - #339

Open
tholop wants to merge 1 commit into
feat/inspect-capsemfrom
feat/inspect-capsem-containers
Open

tholop wants to merge 1 commit into
feat/inspect-capsemfrom
feat/inspect-capsem-containers

Conversation

@tholop

@tholop tholop commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Scope: this PR only touches the Inspect AI sandbox provider (integrations/inspect-ai). It adds no container functionality to capsem-service, the guest runtime, or the SDKs. Capsem 0.7 already runs OCI containers natively — Hypervisor.create(image=...) boots an image session and ExecTarget::Workload (target="container") executes inside it. What was missing is the glue that lets an Inspect AI eval that declares a Docker image or a compose.yaml run on that native path; that glue is this PR (upstream tracking issue #311).

  • Inspect -> Capsem container translation: adds execution_mode="container" / "auto" and image=... to CapsemSandboxConfig, and a single-service compose.yaml / docker-compose.yml parser (PyYAML) that maps the Compose service (image, command/entrypoint, environment, working_dir, user, bind volumes, ${VAR} interpolation incl. Inspect's SAMPLE_METADATA_*) onto Hypervisor.create(image=...) plus target="container" execs. Stacked on feat/inspect-capsem.
  • Drops the legacy 0.6 in-guest dockerd backend entirely (~3,900 lines removed relative to the initial feat(inspect-ai): add inspect-capsem SandboxEnvironment integration #291 draft): no Docker daemon, image cache, or CA trust plumbing inside the guest; the host OCI puller and guest umoci unpack from 0.7 do all of that.
  • Workload ergonomics inside the native container: /workspace staging, non-root user execution (su -m / setpriv), file-mode preservation on single-file and directory bind mounts, and 0700 permissions for staged tool bundles (onedir).
  • Host-boundary guards (Elie's "Host environment and paths" item): default-deny host environment interpolation (CAPSEM_INSPECT_ALLOWED_HOST_ENV) and symlink-safe bind-mount containment (os.path.realpath against the project directory + CAPSEM_INSPECT_ALLOWED_HOST_PATHS), both operator-controlled env vars rather than task config.
  • Fails closed at config/parse time on Compose features 0.7 cannot honour (network_mode: none, multi-service topologies, anonymous single-target volumes, and build: / Dockerfile prior to PR 4) — see Known Limitations below.

Live 0.7 Benchmark & Acceptance Results (google/gemini-3.5-flash)

Suite / Task Mode Samples Status / Result Wall Time Leftover VMs Notes
tests/ironbank/test_sdk_live.py 0.7 VM 2 2/2 passed 24.47s 0 SDK_LIVE_ACCEPTANCE_OK + INSPECT_CAPSEM_VM_ACCEPTANCE_OK
integrations/inspect-ai/tests/live_acceptance.py 0.7 VM + OCI container 3 3/3 passed (44/44 VM + 44/44 container self_check) 121.16s 0 Verifies /workspace write+read, non-root developer (1000:1000) chown/exec/env, onedir 0 700, and numeric 1000:1000 Compose user
inspect_evals/humaneval VM (default) 3 success (0 errors) 12s 0 Clean VM boot, exec, and teardown on stock 0.7 assets
inspect_evals/gsm8k VM (default) 3 success (0 errors) 12s 0 Clean VM boot, exec, and teardown on stock 0.7 assets
inspect_evals/agentdojo (with_injections=false) VM (default) 2 success (0 errors) 11s 0 Clean VM boot, exec, and teardown on stock 0.7 assets
compose_bind_smoke Container (python:3.11-slim) 1 success (accuracy=1.000, 0 errors) 17s 0 Verifies bind mount + ${SAMPLE_METADATA_*} in OCI workload container
inspect_harbor/terminal_bench_2 (build-pov-ray) Container (Compose image:) 1 success (0 errors) 43s 0 Pulls/unpacks Harbor OCI image, runs bash tool loop + harbor_scorer
bigcodebench / swe_bench (default) / cybench (2-service) Compose fast-fail 3 Fast-fail ValueError 6s–7s 0 Rejects network_mode: none and multi-service Compose before VM creation
inspect_evals/swe_bench_verified_mini (-T allow_internet=true) Container (~655 MB compressed / ~1.5 GB uncompressed) 3 3/3 completed (2/3 = 0.667 score) only with a locally patched service (launch.py:478 timeout=30 -> 300, raised CREATE_READY_TIMEOUT); fails on stock 0.7 (HTTP 500 / 504 during image unpack) 902s 0 See Limitation 1 below

Known Limitations & Follow-Ups (Non-Blocking)

None of the items below block this PR: single-service container evals whose images fit the stock 0.7 pull/unpack budget and use default networking work end to end (terminal_bench_2, compose_bind_smoke, and the 44/44 container self_check above run on unpatched 0.7). Each one is a limitation of what 0.7 exposes today, listed here so it is visible from the PR rather than phrased as a review question. We searched google/capsem issues: none of the first three has a dedicated issue yet (proposed titles below, to be filed and linked); the fourth is explicitly out of scope by design in #289.

  1. Very large OCI images can exceed the stock 0.7 create budget — e.g. swe_bench_verified_mini (~655 MB compressed / ~1.5 GB uncompressed). Warm host cache: ~36s in launch.py:assemble() (copy + SHA-256 verify from VirtioFS /run/capsem-image) plus > 30s in umoci unpack trips subprocess.run(..., timeout=30) at guest/artifacts/container/launch.py:478 (HTTP 500 after 68s). Cold cache: pull (104–108s) + unpack exceeds CREATE_READY_TIMEOUT = 110s (crates/capsem-service/src/container_setup.rs:22, HTTP 504 create_timeout; inspect-capsem then deletes the half-created VM via CreateTimeoutError.vm_id). Impact: images above roughly 1 GB uncompressed fail on stock 0.7; typical eval images (< 500 MB) are unaffected. Mitigations available today: 0.7's POST /images/pull (generated SDK operation capsem._operations.pull_image; a Hypervisor-level wrapper is part of 0.7: Polish Python SDK for the image runtime #303) pre-warms the host cache ahead of create and removes the cold-cache case — we can add that pre-pull to inspect-capsem as a small follow-up; Minimal Capsem runtime with OCI application images (replaces profiles) #289's planned per-image .erofs fast path / unpack-once universal path removes most of the unpack cost. Proposed issue: "Container create: launch.py 30s umoci timeout and 110s CREATE_READY_TIMEOUT are too small for >1 GB images".
  2. oci-cache-lock 30s lease contention under concurrent pulls — Puller::blob (crates/capsem-assets/src/oci/pull.rs:343-383) holds cache.lease(&digest) across the whole layer download + publish, while lock() at cache.rs:202 polls with PollOpts::new("oci-cache-lock", Duration::from_secs(30)). Two concurrent POST /vms/create sharing an uncached > 500 MB layer: the second fails with OCI cache lock timed out. Impact: only the first cold pull of a large shared layer under max_sandboxes > 1; retrying after the first pull completes succeeds. Proposed issue: "OCI blob cache: lease timeout (30s) is shorter than PULL_TIMEOUT (300s); hold the lock only around the atomic publish".
  3. No per-VM air-gap / egress-deny toggle in ProvisionRequest — benchmarks that emit network_mode: none (swe_bench_verified_mini default allow_internet=False, bigcodebench, gdm_intercode_ctf) are rejected at parse time rather than silently given network access, because 0.7 has no per-VM egress control. Impact: those tasks must be run with their allow_internet=true style flags (as in the table above). Proposed issue: "Expose a per-VM network egress-deny option in ProvisionRequest so Compose network_mode: none can be enforced by the gateway".
  4. Multi-service Compose (> 1 service / sidecars) is unsupported by design — inspect-capsem fails closed when a Compose file defines more than one service (e.g. cybench default + victim). Minimal Capsem runtime with OCI application images (replaces profiles) #289 states the 0.7 runtime "does not ... introduce multi-container orchestration", so this is a documented limitation rather than a follow-up we expect 0.7.x to lift; inspect-capsem stays strictly single-workload.

@codecov-commenter

codecov-commenter commented Oct 7, 2026 •

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
6016 1 6015 0
View the top 3 failed test(s) by shortest run time
capsem-assets::oci::cache::inventory::tests::reused_materialization_controls_invalidate_inventory_on_acquire_and_after_release
Stack Traces | 0.24s run time
thread 'oci::cache::inventory::tests::reused_materialization_controls_invalidate_inventory_on_acquire_and_after_release' (141295) panicked at .../cache/inventory/tests.rs:130:39:
called `Result::unwrap()` on an `Err` value: cache changed during inventory observation
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
capsem-assets::oci::worker::tests::inventory_worker_publishes_only_quiet_bounded_observations_and_tracks_nested_changes
Stack Traces | 2.45s run time
thread 'oci::worker::tests::inventory_worker_publishes_only_quiet_bounded_observations_and_tracks_nested_changes' (145820) panicked at .../oci/worker/tests.rs:53:6:
called `Result::unwrap()` on an `Err` value: TimedOut { label: "changed-inventory", attempts: 8, timeout: 2s }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
capsem-service::bin/capsem-service::container_setup::tests::registry::catalog_observation_schedules_one_owned_worker_and_refuses_rebinding
Stack Traces | 2.54s run time
thread 'container_setup::tests::registry::catalog_observation_schedules_one_owned_worker_and_refuses_rebinding' (278086) panicked at .../container_setup/tests/registry.rs:27:6:
called `Result::unwrap()` on an `Err` value: TimedOut { label: "service-cache-observed", attempts: 8, timeout: 2s }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
capsem-assets::oci::worker::tests::incompatible_receipts_are_observed_once_until_their_metadata_changes
Stack Traces | 2.67s run time
thread 'oci::worker::tests::incompatible_receipts_are_observed_once_until_their_metadata_changes' (145752) panicked at .../oci/worker/tests.rs:221:10:
called `Result::unwrap()` on an `Err` value: TimedOut { label: "foreign-cache-observed", attempts: 8, timeout: 2s }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
capsem-core::fs_monitor::tests::a_workspace_swapped_for_a_host_link_is_never_walked_or_read
Stack Traces | 360s run time
No failure message available

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

ebursztein added a commit that referenced this pull request Oct 7, 2026
Link assigned issue #342 and source PRs #338, #339 and #340. Keep completed parser components available while removing Inspect VM/workload and Dockerfile/Compose phases from this agent sprint. Shared 0.7 includes all carried code and latest main; complete integration and runtime/package qualification remain Pierre’s work.
Add OCI container execution mode (`execution_mode="container"` / `execution_mode="auto"`, `image=...`, and `compose.yaml` / `docker-compose.yml` config parsing) on top of Capsem VM sandboxes:

- `inspect_capsem/containers/compose.py` + `compose_fields.py` + `compose_inputs.py` + `compose_interpolation.py` + `compose_service.py` + `compose_values.py`: single bounded Compose parser built on `ComposeInputs`, `ComposeLimits` / `_BoundedSafeLoader`, and `InterpolationBudget` with redacted diagnostics, default-deny host environment interpolation (`SAMPLE_METADATA_*` allowlist with `.env` spoofing rejection), support for `image`, `command`/`entrypoint`, `environment`, `working_dir`, `user`, `volumes` (read-write and `:ro` bind mounts with symlink and project-root containment), `healthcheck`, `cpus`/`mem_limit`, and fail-closed validation on unsupported service keys, network isolation overrides, and multi-service topologies.
- `inspect_capsem/containers/runtime.py` + `controller.py`: OCI container staging (`prepare_oci_workload_container`, `_stage_oci_bind_volumes`) and `ContainerController` protocol.
- `inspect_capsem/_compose.py`, `_controller.py`, `_exec.py`, `_files.py`, `_lifecycle.py`, `config.py`, `sandbox.py`: thread container execution mode, non-root `user` execution (`su -m` / `setpriv`), `/workspace` staging, and `SdkCapsemController` `registry_ca_pem` / `Registry(ca_pem=...)` plumbing (moved from PR #340 into PR #339 so #339's hermetic loopback TLS OCI workload fixture authenticates without #340 while `CapsemSandboxConfig` continues to reject `registry_ca_pem` in untrusted task configs).
- Unit tests (`tests/containers/*`, `tests/test_config.py`, `tests/test_sandbox*.py`) and hermetic loopback TLS OCI workload acceptance (`tests/oci_workload_fixture.py`, `tests/live_acceptance.py`).

Proves #310 / #311 / #342 acceptance criteria:
- [x] Compose config coercion (`compose.yaml` / `docker-compose.yml`) and `CapsemSandboxConfig(image=...)` select `execution_mode="container"` and route `exec` to `ExecTarget.WORKLOAD`.
- [x] Hermetic OCI workload acceptance (`tests/oci_workload_fixture.py` + `tests/live_acceptance.py`) builds a digest-pinned OCI image from the guest initrd busybox, serves it over loopback TLS with a per-run CA passed via `SdkCapsemController(registry_ca_pem=...)`, admits its digest in `settings.toml`, runs `sample_init`, `exec` (`ExecTarget.WORKLOAD`), `write_file`/`read_file` (text and binary), and `eval_async` with `SandboxEnvironmentSpec("capsem", ...)` without pulling from Docker Hub, and verifies `history(layer=EXEC)` + `session.db` `exec_events` (`target="workload"`) and zero leaked VMs after `sample_cleanup`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants