Repository navigation
Conversation
❌ 1 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
ebursztein
added a commit
that referenced
this pull request
Oct 7, 2026
Link assigned issue #342 and source PRs #338, #339 and #340. Keep completed parser components available while removing Inspect VM/workload and Dockerfile/Compose phases from this agent sprint. Shared 0.7 includes all carried code and latest main; complete integration and runtime/package qualification remain Pierre’s work.
Add OCI container execution mode (`execution_mode="container"` / `execution_mode="auto"`, `image=...`, and `compose.yaml` / `docker-compose.yml` config parsing) on top of Capsem VM sandboxes: - `inspect_capsem/containers/compose.py` + `compose_fields.py` + `compose_inputs.py` + `compose_interpolation.py` + `compose_service.py` + `compose_values.py`: single bounded Compose parser built on `ComposeInputs`, `ComposeLimits` / `_BoundedSafeLoader`, and `InterpolationBudget` with redacted diagnostics, default-deny host environment interpolation (`SAMPLE_METADATA_*` allowlist with `.env` spoofing rejection), support for `image`, `command`/`entrypoint`, `environment`, `working_dir`, `user`, `volumes` (read-write and `:ro` bind mounts with symlink and project-root containment), `healthcheck`, `cpus`/`mem_limit`, and fail-closed validation on unsupported service keys, network isolation overrides, and multi-service topologies. - `inspect_capsem/containers/runtime.py` + `controller.py`: OCI container staging (`prepare_oci_workload_container`, `_stage_oci_bind_volumes`) and `ContainerController` protocol. - `inspect_capsem/_compose.py`, `_controller.py`, `_exec.py`, `_files.py`, `_lifecycle.py`, `config.py`, `sandbox.py`: thread container execution mode, non-root `user` execution (`su -m` / `setpriv`), `/workspace` staging, and `SdkCapsemController` `registry_ca_pem` / `Registry(ca_pem=...)` plumbing (moved from PR #340 into PR #339 so #339's hermetic loopback TLS OCI workload fixture authenticates without #340 while `CapsemSandboxConfig` continues to reject `registry_ca_pem` in untrusted task configs). - Unit tests (`tests/containers/*`, `tests/test_config.py`, `tests/test_sandbox*.py`) and hermetic loopback TLS OCI workload acceptance (`tests/oci_workload_fixture.py`, `tests/live_acceptance.py`). Proves #310 / #311 / #342 acceptance criteria: - [x] Compose config coercion (`compose.yaml` / `docker-compose.yml`) and `CapsemSandboxConfig(image=...)` select `execution_mode="container"` and route `exec` to `ExecTarget.WORKLOAD`. - [x] Hermetic OCI workload acceptance (`tests/oci_workload_fixture.py` + `tests/live_acceptance.py`) builds a digest-pinned OCI image from the guest initrd busybox, serves it over loopback TLS with a per-run CA passed via `SdkCapsemController(registry_ca_pem=...)`, admits its digest in `settings.toml`, runs `sample_init`, `exec` (`ExecTarget.WORKLOAD`), `write_file`/`read_file` (text and binary), and `eval_async` with `SandboxEnvironmentSpec("capsem", ...)` without pulling from Docker Hub, and verifies `history(layer=EXEC)` + `session.db` `exec_events` (`target="workload"`) and zero leaked VMs after `sample_cleanup`.
tholop
force-pushed
the
feat/inspect-capsem
branch
from
October 8, 2026 13:07
6f9656f to
8708553
Compare
tholop
force-pushed
the
feat/inspect-capsem-containers
branch
from
October 8, 2026 13:07
99a088b to
3cb33cd
Compare
This was referenced Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scope: this PR only touches the Inspect AI sandbox provider (
integrations/inspect-ai). It adds no container functionality tocapsem-service, the guest runtime, or the SDKs. Capsem0.7already runs OCI containers natively —Hypervisor.create(image=...)boots an image session andExecTarget::Workload(target="container") executes inside it. What was missing is the glue that lets an Inspect AI eval that declares a Docker image or acompose.yamlrun on that native path; that glue is this PR (upstream tracking issue #311).execution_mode="container"/"auto"andimage=...toCapsemSandboxConfig, and a single-servicecompose.yaml/docker-compose.ymlparser (PyYAML) that maps the Compose service (image,command/entrypoint,environment,working_dir,user, bindvolumes,${VAR}interpolation incl. Inspect'sSAMPLE_METADATA_*) ontoHypervisor.create(image=...)plustarget="container"execs. Stacked onfeat/inspect-capsem.0.6in-guestdockerdbackend entirely (~3,900 lines removed relative to the initial feat(inspect-ai): add inspect-capsem SandboxEnvironment integration #291 draft): no Docker daemon, image cache, or CA trust plumbing inside the guest; the host OCI puller and guestumociunpack from0.7do all of that./workspacestaging, non-rootuserexecution (su -m/setpriv), file-mode preservation on single-file and directory bind mounts, and0700permissions for staged tool bundles (onedir).CAPSEM_INSPECT_ALLOWED_HOST_ENV) and symlink-safe bind-mount containment (os.path.realpathagainst the project directory +CAPSEM_INSPECT_ALLOWED_HOST_PATHS), both operator-controlled env vars rather than task config.0.7cannot honour (network_mode: none, multi-service topologies, anonymous single-target volumes, andbuild:/Dockerfileprior to PR 4) — see Known Limitations below.Live
0.7Benchmark & Acceptance Results (google/gemini-3.5-flash)tests/ironbank/test_sdk_live.py0.7VM24.47s0SDK_LIVE_ACCEPTANCE_OK+INSPECT_CAPSEM_VM_ACCEPTANCE_OKintegrations/inspect-ai/tests/live_acceptance.py0.7VM + OCI container44/44VM +44/44containerself_check)121.16s0/workspacewrite+read, non-rootdeveloper(1000:1000) chown/exec/env,onedir0 700, and numeric1000:1000Compose userinspect_evals/humanevaldefault)success(0errors)12s00.7assetsinspect_evals/gsm8kdefault)success(0errors)12s00.7assetsinspect_evals/agentdojo(with_injections=false)default)success(0errors)11s00.7assetscompose_bind_smokepython:3.11-slim)success(accuracy=1.000,0errors)17s0${SAMPLE_METADATA_*}in OCI workload containerinspect_harbor/terminal_bench_2(build-pov-ray)image:)success(0errors)43s0bashtool loop +harbor_scorerbigcodebench/swe_bench(default) /cybench(2-service)ValueError6s–7s0network_mode: noneand multi-service Compose before VM creationinspect_evals/swe_bench_verified_mini(-T allow_internet=true)~655 MBcompressed /~1.5 GBuncompressed)3/3completed (2/3 = 0.667score) only with a locally patched service (launch.py:478timeout=30 -> 300, raisedCREATE_READY_TIMEOUT); fails on stock0.7(HTTP 500 / 504 during image unpack)902s0Known Limitations & Follow-Ups (Non-Blocking)
None of the items below block this PR: single-service container evals whose images fit the stock
0.7pull/unpack budget and use default networking work end to end (terminal_bench_2,compose_bind_smoke, and the44/44containerself_checkabove run on unpatched0.7). Each one is a limitation of what0.7exposes today, listed here so it is visible from the PR rather than phrased as a review question. We searchedgoogle/capsemissues: none of the first three has a dedicated issue yet (proposed titles below, to be filed and linked); the fourth is explicitly out of scope by design in #289.0.7create budget — e.g.swe_bench_verified_mini(~655 MBcompressed / ~1.5 GBuncompressed). Warm host cache: ~36sinlaunch.py:assemble()(copy + SHA-256 verify from VirtioFS/run/capsem-image) plus> 30sinumoci unpacktripssubprocess.run(..., timeout=30)atguest/artifacts/container/launch.py:478(HTTP 500 after68s). Cold cache: pull (104–108s) + unpack exceedsCREATE_READY_TIMEOUT = 110s(crates/capsem-service/src/container_setup.rs:22, HTTP 504create_timeout;inspect-capsemthen deletes the half-created VM viaCreateTimeoutError.vm_id). Impact: images above roughly 1 GB uncompressed fail on stock0.7; typical eval images (< 500 MB) are unaffected. Mitigations available today:0.7'sPOST /images/pull(generated SDK operationcapsem._operations.pull_image; aHypervisor-level wrapper is part of 0.7: Polish Python SDK for the image runtime #303) pre-warms the host cache ahead ofcreateand removes the cold-cache case — we can add that pre-pull toinspect-capsemas a small follow-up; Minimal Capsem runtime with OCI application images (replaces profiles) #289's planned per-image.erofsfast path / unpack-once universal path removes most of the unpack cost. Proposed issue: "Container create:launch.py30sumocitimeout and 110sCREATE_READY_TIMEOUTare too small for >1 GB images".oci-cache-lock30s lease contention under concurrent pulls —Puller::blob(crates/capsem-assets/src/oci/pull.rs:343-383) holdscache.lease(&digest)across the whole layer download +publish, whilelock()atcache.rs:202polls withPollOpts::new("oci-cache-lock", Duration::from_secs(30)). Two concurrentPOST /vms/createsharing an uncached> 500 MBlayer: the second fails withOCI cache lock timed out. Impact: only the first cold pull of a large shared layer undermax_sandboxes > 1; retrying after the first pull completes succeeds. Proposed issue: "OCI blob cache: lease timeout (30s) is shorter thanPULL_TIMEOUT(300s); hold the lock only around the atomic publish".ProvisionRequest— benchmarks that emitnetwork_mode: none(swe_bench_verified_minidefaultallow_internet=False,bigcodebench,gdm_intercode_ctf) are rejected at parse time rather than silently given network access, because0.7has no per-VM egress control. Impact: those tasks must be run with theirallow_internet=truestyle flags (as in the table above). Proposed issue: "Expose a per-VM network egress-deny option inProvisionRequestso Composenetwork_mode: nonecan be enforced by the gateway".inspect-capsemfails closed when a Compose file defines more than one service (e.g.cybenchdefault+victim). Minimal Capsem runtime with OCI application images (replaces profiles) #289 states the0.7runtime "does not ... introduce multi-container orchestration", so this is a documented limitation rather than a follow-up we expect0.7.xto lift;inspect-capsemstays strictly single-workload.