Skip to content

fix(podman): trust an operator-supplied additional CA for direct egress - #3996

Open
politerealism wants to merge 1 commit into
NVIDIA:mainfrom
politerealism:3781-podman-additional-ca-minimal
Open

politerealism wants to merge 1 commit into
NVIDIA:mainfrom
politerealism:3781-podman-additional-ca-minimal

Conversation

@politerealism

Copy link
Copy Markdown
Contributor

Summary

Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to a destination signed by that CA fail upstream TLS establishment (curl: (52) Empty reply from server), and the only workaround is rebuilding the supervisor image with the CA baked in per upgrade or rotation.

This adds a Podman-only additional_ca_bundle setting that restores that trust without a custom image.

Related Issue

Fixes #3781.

This is a deliberately narrower, Podman-only alternative to the in-flight cross-driver implementation in #3292 (Docker/Podman/Kubernetes/VM delivery, gateway-level config schema, lifecycle rotation reconciliation), which remains the intended long-term direction — see the note posted on #3781. There are a few other things in flight that need to settle before taking on a change of #3292's size, so this fix is scoped to just what #3781 needs.

Changes

  • Add additional_ca_bundle to PodmanComputeConfig, independent of the existing proxy_ca_bundle (which requires https_proxy to be set) — this applies to direct, inspected egress regardless of whether a corporate proxy is configured.
  • Validate the bundle on the gateway host at sandbox-create time (read and parse it, not just check non-empty), so a misconfigured path or certificate-free bundle fails closed immediately instead of producing a failed sandbox once the supervisor container starts.
  • Bind-mount the bundle read-only into the supervisor container and pass its path via a new --additional-ca-bundle supervisor flag.
  • Fold it into the supervisor's upstream TLS trust store alongside (not instead of) the system CA bundle, reusing the exact additive-append mechanism already proven for the corporate proxy CA case — no changes to tls.rs's trust-store construction itself.
  • Document the new setting in docs/how-it-works/gateways/configuration.mdx and docs/how-it-works/sandboxes/runtimes.mdx.

Deliberately out of scope: Docker/Kubernetes/VM delivery, gateway.toml schema, new proto fields, and lifecycle/rotation reconciliation for stopped sandboxes — all left to #3292 or a future follow-up.

Testing

  • mise run pre-commit passes
  • cargo fmt --all -- --check passes
  • cargo clippy -p openshell-driver-podman -p openshell-supervisor-network -p openshell-supervisor -p openshell-core --all-targets -- -D warnings passes
  • cargo test -p openshell-driver-podman -p openshell-supervisor-network -p openshell-supervisor -p openshell-core --lib — 528 + 232 + 128 + 1458 tests pass, 0 failures
  • Real end-to-end TLS regression test added (additional_ca_enables_trust_of_privately_signed_upstream in tls.rs): a genuine rustls TlsAcceptor/TlsConnector handshake over a real loopback connection, reproducing bug: Podman supervisor cannot trust operator-supplied CA for upstream TLS after v0.1.2 architecture change #3781's exact failure without the CA trusted and proving it succeeds once it is
  • Config→mount→argv wiring tested directly (container.rs), including the no-https_proxy case that's the actual point of this fix
  • Fail-closed validation tested for empty, nonexistent, and certificate-free bundle paths

Checklist

  • Follows Conventional Commits
  • Commit is signed off (DCO)
  • Architecture/user-facing documentation updated

🤖 Generated with Claude Code

Since v0.1.2 the Podman supervisor runs as its own container, separate
from the workload image, so it can no longer inherit a private CA
baked into the workload's system trust store. Policy-allowed,
inspected HTTPS requests to destinations signed by that CA fail
upstream TLS establishment, and the only workaround is rebuilding the
supervisor image with the CA baked in.

Add a Podman-only additional_ca_bundle setting, independent of the
existing proxy_ca_bundle (which requires https_proxy), that bind-mounts
an operator PEM into the supervisor container and folds it into the
upstream trust store alongside the system CA bundle. Validated on the
gateway host at sandbox-create time, matching the existing corporate
proxy CA bundle's validation, so a misconfigured path or
certificate-free bundle fails closed immediately rather than producing
a failed sandbox once the supervisor container starts.

Scoped to Podman only, reusing the existing additive-bundle mechanism
without new gateway.toml schema, proto fields, or cross-driver
plumbing -- a narrower alternative to the in-flight cross-driver
implementation in NVIDIA#3292, which remains the intended long-term
direction.

Fixes NVIDIA#3781.

Signed-off-by: politerealism <burdcat17@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Podman supervisor cannot trust operator-supplied CA for upstream TLS after v0.1.2 architecture change

1 participant