feat: CDI-based NVIDIA GPU passthrough into gVisor actor containers

[eliranw]

Summary

Adds CDI-based NVIDIA GPU passthrough into gVisor actor containers, activated only on worker pods that have a GPU. Non-GPU actors are unchanged.

When a WorkerPool template requests nvidia.com/gpu:

• atecontroller mounts the host NVIDIA toolkit (/usr/local/nvidia/toolkit, read-only) into the worker pod.
• ateom-gvisor, at actor create/restore, detects /dev/nvidia0, enforces the ateom worker's GPU was passed using CDI Mode , generates a per-pod CDI spec once via nvidia-ctk (under reapLock), and injects it into each actor's OCI config.json via the CDI library before runsc create. runsc's --nvproxy auto-enables from the injected device nodes.

Why these choices

• Generate the CDI spec in-pod (not reuse the host /var/run/cdi): NVML in the worker pod sees only the GPU the device-plugin assigned, so the spec is correctly scoped per assigned GPU on multi-GPU / multi-worker nodes — no "which device is mine" resolution.
• Shell nvidia-ctk from the mounted toolkit (not vendor nvcdi): the toolkit mount is needed anyway for the runtime nvidia-cdi-hook binary, so this avoids the whole NVIDIA-toolkit Go dependency. Only the CDI library (tags.cncf.io/container-device-interface) is vendored, for the inject step.
• Require CDI mode: keeps all CDI hooks working under the worker pod. The legacy runtime's /proc/driver/nvidia overmount trips the kernel's mount_too_revealing() when runsc runs update-ldcache deprivileged in the gofer; CDI mode never creates that overmount.

Guarantees

• Non-GPU actors untouched — maybeInjectGPU returns nil before any side effect when /dev/nvidia0 is absent.
• Fail-fast — a GPU actor never silently starts CPU-only: legacy overmount, missing toolkit, generate failure, or unresolved device all abort create with a precise error.
• Both create and restore paths inject per application container (pause containers untouched).

Non-goals (out of scope, per design)

Non-privileged worker pods; rootless runsc; legacy-mode support; per-actor GPU API; MIG/vGPU/GDS; DRA

Testing

• Unit tests (GPU-free): controller GPU-gating, gpuPresent, enforceCDIMode (legacy/missing-toolkit/ok), generateCDISpec (fake nvidia-ctk), injectGPUIntoBundle (fixture CDI + OCI spec), maybeInjectGPU no-op. All green; GOOS=linux and native builds clean.

Live validation

• Control-plane (kind + NVIDIA nvml-mock, CDI mode): ran the full substrate flow — GPU WorkerPool → worker pod carries nvidia.com/gpu + the read-only /usr/local/nvidia/toolkit mount; ateom-gvisor detects /dev/nvidia0, passes the CDI-mode check, generates the per-pod CDI spec, injects it, and runsc create succeeds. A non-GPU control actor starts cleanly. (On the driverless mock, runsc start stops at nvproxy, as expected.)
• Real hardware (A10G, driver 580.159.03, CDI mode): with the injected device nodes and runsc --nvproxy, nvidia-smi runs inside the gVisor sandbox (sees the A10G) and a CUDA kernel (vectorAdd) executes and returns Test PASSED — real GPU compute through nvproxy.
• Negative check: on a legacy-mode cluster the actor fails to create with "GPU injection requires the cluster in CDI mode; detected legacy /proc/driver/nvidia overmount".

---

• Tests pass
• Appropriate changes to documentation are included in the PR

This PR was AI assisted. I tested it on kind, and on real A10G hardware, and reviewed the changes. The CDI-apply approach was contributed by Davanum Srinivas (@dims) (co-authored).

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[Davanum Srinivas (dims)]

eliranw is there a way to do this without introducing all the new vendored components? (21k lines of code is too much!)

[eliranw]

It seems like we decided to commit vendor packages in this repo. I think it’s best to use the cdi package instead of reimplementing it. As far as I understand, the main downside of committing the vendor folder (instead of gitignoring it) is having a large diff. Do you have a different solution?

To make it easier to review, I moved the vendor changes into a separate commit. I noticed many other PRs do the same thing - does this approach make sense to you?

[Davanum Srinivas (dims)]

eliranw it's not whether the vendor-ed stuff is a separate commit or not. just that we should avoid adding stuff if not really required.

see for example - main...dims:substrate:gpu-nocdi

[Benjamin Elder (BenTheElder)]

thanks for working on this

I'd also like to keep imported deps minimal.

I started looking at uVM without privileged and we may have a bigger problem there ... ideally I don't want to require a plugin on the host but we need permission to talk to KVM.

I may have an answer to that wrinkle but broadly speaking I'm also hoping that even if not implemented yet we can figure out how we'll extend this to uVM later, as I'm also working on closing the existing gap (durable directory support) at higher priority ...

[eliranw]

Thanks Davanum Srinivas (@dims) and Benjamin Elder (@BenTheElder). Dropped the vendored CDI as you both suggested and re-tested on a GPU node.

I’ve also gated the whole thing to gVisor-only for now so the NVIDIA toolkit mount only lands on gVisor pools, meaning a micro-VM pool that requests a GPU won’t pick up the nvidia toolkit.

On uVM - I'll start looking into that