vCUDA-kernel

Linux kernel enforcement module for vCUDA GPU virtualization.

vCUDA-kernel provides kernel-side NVIDIA GPU resource tracing, policy control, memory accounting, and compute timeslice enforcement. It is designed to work as the kernel component of the vCUDA stack together with vCUDA-core and the Kubernetes device-plugin integration.

HomePage

CFN-Cloud(In development...)

Build Dependencies

• Linux host with matching kernel headers
• NVIDIA Open GPU Kernel Modules reference submodule
• NVIDIA 570/580 driver target environment
• GSP firmware enabled for the primary development path
• CMake >= 3.16
• make and gcc
• CUDA toolkit, only for example workloads
• root privileges for module load and unload

Ubuntu example:

sudo apt update
sudo apt install build-essential cmake linux-headers-$(uname -r)

How to Use

build

1. initialize submodules

git submodule update --init --recursive

2. build kernel module

make clean
make

3. build example workloads

make example

4. build with CMake wrapper

cmake -S . -B build
cmake --build build

configure

# default dry-run load
make load

# enforcing compute timeslice rewrite
make reload DRY_RUN=0 ALLOW_ENFORCE=1 CLEAR_MEMORY_ON_LAST_CLOSE=0

# optional rewritten timeslice safety clamps
make reload DRY_RUN=0 ALLOW_ENFORCE=1 TIMESLICE_MIN_US=1000 TIMESLICE_MAX_US=10000

Useful load variables:

DRY_RUN=1
ALLOW_ENFORCE=0
MEMORY_TRACE=1
COMPUTE_TRACE=1
MEMORY_ALLOC_MIN_BYTES=16777216
MEMORY_ALLOC_MAX_BYTES=0
MEMORY_TRACE_LIMIT_BYTES=0
CLEAR_MEMORY_ON_LAST_CLOSE=1
TIMESLICE_MIN_US=0
TIMESLICE_MAX_US=0

usage

# inspect module state
make fingerprint
cat /sys/kernel/debug/vgpu/hooks
cat /sys/kernel/debug/vgpu/stats

# run CUDA allocation smoke test
./examples/cuda_malloc_smoke $((256 * 1024 * 1024)) 4 0

# run CUDA VMM smoke test
./examples/cuda_vmm_smoke $((256 * 1024 * 1024)) 4 0

# verify compute timeslice enforcement path
make verify-compute

# verify cgroup identity and policy UAPI path
make verify-cgroup
make verify-cgroup-policy
make verify-cgroup-compute
make verify-cgroup-memory

# unload module
make unload

Debugfs files:

/sys/kernel/debug/vgpu/enabled
/sys/kernel/debug/vgpu/driver_fingerprint
/sys/kernel/debug/vgpu/hooks
/sys/kernel/debug/vgpu/policies
/sys/kernel/debug/vgpu/tasks
/sys/kernel/debug/vgpu/events
/sys/kernel/debug/vgpu/timeslices
/sys/kernel/debug/vgpu/ioctls
/sys/kernel/debug/vgpu/stats

Test

module validation

make clean
make
make load
make fingerprint
nvidia-smi >/dev/null
cat /sys/kernel/debug/vgpu/stats
make unload

example validation

make example
./examples/cuda_malloc_smoke $((256 * 1024 * 1024)) 4 0
./examples/cuda_vmm_smoke $((256 * 1024 * 1024)) 4 0

compute enforcement validation

make verify-compute
make verify-compute VERIFY_COMPUTE_TIMESLICE_MIN_US=1000 VERIFY_COMPUTE_WEIGHT=1
make verify-compute VERIFY_COMPUTE_DRY_RUN=1 VERIFY_COMPUTE_ALLOW_ENFORCE=0

KUnit

make test-kunit

Features

GPU Virtualization Features

Base Features

• ✅ Linux kernel module: vgpu-kernel.ko
• ✅ NVIDIA driver fingerprint detection
• ✅ NVIDIA character-device open/ioctl/release tracing
• ✅ Debugfs diagnostics under /sys/kernel/debug/vgpu
• ✅ Control device: /dev/vgpuctl
• ✅ Per-task GPU context tracking
• ✅ Conservative GPU memory accounting
• ✅ CUDA VMM allocate/free object accounting
• ✅ RM_CONTROL tracing
• ✅ TSG timeslice dry-run tracing
• ✅ TSG timeslice rewrite enforcement
• ✅ KUnit test entrypoint
• ☐ Coverage report publishing
• ☐ cgroup policy ownership
• ☐ Kubernetes device-plugin integration
• ...

Cgroup Policy Roadmap

Cgroup policy control uses existing cgroup v2 identity plus /dev/vgpuctl policy injection. The kernel module will not register a custom Linux cgroup controller. See docs/cgroup-policy-design.md for the full design and docs/cgroup-policy-flows.md for mechanism flowcharts.

Cgroup Identity Plumbing

• ✅ Add core/vgpu_cgroup.c and core/vgpu_cgroup.h
• ✅ Implement vgpu_cgroup_current_id() wrapper
• ✅ Add cgroup_id to task snapshots
• ✅ Record cgroup id in NVIDIA open/ioctl path
• ✅ Show cgroup_id in /sys/kernel/debug/vgpu/tasks
• ✅ Add cgroup identity validation commands

Cgroup Policy UAPI

• ✅ Add struct vgpu_cgroup_policy
• ✅ Add struct vgpu_cgroup_policy_query
• ✅ Add VGPU_IOCTL_SET_CGROUP_POLICY
• ✅ Add VGPU_IOCTL_GET_CGROUP_POLICY
• ✅ Add cgroup policy rhashtable keyed by (cgroup_id, gpu_minor)
• ✅ Add /sys/kernel/debug/vgpu/cgroup_policies
• ✅ Add examples/vgpu_set_cgroup_policy
• ✅ Add KUnit coverage for cgroup policy validation and lookup

Cgroup Compute Enforcement

• ✅ Add policy resolver with priority tgid > cgroup > none
• ✅ Use cgroup compute policy in timeslice rewrite path
• ✅ Extend /sys/kernel/debug/vgpu/timeslices with policy_scope
• ✅ Extend /sys/kernel/debug/vgpu/timeslices with cgroup_id
• ✅ Add cgroup compute verification script

Cgroup Memory Accounting

• ✅ Add cgroup memory stats table
• ✅ Charge RM allocation objects into cgroup aggregate usage
• ✅ Free RM allocation objects from cgroup aggregate usage
• ✅ Apply cgroup memory limit in dry-run first
• ✅ Add /sys/kernel/debug/vgpu/cgroups
• ✅ Add KUnit coverage for cgroup stats charge/free/overflow

Device-Plugin Readiness

• ✅ Keep ../device-plugin workspace aligned with kernel UAPI
• ✅ Document device-plugin call sequence
• ✅ Add make verify-cgroup
• ✅ Add CMake verify-cgroup target
• ✅ Update README usage once cgroup policy path is verified

More Features

• ☐ Remote GPU call over network
• ☐ GPU memory oversubscription control
• ☐ Multi-GPU policy scheduling
• ☐ GPU task hot snapshot
• ...

Why This Project?

This project is built as the kernel enforcement layer for open GPU virtualization experiments.

• Kernel Enforcement: move GPU resource control below user-space LD_PRELOAD boundaries.
• Open Architecture: keep driver ABI research, tracing, and enforcement logic auditable.
• Kubernetes Ready: provide a kernel foundation for device-plugin and cgroup policy injection.
• Dynamic Controllability: allow runtime task policy updates through /dev/vgpuctl.
• Dry-run First: validate policy decisions through debugfs before enabling write-capable paths.

vCUDA-kernel is intended to be published separately from:

• vCUDA-core: user-space CUDA virtualization and remote call transport.
• vCUDA-device-plugin: Kubernetes device-plugin, CDI, and cgroup policy injection.

NVIDIA OKM Reference

This repository includes NVIDIA Open GPU Kernel Modules as a pinned reference submodule under third_party/open-gpu-kernel-modules.

The submodule is used for ABI/layout lookup and RM ioctl structure references. It is not built into vgpu-kernel.ko.

Contributing

Contributing guide

Code of conduct

License

GPL-2.0-only. Source files use SPDX license identifiers. UAPI headers use GPL-2.0-only WITH Linux-syscall-note, matching Linux kernel UAPI convention.