[Bug]: JIT-CDI graphics visibility is inconsistent on multi-GPU hosts: Vulkan works for one single-GPU pod but not another

[tzmtl]

Describe the bug

On a 2-GPU host using the CDI/JIT-CDI path, two pods are created with one GPU allocated to each pod. Vulkan graphics works in the pod assigned one GPU, but fails in the pod assigned the other GPU.

From checking the Toolkit source, CDI does inject graphics files and per-GPU device nodes, including EGL/GLVND/Vulkan configs and matching /dev/dri/* nodes. However, the CDI/JIT-CDI path does not appear to create the legacy NVIDIA application profile:

/etc/nvidia/nvidia-application-profiles-rc.d/10-container.conf

That legacy file sets EGLVisibleDGPUDevices for the allocated GPU minor. Without it, NVIDIA graphics userspace appears to still make some presentation/WSI decisions from a host-global view. This can make the GPU that matches the default/global graphics view work, while the other allocated GPU fails.

A downstream patch that restores this per-container application profile for the allocated GPU minor fixes the issue.

To Reproduce

Detailed steps to reproduce the issue.

1. Use a host with 2 NVIDIA GPUs.
2. Enable Toolkit CDI/JIT-CDI mode.
3. Create two Kubernetes pods, each requesting one NVIDIA GPU.
4. Confirm each pod is assigned a different GPU.
5. In each pod, run a Vulkan WSI/presentation test, for example an app or probe that checks queue presentSupport and attempts swapchain creation.
6. Compare the two pods.

Observed result:

• The pod assigned one GPU works with Vulkan.
• The pod assigned the other GPU fails Vulkan presentation/swapchain support.
• The failing pod has a selected GPU device set, but no /etc/nvidia/nvidia-application-profiles-rc.d/10-container.conf.

Expected behavior

Each pod allocated one full GPU through CDI/JIT-CDI should get a coherent one-GPU graphics view. Vulkan presentation and swapchain creation should work regardless of which physical GPU/minor the pod receives.

Environment
nvidia-container-toolkit version: [v1.19.1]
NVIDIA Driver Version: [580.82.07]
Host OS: [Ubuntu 24.04]
Kernel Version: [6.8.0-generic]
CPU Architecture: [x86_64]
GPU Model(s): [GeForce-RTX-5070-Ti]

[henry118]

Looking at the current CDI/CDI-JIT implementation, nvidia application profile isn't created in the CDI code path. In the legacy mode, the profile is created when the container has the "graphics/display" capability. Maybe we can add a hook to create the profile for CDI use cases?

@cdesiniotis WDYT?