[Jetson][Sandbox] Sandbox GPU passthrough proof fails for the non-root sandbox user on JetPack 6.2 IGX Orin — onboarding aborts

[wangericnv]

Description

On JetPack 6.2 (L4T R36.5.1) IGX Orin, GPU detection succeeds but the end-to-end sandbox GPU passthrough proof fails for the unprivileged in-sandbox user, so nemoclaw onboard aborts at the GPU proof gate. The user can only complete onboarding by forcing CPU behavior with NEMOCLAW_SANDBOX_GPU=0.

Isolation shows the passthrough plumbing is correct and the gap is the non-root user: running cuInit(0) as root inside the same container returns 0 (success), while the unprivileged sandbox user (even with the --group-add video+995 the onboarder applies) cannot initialize CUDA.

Platform scope: Reproduced on JetPack 6.2 (R36.5.1) IGX Orin only; other Jetson variants / JetPack versions not tested. cuInit as root succeeds, so this may be IGX-Orin-specific device-access strictness.

Regression: Unknown — earlier versions not tested.

Environment

Device:        NVIDIA IGX Orin Developer Kit
OS:            Ubuntu 22.04 (JetPack 6.2, L4T R36.5.1)
Architecture:  aarch64
Node.js:       v22.23.1
npm:           10.9.8
Docker:        29.6.2 (installed via official convenience script; nvidia-container-toolkit 1.20.0~rc.1, CSV mode; nvidia runtime configured via `nvidia-ctk runtime configure --runtime=docker`)
OpenShell CLI: 0.0.85
NemoClaw:      v0.0.93
OpenClaw:      2026.7.1

Steps to Reproduce

1. Fresh install on IGX Orin JetPack 6.2:

   curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

2. Configure the NVIDIA container runtime for Docker:

   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker

3. Onboard against NVIDIA Endpoints (any cloud model):

   nemoclaw onboard --name jetson-vdr --agent openclaw

4. Watch [6/8] Creating sandbox → the GPU proof step.

Expected Result

The sandbox GPU proof passes and onboarding completes with sandbox GPU enabled.

Actual Result

Detection / preflight pass:

OK NVIDIA GPU detected (Orin (nvgpu), 54790 MB); Sandbox GPU: enabled (auto)
OK Granting sandbox user access to Jetson Tegra GPU device nodes via --group-add 44, 995
OK Docker container mode: --runtime nvidia (NVIDIA_VISIBLE_DEVICES=all)
OK GPU proof passed: nvidia-smi when available
OK GPU proof passed: /proc/{pid}/task/{tid}/comm write

But the CUDA proof fails, aborting onboarding:

Sandbox CUDA proof failed: cuInit(0) via libcuda.so.1
  NvRmMemInitNvmap failed with Permission denied 356: Memory Manager Not supported
  ****NvRmMemMgrInit failed**** error type: 196626 cuInit(0)=999
GPU proof failed inside an executable sandbox
-> throws at docker-gpu-sandbox-create.js verifyGpuOrExit; onboarding does not finish.

Isolation (device nodes ARE mounted via CSV: /dev/nvmap, /dev/nvgpu/igpu0/*, /dev/nvidia0, /dev/nvidiactl, /dev/nvsciipc, /dev/nvhost-*; libcuda present):

- cuInit(0) as ROOT in the container                 -> 0   (CUDA_SUCCESS)
- cuInit(0) as sandbox user (uid 998, +grp 44,995)   -> 801 (CUDA_ERROR_NOT_SUPPORTED)
- onboard proof (sandbox user)                       -> 999 (NvRmMemInitNvmap Permission denied)

Groups alone (video=44 + 995) are insufficient for the non-root user to init CUDA on this IGX Orin.

Workaround: NEMOCLAW_SANDBOX_GPU=0 (CPU) lets onboarding complete.

Logs

Sandbox CUDA proof failed: cuInit(0) via libcuda.so.1
  NvRmMemInitNvmap failed with Permission denied 356: Memory Manager Not supported
  ****NvRmMemMgrInit failed**** error type: 196626 cuInit(0)=999
proof_error=Sandbox GPU proof returned failed status: cuInit(0) via libcuda.so.1

Related

• VDR 5 checklist: non-DGX findings and current-release verification #7209 (Jetson GPU passthrough E2E verification item), PR test(e2e): migrate test-jetson-nvmap-gpu.sh to vitest #5613.
• [Jetson Orin][CLI&UX] nemoclaw status shows "Sandbox GPU: enabled" but CUDA is unusable inside sandbox — misleading status #4231 (prior "Sandbox GPU enabled but CUDA unusable" status bug, fixed).

[wscurran]

✨ Thanks for the detailed report. The reproduction steps, environment, and isolation findings are clear. Maintainers will investigate the non-root sandbox user GPU device access on JetPack 6.2 IGX Orin.

[wangericnv]

Retested on v0.0.97 — still reproduces (render-group fix insufficient)

Retested on 2026-07-30 on an NVIDIA IGX Orin Developer Kit (aarch64, JetPack 6.2 / L4T R36.5.1) with NemoClaw v0.0.97 (OpenShell 0.0.85). The bug still reproduces — the render-device-group fix is not sufficient on IGX Orin JP6.2.

The fix is applied — onboarding now grants the sandbox user the render group:

Granting sandbox user the detected Jetson GPU device groups via --group-add 44, 109, 995
(so CUDA can initialize as a non-root user)

(44 = video, 109 = render, 995 = debug)

But the end-to-end sandbox GPU proof still fails for the non-root sandbox user with the same error as the original report:

Sandbox CUDA proof failed: cuInit(0) via libcuda.so.1
  NvRmMemInitNvmap failed with Permission denied 356: Memory Manager Not supported
  ****NvRmMemMgrInit failed**** error type: 196626 cuInit(0)=999
GPU proof failed inside an executable sandbox

Onboarding still aborts; NEMOCLAW_SANDBOX_GPU=0 is still required to complete onboarding.

Root cause of the residual failure

The Tegra GPU nodes under /dev/nvgpu/igpu0/* are owned root:video (44) and root:debug (995), which the group-add already covers. But cuInit still fails at NvRmMemInitNvmap — the Tegra nvmap memory-manager path. Adding the render / video / debug groups does not grant the non-root sandbox user whatever access NvRmMemInitNvmap requires on IGX Orin JP6.2.

The fix likely needs to also cover the nvmap access path for the non-root sandbox user, not just the render device group.