[WSL2][Onboard] managed llama.cpp onboarding fails with exit 28 on N1x WoA

[wangericnv]

Note

Re-verified reopening of #11823 — closed via #11955 (fix(inference): honor llama bridge readiness budget), which built on #11444 (fix(inference): host-process llama.cpp bridge probe on Docker Desktop WSL). Both merge commits
are confirmed ancestors of v0.0.128 (0e5f00db0a3a and 1e097a8161fa).
Re-verification on v0.0.128 still reproduces the original symptom, and this
run captured root-cause evidence that was not in the original report — see
Re-verification on v0.0.128 below.

Description

Onboarding with managed llama.cpp on an N1x Windows-on-ARM host (WSL2 + Docker
Desktop) fails at step 3/8 (Configuring inference provider) with Docker llama.cpp private loopback bridge probe failed (exit 28). The managed llama.cpp
container does not become ready before the probe gives up.

Re-verification on v0.0.128 shows the container is not actually failing to
start: it loads the model and serves successfully, but on a different port than
the one the bridge and probe poll. The readiness check therefore can never
succeed and burns its whole budget before aborting.

Platform scope: Reproduced on N1x Windows on ARM (WSL2 ARM64) only; other
platforms not tested.
Regression: Unknown — the host-process probe path was introduced by #11444.
OpenShell issue: No.

Environment

Re-verification run (v0.0.128):

Device:        N1x Windows on ARM
OS:            Windows 11 ARM64 + WSL2 Ubuntu 24.04 LTS
Architecture:  aarch64
Docker:        Docker Desktop backend, server 29.8.0
GPU:           NVIDIA RTX Spark N1X (6144-core Blackwell RTX GPU), 63936 MB
OpenShell CLI: openshell 0.0.116
NemoClaw:      nemoclaw v0.0.128
OpenClaw:      N/A (onboard not completed)

Original report (v0.0.124):

Node.js:       v22.23.2
npm:           10.9.8
Docker:        Docker version 29.7.2 (Docker Desktop, engine 29.7.2)
GPU Driver:    NVIDIA-SMI 615.72 / KMD 616.79 / CUDA UMD 13.4
NemoClaw:      nemoclaw v0.0.124

Steps to Reproduce

1. On an N1x Windows-on-ARM host with Docker Desktop and WSL2 Ubuntu 24.04,
   install the build under test:

   curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=v0.0.128 bash

2. Drive the documented managed recipe onboard on a host that clears the managed
   WSL ARM route's readiness floors:

   HF_TOKEN={non-production canary} \
   NEMOCLAW_PROVIDER=install-llama-cpp \
   NEMOCLAW_LLAMACPP_RECIPE=llama-cpp.qwen3-6-35b-a3b.n1x-wsl.v1 \
   nemoclaw onboard --non-interactive --name {sandbox}

   (The original v0.0.124 report reached the same failure through express
   install: accept the license and the express summary, and let step 3/8 select
   install-llama-cpp.)

3. Wait for onboarding to finish.

Expected Result

Onboarding step 3/8 completes: the readiness probe waits for the llama.cpp
container to start and load the model, managed llama.cpp comes up, and onboarding
finishes with the sandbox Ready.

Actual Result

Onboarding aborts at step 3/8 with exit 28, after waiting out the full probe
budget, with no sandbox created:

[3/8] Configuring inference provider
[non-interactive] Provider: install-llama-cpp
Verifying the exact llama.cpp GGUF in the shared Hugging Face cache
Still working on Provider selection... (30s elapsed)
Pulling pinned managed-inference image ghcr.io/nvidia/nemoclaw/llama-cpp-server@sha256:9d0cddd7bcaf...
Managed llama.cpp install failed: Docker llama.cpp private loopback bridge probe failed (exit 28).
[non-interactive] Aborting: Docker llama.cpp private loopback bridge probe failed (exit 28).

Onboarding did not finish. Resume from the step that failed with:
  nemoclaw onboard --resume

The image itself pulls and caches fine (5.01 GB). The timeout happens during
container startup and readiness polling, not during the pull.

Re-verification on v0.0.128 — evidence not in the original report

Host qualification is not the problem on this run. Docker memory
58,844,774,400 bytes and GPU memory 67,041,755,136 bytes both clear the recipe's
50,331,648,000 floor, the container GPU proof passed, and the product selected
the managed route and proceeded to model acquisition.

The model is acquired and verified — Model download complete, 20.4 GB
Qwen3.6-35B-A3B-Q4_K_M.gguf.

The managed container then starts correctly and serves, per its own log:

srv  load_model: loading model '/models/Qwen3.6-35B-A3B-Q4_K_M.gguf'
srv  llama_server: model loaded
srv  llama_server: listening on http://127.0.0.1:8082

But the bridge and the probe both target 8081:

docker-llama-cpp-private-bridge-process.js --target-host {container ip}
  --target-port 8081 --listen-port 8081
docker-llama-cpp-private-bridge-probe-process.js
  http://127.0.0.1:8081/health 1800

Probing that endpoint directly returns nothing:

curl http://127.0.0.1:8081/health   ->   http=000 after 30s

So the server is healthy on 8082 while readiness is polled on 8081, and
the server is bound to the container's own loopback rather than an address the
bridge can reach. The probe cannot succeed, waits out its 1800 s budget, and
onboarding fails with exit 28.

Related

• [WSL2][Onboard] managed llama.cpp onboarding fails with exit 28 on N1x WoA #11823 — the closed issue this re-verification supersedes.
• [WSL2][Onboard] managed llama.cpp private bridge probe fails on Docker Desktop WSL2 due to --network host namespace isolation #11436 — the earlier exit-7 failure of the same probe, fixed by fix(inference): host-process llama.cpp bridge probe on Docker Desktop WSL #11444.
• [Hermes/docker] Host-local llama.cpp per documented 127.0.0.1:8081 binding is unreachable from the sandbox inference route; onboarding completes without detecting it #11626 — still open, and possibly the same root cause: a host-local llama.cpp
  bound at 127.0.0.1:8081 is unreachable from the sandbox inference route. Both
  turn on what address and port the managed server is expected to be reachable
  on, so they are worth triaging together.