Multiple CUDA GPU's not being utilised

[michaelarmstrong]

Hi, without setting up separate distributed inference instances I cannot seem to achieve a single instance of ds4-server utilising multiple CUDA capable GPU's on a single system (use case is dual RTX Pro 6000). Please advise if this is currently supported or not as I'd like to look into it if not.

[btothec]

I run this on one RTX Pro 6000 with the command below:

CUDA_VISIBLE_DEVICES=0
DS4_CUDA_COPY_MODEL=1
DS4_CUDA_NO_Q8_F16_CACHE=1
DS4_METAL_PREFILL_CHUNK=2048
./ds4-server
--cuda
-m ./ds4flash.gguf
--ctx 450000
--host 0.0.0.0
--port 8002
--cors
--kv-disk-dir ~/.ds4/server-kv
--kv-disk-space-mb 65536
--warm-weights

To be honest I run vllm / sglang 99% of the time so I am not sure, but I think if you just pass CUDA_VISIBLE_DEVICES=0,1 and the rest of my launch command you should be fine.

[edwardtian]

No — DS4's CUDA backend is single-GPU only. ds4_gpu_init() in ds4_cuda.cu hardcodes cudaSetDevice(0) and never enumerates or uses additional GPUs. There's no tensor parallelism, no model parallelism, and no multi-GPU CLI flags within a single process.

The only way to use both GPUs is the distributed mode — separate processes (coordinator + worker), each pinned to one GPU, communicating over TCP.

I forked a repo, you can run build_container.sh and then tune a little on the start-server.sh or run_*.sh