p2p_registry hardcodes "libcudart.so" — FileNotFoundError during NVFP4 dequantize (MiniMax H3)

[lumennoxx-sketch]

Environment

• OS: Windows 11
• ComfyUI: Desktop app, core up to date (commit d8e7bbc)
• ComfyUI-MultiGPU version: 2.6.4 (latest available)
• GPUs: RTX 5080 (cuda:0) + 2x RTX 3080 (cuda:1, cuda:2)
• PyTorch 2.13.0+cu130

Description
When using DisTorch2MultiGPU loaders with MiniMax H3's NVFP4-quantized text
encoder (qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors), execution fails
because p2p_registry.py hardcodes a Linux shared library name with no
Windows fallback:

p2p_registry.py, line 20:
_libcudart = ctypes.CDLL("libcudart.so")

This should resolve to cudart64_*.dll on Windows instead.

Steps to reproduce

1. Load the official MiniMax H3 I2V/T2V workflow
2. Replace UNETLoader/CLIPLoader/VAELoader with the DisTorch2MultiGPU
   equivalents, split across cuda:0/cuda:1/cuda:2 (CLIP donor_device: cpu)
3. Queue a generation
4. NVFP4 dequantization triggers a P2P device-access check that crashes

Expected behavior
The CUDA runtime lookup in _get_libcudart() should work cross-platform,
or the P2P check should skip gracefully when it can't load the library.

Actual behavior
FileNotFoundError, full relevant traceback:

File "...\custom_nodes\comfy_kitchen\backends\cuda\__init__.py", line 658, in dequantize_nvfp4
    _wrap_for_dlpack(qx),
  File "...\custom_nodes\comfyui-multigpu\__init__.py", line 557, in wrap_for_dlpack_with_device_guard
    if tensor_device.index != exec_device.index and not p2p_registry.can_access_peer(tensor_device.index, exec_device.index):
  File "...\custom_nodes\comfyui-multigpu\p2p_registry.py", line 62, in can_access_peer
    result = self._raw_can_access_peer(src_device, dst_device)
  File "...\custom_nodes\comfyui-multigpu\p2p_registry.py", line 38, in _raw_can_access_peer
    lib = _get_libcudart()
  File "...\custom_nodes\comfyui-multigpu\p2p_registry.py", line 20, in _get_libcudart
    _libcudart = ctypes.CDLL("libcudart.so")
FileNotFoundError: Could not find module 'libcudart.so' (or one of its
dependencies). Try using the full path with constructor syntax.

Notes
Also reproduces when the donor_device is cpu (not just GPU-to-GPU),
suggesting the P2P check may be firing in cases where it shouldn't need to
run at all.

[lumennoxx-sketch]

Update: also reproduces in RoPE application during VAE decode, not just NVFP4 dequantize

Confirmed this is broader than the CLIP text-encoder path. After swapping the
CLIP loader to a plain CLIPLoader (single GPU, no MultiGPU wrapper) as a
workaround, text encoding and UNET sampling both completed successfully
(382s, full 20 steps). The identical crash then occurred later, during VAE
decode, while VAELoaderDisTorch2MultiGPU was still in use for the video/audio
VAEs.

This time the call chain enters through comfy.ldm.minimax.vae.py's RoPE
application rather than NVFP4 dequantization:

File "...\comfy\ldm\minimax\vae.py", line 237, in forward
    query[..., :rot], key[..., :rot] = comfy.quant_ops.ck.apply_rope_split_half(
  File "...\comfy_kitchen\__init__.py", line 735, in apply_rope_split_half
    return torch.ops.comfy_kitchen.apply_rope_split_half(xq, xk, freqs_cis)
  File "...\comfy_kitchen\backends\cuda\__init__.py", line 2393, in _apply_rope_cuda
    _wrap_for_dlpack(xq),
  File "...\custom_nodes\comfyui-multigpu\__init__.py", line 557, in wrap_for_dlpack_with_device_guard
    if tensor_device.index != exec_device.index and not p2p_registry.can_access_peer(...)
  File "...\custom_nodes\comfyui-multigpu\p2p_registry.py", line 38, in _raw_can_access_peer
    lib = _get_libcudart()
  File "...\custom_nodes\comfyui-multigpu\p2p_registry.py", line 20, in _get_libcudart
    _libcudart = ctypes.CDLL("libcudart.so")
FileNotFoundError: Could not find module 'libcudart.so' (or one of its dependencies).

Interesting data point: the UNET (minimax_h3_fl2va_pruned_int8_convrot,
INT8 quantization) is split across 3 GPUs via expert_mode_allocations and
completes fine — it never hits this code path. Only operations that route
through comfy_kitchen's custom CUDA ops needing a DLPack bridge (NVFP4
dequant, and apparently RoPE application too) trigger wrap_for_dlpack_with_device_guard.

This suggests the fix in _get_libcudart() (or a Windows-safe skip in
can_access_peer) needs to cover the general DLPack-wrap path, not just the
NVFP4-specific one — any comfy_kitchen op that calls into
wrap_for_dlpack_with_device_guard on Windows will hit this regardless of
which quantization scheme triggered it.

Current workaround: use plain (non-DisTorch2MultiGPU) loaders for CLIP and
VAE, keeping only the UNET on MultiGPU. Full pipeline works end-to-end this way.

[lumennoxx-sketch]

Update: after patching _get_libcudart() for Windows, hit a full system hang instead of a crash — likely a missing P2P fallback between mismatched GPU generations

To get further in testing, I patched _get_libcudart() locally to load the
correct CUDA runtime library on Windows (locating cudart64_*.dll from the
installed PyTorch build instead of hardcoding libcudart.so). This resolved
the original FileNotFoundError from the comment above.

However, with the patch applied, the same workflow (CLIP + VAE on
DisTorch2MultiGPU, UNET split via expert_mode_allocations) now hangs
completely instead of crashing — no exception, no error message, just an
unrecoverable freeze requiring a hard power-off.

Hardware detail that's likely relevant: my setup mixes GPU generations —
an RTX 5080 (Blackwell, cuda:0) with two RTX 3080s (Ampere, cuda:1/cuda:2).
No NVLink, and as far as I know no P2P support between these architectures.
My guess: once the CUDA runtime loads correctly, cudaDeviceCanAccessPeer
correctly returns False for this pair — but whatever is supposed to happen
next (falling back to a host-staged copy instead of direct P2P) either
doesn't exist or hangs indefinitely instead of completing the transfer.

How I confirmed it was a real hang, not just slow:

• Two Task Manager / Core Temp screenshots taken minutes apart showed
  byte-for-byte identical timestamps, load percentages, and per-core
  temperatures — nothing was updating.
• Keyboard Caps Lock LED stopped responding to toggling, which on Windows
  only happens when the OS itself is fully frozen, not just busy/slow.
• No recovery after 5+ minutes; required a hard power-off.

Reproduction: same workflow as before, but the failure point moved past
the original NVFP4/RoPE dequantize calls (those now succeed) and hangs
later in the pipeline, during VAE decode.

This seems more serious than the original crash — an exception at least lets
the user retry; a full system hang risks data loss and forces a hard reset.
Happy to test a fix for the no-P2P fallback path if someone can point me to
where that should live in the codebase.

[ylzbj1-stack]

On the hang you hit after patching the runtime load: that's consistent with what the guard
does next. On GeForce cards P2P is disabled at the driver level, so can_access_peer gets
False and falls through to the staging path - and that path is where packed quantized
tensors die. Same family as #213 (cudaErrorIllegalAddress abort) and #220 (int8 failing on
a non-default compute device): every code path that moves a quantized weight across devices
ends up in comfy_kitchen's dequant/requant dispatch, and that dispatch doesn't reliably
survive the trip.

I hit this on 2x 3080 20G with H3 (int8_convrot, Windows, torch 2.10+cu130). What worked
for me was to stop moving weights at all: split the DiT blocks across both cards once at
load time and keep them there, no per-step cross-device shuffling. Same model family as
yours (mine's int8_convrot rather than NVFP4), so it may drop in directly -
https://github.com/ylzbj1-stack/ComfyUI-StaticPipeline

Can't promise it covers your mixed-generation layout (5080 + 3080 is not something I can
test), but if the DiT can be split statically, the P2P probe and the staging path stop being
in the picture entirely. If you try it and something blows up, paste the "split:" log line
and I'll take a look.

(FWIW the direction in #216 looks right to me - load the runtime cross-platform, and make
the probe non-blocking when peer access isn't there.)