(vllm) ai_python@ubuntu-Super-Server:~/workspace/ComfyUI/models/diffusion_models$ nvidia-smi topo -m
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE 0-27,56-83 0 N/A
GPU1 NODE X 0-27,56-83 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
下面是加载minimaxh3生产1mp画质8s视频的报错，显存双卡共64G（5090），内存128G，我不用这个插件就不会oom了，而且这个插件似乎在0.4mp + 15s视频就没有问题
[MultiGPU CUDA Guard] Switching CUDA current device 0 -> 1 (NodeOverride.load_vae)[INFO] VAE load device: cuda:1, offload device: cpu, dtype: torch.float32[MultiGPU CUDA Guard] Restored CUDA current device 1 -> 0 (NodeOverride.load_vae)[MultiGPU CUDA Guard] Switching CUDA current device 0 -> 1 (NodeOverride.load_vae)[INFO] VAE load device: cuda:1, offload device: cpu, dtype: torch.float16[MultiGPU CUDA Guard] Restored CUDA current device 1 -> 0 (NodeOverride.load_vae)[MultiGPU Core Patching] Successfully patched ModelPatcher.partially_load[INFO] Found quantization metadata version 1[MultiGPU Core Patching] text_encoder_device_patched returning device: cuda:1 (current_text_encoder_device=cuda:1)[INFO] Using MixedPrecisionOps for text encoder[INFO] CLIP/text encoder model load device: cuda:1, offload device: cpu, current: cpu, dtype: torch.float16[MultiGPU DisTorch V2] Full allocation string: #cuda:1;30.0;cpu[INFO] Requested to load MiniMaxH3VideoVAE[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Requested to load MiniMaxH3TEModel_[INFO] Model MiniMaxH3TEModel_ prepared for dynamic VRAM loading. 14956MB Staged. 0 patches attached. Force pre-loaded 410 weights: 4572 KB.[MultiGPU P2P] can_access_peer(1, 0) = False[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[ERROR] !!! Exception during processing !!! CUDA error: an illegal memory access was encounteredSearch for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[ERROR] Traceback (most recent call last):File "/home/ai_python/workspace/ComfyUI/execution.py", line 545, in executeoutput_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/execution.py", line 344, in get_output_datareturn_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/execution.py", line 318, in _async_map_node_over_listawait process_inputs(input_dict, i)File "/home/ai_python/workspace/ComfyUI/execution.py", line 306, in process_inputsresult = f(**inputs)^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy_api/internal/init.py", line 149, in wrapped_funcreturn method(locked_class, **inputs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy_api/latest/_io.py", line 1935, in EXECUTE_NORMALIZEDto_return = cls.execute(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy_extras/nodes_minimax_h3.py", line 277, in executecond = clip.encode_from_tokens_scheduled(tokens)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd.py", line 336, in encode_from_tokens_scheduledpooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd.py", line 405, in encode_from_tokenso = self.cond_stage_model.encode_token_weights(tokens)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd1_clip.py", line 743, in encode_token_weightsout = getattr(self, self.clip).encode_token_weights(token_weight_pairs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/minimax.py", line 110, in encode_token_weightsout = super().encode_token_weights(token_weight_pairs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd1_clip.py", line 45, in encode_token_weightso = self.encode(to_encode)^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd1_clip.py", line 306, in encodereturn self(tokens)^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd1_clip.py", line 279, in forwardoutputs = self.transformer(None, attention_mask_model, embeds=embeds, num_tokens=num_tokens, intermediate_output=intermediate_output, final_layer_norm_intermediate=self.layer_norm_hidden_state, dtype=torch.float32, embeds_info=embeds_info)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/minimax.py", line 96, in forwardreturn super().forward(input_ids, attention_mask=attention_mask, embeds=embeds,^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/qwen3vl.py", line 100, in forwardreturn self.model(^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/llama.py", line 782, in forwardx, current_kv = layer(^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/llama.py", line 602, in forwardx, present_key_value = self.self_attn(^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/llama.py", line 526, in forwardxk = self.k_proj(hidden_states)^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 1407, in forwardoutput = self.forward_comfy_cast_weights(^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 1337, in forward_comfy_cast_weightsweight, bias, offload_stream = cast_bias_weight(^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 374, in cast_bias_weightweight, bias = resolve_cast_module_with_vbar(s, dtype, device, bias_dtype, compute_dtype, want_requant)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 321, in resolve_cast_module_with_vbarweight = post_cast(s, "weight", weight, dtype, prefetch["resident"], update_weight)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 302, in post_castx = to_dequant(x, dtype)^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 298, in to_dequanttensor = tensor.dequantize()^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/tensor/base.py", line 291, in dequantizefull = self.layout_cls.dequantize(qdata, self._params)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/tensor/nvfp4.py", line 88, in dequantizereturn ck.dequantize_nvfp4(qdata, params.scale, params.block_scale, params.orig_dtype)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/init.py", line 249, in dequantize_nvfp4return torch.ops.comfy_kitchen.dequantize_nvfp4(qx, per_tensor_scale, block_scales, dtype_code, hi_first)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_ops.py", line 1279, in callreturn self._op(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_library/custom_ops.py", line 441, in backend_implresult = self._backend_fns[device_type](*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_compile.py", line 54, in innerreturn disable_fn(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fnreturn fn(*args, **kwargs)^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_library/custom_ops.py", line 502, in wrapped_fnreturn fn(*args, **kwargs)^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/backends/eager/quantization.py", line 539, in _op_dequantize_nvfp4return impl(**kwargs)^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/backends/cuda/init.py", line 640, in dequantize_nvfp4_wrap_for_dlpack(qx),^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/custom_nodes/ComfyUI-MultiGPU/init.py", line 565, in wrap_for_dlpack_with_device_guardstaged_tensor = tensor.to("cpu").to(exec_device)^^^^^^^^^^^^^^^^torch.AcceleratorError: CUDA error: an illegal memory access was encounteredSearch for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):File "/home/ai_python/workspace/ComfyUI/execution.py", line 550, in executecomfy.model_management.reset_cast_buffers()File "/home/ai_python/workspace/ComfyUI/comfy/model_management.py", line 1414, in reset_cast_bufferssynchronize()File "/home/ai_python/workspace/ComfyUI/comfy/model_management.py", line 2023, in synchronizetorch.cuda.synchronize()File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/cuda/init.py", line 1219, in synchronizereturn torch._C._cuda_synchronize()^^^^^^^^^^^^^^^^^^^^^^^^^^^^torch.AcceleratorError: CUDA error: an illegal memory access was encounteredSearch for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[INFO] Prompt executed in 12.04 secondsterminate called after throwing an instance of 'c10::AcceleratorError'what(): CUDA error: an illegal memory access was encounteredSearch for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Exception raised from free_impl at /__w/pytorch/pytorch/c10/cuda/CUDAMallocAsyncAllocator.cpp:206 (most recent call first):frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x9d (0x7533de763bfd in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10.so)frame #1: + 0xc4e9 (0x7533e13ab4e9 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)frame #2: + 0x51820 (0x7533e13f0820 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)frame #3: + 0x51c84 (0x7533e13f0c84 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)frame #4: + 0x86ae4 (0x7533de740ae4 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10.so)frame #5: c10::TensorImpl::~TensorImpl() + 0x9 (0x7533de73b219 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10.so)frame #6: + 0x8752b5 (0x7533568752b5 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libtorch_python.so)frame #7: + 0x875341 (0x753356875341 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libtorch_python.so)frame #23: + 0x94ac3 (0x7533e3a94ac3 in /lib/x86_64-linux-gnu/libc.so.6)frame #24: + 0x1268c0 (0x7533e3b268c0 in /lib/x86_64-linux-gnu/libc.so.6)
Fatal Python error: Aborted
Stack (most recent call first):File "/home/ai_python/workspace/ComfyUI/main.py", line 416 in prompt_workerFile "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/threading.py", line 1012 in runFile "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/threading.py", line 1075 in _bootstrap_innerFile "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/threading.py", line 1032 in _bootstrap
Extension modules: sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, greenlet._greenlet, markupsafe._speedups, yaml._yaml, PIL._imaging, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, chardet.models, chardet.pipeline.ascii, chardet.pipeline.confusion, chardet.pipeline.escape, chardet.pipeline.magic, chardet.pipeline.statistical, chardet.pipeline.structural, chardet.pipeline.utf8, chardet.pipeline.utf1632, chardet.pipeline.validity, chardet.pipeline.orchestrator, charset_normalizer.md, charset_normalizer.cd, requests.packages.chardet.models, requests.packages.chardet.pipeline.ascii, requests.packages.chardet.pipeline.confusion, requests.packages.chardet.pipeline.escape, requests.packages.chardet.pipeline.magic, requests.packages.chardet.pipeline.statistical, requests.packages.chardet.pipeline.structural, requests.packages.chardet.pipeline.utf8, requests.packages.chardet.pipeline.utf1632, requests.packages.chardet.pipeline.validity, requests.packages.chardet.pipeline.orchestrator, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, psutil._psutil_linux, PIL._imagingft, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, scipy.special._ufuncs_cxx, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._batched_linalg, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._internal_matfuncs, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg.cython_lapack, scipy.linalg._decomp_update, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, _ni_label, scipy.ndimage._ni_label, regex._regex, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpacklib, scipy.sparse.linalg._propack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial.transform._rotation_cy, scipy.spatial.transform._rigid_transform_cy, scipy.optimize._direct, scipy.interpolate._fitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy.stats._ansari_swilk_statistics, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._rcont.rcont, scipy.stats._qmvnt_cy, av._core, av.logging, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, av.format, av.index, av.utils, av.stream, av.container.streams, av.sidedata.encparams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque, av.packet, av.container.input, av.bitstream, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.filter.loudnorm, av.audio.resampler, av.audio.codeccontext, av.audio.fifo, av.device, av.video.codeccontext, av.subtitles.stream, scipy.signal._sigtools, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.signal._sosfilt, scipy.signal._peak_finding_utils, cuda_utils (total: 205)已中止 (核心已转储)(comfy_new) ai_python@ubuntu-Super-Server:~/workspace/ComfyUI$ nvidia-smiWed Aug 5 09:02:10 2026+-----------------------------------------------------------------------------------------+| NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 |+-----------------------------------------+------------------------+----------------------+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC || Fan Temp Perf Pwr/Cap | Memory-Usage | GPU-Util Compute M. || | | MIG M. ||=========================================+========================+======================|| 0 NVIDIA GeForce RTX 5090 Off | 00000000:31:00.0 Off | N/A ||100% 33C P8 32W / 575W | 15MiB / 32607MiB | 0% Default || | | N/A |+-----------------------------------------+------------------------+----------------------+| 1 NVIDIA GeForce RTX 5090 Off | 00000000:4B:00.0 Off | N/A ||100% 32C P8 42W / 575W | 15MiB / 32607MiB | 0% Default || | | N/A |+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+| Processes: || GPU GI CI PID Type Process name GPU Memory || ID ID Usage ||=========================================================================================|| 0 N/A N/A 2199 G /usr/lib/xorg/Xorg 4MiB || 1 N/A N/A 2199 G /usr/lib/xorg/Xorg 4MiB |+-----------------------------------------------------------------------------------------+(comfy_new) ai_python@ubuntu-Super-Server:/workspace/ComfyUI$ pip list | grep comfyuicomfyui-embedded-docs 0.5.9comfyui_frontend_package 1.47.12comfyui-manager 4.2.2comfyui_workflow_templates 0.11.31comfyui-workflow-templates-core 0.3.295comfyui-workflow-templates-json 0.1.30comfyui-workflow-templates-media-api 0.3.84comfyui-workflow-templates-media-assets-01 0.1.19comfyui-workflow-templates-media-image 0.3.160comfyui-workflow-templates-media-other 0.3.229comfyui-workflow-templates-media-video 0.3.101(comfy_new) ai_python@ubuntu-Super-Server:/workspace/ComfyUI$ pip list | grep torchtorch 2.13.0+cu130torchaudio 2.11.0torchsde 0.2.6torchvision 0.28.0+cu130
(vllm) ai_python@ubuntu-Super-Server:~/workspace/ComfyUI/models/diffusion_models$ nvidia-smi topo -m
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE 0-27,56-83 0 N/A
GPU1 NODE X 0-27,56-83 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
下面是加载minimaxh3生产1mp画质8s视频的报错，显存双卡共64G（5090），内存128G，我不用这个插件就不会oom了，而且这个插件似乎在0.4mp + 15s视频就没有问题
[MultiGPU CUDA Guard] Switching CUDA current device 0 -> 1 (NodeOverride.load_vae)[INFO] VAE load device: cuda:1, offload device: cpu, dtype: torch.float32[MultiGPU CUDA Guard] Restored CUDA current device 1 -> 0 (NodeOverride.load_vae)[MultiGPU CUDA Guard] Switching CUDA current device 0 -> 1 (NodeOverride.load_vae)[INFO] VAE load device: cuda:1, offload device: cpu, dtype: torch.float16[MultiGPU CUDA Guard] Restored CUDA current device 1 -> 0 (NodeOverride.load_vae)[MultiGPU Core Patching] Successfully patched ModelPatcher.partially_load[INFO] Found quantization metadata version 1[MultiGPU Core Patching] text_encoder_device_patched returning device: cuda:1 (current_text_encoder_device=cuda:1)[INFO] Using MixedPrecisionOps for text encoder[INFO] CLIP/text encoder model load device: cuda:1, offload device: cpu, current: cpu, dtype: torch.float16[MultiGPU DisTorch V2] Full allocation string: #cuda:1;30.0;cpu[INFO] Requested to load MiniMaxH3VideoVAE[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.[INFO] Requested to load MiniMaxH3TEModel_[INFO] Model MiniMaxH3TEModel_ prepared for dynamic VRAM loading. 14956MB Staged. 0 patches attached. Force pre-loaded 410 weights: 4572 KB.[MultiGPU P2P] can_access_peer(1, 0) = False[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)[ERROR] !!! Exception during processing !!! CUDA error: an illegal memory access was encounteredSearch for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[ERROR] Traceback (most recent call last):File "/home/ai_python/workspace/ComfyUI/execution.py", line 545, in executeoutput_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/execution.py", line 344, in get_output_datareturn_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/execution.py", line 318, in _async_map_node_over_listawait process_inputs(input_dict, i)File "/home/ai_python/workspace/ComfyUI/execution.py", line 306, in process_inputsresult = f(**inputs)^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy_api/internal/init.py", line 149, in wrapped_funcreturn method(locked_class, **inputs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy_api/latest/_io.py", line 1935, in EXECUTE_NORMALIZEDto_return = cls.execute(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy_extras/nodes_minimax_h3.py", line 277, in executecond = clip.encode_from_tokens_scheduled(tokens)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd.py", line 336, in encode_from_tokens_scheduledpooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd.py", line 405, in encode_from_tokenso = self.cond_stage_model.encode_token_weights(tokens)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd1_clip.py", line 743, in encode_token_weightsout = getattr(self, self.clip).encode_token_weights(token_weight_pairs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/minimax.py", line 110, in encode_token_weightsout = super().encode_token_weights(token_weight_pairs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd1_clip.py", line 45, in encode_token_weightso = self.encode(to_encode)^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd1_clip.py", line 306, in encodereturn self(tokens)^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/sd1_clip.py", line 279, in forwardoutputs = self.transformer(None, attention_mask_model, embeds=embeds, num_tokens=num_tokens, intermediate_output=intermediate_output, final_layer_norm_intermediate=self.layer_norm_hidden_state, dtype=torch.float32, embeds_info=embeds_info)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/minimax.py", line 96, in forwardreturn super().forward(input_ids, attention_mask=attention_mask, embeds=embeds,^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/qwen3vl.py", line 100, in forwardreturn self.model(^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/llama.py", line 782, in forwardx, current_kv = layer(^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/llama.py", line 602, in forwardx, present_key_value = self.self_attn(^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/text_encoders/llama.py", line 526, in forwardxk = self.k_proj(hidden_states)^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 1407, in forwardoutput = self.forward_comfy_cast_weights(^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 1337, in forward_comfy_cast_weightsweight, bias, offload_stream = cast_bias_weight(^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 374, in cast_bias_weightweight, bias = resolve_cast_module_with_vbar(s, dtype, device, bias_dtype, compute_dtype, want_requant)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 321, in resolve_cast_module_with_vbarweight = post_cast(s, "weight", weight, dtype, prefetch["resident"], update_weight)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 302, in post_castx = to_dequant(x, dtype)^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/comfy/ops.py", line 298, in to_dequanttensor = tensor.dequantize()^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/tensor/base.py", line 291, in dequantizefull = self.layout_cls.dequantize(qdata, self._params)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/tensor/nvfp4.py", line 88, in dequantizereturn ck.dequantize_nvfp4(qdata, params.scale, params.block_scale, params.orig_dtype)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/init.py", line 249, in dequantize_nvfp4return torch.ops.comfy_kitchen.dequantize_nvfp4(qx, per_tensor_scale, block_scales, dtype_code, hi_first)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_ops.py", line 1279, in callreturn self._op(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_library/custom_ops.py", line 441, in backend_implresult = self._backend_fns[device_type](*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_compile.py", line 54, in innerreturn disable_fn(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fnreturn fn(*args, **kwargs)^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/_library/custom_ops.py", line 502, in wrapped_fnreturn fn(*args, **kwargs)^^^^^^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/backends/eager/quantization.py", line 539, in _op_dequantize_nvfp4return impl(**kwargs)^^^^^^^^^^^^^^File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/comfy_kitchen/backends/cuda/init.py", line 640, in dequantize_nvfp4_wrap_for_dlpack(qx),^^^^^^^^^^^^^^^^^^^^File "/home/ai_python/workspace/ComfyUI/custom_nodes/ComfyUI-MultiGPU/init.py", line 565, in wrap_for_dlpack_with_device_guardstaged_tensor = tensor.to("cpu").to(exec_device)^^^^^^^^^^^^^^^^torch.AcceleratorError: CUDA error: an illegal memory access was encounteredSearch for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):File "/home/ai_python/workspace/ComfyUI/execution.py", line 550, in executecomfy.model_management.reset_cast_buffers()File "/home/ai_python/workspace/ComfyUI/comfy/model_management.py", line 1414, in reset_cast_bufferssynchronize()File "/home/ai_python/workspace/ComfyUI/comfy/model_management.py", line 2023, in synchronizetorch.cuda.synchronize()File "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/cuda/init.py", line 1219, in synchronizereturn torch._C._cuda_synchronize()^^^^^^^^^^^^^^^^^^^^^^^^^^^^torch.AcceleratorError: CUDA error: an illegal memory access was encounteredSearch for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[INFO] Prompt executed in 12.04 secondsterminate called after throwing an instance of 'c10::AcceleratorError'what(): CUDA error: an illegal memory access was encounteredSearch for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Exception raised from free_impl at /__w/pytorch/pytorch/c10/cuda/CUDAMallocAsyncAllocator.cpp:206 (most recent call first):frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x9d (0x7533de763bfd in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10.so)frame #1: + 0xc4e9 (0x7533e13ab4e9 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)frame #2: + 0x51820 (0x7533e13f0820 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)frame #3: + 0x51c84 (0x7533e13f0c84 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)frame #4: + 0x86ae4 (0x7533de740ae4 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10.so)frame #5: c10::TensorImpl::~TensorImpl() + 0x9 (0x7533de73b219 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libc10.so)frame #6: + 0x8752b5 (0x7533568752b5 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libtorch_python.so)frame #7: + 0x875341 (0x753356875341 in /home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/site-packages/torch/lib/libtorch_python.so)frame #23: + 0x94ac3 (0x7533e3a94ac3 in /lib/x86_64-linux-gnu/libc.so.6)frame #24: + 0x1268c0 (0x7533e3b268c0 in /lib/x86_64-linux-gnu/libc.so.6)
Fatal Python error: Aborted
Stack (most recent call first):File "/home/ai_python/workspace/ComfyUI/main.py", line 416 in prompt_workerFile "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/threading.py", line 1012 in runFile "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/threading.py", line 1075 in _bootstrap_innerFile "/home/ai_python/miniconda3/envs/comfy_new/lib/python3.12/threading.py", line 1032 in _bootstrap
Extension modules: sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, greenlet._greenlet, markupsafe._speedups, yaml._yaml, PIL._imaging, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, chardet.models, chardet.pipeline.ascii, chardet.pipeline.confusion, chardet.pipeline.escape, chardet.pipeline.magic, chardet.pipeline.statistical, chardet.pipeline.structural, chardet.pipeline.utf8, chardet.pipeline.utf1632, chardet.pipeline.validity, chardet.pipeline.orchestrator, charset_normalizer.md, charset_normalizer.cd, requests.packages.chardet.models, requests.packages.chardet.pipeline.ascii, requests.packages.chardet.pipeline.confusion, requests.packages.chardet.pipeline.escape, requests.packages.chardet.pipeline.magic, requests.packages.chardet.pipeline.statistical, requests.packages.chardet.pipeline.structural, requests.packages.chardet.pipeline.utf8, requests.packages.chardet.pipeline.utf1632, requests.packages.chardet.pipeline.validity, requests.packages.chardet.pipeline.orchestrator, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, psutil._psutil_linux, PIL._imagingft, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, scipy.special._ufuncs_cxx, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._batched_linalg, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._internal_matfuncs, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg.cython_lapack, scipy.linalg._decomp_update, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, _ni_label, scipy.ndimage._ni_label, regex._regex, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpacklib, scipy.sparse.linalg._propack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial.transform._rotation_cy, scipy.spatial.transform._rigid_transform_cy, scipy.optimize._direct, scipy.interpolate._fitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy.stats._ansari_swilk_statistics, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._rcont.rcont, scipy.stats._qmvnt_cy, av._core, av.logging, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, av.format, av.index, av.utils, av.stream, av.container.streams, av.sidedata.encparams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque, av.packet, av.container.input, av.bitstream, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.filter.loudnorm, av.audio.resampler, av.audio.codeccontext, av.audio.fifo, av.device, av.video.codeccontext, av.subtitles.stream, scipy.signal._sigtools, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.signal._sosfilt, scipy.signal._peak_finding_utils, cuda_utils (total: 205)已中止 (核心已转储)(comfy_new) ai_python@ubuntu-Super-Server:~/workspace/ComfyUI$ nvidia-smiWed Aug 5 09:02:10 2026+-----------------------------------------------------------------------------------------+| NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 |+-----------------------------------------+------------------------+----------------------+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC || Fan Temp Perf Pwr/Cap | Memory-Usage | GPU-Util Compute M. || | | MIG M. ||=========================================+========================+======================|| 0 NVIDIA GeForce RTX 5090 Off | 00000000:31:00.0 Off | N/A ||100% 33C P8 32W / 575W | 15MiB / 32607MiB | 0% Default || | | N/A |+-----------------------------------------+------------------------+----------------------+| 1 NVIDIA GeForce RTX 5090 Off | 00000000:4B:00.0 Off | N/A ||100% 32C P8 42W / 575W | 15MiB / 32607MiB | 0% Default || | | N/A |+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+| Processes: || GPU GI CI PID Type Process name GPU Memory || ID ID Usage ||=========================================================================================|| 0 N/A N/A 2199 G /usr/lib/xorg/Xorg 4MiB || 1 N/A N/A 2199 G /usr/lib/xorg/Xorg 4MiB |+-----------------------------------------------------------------------------------------+(comfy_new) ai_python@ubuntu-Super-Server:
/workspace/ComfyUI$ pip list | grep comfyuicomfyui-embedded-docs 0.5.9comfyui_frontend_package 1.47.12comfyui-manager 4.2.2comfyui_workflow_templates 0.11.31comfyui-workflow-templates-core 0.3.295comfyui-workflow-templates-json 0.1.30comfyui-workflow-templates-media-api 0.3.84comfyui-workflow-templates-media-assets-01 0.1.19comfyui-workflow-templates-media-image 0.3.160comfyui-workflow-templates-media-other 0.3.229comfyui-workflow-templates-media-video 0.3.101(comfy_new) ai_python@ubuntu-Super-Server:/workspace/ComfyUI$ pip list | grep torchtorch 2.13.0+cu130torchaudio 2.11.0torchsde 0.2.6torchvision 0.28.0+cu130