Is it feasible to run nccl-tests on two node connected to same GPU System over tensor-fusion

[alokprasad-mrvl]

Can two node connect to same GPU using tensor-fusion ?
has anyone tried running collective operations ( maybe running nccl-tests )

[Code2Life]

Does 'two node' refer to two Python applications running in separate pods? If so, theoretically yes - two instances running on vGPU should still be able to use NCCL, though we haven't tested this configuration yet

@alokprasad-mrvl Could you please provide more details about the specific test scenario? Is this intended for distributed training?

[alokprasad]

Basically i want to run all_reduce operation ( provided by nccl-tests) on A and B, but ideally they over some network ( Ethernet/IB)
and each node having physical GPU. But with Tensor-fusion my assumption is to Node A and Node B can avoid having physical GPU.
Is the vGPU mentioned above is NVIDIA vGPU or u meant tensor-fusion vGPU

        +----------------+           +----------------+
        |                |           |                |
        |  Node A (CPU)  |<--------->|                |
        |                |           |                |
        +----------------+           |                |
                                     |                |
                                     |  Node C (GPU)  |
        +----------------+           |                |
        |                |           |                |
        |  Node B (CPU)  |<--------->|                |
        |                |           |                |
        +----------------+           +----------------+

         <-------- TCP/UDP Communication -------->

[alokprasad]

nccl-test needs nccl library and cuda library , not sure without /sys entry and other will they work.

[Code2Life]

@alokprasad gotcha, I think nccl work at most cases, while the communication peers will be changed to vGPU(tensor fusion) guest to guest.

When it's running in remote GPU mode. NVLink/NVSwitch may not working as of now because of API remoting issues, but it's feasible.

we will test nccl cases later and follow-up on this, thanks for your question.

[alokprasad]

@Code2Life i can give it a try , normally i have to install nccl and nccl-tests linked to cuda library on both node
and both node accessible to each other via TCP or RoCE.
How i can use tensor-fusion here. maybe if you can point some good wiki for the same.

[Code2Life]

Hi @alokprasad the simplest way is to install helm chart on existing kubernetes cluster: https://tensor-fusion.ai/guide/recipes/deploy-k8s-local-mode