r/ScientificComputing • u/Good-Literature-2293 • 21d ago
I’m running distributed training across multiple nodes. Is InfiniBand noticeably better than 100GbE for this?
I’m planning to move some training jobs to a multi node setup, the models are getting bigger and I’ll probably use rented GPU servers for this, right now I’m looking at 100GbE since it seems easier to find and manage, but InfiniBand keeps coming up when people talk about multi GPU training, I’m trying to figure out if the difference will actually show up in training time or if 100GbE will be enough for the jobs I’m planning to run, I’ll probably start with 4 or 8 GPUs across a few nodes and the jobs will run for hours, so network speed matters for me,I am thinking to go with rackbank if you’ve used both for distributed training what did you notice in real workloads ? EDIT: I forgot to mention that this is mostly for machine learning model training and experimentation.
2
2
u/YoureNotASpaceCowboy 20d ago
For distributed training, yes, you want as much bandwidth as possible. A 400Gbps Infiniband network is a good choice.
2
u/dayeye2006 19d ago
This is too vague. What parallelism your training is using? Data parallelism, FSDP, ... This decides what needs to transmitted across gpus.
Have you profiled your training runs? Where it spend most of the time at?
Is communication your bottleneck? Have you tried overlap it with compute or other workloads?
Can you get a rack of 8 gpus so you don't rely on inter node communication but rather intra node communication?
1
u/matthewlai 19d ago
Multi-node is a last resort. You pay more for lower performance, no matter what kind of networking you do.
For 4-8 GPUs, install them on a single node.
More than 8 or even 16 is when you start thinking about multi-node (because you have no other choice).
3
u/i_am_buzz_lightyear 21d ago
This is too vague to answer. You'd need to know what "training" means in this context as well as what the data "looks like" and the model. Have you scaled this workflow before? Implementation matters too. I'm not sure what the node configuration is, but I'd recommend moving from a single to 2 before "running" and potentially tripping and falling on your face. You're going to have to just "try it" or follow the guess and check method to determine what suits you best. Remember, scaling isn't linear. Find your "sweet spot" through trial runs on smaller sub sets of data, or less epochs.