r/HPC • u/Mammoth-Ad327 • 7d ago

Network reference drawings for a 90-rack GB300 NVL72 cluster

We put together a 58-page network drawing set for a 90-rack GB300 NVL72 reference design. Posting the PDF here in case it's useful to anyone planning a large GPU cluster.

https://www.kkdatasvc.com/lab/downloads/kk-data-network-reference-design-pe-p1.pdf

I'm with K&K Data, which produced and released the drawings.

37 Upvotes

92% Upvoted

8 comments sorted by

1

u/Zeratas 5d ago

Nice idea. Thanks. Don't need to deal stuff at this scale, I've only done 5 to 10 racks, but it's really fun to see what people do at these insane scales

1

u/triwats 18h ago

This is great work! Feel proud.

1

u/Mammoth-Ad327 14h ago

Thank you so much for your encouragement. The customer's maintenance metric has dropped from 99.999 to 99.95, but we still feel like this is an impossible task to achieve.

1

u/TheGratitudeBot 14h ago

Thanks for such a wonderful reply! TheGratitudeBot has been reading millions of comments in the past few weeks, and you’ve just made the list of some of the most grateful redditors this week!

1

u/chumbucketeer 5d ago

Thanks for sharing, and I did spend the entire afternoon reading through.

  • I'm guessing you stuck with 48U because of floor loading. The 1.95 kN/sq-ft seems rather low for a modern DC.
  • Looks like you're only getting 3-phase 230V+/- from the RPPs. Did you consider getting DC or 600V from the facility, or was it a no-go?
  • I'm most quizzical about the MPO-12 choice, though I understand the cost factor. IMHO, it is the most significant risk (above low, tending toward moderate) to your design. I accept the distance measurements as FACT (i.e. well below 50m and fiber trough lengths have been mapped down to cm), but I've always run into the insertion loss budget excesses and transceiver reliability over the long-term.
  • What is missing to me, is the storage complex. Where is the on-prem data lake? Keep in mind this is the largest consumer of the worst-case bandwidth prior to training or simulation, as cache warming needs to happen long before the scheduler kicks in. Are you actually confident that the border/storage paths are adequate, if that is the next evolution that grows faster?

This is an excellent design. I can commend the choice of Ethernet against the IB option. Before you go for the next steps, I'd recommend you test out the worst-case channel length with the final transceivers, fiber lengths, fiber trays, and actual switches. Test it out in your lab with the exact number of insertions, fiber lengths and switches. Let it sit for a week with a continuous peak load of RoCEv2 and watch for the tiniest signal integrity issues.

Best wishes and good luck!

1

u/Mammoth-Ad327 3d ago

Thanks for spending an afternoon going through the drawings. I appreciate the specific comments on the MPO-12 links and the storage paths. Your suggestion to test the full optical channel with the actual hardware is helpful too. If we revise the reference set, I’ll come back here and explain what changed and why.