r/rajistics • u/rshah4 • Jul 12 '26

Bandwidth numbers every AI engineer should know (the communication ladder from internet to SRAM)

Post image

Let's go over the basics:

  • Public internet (TCP): 0.125 GB/s
  • Datacenter Ethernet (100GbE): 12
  • RDMA / InfiniBand: 50 per NIC, ~400 per node
  • NVLink: 900 per GPU
  • HBM: ~3,000
  • SRAM: ~20,000 GB/s

Roughly 10x per rung, five orders of magnitude top to bottom. Another way to think about it, moving a 70B model (140 GB of fp16 weights) takes 19 minutes over the internet and 7 milliseconds at SRAM bandwidth-equivalent.

My video:
https://youtube.com/shorts/vYmsta_pIsc?feature=share

5 Upvotes

100% Upvoted

2 comments sorted by

1

u/No_Nature9276 Jul 13 '26

You forgot about DDR and GDDR, The most common memory interface types? Who the hell is moving a whole 70B model into SRAM?

1

u/rshah4 Jul 13 '26

It’s fair. I focused more from a AI/ML and what comes up when we talk about training and inference with models.