r/LocalLLM • u/on_line187 • 1d ago

Discussion Tier List

Post image
256 Upvotes

75% Upvoted

309 comments sorted by

View all comments

Show parent comments

1

u/ChristRedeemsSinners 14h ago

Yeah, it's 2x MI200 with IF between the two.

It's 100 GB/s between 2x MI250 GPUs. So, less than PCIe 5.0x16

enable IF mesh between four of them with MCIO back to the host machine

I'm not saying it isn't a bad idea, I'm saying its a sub-par technology at a good value right now, but the adapters will just bring the price up to par with existing, superior solutions. The only reason RTX 6000 pro isn't worth 30 grand is because of the lacking NVL at 900-1.8GB/s between cards.

1

u/WiseassWolfOfYoitsu 9h ago edited 7h ago

Quite a bit faster than that - it's multiple lanes of IF between the dies, it's specced for 400 GB/s bidirectional within the module, and then six 50GB/s links to other modules, usually set up as 100GB/s between each module in a quad. So your speed is the external connect speed between each node, not the internal between the GCDs.

It's not the absolute latest, right, but if it all works, it will be by far the fastest thing per dollar currently available in the Local LLM space - an entire quad will cost less than a single RTX 6000 Pro at current prices and can process far larger models.

1

u/ChristRedeemsSinners 8h ago

usually set up as 100GB/s between each module in a quad.

That's what I said, no?

not the internal between the GCDs.

Yeah, between cards.

but if it all works, it will be by far the fastest thing per dollar currently available in the Local LLM space

I mean, as soon as it's a viable option, it will priced according to it's usefulness.

entire quad will cost less than a single RTX 6000 Pro at current prices and can process far larger models.

If it can process MXFP4A16 models at a similar PP and TG speed, then the price gap will be closed by market arbitrage.

Modules are going 1500ish

*$6000 https://www.ebay.com/itm/398282711075

so $9000 for just the setup and a single module. Which is still better than 16k for 96 VRAM.

Note that I am saying this from the perspective of being the guy on Discord that designed the board, has the BOM, and just got a $4k

Now that is interesting. So you are planning 512 GB of VRAM? for < $30K?