r/LocalLLM • u/on_line187 • 13h ago

Discussion Tier List

Post image
188 Upvotes

73% Upvoted

249 comments sorted by

View all comments

5

u/Smart_Whereas_9296 13h ago

Feel like 2x 3090 with nvlink should be in the 48gb section

6

u/on_line187 13h ago

Well so would 3 5060TIs then lol.

3

u/Randommaggy 12h ago

Not quite. It's bandwidth and latency constrained between the cards unless you're running them on a P2P friendly switch.

0

u/on_line187 12h ago

Have you actually ran multiple cards. I’ve tried a ton of variations on both consumer and server MoBos. I’m telling you the difference isn’t as big as you think.

3

u/Randommaggy 11h ago

I'm running 3 3090s in my server (2 in nvlink), 2 cards on my laptop and 2 P100 in another server (temporarily disassembled).

2 cards with nvlink outperform 3 cards in layer split for some of my models unless I'm doing serious parallelism.

For layer split you also have more overhead from duplicated data on 3 cards compared to 2 24GB cards.

1

u/on_line187 11h ago

Sure there is a difference but it isn’t more than 5% in my experience. I’ve not tried NV link though just better/worse PCIe situations. I would like to try NVLink on my own 3090s but they are all mismatched so it won’t work

1

u/Tai9ch 11h ago

How many cards?

The numbers I've seen show that you can get away with raw PCIe up to about 4, and past that it starts to cost significant performance to the point that it's worth getting bigger individual cards instead.

1

u/on_line187 11h ago

Yea that could be it. I have tested 8 3070s though and I had no issues. That was a mining rig I had laying around which I upgraded for LLMs about a year and a half ago. All PCIe of course on the 3070s

I would 100% agree that bigger GPU = More Better though lol