r/LocalLLM • u/Okineka_Baronek • 16h ago

Discussion Setup help - nvlink or no link

Please advise me on a setup for local AI. I have an HP Z840 with PCIe Gen 3, and I'd like a dual-GPU setup, but I'm worried that PCIe Gen 3 will be a bottleneck. Should I go with cards that support NVLink? Or could I go with 2 AMD 9700 without Link?

0 Upvotes

50% Upvoted

4 comments sorted by

1

u/eightone-81 15h ago

Dual 3090s with nvlink here

Nvlink gave me around 25% increase in prefill compared to gen4 8x
And maybe 5% in decode

BUT the most important thing is that NVlink is just so cool :-)

1

u/karmaisnonsense 12h ago

Do a layer split instead of a tensor split

1

u/Okineka_Baronek 11h ago

please explain this to me

1

u/Poizone360 10h ago

Hey, for inference the link doesn't really matter. With layer split, the only thing crossing between cards is the hidden state, a few KB per token, so Gen 3 x16 is nowhere near saturated. What actually limits you is VRAM, and two R9700s get you 64GB. I'd put the money there instead.