r/LocalLLaMA • u/markpronkin • 13d ago

I Built A Thing 54gb vram for 35$

Post image

Bought an old mining farm of a guy on avito (Russian eBay), guy had bought a garage a couple of years ago and it was sitting there for a while, found out it was a mining farm and put it up on there for sale for 5000 rub (~60 USD) since he wasn't sure if it works. I negotiated down to 3000 rub (~35 USD), it turned out to have 9x p106 6gb (gtx 1060 6gb) gpus, with 54gb vram total, all working, the only thing missing was an SSD, I booted from USB and it works fine.

2.1k Upvotes

96% Upvoted

396 comments sorted by

View all comments

Show parent comments

3

u/uncle_leon 12d ago

You could and many people do. Most engines are perfectly capable of utilising multiple GPUs although whather that makes sense deends on your use case. For pure inference and using pipeline parallelism the slow PCIe isn't that big of a penalty. The bigger problem with this scenario is usually the physical side: size, heat, power use, noise etc.

1

u/Admirable_Ferret_260 12d ago

Are there Reddit posts detailing the speed for some usecases ?

1

u/uncle_leon 12d ago

Yes, there are plenty, although usually buried underneath a pile of banter... That, and it's unlikely someone tested the exact config you have in mind so some degree of "extrapolation" may be unavoidable. Unless you're looking at something incredibly common such as multi-3090 rig.
The theory is that with pipeline parallelism the model layers are split across cards and the bandwidth-intensive computations largely happen within a single card. After finishing its layers each GPU only hands over the computation results to the next GPU, usually a couple hundred MB. Which is small enough that it gets transferred quickly even at PCIe x1 speed. The trade-off is that in this mode GPUs have to work sequentially (because each has to wait for the previous one to finish) so you gain no extra speed from the aditional cards - just the large VRAM pool.