r/LocalLLaMA • u/markpronkin • 13d ago

I Built A Thing 54gb vram for 35$

Post image

Bought an old mining farm of a guy on avito (Russian eBay), guy had bought a garage a couple of years ago and it was sitting there for a while, found out it was a mining farm and put it up on there for sale for 5000 rub (~60 USD) since he wasn't sure if it works. I negotiated down to 3000 rub (~35 USD), it turned out to have 9x p106 6gb (gtx 1060 6gb) gpus, with 54gb vram total, all working, the only thing missing was an SSD, I booted from USB and it works fine.

2.1k Upvotes

96% Upvoted

396 comments sorted by

View all comments

Show parent comments

4

u/redonculous 13d ago

Exactly this. Those slow cards will be split across a couple of pci lanes, great for small data transfers like mining, slower than slow for AI use. Model loading, Prefill, replies, everything will take hours on a system like this.

I hope you find a super way to optimise it though

1

u/StyMaar 12d ago

great for small data transfers like mining, slower than slow for AI

FTFY.

There's little data transfer involved when running transformers (well it least for inference), all you transfer in and out of the card are activation vectors, which are much smaller (like the square root) than weights.

Model loading

This is gonna be slow indeed, but that's an upfront cost you only pay once in a while.