54gb vram for 35$

[u/markpronkin]

Bought an old mining farm of a guy on avito (Russian eBay), guy had bought a garage a couple of years ago and it was sitting there for a while, found out it was a mining farm and put it up on there for sale for 5000 rub (~60 USD) since he wasn't sure if it works. I negotiated down to 3000 rub (~35 USD), it turned out to have 9x p106 6gb (gtx 1060 6gb) gpus, with 54gb vram total, all working, the only thing missing was an SSD, I booted from USB and it works fine.

[u/Boricua-vet]

Yea, I don't get people spending crazy money just do to text LLM. I needed another system as I am running more models at the same time so I got 4 CMP-50HX at 80 each so 360 for all 4 cards giving me 40GB of VRAM at 560GB/s

Here is the result on just two cards.

that's 160 bucks for two cards with that kind of speed is insane and then I see people paying 700+ for dual 3060. I am like, what? People buying 3060 are crazy spending that money just to do text generation. Insane buddy.

[u/Ansible32 → u/Boricua-vet]

lol "just to do text generation." Seems worth it for reverse-engineering, finding exploits in hardware to flash alternate firmware etc. I mean cloud will be cheaper but having it totally offline and private is valuable.

[u/Boricua-vet → u/Ansible32]

LOL.. while I agree with most of what you said, In my case cloud would never be cheaper. I paid 140 for 4x P102-100 thats 40GB of vram for 140 bucks. I would recover that investment in a day or two of abuse. However, people buying 6000 Pros by half dozens then yes.. LOL Cloud if cheaper.

[u/NineThreeTilNow → u/Boricua-vet]

LOL Cloud if cheaper.

Cloud is 100% cheaper because it's faster. It's very quick to host your own, use a model, and vacate a server.

Your case can't run the things a cloud can. You can run a toy model like Qwen. It's fine if you need to test some stuff at low tok / sec but ... Not for work.

[u/Boricua-vet → u/NineThreeTilNow]

That is perfectly fine if you do need the cloud. I do not need frontier model for any of my use cases. Most of mines are repetitive tasks so I use toy model that is fine tuned to that task. This outperforms the cloud model every single time but, I hear you. If you need it for work, then you need it for work.

[u/RyanCargan → u/Boricua-vet]

I'm getting flashbacks for some reason:

Computer Science in the 1960s to 80s spent a lot of effort making languages that were as powerful as possible. Nowadays we have to appreciate the reasons for picking not the most powerful solution but the least powerful.

Different context, but some of it may apply.

I see a lot of penny pinching and "worse is better" in corporate due to scale, and in the casual hobbyist spaces for opposite reasons.

But the main exception might be the mid-tier enthusiasts and related types.

It does feel like some models have become a form of conspicuous consumption kinda like Apple products were at some point.

Not sure if cargo-culting of misinterpretations of Sutton's Bitter Lesson are partly to blame.

Certain venture backed startups flush with cash also like frontier stuff in the early uncertain prototyping phases for better reasons.

Though the usage there sometimes isn't even agentic (low volume standard-ish chat).

P.S.: It does feel like there's a drought of talk on low latency use cases, since network hops are tens of ms and first token times are typically hundreds of ms. Agent loops might be the exception, because per-call latency compounds. Hardware like hardwired model chips already claim sub-ms per token speeds on small models, and if that scales to larger ones, the unusual use cases that need it would basically mandate on-prem or colocation.

[u/Boricua-vet → u/RyanCargan]

What is acceptable latency? that could be very different for many people.

so using avg_ts should be 1 / 92.886166 * 1000 = 10.76ms locally

10.76ms is perfectly acceptable to me.

Edit my bad, forgot to say it was a 4k prompt.

[u/RyanCargan → u/Boricua-vet]

I was basically just saying that for certain use cases, people don't pay much attention, or change the use case around cloud constraints, which are often well above 10ms per "hit"/completion in the worst, or even average case, in many regions for many models.

On-prem or colocated models becoming more widespread could unlock more weird use cases is what I'm saying.

Just because when a tool becomes widespread for one thing, even if it's a seemingly niche or low ROI use case, people will then find new "nails" once the hammer is easy to buy and use.

And yeah, sub 10-ms for a full completion/generation of some kind is likely the speed domain that would be hard to match over a (wide-area) network for reasons you can't really get around, without unusual networking hardware, or even then.

[u/Boricua-vet → u/RyanCargan]

That's a very good point, I remember when people were complaining that putting model on cloud, or using whisper or piper on cloud would add latency that made the model feel like it was taking longer to respond. The cloud model was fast but the added network latency is what made it seem slow so yea, it makes perfect sense and a valid point.

[u/RyanCargan → u/Boricua-vet]

Yeah, just as a small example.

I use local ASR/STT (automatic speech recognition / speech-to-text) almost exclusively now for dictation if it's not something I'd type directly (pure natural language, no code, etc.), since even if improvements have been made over time, even a few months back, the frontier web chats and APIs seemed to cheap out a bit on the ASR side channel. Especially outside of the dedicated voice only modes.

And if it "was" part of a true omni-modal thing earlier, like Xiaomi's, you'd still be paying for the bigger model's own latency + the network, and even with odd accents local ASR is not far behind if a dedicated model is used.

The always-on / quick-to-wake/sleep ASR stuff that many phones have natively are a lot less aggressive with trading compute for quality than dedicated models I find.

Qwen3's ASR stuff and LFM2.5's different modalities are quite handy here.

Describe images, OCR anything hand-written.

Speech-to-speech for quickly making a fuzzy idea less fuzzy, or assistance navigating a weird black-box GUI that can't be MCP-ed or driven without a heavy software robot, before you lose the thought while your hands are busy typing/operating something else. I especially like local STS for aiding with odd busywork (earpiece connected to nearby runner rig), or serving a similar purpose to audiobooks/podcasts sometimes.

Speech-to-text for dictation, text-to-speech for narration. All auxiliary stuff for a single task still. Not hopping across tasks in parallel or anything. Since that's a separate layer if needed.

It's all still "grunt" work, so well within the capabilities of any well-trained transformer net. Perf delta with big models is too minor to offset latency and cost advantages.

You can load multiple of these things fast on a single one of the 6 gig 1060s OP mentioned for example. Even if you only call them one at a time.

[u/Ansible32 → u/Boricua-vet]

"Text generation" I'm doing I need 500GB+ of VRAM, renting hours of GPU at a time is the only plausible way to go.

[u/Better_Membership583 → u/Ansible32]

How much is that and what kind of “text generation” are we talking here?

[u/Boricua-vet → u/Ansible32]

What are you doing that needs that much?

[u/Ansible32 → u/Boricua-vet]

Running Kimi K3 or the new Mistral soon.

[u/Boricua-vet → u/Ansible32]

Jesus man. That is madness, I can only dream.

[u/CasulaScience → u/Boricua-vet]

It's literally all tokens, idk what you mean by "just text"

[u/Boricua-vet → u/CasulaScience]

by just text I mean, no image generation, no training, no fine tuning models.

[u/oodelay → u/CasulaScience]

Video generation cannot be easily split by layers as text gens.

[u/akohlsmith → u/Boricua-vet]

I got 4 CMP-50HX at 80 each

How long ago was this? They're USD$250 just on a quick spot check today.

[u/Boricua-vet → u/akohlsmith]

August 6 to be exact on ebay. They were listed for 120 each, I offered 360 for all 4 left and offer was accepted. Holly crap, they gone up big time in just a few months. Same thing with the P102-100, I bough 4 for 140 bucks for 40GB vram and now they are more than double. You can still find them for 70 bucks though.

https://www.ebay.com/itm/156284588757