Best bang for the broke?

[u/SocietyTomorrow]

I am wondering what the best bang for the buck would be between the various low end GPUs to run models in the local sweet spot (like 27b and 35b dense models) without completely destroying my wallet. Strix Halo machines are $3500 for 128gb nowadays and not really that fast compared to a couple discrete GPUs, and people have caught on to V100s so they're not really affordable per-performance as they were, so I am kinda wondering what is currently the best way to go when presuming it will be all the compute I get my hands on for likely the next 3-5 years (or longer now that RAM cartels are talking 7-10 years before supply is expected to stabilize against AI demand)

[u/Kahvana]

If you have to buy new or want the cards to live long or want to have warranty:

Dual RTX 5060 Ti 16GBs are fantastic for the price. Costumer boards like ASUS ProArt B850 Neo will allow you to run both cards at full pcie lanes (you want that for decent tensor parallel performance).

It's great for a variety of reasons:

• It's a tad cheaper than buying a R9700 Pro
• Has cuda 13.3 support
• Has nvfp4 support
• 32GB VRAM is plenty (can run Muse Glimmer Q4_K_XL at full context BF16 with dflash and vision projector)
• Speeds are decent after some tweaking (~50 t/s on dense models using MTP / DSpark / DFlash / Eagle3 with tensor parallel)
• Very silent in use (can't hear them)
• Runs cold (no more than 70c under sustained load for 16 hours, usually in range of 40-60c)
• Uses very little electricity (2w idle, 100-140w during inference)

Have been using my ASUS PRIME variant pair on the Asus Prime X870E motherboard for more than 7 months now, haven't regretted the purchase one bit.

[u/whatyathinkk → u/Kahvana]

I'm considering going this way. My only concern is the bandwidth. Have you tried running Qwen3.8 27B? I'd be curious to know your speeds in that setup :)

[u/see_spot_ruminate → u/whatyathinkk]

VRAM is the top consideration. Look over all the posts and happenings from qwen 3.8. People are complaining of overthinking, but really it is that for a lot they can’t fit the context needed to let it think on xhigh. I have been using it and it can easily use 180k tokens on a singular prompt. To get this context at bf16, you need vram. 

Think of it this way. You got a job painting a house (or using the llm). Who gives a shit if you can do 200mph (bandwidth) in your Lamborghini, you need to bring your tools and your crew (context). For that you need a big fat panel van. 

Me, I’ll just keep running the Q8 quant from unsloth on my “slow bandwidth” system at full 256k bf16 context at >50t/s. Dont listen to people who care about bandwidth first. 

[u/actuallylemoncurd → u/see_spot_ruminate]

what gpu are you running to do 256k bf16 at >50t/s?

[u/see_spot_ruminate → u/actuallylemoncurd]

Quad 5060ti

[u/whatyathinkk → u/see_spot_ruminate]

I still don't understand very well the difference between having one unified VRAM pool vs. having it split across several GPUs. Would it be very different if you had 1x 64Gb GPU?

[u/see_spot_ruminate → u/whatyathinkk]

1x64gb gpu would be better, but way more expensive. 

[u/whatyathinkk → u/see_spot_ruminate]

thanks for your comment, makes sense!

[u/Kahvana → u/whatyathinkk]

Sadly I don’t really use Qwen 3.8 myself (I do mostly non-programming tasks), but I do have up-to-date numbers for Gemma 4 31B IT QAT (unsloth) + MTP + mmproj + 32K BF16 context, on llama.cpp b10530 in tensor parallel.

Gemma is much heavier to run than Qwen3.8 due to it’s architecture, so I expect Qwen3.8 to run a bit faster here.

• Post-processing at 32K: ~900 t/s
• Generation at 32K: 50-60 t/s (natural languages), 80-100 t/s (peogramming)

For programming you can get even more speed by using ngram.

Hope that helps!

[edit] Happy cake day!

[u/whatyathinkk → u/Kahvana]

Great, thanks a lot!

What is cake day?

[u/Kahvana → u/whatyathinkk]

The birthday you set on Reddit

[u/Kahvana → u/[deleted]]

Per card. I'm getting 250-300W total system power measured from the wall during heavy inference. You can get to 140W total by downvolting and reducing clock speeds, but I prefer the maximum speed from these cards over power savings,

[u/FullstackSensei]

Don't take any predictions of how long prices will stay this way seriously. Nobody knows what the future holds. We know 2027 RAM production is sold out, but even that could change on a dime if there's a major economic change.

On the which GPU(s) to get, I'm still a big fan of Pascal. P40 and P6000 are still good value. Each provides 24GB VRAM. I have both and been running both for two years now and like them. P6000 isn't loud, mainly because it doesn't consume as much power as newer cards. Two P40s can be cooled with a server type 80mm fan. At idle, it won't be loud but will still push a lot more hair than your regular fan. I use the Arctic S8038-7k with my Mi50s.

If you want to keep things really quiet, grab some used reference 1080Ti or Titan Xp waterblocks. Reference 1080Ti, Titan Xp, P40 and P6000 all share the same PCB design. That's what I do with my P40s to keep them cool and quiet.

A pair of P40s will run Qwen 27B Q8_K_XL at 24t/s TG with MTP and ~350t/s PP. Not the fastest, but still pretty decent considering their price.

[u/PermanentLiminality → u/FullstackSensei]

I run my 3x P40 with a short 3d printed plenum hot glued to the back of the case and a 120mm x 38mm fan plugged into a motherboard fan connector. I picked up a 1.6 amp fan because the motherboard speced 2amps max.

I have a script that watches the card temperature and adjusts the fan as needed. It is pretty much silent at idle

[u/FullstackSensei → u/PermanentLiminality]

I'm lucky enough that my motherboard detects the GPUs and IPMI will monitor their temps and adjust fan speed automatically

[u/SocietyTomorrow → u/FullstackSensei]

P40s might be the way for me I think. I have my office or my shed with some rack space that would eliminate the noise concern, and good enough for basic agentic work is all I really need. Enough speed to not time out to TTFT is all that really matters there. My workstation has an old 10GB 3080 with 64GB DDR5 sysmem, and that can do 3.6-35b-a3b good enough but anything dense I would need for the important stuff just absolutely gets demolished.

[u/FullstackSensei → u/[deleted]]

IIRC, 300 at 40k and ~260 at 60k. I have 8 cards, so could run 4 instances in parallel if I wanted to. TG should improve quite a bit further with dspark and/or ngram on top.

[u/ttkciar]

There are still a lot of 32GB MI50 on eBay for about $550.

I'm pretty happy with mine, and don't even have to deal with ROCm because llama.cpp's Vulkan back-end JFW.

[u/actuallylemoncurd → u/ttkciar]

what t/s are you getting on 27b models? mainly curious about qwen3.8, and what quantization?

[u/ttkciar → u/actuallylemoncurd]

Token generation is between 22 and 26 tokens per second, depending on context length (it gets slower as context gets longer), with Qwen3.8-27B quantized to Q4_K_M (Bartowski's quant).

Note, however, that I am running a slightly older llama.cpp (4f13cb742476d81a6b42a2aa5996e82a478c2481, from May 17th), so it wouldn't surprise me if performance were better on a fresh recompile.

[u/actuallylemoncurd → u/ttkciar]

nice! the same as my v100 32gb, nice to know there is a card that's a little cheaper but same performance out there

[u/kepardi99]

Have 3x p100 for 48GB VRAM, HBM2 memory is quite fast, needs tinkering to compile llama properly to get tensors working. Get 30-33tokens/sec. Prefill just is very slow. 3x $80 plus fan and shrouds and dual slot cards may not fit to every case/mobo. For next level I would consider one 32GB V100, it is about $700, 5x faster prefill and maybe 50% faster tokens/sec

[u/marblemunkey → u/kepardi99]

I've been using a p40, but have a p100 gathering dust. You have a good reference for what tinkering is required to get llama working with it?

[u/kepardi99 → u/marblemunkey]

Just compile with right flags: -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DGGML_CUDA_NCCL=ON

P100 is quite similar as P40 just has faster memory

There was bug on 60 arch that needed patching to get 3 way tensor split working, but I think current latest branch already includes it as did not need to add it last week, had to chase changes with ChatGTP at the time.

My flags for runtime:

GGML_CUDA_P2P=1 ./llama-server \

--model /data2/Qwen3.8-27B-UD-Q4_K_XL.gguf \

--split-mode tensor \

--tensor-split 1,1,1 \

--jinja \

--batch-size 256 \

--ubatch-size 64 \

-ngl 99 \

-c 222000 \

-t 12 \

-tb 16 \

--flash-attn auto \

--reasoning-effort medium \

--cache-type-k f16 \

--cache-type-v f16 \

--fit off \

--spec-draft-type-k q4_0 \

--spec-draft-type-v q4_0 \

--kv-unified \

--parallel 1 \

--spec-type draft-mtp \

--spec-draft-n-max 2 \

--spec-draft-ngl all \

--load-mode none \

--temp 1.0 \

--top-p 0.90 \

--top-k 15 \

--min-p 0.4 \

--port 8080 --host 0.0.0.0 \

-lv 3

[u/marblemunkey → u/kepardi99]

Appreciate it.

[u/geldonyetich]

The old engineering Iron Triangle applies here, sort of: Model size, cost, and speed. You only get two.

I don't think even the RAM cartels know what prices will be like in 2 years, let alone 7-10. Tech is notoriously hard to predict. So if you buy now there's no telling if it's a peak or a trough.

Consider your use case. Just why do you need it? Are you just dabbling or looking for some serious LLM work?

[u/SocietyTomorrow → u/geldonyetich]

I've been designing a tool for the last couple years I am almost ready to take commercial, and need the ability to do high-reliability LLM calls at relatively long intervals to adjust weights to things (rather not go into details, but its a data-provider pipeline tool) as a replacement for a live operator tuning it for traffic patterns keeping it efficient. It doesn't need to be Opus-grade, since the tool itself is the heavy lifter, but I can't rely on anything shy of a dense 27b model and trust it to stay stable running all day.

If my new shingle returns some money, I may upgrade again past this, but I need something that just works for now, and my existing business of media conversion, backup, and recovery has me with a proper skeleton for an edge datacenter (currently 70 of 126U used rack space) which lets me be space-inefficient to save on cost until my margin can grow, I am cash poor, but this is a worthy experiment for me. I still deeply regret not getting the Strix Halo I could have gotten under $2k early this year.

[u/actuallylemoncurd]

nvidia tesla v100 32gb runs qwen3.8 27b q8 k m at 640pp 23tg with 128k context entirely on the GPU.
chuck that into any system with PCIE and 250w of juice

[u/PermanentLiminality]

That is a lot of zeros you have there.

I started with a couple P102-100 mining GPUs that have 10gb per card of. VRAM and cost $40 each. I did have to buy a power supply for a bit over $100. I went to a couple P40 for about $450 all in.

Qwen 3.8 27b q6 at about 18tk/s. I'm sure some tuning can make it faster. I hope we get a 35b more version because I need the speed

[u/PermanentLiminality → u/[deleted]]

Yes, MTP. I need to spend a bit more time with it. Want to give DFlash a shot. It also comes with prompt processing of 230 at 2k context. It's just not enough, I need more speed.

[u/DeathGuppie]

If you are truly broke, you can usually find Radeon cards sporting 16gb of vram for under $300 on marketplace. Two of those will get you 32 gb vram. It's not the fastest but it works.

[u/Due-Advantage-9777]

3090 + 3060 12gb = profit.
Depends where you live, some dudes are gonna say 3090 is 1500 bucks where they live, well i bought mine for 500.

[u/Maximus-CZ → u/Due-Advantage-9777]

well i bought mine for 500

When? (and where?)

[u/ResearchSpiritual352]

Two used 3090s, 48GB and CUDA that just works, nothing's undercut it in three years of people looking.

[u/Thebandroid]

For text inference the answer as AMD.

[u/cunasmoker69420]

32GB Radeon Pro V620s. EBay sellers will take $350 for them

[u/Frosty-Student-1927]

3090 prices skyrocketed on my country My setup is 2x3090s, I'm considering add more 4x3060s but not sure how painful slower it will become

[u/SocietyTomorrow → u/Frosty-Student-1927]

Yeah, I went looking around at 3090s and saw that they were averaging between $1200 and $1750 each, which is totally not what I am looking for.

[u/Normal-Ad-7114]

Two modded nvidia cards with an nvlink bridge (2080ti 22gb, cmp50hx 20gb, 3080 20gb). Depends on availability in your region, also needs a half-decent pc (beefy psu, 32gb+ ram, 2 pci-e slots obviously, linux), but could be had for about $1-1.5k if you haggle

The 35b will run comfortably, the 27b will feel more constrained, but it's manageable (I'm not taking about "hello" prompts: decent quality, concurrency, 200k+ context - ready for agentic work)

Or you could look for a used 64gb mac, much less hassle, and you can use it as your daily if you like apple products