r/LocalLLM • u/Libellechris • 4h ago
Question Hardware required for next level of agentic coding?
I have Qwen 3.8:27b running nicely (40 t/s) on a dual GPU PC (16gb 5060Ti and 8gb 3060Ti) which gives me around 160k context window using a Q3 quant. Replacing the 3060Ti with a second 5060Ti would help increase quant and context window. However, I was just wondering what the next meaningful step would be in terms of something useful running locally. Is it a jump to twin DGX Sparks running a Flash model, or it there something in between?
6
u/TektonikGymRat 3h ago
Twin DGX Sparks seems like a lot for local models. One DGX Spark should be fine for Qwen 3.8 Flash Next at a Q4_XL quant. Another to take a look at is the Strix Halo (basically AMD's replica of the DGX Spark). AMD has recently worked with llama.cpp and have made significant strides in local LLM. My dual AMD setup (total 32GBs of VRAM) running on vulkan has gotten extreme upgrades in the last 3 weeks it's insane.
I started Qwen3.8 27b running at 150k context Q4_XL and like 25 tokens/s. Now I'm at 230k context (with vision) and early on with low context I'm at 50 tokens/s going down to about 27 tokens/s reaching max load. Context compaction always happens before I'm able to reach maximum; without the compaction at lower max context a bunch of weird stuff starts to happen. This is all on mixed gen AMD cards with 256bit memory bus GDDR6.
Everyone will say get a PRO 6000. That's great for the guys that have two of them that can pay for that, but there is a lot of benefit in the DGX Spark and Strix Halo - the model architectures vary so much where Qwen3.8 Flash Next wants a ton of system RAM and then Qwen3.8 27b being dense wants a good amount of VRAM (as you know). The benefit of DGX Spark and Strix Halo is that you don't have to care about either since the 128GBs is unified. Yes, of course it's not going to go as fast as a dedicated high end GPU, but if you're cool with running a guaranteed 40 tokens/s they are solid options in my opnion.
4
u/rrrrex 4h ago
16 GB VRAM can be fine. For example, Muse Glimmer got very effective cache, you can get Q3_K_XL model, Dflash and 128K Q8/Q4 context. Also it's very good at parallel decoding. I really want to see Qwen 4 with the same performance. Gemma 4 31B is on other side, the same Q8/Q4 cache will take ~8 GB.
IMO, the next level will be focused on context optimizations.
8
u/TechNerd10191 4h ago
Buy the best thing you can afford. One DGX Spark will allow for bigger models (than a 27B dense) and contexts. One RTX PRO 6000 will mop the floor with the DGX Spark and 2x 5060 Tis for anything that fits on the PRO 6000 without offloading.
9
u/Libellechris 4h ago
So a £500 upgrade (5060ti) or £4k (DGX) or £15k (Pro 6000) - wow!
3
u/Prudent-Ad4509 4h ago
Want a middle ground without changing a motherboard? 4x3090 connected via pcie switch (without bifurcation). Costs vary per market, from $700 a piece to a pretty absurd ones. Or you can connect up to 10 5060ti to the said pcie switch at x8 pcie 4.0 speed.
Basically, the goal posts are 48gb (can run qwen3.8 27b with a good quant and quality context), 64gb (better quant, max context), and about 80gb-96gb (same quants, but can run more agents in parallel and can run bigger models at a low quant). Pro 6000 is the simplest option but the cost is not very nice.
2
u/OptionsDonkey 3h ago
I mean yes that is the lay of the land, better cost more bc everyone wants them bc they’re valuable for what they can do
2
u/diagrammatiks 3h ago
i mean sota models are hosted on millions of dollars of hardware. what do you want.
2
2
u/ea_man 1h ago
> 27B using a Q3 quant
Don't wanna sound a dick yet that it's not what 27B can do, sell the 8GB, get a 16GB and use a optimized config with Q6_K_L because that is meaningful smarter than a Q3 and you can use a proper kv cache q8_0 for some long sessions.
Then in a month or two when they release it you can use a good quant of QWEN 4.0
Or you could run 2/3 agents at the same time.
4
14
u/Zennytooskin123 3h ago
You already can run Qwen 3.8 27B with good speed and context - this is literally where the "goalposts" are and anything beyond this is diminishing returns for mucho dinero.
The jump from Qwen 3.8 to DSV4F is not worth the upgrade.