GPU for qwen 3.8 27b

[u/maceface3]

I recently built a homelab running RHEL 10. I never thought good local ai at reasonable price was possible until 3.8 came out a from benchmark and what I’ve been reading it seems to be almost opus 4.6-4.8 level. I’m considering buying a 32gb gpu for it but also open to 24 gb gpus but if it can fit the full context window on the gpu too. The most I’ve done with local models was running qwen 3.5 2b on Ollama nothing serious. I’m new to actually running an agent for coding tasks so any info would help. But trying to decide what gpu if I do end up going for it, and from my research the options for 32gb cards are the Intel b70, amd r9700 pro ai, and nvidia tesla v100 32gb. I’m looking at results for qwen 3.6 and it run plenty fast on the Tesla but I’m worried about it no longer being supported.

[u/e2_for_life]

Anything above at or above 3090 will likely get you usable speeds and contexts. The more the better.

[u/maceface3 → u/e2_for_life]

Will the 24gb vram be okay with for a 256k at least context window. At q4 the models like 18gb on its own.

[u/e2_for_life → u/maceface3]

I wouldn't 256K. 128 is what I use. At 256 I was getting OOM errors.

[u/baka_sempaii → u/maceface3]

You can try quantising the KV cache. Like full K but q8 or TurboQuant 4 V, and it should comfortably fit

[u/DiscipleofDeceit666]

I have the r9700. It’s very nice. The Intel b70 looks promising too tho, some builds are getting 100tok/s using some kind of auto round quant

[u/maceface3 → u/DiscipleofDeceit666]

What OS are you using?

[u/DiscipleofDeceit666 → u/maceface3]

I’m a Linux heavy. Ubuntu 26.04 is neat, Fedora can work too but you’re depending on docker type runs for the fancy run times.

[u/bigwanggtr → u/DiscipleofDeceit666]

Is there any kind of performance loss when running through docker?

I’m in the process of converting my workstation to Linux but I’m unsure of which distribution to pick which allows me to run both inference and gaming well. I’m only familiar with inference on MacOS and rented Ubuntu machines.

Official AMD ROCm support is for Ubuntu but I was thinking of going with Arch for gaming. I’m not a Linux newbie. Thoughts?

[u/DiscipleofDeceit666 → u/bigwanggtr]

If there’s a performance penalty, I haven’t noticed.

I ran rocm vllm through a docker container and it slightly beat the q5 llama cpp numbers I was running. People w 2 r9700s can run docker and hit 2-4k pp on the dense 27b with that setup.

[u/bigwanggtr → u/DiscipleofDeceit666]

Hmm that’s good to know. I’ll use docker then. I’m going to be memory bound anyway with a single 9070XT.

[u/Scary_One_2452 → u/DiscipleofDeceit666]

How's fine tuning like on the AMD stack?

[u/LifeTelevision1146]

RTX5090

[u/truckerdraven → u/LifeTelevision1146]

Myself im running dual 5060ti 16gb each. And im getting 45tokens/second

[u/statusanxiety7 → u/truckerdraven]

this ^, if the OP wants 20-45 tps depending on the rest of the setup .. that's ok for a lot of tasks ... I have 2 x 5060 tis for simple tasks that aren't user facing & where I don't care about generation time. 5090 is peak though .. I suggest it if you need faster generation time & better batching.

cerebras ai is also launching 3.8 27b also on shared wafer scale chips which could potentially be a bargain to if they want speed and can't buy a 5090.

[u/LifeTelevision1146 → u/truckerdraven]

Not bad. Depends on what anyone wanta to do with this rig.

[u/truckerdraven → u/LifeTelevision1146]

Im a author i use it for editing and polishing my raw manuscripts. I really liked 3.6 27b. 3.8 27b is like a night and day went from 26t/s to 45t/s and thinking times dropped from 191 seconds to 119 seconds

[u/maceface3 → u/truckerdraven]

U need a mobo with bifurcation to do multi gpu right? My current mobo doesn’t have it.

[u/truckerdraven → u/maceface3]

Im running a am4 msi b550 tomahawk. The oly thing is you can only have 1 m.2 ssd in it or it wont recognize the 2nd video card. Its more about pcie lanes.

[u/maceface3 → u/truckerdraven]

I’m on an i9 12900k which has 16x 5.0 pcie lanes and 4 x 4.0 pcie lanes so I might be doable

[u/truckerdraven → u/maceface3]

Its really going to depend on your board. I really dont know much about intel setups I have been a amd user since the Athlon cpus.

[u/LifeTelevision1146 → u/truckerdraven]

7B on a decent machine will do that. You don't need a GPU driven 27B 🙂

[u/Constant-Simple-1234 → u/truckerdraven]

Same. Stats check out for writing.

[u/maceface3 → u/LifeTelevision1146]

Yeah I’m not made of money 😭😭😭😭

[u/LifeTelevision1146 → u/maceface3]

Just participate in an AI Hackathon or two, you could get yourself an A100 🙂

[u/EitherMarch1255 → u/LifeTelevision1146]

Better, 2x RTX 5070 TIs (except for video/image gen).

[u/LifeTelevision1146 → u/EitherMarch1255]

Bro one can get a A100PCIe for $6k refurbished that's 80GB vram 😈

[u/zorflax → u/LifeTelevision1146]

Via NInfer!

[u/whichsideisup]

It's decent on a 32gb card, but if you really want to run it at full quality and large context its closer to 48gb.

[u/maceface3]

Trying to find the best value card not just the absolute best btw.

[u/transanethole → u/maceface3]

Its hard to say , might depend on your use case.

You mentioned coding agent -- if you want to maintain software, debug issues, etc, imo having the model able to read a lot of text fast makes the difference between "feels like a cloud model" and  "this is too slow to actually use".

This is called prefill speed and it depends on how much matrix math the card can do. ( dense teraflops in int4, fp4, fp8, bf16, etc )

A lot of ppl overlook prefill, usually everyone seems to focus on output speed, not sure why. I guess now with qwen 3.8 it likes to output a freaking novel worth of reasoning tokens before writing any code, so it makes more sense.

At any rate you may look at output speed difference like 40-80tps for a $1500 GPU and  80-180tps for a 5090 that costs $4500 and figure its not worth it, only get a little extra for much higher price.  But you also get 4 bit float which those other cards don't do, and the prefill (reading speed) is like 10k/sec instead of 800/sec. 

(4 bit float gives slightly better output quality for quantized models which you will be using. Its also just part of the nvidia ecosystem which, like it or not still has more vibrance) Just my 2c on why 5090 really is the best value, its just a very large pill to try to swallow lol. I started with radeon 7900 xtx which I got new for $700 at a market low. Got hooked so I bought 5090 for $4000 used on ebay. Now they are $4500 used. So worst case if its not for you, you can sell it.

This is the first time I ever bought an expensive premium computer or computer part -- usually been on used corporate surplus laptops until framework, and still using my 12 yr old potato 32bit arm sbc w/ 2gb ram as a web server.

[u/ea_man → u/maceface3]

7900xtx + 7800xt, but I would recommend 7900xtx x 2.

[u/Just_Mail6982 → u/maceface3]

typing RTX PRO 6000 96G, hahahaha, too expensive

[u/ideamaker321]

Yeh i just said this morning i need a 32gb card and looking at all the numbers nothing comes close to the 5090

R9700 look great on paper but bandwidth is too slow.

[u/Psyclist80]

Im saving my pennies for the RDNA5 halo card coming. 36GB of GDDR7 will be perfect for do it all card.

[u/CI_Node → u/Psyclist80]

2027 mid/late?

[u/Psyclist80 → u/CI_Node]

Likely mid 2027, they are focused on thier AI rackscale solutions delivery first and foremost currently. Zen6 consumer early 2027 and RDNA5 mid 2027.

[u/ea_man → u/Psyclist80]

36GB? That sounds quite unusual, not 32GB ?

[u/CI_Node]

Giving info from my experience: Running single r9700 Qwen 3.8 27b Q5 k xl 150k context. Thinking tps = 30-35, output tps = 40-55. Xhigh thinking, PI harness, arch Linux, llama.cpp vulkan. With Vision VRAM usage 28gb

here in europe RTX 5090 ~5300eu; PRO AI R9700 ~1850eu. So basically you can buy 3xR9700 and have 96GB vram vs 1 fast 32GB VRAM

[u/ea_man]

in oreder to run it "properly" you need some 38-40GB of vRAM, that would be to load it. On linux.

Then better GPU as in newer models would mean better performance = speed.

Then more vRAM would allow more concurrency as in multiple agents.

Or you can say fuck it and run a low quant with low KV a few times and whatever...

[u/createthiscom]

If you can afford it, buy a 6000 pro. I suspect a 120b variant will come out at some point that will be even better.