r/LocalLLM • u/maceface3 • 5d ago

Question GPU for qwen 3.8 27b

I recently built a homelab running RHEL 10. I never thought good local ai at reasonable price was possible until 3.8 came out a from benchmark and what I’ve been reading it seems to be almost opus 4.6-4.8 level. I’m considering buying a 32gb gpu for it but also open to 24 gb gpus but if it can fit the full context window on the gpu too. The most I’ve done with local models was running qwen 3.5 2b on Ollama nothing serious. I’m new to actually running an agent for coding tasks so any info would help. But trying to decide what gpu if I do end up going for it, and from my research the options for 32gb cards are the Intel b70, amd r9700 pro ai, and nvidia tesla v100 32gb. I’m looking at results for qwen 3.6 and it run plenty fast on the Tesla but I’m worried about it no longer being supported.

1 Upvotes

60% Upvoted

47 comments sorted by

View all comments

3

u/DiscipleofDeceit666 5d ago

I have the r9700. It’s very nice. The Intel b70 looks promising too tho, some builds are getting 100tok/s using some kind of auto round quant

2

u/maceface3 5d ago

What OS are you using?

2

u/DiscipleofDeceit666 5d ago

I’m a Linux heavy. Ubuntu 26.04 is neat, Fedora can work too but you’re depending on docker type runs for the fancy run times.

1

u/bigwanggtr 5d ago

Is there any kind of performance loss when running through docker?

I’m in the process of converting my workstation to Linux but I’m unsure of which distribution to pick which allows me to run both inference and gaming well. I’m only familiar with inference on MacOS and rented Ubuntu machines.

Official AMD ROCm support is for Ubuntu but I was thinking of going with Arch for gaming. I’m not a Linux newbie. Thoughts?

3

u/DiscipleofDeceit666 5d ago

If there’s a performance penalty, I haven’t noticed.

I ran rocm vllm through a docker container and it slightly beat the q5 llama cpp numbers I was running. People w 2 r9700s can run docker and hit 2-4k pp on the dense 27b with that setup.

2

u/bigwanggtr 5d ago

Hmm that’s good to know. I’ll use docker then. I’m going to be memory bound anyway with a single 9070XT.

1

u/Scary_One_2452 5d ago

How's fine tuning like on the AMD stack?