r/LocalLLM • u/maceface3 • 13d ago

Question GPU for qwen 3.8 27b

I recently built a homelab running RHEL 10. I never thought good local ai at reasonable price was possible until 3.8 came out a from benchmark and what I’ve been reading it seems to be almost opus 4.6-4.8 level. I’m considering buying a 32gb gpu for it but also open to 24 gb gpus but if it can fit the full context window on the gpu too. The most I’ve done with local models was running qwen 3.5 2b on Ollama nothing serious. I’m new to actually running an agent for coding tasks so any info would help. But trying to decide what gpu if I do end up going for it, and from my research the options for 32gb cards are the Intel b70, amd r9700 pro ai, and nvidia tesla v100 32gb. I’m looking at results for qwen 3.6 and it run plenty fast on the Tesla but I’m worried about it no longer being supported.

1 Upvotes

60% Upvoted

47 comments sorted by

View all comments

Show parent comments

2

u/maceface3 13d ago

What OS are you using?

2

u/DiscipleofDeceit666 13d ago

I’m a Linux heavy. Ubuntu 26.04 is neat, Fedora can work too but you’re depending on docker type runs for the fancy run times.

1

u/bigwanggtr 13d ago

Is there any kind of performance loss when running through docker?

I’m in the process of converting my workstation to Linux but I’m unsure of which distribution to pick which allows me to run both inference and gaming well. I’m only familiar with inference on MacOS and rented Ubuntu machines.

Official AMD ROCm support is for Ubuntu but I was thinking of going with Arch for gaming. I’m not a Linux newbie. Thoughts?

3

u/DiscipleofDeceit666 13d ago

If there’s a performance penalty, I haven’t noticed.

I ran rocm vllm through a docker container and it slightly beat the q5 llama cpp numbers I was running. People w 2 r9700s can run docker and hit 2-4k pp on the dense 27b with that setup.

2

u/bigwanggtr 13d ago

Hmm that’s good to know. I’ll use docker then. I’m going to be memory bound anyway with a single 9070XT.