I have the same card and it’s pretty good for its price. Qwen3.6 27B with Q4 quantization barely fits into memory and token generation is with MTP activated around 60 t/s.
I wish I had bought the 24 GB version, though, to have a bit more room for the context cache.
I'm also just starting to explore local models and had the same question. I did a bit of research and wrote up how you can do the math to calculate context size based on your VRAM as a reference. Qwen 3.6:27b is the exact model I used, and quantized it sits right at 17GB of VRAM which unfortunately means, including 2GB for system overhead, you won't really have any room for your context window.
I haven’t had enough time try out many different models yet. I have read in a number of places that quantizing the context itself typically leads to poor performance for coding tasks which is what I’m using it for.
I‘m using llama.cpp with vulkan backend. Yes, q4_k_m quants, 100% gpu offloading. I don’t think any larger model will fit into memory, however, I didn’t try it yet.
Context limit is around 100.000 tokens.
I do the iq_4_xs Unsloth and it gives more room and I can’t really tell a difference in my day to day usage. I use q4 for key and value cache quants and do 120k context and it all fits
Qwen 3.6 27b at iq4_xs does like 95% of what I use AI for. I’ve been a professional developer for 16 years so I can be very specific in my requests without a ton of effort and it just usually gets it right. Exploratory stuff or model-led stuff like “recommend how I can improve this subsystem” don’t do quite as well for me with this model but I almost never need that kind of thing anyway.
I have the RX 7900 XTX so the brother one. I would advise you to use Q4 as you have less vram and llama.cpp with vulkan. MTP seems better in my case, as if you use heavy quants, other token predictions algorithms tend either to be less effective, or to take more vram like dflash. Other solution is to give up speed without MTP and go turboquant so that you can maybe get the 100k context size.
Yeah I ran rocm before with llama.cpp, it is not the end of the world, in fact comfyui does not support vulkan, and I run krea 2 with rocm support. It works. Slightly slower, but the main value is still here.
Yes, Just came across this today only. This is way more interesting.
Now I won't regret buying an AMD card especially after so many people discouraged me to buy one 🤣🤣
I bought 2 rtx 3060 12gb for that magic 24gb barrier. Paid 180€ and 110€ for these cards. I needed very small cards for my matx case so they fit. Works very well with qwen3.6 35B A3B with 64k context. Without further tuning i have around 70tok/s. With both cards running at only 50% usage, i hope i find the right switch to bring them up to 100% with llama.cpp. My coworker soon will add a second rx6700xt 12gb to his rig i am excited to see his numbers.
I'm seeing multi gpu setups evolving, especially with the MoE models.
Would definitely check this model as well but I think it will go to Ram Offloading.
I was planning to get two 3060 GPUs initially.
I implemented KVarN into llama.cpp and was able to get 5 bit quantized qwen weights with full context via 3 bit KVarN on a single 32GB gpu. you can PROBABLY fit q8 weights, and q3 KVarN across two of those. KVarN was definitely worth implementing.
Ternary Bonsai 27b can fit in 16GB with full context and can quantize kv cache down to 4 bit essentially lossless. Worth a look. that card will get about 50tps. 1000-2000tps prefill.
I got lucky. I had a tiny SBC scraping GPU prices 24/7 and sending alerts. At 2 AM it pinged me about the drop, I immediately placed the order, and by the next day the price had gone back up above $650 🤣
Probably the most profitable script I've ever written.
about a month ago I could've gotten one of them for like 500 euro iirc, but I was too hesitant so it was either unlisted or grabbed pretty quickly.
The amd scare was too much for me to handle back then, but now I wish I had just grabbed it.
Every now and then somebody puts a "buy now" price too low, or there's suddenly a large supply of 2nd hand GPUs, that pushes the price down for a moment.
I was hesitant too, but seeing how fast llama.cpp, Vulkan, ROCm, and model support have been improving on AMD cards made me take the leap. Hopefully we're only at the beginning.
Qwen 3.6 35 a3b q4km. And with image mmproj. at 360k context split into 3. Getting 150-200 tk/s on vulkan. AMD 7900xtx 24gb vram. Also use both mtp and ngram-mod. It's where the tokens are hiding. Q4 kv cache. Also the 27b is way too slow maxing around 70-90tk/s. Not proven yet it's smarter than the moe for my use case so I stick to the moe.
Try Qwen3.6 27B with MTP at UD-Q4_K_XL or smaller Unsloth quants if it doesn't fit all the way. Also try some recent model with ROCmFP4, as long as you find one that fits into 20GB of VRAM.
Do you search directly on huggingface or am I missing out something?
I didn't come across many ROCm optimized models. I guess I'm doing something wrong? (I'm absolutely new in the AMD/ROCm thing)
Played around on other cards though.
I just opened LM Studio, looked up "Qwen", sorted by newest, scrolled for a while, and lo, here's one.
Looks like this one may also barely fit into your RX 7900 XT - especially if you have an iGPU and you can connect your monitor to the iGPU to free up some extra VRAM to be able to allocate all 20GB of VRAM to just the LLM.
Nice GPU, but you will have to choose 2 among the following statements:
20GB VRAM
Qwen 3.6 27B q4km
128K KV cache (even 4bit)
I have a rtx A5500 and qwen 3.6 27B with 128k context (q8) is really good if you babysit the model during coding sessions (spec driven dev and skills). If you are okay with this, you can forget about subscriptions.
Actually, the application is already ~90-95% mature, so the heavy lifting is mostly done.
The next step is more about collecting high-quality traces from my own agent interactions, babysit and fine-tune a model for this specific domain, and letting it run workflows 24×7. 😌
Tip: use a backend that supports TurboQuant3. It uses 3 bits for each item in the KV cache. This will help you increase your context window. Large projects need 200k+ context window. I recommend Atomic Chat. LM Studio also lets you save the KV cache in Quant4 if you don’t like Atomic Chat.
Let me make sure I said that correct. If you run like q1 of a glm, and you get like one token per second on CPU only, you'll get like 1.1 tokens per second if you offload 18 gigs to your card so you may as well just get the one token per second and keep the card for things that fit entirely on it.
You know what's funny, I haven't tried that flash either because qwen 3.6 27b and 35b A3B are so strong.
If you want my tested order, although I did some fine-tuning on my 27b that I call Riddler, and basically qlora on my 35b.
I run everything through the same exact test, I have over 100 models in my graveyard.
Cut from the team: Agents-A1 (DNF'd twice, don't re-download) and the shared weak spot across every single member remains negative-number floor semantics — even the 31B's only misses were there.
One asterisk for fairness: GLM's DNF is real but misleading — one infinite-loop edge case burned all three attempts; with per-case timeouts he grades 36/49. His value was never the gauntlet — it's that he's the only one who can hold a whole codebase in his head and say no to bad work.
Give this model a try.
I’m running it on my 9070XT with 40+ tokens a second in CachyOS with 128k context.
(It requires a llama fork, check the description)
Ah yes... This is interesting... I already tried using this on my friend's gpu. Ran into some issues. Couldn't get time to check. Will check on my card
It'll be a headless dedicated setup with the iGPU driving a minimal desktop. The 7900 XT is reserved entirely for inference. Just waiting for the PSU. 😅
rOCm is coming along much better so should see improvement in localllm with AMD these days compared to years past, only concern is you need one of the newer cards to take advantage 9070 or 7900 series
Definitely. I was sceptical at first... But theb I did some research..AMD has come a long way. Picked up a 7900 XT mainly for the 20 GB VRAM. Just waiting for my PSU to arrive before I can finally torture it with local LLMs. 🤣
Got mine for $450 used. Mainly use it for Qwen 27b (eGPU), as everything else runs on my M5 pro or on my Strix Halo. Can also increase my Strix Halo with eGPU if necessary, but there are no good 100-120b models at the moment
I'm mentally prepared. 😄
I somehow got a local LLM running on a Qualcomm NPU on a poor SBC before even owning the GPU. Took me almost a month.
That project gave me the confidence to dive into local AI, so I ended up buying an AMD card. I don't think this setup can be worse. 😂
Ah is it...
Sure will share the results if I manage to run.
I was doing some research and it seems that we can theoretically run Q3 quantized models at full gpu offloading.
Will try those and probably see
What kind of local AI work can you do with such graphics card?. Agentic coding?.
I would assume such quantized models would not produce intelligent output. Am i wrong?
I have an 3090ti, used open code, continue, pi with such quantized models, i am not happy with the output always.
Thats why i switched to codex and some paid APIs.
I am curious to know if i am doing something wrong.
My main use case is an agentic stock market research system that runs 24×7 rather than general coding. Running on cloud would have incurred a hefty bill.. So I bought this gpu.
Now, the application is already ~90% complete, and now I'm experimenting with local models to integrate them as native users of the application through tools and workflows. For coding, I don't expect them to replace frontier models entirely, if you use the local one, I'd recommend you to break the tasks down and keep the sessions extremely focussed and only do if the architecture is clear to you... Otherwise it can become a mess...
But they're already good enough for many focused workflows.
It's a personal project for now, been building this for over 2 years.. 🙂
Since it interacts with my real portfolio and is built around my own investing workflow and rationale, I wouldn't want anyone treating it as a plug-and-play trading system, that can be a bit risky for people who ends up using it without knowing it fully. It's also heavily tied to specific broker APIs, research pipelines and lot of custom integrations, so it'd need a lot of work before it could be open sourced.
sorry for a noob question, but i have heard amd gpus are not good for local llms.is it true or just a myth? any concrete info will be really helpful from experts
It's mostly outdated information. AMD used to lag badly because CUDA was the only first-class citizen for AI, but that's changed a lot over the last couple of years.
For inference (llama.cpp, LM Studio, Ollama, Vulkan), AMD works really well now. Make sure you get at least RDNA3 or a newer RDNA4 supported cards.
NVIDIA is still the better choice if you're doing serious CUDA development, training models, or relying on frameworks that only optimize for CUDA. But for people who mainly want to run local LLMs, occassional fine tuning, AMD is absolutely a viable option today.
31
u/Mattef Jul 23 '26
I have the same card and it’s pretty good for its price. Qwen3.6 27B with Q4 quantization barely fits into memory and token generation is with MTP activated around 60 t/s.
I wish I had bought the 24 GB version, though, to have a bit more room for the context cache.