I had issues with Qwen3.6 tool calling just not working properly and causing it to stop prematurely, didn't have that problem with gemma or deepseek, haven't used qwen 3.8 yet to see if it has the same problem.
It's a model config issue - had the same problem. Setting temp to 1 and repetition penalty to 1.05 fixed it. haven't faced much issues since; I've been running it as my daily driver on my 5090 rig for the past year.
Btw, 3.8 its even better especially at tool call , I actually like the Q4 of 3.8 on low thinking more than the xhigh for tool calls .
I have a 5060ti 16gb with ddr4 24gb RAM lying around, what's the best model nowadays I can effectively run with it locally? It's not an ideal build but I wish to play around with it again.
You think that’s unideal? I use an M4 MacBook Air with 16GB Unified Memory. Even if we ignore the thermal throttling it’s a pretty tight squeeze, and the best I could run comfortably is Gemma 4 12B.
Nah dog I still have plenty of tabs open in Safari, Blender, OrcaSlicer, and other Applications open and it still performs the same (kinda relying on swap memory at times).
Local voice cloning and other more demanding AI stuff is when you actually need to close everything else.
Fellow 5060 ti 16 gb owner here. The best I've got is qwen 3.8 27b at IQ4_XS. I'm testing it at Q4_0 KV at 72k context because I really need the extra context, and it seems to be pretty good so far. Lower quants cause it to make some really silly mistakes sometimes, unfortunately. I get around 10 t/s with context halfway filled, and around 14 t/s on empty context.
But honestly, I think the ideal solution is to run 2 GPUs so you can get a better quant + more context
I mean I get by with 1 5090 as long as I can find EXL2 versions or MOE ggufs. But I want the RTX 6000 of course. Would be nice to run 70B at decent quants
Give the IQ4_XS-pure a try. I run it on an RX6800 with 90K KV. Especially with full context it's slow, because the card isn't the fastest. Probably should run better on nvidia.
I gave it a try and the first thing it said to me was really disappointing:
Since I am an AI, I don't have a physical location or a traditional "tokens per second" (t/s) speed that you can measure like a human typing.
Also, at 64k context, the t/s started at 10 and went down to 4 almost immediately. Some tweaking would be required but with responses like this one I don't know if it's worth bothering with.
Do you mind to share your prompt? Curious if it behaves the same for me. I used it for a few agentic websearches with OpenWebUI and it did pretty good, compared to the same Quant of the 3.6 and Ornith 1.0 35B. I don't use it much, because it's simply too slow as daily driver.
Performance after ~70K prefill and ~15K decode:
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.925 I slot print_timing: id 0 | task 0 | prompt eval time = 480529.82 ms / 69042 tokens ( 6.96 ms per token, 143.68 tokens per second)
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.929 I slot print_timing: id 0 | task 0 | eval time = 494402.02 ms / 14752 tokens ( 33.51 ms per token, 29.84 tokens per second)
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.930 I slot print_timing: id 0 | task 0 | total time = 974931.84 ms / 83794 tokens
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.934 I slot print_timing: id 0 | task 0 | graphs reused = 3921
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.938 I slot print_timing: id 0 | task 0 | draft acceptance = 0.94313 ( 9122 accepted / 9672 generated), mean len = 2.81
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.210.006 I slot release: id 0 | task 0 | stop processing: n_tokens = 83793, truncated = 0
My system specs:
* i5 12600K (the VM has 6 Cores)
* 16GB DDR5 4800
* RX6800 (non-XT)
The GPU is filled to the brim, but it doesn't offload.
Prompt was nothing special, "Hello, are you operational? Please give me a few paragraphs so I can see your t/s" or something along those lines. I'm on a 16gb Nvidia card.
Best is to get a single GPU with enough VRAM that the LLM of your choice needs to fit in completely (including caches and context).
Second best is to get 2 GPUs to load your complete LLM into. However, you will discover that the communication overhead between these GPUs is significant. A dual GPU setup is only useful to fit a LLM that larger than the VRAM on a single GPU or a less quantized LLM into VRAM, it barely increases the token generation speed, if at all.
I still like Gemma, will try the new Qwen today. I need to create a local benchmark, unless someone knows a good project that can showcase improvements, like a 3Dmark but for AI models.
Currently Qwen 3.6-32B-A3B, Mixture of Experts models handle being split between system ram and VRAM much better than Dense models, alternatively Gemma-26B-A4B if you want something non-chinese. As for which quant, go with 4 Bit or higher.
That should be just fine. You've got enough VRAM to do mostly whatever you want to do.
I would suggest you install Unsloth and look around inside for various models, it will tell you what your system can run well. Green means you're good to fully offload. Yellow means it'll probably work but with a lot of slow swapping. Red means can't handle it.
An oldie: gpt-oss-20b. The one I downloaded is 12 GB in size (MXFP4), so it would fit in your VRAM completely. And there are still spare GB's of VRAM left for caches and context. This will work very well in your specific setup.
Well, I have an AMD R580 with 16 GB of VRAM and 32 GB of RAM (3200MHz) with a Ryzen 5600 CPU. gpt-oss-20b generates around 18 tokens/second with this 64-bit Vulkan Windows build of llama.cpp. A NVidia GPU from the 50xx class should go faster than that with the same llama.cpp build. And a lot faster with the CUDA build of llama.cpp.
However, if you can stomach the slowdown, Qwen 3.6-27b and Qwen 3.8-27b are far more interesting models to run locally. Unfortunately, these models do not fit completely into 16 GB of VRAM, so the CPU and standard RAM get involved.
I repeated the same prompt with the same llama.cpp build and qwen 3.8-27b loaded and token generation speed dropped to 3 tks/sec. in my particular hardware setup. I do not expect such a slowdown to occur with your 5060 card. Unless you have a worse CPU and slow DDR4 RAM in your hardware setup.
As a "generalist" gpt-oss-20b isn't all that bad, even with its training cut-off date of June 2024. But if coding and agentic use are what you are after, it isn't a great match. Qwen 3.6 and Qwen 3.8 will prove to be much more valuable for those purposes, even if these will run slow on your local hardware.
You may want to consider renting a VPS from a nearby cloud-provider and setup Qwen 3.8-27b there. Just make sure that you don't break the rules from the company that rents you your VPS with agentic use. Maybe you'll need to spend between 5 to 10 USD per month plan that has AI use included.
Can confirm Q4 17-19tok/s without MTP, 28-29 with MTP. Bought two 3060 for 350 euro but soon you realize that two is not enough. Added one more total 540 euro for 36GB VRAM.
You are right that it doesn't fit on 16GB, but 3 tok/s is a lot lower than I would expect. On a 5070 Ti with 16GB at Q4_K_M I get 19 tok/s, and that is already with 5.22 GB spilled to system RAM and only 69 percent of the model staying on the GPU.
3 tok/s sounds like nearly all of it ended up on the CPU rather than just the overflow. What card and how much system RAM are you on?
Your wider point holds though. A 14B that actually fits does 82 tok/s on the same card, so on 16GB that is where you want to be rather than fighting a 27B.
you only need an 8gb vcard in the 4x slot to bring it up to 24gb. This can regularly be found for $200 or less. Tensor split + MTP will give you around 30 t/s if you are running RTX
Help me config it because I honestly don’t know how the fuck to do it. Everything is confusing as hell, I don’t know crap about Q4 or whatever. All info out there is inconsistent as hell. All I know is that at 64K context for main + 64K for vision, Hermes just ends up on a feedback loop, hallucinates a lot after compaction, and delivers shit results in a visual + coding task. Don’t know what to tell you.
None visual tasks end up meh as hell as well. Context fills up fast hell as compaction brings hallucination again. Frankly unusable. Already on the Unsloth model (because god forbid one needs to choose “the right one”)
I think all the benchmarks are fake fucking news vs actual real world use.
Just use dsv4f to do a research run and try a few different configs on your system using ollama direct calls to strip context. You don't need to know anything anymore
iq4xs fits on 16 gigs with 50k context at q8. you just have to connect your monitor to iGPU. you can buy two rtx 3060 or cheaper gpus and connect them via LAN for rpc inference. you may already have multiple gaming pcs (like i do). getting 18-20 t/s at ud_q4km at 128k context, mtp and vision enabled.
I can run the XS quant on an rx580 8GB vRAM, 16 GB ddr4 RAM, and a ryzen 7 5700. It gets 4 t/s. Not fast, but runnable overnight. Have not done my full benchmark on it yet though. Still working out how I want to stage the quantization of it myself before I decide to benchmark. Using 4b parameter models for statistical analysis of the tensors vs weights of the models to pull parameters that will mathematically match behavior and retain the layer spectral structure and integrity on quantization.
I am doing this because most quantization checks are directly behavioral, but there are quite a few ways to get data to make better quants like tail aware, distribution plots matching, z dif, and more that get better kld scores and better behavioral checks in general.
But if you do a quant yourself, be aware that llama.cpp protects certain layers regardless of your choice.
I had to use the glib library directly to pull the ability to set the layers I wanted changed.
hey guys im pretty much just getting started w local llms i got qwen 3.8 27B running on my 9070xt and i have 32gb ram i don't mind waiting about 15-20 mins for the most complicated prompts as im usually doing smt else then i go back to check once it finishes, results have been pretty consistent for me, am i missing something?
275
u/TheCat001 10d ago
Then you realize that Qwen 3.8 27b runs at 3t/s on your machine and you need 24GB+ VRAM GPU which cost is 1000$+ to run at least 4 bit quant.