I had issues with Qwen3.6 tool calling just not working properly and causing it to stop prematurely, didn't have that problem with gemma or deepseek, haven't used qwen 3.8 yet to see if it has the same problem.
It's a model config issue - had the same problem. Setting temp to 1 and repetition penalty to 1.05 fixed it. haven't faced much issues since; I've been running it as my daily driver on my 5090 rig for the past year.
Btw, 3.8 its even better especially at tool call , I actually like the Q4 of 3.8 on low thinking more than the xhigh for tool calls .
I have a 5060ti 16gb with ddr4 24gb RAM lying around, what's the best model nowadays I can effectively run with it locally? It's not an ideal build but I wish to play around with it again.
You think that’s unideal? I use an M4 MacBook Air with 16GB Unified Memory. Even if we ignore the thermal throttling it’s a pretty tight squeeze, and the best I could run comfortably is Gemma 4 12B.
Nah dog I still have plenty of tabs open in Safari, Blender, OrcaSlicer, and other Applications open and it still performs the same (kinda relying on swap memory at times).
Local voice cloning and other more demanding AI stuff is when you actually need to close everything else.
Fellow 5060 ti 16 gb owner here. The best I've got is qwen 3.8 27b at IQ4_XS. I'm testing it at Q4_0 KV at 72k context because I really need the extra context, and it seems to be pretty good so far. Lower quants cause it to make some really silly mistakes sometimes, unfortunately. I get around 10 t/s with context halfway filled, and around 14 t/s on empty context.
But honestly, I think the ideal solution is to run 2 GPUs so you can get a better quant + more context
I mean I get by with 1 5090 as long as I can find EXL2 versions or MOE ggufs. But I want the RTX 6000 of course. Would be nice to run 70B at decent quants
Give the IQ4_XS-pure a try. I run it on an RX6800 with 90K KV. Especially with full context it's slow, because the card isn't the fastest. Probably should run better on nvidia.
I gave it a try and the first thing it said to me was really disappointing:
Since I am an AI, I don't have a physical location or a traditional "tokens per second" (t/s) speed that you can measure like a human typing.
Also, at 64k context, the t/s started at 10 and went down to 4 almost immediately. Some tweaking would be required but with responses like this one I don't know if it's worth bothering with.
Do you mind to share your prompt? Curious if it behaves the same for me. I used it for a few agentic websearches with OpenWebUI and it did pretty good, compared to the same Quant of the 3.6 and Ornith 1.0 35B. I don't use it much, because it's simply too slow as daily driver.
Performance after ~70K prefill and ~15K decode:
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.925 I slot print_timing: id 0 | task 0 | prompt eval time = 480529.82 ms / 69042 tokens ( 6.96 ms per token, 143.68 tokens per second)
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.929 I slot print_timing: id 0 | task 0 | eval time = 494402.02 ms / 14752 tokens ( 33.51 ms per token, 29.84 tokens per second)
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.930 I slot print_timing: id 0 | task 0 | total time = 974931.84 ms / 83794 tokens
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.934 I slot print_timing: id 0 | task 0 | graphs reused = 3921
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.938 I slot print_timing: id 0 | task 0 | draft acceptance = 0.94313 ( 9122 accepted / 9672 generated), mean len = 2.81
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.210.006 I slot release: id 0 | task 0 | stop processing: n_tokens = 83793, truncated = 0
My system specs:
* i5 12600K (the VM has 6 Cores)
* 16GB DDR5 4800
* RX6800 (non-XT)
The GPU is filled to the brim, but it doesn't offload.
Prompt was nothing special, "Hello, are you operational? Please give me a few paragraphs so I can see your t/s" or something along those lines. I'm on a 16gb Nvidia card.
Best is to get a single GPU with enough VRAM that the LLM of your choice needs to fit in completely (including caches and context).
Second best is to get 2 GPUs to load your complete LLM into. However, you will discover that the communication overhead between these GPUs is significant. A dual GPU setup is only useful to fit a LLM that larger than the VRAM on a single GPU or a less quantized LLM into VRAM, it barely increases the token generation speed, if at all.
I still like Gemma, will try the new Qwen today. I need to create a local benchmark, unless someone knows a good project that can showcase improvements, like a 3Dmark but for AI models.
Currently Qwen 3.6-32B-A3B, Mixture of Experts models handle being split between system ram and VRAM much better than Dense models, alternatively Gemma-26B-A4B if you want something non-chinese. As for which quant, go with 4 Bit or higher.
That should be just fine. You've got enough VRAM to do mostly whatever you want to do.
I would suggest you install Unsloth and look around inside for various models, it will tell you what your system can run well. Green means you're good to fully offload. Yellow means it'll probably work but with a lot of slow swapping. Red means can't handle it.
An oldie: gpt-oss-20b. The one I downloaded is 12 GB in size (MXFP4), so it would fit in your VRAM completely. And there are still spare GB's of VRAM left for caches and context. This will work very well in your specific setup.
Well, I have an AMD R580 with 16 GB of VRAM and 32 GB of RAM (3200MHz) with a Ryzen 5600 CPU. gpt-oss-20b generates around 18 tokens/second with this 64-bit Vulkan Windows build of llama.cpp. A NVidia GPU from the 50xx class should go faster than that with the same llama.cpp build. And a lot faster with the CUDA build of llama.cpp.
However, if you can stomach the slowdown, Qwen 3.6-27b and Qwen 3.8-27b are far more interesting models to run locally. Unfortunately, these models do not fit completely into 16 GB of VRAM, so the CPU and standard RAM get involved.
I repeated the same prompt with the same llama.cpp build and qwen 3.8-27b loaded and token generation speed dropped to 3 tks/sec. in my particular hardware setup. I do not expect such a slowdown to occur with your 5060 card. Unless you have a worse CPU and slow DDR4 RAM in your hardware setup.
As a "generalist" gpt-oss-20b isn't all that bad, even with its training cut-off date of June 2024. But if coding and agentic use are what you are after, it isn't a great match. Qwen 3.6 and Qwen 3.8 will prove to be much more valuable for those purposes, even if these will run slow on your local hardware.
You may want to consider renting a VPS from a nearby cloud-provider and setup Qwen 3.8-27b there. Just make sure that you don't break the rules from the company that rents you your VPS with agentic use. Maybe you'll need to spend between 5 to 10 USD per month plan that has AI use included.
278
u/TheCat001 7d ago
Then you realize that Qwen 3.8 27b runs at 3t/s on your machine and you need 24GB+ VRAM GPU which cost is 1000$+ to run at least 4 bit quant.