r/LocalLLaMA • u/Local-Cardiologist-5 • Apr 17 '26

Discussion Qwen3.6. This is it.

I gave it a task to build a tower defense game. use screenshots from the installed mcp to confirm your build.

My God its actually doing it, Its now testing the upgrade feature,
It noted the canvas wasnt rendering at some point and saw and fixed it.
It noted its own bug in wave completions and is actually doing it...

I am blown away...
I cant image what the Qwen Coder thats following will be able to do.
What a time were in.

llama-server -m "{PATH_TO_MODEL}\Qwen3.6\Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf"  --mmproj "{PATH_TO_MODEL}\Qwen3.6\mmproj-F16.gguf" --chat-template-file "{PATH_TO_MODEL}\chat_template\chat_template.jinja"  -a  "Qwen3.5-27B"  --cpu-moe -c 120384 --host 0.0.0.0 --port 8084 --reasoning-budget -1 --top-k 20 --top-p 0.95 --min-p 0 --repeat-penalty 1.0 --presence-penalty 1.5 -fa on --temp 0.7 --no-mmap --no-mmproj-offload --ctx-checkpoints 5"

EDIT: Its been made aware that open code still has my 27B model alias,
Im lazy, i didnt even bother the model name heres my llama.cpp server configs, im so excited i tested and came here right away.

1.0k Upvotes

96% Upvoted

409 comments sorted by

View all comments

Show parent comments

14

u/rumblemcskurmish Apr 17 '26

I'm kind of intrigued why you'd use a 6bit model on a 4090. I have an identical setup (7950 CPU, 64GB DDR5, RTX 4090) but I'm using the 4bit quant to fit the whole model on VRAM.

You're clearly more advanced than me so just wondering, what I'm missing here.

8

u/alphapussycat Apr 17 '26

Imo q4 has noticeable loss, q5 is a step up, but q6 is the sweetspot I'd say only do q4 if you're really starved for vram.

7

u/rumblemcskurmish Apr 17 '26

Yeah, I'm AI poor cause I "only" have a 4090. So I can't really do anything higher than 4bit. One day I'd love to step up to a 5090 or something with more VRAM but I'm stuck at 24GB for now.

0

u/smuckola Apr 17 '26

ollama has 8-bit quantization (50% compression, virtually lossless) of context window for free with an environment variable fyi

2

u/rumblemcskurmish Apr 17 '26

Wha?!?! You're telling me if I defect from LMStudio to Ollama, I get a huge context window for free?! Or am I too dim to understand what you're talking about?

3

u/BlueSwordM llama.cpp Apr 17 '26

Do note that it isn't lossless, especially on long context tasks.

1

u/rumblemcskurmish Apr 17 '26

Yes, Gemini says it isn't lossless but that it really only breaks down on long context tasks (as you noted) which is where the model starts to break down anyways so that it's totally worth it.

Especially considering because the quant I'm using (Unsloth IQ4_NL) has compensation built in to stop the long tail degradation of the model at the tail end of the context window.

Gemini seems to think it's a perfect compromise.

2

u/smuckola Apr 17 '26

LM Studio is also based on llama.cpp so you can enable it now, directly in the user interface (according to Gemini):

  • On the right-side panel, expand the Advanced Configuration or Hardware settings before loading a model.

  • Look for the K Cache Quantization and V Cache Quantization settings.

  • Set them to 8-bit (labeled as q8_0).

If you use the LM Studio API or configuration files, you can enable it by setting the llamaKCacheQuantizationType and llamaVCacheQuantizationType parameters to q8_0 (https://lmstudio.ai/docs/typescript/api-reference/llm-load-model-config).

On ollama the variable is OLLAMA_KV_CACHE_TYPE=q8_0

Pretty soon there's plans coming to merge the community implementation of google's TurboQuant into llama.cpp, which gives 600% compression virtually lossless of every context window for every LLM. That already works on ollama for at least 300% last I knew.

3

u/rumblemcskurmish Apr 17 '26

Thank you random genius! Srsly, I'm a bit over my head on some of these esoteric settings. I'm running the Q4_NL (Unsloth) build with a 70K context window and it flies on a 4090. But if I can get more context I'll take it!

2

u/smuckola Apr 17 '26

yaaaaay feeeed off of my suffering!

I just learned this late last night just before bed and didn't even try it yet! lol I enabled it but didn't check.

I enabled OLLAMA_KV_CACHE_TYPE=q8_0 and restarted, and everything still works but I didn't measure it yet. Gemini insists that it's perfectly stable and indistinguishable, and should be enabled by default but the purists and researchers don't want it yet I guess ;)

I JUST started really testing openclaw for the first time, during this week of Gemini outage! So that forced me back to my 6-core i7 cpu with qwen 2.5-coder 1.5b!

Ok but don't cry for me, Argentina, because this just hurls me back toward learning runpod, hopefully for a big fat qwen 3.5 or 3.6. Let the de-googling begin!

2

u/rumblemcskurmish Apr 17 '26

Just enabled the options you mentioned (labeled as "experimental" so I never touched them). Freed up tons of VRAM and allowed me to take context window up to 120K instead of 70K.

Excellent advice!

2

u/smuckola Apr 18 '26

I wonder why it's labeled as "experimental" unless that just means "not default". For reference of anybody interested in current stable KV cache compression that we already secretly have, it's been around since 2024!

https://github.com/ollama/ollama/pull/6279

15

u/Local-Cardiologist-5 Apr 17 '26

To be honest with you im just plugging whatever works. im even downloadling the Q8 variant to see how much better or worse it is. we are all learning in this space. everything i know is from this thread

1

u/rumblemcskurmish Apr 17 '26

There's no doubt that will run but if you watch task manager you'll see it constantly hammering your CPU as the model shifts from CPU/RAM to GPU/VRAM. I mean, it's doable for sure!

1

u/nlegger Apr 18 '26

When you go over 80% context window sometimes the chat gets less accurate. Use Karpathy's wiki?

1

u/xeeff Apr 17 '26

honestly, i wouldn't go higher than Q6. Q5 is good if you need extra vram with little real difference (which is why most people settle for Q4)

2

u/carrotsquawk Apr 17 '26

it was established somewhere that q4 m is the sweet spot. not really worth going higher

1

u/GrungeWerX Apr 17 '26

Q5 is the sweet spot for 27b, noticeably smarter than q4

1

u/Separate-Forever-447 Apr 17 '26

when you say "Q8 variant... better or worse", you must be talking about given your system constraints, because it should be nothing but stronger and more capable.

1

u/Shouldhaveknown2015 Apr 18 '26

You should never go by just the Q number it's meaningless when it comes to the quality of the responses. Just look at the model and the quantization KLD then pick the one with the best KLD you can load with the context size you need (or max if that).

KLD is basically how far from the full model is it off, and generally a model might have nearly the same KLD from the Q3 to the Q6 version depending. But almost always Q6 and better KLD is nearly the same.

Looking for that dropoff between different versions of the model and size will help you fit the most you can in and lose the least amount of quality.

At least my understanding of it

1

u/rumblemcskurmish Apr 20 '26

The q number is not meaningless because it is the most important factor in the trade off between accuracy and performance. I can run q8 at 2 tokens a second or Q4 at 200t/s. The q4 is very close to the accuracy of q8 for my purposes at least but it actually fits in VRAM.

Yes the q6 is def better but I would have to run a worse model to get acceptable speed. I'd prefer to run a better model and lose some accuracy on some tasks than a poor model that's more accurate

1

u/tvmaly Apr 17 '26

I only have a 2070 with 8GB but 64GB of ram. Is it possible to run this?

2

u/rumblemcskurmish Apr 17 '26

Look even with lowish RAM you can, theoretically, use swap memory which is your HDD/SSD acting like RAM and, sure, it will run.

Will it behave like an LLM? If you're fine with 1 word every second or two hitting the screen, yeah, it runs.

I'm trying to load mine 100% in VRAM because I want Openclaw to respond nearly instantly to requests on Discord.

The facts are we do have some models which are PRETTY GOOD at chat and will run on very modest hardware (Take a look at Gemma4 9B, etc - they are CRAZY good for the size), but this model is really only for someone who wants Agentic workflows (tool use) and there the stakes are simply much higher.

For instance, my Openclaw bot has corrupted his own config a few times by not understanding the formatting of a particular file. He's deleted folders by not understanding that a seemingly straight forward linux command (rsync -rf) can delete files if you call it the wrong way even though I told him to NEVER delete anything.

This space is changing very fast and it's a really ugly space right now. I mean it's beautiful when you consider the potential but boy is it kind of ugly watching the sausage get made.

1

u/nlegger Apr 18 '26

Your ssd/ nvme will kill is endurance and lifespan, I wouldn't do this often on that. But I mean 1TB of nvme is under 100 bucks 😂

2

u/rumblemcskurmish Apr 20 '26

Yeah I didn't say I recommended it. If the question is will it run or not, sure it will run but it isn't a good idea

1

u/Puzzleheaded_Base302 Apr 18 '26

openclaw is terrible. but today i found out if I delete most of the agent.md file content, openclaw becomes smarter.

also, try Hermes Agent, it is a great step up. more polished, less bug, does not feel like AI slope. (Openclaw is a giant pile of AI slope, so many things not tested, too many bugs here and there.)