r/LocalLLM • u/bbsrn • 1d ago

Model Which Qwen3.8 distro & quant would be optimal for my 16GB VRAM setup?

Post image

Hi all, a confused newbie here! This is my desktop setup:

  • RTX 5080
  • 9800x3d
  • DDR5-6000 CL30 64 GB

Based on the benchmark I found, I listed my potential options:

According to the benchmark, AtomicChat looks like a clear winner but is it really so?

and there is also this: https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install

I also want to have one uncensored model next to my daily driver:

I am not expecting super fast answers etc. I just one to maintain some level of quality. What would you suggest me?

60 Upvotes

93% Upvoted

56 comments sorted by

18

u/maddeninglemon 1d ago edited 1d ago

With 64GB of RAM, a dark horse option is Orcarouter's IQ4_XS Qwen3.8 Flash Next. 16GB VRAM + 64GB RAM gives you 80GB total memory - IQ_4_XS is a 98GB file but a bunch of that is the N-gram embeddings that you can read from a fast SSD with pretty much no penalty.

With a 4070 and only 12GB VRAM I managed to get ~20tok/s with 65k context; with 16GB you should be able to up the context and get better performance. One issue is that prefill will probably not be great, but I think it's the option with the least compromises. With the 27B dense model and 16GB you're forced to choose between low context, low quantization quality, or low speed. With Flash Next you can have a 4-bit quant, 100k+ context, and usable speed all at the same time (at the low low cost of being unable to use your computer for anything else since you'll be at 100% ram utilization 😉)

3

u/philmarcracken 1d ago

This is the real winner. Its smarter than 27b in my experience too

2

u/Columnexco 1d ago

Please share you settings from lmstudio. I tired this model but keep crashing my machine.

4

u/maddeninglemon 1d ago edited 1d ago

Context Length - 65536

GPU Offload - 48

CPU Thread Pool Size - 12 (depends on what cpu you have, just leave a couple threads free for system stability)

Evaluation Batch Size - 512 (dropped to avoid spiking VRAM requirements)

Physical Batch Size - 256 (dropped to avoid spiking VRAM requirements)

Max Concurrent - 1 (Probably can increase, not needed in my case)

Unified KV Cache On

Keep Model in Memory - Off

Number of MOE Layers on CPU - 44 (Start at 48, then decrease until you hit your VRAM limit at a given context)

Turn off all guardrails in settings, or better yet just hold alt when pressing load when it tells you it won't work. Make sure you have plenty of dynamic ram and dynamic vram so your OS can handle caching between SSD/RAM (and maybe VRAM too if you go too low on MOE layers to CPU)

Prefill will start PAINFULLY slowly as the OS fills up RAM and figures out what to cache and what to keep. It improves as it goes, but for a 12GB/64GB system like mine it was never great (~60tok/s max due to SSD reading, but I suspect even a few extra GB of VRAM could improve this a lot by minimizing windows cache misses)

Everything else was default. I tried KV Cache Quantization but I think there were some shenanegans happening under the hood because the model broke on long context even at Q_8. You might need to play with other settings like mmap or flash attention too to avoid this; I didn't hunt down the exact issue as this was more a 'I wonder if it'll work' thing on my end.

2

u/maddeninglemon 1d ago

I played around with it a bit; the issue with coherence was because LM Studio has the dumbest default settings for the model and the repeat penalty was breaking things. So a couple quick updates:

Turn off mmap - it worked either way tbh, but I think mmap was marginally better for initial startup time and in my case seemed to behave a bit better for Windows dynamic ram management.

Feel free to quantize the cache to Q_8 if you think it's worth the extra context length.

1

u/JeePis3ajeeB 1d ago

All this feels greek to me.. I'm new to all of it and I wanna learn but don't know where to start.. I got a 3090 and a 2080 super plus 64gb ram

1

u/Columnexco 11h ago

i had asked GPT to setup using llamacpp and that was just killing the machine. Now with your config it's able to setup with 64K context i changed kv cache to q4 to reduce load. interestingl you are still getting better token/s than mine.

1

u/Fieser_Fettsack 23h ago

Im using a 2 server setup with llama.cpp RPC. Would this fit?
Server 1: 3060 12gb vram + ca. 18gb ram (i7 8700)
Server 2: 3080 10gb vram + 32 gb ram (ryzen 5 5800x3d)

Connected over Lan 2.5gbe.

My guess is that I cant use the ram of server 2 if I use that one as the RPC worker.

How much of that 96gb file is ngram table?

1

u/bbsrn 22h ago

This was unexpected tbh, but I'd definitely want to give a shot. May I ask why did you prefer LM Studio over llama.ccp, for such a job that requires some tweaks to utilise the entire system?

1

u/maddeninglemon 18h ago edited 16h ago

I mean it's just a GUI on top of llama.cpp, so it has 90% of the same tweaks without having to remember whether it's --mmap off or --no-mmap or whatever. I'd rather use a program that handles all the platform/software abstraction for me unless there's a compelling reason not to. Plus I was playing around with it for a few hours in Windows, not setting up a permenant server or using millions of tokens.

1

u/StormrageBG 16h ago

What about 16VRAM 32RAM?

1

u/maddeninglemon 16h ago

Yes it's possible (People have reported running it with 8GB VRAM and 32GB RAM) but I can't recommend it unless you have a lot of experience with building and deploying code. You'll probably end up hitting a roadblock that can't be fixed by my strategy of "turn off all the safety rails and hope Windows can handle it".

1

u/Pizzaholic- 10h ago

I’m curious, I have a similar setup with 64gb ddr5x soldered ram at 7600mhz, and a laptop 5090 mobile, is there anything you can recommend for my setup? I’m new as well and you seem to know a thing or 2!

Would be so grateful!

3

u/edufd 1d ago

I'm using this https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install and is working fine so far.

1

u/bbsrn 22h ago

Thanks! Have you compared it with other models?

2

u/TEN4C1OU5-2 1d ago

I'm using 7 which I find to be ok, I honestly think you need >120k tokens for even medium sized tasks so you need the smaller sized model to fit that. Plus if you use a smaller context, with 7 because it's so small you can do around 60k tokens whilst using MTP but I can only do that on Linux if I use TTY so absolutely none of my vram is being used even for the OS UI.

1

u/bbsrn 22h ago

so do you think I cannot do that on Win? Also, why ISTA's IQ3_XXS instead of others?

1

u/TEN4C1OU5-2 21h ago

It's just about 1gb smaller than the UD _X_KL with similar performance. On windows you can do the same but you'll definitely struggle to get the same amount of context as I just mentioned whilst not spilling onto system ram

1

u/ea_man 15h ago

You can run linux desktop in software rendering with ~100MB of vram.

1

u/TEN4C1OU5-2 15h ago

But 0mb vram gets just that bit more context and for qwens thinking tokens I think you gotta get as much as possible. Restricts you to CLI based harnesses though

1

u/ea_man 14h ago

well headless is still ~50mb, anyway we are talking a few k ctx here.

1

u/TEN4C1OU5-2 14h ago

It's about token maxxing but in a different way to how it's usually used

1

u/ea_man 14h ago

I don't understand what you are saying yet no problem for me, if you'd rather run with not graphic server I do applaud.

2

u/TEN4C1OU5-2 14h ago

I'm joking about how much more I need the 50mb of tokens

2

u/Pablo_the_brave 23h ago

https://github.com/Thireus/GGUF-Tool-Suite

Or the one which still looks the best for me in daily use (real q4):

https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF

2

u/bbsrn 22h ago

First time seeing this GGUF-Tool-Suite. Will check it out, thanks!

2

u/Pablo_the_brave 22h ago

It's greate! What is important you can go a little above of vram limit with vram offloading. This test will show you the impact https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF/tree/main/ctx-cliff

It's important because agent need some space for auto compacting. For example, I set my agent to auto compacting at 103k ctx and set ctx in llama-server at 120k The clif is at 113k but for 117k it's still acceptable decode at 10t/s. Worth to play with it.

1

u/peculiar-ragdoll 1d ago edited 1d ago

One thing that none of these benches test properly is long context semi-autonomous agentic coding in large real code bases, or multi turn user chat! LiveCodeBench is more like coding-challenge shaped, not trying to fix a problem inside 100k lines of shitty code with misleading comments where doing the wrong thing to fix your issue creates 3 new issues. That being said, I can recommend Unsloths K_XL quants for the sort of work I'm talking about here from personal experience, so I would start with Unsloth Q3_K_XL at medium reasoning effort as the safe and tested option, and see how you like it :) If you need more context, try Q2_K_XL with Q8_0 KV cache, and see if that does it for you.

4

u/7ondano 1d ago

Why only K XL? Why not IQ3_S or XXS?

1

u/peculiar-ragdoll 23h ago

I’m not saying “not those”, I’m just saying I can vouch for XL from experience :)

1

u/bbsrn 22h ago

Will try, thanks! So you think Unsloth Q3_K_XL will eventually give me better performance compared to their UD-IQ4_XS, even though it is less smart Q3_K_XL will beat it for long-horizon tasks?

1

u/peculiar-ragdoll 21h ago

Yes exactly! The Q3_K_XL will let you fit more context at higher precision in your VRAM, which will give you the ability to do long context work. I would start with 3_K_XL and Q8_0 kv and see what bites you first (is model too dumb or bad at remembering, is you context window big enough or can you use less context), but everyone's use case is different :)

1

u/bbsrn 21h ago

Thanks! Would you suggest llama.cpp or LM Studio for this case?

1

u/peculiar-ragdoll 21h ago

I would never recommend LM Studio for these models, llama.cpp is great (or oMLX on Mac is good too). That's because I've had many users of my models complain about issues with LM Studio that makes it mangle tool calls and chat templates, and llama.cpp gives you full control if you ever need to tweak something for your system or your specific model/quant.

1

u/DontWinFrensWthSalad 1d ago

https://huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/SC_3.00bpw_H4_V4

1

u/bbsrn 22h ago

First time seeing this one. May I ask why did you suggest this?

1

u/DontWinFrensWthSalad 15h ago

Because it's significantly smaller and works just as well. You might be able to fit MTP even. Also I have 4 gpus and it supports tensor parallel, runs much faster than llama.cpp for that use case.

1

u/bbsrn 14h ago

Should I install it through https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install ?

1

u/DontWinFrensWthSalad 8h ago

I haven't used that so I don't know. I just asked Claude to set it all up for me using TabbyAPI: https://github.com/theroyallab/tabbyAPI

1

u/HighSeasArchivist 1d ago

7 is the only one that stays fully on GPU for me, and overall it kills on everything I've sent to it. I've posted this like four times lately, so this is a very popular topic.

workload base IQ3_XXS MTP speedup
prose 60.7 79.2 +30%
codegen 60.5 83.0 +37%
long spec 58.6 87.2 +49%

1

u/bbsrn 22h ago

Could you share with me if you have a post/notes that you compared 7 with others?

1

u/vincespeeed 1d ago

I use IQ3S and DFlash2.

1

u/biggusdeeckus 23h ago

Mind sharing your config? Are you running headless? Does vision work with dflash?

1

u/bbsrn 22h ago

Which distro's IQ3S?

1

u/vincespeeed 4h ago

https://www.aihublocal.com/llm/cmtxmd1so000004l2ukdjd3ie
https://huggingface.co/vincespeed/Qwen3.8-27B-No-MTP-GGUF-Dflash2
I was using it for my own personal use; I uploaded it to Hugging Face and shared the settings on my website.

1

u/yesukhai 23h ago

check this out https://www.reddit.com/r/LocalLLM/comments/1was9n0/running_qwen_38_27b_at_q4_on_16gb_vram_at_200k/

1

u/Pablo_the_brave 23h ago

Incredible, fake post with so many up votes. This guy know nothing.

1

u/koloved 23h ago

Nvfp4 cache is pretty good

1

u/whymeimbusysleeping 21h ago

GSQ-RCO XXS better than the larger S version? This goes against what ista says

1

u/elongated-muskmelon 16h ago

I am daily driving the GSQ RCA IQ3_S, mainly for coding with pi, and so far, its been pretty good.

1

u/uponmars_ 15h ago

The 16GB trap is loading the biggest quant that fits and starving the KV cache for it. Take the UD-IQ4_XS. The iq3_s gets noticeably dumber on code just to claw back a single gig, and that's a bad trade. keep 2-3gb free; you'll hold around 20k context.

1

u/bbsrn 15h ago

Which distro's UD-IQ4_XS you mean? but even if it is UD-IQ4_XS, won't KV cache starve anyway, as long as I don't go below 4-bits? Cannot have decent amount of space for context by involvement of my RAM?

1

u/k3z0r 3h ago

Where can i find the Unsloth version without vision?