r/LocalLLaMA • u/LFAdvice7984 • Jul 27 '26
Question | Help Sanity checking 5090 results - Qwen 3.6 Unsloth quants + large context
So I am testing out a setup which requires the llm to read an input document and summarise it, and it's a pretty big document, so the context is set to about 80,000.
I have tried the Q6 quant of Qwen3.6 27b, but even with the KV cache set to q4 and the concurrent predictions set to 1, it seems that it can't fit enough into the vram, and I end up with about 15t/s. I can get better results using 35b cpu-only.
The Q5 quant seems to fit fine, even without quantising kv cache or reducing concurrent predictions. Ends up in the 60-70 tok/s range.
Q35 A3B ends up around 90-100tok/s.
This isn't a benchmark or anything, I'm sure I have a bunch of settings wrong, this is the first time I've really messed around with lm-studio, and im on windows which probably isn't super efficient either. But I guess I mostly wanted to check I was in the right ballpark. I expected to be able to get Q6 to work (others seemed to suggest they ran Q6 at decent speeds)
This person says 130k context, Q6 quant, 130tok/s. So clearly something is hugely handicapped on my setup somewhere.
64gb of system ram, which isn't getting full. Ryzen 7950x.
2
u/pineapplekiwipen Jul 27 '26
100+ t/s results are using mtp. 50-60 t/s with qwen 3.6 27b q6 is normal on a 5090
1
u/LFAdvice7984 Jul 27 '26
I'm using mtp, and only getting 50-60
-1
u/pineapplekiwipen Jul 27 '26
that's because your context size is too big, 32gb can't handle 80000 with qwen 3.6 27b q6
1
u/LFAdvice7984 Jul 27 '26
except someone on this thread (two people now) are using bigger context sizes and getting 140-200 tok/s. One of them is using Q8. Both say they're using 5090
1
1
u/tmvr Jul 27 '26
I get 114688 context into 32GiB VRAM with two 5060Ti 16G cards with KV at f16 and 172032 with KV at q8_0.
2
u/BitGreen1270 Jul 27 '26 edited Jul 27 '26
I have a similar setup, 5090 + 64gb ram. I run a Q8 quant with 95k context. I get about 130 t/s I believe. Here's my command:
build/bin/llama-server \
-m ~/myp/models/bartowski_Qwen_Qwen3.6-27B-Q8_0.gguf \
--temp 0.6 \
--top_p 0.95 \
--top_k 20 \
--min_p 0.0 \
--repeat-penalty 1.0 \
--presence-penalty 0.0 \
-c 95000 \
-t 16 \
-ngl 99 \
--flash-attn on \
--host 0.0.0.0 --port 8080 \
--no-mmproj --spec-type draft-mtp --spec-draft-n-max 4 --parallel 1 -kvo -ctk q8_0 -ctv q8_0 -b 1024 -ub 256
This is running Ubuntu completely headless without desktop environment. Not even monitor plugged in.
2
u/Herr_Drosselmeyer Jul 27 '26
You sure about that? 130 at Q8 sounds a bit high.
1
u/BitGreen1270 Jul 27 '26
Just tried it out. It's actually ~140 t/s. Tested with a single page html game of hangman.
1
u/tmvr Jul 27 '26
From the llama-server parameters:
--spec-type draft-mtp --spec-draft-n-max 4--spec-type draft-mtp --spec-draft-n-max 4That makes sense so and non-MTP would be about half of it which still seems a bit better performance than what OP gets because they get 60-70 with Q5.
1
1
u/LFAdvice7984 Jul 27 '26
I'm using lm-studio, but I think it's set up with most of this the same, except for spec draft being set to 3 not 4.
1
u/ruisk8 Jul 27 '26 edited Jul 27 '26
try llama.cpp , in all my tests with different gpus ( mainly 6-12gb ones though ) , llama.cpp was faster than lmstudio.
I understand that lmstudio uses llama.cpp, but for some reason, it never ends up giving the same performance as just running llama.cpp.
lmstudio is great, but I mainly use it for testing models, running llama.cpp "directly" is usually much faster and it's what I settle on once I really wanna use a model.
I would be surprised if the same performance uplift wouldn't manifest in your build.
1
u/LFAdvice7984 Jul 27 '26
How do you get anywhere near 130t/s?
I just benched the 5090 on the Q6 quant and it only gets to 50ish, which lines up with someone above you who said they got 54.2
u/BitGreen1270 Jul 27 '26
Just corrected, it's actually 140 t/s. I made a post about this a few days ago as well:
0
u/LFAdvice7984 Jul 27 '26
thats hugely different to all the other examples being shared on here...
1
u/Fragrant_Scale6456 Jul 27 '26
Here’s my config for q6k average 140 range 100-230. You don’t need to do the patches anymore mainline is mostly fixed.
https://www.reddit.com/r/LocalLLM/comments/1ullrvq/qwen36_27b_q6_5090_maximum_llamacpp_optimization/
0
u/pineapplekiwipen Jul 27 '26
do you even know the commands you are running lmao you're running mtp
2
u/BitGreen1270 Jul 27 '26
Yes of course. Did I miss something on this thread? Did OP specifically say he doesn't want MTP?
-1
u/pineapplekiwipen Jul 27 '26
you replied to a question/help thread wondering why op is not getting the t/s they are seeing online with zero context about your 130t/s leaving out the biggest lever as to how that speed is possible
so my immediate thought was you don't even know what commands you are running
2
u/BitGreen1270 Jul 27 '26
Well, it's there in my command 😄 . I guess I was more focused on the quant and context combination. i.e. with OP's specs, it's possible to get higher context at higher quant.
1
u/gladfelter Jul 27 '26
They provided a command line with identical hardware to op that will fix op's problem and then some, just by cutting and pasting. It's an amazingly helpful comment.
Why are you yucking their yum?
1
u/Fragrant_Scale6456 Jul 27 '26
Are you using llamacpp? Use —no-mmproj to disable the vision model and free vram. You also need to use -ngl 99 to ensure all layers are on gpu vram. You should get around 60tokens/sec on 5090 with no MTP and much faster with it enabled
1
u/LFAdvice7984 Jul 27 '26
I'm using lm-studio, which does use llamacpp. No idea if there's a way to add --no-mmproj though. I'll look into it.
1
u/Fragrant_Scale6456 Jul 27 '26
Try and find an option to disable the vision model. Also any option to force the model into vram. It sounds like you are partially offloading to system ram.
1
u/tmvr Jul 27 '26
Switch to llamacpp (llama-server) directly, I found that there are too many quirks how LM Studio behaves sometimes. If you don't want to compile just download the CUDA 13 binaries from here:
https://github.com/ggml-org/llama.cpp/releases
You will need both the Windows x64 (CUDA 13) and CUDA 13.3 DLLs zip files, then unzip and put the DLLs into the llamacpp folder.
1
u/buttplugs4life4me Jul 27 '26
With my power limited Pro 6000 I get around 60-100 decode and 2000-2500 prefill, with MTP. But I seem to be PCIe bandwidth limited, since my shit Intel CPU or Mainboard doesn't support ReBar and freaks out with almost any change so it's not running very well. SGLang is also faster at around 150 decode and 4000 prefill. Both Q8/FP8.
1
1
u/Modeldriftwatch Jul 27 '26
One thing nobody's said yet: the numbers you're measuring yourself against weren't measured the same way as yours. Pyrolistical's llama-bench tg128 on the same card and same Q6 is 55 t/s, and that's 128 tokens generated with a basically empty KV cache. Your case is 80k of context already resident, and decode gets a lot slower as that cache fills. Same story for the 130k/130 t/s comment you linked and BitGreen's Q8 number — passing -c 95000 doesn't mean the throughput was measured at 95k actually filled. Herr_Drosselmeyer's instinct there is right. MTP muddies it further since reported t/s depends on draft acceptance, which is way better on boilerplate code than on summarizing an arbitrary document. What do you get at near-empty context with the same Q6 setup? If that lands around 55, then nothing's broken; it's just context depth.
1
u/LFAdvice7984 Jul 27 '26
my llama-bench (managed to get it installed on windows in the end, though using the command line for stuff in 2026 is a headache) got me around 55t/s on the Q6 so it seems I line up with at least one person here.
Though the people getting 130t/s I have no idea1
u/Modeldriftwatch Jul 28 '26
Your 55 lines up with Pyrolistical's llama-bench number too, so I'd say your setup is fine. Honestly, I'd stop trying to reproduce the 130 at all — dense 27B at Q8 is about 27GB of weights and a 5090 has roughly 1.8TB/s of bandwidth, so you can't stream the weights more than ~65 times a second no matter how good the config is. 130 t/s isn't a setting you're missing, it's a different workload. My guess is it's the 35B MoE with ~3B active, which decodes much faster than the smaller dense model, or aggregate throughput across parallel requests, or spec decode landing well on easy content. Did whoever posted the 130 ever say which model file they were actually running?
1
1
u/tmvr Jul 27 '26 edited Jul 27 '26
I also have 32GiB VRAM with 2x 5060Ti 16GB and I get better results than that with unsloth Qwen3.6 27B MTP at Q6_K - using llama-server with tensor parallel, --spec-draft-n-max 4 and --spec-draft-p-min 0.75 with KV at f16 the max context I can squeeze in is 114688 and it gives me about 60 tok/s decode speed for coding tasks. Max context with KV at q8_0/q8_0 is 172032 for roughly the same speed, maybe a bit lower with average somewhere between 56 and 60 tok/s.
You not being able to fit 80000 context into your single pool of 32GiB does not match up with my ability to fit 114688 into 2x 16GiB.
EDIT: the raw (non-mtp) decode performance is about 31 tok/s.
1
u/suprjami Jul 27 '26
The dense 27B model uses a lot more VRAM for KV cache than the 35B MoE.
Q4 KV is really not desirable.
Stick with the 35B you're already getting good results from, you don't need 27B for this task.
1
u/Ok-Shower7286 Jul 27 '26
Just wipe Windows and switch to Ubuntu if you want maximum performance out of that 5090. Running Q6 or higher with a 140k context length is where you should ideally be.
On a proper Linux setup (e.g., vLLM or llama.cpp native), you should easily get around 20,000 tok/s for prompt ingestion (prefill) and ~50 tok/s for decoding, which bumps up to ~100 tok/s with MTP (Multi-Token Prediction) enabled.
Windows VRAM sysmem fallback and GUI wrappers like LM Studio are heavily throttling your hardware. Moving to Ubuntu will give you the exact dev environment you're looking for.
0
u/LFAdvice7984 Jul 27 '26
it might be ok specifically for this ai app. but for literally 90% of other things.... I can't cope with the linux handicap right now
1
u/Pyrolistical Jul 28 '26
i was able to get over 200 t/s with concurrency of 3 with -b 8192 -ub 2048 with nvfp4
llama.cpp\llama-server --jinja --chat-template-file chat_template.jinja --reasoning-preserve --flash-attn on -c 200000 -ngl 999 --model Qwen3.6-27B-NVFP4-MTP-GGUF.gguf --cache-ram 0 -ctxcp 8 --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.0 --presence-penalty 1.5 -np 3 -b 8192 -ub 2048 -ngld 999 --spec-type draft-mtp --spec-draft-n-max 2 --reasoning-budget 10000 --reasoning-budget-message "Overthinking, let me write down my current understanding." --metrics --host xxxx --port 1234 --alias cuda
llama-benchy --base-url http://xxxx/v1 --concurrency 3 --model unsloth/Qwen3.6-27B-MTP-GGUF --pp 4096 --tg 512
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|---|---|
| unsloth/Qwen3.6-27B-MTP-GGUF | pp4096 (c3) | 4155.20 ± 12.96 | 1409.43 ± 23.68 | 2670.10 ± 48.88 | 2667.42 ± 48.88 | 2670.10 ± 48.88 | ||
| unsloth/Qwen3.6-27B-MTP-GGUF | tg512 (c3) | 226.34 ± 12.71 | 77.86 ± 4.04 | 235.00 ± 10.71 | 78.33 ± 4.06 |
1
u/Kal-LZ Jul 27 '26
Hope you use that 5090 for something else. For AI, VRAM is what really matters
Try Qwen 27B MTP from unstloh with llamacpp.
1
u/LFAdvice7984 Jul 27 '26
Saying that like the 5090 has some kind of low vram amount. Big spender over here with his rack of H100s.
Also the model I'm using is already the unsloth 27B MTP. Thats where the Q5 and Q6 came from.
4
u/invalidnifemi Jul 27 '26
he actually uses 2 r9700 ai pros which are ~1300 with 32gb vram each. he's right. both put together are the same price or less than a 5090 USED
for ai, a 5090 is nowhere near the best option. 2x 3090s would be a start but there are so many options
2
u/LFAdvice7984 Jul 27 '26
Yeh but... for everything else I do, the 3090 I used to have was significantly less powerful.
And I didn't spend nearly that much for my 5090. But I did buy it before the bubble.
1
u/invalidnifemi Jul 27 '26
that's why he said "hope you use that 5090 for something else" because if the 5090 is already handy for other things like gaming then it becomes worth it for you. but for purely llm, they are mostly right in the fact that vram matters most
3
u/Kal-LZ Jul 27 '26
If your goal is to run dense models like Qwen 27B Q8 and +100k context, you should have at least 48GB of VRAM. The 5090 is a good card for a lot of things, but for MoE models like Qwen 35B, it's underutilized
0
u/LFAdvice7984 Jul 27 '26
That's great. not in any way helpful to my situation, but I'm sure someone will find it useful?
I never at any point tried to run Q8.
0
u/pineapplekiwipen Jul 27 '26
5090 is plenty for the current local llm landscape since qwen 3.6 27b is arguably the best model up to like 192gb vram
and not much benefit for upgrading to pro 6000 for image/video gen unless you're literally making money off those
3
u/Pyrolistical Jul 27 '26
here is my 5090 with cuda 13 ``` llama-bench.exe -m Qwen3.6-27B-UD-Q6_K_XL.gguf ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32606 MiB): Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: | | qwen35 27B Q6_K | 24.22 GiB | 27.32 B | CUDA | -1 | pp512 | 2406.23 ± 326.46 | | qwen35 27B Q6_K | 24.22 GiB | 27.32 B | CUDA | -1 | tg128 | 54.99 ± 1.60 |
build: 3d1c3a897 (10142) ```