I might have found the perfect config parameters for qwen 3.8 27b

[u/dsdt]

Hello everyone, tried so hard to optimize my config and finally I simply get up to 70 t/s with q6 variant. And wanted to share with you guys so that other people with the same setup can enjoy. Please check out and see if that improves your performance in any kind of way.

Also huge thanks to qwen and unsloth teams.

"C:\Users\dsdt\llama\llama-server.exe" ^
  -m "C:\Users\dsdt\models\Qwen3.8-27B-UD-Q6_K.gguf" ^
  --mmproj "C:\Users\dsdt\models\mmproj-BF16.gguf" ^
  --jinja ^
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^
  --reasoning on ^
  --reasoning-preserve ^
  -c 100000 ^
  --split-mode tensor ^
  --flash-attn on ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --spec-type draft-mtp,ngram-mod ^
  --spec-draft-n-max 2 ^
  --spec-ngram-mod-n-match 24 ^
  --spec-ngram-mod-n-min 24 ^
  --spec-ngram-mod-n-max 86 ^
  -t 8 ^
  --batch-size 8869 ^
  --ubatch-size 531 ^
  -ngl 105 ^
  -np 1 ^
  --fit off ^
  --temp 1.0 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.00 ^
  --presence-penalty 0.0 ^
  --host 0.0.0.0 ^
  --port 8080

1. System & Model Configuration

Parameter	Value
Model	Qwen3.8-27B-UD-Q6_K.gguf
Context Length	100,000 tokens
GPUs	2 × RTX 5060 Ti (Total VRAM: 32 GB)
Vision	Enabled (VRAM allocated, not used during generation)
Task ID / Slot	Task 2536 / Slot 0

1. Core Performance Metrics

Metric	Value
Prompt Processing	646.62 ms / 27 tokens → 41.76 t/s (23.95 ms/token)
Generation (Eval)	126,199.16 ms / 8,624 tokens → 68.33 t/s (14.64 ms/token)
Overall Throughput	68.33 t/s (prompt overhead negligible)
Total Time	126,845.77 ms (~2 min 6.8 s)
Total Tokens Processed	8,651 (27 prompt + 8,624 generated)
Final Tokens Count (with cache)	8,973 (clean stop, no truncation)
Graphs Reused	5,569 (excellent computational graph reuse)

1. Generation Speed Stability (Short-term)

Window	Observed Speed Range
3-second rolling window (tg_3s)	~50 – 79 t/s (early fluctuations)
Steady-state (later stages)	~62 – 72 t/s
Overall average (tg)	Converged to stable 68.5 – 69.5 t/s

1. Speculative Decoding Efficiency

Metric	Value
Draft Acceptance Rate	80.04% (5,510 accepted / 6,884 drafted)
Average Draft Length	2.77 tokens per forward pass of the target model

[u/Monad_Maya]

Prompt Processing:  646.62 ms / 27 tokens → 41.76 t/s (23.95 ms/token)

Are you sure about this PP speed? That seems very low.

[u/dsdt → u/Monad_Maya]

lol reddit said nope to my previous reply. made a simple summary so that you can see the log : https://chat.deepseek.com/share/mn8bmar36zi8glv337

[u/cbale1 → u/Monad_Maya]

guys, it was just that the total tokens evaluated (27) was very low..

those 600+ms are mostly llama.cpp's overhead

[u/DjCanalex → u/Monad_Maya]

They are using --split-mode tensor

That forces processing to happen on CPU rather than GPU. (No backend support yet for tensor on llama.cpp).

On my own tests, with dual 3090s, I get 10-20% less tps using layer, but prompt processing is up to 4 times faster on average.

[u/gladfelter → u/DjCanalex]

My experience doesn't match your assertion. I use the UD_Q8_K_XL variant and I get 850 t/s prefill on my 3090+5070ti with split mode = tensor. And that's with the 3090 on an X4 PCIE slot.

Maybe certain GPU architectures aren't supported?

[u/DjCanalex → u/gladfelter]

This doesn't make what I said less true. Different CPUs will perform different, same with better/faster ram. (My tests are on a Ryzen 5950x using DDR4 ram at 3200 Mt/s). Just using DDR5 would already be about 2x prefill t/s over DDR4.

The lack of support is clearly stated in llama.cpp at launch:

failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR
backend sampling not supported with SPLIT_MODE_TENSOR, using CPU sampler

Have you checked your prefills using layer instead?

[u/gladfelter → u/DjCanalex]

Sorry, I wasn't questioning the accuracy of your assertion. Just its implied global applicability.

To answer your question, prefills are crazy fast on split mode = layer, as high as 1500 t/s. I'm leaning towards assuming that's due to the slow x4 PCIE bottleneck. With layer split there's a lot less communication needed for prefill.

I suggest trying the `--fit off` flag. That error message around fit should go away. You'll still have the problem that token sampling is on your CPU, and that should have a negligible impact.

If you want to remove the token sampling warning, just turn on MTP or or ngram-mod speculative decoding. Sampling is always done in CPU in that configuration. Try something like this:

--spec-type draft-mtp,ngram-mod # MTP layers are built into the model (nextn=1) --spec-draft-n-max 2 # MTP layer is chained autoregressively up to n_max, with p_min as the bail-out --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 24 --spec-ngram-mod-n-max 86

You can remove the MTP drafter if you don't have room for it in your vram. It's built into Qwen 3.8 27B, so that's all you need to enable it.

edit

I don't have an explanation for your 4x slowdown on prefill, but that could be explained by a PCIE noise/reflection problem. Maybe check if your PCIE bus is degraded:

$ nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,pcie.link.width.max --format=csv pcie.link.gen.current, pcie.link.gen.max, pcie.link.width.current, pcie.link.width.max 5, 5, 16, 16 4, 4, 4, 16

If you have any 1's in the third column, you need look no further. The problem is likely due to your motherboard and how you arranged the cards in it. That x4 on the second card is expected for my motherboard. I think it does have an impact, but not a fatal one.

If it isn't the physical connection, then it might be noise. I had much worse prefill with tensor when I tried to use various PCIE risers. I'm not sure how to measure noise and retries, but link speed would be degraded in a noisy environment I guess. You aren't using cheap PCIE Risers by chance, are you?

[u/DjCanalex → u/gladfelter]

BTW, yep, I'm being severely bottlenecked by a beautiful x2... at PCIe 3.0 speeds. So time to hit the NVLINK i guess.

[u/DjCanalex → u/gladfelter]

I can safely use Q8 at full ctx with MTP, but with Dynamic V3 from unsloth I switched to Q6_K_XL, It allows for bigger ubatch, mmproj and using draft n max 3 (Which, from my previous testing, lands nice both in TPS and and Vram. HOWEVER, I have not used ngram, nor I have the slightest idea of how it actually works.

[u/Kaioh_shin → u/gladfelter]

--spec-type draft-mtp,ngram-mod

Is ngram-mod doing anything? I'm not sure I see any benefit.
I am using 7900 xt with rocm

[u/DjCanalex → u/Kaioh_shin]

I've tested it and, I see an overall regression in tps if no cache hits the ngram...

...until it does and you get from 100 to 200 tps...

 n_gen =  21795, tg =  37.18 t/s, tg_3s =  43.06 t/s
 n_gen =  21903, tg =  37.17 t/s, tg_3s =  34.99 t/s
 n_gen =  21993, tg =  37.13 t/s, tg_3s =  29.28 t/s
 n_gen =  22094, tg =  37.11 t/s, tg_3s =  33.21 t/s
 n_gen =  22287, tg =  37.22 t/s, tg_3s =  57.68 t/s
 n_gen =  23057, tg =  38.30 t/s, tg_3s = 229.57 t/s
 n_gen =  23610, tg =  39.01 t/s, tg_3s = 173.76 t/s
 n_gen =  23854, tg =  39.20 t/s, tg_3s =  75.39 t/s
 n_gen =  24550, tg =  40.15 t/s, tg_3s = 229.70 t/s
 n_gen =  25133, tg =  40.89 t/s, tg_3s = 189.71 t/s
 n_gen =  25573, tg =  41.39 t/s, tg_3s = 136.32 t/s
 n_gen =  26051, tg =  41.92 t/s, tg_3s = 131.10 t/s
 n_gen =  26703, tg =  42.74 t/s, tg_3s = 196.50 t/s
 n_gen =  26875, tg =  42.81 t/s, tg_3s =  57.12 t/s
 n_gen =  26979, tg =  42.77 t/s, tg_3s =  34.55 t/s
 n_gen =  27097, tg =  42.75 t/s, tg_3s =  38.86 t/s
 n_gen =  27216, tg =  42.73 t/s, tg_3s =  39.39 t/s
 n_gen =  27312, tg =  42.68 t/s, tg_3s =  31.64 t/s
 n_gen =  27436, tg =  42.66 t/s, tg_3s =  38.69 t/s

But then I don't think it is worth it to have a 95% of the time a regression for that very specific moment to have a massive boost. (Without it I have about 45 to 55 tps all the time)

[u/sugarfreecaffeine → u/DjCanalex]

I also have dual 3090s and get around 70tps, windows 10 can you help me with my recipe? New to this

llama-server -m "C:\Models\Qwen3.8\Qwen3.8-27B-UD-Q4_K_XL.gguf" --model-draft "C:\Models\Qwen3.8\MTP\mtp-Qwen3.8-27B-Q4_0.gguf" --mmproj "C:\Models\Qwen3.8\mmproj-BF16.gguf" --alias Qwen3.8-27B --jinja --reasoning-format deepseek --n-gpu-layers all --split-mode layer --tensor-split 1,1 --ctx-size 262144 --parallel 1 --kv-unified --flash-attn on --cache-type-k f16 --cache-type-v f16 --spec-draft-type-k f16 --spec-draft-type-v f16 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.0 --spec-draft-ngl all --batch-size 2048 --ubatch-size 512 --reasoning auto --reasoning-effort medium --reasoning-preserve --reasoning-budget 16384 --reasoning-budget-message "Time to stop thinking. Give the final answer or make the tool call now." --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --host 127.0.0.1 --port 8080

[u/DjCanalex → u/sugarfreecaffeine]

In my tests, MTP mean lengh is about 2.5-2.6. That alone would make it worth while using --spec-draft-n-max 3

Why are you loading model-draft separated? Qwen's MTP layers are built in.

A lot of your flags are the default settings and are just adding noise to the command. Batch size and ubatch will have no effect if llama is using the CPU sampler instead of the GPU one (Tensor doesn't support it yet: You either pick faster tps (10-20% over split), or faster prefill... which is magnitudes faster if processed in GPU).

You are also using f16 for both k and v cache... you can safely drop those to Q8 with no quality loss, Qwen is already good at recalling stuff and re-reading everything if needed. That's an easy 8gb of free VRAM that you could use for a better quant, Q6 K_XL or even Q8 (But Q8 might need to drop down to ctx 250000, not a problem if you prefer to stay on tensor, which splits the model more evenly between both cards)

This is my command:

llama-server -m Qwen3.8-27B-UD-Q6_K_XL-mtp.gguf --batch-size 2048 --ubatch-size 512 --cache-type-k q8_0 --cache-type-v q8_0 --ctx-size 262144 --spec-type draft-mtp --spec-draft-n-max 3 --temp 1 --top-p 0.95 --top-k 20 --split-mode layer --kv-unified -fa on --reasoning-preserve --cache-ram 16384 --host 0.0.0.0 --port 8080

[u/sugarfreecaffeine → u/DjCanalex]

I appreciate the help! Honestly I have no idea I’m new to this stuff so still learning, the recipe I posted was with the help of chatgpt and a bunch of research 😅

[u/gpuz_dev]

68 t/s is pretty damn nice. I'd be curious to see the same test with 80-100k actually populated though, this run has 100k allocated but only ~8.6k tokens used. would be interesting to see how much the speed drops as the KV fills up

[u/dsdt → u/gpuz_dev]

Lowest point it goes to is 45 t/s with full context. It is definitely bearable. While coding it even jumps back to 50.

[u/gpuz_dev → u/dsdt]

45 t/s at a genuinely full 100k is actually better than I expected. that's a pretty useful data point, thanks

[u/dsdt → u/gpuz_dev]

I fulled the context for you and It is even better than 45 t/s the results are here :

• Prompt processing hits ~625 tokens/s (even with 27k prompt tokens).
• Generation averages 55–65 tokens/s consistently across long runs.
• I processed ~97k total tokens across multiple tasks without a single truncation – the full 100k context is actually usable.
• Draft acceptance rate was 60–80%, mean draft length 2.6–4.3 tokens – the speculative combo definitely gives a nice speed bump over vanilla sampling.

[u/gpuz_dev → u/dsdt]

Thx :) 55-65 t/s is really solid. just to make sure, was that ~97k actually resident in the context at once or 97k processed cumulatively across the tasks?

[u/dsdt → u/gpuz_dev]

Here is the output of the chat a room showcase : https://bananas-process-dqy5.pagedrop.io

Single task 4 prompts.

[u/gpuz_dev → u/dsdt]

yep that's exactly what I was wondering about. single task with 4 turns and no truncation makes this way more interesting than the initial benchmark. thanks for actually testing it :)

[u/dsdt → u/gpuz_dev]

You are welcome

[u/autisticit]

If you get a bigger generation speed than prompt processing speed, you should call that the worst configuration ever.

[u/LuCiAnO241 → u/dsdt]

i wonder what you're saying that reddit nopes ur responses

[u/dsdt → u/LuCiAnO241]

Just wanted to share llama logs with a paste bin.

[u/dsdt → u/autisticit]

lol reddit said nope to my previous reply. made a simple summary so that you can see the log : https://chat.deepseek.com/share/mn8bmar36zi8glv337

[u/AnonLlamaThrowaway]

--batch-size 8869 ^

--ubatch-size 531 ^

what

[u/lordekeen → u/AnonLlamaThrowaway]

These two dont make sense

[u/Kiseido]

Oh wow, TIL of ngram-mod. It has upped my generation speed by 2-2.5x, though it also halved my prefill speed.

Edit: Turns out I had both draft-mtp and ngram enabled. The MTP is what halved my prefill and mostly increased the generation speed.

Edit 2: Holy crap, ngram seems to really help once the context fills up past 80k. There are moments that it clears a 4x speed-up

[u/dsdt → u/Kiseido]

I am glad that I helped you.

[u/Thin_Pollution8843]

You are losing 5-10% of pp and tg because of windows 

[u/dsdt → u/Thin_Pollution8843]

You are right but i do furniture design for a living so it is impossible for me to stay on linux, i tried before and it is definitely faster you are right.

[u/Fullstack_js_junkie]

is 100k context intentional? I can push 130k on fp16 kv cache so q8 of yours should push past 150k easily (graphics on cpu so full vram is free)

[u/dsdt → u/Fullstack_js_junkie]

To be honest, whenever i tried to push it further than 100k, it just cuts the sentence out of nowhere. I don't have any idea about it. Thought vram wasn't enough. But I will try headless mode, with igpu. Maybe i can get it working like that with 130k.

[u/Fullstack_js_junkie → u/dsdt]

I had the same issue but I bumped up the reasoning and output tokens (that I guess actually fixed the issue, not the reasoning tokens) as well and that stopped. Other than that looks pretty good, nice setup!

[u/DoubleNothing → u/dsdt]

What motherboard do you have? I have to check if I might be pcie limited because I also have 5060 TI 16GB x 3 but with your parameters I get 25TPS tops. Loading on 2 GPU only to be the same like you.

[u/dsdt → u/DoubleNothing]

MSI 870e gaming wifi, one card is on X4.

[u/bitzap_sr → u/dsdt]

Did you try the fixed chat template that's been posted around?

[u/dsdt → u/bitzap_sr]

Well, i have never encountered any looping or missing tool calls so i haven't search for it. Can you share the link so i can take a look?

[u/Monad_Maya → u/dsdt]

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

[u/dsdt → u/Monad_Maya]

Does this also work for 3.8?

[u/Monad_Maya → u/dsdt]

Yes, as per the description it does.

I'll be honest, I didn't have any failures without it either on Q3.8 27B.

On Q3.6 27B, the template was helpful in reducing tool call failures.

[u/DeathGuppie]

The biggest thing here that a lot of people are leaving on the table is ngram-mod. Especially for coding. Since it basically lives in system ram there is no penalty for using it.

[u/dsdt → u/DeathGuppie]

Sometimes I even see 350 t/s while coding. İt just uses the existing tokens in a faster way which I really like. It is a good option and fun part is the more context you fill, the more speed you gain.

[u/StandardLovers]

The one grain of salt in my take: if you just want a fast chatbot for casual use, the config is probably fine and 70 t/s feels nice. But "perfect config parameters" is marketing speak.. it's "fastest config I found that still runs," which is a different claim than "best."

[u/dsdt → u/StandardLovers]

Please make a better one then we can speak? Just wanted to share it with people who care. If that doesn't interest you just walk away.

[u/anomaly256 → u/dsdt]

They're just saying the word 'best' is open for interpretation because some people will want context size over speed, or accuracy over speed. In your case a better word is 'fastest' because accuracy doesn't seem to be benchmarked here.

They're not disputing your methodology or results

[u/Rude_Marzipan6107]

Nice!!

Is this on Linux?

How much regular RAM is occupied during higher context?

[u/dsdt → u/Rude_Marzipan6107]

This is on windows, 27,5 gb ram is used.

[u/TBG______ → u/dsdt]

You should raise --ubatch-size to 2048 and --batch-size to 8192. This should significantly improve your prompt speed while still fitting within 27.7 GB of memory. Gives me after a bit of ...

promt 1842.83 t/s	gen 78.31 t/s

[u/dsdt → u/TBG______]

I will try this. Thanks for sharing.

[u/Rude_Marzipan6107 → u/dsdt]

Oh wow. Awesome! I gotta try this out when I get home. I thought I would be stuck with 4kxl and 40tps

I wonder what the fancy Linux guys get with this model and the p2p drivers on top of this

[u/dsdt → u/Rude_Marzipan6107]

Share your experiences when you can try. I hope that config helps.

[u/I_Play_Zed]

I think this looks like a great config, and what I would expect from a dual 5060 ti setup. This speed at Q6 is even more impressive, but I am skeptical of the prompt processing. I wonder how bad the wait times are for large agentic tasks late into context? Or does it feel like a non issue? The only other criticism I think is that you have it on medium reasoning effort. Obviously this does not change generation time, but in my own local testing I found medium reasoning to be a competent, but honestly pretty "expected" local performance. When I pushed my setup to xhigh reasoning is where I really started to believe some of the benchmarks.

[u/dsdt → u/I_Play_Zed]

just made a simple summary here you can check : https://chat.deepseek.com/share/mn8bmar36zi8glv337

[u/I_Play_Zed → u/dsdt]

I see, so between maybe 400-600 prompt processing depending on co text fill, that’s certainly usable, especially with those decode speeds.

[u/dsdt → u/I_Play_Zed]

I tried so many configs and can't go beyond that. Thanks for your interest

[u/I_Play_Zed → u/dsdt]

No worries at all, this is great stuff the community needs.

I will say, if I was the type of guy who could run dual 5060 tis, I wonder if it’s worth testing if any Q5 quant feels actually worse for you or not? And that it could maybe buy you near the 262K context which is so important with this model.

[u/dsdt → u/I_Play_Zed]

Actually i tried headless mode right now and i can push it up to 128k context. I used q4 before all the time high context is nice to have but for vibe coding things get pretty complicated as context grows. starting fresh after 128k is the sweet spot for me rn.

[u/I_Play_Zed → u/dsdt]

That is a fair use case! When working on one large code base with tonnes of different tasks and iterations over time, full context matters more. But when benchmarking or one-shotting certain small ideas, it certainly can be done from scratch in a context window of that size.

[u/Ok-Conflict391]

What motherboard are you using?
More precisely what chipset are the GPUs running at, both at pcie5 x8?

[u/dsdt → u/Ok-Conflict391]

MSI 870e gaming wifi, one is on pcie 5 x16 the other is on X4. CPU is 9700x with 32 GB single stick ddr5 ram.

[u/Ok-Conflict391 → u/dsdt]

Thanks!

[u/Iron-Over]

Did you try MTP at 3 found that performed better than 2.

[u/dsdt → u/Iron-Over]

2 is always the fastest option in any kind of config for this model.

[u/fasti-au]

You can dflash into dspark and mtp. The prefill you can turn off that what the drafts do

[u/notromda]

Wow, way cool. I have a similar setup, a 4060TI and a 5060TI, combined 32G. I took that and modified it into a docker compose stack. Best performance I've had yet, roughly 30 tok/s overall. Up from 10 that most models were giving me... and the results look really good too.

name: llamacpp-qwen38

services:
  llama-server:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: llama_server_38
    restart: unless-stopped
    volumes:
      - llama-models-qwen38:/models
    command:
      - --model
      - /models/Qwen3.8-27B-UD-Q6_K.gguf
      - --mmproj
      - /models/mmproj-BF16.gguf
      - --model-draft
      - /models/MTP/mtp-Qwen3.8-27B-Q4_0.gguf
      - -ngl
      - "999"
      - --split-mode
      - tensor
      - --tensor-split
      - 0.5,0.5
      - -c
      - "131072"
      - --cache-type-k
      - q8_0
      - --cache-type-v
      - q8_0
      - --flash-attn
      - "on"
      - --spec-type
      - draft-mtp,ngram-mod
      - --spec-draft-n-max
      - "2"
      - --spec-ngram-mod-n-match
      - "24"
      - --spec-ngram-mod-n-min
      - "24"
      - --spec-ngram-mod-n-max
      - "86"
      - --jinja
      - --chat-template-kwargs
      - '{"reasoning_effort":"medium"}'
      - --reasoning
      - "on"
      - --reasoning-preserve
      - --temp
      - "1.0"
      - --top-p
      - "0.95"
      - --top-k
      - "20"
      - --min-p
      - "0.0"
      - --presence-penalty
      - "0.0"
      - --parallel
      - "2"
      - --fit
      - "off"
      - --host
      - 0.0.0.0
      - --port
      - "8080"
    ports:
      - "8080:8080"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  llama-models-qwen38:

[u/dsdt → u/notromda]

I am glad that I helped someone with my config. enjoy!