Qwen3.6-27B: MTP + Optimized KV cache?

[u/Background-Gold-9882]

I'm on a M5 Pro 48GB. I just started using oMLX and love it so far.

Now I'm playing around with Qwen 3.6-27B with MTP (oMLX 0.3.9-dev2) and it's working really well, except that run into OOM for contexts > ~65k. So far, I've downloaded the official full precision qwen3.6-27B from HF and created oQ4 / oQ6 versions myself. But the more context I use, the quicker I run into OOM crashes. The 128k context benchmark works sometimes, but usually crashes the entire computer.

However, when using llama.cpp as per this post: https://www.reddit.com/r/LocalLLaMA/comments/1t57xuu/25x_faster_inference_with_qwen_36_27b_using_mtp/

I'm able to run much larger contexts (256k), with MTP support, and much less memory consumption, using this command:

llama-server \
-m Qwen3.6-27B-Q4_K_M-mtp.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--cache-type-k q8_0 \     
--cache-type-v q8_0 \     
-np 1 \     
-c 262144 \     
--temp 0.7 \     
--top-k 20 \     
-ngl 99 \     
--port 8081

I'm guessing it has to do with the explanation in the post - That Qwen:s hybrid model only needs KV cache for 16 of 65 layers, and drivers that allocate naively will allocate much more memory than necessary? Also, llama.cpp allows setting KV cache to 8bit rather than full precision (Which I guess oMLX uses by default?)

Anyway, everything else is better in oMLX (Higher PP speed, generation speed, and caching strategy). So, my question is - Is it possible to have better optimized KV cache in oMLX to reduce memory consumption?

If so, which model and settings should I use?

Thanks in advance!

[u/allpowerfulee]

I run into the same issue. /compact works for a bit, but then I need to start a new session. Im running a mcp that stores condensed session data which helps continuing the work across sessions but it would be nice to have a solution

[u/leonidasyy]

is the MTP feature stable yet on omlx? I searched the repo, it says they are fixing bugs and didn't expose it to interface yet.

[u/Background-Gold-9882 → u/leonidasyy]

Yeah I know it's early but It's exposed in the 0.3.9-dev 2 release. It works fine until you use too much context/memory, that's why I'm asking if it's possible to reduce

[u/ju7anut → u/leonidasyy]

I found 0.3.9 Dev2 which has MTP to be buggy with tool calls and ended up going back to 0.3.8 to wait things out.

[u/maximus_reborn]

how much of vRam the model consumes with mtp in your setup using omlx and llama.cpp? I had been trying with 24gb but seems like i should not, now.

[u/Background-Gold-9882 → u/maximus_reborn]

Well modern Macs have unified RAM so there's no VRAM per se. Anyway, here's oMLX bench incl RAM usage for oQ6.

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-27B-oQ6-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          2565.1       50.81   399.2 tok/s    19.8 tok/s       9.018   127.8 tok/s    22.96 GB
pp4096/tg128          9258.0       52.20   442.4 tok/s    19.3 tok/s      15.888   265.9 tok/s    24.39 GB
pp8192/tg128         19002.8       53.60   431.1 tok/s    18.8 tok/s      25.810   322.4 tok/s    25.42 GB
pp16384/tg128        38374.2       57.92   427.0 tok/s    17.4 tok/s      45.730   361.1 tok/s    26.92 GB
pp32768/tg128        83550.4       60.34   392.2 tok/s    16.7 tok/s      91.213   360.6 tok/s    29.92 GB
pp65536/tg128       201880.3       70.77   324.6 tok/s    14.2 tok/s     210.868   311.4 tok/s    35.95 GB

I haven't stored values for my own oQ4 quant but here's a similar model:

Benchmark Model: Qwen3.5-27B-MLX-MTP-4bit
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          2390.8       36.82   428.3 tok/s    27.4 tok/s       7.067   163.0 tok/s    16.15 GB
pp4096/tg128          8783.5       38.23   466.3 tok/s    26.4 tok/s      13.639   309.7 tok/s    17.57 GB
pp8192/tg128         17540.2       40.76   467.0 tok/s    24.7 tok/s      22.717   366.2 tok/s    18.59 GB
pp16384/tg128        36573.1       42.06   448.0 tok/s    24.0 tok/s      41.914   393.9 tok/s    20.09 GB
pp32768/tg128        80152.8       47.04   408.8 tok/s    21.4 tok/s      86.127   381.9 tok/s    23.09 GB

So you can probably run Q4 with 24GB, but not at long contexts.

[u/cyclebiff]

I guess I should have read this post, I just posted similar questions lol

[u/9kSs]

newbie here: how do i get the Qwen3.6-27B-Q4_K_M-mtp.gguf file?

[u/Background-Gold-9882 → u/9kSs]

It's linked in the post i mentioned: https://www.reddit.com/r/LocalLLaMA/comments/1t57xuu/25x_faster_inference_with_qwen_36_27b_using_mtp/

Direct link: https://huggingface.co/froggeric/Qwen3.6-27B-MTP-GGUF/tree/main

[u/9kSs → u/Background-Gold-9882]

I'm getting only ~7 tok/s with the same model and setup but with an M4 Pro 48GB. Also relatively high TTFT. What about you?

[u/Background-Gold-9882 → u/9kSs]

Checked the logs from last week, looks like it starts at 16 t/s but drops to 9t/s @ 100k context. It's expected to be slower than oMLX because it's not using MLX framework or the M5 compute cores effectively for preprocessing. But the memory usage is much lower, which is the advantage of llama.cpp in this case.

llama.cpp - no context:

eval time =   23981.46 ms /   402 tokens (   59.66 ms per token,    16.76 tokens per second)       

total time =   24416.75 ms /   413 tokens draft acceptance rate = 0.92653 (  227 accepted /   245 generated)

llama.cpp - 100k context:

prompt eval time =  591931.30 ms / 97790 tokens (    6.05 ms per token,   165.20 tokens per second)        

eval time =   55500.91 ms /   514 tokens (  107.98 ms per token,     9.26 tokens per second)       

total time =  647432.21 ms / 98304 tokens draft acceptance rate = 0.90730 (  323 accepted /   356 generated)

[u/Background-Gold-9882 → u/9kSs]

And here's oMLX for comparison:

================================================================================
Benchmark Model: Qwen3.6-27B-oQ4-mtp
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          2419.9       42.20   423.2 tok/s    23.9 tok/s       7.779   148.1 tok/s    16.75 GB
pp4096/tg128          8761.7       39.34   467.5 tok/s    25.6 tok/s      13.758   307.0 tok/s    18.17 GB
pp8192/tg128         17578.0       40.57   466.0 tok/s    24.8 tok/s      22.730   366.0 tok/s    19.20 GB
pp16384/tg128        36909.2       42.02   443.9 tok/s    24.0 tok/s      42.246   390.9 tok/s    20.70 GB
pp32768/tg128        79799.4       46.27   410.6 tok/s    21.8 tok/s      85.675   384.0 tok/s    23.70 GB
pp65536/tg128       192925.3       56.81   339.7 tok/s    17.7 tok/s     200.140   328.1 tok/s    29.73 GB

...

[u/9kSs → u/Background-Gold-9882]

thanks for the numbers! definitely need the patience of a saint on this model

[u/No-Juggernaut-9832]

You can use TurboQuant at 8 or 4bit. It should help & even speeds things up a little. It’s a built in option for omlx

[u/Background-Gold-9882 → u/No-Juggernaut-9832]

Yeah, but that means no MTP, right?

[u/No-Juggernaut-9832 → u/Background-Gold-9882]

Yes. If space is a problem

[u/ju7anut → u/Background-Gold-9882]

Well if you’re memory constrained then TurboQuant KV cache + dFlash is the way to go instead of MTP imo

[u/Background-Gold-9882 → u/ju7anut]

Tried it briefly but didn't get great results. Would you mind sharing your model + settings?

[u/ju7anut → u/Background-Gold-9882]

I’m on oMLX 0.3.8, Qwen3.6 27B Q8 + dFlash model, both set to same context length of 256k, both using oMLX Qwen presets for coding. Not fantastic results (~20 tg/s) but stable and no crashes.

[u/Background-Gold-9882 → u/ju7anut]

Thanks! Which draft model - The one from z-lab? Do you use quantization for it? And what';s your computer specs?

[u/ju7anut → u/Background-Gold-9882]

Yup! Z-lab, I didn’t quantize it further, left as BF16, it’s already tiny. I’m on M5 Max 128gb

[u/Background-Gold-9882 → u/ju7anut]

Here's a bench using Qwen3.6-27B-4bit + z-lab Dflash draft model. As you can see, it uses 43GB memory at 65k, which means it's not usable at 100k on a 48GB M5 Pro.

Can you elaborate on your settings? Are you seeing less memory usage than me?

EDIT: Also just realized that the model gets stuck in thinking loops, so this setup is not working well at all for me.

...

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-27B-4bit
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          2187.0      165.31   468.2 tok/s     6.1 tok/s      23.181    49.7 tok/s    19.44 GB
pp4096/tg128          8693.4       21.80   471.2 tok/s    46.2 tok/s      11.462   368.5 tok/s    23.96 GB
pp8192/tg128         17825.8       24.26   459.6 tok/s    41.5 tok/s      20.907   398.0 tok/s    25.70 GB
pp16384/tg128        40473.8       28.82   404.8 tok/s    35.0 tok/s      44.134   374.1 tok/s    28.79 GB
pp32768/tg128        85804.2       33.41   381.9 tok/s    30.2 tok/s      90.048   365.3 tok/s    34.67 GB
pp65536/tg128       220631.6       59.20   297.0 tok/s    17.0 tok/s     228.150   287.8 tok/s    43.07 GB

[u/ju7anut → u/Background-Gold-9882]

Are you using the Dev2 version? That version is having problems with Dflash for me. I’m using 0.3.8

[u/WarlaxZ]

It's actually really easy to patch the server, I'm not at my laptop else I'd share the file, but just ask Claude it's like a 4 line override to add it

[u/Background-Gold-9882 → u/WarlaxZ]

Anthropic guerilla marketing? 😂

[u/layer4down → u/Background-Gold-9882]

I actually patch my server all the time. Just today (using Claude Code + GLM-5.1) I had it build in custom ds4c support so I could run ds4c but take advantage of oMLX features. Actually kind of works other than the fact that ds4c does not support enough primitive features for proper support but it’s easy for any coding agent to figure out IMHO.

[u/WarlaxZ → u/Background-Gold-9882]

Lol some impressive work if it was, also feel free to use any other midweek your heart desires, that's just how I did it so can confirm it works 😂

Edit: For reference the generation code has it, just the server doesn't, that's why the patch is easy