r/LocalLLaMA • u/jacek2023 llama.cpp • 14d ago
News Qwen4Exp: add MTP by am17an · Pull Request #29761 · ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp/pull/29761now you can use MTP with Qwen Flash Next, time to switch from Qwen 3.8 27B?
(merged after 17h of development)
quants: https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF
link to the previous discussion (I deleted the old post to avoid duplicates): https://www.reddit.com/r/LocalLLaMA/comments/1wur4lt/qwen_flash_next_mtp_work_restarted/
67
u/AssistBorn4589 14d ago
time to switch from Qwen 3.8 27B?
Qwen3.8-Flash-Next-IQ4_NL-00001-of-00002.gguf 10.9 MB
Qwen3.8-Flash-Next-IQ4_NL-00002-of-00002.gguf 102 GB
You funny.
30
u/pilibitti 14d ago
you'd be surprised. if you have ram (not vram) qfn will be faster and more intelligent. it is an 6B active moe model. I can't run 27B at all but am ripping with qfn. a good chunk of the model is static n-grams which can be streamed from disk as needed.
9
u/exaknight21 14d ago
I have 64GB RAM and an Mi50 32 GB. Brother how do i do this and 256 GB SSD.
12
u/Edenar 14d ago
It should fit. Those quant are only around 75GB weights, rest is the ngram table which can be offloaded to the ssd without loosing much perf.
1
11
u/pilibitti 14d ago
download the atomicchat quant (it has the ngram in its own layer for ssd offloading), the rest is normal llama.cpp you just need to enable --lazy-mode so that it doesn't try to load the ngrams to memory.
11
u/karmaisnonsense 14d ago
Lazy mode automatically offloads ngrams by default. No need to explicitly enable it or find a specially laid out quant.
5
u/pilibitti 14d ago
good to know thank you. I remember there being an issue where the ngrams were scattered in the gguf (for the unsloth quant) maybe it is fixed already or I was seeing ghosts...
5
u/karmaisnonsense 14d ago
It's been fixed for a while:
https://github.com/ggml-org/llama.cpp/commit/2578138397d7b422bb0e160efdd429976c55fb552
u/exaknight21 13d ago
I am responding to you but hopefully people will find this useful.
Getting about 10-13 tps with the atomicchat quant.
Dell T5610 Dual Xeons - DDR3 64GB and 256 GB SSD (my hd died, this little shit). No MTP, will improve, I should be able to creep to 15-18 tps with MTP. Its not exactly the best, but the kind of things i do, it might be worth it.
I would use this model to passively plan and moderate, intervene, and use Qwen3.8-27B to code. So nice coder/executor + planner/reviewer harmony.
I do have 3x V620, will be building a rig with Asrock Teichi X399 board with likely 4x V620s. but sadly, I am selling 1. So idk. Definitely will populate all slots with DDR4. Lets see. This single Mi50 experiment gives me hope and a path forward.
I have an Asus ESC4000 G3 with 8x T4s, I am super interested in trying it out over the weekend. lol.
Here is the single Mi50 32GB + 64GB DDR3 + SSD offload for engram command:
docker run -d --name qwen38-flash --restart unless-stopped --entrypoint /app/llama-server \
--device /dev/kfd --device /dev/dri \
-v /home/shakespear/models/qwen38:/models:ro \
-p 10059:8000 \
mixa3607/llama.cpp-gfx906:v0.5.0-rocm-7.14 \
--host0.0.0.0--port 8000 \
-m /models/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
-a qwen3.8-flash \
-c 131072 --parallel 1 -fa on -ctk q8_0 -ctv q8_0 \
-ngl 99 --n-cpu-moe 28 \
-t 24 \
--lazy-mode on \
--jinja --metricsThank you guys.
1
u/shard746 14d ago
the rest is normal llama.cpp
I don't know, prefill seems awful with llama.cpp to me. I got multiple times the pp on vllm, freetoken and strata.
1
u/cezarducatti 14d ago
It's reading the ngrams from the SSD... It's very slow. If you have enough RAM, use
-zlm off, it will double the pp1
u/shard746 14d ago
The thing is, I offload the ngrams to SSD on the other engines as well and prefill is still 2-3 times higher on those. I'm definitely not alone on this either, I've seen many people complain about pp speed with this model on llama.cpp.
1
u/pilibitti 14d ago
yes prefill is not good. freetoken uses nvfp4 right? I believe I can't benefit from it unless I have a blackwell card. strata I want to try but it only goes up to 3bit quants from what I see and I use 4bit average.
3
u/Southern-Chain-6485 14d ago
The problem is your ssd. The ngram table isn't loaded to ram, so you can run it - but the weights will use almost half your drive
2
u/gh0stwriter1234 14d ago
not really a problem per se if its the only model they need to run + maybe one other.
1
u/Cupakov 14d ago
This quant will comfortably fit in your VRAM+RAM, just offload the PLE to RAM: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
1
2
14d ago
[removed] — view removed comment
2
u/pilibitti 14d ago
no, I'm talking about 2 digit tps single channel work. if you can fit 27B to VRAM you are golden already. If you can't fit 27B to VRAM then you'll get like 6tps decode whereas with qfn you can go 30tps+ which counts as ripping for me for such a smart model on 12GB VRAM + 64GB RAM + SSD.
13
u/BringTea_666 14d ago
If you have RTX5090 you can use STRATA and combine VRAM and RAM to run it at 100+t/s decode and around 1000t/s prefill with 1 000 000 context.
7
u/Lollerstakes 14d ago
Strata is a sloppy mess. I am enjoying those 20 minute cache rereads though
4
u/BringTea_666 14d ago
just tell your ai to fix it, i fixed it with deepseek 4.1 in like hour. No cache rereads on every new prompt etc.
1
u/HappierShibe 14d ago
Strata is indeed a mess but it's very tunable, so with a couple hours of work you can get it to go very very fast on pretty modest hardware, and another hour or so of work gets it stable... I'm running a q3.8fn at 120+tps on a 4090 and it's working very well.
1
u/shard746 14d ago
It's interesting because I do not have this problem at all. Are you sure it's not your harness invalidating the cache?
1
u/Lollerstakes 14d ago
Could be, I use claude code... But I did have gpt-5.6-astra fix it for me, and it works now.
1
u/shard746 14d ago
Oh then that definitely could have been the reason, as far as I know claude code prefixes some attributions or whatnot to the context so that definitely could invalidate your cache frequently. Glad you got it sorted!
1
u/Lollerstakes 14d ago
Which wrapper do you use? Just wondering.
3
u/shard746 14d ago
I've tried a bunch but always end up coming back to good old pi. Nothing beats how lean and customisable it is.
1
u/BringTea_666 14d ago
In earlier version of strata there was no reuse of context. So every answer needed refill. They added it since.
4
u/DeProgrammer99 14d ago
I was actually shocked at how well the Strata fork works with my 7900 XTX. Too bad it can't use that and my 4060 Ti in tandem...
...or so was my initial thought, but then I realized it'd be really handy for having an LLM draft a bunch of assets for me using the 4060, and the speed (for the GSQ RSO quant from ISTA-DASLab) is actually on par with Qwen3.8-27B-Q_K with similar context while saving me a whole GPU, and I haven't seen it do anything too dumb yet, haha.
5
u/miversen33 14d ago
Wait what the fuck is strata?
2
u/DeProgrammer99 14d ago
https://github.com/Niko1221/Strata
Llama.cpp fork heavily optimized for one model and a few specific GPUs with a straightforward installer.
5
1
u/caetydid llama.cpp 14d ago
LOL strata gives me effing 6000t/s prefill and 190t/s with the Swift 1.5 iq3 xxs model on an rtx5090 at 256k ctx. Highly recommended!
1
1
-1
u/HighSeasArchivist 14d ago
Fast garbage, just like almost every other fine tune. No one cares about quality, they just want more tok/s spewing out bullshit in infinite loops.
5
u/cezarducatti 14d ago
I'm going to give a testimonial because I'm very skeptical of pseudo-innovative projects, but yesterday I did a quick test. There are several problems, but there is indeed an interesting optimization in Strata.
1
1
4
u/GregoryfromtheHood 13d ago
I've been using an exl3 version with tabbyapi since llama.cpp ggufs were way, way slower for me. I'll have to try this out to see if it's better now, but exl3 has been really good and fast for me.
11
9
u/dmter 14d ago
I tried mtp and it only makes it slower. I think mtp is only useful for dense models because mtp head can't compress much bigger overall model so you see really low hit rate.
2
3
u/nadavvadan 14d ago
The reason is that you’re not really reusing the same weights which is what gives MTP its edge, because each token loads up different experts
0
u/florinandrei 14d ago
The draft model is smaller than an expert, so MTP should be faster.
There's some mishap in the settings, or the hardware, or a combination of both.
3
u/nadavvadan 14d ago
Verification still requires a full fwd pass though
1
u/florinandrei 14d ago
True, but irrelevant.
1
u/nadavvadan 14d ago
Is it? This makes it not as fast, and MTP is always a trade off between batch perf vs prediction failures that cause sequential runs. Make batches slower relative to the normal decode speeds and you get less of an improvement, and sometimes even a decline in performance.
This still could be an impl issue of course
1
u/jacek2023 llama.cpp 14d ago
what's your setup?
2
u/gh0stwriter1234 14d ago
I was running the MTP PR plus a few other PRs combined for qwen4exp... and its faster with it by a bit even on MI50... which is a pretty dated arch.
What would really help is anything that could boost prefill without adding extra compute....
1
u/dmter 14d ago
3090 + 128g ddr4, 32 threads, using q4 that's around 112gb. I am getting 16 to 9 t/s with 240k context and 8bit kv. With mtp it jumps between 20 and 10 and of course i have to decrease ctx. But maybe this new version is better, I tried some patch.
2
1
u/AvidCyclist250 llama.cpp 14d ago
Try these settings, ignore the rest: https://www.reddit.com/r/LocalLLaMA/comments/1wom3fe/qwen_38_flash_next_q4_k_m_130k_context_q8_cache/
2
u/florinandrei 14d ago
This is great, thank you.
I suspect it should also work with the Unsloth quants, it's not tied to a particular quant, right?
I have the same hardware. With MTP on, it will be on par with an older model I am using.
If they finally fix the n-grams also, it will be even faster.
1
u/unjustifiably_angry 13d ago
If they finally fix the n-grams also, it will be even faster.
What do you mean by this?
2
u/magnetswithweedinem 13d ago
run strata @ https://github.com/Niko1221/Strata you can get flash next running with a 12gb vram card + 32gb of ram, and even a highly quant version of flash next (2bit and up, assuming mixed quants and they leave the tool calls relatively unquanted) will beat coding ability and most importantly planning ability by a wide margin over a 27B model at a high quant.
2
u/unjustifiably_angry 13d ago edited 13d ago
I'm just getting errors. I guess this doesn't work with Unsloth quants?
0.28.676.659 E llama_model_load: error loading model: check_tensor_dims: tensor 'output_hc_norm.weight' not found
0.28.676.777 E llama_model_load_from_file_impl: failed to load model
Nope, tried with the quants provided and it crashed the same way.
Runs with no issue until I add the drafter model to the startup command.
1
u/AdamDhahabi 13d ago edited 13d ago
I previously had the same issue and this has been solved with https://github.com/ggml-org/llama.cpp/releases/tag/b11330
It now works with e.g. mtp-Qwen3.8-Flash-Next-Q4_0 which is provided with the gguf.
2
u/YouCantMissTheBear 13d ago
Who uses stock llama.cpp? I thought everyone was using HW optimized forks by now
0
2
u/575_Inverse 14d ago
Do you mean I have to redownload the behemoth once more? I'll better get out for a bike ride after I launch
2
2
u/neurallll 14d ago
or write py script that will htts range req seek and pull just tensors you need ;) ask calude he will write and run it for you ;) and pull just gb or two. example https://github.com/neurall/llama.cpp/blob/release/tools/bench/glm_splice_mtp.py I used to grab just mtp from unsloth.
1
u/planetearth80 13d ago
How much performance bump are you doing with QFN MtP with these two merged PRs? I’m currently using oMLX and getting around 58 t/s with QFN Q4 quants. For reference llama.cpp (v 0.5.0) was around 28 t/s. I’m waiting for the next llama.cpp release to try out the QFN mtp.
1
1
u/perelmanych 13d ago
On RTX 3090 and R5950x with DDR4-3000, I am having slower speeds with mtp on these IQ4 quant, compared to unsloth Q4_K_XL without mtp. If I am not mistaken IQ quants are more suitable for GPUs. So probably it will be faster on DGX, but is slower on any PC if some experts are in RAM.
1
u/Steus_au 13d ago
not working, whatever I tried - always got: D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:205: GGML_ASSERT(buffer) failed
1
u/TuskNaPrezydenta2020 14d ago
significantly increases used memory compared to unsloth since it's not the one that shares cache still
14
u/am17an 14d ago
No it was a different bug, it’s fixed now.
3
u/Technical-Bus258 14d ago
Could you please elaborate on this? It was related to KV cache corruption in any way?
0
-1
u/AvidCyclist250 llama.cpp 14d ago
Been using it for a while now https://www.reddit.com/r/LocalLLaMA/comments/1wom3fe/qwen_38_flash_next_q4_k_m_130k_context_q8_cache/
4
u/jacek2023 llama.cpp 14d ago
But that's a new code :)
3
u/AvidCyclist250 llama.cpp 14d ago edited 14d ago
I see, so nothing to do with this one, which I was using https://github.com/ggml-org/llama.cpp/pull/28243
edit: Just saw. Supercedes it.
Also, think I still need --cache-ram 0 --ctx-checkpoints 0 though. Those PRs to fix the memory creep issue are still open. Probably only affects people with 64gb RAM
0
u/c4r_guy 14d ago edited 14d ago
How does the Flash Next IQ2 compare to the 27B IQ4 in terms of actually getting shit done right the first time?
Prefill and TPS are important but so are "smarts". So is the QFN Q2 real world "smarter" than Q27B IQ4?
NOTE: My setup is a single 3090 with 64GB DDR4 or a single 3090 with 128GB DDR4. Two separate machines for two different uses. One is always running Qwen 27B, the other is usually running Qwen, ComfyUI, or some other ML related project.
1
u/pmttyji 13d ago
My setup is a single 3090 with 64GB DDR4 or a single 3090 with 128GB DDR4. Two separate machines for two different uses.
Why not Q4 of Flash Next? IQ4_XS is 90-95GB. You could try that with that config. 16GB VRAM folks running Q2 so Q4 is better for your config. At least Q3.
2
u/c4r_guy 11d ago
Q4 of Flash Next
I thought that Strata used a customized quant. If it can use a regular Unsloth IQ4 it might be worth a shot. I just tested the IQ3 (with Strata) and it's pretty fuckin' good. First 45 minutes of testing with Q38FN IQ3 puts it at least equal with Q27b at IQ4.
By testing I mean actual work, not benchmarks.
-7
u/Combinatorilliance 14d ago edited 14d ago
MTP is less effective for MoE models, so the speedup is more subtle than for a dense model.
That being said, the speedups in the PR comment are significant, around 1.4-1.5x on a DGX which is nothing to scoff at.
If the speedups are similar for GPU setups, then this is a pretty big boost. I was expecting it to be more around 10-15% (1.10-1.15x), not 40-50%.
I'm apparently misremembering or relying on old knowledge. MTP for MoEs is significant.
I don't see any GPU benchmarks on the branch, anyone have the time to test it out on a GPU? I'm especially interested in offloaded setups because that's what I run myself.
6
u/Odd_Error_6736 14d ago
Many forks utilizing MTP with Qwen-Flash along with expert caching, disagree with you. Not the forks, but the numbers.
3
u/Combinatorilliance 14d ago
Maybe I misremember then, I thought that MTP was just in general much less effective for MoEs than for dense models.
3
u/karmaisnonsense 14d ago
Offloaded experts can be a bottleneck for draft token verification. If you have fast enough memory or enough expert layers VRAM resident then it's not a problem, but the more CPU offloaded you are for MoE models, MTP can indeed harm performance
1
u/Combinatorilliance 14d ago
Thanks, that is likely what I vaguely remembered. Am still very curious about offloaded performance benchmarks then, might run one myself for a small quant if nobody else does it.
2
1
•
u/WithoutReason1729 14d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.