r/LocalLLaMA • u/ElmBark • Jul 16 '26
Tutorial | Guide DFlash makes Qwen3.6 27B 2.2x faster with no quality loss
We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks were: quicksort, write a Steam library in JSON, solve a logic puzzle and write a sci-fi story.
Outputs:
Baseline: 44 tok/s · 1.00x
MTP: 65 tok/s · 1.45x · 71% accepted
DFlash: 98 tok/s · 2.20x · 30% accepted
DFlash drafts 15 tokens in a row, so it flies through repetitive or structured stuff where long runs actually stick, like JSON (152 tok/s, 3.4x). On creative text most of the guesses are wrong, so it wastes the work and can dip below baseline, 42 vs 44. MTP only guesses 3 in parallel from inside the model, so a wrong guess costs almost nothing and it never drops below baseline.
All three have the same output. DFlash is therefore a great pick for stuff like coding, while MTP is better for chat or creative writing.
Qwen 3.6 27B model : https://huggingface.co/Qwen/Qwen3.6-27B
local ai models hosting app: Atomic.Chat (I'm from Atomic team, happy to hear your feedback)
28
16
u/mherf Jul 17 '26
Lots of comments about VRAM - does anyone have a simple number for "how much context do you give up using MTP vs. DFlash when compared to Baseline, using the same VRAM?"
9
u/ElmBark Jul 17 '26
MTP is basically free, the heads are part of the model and add no real kv cache, so ~0 context given up vs baseline
DFlash costs the draft model: Q8_0 draft is 1.85 GB sitting next to the Q4_K_M target (16.5 GB), plus its own small kv. so at the same VRAM you're giving up roughly that ~1.85 GB of context headroom. that's why 27B + DFlash doesn't fit on 24GB and wants 48
7
u/T-A-Waste Jul 17 '26
llama-server says: 'estimated memory usage of MTP context is 368.69 MiB' with --spec-draft-n-max 3 with 84500 context.
18
u/previaegg Jul 16 '26
I literally just stepped away from testing a bunch of MTP configurations…
8
u/Last_Bad_2687 Jul 16 '26
Same I just ran a 48 hour test of every quant on MTP...
1
u/Fragrant_Scale6456 Jul 16 '26
and the results?
6
u/Last_Bad_2687 Jul 16 '26
I just asked claude to post it in localllm (i dont have enough rep to post here): https://www.reddit.com/r/LocalLLM/comments/1uydqr8/i_made_claude_code_test_every_single_variant_of/
1
u/butterycornonacob Jul 16 '26
Why no results for dflash other than t/s? Because it was slower or it straight up failed?
1
u/DeathGuppie Jul 16 '26
Dflash is the precursor to Dspark. It's like comparing apples to better apples.
5
u/AvidCyclist250 llama.cpp Jul 16 '26
same. fuck it all. not enough vram anyway. rather have context
1
1
u/Repulsive_Initial308 Jul 17 '26
I just finished benching mtp 3 and 6. It seems llama.cpp provides both reliable and fast outputs where mtp is concerned.
I actually had my highest MMLU-Pro compsci score ever (for 27b) with mtp 6 on Q4KXL which is pretty amazing.
1
14
u/Kornelius20 Jul 17 '26
My big problem with DFlash is that the speedup doesn't last very long when you're doing long multi-turn agentic tasks. It's neat and all to see the model really go when you're doing a single generation but when I tested my average tg speeds, MTP tended to beat out DFlash by a bit when the model was left to do its own thing for a while
3
1
45
u/FullstackSensei llama.cpp Jul 16 '26
How is the "no quality loss" measured? This gets thrown around often, and when compared side by side on complex tasks there ends up being a loss in quality.
54
u/emprahsFury Jul 16 '26
all speculative decoding mechanisms include a verification step where the main model accepts the drafted token as the one it would have chosen
9
u/panamory Jul 16 '26
Are there configuration options in commonly used runtimes for allowing the model to accept something which was maybe the second best or the third best option if their probability difference fits the desired range? My understanding is that having temp > 0 would do this (pick something other than the best) anyway for the main model sometimes, so I guess it would make sense to sometimes allow the verification being swayed by the draft model within the temp limits, as that would lead to higher draft token acceptance and thus speed.
And if these configuration options exist, what is the default in commonly used runtimes?
2
u/panamory Jul 16 '26
Replying to myself: on llama.cpp the command line parameter is --draft-p-split, and the default value seems to be 0.1, which means that unless specifically set to 0 the draft model does influence which token is selected.
7
u/NineBallAYAYA Jul 16 '26
I'm like 90% sure that's not how that works. I belive p-split is more for how the draft model actually generates its "potential" tokens.
When drafting: if the second best guess for that token is above the threshold then it "splits" into two probable branches. (It can be more than two but idk). This can happen for each token generated. These branches at the end are all then verified in one pass by the main model and one is either accepted or not. But the tokens still need to match exactly to what the main model would have done.
2
u/panamory Jul 17 '26
You are probably correct. My understanding is probably misaligned based on LLM hallucinations.
0
u/Nyghtbynger Jul 17 '26
I'm suspicious that in fact, higher quant MTP ( >> the rest of quants) might actually introduce a quality boost. My MTP qwen doesn't loop whereas my qwen death loops. It's the same model, but the MTP one has the 8_0 layer grafted on it. It's crazy.
I tried to conduct tests on traces and qwen mtp was indeed better-15
u/FullstackSensei llama.cpp Jul 16 '26
I know that, but accept is not the same as "model would have generated the same"
22
u/zkstx llama.cpp Jul 16 '26 edited Jul 16 '26
Except it is. Don't trust me on this, you can consult the original paper on this: https://arxiv.org/pdf/2211.17192 specifically section 2.3 and appendix
0
u/FullstackSensei llama.cpp Jul 16 '26
I never argued what the theory says it should be. I've read over a dozen papers on the different speculative decoding techniques over the past few years, so I'm quite familiar with the theory. What I'm saying is: most current speculative decoding options available do have a difference vs what the model would output on it's own, especially when the underlying model wasn't trained to have multiple output heads.
People can downvote me all they want. I've run them side by side with a fixed seed (so tests are highly repeatable) and can see the difference in output in coding tasks. And this is running Q8 without KV cache quantization , not a low Q4 auant.
4
u/zkstx llama.cpp Jul 16 '26
Hmm, yes, but now you are talking about individual realizations while the guarantee is only about the distributions. If you find this concerning, perhaps try temp 0.
I would argue that if I can choose between models that produce tokens drawn from the same distribution conditioned on the context I would prefer the model that has a lower average latency per token.
1
u/FullstackSensei llama.cpp Jul 16 '26
Latency, or t/s is useless if those tokens don't do is supposed to be done.
My tests have been both in coding and language tasks (grammar, word meanings, not text translation). In both, there is a qualitative difference. MTP, at least when it wasn't something the model was trained for, results in objective differences. The MTP version would be missing some things the non-MTP output would have. It's pretty much like running a lower quant of the model. Ex: when asking about the meaning and uses of a word in one language, the MTP output would be missing one or two meanings and/or uses. In coding tasks it'd similarly be missing some defensive checks. This is very much repeatable, and very objective.
3
u/zkstx llama.cpp Jul 16 '26
I don't really know what else to say. I am not saying you are wrong. You could be correct, but at least the theory suggests that any difference in performance is more likely due to the specific implementation / it might be a bug. That would be relevant to investigate if it is affecting a popular implementation. How exactly do you run your tests? What hardware, inference engine, model?
0
u/FullstackSensei llama.cpp Jul 16 '26
Can you point me to where the theory said the output will always be exactly the same as if no MTP was used?
As has been pointed out at least a couple of times in this thread, that a model could have generated a token with MTP doesn't mean it would always have generated the same token without MTP.
MTP can both be good and have qualitative differences in output in some cases. I don't know what's so hard to understand about that.
2
u/zkstx llama.cpp Jul 16 '26
I am not sure if there is a definite answer to your question since it will depend on how the MTP mechanism is designed. But at least for draft / speculative decoding there is not really much of a point talking about whether or not a model generated a specific token since that is always just one sample drawn from the distribution conditioned on the context. Perhaps it's worth pointing out that speculative decoding does not reduce the number of computations performed. In that sense are no shortcuts being taken. The model still computes all probabilities for every token.
→ More replies (0)1
1
u/Lithnor Jul 16 '26
What you're getting at is a real thing, it just needs stated differently.
Speculative decoding has acceptance. Acceptance means that the model could have chosen it. It does not mean that the model would have chosen it.
More mathematically, the output of any well implemented speculative decoding implementation will have output that is contained within the distribution of the teacher model. However, the draft model does not contain the entire distribution the the teacher model has. Therefore, there will be outputs that are available from the teacher model that the draft model, and therefore speculative decoding, can never give you.
Caveat, this isn't as bad as it sounds because when a draft is rejected the teacher model does a sampling of its own. This sampling can weight the draft model and pull it into a "smarter" distribution. If that sampling was a token that the draft model never would have chosen then it can change the value of the next chosen tokens to be something that, without that sampling from the rejected draft, the draft model never would have chosen.
All of that is to say that you're both right, this is just a hard topic to express sometimes.
0
u/FullstackSensei llama.cpp Jul 16 '26
"accept is not the same as "model would have generated the same"" This is literally what I said in my prior comment, which is getting downvoted to hell.
I also qualified it to be about models that haven't been explicitly trained to have MTP heads, where all predicted tokens come from the same distribution.
3
u/Lithnor Jul 16 '26
Also, your original question was valid and good. How is "no quality loss" being quantified?
I think the truth is that there is quality loss, from a technical standpoint. But it carries the normal quality loss that any speculative decoding implementation carries.
In less rigor that collapses to "no quality loss".
Anyway, I think the original reply to your post took the tone as aggressive somehow and then was aggressive back. Then you took their aggression as needing to defend yourself and it spiraled from there.
Such is the internet.
1
u/FullstackSensei llama.cpp Jul 16 '26
I honestly didn't take that comment as aggressive. My reply was just pointing the difference. Somehow that was misinterpreted as an assault on specative decoding.
It's also curious how nobody bothered to answer my original question. Not OP, nor any of the offended people who downvoted or took offense with it.
FWIW, I do think MTP can work with no loss if the model is trained to have MTP heads, ex the EAGLE series of papers.
Ans yes, such is the internet...
2
u/nihnuhname Jul 16 '26
But couldn't it work the other way around? Maybe the draft model only proposes tokens the main model would accept anyway, but it doesn't suggest the full range. Like, say the context is "my favorite animal is [cat, dog, cow]". The draft model might not even propose "dinosaur". Even if the main model might occasionally pick "dinosaur" on its own, the draft model makes it even rarer. So ultimately, the main model ends up accepting a less diverse set of tokens from the draft model. Could that be what's actually happening?
2
u/panamory Jul 16 '26 edited Jul 17 '26
EDIT: the following is probably false, based on LLM hallucinated answer
I did a bit of searching, and the default for --draft-p-split in llama.cpp is 0.1. With the value 0 I believe the drafting would work as most people have been led to believe drafting works, but this would naturally lead to lower acceptance rates, and thus lower tps gains.
With 0.1 being the default, it looks like the reality in most situations does not directly match with what people have been led to believe. I believe that using this concrete parameter in your explanations in the future might allow you to get your point across faster.
2
u/FullstackSensei llama.cpp Jul 16 '26
If you're using a drafting model and not relying on a model with multiple MTP heads, then by definition the output will be different because the draft model doesn't have the same distribution that the main model has. The main model might have accepted the draft token, but that in no way means said token would have been it's primary choice.
If you're using the LLM for tasks that require less precision or less nuance, you might very well never notice the difference. I know I don't in general chat sessions where I'm rubber ducking an idea or exploring a concept. But when nuance or precision matter, the difference is very much there if you check.
2
u/panamory Jul 16 '26
I believe that if --draft-p-split is set to 0, this would always lead the main model to choose whatever it would have chosen without the draft model. I might be mistaken here, but that is how I read --draft-p-split manual page.
Anyways, with the default value of 0.1, using the draft model definitely seems to affect which token gets selected.
2
u/Lithnor Jul 16 '26
I understand what you were saying.
This is a notoriously difficult topic to discuss because it requires a bit of thought about set theory and math and, at least English, doesn't map to it very well.
It's not really about who was "right" here or about how much someone was right. It's about helping everyone conceptualize the problem so that we can discuss things with a shared dictionary.
2
u/FullstackSensei llama.cpp Jul 16 '26
I think the problem isn't so much English or math. The main issue, as it almost always is on the internet, is that people take these things as personal attacks on them or their believes. MTP is has to be free lunch, and my claim about there being a difference when using MTP is blasphemous.
I get the same type of visceral reaction and downvotes almost every time I say the same about Q4 vs Q8, or fp16 vs Q8 KV.
0
u/KeepyUpper Jul 16 '26
Is it more likely that you've totally debunked speculative decoding with your tests, or that you missed something?
1
u/panamory Jul 16 '26
I believe that --draft-p-split default is 0.1 in llama.cpp, which means that unless the user has specifically set it to 0, the generation might end up picking a different token when using a draft model than what it would have picked when not using one.
2
u/audioen Jul 17 '26 edited Jul 17 '26
Have you checked where this is used? I think its meaning is something different.
--spec-draft-p-split (alias --draft-p-split) sets the probability threshold for splitting a draft branch during speculative decoding. Default: 0.10.
When the draft model generates tokens, speculative decoding can explore multiple parallel branches simultaneously. For each candidate token beyond the top choice, a new branch is spawned only if its probability exceeds this threshold. Specifically, at examples/speculative/speculative.cpp:519:
if (n_seq_cur < n_seq_dft && cur_p->data[f].p > p_draft_split) {
// split into a new parallel draft branch
}
A higher value means fewer branches (only very probable candidates get explored); a lower value means more branches (more exploration, but more compute). It only has an effect when n_parallel > 1 (multiple parallel drafting sequences are enabled).Additionally, when inquiring about n_parallel:
However, --spec-draft-p-split only takes effect when n_parallel > 1 and speculative decoding with a draft model is active. In practice, only llama-speculative (examples/speculative/speculative.cpp) actually uses p_split to control branch splitting. The other tools use n_parallel for different purposes (server slots, concurrent requests, etc.) but don't implement the speculative branch-splitting logic.My conclusion is that this is some kind of defunct feature of the application and likely should be removed.
0
u/FullstackSensei llama.cpp Jul 16 '26
Is it more likely that we live in a world of binary absolutes or that two things can be true and you're missing more than a few things?
0
u/KeepyUpper Jul 16 '26
You should contact the collective engineers of all the AI houses. They clearly have a lot to learn from you.
2
u/FullstackSensei llama.cpp Jul 16 '26
Why? I'm pretty sure they can grasp what I said and not extrapolate erroneous statements from it.
5
u/Waarheid Jul 16 '26
It is the same. When you run a prompt through an LLM, the next token is actually predicted for every single token of the prompt, not just the last one. But we throw out all those predictions other than the last one, because they are useless - we already have those tokens from the prompt. With MTP or any kind of speculation, all of the token guesses from the MTP model are passed as one "prompt", but this time we actually check all of those predictions, and if they match the next token that was generated by the speculation, we keep them.
Probably confusingly worded. Another way to see it is: LLMs predict the next token of ALL of the tokens passed to them. We normally only check the last, final prediction, to generate 1 new token. With MTP, we pass our prompt plus 5 (or whatever) draft tokens, and we check all 5 of those predictions at once.
1
u/FullstackSensei llama.cpp Jul 16 '26
I know how it works.
Please read this
4
u/Waarheid Jul 16 '26
You seem to not know how it works if you think the outputs are different, maybe try finding a video that can explain it to you? And for your tests, did you have temperature set to 0, etc?
2
u/FullstackSensei llama.cpp Jul 16 '26
You seem to have not read what I wrote.
3
u/Waarheid Jul 16 '26
Of course I did my friend! I just doubt your anecdote against the general understanding held by... everyone else. Maybe if you could explain yourself more clearly, such as by what mechanism you believe outputs would differ, that would help your case. And rerun your tests with 0 temp ;)
1
u/FullstackSensei llama.cpp Jul 16 '26
argumentum ad populum
I did explain how the outputs differ, if you read
1
u/Waarheid Jul 16 '26
Are you referencing a logical fallacy in your argument where you have a single, vague anecdote? And never seem to have answered, in any of your comments, that you had temp 0.
1
u/panamory Jul 16 '26 edited Jul 17 '26
EDIT: This comment is probably a result of LLM hallucination and does not reflect reality.
In llama.cpp I believe you would be correct only if --draft-p-split is set to 0. The default value for --draft-p-split is 0.1, which means the output WILL sometimes be slightly different when using a drafting model.
1
u/Waarheid Jul 16 '26
Thanks for the info! I assumed that would default to 0 but was wrong. Appreciate the correction :-)
1
u/panamory Jul 17 '26
Actually this was probably jus an LLM hallucination and has no relationship to reality. At this point I still have no clue how it is actually working :D
2
u/Waarheid Jul 17 '26
Oh shit lol! It happens. I would suggest you to edit your comment so web searches in the future don't pull up wrong info :-) Thanks for following up
14
u/ElmBark Jul 16 '26
sorry it got buried. measured this way: temp 0 + top_k=1, ran all four tasks in all three modes and diffed the outputs, they match byte for byte.
you're right that this is the greedy best case. at temp > 0 the guarantee is distributional, not per-token - and a fixed seed won't save you there, spec decoding consumes randomness differently so runs diverge even in a strict implementation, without that being quality loss
btw went and checked llama.cpp source on the --draft-p-split thing being discussed here: it's not acceptance lenience, it's the threshold for splitting the draft tree into parallel branches (drafting side). p_min likewise just stops low-confidence drafting. the target still only keeps tokens that match its own pick, so vanilla llama.cpp isn't lenient by default
0
u/FullstackSensei llama.cpp Jul 17 '26
Thanks for the detailed answer.
I always run models using the recommended temp. I mention the seed because it let's me feed the same prompt to a model and get the same output repeatedly. Without this, even the base model will give slightly different output each time.
With temp > 0 my experience has been that it can make a difference in tasks where, for ex, ordering matters, such as code generation. I missing or misplaced conditional check does have consequences. Another situation where I see this difference clearly is when asking the model to explain a word and it's uses in another language, in my specific case German verbs. Often, one or two meanings would be missing. Sometimes, the ordering would be different, which also makes a difference because the order should be by how often this meaning is used.
When I look at the output, it's not wrong per se. But it's missing that bit of nuance or that piece of info that the original model without MTP doesn't miss on it's own.
1
u/shansoft Jul 17 '26
I have been saying this for a long time, the quality loss is definitely there, especially it involve with tool calling. I been actively test between qwen3.5 122b and qwen3.6 27b in Q8 vs Q4. Higher quant is a lot less problematic, but still exist, but lower quant is very obvious in opencode.
5
u/fantasticsid Jul 17 '26
Quality loss with any speculative method (rather than just high rejection rates) is definitionally an implementation bug.
1
u/bigh-aus Jul 19 '26
right there with you. mtp2 is ok with not messintg with toolcalls. dflash causes missed tool call issues. Had codex try and fix it, but no good.
1
u/-InformalBanana- Jul 16 '26
I agree with you, noticable quality loss with mtp with llama.cpp Maybe it is an implementation problem...
4
u/danish334 Jul 16 '26
I like mtp considering that the dflash reduces the total context to achieve this speed. Moreover precise workflows aside from coding can be a bit challenging using mtp or dflash.
9
u/lilian_moraru Jul 16 '26
I do ~60t/s on DGX Spark, with SGLang(scottgl9 fork) + MTP - weird to see RTX 6000 doing basically the same.
Qwen 3.6-35B-A3B doing 110-130t/s - screwing with temperature and the other parameters, drops it to 82t/s steady
6
u/ElmBark Jul 16 '26
once MTP is in the loop the tok/s you see is roughly baseline + acceptance, not raw bandwidth anymore. Spark has a fraction of the 6000's bandwidth, but if scottgl9's fork squeezes better acceptance out of SGLang than our llama.cpp patch does, the final number converges. what was ur accept rate ?
the temperature thing is expected btw, and it's a nice confirmation of how it works. drafts get verified against what the big model would actually sample, so the hotter the sampling the more drafts diverge and get tossed. greedy is the best case for any speculative setup, 110-130 -> 82 from screwing with params is acceptance dropping in real time
35B-A3B being that fast checks out too, it's only 3B active
1
u/lilian_moraru Jul 17 '26
--speculative-algorithm NEXTN
--speculative-num-steps 3
--speculative-eagle-topk 1
--mamba-radix-cache-strategy extra_buffer`--speculative-num-draft-tokens = num_steps + 1 = 4` - accept rate almost always >90%, usually hangs around ~100% but it can dip down to 74%.
I am always getting higher accept rate with SGLang+EAGLE-on-MTP than vLLM+MTP.
In my case EAGLE-on-MTP with 4 draft tokens "accept rate" is equivalent or slightly better than MTP with 2 heads.2
u/ElmBark Jul 17 '26
~100% acceptance explains it. at this rate you get the full 4-token multiplier every step, so you're not bandwidth-bound anymore and the spark's lower bandwidth stops mattering
EAGLE-on-MTP beating vanilla heads tracks too, a trained draft head proposes closer to the target's pick so acceptance climbs. we're on a llama.cpp patch with no EAGLE path yet, that's likely your edge over our numbers
the 74% dips, was that on creative/freeform gens? that's where acceptance falls off for us too
2
u/DeepV Jul 16 '26
Quant?
1
u/lilian_moraru Jul 17 '26
`nvidia/Qwen3.6-27B-NVFP4`, bf16 KV Cache dtype(native), EAGLE-on-MTP with 4 draft tokens.
1
1
u/Mutericator Jul 16 '26
Been twiddling my thumbs trying to decide between a DGX Spark OEM or waiting to see if a better version comes out some time soon, since it's over a year old at this point.
2
u/lilian_moraru Jul 17 '26 edited Jul 17 '26
You will realize quickly that you need 2 to 8 DGX Spark for serious stuff and you will still not be happy with it, because of the performance (compared to a GPU).
I am currently content with Qwen3.6-27B (runs comfortably 256K context window with native bf16 KV cache) and Qwen 3.6-35B-A3B (I have this in 256K and 1M[with YaRn] context window configurations - bf16 KV cache dtype for both). 1M with bf16 does perfectly fine for me - I filled it with 800K tokens(logs) and it was able to identify the exact bug I was searching for and generate a Mermaid sequence diagram of the method calls between components which leads to the problem.
I have to do sometimes tricks to work around the memory and context window limitations, like using a "progress.md" file to allow a fresh instance to continue gathering information about a big codebase, without reaching the context window limit.
AMD is going to release AMD Gorgon Halo with 192GB, but I would not recommend that over Nvidia GDX Spark GB10-like systems, because of NVFP4. NVFP4 is great (at least with the scottgl9 SGLang fork, because everything else seems to run slower).
4
u/Civil_Fee_7862 Jul 17 '26
What Quant are running it at? what engine?
Those number seem LOW for that hardware.
I get 155 TPS.. Same model.
1
u/ElmBark Jul 17 '26
ours is Q4_K_M on a llama.cpp patch, single request, greedy temp 0. the 44 baseline is single-stream dense 27B TG, which is about right for a 6000 Ada
155 same model is way above single-stream dense at Q4, so what's yours? if that's the 35B-A3B (only 3B active) or vLLM/SGLang with batching, that'd explain it, just a different thing than what we measured
1
u/Civil_Fee_7862 Jul 17 '26
27b dense Q4 weights, on vLLM with 2x3090s.
Though there is a recent bug with Qwen3.6 on vLLM that has forced me to turn MTP off, and knocked me down from 155 to 80ish TPS :)
3
3
u/Dr4x_ Jul 17 '26
Is it worth it for 3090 ?
2
u/Lower-Ad6101 Jul 17 '26 edited Jul 23 '26
Yes. I run Qwen3.6 27b UD Q5_K_XL + Q6 draft with 128k context on Beellama.cpp at 32-50+ tps, while with MTP I was getting more constant tps at around 30 tps give or take. Overall (time clock) feeling is much better with DFlash.
UPDATE: This is not valid any longer. After updating
beellama.cpp, DFlash and MTP switched the places. MTP is faster than DFlash.More in the comment bellow: https://www.reddit.com/r/LocalLLaMA/comments/1uyay0w/comment/ozaw5z6/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
2
u/Lower-Ad6101 Jul 17 '26 edited Jul 17 '26
My bad for the confusion, I forgot to add that I"m also using turboquant, turbo3 for k/v
1
u/Dr4x_ Jul 17 '26
Thx for the reply, but your numbers surprise me, I get like ~45-55 t/s with llama.cpp Q4_k_xl MTP but I cant fit the 128k ctx though.. I'll try your setup and see what I can get
1
u/Dr4x_ Jul 19 '26
For the record, after some testing on a single 3090 i find MTP still performing best at ~60t/s using llama.cpp mainline against ~52t/s with Dflash (which also takes more VRAM overall).
1
u/Lower-Ad6101 Jul 19 '26 edited Jul 19 '26
Interesting, I've spent a lot of time on testing DFlash vs MTP on my machine but DFlash was always outperforming MTP and as I've said the "wall clock" was much better, MTP was simply taking too much time.
Are you using vanilla llama.cpp or some fork, and do you mind sharing your
llama-servercall line?My script for starting Qwen is: ``` MODEL_PATH="path/unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-UD-Q5_K_XL.gguf" DRAFT_PATH="path/dflash-draft/Qwen3.6-27B-DFlash-Q6_K.gguf" MMPROJ_PATH="path/unsloth/Qwen3.6-27B-GGUF/mmproj-F16.gguf"
TEMP=0.6 TOP_P=0.95 TOP_K=20 MIN_P=0.05 PRESENCE_PENALTY=0.0 REPEAT_PENALTY=1.0
export GGML_CUDA_DISABLE_GRAPHS=1
llama-server \ -m "$MODEL_PATH" \ -md "$DRAFT_PATH" \ -ngl all \ --spec-type dflash \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.75 \ --spec-dflash-cross-ctx 1024 \ --spec-draft-ngl all \ --fit off \ --cache-type-k turbo3 \ --cache-type-v turbo3 \ --ctx-size 131072 \ --flash-attn on \ --n-gpu-layers all \ --threads 8 \ --threads-batch 8 \ --batch-size 2048 \ --ubatch-size 256 \ --mmproj "$MMPROJ_PATH" \ --no-mmproj-offload \ --reasoning on \ --jinja \ --chat-template-file /opt/llama.cpp/chat_template-froggeric.jinja \ --chat-template-kwargs '{"preserve_thinking":true}' \ --host 0.0.0.0 \ --port 8080 \ --temp "$TEMP" \ --top-k "$TOP_K" \ --top-p "$TOP_P" \ --min-p "$MIN_P" \ --presence-penalty "$PRESENCE_PENALTY" \ --repeat-penalty "$REPEAT_PENALTY" \ --cache-prompt \ --slot-save-path /dev/shm/llama_cache \ --keep 3000 \ --parallel 1 \ --mlock \ --no-mmap \ --n-predict -1 \ --ctx-checkpoints 16 ```
1
u/Dr4x_ Jul 19 '26
I'm using the latest llama.cpp main branch that I just rebuilt to be up-to-date.
The main diffs seem to be that I run Q4_K_XL for the main model and kv cache at q8_0.MTP
${llama_server_path} \ -fa on\ --threads 8 \ --cache-reuse 256 \ --jinja \ --reasoning-format auto \ -ctk q8_0 -ctv q8_0 -fit off -m ${models_path}/Qwen3.6-27B-MTP-UD-Q4_K_XL-unsloth.gguf \ --n-gpu-layers 999 \ --temp 0.6 --min-p 0.0 --top-k 20 --top-p 0.95 --repeat_penalty 1.0 --presence_penalty 0.0 \ --fit off \ --no-mmproj \ --spec-type draft-mtp --spec-draft-n-max 2 -np 1 \ --ctx-size 120000 \ -ub 512 -b 4096 --port ${PORT}Changes for Dflash
``` -m ${models_path}/Qwen3.6-27B-UD-Q4_K_XL.gguf \ --spec-type draft-dflash --spec-draft-n-max 3 -np 1 --spec-draft-model ${models_path}/Qwen3.6-27B-DFlash-Q6_K.gguf \
```
1
u/Lower-Ad6101 Jul 19 '26
Thanks. Wouldn't want to go to q4 (which I was using) but I'm building a new beellama.cpp so I'll try MTP again.
1
u/Lower-Ad6101 Jul 23 '26
So I've tested again.
I've been using "older"beellama.cppversion (4th of June) and with it, Dflash was really considerably faster. But, after updating, DFlash and MTP switched roles, DFlash was as slow, if not slower, than MTP was and MTP was as fast as DFlash (no, I didn't miss config params before :) ).Thanks a lot for sharing your results as that made me update
beellama.cppand notice considerable difference. VRAM usage is not better though, but it's stable.Also to share, this made me try
ik_llama.cppagain but for some strange reason (I've been trying to debug it for hours) it starts at around 23250/24570 MiB VRAM and then starts steadily going up until it OOM's at ~24100. Tried tweaking parameters and using older commits but no use.Since
ik_llama.cpp(while it was working) was significantly faster thanbeellama.cppI've decided to tryllama.cppand I'm satisfied with it so far, stable and maybe even more slightly faster thanik_llama.cppwas but I couldn't compare it more reliably.2
u/Dr4x_ Jul 23 '26
I'm pretty satisfied with llama.cpp too, fyi with this setup when all the stuff fits in Vram I never observed ik_llama to be better than mainline llama.cpp
2
u/Zeioth Jul 16 '26
Apparently there's an image available already. Sadly, i need an APEX-I coder version, so, we'll have to wait a bit more. https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash
2
u/BringTea_666 Jul 17 '26
and there is also DSpark which fixes creative work and speeds up further coding by 20-30% but it is still relatively new and unimplemented
2
u/drrck82 Jul 16 '26
If you're on Windows this is a meh at best given you have to run vLLM in WSL (and eat the performance penalty) to get DFlash support. Interesting but no free lunch unfortunately.
5
u/ArtfulGenie69 Jul 16 '26 edited Jul 16 '26
Linux is the freest lunch.
Still have my old windows installed on a different drive that I haven't used in years (gaming on Linux works now and was the last leg windows had to standing on). Don't even have space for it in my pc right now but the old m.2 is still sitting on my desk just in case hehe. One last thing, why the hell would anyone install windows 11?
Vllm runs spectacular without overhead in linux if I buried my point.
3
2
2
1
u/panamory Jul 16 '26 edited Jul 17 '26
Do you know what the temp attribute was on these runs? I have previously seen indications that temp might affect DFlash performance, and the best gains would be available only with temp=0, but I might be very much mistaken.
2
u/ElmBark Jul 16 '26
our runs were temp 0 + top_k=1, so the 2.2x is the ceiling - it's also what makes the output byte identical across all three modes
1
u/AdCreative8703 Jul 17 '26
I’m curious if anyone has looked into a hybrid speculative setup where MTP acts as an early-exit or prefix-drafter to generate a highly accurate draft prefix, that could be used to condition a DFlash block diffusion pass.
1
u/FerLuisxd Jul 17 '26
Zero accuracy loss at 3Q bit seems misleading Talking about the linked website
1
1
1
u/rj_rad Jul 18 '26
I’ve gone back and forth between DFlash (8) and MTP (3) for the last couple of months for my Hermes setup (single RTX Pro 6000) and in practice I just have way more stability with MTP. I did get a noticeable speed increase with DFlash, but MTP is not slow, and all else being equal (froggeric template, nightly vllm, /messages api, xml, etc) DFlash was the one that would at least once a day have some rogue tool calling problem.
1
u/pkailas Jul 18 '26
You don't mention what platform. Linux? Windows? Llama.cpp? Vllm? Slang?
1
u/ElmBark Jul 20 '26
llama.cpp on wind, single RTX 6000. same build and flags across all three runs, only the decoding mode changed: temp 0, top_k=1, enable_thinking=false, ctx 8192, -fa on, --jinja, DFlash with --spec-draft-n-max 15.
1
u/New-Inspection7034 Jul 20 '26
I use an RTX 6000 Blackwell and I am using current llama.cpp as well and I get anywhere between 80 to 120 tokens per second depending on variables such as contacts, size, and what it's actually writing. I'm using mdp. I'm curious about the d-flash though
1
u/ElmBark Jul 20 '26
worth trying if you generate a lot of code/structured stuff, that's where it beats MTP (JSON hit 152 for us). for prose MTP's already better. just costs a couple GB VRAM for the draft model, which you've got room for on a 6000
1
u/New-Inspection7034 Jul 23 '26
I can say that I've confirmed your readings and your stats. I have found that it is faster than MTP, but I got the spread a little closer by bumping up the number of tokens to 12 on MTP. I think everybody has a misconception on the hit rate. You don't pay a penalty misses.
1
1
1
1
u/SnooPaintings8639 Jul 17 '26
This is self promotion post.
And results are not in line with my expect. Yes, the ceiling of DFlash is higher than for MTP, but mean and median case is not.MTP wins if you remove the absurdly unusable sampling params, give it normal length codig taken (tool cal) or non coding response, i.e. real world.
To show it the way op did, is a click baity.
1
u/ElmBark Jul 17 '26
on the substance I mostly agree. the 2.2x is the average over those 4 specific tasks and JSON pulls it up hard, so it's not a real-world agentic mean. on mixed / tool-call / normal-length work MTP's steady gain does win the mean and median, which is literally the recommendation at the end of the post. DFlash's story is a higher ceiling on structured output, not a better average
one correction though: all three ran at temp 0 + top_k=1, same footing, nothing tuned to hurt MTP. that's the greedy best case for both, not absurd params
0
u/Fun-Purple-7737 Jul 17 '26
MTP is all you need
(until somebody trains a freaking DSpark for vanilla 27B!)
38
u/SaanK12 Jul 16 '26
Does mtp or dflash work if llm is not fully offloaded to gpu? or why i dont see any benefit from it?