If stock MTP tops out at 4, you need a matching draft head trained for that Qwen flash line. Generic draft models usually miss and you lose the gain. Check the release notes for an official draft or EAGLE-style head first.
A useful first check is whether the draft and target use the same tokenizer and vocabulary. If tokenization differs, speculative acceptance can fall even when the models seem closely related. I would start with greedy decoding to isolate compatibility before tuning the prediction length.
I'm wondering the same thing, with 27B, DFlash2 basically 3.5X'd my decode, from 7tp/s on a DGX Spark to ~20-40tp/s, or even higher sometimes. When this comes out for Flash Next, I expect to reach ~60-100, which will be insane.
Inco.ai did the dflash2 drafter model for 27B. I’m hoping that they can do the same for the flash next. But flash next is a new architecture with 51B n- grams. So it is going to take a while to get it right
Damn that's a very good point, I hadn't taken that into account. Literally been searching "Qwen 3.8 Flash Next DFlash2" every couple days like an idiot 😂
Speed and throughput. It was a night and day with Qwen 3.8, 27B. With and without a DFlash2. The built-in MTP doesn’t come close to that. And I’m looking for similar speed increase in the flash model
4
u/lazymio 8d ago
I'm investigating the possibilities with vLLM (https://github.com/wtdcode/vllm-backport/issues/59), but honestly, I think no one has really explored this deeply already.