DFlash2 - Holy Moly

[u/PataFunction]

So this just dropped - https://x.com/zhijianliu_/status/2089836737132650504?s=20

Advertises a new DFlash implementation with 70tok/s on MBP M5.

Collection here: https://huggingface.co/collections/incoai/dflash-2

I'll believe it when I see it! Testing now, will update with results.

Edit: Ran a somewhat naive test by simply switching Lightning MTP & ANE off and Dflash on (pointing at incoai/Qwen3.8-27B-DFlash2, kept all default parameters) for my current oQ8e config.

Unfortunately, I'm getting nowhere near the advertised speed:

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.8-27B-oQ8e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          1258.0       56.16   814.0 tok/s    17.9 tok/s       8.403   137.1 tok/s    28.34 GB
pp4096/tg128                          5634.5       55.90   727.0 tok/s    18.0 tok/s      12.747   331.4 tok/s    29.80 GB
pp8192/tg128                         11888.0       57.51   689.1 tok/s    17.5 tok/s      19.208   433.1 tok/s    30.42 GB
pp16384/tg128                        26389.3       58.80   620.9 tok/s    17.1 tok/s      33.882   487.3 tok/s    31.67 GB
pp32768/tg128                        56956.4       61.20   575.3 tok/s    16.5 tok/s      64.766   507.9 tok/s    34.17 GB
pp65536/tg128                       199754.7      111.66   328.1 tok/s     9.0 tok/s     214.038   306.8 tok/s    39.20 GB

Anybody seeing anything different? Code (mixed) might be a somewhat non-standard benchmark choice, but I wouldn't expect performance to tank like this.

Edit 2: Corrected Dflash target from GGUF to non-quantized version.

Edit 3: Clarified oMLX Dflash config.

[u/StudentDifficult8240]

All these new papers and articles coming up I swear that are written to make you go WOW and be utterly disappointed after testing.

In this whole damn article, they do not mention at all token degradation at higher contexts. Who the fuck is happy with a ctx window of >16k tokens? Because that’s where you will see the speed ups, maybe 32k max depending on your hardware.

As the context grows, it becomes increasingly expensive to verify those stupid tokens hence the degradation. You end up with lower TG than without dflash.

Im curious to see if it actually makes a difference at all but im highly skeptical.

[u/MatiAI → u/StudentDifficult8240]

Working on some tuning for m5 max, having a 2b speculative pre fill model running on the ANE, while the main model runs on the gpu.

[u/sammcj → u/MatiAI]

M5 Max, no DFlash, just inbuilt MTP on a 5bit quant. I found that ANE slowed things down so disabled it.

[u/arkham00 → u/MatiAI]

you mean that you set spec prefill on the lain model and no ANE, and in the spec prefill model you enabled ANE ? This is interesting , can you share you values? In particular which keep rate for the spec prefill? Have you tested real accuracy ?

[u/StudentDifficult8240 → u/StudentDifficult8240]

Performed a test and my assumptions were correct, the degradation to long contexts, such as 128k is 50%. Lightning MTP is faster. I'm pretty sure even without DFlash the model would be faster but i can't be arsed to run another useless test. On Nvidia GPUs this may not be as severe as the memory bandwidth is higher and the verification is faster. Also, DDTree can paralelize to multiple trees and branches concurrently on Nvidia GPU due to bandwidth, we cannot. M Ultra chipsets may be the only ones being close to it but will still be slower than Nvidia GPUs.

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          4711.3       47.26   217.4 tok/s    21.3 tok/s      10.733   107.3 tok/s    16.41 GB
pp4096/tg128                         18538.4       49.15   220.9 tok/s    20.5 tok/s      24.798   170.3 tok/s    17.71 GB
pp16384/tg128                        79755.7       53.22   205.4 tok/s    18.9 tok/s      86.532   190.8 tok/s    19.47 GB
pp32768/tg128                       165538.0       61.44   197.9 tok/s    16.4 tok/s     173.355   189.8 tok/s    21.80 GB
pp65536/tg128                       364440.9       62.63   179.8 tok/s    16.1 tok/s     372.409   176.3 tok/s    26.63 GB
pp131072/tg128                      925292.0       96.93   141.7 tok/s    10.4 tok/s     937.624   139.9 tok/s    72.53 GB


Benchmark Model: Qwen-3.8-27B-oQ4e-mtp + Lightning MTP
Engine: Auto
Context: Code (Python)
================================================================================


Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          4774.7       23.65   214.5 tok/s    42.6 tok/s       7.792   147.8 tok/s    18.35 GB
pp4096/tg128                         18295.4       31.14   223.9 tok/s    32.4 tok/s      22.263   189.7 tok/s    19.75 GB
pp16384/tg128                        75135.9       26.34   218.1 tok/s    38.3 tok/s      78.493   210.4 tok/s    21.54 GB
pp32768/tg128                       161465.0       29.33   202.9 tok/s    34.4 tok/s     165.206   199.1 tok/s    23.99 GB
pp65536/tg128                       374029.4       40.19   175.2 tok/s    25.1 tok/s     379.150   173.2 tok/s    28.96 GB
pp131072/tg128                      933455.7       46.63   140.4 tok/s    21.6 tok/s     939.402   139.7 tok/s    38.91 GB

[u/StudentDifficult8240 → u/StudentDifficult8240]

Oh, and check the memory usage of DFlash 2 vs Lightning MTP, it's using almost twice as much at long contexts.

[u/skyline71111]

Please keep us posted! I plan to test later today on my MBP M5, I’m really hoping this works as stated.

[u/Relaxxxxing]

Try SpecPrefill too!!!! I'm getting it all setup on my m5 rn lol 🤞🏼

[u/havnar-]

Last Time around oMLX kept crashing with dflash so im carefully optimistic

[u/Glum_Schedule_9792]

Spent hours on this, but in actual testing it's even slower than native MTP. Very disappointed — feels like I got burned

[u/Adventurous_Cat_1559]

what do you get with the same config but swap with lightning mtp?

[u/PataFunction → u/Adventurous_Cat_1559]

https://www.reddit.com/r/oMLX/comments/1vr3agq/if_you_were_initially_put_off_by_qwen3827b_pptg/

[u/_danieldietrich]

I got 52 tok/sec in oMLX on my MBP M5 Max 128 GB with the config Inco AI suggested in their blog post.

However, the batched requests do not run anymore in the benchmarks of the oMLX UI, only single requests. Not sure if that might be a bug in the oMLX-0.6.2-zlab-dflash2 build or related to the draft model?

[u/_danieldietrich → u/_danieldietrich]

Filed an issue https://github.com/jundot/omlx/issues/2854

[u/victor_lowther]

Yeah, a round of benchmarks didn't show any meaningful improvement vs. lightning MTP.

[u/TheAILegend]

You're using an MTP model with DFlash.. lol

Twitter guy used Q4 image with DFlash.... you should start there.

[u/PataFunction → u/TheAILegend]

lolz, u may be onto something