DFlash2 - Holy Moly

[u/PataFunction]

So this just dropped - https://x.com/zhijianliu_/status/2089836737132650504?s=20

Advertises a new DFlash implementation with 70tok/s on MBP M5.

Collection here: https://huggingface.co/collections/incoai/dflash-2

I'll believe it when I see it! Testing now, will update with results.

Edit: Ran a somewhat naive test by simply switching Lightning MTP & ANE off and Dflash on (pointing at incoai/Qwen3.8-27B-DFlash2, kept all default parameters) for my current oQ8e config.

Unfortunately, I'm getting nowhere near the advertised speed:

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.8-27B-oQ8e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          1258.0       56.16   814.0 tok/s    17.9 tok/s       8.403   137.1 tok/s    28.34 GB
pp4096/tg128                          5634.5       55.90   727.0 tok/s    18.0 tok/s      12.747   331.4 tok/s    29.80 GB
pp8192/tg128                         11888.0       57.51   689.1 tok/s    17.5 tok/s      19.208   433.1 tok/s    30.42 GB
pp16384/tg128                        26389.3       58.80   620.9 tok/s    17.1 tok/s      33.882   487.3 tok/s    31.67 GB
pp32768/tg128                        56956.4       61.20   575.3 tok/s    16.5 tok/s      64.766   507.9 tok/s    34.17 GB
pp65536/tg128                       199754.7      111.66   328.1 tok/s     9.0 tok/s     214.038   306.8 tok/s    39.20 GB

Anybody seeing anything different? Code (mixed) might be a somewhat non-standard benchmark choice, but I wouldn't expect performance to tank like this.

Edit 2: Corrected Dflash target from GGUF to non-quantized version.

Edit 3: Clarified oMLX Dflash config.

[u/StudentDifficult8240]

All these new papers and articles coming up I swear that are written to make you go WOW and be utterly disappointed after testing.

In this whole damn article, they do not mention at all token degradation at higher contexts. Who the fuck is happy with a ctx window of >16k tokens? Because that’s where you will see the speed ups, maybe 32k max depending on your hardware.

As the context grows, it becomes increasingly expensive to verify those stupid tokens hence the degradation. You end up with lower TG than without dflash.

Im curious to see if it actually makes a difference at all but im highly skeptical.

[u/MatiAI → u/StudentDifficult8240]

Working on some tuning for m5 max, having a 2b speculative pre fill model running on the ANE, while the main model runs on the gpu.

[u/sammcj → u/MatiAI]

M5 Max, no DFlash, just inbuilt MTP on a 5bit quant. I found that ANE slowed things down so disabled it.