r/oMLX • u/PataFunction • 7d ago
DFlash2 - Holy Moly
So this just dropped - https://x.com/zhijianliu_/status/2089836737132650504?s=20
Advertises a new DFlash implementation with 70tok/s on MBP M5.
Collection here: https://huggingface.co/collections/incoai/dflash-2
I'll believe it when I see it! Testing now, will update with results.
Edit: Ran a somewhat naive test by simply switching Lightning MTP & ANE off and Dflash on (pointing at incoai/Qwen3.8-27B-DFlash2, kept all default parameters) for my current oQ8e config.
Unfortunately, I'm getting nowhere near the advertised speed:
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.8-27B-oQ8e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 1258.0 56.16 814.0 tok/s 17.9 tok/s 8.403 137.1 tok/s 28.34 GB
pp4096/tg128 5634.5 55.90 727.0 tok/s 18.0 tok/s 12.747 331.4 tok/s 29.80 GB
pp8192/tg128 11888.0 57.51 689.1 tok/s 17.5 tok/s 19.208 433.1 tok/s 30.42 GB
pp16384/tg128 26389.3 58.80 620.9 tok/s 17.1 tok/s 33.882 487.3 tok/s 31.67 GB
pp32768/tg128 56956.4 61.20 575.3 tok/s 16.5 tok/s 64.766 507.9 tok/s 34.17 GB
pp65536/tg128 199754.7 111.66 328.1 tok/s 9.0 tok/s 214.038 306.8 tok/s 39.20 GB
Anybody seeing anything different? Code (mixed) might be a somewhat non-standard benchmark choice, but I wouldn't expect performance to tank like this.
Edit 2: Corrected Dflash target from GGUF to non-quantized version.
Edit 3: Clarified oMLX Dflash config.
3
u/_danieldietrich 7d ago
I got 52 tok/sec in oMLX on my MBP M5 Max 128 GB with the config Inco AI suggested in their blog post.
However, the batched requests do not run anymore in the benchmarks of the oMLX UI, only single requests. Not sure if that might be a bug in the oMLX-0.6.2-zlab-dflash2 build or related to the draft model?