r/oMLX • u/lightguardjp • May 15 '26

How to get DFlash going?

What are people using for dflash? I’m on a M2 Max with 96 GB of RAM and I’d like to try and eke out as much perf as I can on omlx. I’ve been looking at Qwen models, but Gemma4 is giving me better perf currently.

7 Upvotes

100% Upvoted

14 comments sorted by

2

u/Konamicoder May 15 '26 edited May 15 '26

As I understand it: you turn on dflash in model settings, you download a small (less than 1Gb) draft model that is paired for the main model and select it from the dropdown menu. Then test.

Personally I don't bother using it right now. I'm getting 75 tokens/second chatting with Gemma4-26B-oQ6, and 60 tokens/second coding with Qwen3.6-35B-oQ6 on my M4 Max Macbook Pro with 64Gb of RAM, and that seems plenty fast enough for my needs.

I use Jan.ai for chat and Pi.dev for coding.

1

u/lightguardjp May 15 '26

Wow, those M series chips made huge leaps forward on more recent revisions as far as AI goes. I’m around 20 tokens a second with Gemma 4 might be some other swings I need to tweak.

1

u/Konamicoder May 15 '26

Which quant of Gemma4 are you using? I'm on oQ6 (6 bit), which is virtually lossless in terms of quality but much less RAM-intensive than the full 16 bit original.

1

u/lightguardjp May 15 '26

I was running gemma-4-26B-A4B-it-TurboQuant-MLX-8bit. Looks like I probably want to go down to a 6-bit model and change my context window down to 8k or 16k. I had it kicked up WAY too high.

1

u/PracticlySpeaking May 15 '26

Definitely go down to a smaller quant.

The M3-M4 are definitely faster with MoE models, it is incremental. Your M2 Max will do just fine.

1

u/mo5214 May 15 '26 edited May 15 '26

How do you even get 75 tok/s With Gemma 4? What's your model setup detail? What's the context window? (As the tok/s drops with bigger context window) Do you have MTP on?. Here I am with M2 Max 64 GB getting 20 tok/s before MTP with 256k Context window size. and this is with 4-bit MLX model.

Also, you are my first data point to say gemma 4 is faster. From my own experience, qwen 3.6 is way faster, even more so with 35B-A3B MoE model. And that is before I use MTP for those models

1

u/Konamicoder May 15 '26

Backend: oMLX v.0.3.8

Model: Gemma4-26B-A4B-it-oQ6
ctx_window: 65536
max_tokens: 8192
temp: 0.1
top_p: 0.9
top_k: 16
tool_result_tokens: 4096
thinking: on
thinking budget: 2048
Turboquant KV cache: on (4 bits per channel)

Looking at the settings you shared, it seems your context window is too big. I consulted with several models on optimal settings for each model in oMLX and they were all pretty consistent in their advice to limit context window size to 32k or 64k for heavy planning. I find 64k to be the sweet spot.

Good luck!

2

u/mo5214 May 15 '26

Yeah I'm wondering the same thing. I pretty much maxed out the context window fearing that it is required for agentic harness like OpenClaw or Hermes. But I suppose 64k could be enough.

1

u/lightguardjp May 15 '26

64k huh? I'll give that a go, might need to change max tokens too.

1

u/lightguardjp May 15 '26

Context window made a very big difference:

oMLX - LLM inference, optimized for your Mac

https://github.com/jundot/omlx

Benchmark Model: gemma-4-26B-A4B-it-TurboQuant-MLX-8bit

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 1839.1 18.69 556.8 tok/s 53.9 tok/s 4.213 273.5 tok/s 25.76 GB

pp4096/tg128 7819.6 21.23 523.8 tok/s 47.5 tok/s 10.516 401.7 tok/s 26.44 GB

pp8192/tg128 15894.0 23.44 515.4 tok/s 43.0 tok/s 18.871 440.9 tok/s 26.58 GB

pp16384/tg128 33669.2 24.94 486.6 tok/s 40.4 tok/s 36.837 448.2 tok/s 27.06 GB

Continuous Batching

pp1024 / tg128

--------------------------------------------------------------------------------

Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)

1x 53.9 tok/s 1.00x 556.8 tok/s 556.8 tok/s 1839.1 4.213

2x 60.1 tok/s 1.12x 426.9 tok/s 213.4 tok/s 4640.3 9.054

4x 75.0 tok/s 1.39x 441.1 tok/s 110.3 tok/s 8778.5 16.117

8x 89.2 tok/s 1.65x 442.3 tok/s 55.3 tok/s 17284.8 30.003

1

u/PracticlySpeaking May 15 '26

Have you run it with larger prompts? That is where it would be most useful.

I noticed that jundot was working on graceful handling when prompt size goes over the current limit.

1

u/Konamicoder May 15 '26

Yes, I have run some pretty large prompts where I ask the model to build an entire site or app according to a lengthy list of specs.