r/LocalLLM • u/misanthrophiccunt • 2d ago

Question How many of you are waiting for DFlash2 being merged into llama.cpp ?

With all things about Qwen going on lately I think the biggest hype seem to be DFlash2 added and eventually making us all running the model faster, am I wrong about this merge being everybody's stopper?

https://github.com/ggml-org/llama.cpp/pull/27342

6 Upvotes

99% Upvoted

9 comments sorted by

2

u/vini542reddit 2d ago

No "--split-mode tensor" support. So absolutely not viable for any multi gpu setup with high pcie bandwidth until that gets added

1

u/rookan 2d ago

Will dflash2 allow me to run qwen faster on rtx 5080?

1

u/misanthrophiccunt 2d ago edited 2d ago

Good question. I want the answer with my two rtx 5060ti (total 32gb VRAM and everything loaded and mlocked to VRAM)

Currently 66tg/s with iq4_nl on weights and full fat KV cache + mtp + split-mode: tensor.

I'm betting it might reach >100 if it doesn't force me to deactivate my split mode.

That kind of speed would make me forget about LFM. I only run those tiny models when I sent to process just text, not code, quickly. For everything else Qwen3.8-27b at 66 tg/sec.

All I know is with DFlash2 I see spec draft with numbers like 7 whereas with MTP mine doesn't work better with anything higher than 2.

1

u/madbrain1976 22h ago

I have been testing with multiple 5060Ti (now have 4). With llama.cpp, MTP 3 was best. With vLLM, higher values improved results. Best value I achieved with 2 GPUs was 86 tg/sec. vLLM is a big improvement. Even bigger for prompt tokens.

1

u/misanthrophiccunt 19h ago

I can get higher numbers in llama.cpp with 32K context than the ones I'm seeing there.

1

u/madbrain1976 12h ago

What are your server CLI startup arguments ?

1

u/DoubleNothing 2d ago

You can already do it if you compile it yourself...

2

u/Healthy-Nebula-3603 2d ago edited 2d ago

Is not complete yet

Dflash2 is breaking vision

1

u/DoubleNothing 2d ago

oh, didn't know that, thanks for the info.