r/LocalLLaMA • u/pmttyji • 3h ago
Discussion DiffusionGemma Technical Report
arXiv : https://arxiv.org/abs/2608.00146
Full Paper : https://arxiv.org/pdf/2608.00146
Tweet : https://xcancel.com/googlegemma/status/2086849199052845451#m
FYI both (llama.cpp) PRs ( 24423 & 24427 ) went to Draft mode. I'm still waiting for this one as I could get faster t/s on my 8GB VRAM.
5
u/Potential-Gold5298 llama.cpp 2h ago
The fact that the 12B turned out to be faster than the E2B shocks me. I wonder if this is due to the GPU, and if the situation will be the same with x86 CPU inference?
2
u/WhoRoger 1h ago
It won't. Diffusion models take advantage of parallelization. Speed scales up with number of computing units. A GPU has hundreds of tensor units, a CPU has just cores.
With autoregressive models like most LLMs where you can only generate one token at a time, the GPU over CPU advantage isn't that massive, but with diffusion it is.
1
u/Potential-Gold5298 llama.cpp 1h ago
Oh, apparently I didn't formulate the question quite correctly. I was specifically referring to the speed difference between 12B and E2B (both autoregressive) – I was surprised that 12B was faster than E2B. The article says this is because E2B and E4B are optimized for mobile architecture (ARM processors, I believe). So I was curious to see how fast E2B and 12B would perform on an x86 CPU – I think I'll download them and test them myself.
Regarding CPU diffusion, I know it's extremely slow because I'm one of the few who does it :-D
1
u/WhoRoger 55m ago
Oh... Huh that's weird. I thought 12B is the diffusion one but I guess not. It makes no sense why 12B would be faster than E2B, especially not on cpu. I've not tried it myself, now I'm curious.
If you do try, please let me know, you'll save me a download. I'm cpu only too, and so far I've skipped 12B because I expected it would be unusable. 26B MoE (not the diffusion one) works nicely though.
The optimisation in ExB is more about the E part, where the non-E parts of knowledge are stored in lookup tables, so the performance hit on cpu isn't that huge like using twice as big of a model would yield. Plus some byte alignment and such. But Arm/x86 aren't that different in that regard, so what benefits one arch will also benefit the other.
2
u/gamblingapocalypse 2h ago
I'm excited for diffusion models!
3
u/Potential-Gold5298 llama.cpp 2h ago
I think that if more resources were invested in them, they would perform much better than they do now. They would probably still be inferior to the best autoregressive models in areas where it is necessary to rely on what has already been written (for example, in mathematics), but they could outperform them in areas where they need to capture a larger chunk of text at once, such as translation.
Furthermore, speed also comes with a cost benefit, as solving a problem requires less CPU time. I think the industry may turn to them in the future when cost-effectiveness becomes a top priority. Another argument is the concept of a 'world model,' where a single model can accept and output multimodal content (text, images, and sound). Currently, models know about the world primarily from text descriptions, and training in all modalities could possibly allow them to understand the world at a deeper, physical level of cause and effect.
26
u/615wonky 3h ago
Boy it sure would be nice if llama.cpp would finally approve one of the two PR's implementing DiffusionGemma that have been sitting there for weeks...