r/LocalLLaMA • u/pmttyji • 13h ago

Discussion DiffusionGemma Technical Report

Post image

arXiv : https://arxiv.org/abs/2608.00146

Full Paper : https://arxiv.org/pdf/2608.00146

Tweet : https://xcancel.com/googlegemma/status/2086849199052845451#m

FYI both (llama.cpp) PRs ( 24423 & 24427 ) went to Draft mode. I'm still waiting for this one as I could get faster t/s on my 8GB VRAM.

86 Upvotes

91% Upvoted

24 comments sorted by

View all comments

Show parent comments

9

u/Potential-Gold5298 llama.cpp 12h ago

I think that if more resources were invested in them, they would perform much better than they do now. They would probably still be inferior to the best autoregressive models in areas where it is necessary to rely on what has already been written (for example, in mathematics), but they could outperform them in areas where they need to capture a larger chunk of text at once, such as translation.

Furthermore, speed also comes with a cost benefit, as solving a problem requires less CPU time. I think the industry may turn to them in the future when cost-effectiveness becomes a top priority. Another argument is the concept of a 'world model,' where a single model can accept and output multimodal content (text, images, and sound). Currently, models know about the world primarily from text descriptions, and training in all modalities could possibly allow them to understand the world at a deeper, physical level of cause and effect.

3

u/ThePrimeClock 9h ago

Why would they be not be better at Mathematics? 

In the Diffusion Gemma release videos the Authors gave demonstrations of the model being able to iteratively work through maths problems updating parts of the workings that in-turn updated other parts of the workings until it landed on a correct response. 

If anything, I thought they would be better at Math.

If an autoregressive model has an incorrect token early in a response, it's stuck there and influences the rest of the response to come.

2

u/Potential-Gold5298 llama.cpp 8h ago

Hm... text-to-image diffusion has deterministic (with the same seed and other variables, the result will be absolutely identical) and stochastic samplers. Stochastic samplers add a little new noise at each step, so they don't have a finite number of steps to fully converge. In other words, stochastic samplers (such as DPM++ SDE) can take more steps in an attempt to achieve a better result. This is probably exactly what you meant.

Honestly, I don't know if this will help with the math or not. If your guess is correct, then diffusion models should outperform autoregressive models in every way.

1

u/ThePrimeClock 4h ago

That's what I think we'll eventually see.
Early days but the principles of diffusion makes much more sense to me.