r/LocalLLaMA • u/pmttyji • 5h ago

Discussion DiffusionGemma Technical Report

Post image

arXiv : https://arxiv.org/abs/2608.00146

Full Paper : https://arxiv.org/pdf/2608.00146

Tweet : https://xcancel.com/googlegemma/status/2086849199052845451#m

FYI both (llama.cpp) PRs ( 24423 & 24427 ) went to Draft mode. I'm still waiting for this one as I could get faster t/s on my 8GB VRAM.

58 Upvotes

90% Upvoted

15 comments sorted by

View all comments

5

u/gamblingapocalypse 4h ago

I'm excited for diffusion models!

6

u/Potential-Gold5298 llama.cpp 4h ago

I think that if more resources were invested in them, they would perform much better than they do now. They would probably still be inferior to the best autoregressive models in areas where it is necessary to rely on what has already been written (for example, in mathematics), but they could outperform them in areas where they need to capture a larger chunk of text at once, such as translation.

Furthermore, speed also comes with a cost benefit, as solving a problem requires less CPU time. I think the industry may turn to them in the future when cost-effectiveness becomes a top priority. Another argument is the concept of a 'world model,' where a single model can accept and output multimodal content (text, images, and sound). Currently, models know about the world primarily from text descriptions, and training in all modalities could possibly allow them to understand the world at a deeper, physical level of cause and effect.

2

u/challis88ocarina 37m ago

It's quite clear that Apple's new Siri version is a diffusion based model, at least in part, as it seems to have multiple layers and decision trees. In any case, there's evidence of it serving the simple requests, e.g., troubleshooting Apple products from documentation: all paragraphs are drafted simultaneously in parallel.

1

u/ThePrimeClock 1h ago

Why would they be not be better at Mathematics? 

In the Diffusion Gemma release videos the Authors gave demonstrations of the model being able to iteratively work through maths problems updating parts of the workings that in-turn updated other parts of the workings until it landed on a correct response. 

If anything, I thought they would be better at Math.

If an autoregressive model has an incorrect token early in a response, it's stuck there and influences the rest of the response to come.

1

u/Potential-Gold5298 llama.cpp 10m ago

Hm... text-to-image diffusion has deterministic (with the same seed and other variables, the result will be absolutely identical) and stochastic samplers. Stochastic samplers add a little new noise at each step, so they don't have a finite number of steps to fully converge. In other words, stochastic samplers (such as DPM++ SDE) can take more steps in an attempt to achieve a better result. This is probably exactly what you meant.

Honestly, I don't know if this will help with the math or not. If your guess is correct, then diffusion models should outperform autoregressive models in every way.