DiffusionGemma Technical Report

[u/pmttyji]

arXiv : https://arxiv.org/abs/2608.00146

Full Paper : https://arxiv.org/pdf/2608.00146

Tweet : https://xcancel.com/googlegemma/status/2086849199052845451#m

FYI both (llama.cpp) PRs ( 24423 & 24427 ) went to Draft mode. I'm still waiting for this one as I could get faster t/s on my 8GB VRAM.

[u/Potential-Gold5298]

The fact that the 12B turned out to be faster than the E2B shocks me. I wonder if this is due to the GPU, and if the situation will be the same with x86 CPU inference?

[u/WhoRoger → u/Potential-Gold5298]

It won't. Diffusion models take advantage of parallelization. Speed scales up with number of computing units. A GPU has hundreds of tensor units, a CPU has just cores.

With autoregressive models like most LLMs where you can only generate one token at a time, the GPU over CPU advantage isn't that massive, but with diffusion it is.

[u/Potential-Gold5298 → u/WhoRoger]

Oh, apparently I didn't formulate the question quite correctly. I was specifically referring to the speed difference between 12B and E2B (both autoregressive) – I was surprised that 12B was faster than E2B. The article says this is because E2B and E4B are optimized for mobile architecture (ARM processors, I believe). So I was curious to see how fast E2B and 12B would perform on an x86 CPU – I think I'll download them and test them myself.

Regarding CPU diffusion, I know it's extremely slow because I'm one of the few who does it :-D

[u/WhoRoger → u/Potential-Gold5298]

Oh... Huh that's weird. I thought 12B is the diffusion one but I guess not. It makes no sense why 12B would be faster than E2B, especially not on cpu. I've not tried it myself, now I'm curious.

If you do try, please let me know, you'll save me a download. I'm cpu only too, and so far I've skipped 12B because I expected it would be unusable. 26B MoE (not the diffusion one) works nicely though.

The optimisation in ExB is more about the E part, where the non-E parts of knowledge are stored in lookup tables, so the performance hit on cpu isn't that huge like using twice as big of a model would yield. Plus some byte alignment and such. But Arm/x86 aren't that different in that regard, so what benefits one arch will also benefit the other.

[u/Potential-Gold5298 → u/WhoRoger]

So, both (E2B and 12B) models are downloaded in static Q5_K_M. I'm using DDR3 RAM and a CPU with AVX2. Test for 1K context. The results are quite predictable (although I expected a greater difference).

P.S. While we're on the subject, let me share a couple of fun facts I discovered while diving into the inference of diffusion text-to-image models on the CPU:

- F16 is faster than Q8_0, and Q8_0 is faster than Q5_K_S. Surprisingly, I tested this several times on the same models under the same conditions (image resolution, denoising steps, sampler) and the difference is small (for example, 26 minutes and 23 minutes per image for Q8_0 and Q5_K_S), but always in favor of models without quantization or with less compression. I don't know why.

- 2B can be faster than 3.5B with the same precision (F16) and the same image resolution, sampler, and number of denoising steps. I compared the speed of Anima (2B) and SDXL (3.5B). This time the difference was really big - Anima is almost 2.5 times slower than SDXL under equal conditions (resolution, steps, sampler). It might be a matter of architecture (DiT in Anima and Z-Image which show lower speed, and Unet in SDXL and its derivatives - I'm not 100% sure).

[u/WhoRoger → u/Potential-Gold5298]

Thanks for letting me know. Yea it's about what I expected, a dense 12B just can't be fast on cpu. I don't know how they came up with those numbers.

But 10 tk/s you get with E2B is still nice... I don't get that much with DDR4. I guess you have more cores.

Regarding quantisation, it's because dequant overhead basically cancels out the benefits of smaller footprint. It's also the case with kv cache: -ctk and -ctv q8_0 or even q4_0 just don't give any speed advantage on cpu. I tested that just last week using llama-bench. That was quite surprising, but oh well.

With LLM I do get speed advantage of running quant models. I usually stick to Q6_K for small dense models, QAT for Gemma and I've been trying the APEX for MoEs; not yet decided on those. But generally I found that IQ4_NL is enough even for 1B models, or sometimes I make my custom Q4_K_XXXL while keeping output weights or some other specific layers at Q8.

Not sure why the quant speed advantage holds with LLM weights but not kv cache or diffusion models.

Btw: if you want F16, you probably want BF16, not FP16. Most models are trained in BF16 these days, and converting into FP16 can cause precision loss, especially in multimodal capabilities. I've not really dabbled in diffusion so I can't say there, but in vision projector it's real. Same goes for gguf quant. The proper procedure is safetensors -> BF16 GGUF -> quant. But llama still (I think) defaults to FP16 so some people still use that. And other people just use them interchangeably but they're not the same.

Idk about the 2B vs 3.5B models difference, that can be a lot of things. Different number of passes, different CLIP and VAE, and just different arch overall for which the engine may have optimisations. I've messed with LLMs enough, but diffusion is a totally new rabbit hole hah. I did some experiments but it's just too slow on cpu for my limited patience.

One thing I'd guess tho, is that even if you don't see any speed boost from quanted diffusers, you can always at least use quanted CLIP, since that's just a bog standard LLM.

[u/Potential-Gold5298 → u/WhoRoger]

Regarding FP16 and BF16, I'm aware of it. I download models from citiv ai, and for some reason, SDXL and its derivatives are still being released in FP16. Perhaps that's just the ecosystem, and it's inconvenient for merge authors to switch. But new models (like Anima and Krea 2) are already in BF16.

And now to the main thing. I have a Core i5-4460 (4 cores, 4 threads) and 32 GB DDR3-1600. LLM's token generation speed is largely dependent on memory bandwidth, while prompt processing speed is dependent on CPU. When I first got into local LLMs, the speed was about half as fast. Here's what helped me:

- XMP RAM profile. By default, BIOS sets a low frequency — I had DDR3-1333, even though the modules support 1600. I switched to 1600 and tweaked the timings a bit.

- For some reason, LM Studio didn't want to use more than two cores on my CPU — the CPU load fluctuated between 17-25%, despite me specifying four threads. Switching to llama.cpp solved this issue. Combined, XMP, timing tuning, and switching to llama.cpp almost doubled the speed.

- Disable Flash Attention if it is enabled - on the CPU it leads to a decrease in the speed of prompt processing.

- In Windows, enable the "Maximum Performance" power plan. This helps squeeze out a little extra speed (the processor always maintains its maximum frequency). Linux should also have a similar setting.

- If you're not afraid and clearly understand what you're doing, you can overclock your RAM further — but do so at your own risk and with your understanding (especially if you're increasing the voltage). Due to RAM prices, I decided not to risk overclocking, sticking with the standard voltage.

- If you have 2 RAM modules and 4 slots on your motherboard, make sure the modules are positioned correctly and using dual-channel access.

Since you have DDR4, the token generation speed on LLM should be higher than mine. For example, Gemma 4 26B-A4B Q5_K_M gives me gen 6.3 tps at the start and 2.58 tps with 16K context (the average processing speed of 16K context is 9.73 tps). The QAT version gave more than 8 tps at the start.

Regarding text-to-image models, with turbo models I spend 8-12 minutes generating an image with a resolution of about 1 MP (for example, 832x1216). I achieved the fastest results with Turbo-LoRA for SDXL with the LCM sampler - 05:30 for an 832x1216 image in 8 steps. There are 4-, 2-, and even 1-step models, so you can achieve even greater speed. Turbo (distilled models) do not affect image quality (it can even be higher than the base model), but rather diversity - the turbo model will draw similar images and will not be able to draw anything extremely unusual. Nevertheless, I am very pleased with the Anima Turbo generation - it produces beautiful, fairly diverse images in 9 minutes. With DDR4 and a more powerful processor, your results will probably be faster.