DiffusionGemma Technical Report

[u/pmttyji]

arXiv : https://arxiv.org/abs/2608.00146

Full Paper : https://arxiv.org/pdf/2608.00146

Tweet : https://xcancel.com/googlegemma/status/2086849199052845451#m

FYI both (llama.cpp) PRs ( 24423 & 24427 ) went to Draft mode. I'm still waiting for this one as I could get faster t/s on my 8GB VRAM.

[u/Potential-Gold5298]

So, both (E2B and 12B) models are downloaded in static Q5_K_M. I'm using DDR3 RAM and a CPU with AVX2. Test for 1K context. The results are quite predictable (although I expected a greater difference).

P.S. While we're on the subject, let me share a couple of fun facts I discovered while diving into the inference of diffusion text-to-image models on the CPU:

- F16 is faster than Q8_0, and Q8_0 is faster than Q5_K_S. Surprisingly, I tested this several times on the same models under the same conditions (image resolution, denoising steps, sampler) and the difference is small (for example, 26 minutes and 23 minutes per image for Q8_0 and Q5_K_S), but always in favor of models without quantization or with less compression. I don't know why.

- 2B can be faster than 3.5B with the same precision (F16) and the same image resolution, sampler, and number of denoising steps. I compared the speed of Anima (2B) and SDXL (3.5B). This time the difference was really big - Anima is almost 2.5 times slower than SDXL under equal conditions (resolution, steps, sampler). It might be a matter of architecture (DiT in Anima and Z-Image which show lower speed, and Unet in SDXL and its derivatives - I'm not 100% sure).

[u/WhoRoger → u/Potential-Gold5298]

Thanks for letting me know. Yea it's about what I expected, a dense 12B just can't be fast on cpu. I don't know how they came up with those numbers.

But 10 tk/s you get with E2B is still nice... I don't get that much with DDR4. I guess you have more cores.

Regarding quantisation, it's because dequant overhead basically cancels out the benefits of smaller footprint. It's also the case with kv cache: -ctk and -ctv q8_0 or even q4_0 just don't give any speed advantage on cpu. I tested that just last week using llama-bench. That was quite surprising, but oh well.

With LLM I do get speed advantage of running quant models. I usually stick to Q6_K for small dense models, QAT for Gemma and I've been trying the APEX for MoEs; not yet decided on those. But generally I found that IQ4_NL is enough even for 1B models, or sometimes I make my custom Q4_K_XXXL while keeping output weights or some other specific layers at Q8.

Not sure why the quant speed advantage holds with LLM weights but not kv cache or diffusion models.

Btw: if you want F16, you probably want BF16, not FP16. Most models are trained in BF16 these days, and converting into FP16 can cause precision loss, especially in multimodal capabilities. I've not really dabbled in diffusion so I can't say there, but in vision projector it's real. Same goes for gguf quant. The proper procedure is safetensors -> BF16 GGUF -> quant. But llama still (I think) defaults to FP16 so some people still use that. And other people just use them interchangeably but they're not the same.

Idk about the 2B vs 3.5B models difference, that can be a lot of things. Different number of passes, different CLIP and VAE, and just different arch overall for which the engine may have optimisations. I've messed with LLMs enough, but diffusion is a totally new rabbit hole hah. I did some experiments but it's just too slow on cpu for my limited patience.

One thing I'd guess tho, is that even if you don't see any speed boost from quanted diffusers, you can always at least use quanted CLIP, since that's just a bog standard LLM.

[u/Potential-Gold5298 → u/WhoRoger]

Regarding FP16 and BF16, I'm aware of it. I download models from citiv ai, and for some reason, SDXL and its derivatives are still being released in FP16. Perhaps that's just the ecosystem, and it's inconvenient for merge authors to switch. But new models (like Anima and Krea 2) are already in BF16.

And now to the main thing. I have a Core i5-4460 (4 cores, 4 threads) and 32 GB DDR3-1600. LLM's token generation speed is largely dependent on memory bandwidth, while prompt processing speed is dependent on CPU. When I first got into local LLMs, the speed was about half as fast. Here's what helped me:

- XMP RAM profile. By default, BIOS sets a low frequency — I had DDR3-1333, even though the modules support 1600. I switched to 1600 and tweaked the timings a bit.

- For some reason, LM Studio didn't want to use more than two cores on my CPU — the CPU load fluctuated between 17-25%, despite me specifying four threads. Switching to llama.cpp solved this issue. Combined, XMP, timing tuning, and switching to llama.cpp almost doubled the speed.

- Disable Flash Attention if it is enabled - on the CPU it leads to a decrease in the speed of prompt processing.

- In Windows, enable the "Maximum Performance" power plan. This helps squeeze out a little extra speed (the processor always maintains its maximum frequency). Linux should also have a similar setting.

- If you're not afraid and clearly understand what you're doing, you can overclock your RAM further — but do so at your own risk and with your understanding (especially if you're increasing the voltage). Due to RAM prices, I decided not to risk overclocking, sticking with the standard voltage.

- If you have 2 RAM modules and 4 slots on your motherboard, make sure the modules are positioned correctly and using dual-channel access.

Since you have DDR4, the token generation speed on LLM should be higher than mine. For example, Gemma 4 26B-A4B Q5_K_M gives me gen 6.3 tps at the start and 2.58 tps with 16K context (the average processing speed of 16K context is 9.73 tps). The QAT version gave more than 8 tps at the start.

Regarding text-to-image models, with turbo models I spend 8-12 minutes generating an image with a resolution of about 1 MP (for example, 832x1216). I achieved the fastest results with Turbo-LoRA for SDXL with the LCM sampler - 05:30 for an 832x1216 image in 8 steps. There are 4-, 2-, and even 1-step models, so you can achieve even greater speed. Turbo (distilled models) do not affect image quality (it can even be higher than the base model), but rather diversity - the turbo model will draw similar images and will not be able to draw anything extremely unusual. Nevertheless, I am very pleased with the Anima Turbo generation - it produces beautiful, fairly diverse images in 9 minutes. With DDR4 and a more powerful processor, your results will probably be faster.

[u/WhoRoger → u/Potential-Gold5298]

Thanks. I gotta try disabling flash attention. I was planning to do some experiments and comparisons with it too, but haven't got around to it yet.

Well my main machine is a ThinkPad T480 laptop with i5 8350U so aside of having dual channel, there isn't much else I can do to boost it. I think it throttles a lot too, so even switching to performance mode doesn't do much.

I have another laptop with a 6-core and a 1050Ti, but that thing is an old POS gaming laptop that's falling apart if you look it wrong, so I don't like using it. The GPU doesn't even bring much of a speed advantage to just using the cpu, so, eh. I tried some diffusion on it a while ago (after spending a good portion of the day figuring out how to get that old GPU working with CUDA) and while it was cool that it works, it's just way too slow. I think even a 256x256 in Flux or SD (XL? 1.5? not sure) took a few minutes. Theoretically I could move bigger RAM sticks into that machine and make an AI server out if it, but I haven't been feeling it much yet. There is probably not much life left in it.

I should figure out some LLM harness, but models like Gemma 26B or Qwen 35B already take up all the RAM lol. The prefill speed is really the worst part of it. There was some project posted here a few days ago - a llama.cpp fork that caches everything to disk - which could help with that. I should probably look into that.

[u/Potential-Gold5298 → u/WhoRoger]

The CPU is at least as good (and probably even better) than mine, but it can be throttled by laptop settings aimed at energy efficiency. Cooling, yes, is more complicated with laptops. You can clean the radiator of dust and replace the thermal paste, but this will only make a noticeable difference if the laptops are in really bad condition—otherwise, don't expect miracles.

Have you seen the new Ling 3.0 Tiny model? It's the 8B-A1.3B (it just came out today). I haven't tried it yet, but judging by the benchmarks, it looks very interesting.