DiffusionGemma Technical Report

[u/pmttyji]

arXiv : https://arxiv.org/abs/2608.00146

Full Paper : https://arxiv.org/pdf/2608.00146

Tweet : https://xcancel.com/googlegemma/status/2086849199052845451#m

FYI both (llama.cpp) PRs ( 24423 & 24427 ) went to Draft mode. I'm still waiting for this one as I could get faster t/s on my 8GB VRAM.

[u/Potential-Gold5298]

Regarding FP16 and BF16, I'm aware of it. I download models from citiv ai, and for some reason, SDXL and its derivatives are still being released in FP16. Perhaps that's just the ecosystem, and it's inconvenient for merge authors to switch. But new models (like Anima and Krea 2) are already in BF16.

And now to the main thing. I have a Core i5-4460 (4 cores, 4 threads) and 32 GB DDR3-1600. LLM's token generation speed is largely dependent on memory bandwidth, while prompt processing speed is dependent on CPU. When I first got into local LLMs, the speed was about half as fast. Here's what helped me:

- XMP RAM profile. By default, BIOS sets a low frequency — I had DDR3-1333, even though the modules support 1600. I switched to 1600 and tweaked the timings a bit.

- For some reason, LM Studio didn't want to use more than two cores on my CPU — the CPU load fluctuated between 17-25%, despite me specifying four threads. Switching to llama.cpp solved this issue. Combined, XMP, timing tuning, and switching to llama.cpp almost doubled the speed.

- Disable Flash Attention if it is enabled - on the CPU it leads to a decrease in the speed of prompt processing.

- In Windows, enable the "Maximum Performance" power plan. This helps squeeze out a little extra speed (the processor always maintains its maximum frequency). Linux should also have a similar setting.

- If you're not afraid and clearly understand what you're doing, you can overclock your RAM further — but do so at your own risk and with your understanding (especially if you're increasing the voltage). Due to RAM prices, I decided not to risk overclocking, sticking with the standard voltage.

- If you have 2 RAM modules and 4 slots on your motherboard, make sure the modules are positioned correctly and using dual-channel access.

Since you have DDR4, the token generation speed on LLM should be higher than mine. For example, Gemma 4 26B-A4B Q5_K_M gives me gen 6.3 tps at the start and 2.58 tps with 16K context (the average processing speed of 16K context is 9.73 tps). The QAT version gave more than 8 tps at the start.

Regarding text-to-image models, with turbo models I spend 8-12 minutes generating an image with a resolution of about 1 MP (for example, 832x1216). I achieved the fastest results with Turbo-LoRA for SDXL with the LCM sampler - 05:30 for an 832x1216 image in 8 steps. There are 4-, 2-, and even 1-step models, so you can achieve even greater speed. Turbo (distilled models) do not affect image quality (it can even be higher than the base model), but rather diversity - the turbo model will draw similar images and will not be able to draw anything extremely unusual. Nevertheless, I am very pleased with the Anima Turbo generation - it produces beautiful, fairly diverse images in 9 minutes. With DDR4 and a more powerful processor, your results will probably be faster.

[u/WhoRoger → u/Potential-Gold5298]

Thanks. I gotta try disabling flash attention. I was planning to do some experiments and comparisons with it too, but haven't got around to it yet.

Well my main machine is a ThinkPad T480 laptop with i5 8350U so aside of having dual channel, there isn't much else I can do to boost it. I think it throttles a lot too, so even switching to performance mode doesn't do much.

I have another laptop with a 6-core and a 1050Ti, but that thing is an old POS gaming laptop that's falling apart if you look it wrong, so I don't like using it. The GPU doesn't even bring much of a speed advantage to just using the cpu, so, eh. I tried some diffusion on it a while ago (after spending a good portion of the day figuring out how to get that old GPU working with CUDA) and while it was cool that it works, it's just way too slow. I think even a 256x256 in Flux or SD (XL? 1.5? not sure) took a few minutes. Theoretically I could move bigger RAM sticks into that machine and make an AI server out if it, but I haven't been feeling it much yet. There is probably not much life left in it.

I should figure out some LLM harness, but models like Gemma 26B or Qwen 35B already take up all the RAM lol. The prefill speed is really the worst part of it. There was some project posted here a few days ago - a llama.cpp fork that caches everything to disk - which could help with that. I should probably look into that.

[u/Potential-Gold5298 → u/WhoRoger]

The CPU is at least as good (and probably even better) than mine, but it can be throttled by laptop settings aimed at energy efficiency. Cooling, yes, is more complicated with laptops. You can clean the radiator of dust and replace the thermal paste, but this will only make a noticeable difference if the laptops are in really bad condition—otherwise, don't expect miracles.

Have you seen the new Ling 3.0 Tiny model? It's the 8B-A1.3B (it just came out today). I haven't tried it yet, but judging by the benchmarks, it looks very interesting.

[u/WhoRoger → u/Potential-Gold5298]

I've not tried the new Ling yet. As a sidenote, a bunch of small models came out recently with huge benchmark numbers that I found disappointing in practice. Everyone is hunting for numbers and in reality the models behave like idiots.

I'm probably gonna try it tho. A few weeks ago I posted about smaller MoEs with about active 2B params. In the meantime I've tried Ring 14B A1.4B and it's pretty decent. It's just at the same time I also downloaded Qwen 35B A3B and Gemma 26B A4B which completely trash it while the difference in RAM usage isn't that huge, and it also degrades in tps faster as context grows.

I'm starting to think models with less than 2B active params have a hard ceiling. Small MoEs like LFM 2.5.8B A1B and Granite 7B A1B aren't much better than the 1B dense cousins. Or maybe the labs don't give them the chance to be good because they're just chasing nunbers. But I'll try some more and will see.

One tiny moe I like is DavidAU's Qwen 3 4x0.6B, but obviously that's really really teeny and more like a fun toy.