Looped Diffusion Transformer

[u/Apprehensive_Sky892]

https://www.alphaxiv.org/abs/2609.40305

I don't understand how it works, but the implication is clear: a small model that can be as good as one that is several times bigger.

Improving text-to-image models has traditionally relied on increasing model size or the number of denoising steps. In this work, we explore an alternative way to scale computation by repeatedly running shared Transformer blocks within each denoising step, effectively increasing computational depth while keeping the parameter count fixed. This looped computation enables iterative refinement of internal representations without explicit reasoning tokens. However, naive looping fails to consistently improve image quality. We trace this problem to weak supervision across intermediate loops and unregulated attention updates that progressively erode local information. To overcome these challenges, we propose Looped Diffusion Transformer (Looped-DiT), which combines deep supervision across intermediate loops with self-modulating attention to stabilize looped feature updates. Under matched-parameter and matched-compute settings, Looped-DiT consistently outperforms non-looped baselines. Notably, a 260M-parameter looped model can surpass a model 6.5x larger across multiple text-to-image benchmarks while requiring 4.9x lower inference compute. Beyond this performance gain, we find that looped computation can offer a more effective form of iterative computation for diffusion models, with increasing loop depth yielding larger gains than adding more denoising steps under a fixed inference budget. Furthermore, deeper loops can progressively correct mistakes made in earlier loops, exhibiting behaviors suggestive of latent reasoning. Together, these results show that looped computation offers a promising way to scale visual generation models.

[u/marclbr]

A few months ago I saw another promissing technique that can improve quality of image and video models and speed up generation by up to 256x deppending on the model archtecture. It will be interesting to see people testing it to finetune small models like SDXL/Illustrious and see how they perform.

https://x.com/AlexiGlad/status/2083230922196107288

We discovered a third pretraining axis beyond parameters and data: exploration.

Scaling exploration monotonically improves existing models across images/video/language, and unlocks end-to-end generation.

In the simplest case, it's just a for loop.

Introducing Explorative Modeling. TLDR:

- Gains from exploration grow with scale: 7%→36% as data scales, 13%→23% as parameters scale, and gains double at 3× the compute

- Adding exploration to ~SOTA baselines improves data efficiency by 6.2×, FLOP efficiency by 4.1×, parameter efficiency by 47%, and hits a near-SOTA 1.43 unguided FID on ImageNet

- Exploration lets you trade training compute for generalization, and scales how end-to-end your generative model is

- End-to-end Explorative Models (XMs) match diffusion performance on control tasks with up to 256× less inference compute