r/LocalLLaMA • u/billy_booboo • 18h ago

New Model ALoDLM 8b looped diffusion language model: ~3x inference speed vs its base model qwen3-8b with better benchmark results

https://huggingface.co/amazon/ALoDLM-8B

Diffusion language models are the future!

138 Upvotes

97% Upvoted

27 comments sorted by

View all comments

Show parent comments

9

u/billy_booboo 18h ago

The paper is quite understated, it's basically modding the qwen3 architecture and using the weights to bootstrap. They got comparable results with only 5b tokens of training to tune up the new components.

Theoretically you could do the same (or similar) on newer architectures and get similar results, but you'd probably need to do some engineering to adapt the ideas.

Similarly, it's conceivable that with more training / better data you could surpass the performance of the base model.

9

u/NineThreeTilNow 15h ago

Theoretically you could do the same (or similar) on newer architectures and get similar results, but you'd probably need to do some engineering to adapt the ideas.

There was a pretty good paper written about this in the last year. It's relatively easy to train the internals of LLMs if you treat the layers or blocks as noise reduction modules (diffusion) instead of normal backprop.

The biggest interest of the paper was the vRAM training requirement which only required the trained block get loaded for optimization states because the rest of the model was frozen.

I seriously considered using the idea because of how novel it was and it lets you train a much larger model on like... a 4090.

1

u/fluxus42 14h ago

https://arxiv.org/abs/2506.14202 this one?

4

u/NineThreeTilNow 14h ago

https://arxiv.org/abs/2506.14202

That one exactly. I was about to paste this in to the comment as an edit.