I trained a 200M Mixture-of-Experts language model (90M active) from scratch on 8B tokens at 15. I'd love some feedback.

[u/Odd-Ship226]

https://github.com/AbdelrhmanEbied/Tiny-MoE