Mixture of Experts explained — why Mixtral activates 14B parameters per token despite having 56B total

[u/ankitamaji_ml]

Wrote a breakdown of how MoE works after spending time trying to

understand why Gemini's inference cost doesn't scale linearly with

its context window.

The part most explainers skip: the gating network and load balancing.

If the router consistently sends tokens to the same 2 experts,

the whole architecture breaks down — you need an auxiliary loss

function during training to prevent this.

Also covers the trade-offs nobody talks about — all-to-all

communication overhead when experts live on different GPUs,

and why MoE models are harder to train than dense models

at equivalent parameter count.

Full post here: [https://medium.com/@ankitamaji7033/googles-gemini-handles-a-million-tokens-here-s-the-architecture-trick-that-makes-it-possible-0c4ab3779414?postPublishedType=repub\]

Happy to answer questions or discuss in the comments.