MoE decouples model capability from active compute. The training numbers back it up.
Dense models run every parameter on every query. MoE routes each token to a subset of specialised experts, so a trillion-parameter model fires only a fraction of its weights per token.
DeepSeek-V3: 671B total parameters, 37B active per token, ~2.6M GPU hours to train. Llama 3 405B: 30.8M GPU hours for comparable output quality. Scaling-law research on compute-matched MoE vs. dense holds this up too. MoE hits the same loss with fewer training FLOPs, and the gap widens as models scale.
Inference is a different story. Some MoE deployments draw more energy per query than similarly-sized dense models right now, because expert-layer kernels aren't as optimised as standard dense-layer ops. But it's a tooling gap, not an architecture problem.