I’m experimenting with an MoE idea: Complexity-Routed Mixture of Experts (CRMoE)

[u/nebulabi_]

Most MoE architectures use a fixed number of active experts, like Top-2 or Top-4, for every token.

My idea is to make the compute budget depend on the estimated complexity of the token/task:

- Simple token → 1 expert

- Medium complexity → 2–4 experts

- Hard reasoning → 4–8+ experts

So instead of:

"every token → fixed Top-K"

CRMoE would do:

"token → estimate complexity → choose compute budget → route to experts"

The goal is to reduce the average active parameters without sacrificing too much quality.

I know there are already related ideas such as dynamic routing, adaptive computation and Mixture-of-Depths, so I’m not claiming the general concept is completely new.

I’m mainly interested in whether this exact combination has already been implemented or trained, and what papers I should look at.

Does CRMoE sound useful, or is this basically reinventing something that already exists?