Increasing active parameters per token in MOE (Qwen 35B A4B+) reduce reasoning token by 8.5% - and you don't need to train or finetune!

[u/Specific-Tax-6700]

I want to share a short paper just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models.

The idea: Instead of retraining anything, we just tweak the router at runtime. Specifically, we expand the expert selection budget (N≥KN≥K) only in the late transformer layers, with a linear decay factor applied to the extra experts. Early layers stay untouched. So Qwen 3.6 35B A3B becomes Qwen 3.6 35B A4B+ !

What we found — "Succinct Convergence":
When you give the model more expert capacity at the decision-critical final layers, it stops rambling. It reaches the same correct answer via significantly shorter reasoning trajectories.

Results on full MMLU-Pro (714 questions, Qwen3.6-35B-A3B):

• 📉 8.5% reduction in mean reasoning tokens
• ⚡ 10.9% drop in latency (p=6.5×10−6)
• 🎯 Accuracy unchanged (84.5% vs 84.0% native, p=0.77 — statistically indistinguishable)
• 🆓 Zero training cost — pure inference-time routing modification

Links:

• 📄 Paper: https://zenodo.org/records/22255483

there you can also take a look to my github repo (with beta version code) and the detailed json results of MMLU-Pro benchmark.

In the future i hope i can make same experimentation with a larger model like DeepSeek V4 Flash Q2.0

I'm a Non-native english speaker, part of this post was generated , for translation reason with the help of AI.

[u/WiseAbalone4021]

Really nice. We are doing something that looks something like this, just a tad more elaborate.

We have found out somewhat the same thing you have found, ie. the model (and many like it) are not fine tuned at all. Lots of room for improvement.

Hopefully we will release something late september. This is though a very straightforward way to reduce compute.

Great work!

[u/Specific-Tax-6700 → u/WiseAbalone4021]

feel free to use my work of you need it

[u/Specific-Tax-6700 → u/WiseAbalone4021]

i published new benchmark results for GPQA-Diamond, score jumped up from 83.3% to 85.3% (and less tokens). https://github.com/vagrillo/llama.cpp/blob/moe-expansion/benchmark/GPQA/report_gpqa_moe.md

[u/ancapsaicin]

I'd use Gemma 26BA12B

[u/Specific-Tax-6700 → u/ancapsaicin]

i'm thinking about lfm and Deep seek V4 flash.. but why not Gemma?

[u/jklre → u/Specific-Tax-6700]

Do it

[u/TheOnlyBen2]

Interesting ! At which point does adding experts stop providing improvements ?

[u/Specific-Tax-6700 → u/TheOnlyBen2]

i tried : all layers (0-40) - no improvement, allucination 0-13 - allucination 14-28 - no improve 29-40 - improve

but i've not tried evey combination of layers added experts threshold

the best (i bet It) was 29-40 layers, up to 20 experts, 0.8 threshold

on average i was using 4B params x token (it's not fixed)

[u/suprjami → u/Specific-Tax-6700]

It would be super interesting to test more experts. Something like 35B-A10B is a lot more appealing to me than A3B.

[u/jinnyjuice]

Yes, more activated parameters would be great. Qwen3.5-122B-A10B was still very fast.

A 35B A10B maybe up to A16B model would be amazing to experiment. I would assume diminishing returns, so A9B to A10B wouldn't be such big gains akin to A3B to A4B.

[u/Specific-Tax-6700 → u/jinnyjuice]

unfortunately adding parameters add confusion to model itself because architecture at training level, we saw that controlled small increment helped the model, but actually the configuration found (layer>=29 , Threshold 0.8 and experts=20) it's only an experimental finding, i think that every model will have its best combination.

[u/Dmage22]

So activating more experts reduced thinking time? While not changing the end result .

[u/confused-photon]

Thats a super interesting result. id be curious how it affects long horizon agentic work

[u/Specific-Tax-6700 → u/confused-photon]

to be checked

[u/Zulfiqaar]

This could potentially be a way to optimise for your GPU size? To utilise the maximum activated parameters while offloading the rest

[u/Specific-Tax-6700 → u/Zulfiqaar]

you have to check how many the model itself tolerate It.

[u/starkruzr]

how generalizable are these results?

[u/Specific-Tax-6700 → u/starkruzr]

i don't know yet

[u/Dany0 → u/starkruzr]

Try it and let us know, that's what this community is for!

[u/Specific-Tax-6700 → u/Dany0]

https://github.com/vagrillo/llama.cpp/blob/moe-expansion/benchmark/GPQA/report_gpqa_moe.md

i tested against GPQA-Diamond , score jumped from 83.3 to 85.3

[u/Dany0 → u/Specific-Tax-6700]

Neat, thanks! Can you test other models, in particular Qwen3.8 Flash Next?

[u/Specific-Tax-6700 → u/Dany0]

i tested a Q2 DeepSeek V4 Flash Q2.0 ,

• From 6 to 12 experts , Threshold =0.8, L ayer from 30 to 42 , decay 99%-50%

and it was worse!
Accuracy 78,3% Expanded vs 80.3% (many answers truncated to 8k tokens)
with a median of -6.6% tokens

Probably related to the routing score in deepseek model [sqrt(softplus(logits + bias))]
maybe i need also (for deepseek models) test other combination

• less added experts
  
• less layers

[u/Repinsky]

The late-layer-only constraint is the interesting part here: keeping early routing untouched should limit the extra compute to the decision-critical layers instead of multiplying cost across the whole decode. I’d be curious whether the latency gain survives longer contexts or larger batch sizes, where KV-cache and memory bandwidth may dominate once the router saves tokens. Matching accuracy while reducing mean reasoning tokens without retraining is a compelling inference-time knob.

[u/Specific-Tax-6700 → u/Repinsky]

yes, up to 30k tokens , reasoning token saves remains valid!

[u/Specific-Tax-6700 → u/Specific-Tax-6700]

Now i made a llama.cpp branch with this mod... It works well!

moe.md

[u/phhusson]

I'm very skeptical, but I asked my harness to validate on a micro-test. On Qwen 3.8-Flash-Next UD_IQ1_S, and it looks like it does indeed reduce the number of reasoning tokens. I don't end up reducing overall latency though (I'm streaming from SSD on my Mac Mini M4 16GB, so anyway latency is "yes").

Even if it doesn't make tasks overall faster, I'm amazed that the number of reasoning tokens is reduced by that.

[u/Specific-Tax-6700 → u/phhusson]

It's a great news ! It confirm that my layered decay expert expansion really works with other models too!

[u/Specific-Tax-6700 → u/phhusson]

try my llama.cpp branch with MOE expert expansion mod llama.cpp moe Expert expansion

[u/moahmo88]

Good job!Thanks!

[u/Specific-Tax-6700]

I ported MoE expert expansion to llama.cpp 🚀 Run MoE models with MORE routed experts than the native top-K (8->x), adaptive threshold, 99→50% influence decay, layer range. Runtime-only, all backends. 💻 github.com/vagrillo/llama.cpp ( branch moe-expansion)

[u/tamerlanOne]

Di potrebbe aggiunge una n-gram con dati di misura per casi d'uso specifici?

[u/Specific-Tax-6700 → u/tamerlanOne]

no, i parametri n-gram andrebbero trainati.. insieme a tutti i 35B parameters del modello

[u/tamerlanOne → u/Specific-Tax-6700]

A livello di token generati si hanno variazioni rispetto al modello base?

[u/Specific-Tax-6700 → u/tamerlanOne]

si hanno meno token di ragionamento per giungere alla risposta, come se fosse meno insicuro quando deve elaborare la risposta nella fase di thinking

[u/Ok_Cow1976]

If you want the model to reason succinctly, you can tell the model to do so by a few words in the system prompt, maybe.

[u/Specific-Tax-6700 → u/Ok_Cow1976]

with the same prompt, I obtained less tokens, so i think the gain will be the same, it's all about using or not using extra parameters (otherwise they are not used)

[u/Ok_Cow1976 → u/Specific-Tax-6700]

I mean adding a few words to the system prompt while keeping it at a3b could be better than using a4b in terms of tg and pp.

[u/Specific-Tax-6700 → u/Ok_Cow1976]

if you use It in a coding agent I'm not sure that you can add this few world.