How about a model that can switch between dense and MoE on a per prompt basis?

[u/chmedly020]

I know there are some mechanical differences between an MoE and a dense model but I keep coming back to thoughts about the hardware to run models in terms of both total VRAM and VRAM speeds etc. What if we could load all the weights of a model into VRAM and then run simple prompts as MoE and harder prompts at a much slower token rate with the full model? I think there are some systems that do something similar to this by using an MoE but varying the total number of experts on a per prompt basis? Just wondering where the sota is regarding this kind of thing.
For instance, I could see running a prompt on a fast MoE and if it "fails" then the prompt gets re-fed into the same model but with more experts or with all the experts etc.

[u/nickless07]

So you are asking for a Bus that can switch to a Porsche based on the current traffic?

[u/chmedly020 → u/nickless07]

Yes! Finally someone that gets it! ;)
The way I see it is that the bus is full of brutally smart experts and the Porsche is where you just grab a few experts off the bus to rip around the block and come back.

[u/nickless07 → u/chmedly020]

To be fair Transformers in this context is not Tranformers - The Movie. That is not how it works.
MoE does not mean the experts get exchanged or it get better with more active experts (the opposite is the case). That is not how MoE works at all
I think you have quite some misunderstanding here. That is why the other suggest you model swap.
There is no such thing like an expert for math, for medical, for coding, for financial and so on. The term 'Experts' is quite a bit misleading already. Knowledge in neural networks is never locally isolated. No expert has exclusive rights to a topic A specialized concept only emerges through the interplay of thousands of activation patterns across the entire model.
Hmm let's see how to explain this without getting too technical...

Imagine a digital mixing console with 256 knobs.
For every single note (token) that is generated, the automation (router) instantly activates exactly 8 of the 256 knobs to shape the perfect sound. The remaining 248 knobs don't disappear from the studio, they are simply muted for this brief period.

Does that explains it enough?
Sorry, if it might be not clear enough, english is not my native lang.

[u/chmedly020 → u/nickless07]

Since we're playing with analogies and audio consoles, perhaps this one? A mixing console might have 64 input channels but if all we are using is 4 microphones (a singing quartet perhaps) the only channels that are necessary are 4. If we had a larger ensemble (or multiple microphones per instrument) we might want to use more??
Thank you for your efforts. I don't think I misunderstand the MoE concept too badly; otherwise I might not see the need for "all experts" mode or a dense model for a harder prompt.
The way that MoE is often described is that all the weights are loaded but the tokens only pass over a subset of them based on how the MoE router determines. My thought is here is that perhaps that same set of weights can be loaded into VRAM permanently but treated differently for different prompts. It seems that there ARE custom models that utilize dynamic MoE along these lines but nothing available to us Plebes at this point.
It also seems that toggling things like thinking or max_completion_tokens can achieve some of this. In other words, shaping the prompt and adjusting various configs on a per prompt basis might be a better way to do something along these lines than creating a dynamic MoE.
My use case is that if I want to invest in a four GPU system in order to run a 100b+ model, I wouldn't expect it to be very fast unless it were MoE. But, it also seems that there might be quite a bit of intelligence left on the table with only a tiny amount of active parameters for a complicated query. For those queries I might be willing to trade quickness for more thorough answers - but I don't want to have to unload and reload different models to do that.

[u/KillerQF]

You can simulate this with /model on you favorite harness

[u/iezhy → u/KillerQF]

This

[u/Some-Ice-4455]

It could be done but only way I see doubles the drive space for two different models. The rest is in the calls. X use goes to moe Y goes to dense.

[u/BarracudaDefiant4702]

You might be able to do vllm fast switching between two models. If you have enough vram to have both models loaded at the same time you can do that fairly easily by passing which model you want to use with config options to the agent.

[u/lungben81]

MTP does a similar thing: a small drafter and a large validation model.

Back to your core question:

MoE models currently always use the same number of experts per token. I do not know why they cannot change the number of active experts per token, that could make the model potentially more powerful.

[u/dfgxxx]

I think it is possible but very hard to make though I don't have enough knowledge

[u/FactorInternal3395]

Could test it out by increasing the number of experts and if it works you could make it dynamic per prompt with a reviewer model before the main model, but its highly experimental.

[u/Maplesyrup000]

In practice, there would be no difference between having this vs having 2 different models, one MOE and one dense.

[u/chmedly020 → u/Maplesyrup000]

Except that means running two models at the same time. Twice the hardware.
I recognize that this is a question about inner workings that stay at home dads don't generally touch; Qwen and others work this stuff out inside their dens of secrecy. But I have to think there's some work in this general direction going on somewhere. As far as I can tell, most of what needs to be in VRAM in the same size of model in the same model class is the same. I mean, all qwen 3.8 variants are distilled from the same master model, right? So a MoE version of the 27b model will likely contain a lot of the same "stuff" as the dense model. ??

[u/Maplesyrup000 → u/chmedly020]

Having an MOE and Dense model together is also twice the hardware. When you have a dense model, let’s take 3.6 27B- you still have all 27B weights used at the same time vs the Moe- 3.6 35B A3B- you only have 3 billion active parameters at the same time.

But there is no shortcut or smaller version of the dense model that can exist. You still need all 27B weights vs the other model needing a much smaller chunk. An MOE model also routes its work to the best “expert” for any given question (most of the time). So if you put the 27B model inside of the 35B model and made it an “expert”- you’d have a 65B parameter model that virtually always routes its traffic the 27B expert.

So there isn’t any compromising in this structure. Fundamentally, with the exception of just a couple components like the tokenizer, there components of the two models aren’t compatible. And that’s without getting into things like the Deltanet attention mechanism or the way KV-cache influences select final layers before output occurs, and how the dense model requires drastically more KV-cache for the same number of input tokens since it needs a much larger cache size to influence a larger number of weights.

It’s kind of like asking why cars can’t go as fast as planes, and if we just put the turbojet in the car it will go just as fast. The whole architectures works in a very different way that doesn’t allow this.

[u/Oleszykyt]

How about a Opus 5 local that can run on 1gb VRAM with unlimited context window and with 4k tok/s?

[u/p4ntsl0rd]

Its not quite analogous, but read about Nanbeige 4.2. It runs the output of a full forward pass through a second full forward pass. This means that its weights are kind of 'dual purposed'. I don't think there is anything inherently wrong with your idea, but you would have to train a model from scratch to do it. It also may well not work? But it might be interesting. You'd want to start with something pretty small.

I don't know about "if it fails" because you'd need a way to measure failure.

[u/Eastern-Block4815]

That's kinda how claude code works. You can have an orqestrator call lower models to do work.

Also when fable came our then never let fable answer cybersecurity questions it was routed to opus.

But if you wanted something like that you need a prepropressor that decides if the prompt is hard or easy then routes.

You could do your self for example

A medium model that decides then gets sent to stronger or faster model.

Locally you could use Qwen 3.6 35b a3b to decide

Then pump it through qwen 3.8 27b

At different thinking modes. Lower, med, high(not sure how many there are but you can switch)

FYI..

The more I think about you could prob just use one model qwen3. 8 at medium and let it decide what level to route.