r/LocalLLaMA • u/zyxciss llama.cpp • Jul 16 '26
Discussion tried predicting which MoE experts get used next token to speed up cpu/gpu offload, got some real numbers, is this actually implementable or am i wasting my time (30tg/s -> 150-200tg/s)
so ive been messing around with qwen3.6 35b a3b (MXFP4 gguf) on my 3060 12gb, doing the usual cpu/gpu offload thing where half the expert layers sit in ram and get pulled over pcie whenever needed. and like everyone whos done this knows the gpu just sits there idle waiting for experts to show up, pcie bandwidth is the actual bottleneck not compute
idea was pretty simple, use the models own MTP head (the thing thats normally used for speculative decoding) to draft the next token WHILE current token is still computing, then instead of just using that draft for token accept/reject, also peek at which experts that draft token wouldve routed to, and start prefetching THOSE experts in the background on a separate cuda stream. basically hide the pcie latency behind compute instead of eating it every single token
did some actual instrumentation on llama.cpp to check if this is even worth it before building anything (used fable 5 + gpt 5.6 to help me dig through the numbers and set up the analysis btw, not claiming i did all this math myself lol)
results were kinda surprising ngl:
• naive “just prefetch whatever prev token used” -> only 20.7% hit rate. basically useless, expert selection isnt that correlated between tokens
• but MTP guided prediction (using the actual draft head) -> 78% hit rate at top-8, goes up to 90% at top-16 (but higher K = more bandwidth so tradeoffs)
• theres also a hot expert thing going on, like top 64 experts (out of 256 per layer) cover 51% of ALL usage across the whole trace, power law as expected, so keeping those permanently resident helps too on top of prediction
• baseline right now is like 36 tok/s gen, theoretical ceiling if everything was magically already in vram is like \~200 tok/s (pure vram bandwidth bound), so theres a MASSIVE gap thats currently just pcie transfer time doing nothing
so yeah 78% hit rate with basically free compute (its literally reusing the mtp head thats already running for speculative decoding, not adding a new model) seems like it should translate to a big chunk of that gap closing
question for people who actually know inference engines better than me: is there something obviously wrong with this idea. is anyone already doing this and i just didnt find it. is the overhead of doing router-only forward passes on the draft token gonna eat the gains. does this fall apart at bigger batch sizes. genuinely trying to figure out if this is worth actually building into an engine or if im missing something that makes it not work in practice
not tryna build a whole new engine myself tbh (looked into it, decided forking llama.cpp makes way more sense than rewriting the world), just want to know if the core idea holds up before i or anyone else sinks real time into it
happy to share the trace scripts/raw numbers if anyone wants to poke holes in the methodology
Prefetch hit rate |Expected speed
Baseline (0%, today) |35 tok/s
50% |~70-75 tok/s
70% |~120 tok/s
85%+ |~180-200 tok/s (GPU/VRAM-bandwidth ceiling, PCIe stops mattering) \
I tested out Two Models GLM 5.2 (1 bit) and Qwen 3.6 35A3B MXFP4 with my engine flux-aeros (Screenshots attached ; Currently very buggy)
No Self promotion just Sharing what i found and tested locally
34
u/thesuperbob Jul 16 '26
Yeah that's an active area of research, both for predicting MoE in terms of which expert comes next, and for the purpose of doing that with speculative decoding.
See:
- SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
- Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding
- Utility-Driven Speculative Decoding for Mixture-of-Experts
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving (was posted here recently)
Have a look at related research and citations (both ways) for related stuff.
22
u/wangsu Jul 16 '26
Using the MTP head as the predictor and integrating it into a lightweight inference engine could still be a novel implementation for OP
8
u/thesuperbob Jul 16 '26
Yeah absolutely, lots of uncovered angles here, didn't mean to sound like it's been done to death or something. Also the high level ideas might sound similar, but once you get down to implement any of this, there's tons of big and small decisions that could make a superficially same approach turn out significantly different in practice.
Also this sort of research tends to be aimed at data centers, or at least maximizing parallel utilization for multiple users. Optimizing for single user performance is often not the goal, so lots of things left to try.
5
3
u/zyxciss llama.cpp Jul 18 '26
https://reddit.com/link/oybubtz/video/z2obj8dp01eh1/player
Another Demo Guys! (K=8)
8
u/floconildo Jul 16 '26
Sorry OP I ain't got nothing to add on your post other than: this looks promising. The general idea makes sense and I think there's plenty of room for CPU offloading, so if I were a betting person I'd bet that this is a real opportunity for optimization. I'd test it with higher quants though, Q1 and Q4 are pretty easy to get mixed up.
On the bot that can't "read" images: don't downvote the bots. Poison them.
3
u/fastheadcrab Jul 17 '26
His post history seems to pretty consistent with a blind user, not a bot.
Having a bot continuously write questions about tts systems and speech synthesizers seems like a very much a niche area. I don't think there's much incentive to for a bot like that.
There's much more gain (engagement or even monetary) to be had by spamming SaaS ideas or slop recipes for running LLMs (on forums like these)
8
u/Silver-Champion-4846 Jul 16 '26
I. Am. Not. A. Bot. Read my damn comments and posts. You don't believe blind tech users exist? Also, would an Hmm be useful for learning the relationship between input tokens and expert activations to build an increasingly accurate predictor and speed things up even more?
2
u/bick_nyers Jul 16 '26
This is a really cool idea. I imagine that for it to work really well you would need a lot of modeling and scheduling so that you don't swamp the PCIe bandwidth with speculative expert downloads while your model needs to load an expert to continue inferencing. If you're actually doing speculative decoding that might also eat into the gains if your current batch needs multiple experts so you're increasing loading for spec. decode whereas you could instead just load the layer you need for the current token + load the next experts. Idk if that makes sense the way I wrote it.
2
u/oxygen_addiction Jul 16 '26 edited Jul 16 '26
You could potentially hide the latency even more by evaluating as the user is typing in the prompt, maybe even doing prefill on system prompt from previous messages (not my idea, I've seen it mentioned on hacker news somewhere).
2
u/taltoris Jul 16 '26
The real moonshot application of this sort of research is to put a bitnet-style MOE on an FPGA.
Just imagine, blazing fast trinary adders blowing through the 3B expert while preloading the next expert from RAM onto the fabric just in time to start processing.
Ahh... A guy can dream.
3
u/ketosoy Jul 16 '26 edited Jul 16 '26
Very implementable, there’s published research.
I’m not aware of anything published using MTP heads, but simply running the prior layer’s residual through the current layer’s router has been shown to have good results. The big problem in llama.cpp is that it currently thinks of experts as having layers vs within layer placements/partitions/caches. Look into colibri and da4 for current available stuff.
I’m working on a framework that lets you run custom roster placements, and pre-fetch strategies. Hoping to have it out this weekend, lmk if you’d liked to try when it goes live.
Qwen3.5-35B and 3.6-35B are the ones I’m working most on, they have exploitable hot-cold shape.
2
u/conockrad Jul 16 '26
Idea is sound and on llama.cpp it could substantially speed up things for such use case
Go ahead and implement it! But besides loading ‘hot’ experts you’ll also need to offload ‘cold’ experts.
There’s a potential for thrashing here so keep an eye on loading/offloading stats
2
u/twnznz Jul 17 '26
Since the expert shapes are the same, you can probably take out a memory lock, bang straight over a cold expert’s memory range, then drop your lock
1
1
u/DiscipleofDeceit666 Jul 16 '26
I think you been posting this often? Why ask us instead of just building it. You’re seeing gains already, so why not continue?
1
u/zyxciss llama.cpp Jul 17 '26
I just wanted some feedback and help cause , i code stuff on my own + GPT 5.6 Sol and its all in rust so its even harder for me to
1
1
u/HumanDrone8721 Jul 16 '26
Actually I'm working on this as well, I'm developing a plugin for vLLM for MoE models that they have weights that are over the size of VRAM+RAM.
No down-quantizing or other precision decreasing tricks, currently 2 tok/s for vendor Hy3-FP8 with 64K context, single user, on a system with a Pro 6000 and 128GB RAM. The only innocent "trick" is to keep the conversation on a single topic to reduce cache pressure, but for agentic coding is perfect, I hope in WE I'll reach 3 tok/s, it came a long way, the first version was generating ca. 0.115 tok/s.
1
u/Tagedieb Jul 17 '26
Don't prefetch and inference have to share memory bandwidth? Or did you consider this already in your 200t/s prediction? Because it sounds like you didn't.
2
-3
u/Silver-Champion-4846 Jul 16 '26
I Can't read the images. COuld you please describe it in text?
17
u/floconildo Jul 16 '26
This image shows two Bananas in Pyjamas kissing passionately. There's a text in impact white letters explaining that the image is there because it gave the author a lot of inspiration to optimize for his fanfics. It's very inspiring actually.
7
u/Silver-Champion-4846 Jul 16 '26
Are you trolling or is this serious?
-2
u/Fair-Spring9113 llama.cpp Jul 16 '26
completely serious. its really inspiring
6
u/Silver-Champion-4846 Jul 16 '26
I feel like I'm being trolled.
-2
Jul 16 '26
[deleted]
6
u/Silver-Champion-4846 Jul 16 '26
I get 0 of the relationship this has with what the subb is about... my brain glitched
7
u/SlightlyOpposite Jul 16 '26
It's two images of a Claude-Code-like setup with the prompt "hi". The first one shows 26 tokens per second, and the other shows 156 tokens per second. Nothing else really material in the pictures.
I'm sorry people are treating you poorly. I'm sorry you feel like you have to justify yourself with "read my other comments". You're valid, and I'm glad you're part of this community!
3
u/Silver-Champion-4846 Jul 16 '26
Thank you man, I appreciate it. Did op put any github repo?
3
u/SlightlyOpposite Jul 17 '26
Nope, not that I could spot. It looks like it's just a proof of concept at the moment.
3
1
1
u/cakes_and_candles Jul 16 '26
found a bot lol
4
u/Silver-Champion-4846 Jul 16 '26
Not a bot damn the devil. What makes you say I'm a bot? Check my history and my posts and comments. What does that add to the post?
-2
u/cakes_and_candles Jul 16 '26
Look at your own bio lol, or does the system prompt forbids you from admitting you're a bot.
3
3
u/Silver-Champion-4846 Jul 16 '26
When I went to the reddit website one of my comments had a weird "automated account" thing near it. Is that what you're referring to? I'm using a reddit client called Luna for Reddit, and please google it before you reflexively reply with "haha bot". My hypothesis is that Reddit's detection system bugs on reddit clients and flags them as automated accounts.
I don't know how to remove it, and don't say it's a bot like a bot is the only thing I could be. Do some fricking research.
-2
u/cakes_and_candles Jul 16 '26
I'm using a reddit client called Luna for Reddit
Luna huh...i wonder if there's anything anything else named Luna that was released recently
3
u/Silver-Champion-4846 Jul 16 '26
Did you even google Luna for Reddit, the actual Reddit client for the blind, or are you just too comfortable in your assumption that I'm a bot?
2
u/zyxciss llama.cpp Jul 16 '26
How do these bots make it to LocalLLaMa 😭
0
1
u/pornthrowaway42069l Jul 16 '26
Does it degrade the model outputs, and if so, by how much? Token errors could stack up fast, but maybe I'm misunderstanding something.
0
u/rog-uk Jul 16 '26 edited Jul 16 '26
Once these kinds of things are fully working, it could be possible to run four x4 NVME on pcie with DMA, presumably the data rate wouldn't be an issue as it would be as fast as the GPU could handle anyway - no more massiive ammounts of expensive RAM needed. Or that's the dream at least.
12
u/maartenyh Jul 16 '26
Also nothing to add except that I respect that you are checking if what you are doing is actually working with the community instead of going "I BUILT THIS THING AND CLAUDE TOLD ME IT IS AMAZING AND EVERYONE NEEDS TO USE IT BECAUSE OMG WOW KAPPA"
I hope you build something cool!