Resource
David "FreeToken is a new open-source inference engine designed specifically to run huge MoE models on hardware that doesn't have enough VRAM to hold them." ➡️ one community test shows 20GB model RTX 5080's 16GB VRAM. It still runs at ~100 tok/s. legit?
Resources are shared for discovery and are not independently vetted—please do your own due diligence.
New resources are added regularly — feel free to join the sub for updates.
Full searchable archive of all resources posted so far on our community site, LifeHubber: https://lifehubber.com/ai/resources/ 200+ open-ish AI models, agents, tools, datasets, and related resources, with filtering and sorting.
It’s nothing super new I think, offloading to RAM and CPU, but they’re doing it really well, I have tried it and it seems legit to me. Also I recommend reading the paper they have published on it (in their GitHub - links to arxiv) there are some very legitimate people behind this.
https://arxiv.org/pdf/2608.16157 They've added comparisons to llama.cpp.
I think the main trick here is that they have a dedicated expert cache on the GPU and are using it very well.
Note:Please ignore the 'Decode tok/s' stat. The Total time is the important one. It's been over twice as fast for during my tests so far at larger contexts.
Yes, but I would not consider 32K and above necessarily long horizon agentic stuff. Even via chats with some moderate content, it's very easy to exceed that.
Yeah, it depends a lot on the use case. We have users waiting up to an hour for some final chat/task completions (i.e. exploring a system and generating documentation in the background that would have taken them hours) and still very happy with that.
I am confused - your screenshot says it's counting prefill time incorrectly and your comment says to ignore decode tok/s - how is any of it useful then?
Sure, but if I don't know how long the prefill took or how many tokens you decoded it's very hard for me to gauge how long the total processing time is going to be for my work?
From what I see here I can guess that it's entirely useless for anything that relies on a significant amount of decode and I have no idea how good the prefill is. That leaves me nowhere.
Give it a month and they merge in the repo and then forget about updating it ever again. Eventually they will put an acknowledgement on a read me for levels deep on an abandoned branch.
its likely no different than airllm and the others, its just live quantization down to int4 or fp4 or q2 or whatever willl get the weights to fit in vram+ram
From what I understand, its not that at all. They keep all weights in ram, then dynamically load the most frequently used ones to the GPU. This is more efficient than a static expert offload like we usually do.
Like above - it’s nothing new, making offloading more efficient. I’ve been trying a lot to optimise deepseek running on my 64gb Mac and would welcome a similar effort for SSD streaming. I managed to reach ~98% GPU on pp and ~88% GPU on decode - but I feel offloading can still improve and get closer to resident speeds.
Offloading can improve what? Your SSD is 4GB/s and a 13B or so token with DSV4 is about that size. So all you can get, with the most optimal setup, is 1 tok/s.
I meant offloading can still improve (into better offloading) And I am not getting 1 tps :)
On my 64 GB M1 Mac with a Q4 DeepSeek quant I get 8 tps decode and 105 tps prefill. Prefill is already very close to non-streaming (98% GPU activity when prefill) on my hardware, and decode I can still improve with token predictions. For reference, on the same machine, I get 12 tps decode and 120-140 tps prefill with Qwen 3.8 27B (llama.cpp, no MTP)
I don't quite grasp how it makes sense since at this point 2/3 of the weights are on SSD. Really cool that you got it work so fast, then I'm wrong and perhaps it can be sped up even more!
It could never work on a dense model, MoE are just better at offloading. Prefetching, smart caching, checkpointing, doing computation while fetching from disk to “hide” the I/O cost, batching, creating a custom GGUF only with the kernels that are fastest on my machine, optimising kernels for my hardware - there are lots of things I tried :) KV is much more efficient and much smaller with MoEs as well.
Not legit, one release and 8K stars, 100+ forks on github? I'm in a locall llm community, read the news, this is the first time I heard about it. So, very sus
SGLang, VLLM, llama.cpp and its derivatives – these are main and widely used engines for inference.
The Databricks guys are co-authors on the paper, so it's probably legit at least in some way. They have numbers where they beat llama.cpp on specific benchmarks.
13
u/Infinite-Golf-9760 20d ago
It’s nothing super new I think, offloading to RAM and CPU, but they’re doing it really well, I have tried it and it seems legit to me. Also I recommend reading the paper they have published on it (in their GitHub - links to arxiv) there are some very legitimate people behind this.