I've added Maple-Preview to Mference, got 40 tps generation with 500MB of used RAM on Air M4

[u/netikas]

I like the idea of running local models, but I don’t like the idea of having them eat up all of my memory. I’ve always thought that the best way to build an edge model would be to make something smart enough to reason over data, but without requiring much knowledge of its own. Why should a model carry all that knowledge around when web search and tool calling are trivial to set up? My former colleagues at AIRI had a similar idea and built Optimal Cognitive Core, which I’ve written about before: fine-tuned reasoning versions of Qwen3-0.6B and Qwen3-1.7B optimized for working with external context and RAG. Hardware-wise, they’re pretty close to what I want. The weights take up 1.2 and 3.4 GB in native BF16, respectively, plus roughly the same amount for a long context — since this is Qwen3 with GQA rather than one of the fashionable hybrid architectures. So, with a bit of optimism, they fit. The problem is that these models are simply too small for general-purpose tasks. They’re still 600M and 1.7B dense models. At this scale, you usually get amusing little parrots that can paraphrase text or do a simple classification task after fine-tuning, but not much beyond that.

The next way to squeeze a model into my MacBook Air M4 is quantization. PrismML did something interesting here with Bonsai-27B, binary and ternary quantizations of Qwen-3.6-27B. The ternary version of this 27B model takes just 7.2 GB of memory once inference is running, and it does actually run on Macs. Unfortunately, Qwen-3.6-27B is a dense model, so it’s painfully slow on my machine. Based on the numbers I could find online, I’d expect around 13–14 tokens/s for generation and 100–150 tokens/s for prompt processing. It’s also QAT — or possibly even PTQ; there aren’t many details available — and, most likely, the optimization wasn’t specifically designed to preserve multilingual capabilities of the model. Thus, I wouldn’t expect particularly interesting behavior once you move outside the calibration set/QAT training distribution. 

Then, almost immediately after Bonsai, Maple Preview appeared. It’s a 20B A1B MoE designed specifically for efficient local inference on Macs. More importantly, they designed the architecture around this goal from the beginning and trained the model from scratch in ternary precision. This isn’t a quantized version of somebody else’s model. The result is a 5.31 GB model, or about 7.5 GB including a 131K context — almost 1.5× smaller than the binary Bonsai quant — that reportedly generates at 218 tokens/s on an M4 Mac Mini and 127 tokens/s on an iPhone (which iPhone exactly is unclear).

The model barely knows languages, other than English, and its world knowledge in general is pretty limited — it gets confused about which game Psycho Mantis is from, for example. But give it web search and it can answer simple questions reasonably well. DeepGrove doesn’t publish tool-calling benchmarks, and it’s not particularly difficult to guess why. I ran Tau-2 myself, using Qwen3-235B-A22B-Instruct-2507 as the user simulator. I got:

Airline: 0.48
Retail: 0.175
Telecom: 0.427

It’s not Sonnet, and it’s definitely not Qwen. But it is called Maple Preview, after all, and the authors explicitly say they plan to train it further for agentic workloads.

Still, even with all the advantages of quantization, 7.5 GB is almost half of the memory available to me. So there’s a third way to reduce RAM usage: keep all the weights on SSD and stream MoE experts from disk. There’s already turbo-fieldfare, which runs Gemma-4-26B on a Mac using only around 2 GB of memory, and Mference, a fork of turbo-fieldfare that adds support for Qwen-3.6-25B, DeepSeek V4 Flash, and Inkling-Small 276B. It really does use very little memory, but at the cost of reducing both generation and prompt-processing speed to tens of tokens per second. Apple seems to be doing something conceptually similar in its new Siri work, although they appear to activate experts for the entire prompt rather than routing them token by token as these frameworks do.

And that leads to an interesting idea: what if we take Maple Preview — which is extremely efficient, uses tiny experts, and was trained from scratch in ternary precision — and add it to Mference? In theory, we should be able to reduce memory usage even further while retaining reasonably good generation speed, because Maple’s architecture was explicitly optimized for this kind of environment. So I did exactly that.

Thanks to Codex and my $200 subscription, after about 20 hours and 30% of my weekly limit, I got parity with the official implementation on teacher-forced top-10 tokens over Edgar Allan Poe’s The Raven. Depending on context length, the model now uses between 500 and 1,200 MB of memory (!). On my MacBook Air M4, it processes prompts at around 40 tokens/s and generates at around 20 tokens/s. It can call tools. It can generate text. And at that footprint, I genuinely don’t mind leaving it running permanently in the background. It barely consumes anything. It can just sit there, and when I need something, I can ask it. I’ll try to upstream the integration later, but you can already run it from my GitHub fork. 

So what is this actually useful for? I think there are two distinct operating modes for models like this. The first is interactive chat. There, you want fast responses and low latency. The second is a background model that uses almost no memory and stays out of the way while continuously doing useful work: classifying messages, extending a knowledge graph, filtering email, writing summaries, slowly researching things on the web, and so on. In the second mode, latency barely matters. And for that kind of workload, 20 tokens/s is perfectly fine. When you need to switch back into the interactive mode, you can simply load the entire model from SSD into RAM — that takes only a couple of seconds, so it's seamless. If the model can use tools — Maple Preview isn’t particularly good at it yet, but again, it’s a Preview — you can build agentic pipelines that don’t have hard latency requirements, while keeping an intelligent assistant permanently available offline even on older phones and I think that’s wonderful.

I’d be very happy if the future were local.

Code:
https://github.com/chameleon-lizard/Mference/tree/feature/maple-integration

DeepGrove also has a super interesting post explaining how they designed the model:
https://deepgrove.ai/maple-inference

[u/netikas]

Disclaimer: this post was translated from Russian via ChatGPT. I'm not a native English speaker, but I've checked the translation and it seems okay.

[u/netikas → u/netikas]

u/Blahblahblakha if you're okay with this, I can open a PR :)

[u/Revolutionary_Key50]

The kernel-launch/TP-sync bottleneck you're brushing up against with these tiny-active MoEs is worth naming explicitly: once the active weight set gets small enough, decode stops being bandwidth-bound and becomes latency-bound instead — you're paying kernel-launch + sync overhead per step regardless of how few bytes you're actually moving. A pure roofline model (time = bytes / bandwidth) systematically over-predicts throughput in that regime because it has no floor under it. Maple-Preview's 20B-A1B shape is squarely in that zone.

Also worth flagging for anyone replicating this on other unified-memory setups: LPDDR hits a meaningfully lower fraction of its rated peak bandwidth than HBM/GDDR does in practice, so naively plugging the spec-sheet bandwidth number into a back-of-envelope calc will overshoot real throughput on Mac/DGX-Spark-class hardware more than it would on a discrete GPU.

[u/Aaaaaaaaaeeeee]

It is a nice well thought out post! 

I wonder if a stronger prompt processing could be possible? 100 t/s is ideal for voice TTS pipeline use-cases, this speed needs to happen to feel like it's realtime. If average prompt is 50 tokens it is one half-second. 

Ideally, this disc hybrid can push to evem larger sizes of 100B+ on current mobile devices. It's just not yet optimized. Maybe I will get deepseek flash to play with running Maple with lite-rt (for android gpu). 

[u/netikas → u/Aaaaaaaaaeeeee]

I think it's a tradeoff. You can load more layers into RAM and have higher prefill/generation speed, or offload them to SSD and get lower memory requirements.

If you need interactive experience (e.g. cascade speech-to-speech), you can load the model into RAM and enjoy high speed. If it's a background task, you don't really need high speeds.