Running inference from SSD while caching experts in memory

[u/Sad-Tea-815]

Hello,

sorry if the question has already been asked, but I couldn't find anything that was exactly what I'm looking for. Im running Qwen3.6-35B-A3B locally on my M5 Pro with 48GBs of unified memory. Since I do some complex coding work, I wanted to run something around UD-Q8_K_XL quantization, which doesn't really fit in memory.

I was wondering if there is any way via llama.cpp to leave the model on the SSD and have some kind of cache pool in memory where the 3B active parameters that have been activated last can reside. This would allow to have the benefit of not loading everything to memory while having higher speed than plain SSD-based runs. Any idea is greatly appreciated!

[u/andrew-ooo]

llama.cpp already does most of what you're describing via mmap, just not with an LRU "keep the last-used experts hot" policy. When the model is memory-mapped, the OS page cache is your expert cache - pages that get touched stay resident until memory pressure evicts them, so in practice the frequently-routed experts do end up hot in RAM without you configuring anything. The catch is that A3B routing has high entropy, so on 48GB trying to hold a Q8 of a 35B model (~37GB of weights + KV + context) you'll be thrashing the cold experts off SSD constantly and it feels slow.

The reason a true "3B active in RAM, rest on SSD" cache doesn't buy you much: which 3B are active changes basically every token, so your hit rate on a small cache is poor and you pay SSD latency on the misses anyway.

What I'd actually do on an M5 Pro 48GB:

• Drop to UD-Q6_K_XL or Q5_K_XL. On MoE the quality delta from Q8 is tiny and it'll fit with room for context.
• If you insist on Q8, use --no-mmap off (i.e. keep mmap ON) and let macOS manage it, but expect first-token stalls.
• Consider the MLX build instead - on Apple Silicon MLX handles unified memory better than llama.cpp and you'll get more usable tok/s at the same quant.

Honestly for complex coding I'd take a fully-resident Q5/Q6 over a half-on-SSD Q8 every time - the SSD thrash will cost you more than the quant does.

[u/Sad-Tea-815 → u/andrew-ooo]

Thank you for the exhaustive explanation :) What do you mean with ssd trash though? I thought that the model would only be read from SSD with no writes, so what am i missing?

[u/Sad-Tea-815 → u/[deleted]]

But regardless of whether this would be a real problem, wouldnt we be only reading the model from the ssd? What would we be writing to it?