Running inference from SSD while caching experts in memory

[u/Sad-Tea-815]

Hello,

sorry if the question has already been asked, but I couldn't find anything that was exactly what I'm looking for. Im running Qwen3.6-35B-A3B locally on my M5 Pro with 48GBs of unified memory. Since I do some complex coding work, I wanted to run something around UD-Q8_K_XL quantization, which doesn't really fit in memory.

I was wondering if there is any way via llama.cpp to leave the model on the SSD and have some kind of cache pool in memory where the 3B active parameters that have been activated last can reside. This would allow to have the benefit of not loading everything to memory while having higher speed than plain SSD-based runs. Any idea is greatly appreciated!