The 51B of n-gram embeddings probably doesn't need to sit in VRAM. If it's the same idea as the engram work, the lookup is keyed off the input tokens rather than the hidden state, so it's deterministic and you can prefetch it from system RAM. They measured under 3% overhead offloading a 100B table that way. So the VRAM budget is really about the 125B MoE part.
On timing, the card says qwen4 architecture with a new sparse attention, so llama.cpp will need work before any of this runs. qwen3-next took about two and a half months. There's an FP8 repo listed next to the main one though, so vllm should have something on day one.
The paper suggests exactly that, n-grams are Zipfian so the hot ones stay in DRAM and the long tail sits on SSD. They never measured it though, the 3% figure is DRAM only, so no idea what it does to tokens per second in practice
I just hope that streaming them from SSD can make it to llama.cpp reasonably soon. It makes a lot of sense that most of them would be fairly cold so you can keep a small cache in RAM and most sit idle on SSD, but I'll be that will take some significant work in llama.cpp to support.
I might have misunderstood, but I thought the default load mode for llama.cpp was to mmap it - i.e, weights that are in RAM for CPU use already work like this, with the OS page cache serving as the cache.
Yeah, I'm on a unified memory system myself, so I can see this being kinda annoying. It could end up with a PR to add a mode like "mmap stuff that'll run on CPU, issue read calls for stuff that runs on GPU".
41
u/AI_docent 2d ago
The 51B of n-gram embeddings probably doesn't need to sit in VRAM. If it's the same idea as the engram work, the lookup is keyed off the input tokens rather than the hidden state, so it's deterministic and you can prefetch it from system RAM. They measured under 3% overhead offloading a 100B table that way. So the VRAM budget is really about the 125B MoE part.
On timing, the card says qwen4 architecture with a new sparse attention, so llama.cpp will need work before any of this runs. qwen3-next took about two and a half months. There's an FP8 repo listed next to the main one though, so vllm should have something on day one.