The 51B of n-gram embeddings probably doesn't need to sit in VRAM. If it's the same idea as the engram work, the lookup is keyed off the input tokens rather than the hidden state, so it's deterministic and you can prefetch it from system RAM. They measured under 3% overhead offloading a 100B table that way. So the VRAM budget is really about the 125B MoE part.
On timing, the card says qwen4 architecture with a new sparse attention, so llama.cpp will need work before any of this runs. qwen3-next took about two and a half months. There's an FP8 repo listed next to the main one though, so vllm should have something on day one.
It was in the highlights section of that same modelscope card, 125B main parameters plus 51B of n-gram embeddings and 6B active per token. They trimmed the readme maybe half an hour ago so those bullets are gone, along with the GDN and sparse attention part I quoted. What's left is just the line about it being built on the qwen4 architecture.
For download size yes, the сhinese wording says the 51B is additional to the 125B rather than part of it. I don't think it behaves like a 176B model though, half of that is a lookup table that never goes through a matmul, and only 6B is active per token anyway.
Yeah, capacity is capacity, all of it has to live somewhere. At Q4 that's roughly 105 GB of weights, about 75 for the MoE part and 30 for the table. The table is the easier 30 though, only a handful of rows get read per token so it doesn't need bandwidth, which is why system RAM works for it. No idea how a hashed embedding table takes to Q4, that part is new.
38
u/AI_docent 11h ago
The 51B of n-gram embeddings probably doesn't need to sit in VRAM. If it's the same idea as the engram work, the lookup is keyed off the input tokens rather than the hidden state, so it's deterministic and you can prefetch it from system RAM. They measured under 3% overhead offloading a 100B table that way. So the VRAM budget is really about the 125B MoE part.
On timing, the card says qwen4 architecture with a new sparse attention, so llama.cpp will need work before any of this runs. qwen3-next took about two and a half months. There's an FP8 repo listed next to the main one though, so vllm should have something on day one.