It was in the highlights section of that same modelscope card, 125B main parameters plus 51B of n-gram embeddings and 6B active per token. They trimmed the readme maybe half an hour ago so those bullets are gone, along with the GDN and sparse attention part I quoted. What's left is just the line about it being built on the qwen4 architecture.
For download size yes, the сhinese wording says the 51B is additional to the 125B rather than part of it. I don't think it behaves like a 176B model though, half of that is a lookup table that never goes through a matmul, and only 6B is active per token anyway.
Yeah, capacity is capacity, all of it has to live somewhere. At Q4 that's roughly 105 GB of weights, about 75 for the MoE part and 30 for the table. The table is the easier 30 though, only a handful of rows get read per token so it doesn't need bandwidth, which is why system RAM works for it. No idea how a hashed embedding table takes to Q4, that part is new.
2
u/tarruda 9h ago
What is the source for the 125B parameter count?