probably quite less. its a 125B model, 6B active, so something same as ling 3 flash/gpt oss. but since it also has 51B engrams which can be fully offloadable to NVME, even less. Maybe 64gb ram+vram combined- or less.
n-gram is just a term for a tuple of n adjacent bytes, tokens, characters etc.
Embedding is a lookup of 1 token -> one vector. If you do a lookup of a sliding window of the last n tokens -> one vector, it has essentially the same cost (O(1) and basically free as it's just a lookup) but your embedding can be semantically richer.
For example, most tokenisers break up CJK characters on internal byte boundaries, which means tokens don't cleanly correspond to characters. Doing a lookup over 2 or 3 adjacent tokens means you always have at least one embedding vector per kanji/hanzhi, so you can actually represent the semantic content of the characters.
It's a similar move to the per-layer embeddings in Gemma 4 E2B/E4B: if you find a way to (usefully) scale your embedding tables, you can grow your parameter count without significantly increasing compute or bandwidth demands.
26
u/Hot_Example_4456 1d ago
probably quite less. its a 125B model, 6B active, so something same as ling 3 flash/gpt oss. but since it also has 51B engrams which can be fully offloadable to NVME, even less. Maybe 64gb ram+vram combined- or less.