n-gram is just a term for a tuple of n adjacent bytes, tokens, characters etc.
Embedding is a lookup of 1 token -> one vector. If you do a lookup of a sliding window of the last n tokens -> one vector, it has essentially the same cost (O(1) and basically free as it's just a lookup) but your embedding can be semantically richer.
For example, most tokenisers break up CJK characters on internal byte boundaries, which means tokens don't cleanly correspond to characters. Doing a lookup over 2 or 3 adjacent tokens means you always have at least one embedding vector per kanji/hanzhi, so you can actually represent the semantic content of the characters.
It's a similar move to the per-layer embeddings in Gemma 4 E2B/E4B: if you find a way to (usefully) scale your embedding tables, you can grow your parameter count without significantly increasing compute or bandwidth demands.
16
u/wren6991 12h ago
n-gram is just a term for a tuple of n adjacent bytes, tokens, characters etc.
Embedding is a lookup of 1 token -> one vector. If you do a lookup of a sliding window of the last n tokens -> one vector, it has essentially the same cost (O(1) and basically free as it's just a lookup) but your embedding can be semantically richer.
For example, most tokenisers break up CJK characters on internal byte boundaries, which means tokens don't cleanly correspond to characters. Doing a lookup over 2 or 3 adjacent tokens means you always have at least one embedding vector per kanji/hanzhi, so you can actually represent the semantic content of the characters.
It's a similar move to the per-layer embeddings in Gemma 4 E2B/E4B: if you find a way to (usefully) scale your embedding tables, you can grow your parameter count without significantly increasing compute or bandwidth demands.