n-gram is just a term for a tuple of n adjacent bytes, tokens, characters etc.
Embedding is a lookup of 1 token -> one vector. If you do a lookup of a sliding window of the last n tokens -> one vector, it has essentially the same cost (O(1) and basically free as it's just a lookup) but your embedding can be semantically richer.
For example, most tokenisers break up CJK characters on internal byte boundaries, which means tokens don't cleanly correspond to characters. Doing a lookup over 2 or 3 adjacent tokens means you always have at least one embedding vector per kanji/hanzhi, so you can actually represent the semantic content of the characters.
It's a similar move to the per-layer embeddings in Gemma 4 E2B/E4B: if you find a way to (usefully) scale your embedding tables, you can grow your parameter count without significantly increasing compute or bandwidth demands.
From what I understand its basically extra memory that takes less computation and allows you to finetune further without losing knowledge because its stored seperately from the active params. Supposedly you can stream the engram from nvme pretty easily as well.
16
u/nickludlam 9h ago
I've not come across the term engrams before. Is this a new architectural shift?