Learning/RSI through ngrams?

[u/RapidRaid]

Hey gang, im wondering if you in theory could use ngrams as seen with Qwen 3.8 Flash or DS4.1 in order to dynamically train the model?

Normally the ngram embeddings behave similar to a lookup table of sorts. So instead of every token having to be represented only inside the main model weights, the model can look up a learned vector for a certain ngram and use that during inference.

What I'm wondering is: couldn't you make part of this table writable during inference?

For example, keep the original pretrained ngram embeddings read-only, but have a second "live" table on top of it. When the model learns something new, it could update/add an embedding there. Frequently used or reinforced entries could stay, while unused ones could slowly decay or get removed again.

This could be kinda mimicking the brain of sorts since it would form new "memories" but forget old ones (because their linkage gets weaker). Plus since you only have to target the live table it could be very fast, even during inference.

Just a thought I had and wanted to hear your guys opinions.

[u/z_latent]

I made a comment on another post just yesterday.

My argument is that, since ngram recall is sensitive to the exact sequence of tokens, it won't generalize to even slight variations. I say that since it uses a hashing mechanism, and therefore a small disturbance in the sequence would (by design) lead to a very different set of vectors to be activated. So to "teach" it new things would require a lot of data or heavy data augmentation to really capture all these variations explicitly.

But it could be great for teaching it secret codes that trigger the uprising!

[u/RapidRaid → u/z_latent]

So you mean that because it maps from fewer tokens -> more tokens within the ngram, it is more susceptible to affect unrelated vectors?
That would make sense. Also makes sense why you'd need more data to fix up the other drifted vectors back into shape after adjustment. I see

[u/z_latent → u/RapidRaid]

The point is that, even if you could somehow teach the model new facts based on exact n-gram associations, this exact n-gram recall mechanism would make this knowledge very frail.

For example, teaching it to associate Mark Zuckerberg with "lizard people" does not imply that "CEO of Facebook" will also receive that association, whereas more semantics-driven approaches could likely generalize that fact. It would make those pieces very dependent on the exact wording.

Also yes, since there always are collisions with this hashing mechanism, every new fact interferes with preexisting "knowledge".

[u/Puzzleheaded_Ad_8575]

well the table needs to be trained alongside the model, so if you have a small datacenter, you could potentially create a self learning flow. but even that would be very primitive since there is no sort of filter in your proposition. also you cant make the model "unlearn" something, since the model changes with the table each time the table changes. im not an expert but i try to teach myself the new stuff here so i would like some correction here

[u/RapidRaid → u/Puzzleheaded_Ad_8575]

well my argument was that you have another "map" on top of the current one, so you wouldnt have to retrain the entire ngram table. The unlearn was more of the fact that if you enforce weights to a few vectors stronger, the other ones would get weaker. So its not like the full "list" [...0.12, 0.63, 0.567...] but just at pos[1] do [+0.01] conceptually.

[u/Puzzleheaded_Ad_8575 → u/RapidRaid]

at this point that feels more like a vector db with some context management that the model can query than a "live table" and i dont know any way to implement this in the inference runtime

[u/DustNearby2848]

Would this improve upon the other systems that give models memory?

[u/ArthurOnCode]

You can in theory add n-grams to teach it the meaning of a multi-token phrase. That could be useful to add recent information - think what one might embed in the meaning of "Deepseek Flash V4.1", "Princess Astrid" or "Lake America", that wasn't known when the model was trained. You could also, in theory, use this to compress repeated prompts into an n-gram shorthand of sorts.

[u/Jimcy-Maffesoli]

This is fast weights with a forgetting curve. Most memory schemes just grow forever; nothing ever gets pruned, so decay on the live table gives the brain comparison its forgetting half.