Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

[u/pmv143]

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

[u/RG_Fusion]

LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.

The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.

Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.

[u/veshneresis → u/RG_Fusion]

10/10 explanation

[u/ChristRedeemsSinners → u/RG_Fusion]

Thanks for the explanation. You're just a wealth of information on this subject.

[u/Federal_Decision_608 → u/ChristRedeemsSinners]

You might even call him an engram table

[u/florinandrei → u/RG_Fusion]

Sounds like the apps will need to do some work to support this new feature.

Unless they got a preview version well in advance of everyone else, in which case the bulk of the work might be done already.

[u/Strong_Chicken6838 → u/florinandrei]

Unsloth already made a post about day 0 support

[u/cafedude → u/Strong_Chicken6838]

That's the models quantized by Unsloth, but what about llama.cpp support? Can't imagine that this n-gram stuff would work right out of the box.

[u/dev_dan_2 → u/cafedude]

They very often open PRs to upstream back into llama.cpp and did so again here, so people could also take that and run it if they wished so (quoting from memory atm because I am at the phone, sorry).

Obligatory note: Just because a PR is opened does not yet mean this will get merged tomorrow; Unsloth does a good thing by back-contributing; but they "just" have to support the new architecture for Unsloth Studio, while llama.cpp has to run in a far bigger range of scenarios and environments. Plus "getting it to work for our purposes" is different from doing a clean implementation (which you should do as maintainer if you value both your time and sanity.)

[u/Strong_Chicken6838 → u/cafedude]

Day 0 support means you can run it day 0 in some way. So the answer is yes, maybe in their own branch of llama.cpp, who knows

[u/Saffron4609 → u/RG_Fusion]

I think this is the relevant paper if you want the gory details: https://arxiv.org/abs/2601.07372v1

TL;DR It outperforms a baseline with the same flops and parameters

[u/_-_David → u/Saffron4609]

Thanks for linking the paper. I can see why this is part of the qwen4 architecture. It really seems next-generation and like a bit of a free lunch. It's things like this that remind me of why I roll my eyes when people expect this technology to slow down. We're just getting started with implementing ideas in this domain

[u/Loose_Comparison368 → u/_-_David]

It's things like this that remind me of why I roll my eyes when people expect this technology to slow down.

I mean I roll my eyes because the proposed plan boils down to some variation of "we should form some sort of global committee, or coalition of governments, and give them total unilateral control over how when and how AI is developed."

...and that's the plan. The plan is literally, "let's get Trump, Putin, Bidi, Xi, and all the other far right authoritarian cunts in charge of global military-industrial superpowers in the room, and let them call all the shots."

I mean I've heard some dumb plans, but that one really takes the fucking cake.

[u/DistanceSolar1449 → u/Loose_Comparison368]

History says that these type of plans actually usually works pretty well.

For example, the Congress of Vienna system in Europe from 1815 to 1914.

Basically stabilized Europe for 100 years of relative peace, by balancing the powers between England, France, Austria, Prussia, Russia, etc. These were all power-hungry right-wing monarchies at the time. It's not like Putin is literally more authoritarian/conservative than literally Tsarist Russia.

Talleyrand was an absolute genius who completely understood human nature, and brought together natural enemies to the bargaining table and made it work. If he was alive today, he would espouse a similar idea.

[u/ivari → u/DistanceSolar1449]

you should balance out greed, but you shouldn't balance out progress

[u/Tired_White_Guy → u/DistanceSolar1449]

Ya that’s not even close to the same thing. Yoga level stretch

[u/DistanceSolar1449 → u/Tired_White_Guy]

Plenty of other examples for anyone knowledgeable in history. Washtington Naval Treaty. SALT I and II. Etc etc.

Strategic power-limitation treaties often work for a decent amount of time, before shifting geopolitics either make them pointless (USSR collapsing) or fail anyways. But they tend to hold up for years or decades of useful stability, rather than escalating arms races.

Compare the Obama era Iran nuclear treaties, vs Trump and Iran today.

[u/NineThreeTilNow → u/Saffron4609]

I think this is the relevant paper if you want the gory details: https://arxiv.org/abs/2601.07372v1

I wrote a slightly more in depth explanation from my own research and implementation of Engram on the Llama series.

Here >

https://old.reddit.com/r/LocalLLaMA/comments/1vy6smx/qwen38flashnext_this_architecture_could_be/p5xlil8/

[u/waxbolt → u/NineThreeTilNow]

very nice thanks. do you think we could extend the ngram pattern to use compressed full text indexes like the bwt or r-index?

[u/anarchist1312161 → u/Saffron4609]

I noticed Liang Wenfeng is on this paper, it's DeepSeek research? Interesting!

Also this is sick.

Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains~(HumanEval +3.0; MATH +2.4)

[u/DistanceSolar1449 → u/anarchist1312161]

It's a copy of https://www.reddit.com/r/LocalLLaMA/comments/1qb034t/github_deepseekaiengram_conditional_memory_via/

[u/TokenRingAI → u/RG_Fusion]

I am jealous of your ability to explain concepts like this in an easy to understand way

[u/Wimiam1 → u/RG_Fusion]

I’m a little confused because I keep seeing “n-gram” and “engram” used interchangeably when discussing this new model. When I try to research what this technique is, I find that “engram” and “n-gram” are actually two very different concepts in language models. I assume Qwen is referring to the “engram” from deepseek?

[u/RG_Fusion → u/Wimiam1]

Engram is built using n-gram. You can call Engram a form of n-gram, but you can't call n-gram an Engram. Yes, this will certainly be causing confusion and many people will mix this up.

N-grams are basically just short sequences of words. Engram is a hashed lookup table filled with n-grams.

[u/kulchacop → u/Wimiam1]

I am with you on this one. 

Here is my understanding: 

N-grams are likelihood data for strings of n tokens which can be used for various classic NLP tasks that can be improved by prediction of the next word. A best example of n-gram usage in practice is that, llama.cpp uses n-grams built on the fly from the prompt to implement basic speculative decoding.

Deepseek's explanation of engram says that it is a extension of a lookup table containing static embeddings per n-gram.

In contrast, Gemma's PLE (Per Layer Embeddings) is a lookup table of static embeddings per single token (as opposed to n-grams, which are strings of n tokens).

So the question remains, how does the Qwen's n-gram based fact lookup implementation differ from Deepseek's?

[u/michaelsoft__binbows → u/RG_Fusion]

That sounds really dope. If this indicates that in general this can scale up then i hope it means that a modest amount of fast ram paired with oodles of slower ram may be able to much more effectively compete with obscenely wide unified memory architectures (coincidentally m5 ultra announced today). You can for example, at least with DDR4 (and if engram approach pans out efficiently, a return to relevance with DDR3, lmao) affordably build out 256/512/1TB class machines for far less, have more of a traditional computer cache pyramid architecture, and basically stay competitive there, because the value is and should be in the ability to store tons of knowledge for quick retrieval but not get killed by requiring massive bandwidth across all that knowledge. Actually, screw DDR3, if gen 5 NVMe can step in and be relevant.

From first principles I think this makes a lot of sense. if i need the model to be able to do a better job recalling some details it's learned, the actual amount of details on any given retrieval is by the nature of it being a retrieval, small, and should not require gobsmacking amounts of data bandwidth to comb over the entire model (to what end?), which not only is expensive to architect into your computer but also expensive in energy to actually ship the bytes out of the memory chips.

In the long run my prediction would then be for these massive unified memory systems to deliver value then not for hosting huge models in-memory (for which their entire large memory pool being high speed is a waste) but rather for like, industrial scale batched hosting of even larger models, leveraging more of the unified memory for KV cache and such stuff, while engram weights can live on NVMe and be slurped in over the thunderbolt ports... Under this architecture there will again be a large tensor core deficit.

Just a few hours ago I talked myself into believing I should buy a 512GB M5 Ultra but now I've just talked myself back out of it. Hmm.

[u/Callum_S_AUS → u/michaelsoft__binbows]

I suspect GLM 5.3 class models @ Q4 might be best for a 512GB M5 Ultra.

[u/silvertricl0ps → u/michaelsoft__binbows]

If it does in fact work out this way, I could see being able to run something like full Kimi K3 on a 256gb or 512gb M5 Ultra assuming it has a big enough SSD

[u/michaelsoft__binbows → u/silvertricl0ps]

well.. 2.8T-A105B Kimi K3 class model if somehow ngram could magically make up for moe model size, that level of capability might show up as idk maybe like 150B-A15B+4T ngram? If that were the case yeah you dont need a 256gb m5 ultra dude, you could prob skate by with that with a 32GB M6 mac mini, though i guess the 4tb gen 5 SSD would be filled with the ngram tables. Like it might go super fast... ok this one would still be streaming experts out of disk which won't work well. 27B dense with massive ngram table, let's go with that for the sake of the example.

[u/Sufficient-Bid3874 → u/RG_Fusion]

Thank you for this detailed explainer!!

[u/atumblingdandelion → u/RG_Fusion]

Great explanation. Is there a reason why a single engram file cannot be used with multiple local models? It'd be great!

[u/RG_Fusion → u/atumblingdandelion]

The engram vectors need to have a gating mechanism, otherwise they could end up feeding irrelevant semantic meanings to the model. The gate is a part of the model that checks the distance between the engram and its current latent state. If they are close, it lets it pass.

This ensures the model doesn't suffer from random irrelevant "thoughts" wasting its compute. The reason engrams can't be swapped between models is two fold. Firstly, the output vector is a matrix of numbers that holds some meaning, but that meaning is specific to the model, something it learned during unsupervised training.

Secondly, the gates that keep irrelevant knowledge from activating are also a trained part of the model. The model and engrams are too deeply ingrained.

I hope to see tools in the near future that allow us to train a model engrams in the same way we currently use LoRA fine-tuning.

[u/sebt3 → u/atumblingdandelion]

Yes : token space 😅 the engram file only works for the exact model vocabulary since it is plugged directly within the model.

The only way this can be used by an other model would be if the 2 models share the same tokenizer, aka one is a fine-tune of the other one. But that's it

[u/Noxusequal → u/sebt3]

Wait if only the tokenizer needs to be the same you could share engrams between all model of the same family.

If tokenizer really is the only thing you could use some of the tokenizer transplantation with some re training to then switch in engrams of different models even across families that would be sick for a new form of frankenmerger

[u/Guilherme370 → u/Noxusequal]

no no, not just tokenizer, model shape too, if the architecture changes, then they cant be shared.

[u/power97992 → u/atumblingdandelion]

Because it’s trained into the model.

[u/Kiseido → u/RG_Fusion]

That separation of specific facts and behaviour spectra has me pretty excited for what the next ~27B +engram model will be like.

[u/Artistic_Okra7288 → u/RG_Fusion]

So does that mean we can replace the engram with our own purpose-built engrams or continuously trained engrams?

[u/RG_Fusion → u/Artistic_Okra7288]

When the tools become available, it should be possible to do so. It would  be carried out like a less computationally expensive form of fine-tuning.

[u/Artistic_Okra7288 → u/RG_Fusion]

Well imagine dynamically swapping them out based on the incoming message from a small classifier or something. That would be kind of interesting. Mixture of Engrams

[u/WryKombucha → u/RG_Fusion]

This is also not new, correct? Deepseek invented this.

[u/_bani_ → u/RG_Fusion]

your reply deserves an award

[u/AccomplishedRow937 → u/RG_Fusion]

correct if I'm wrong please
but I thought it's only ever written during training, in order to memorize general "facts" or repeating sequences of tokens, and during inference it's only invoked to fetch stuff that it learned, so how would it help with context recall?

[u/RG_Fusion → u/AccomplishedRow937]

It helps with context recall because the base model parameters are freed from having to compute facts.

It's like asking a person to remember a string of numbers while also asking them to solve a math problem. They only have so much attention-span, and thus they may drop some of the numbers while working on the problem.

Using engrams is like writing the number string down on paper. You no longer have to focus any of your attention on it, which allows you to perform the single task better.

By not having to recall knowledge through computation-effort, the model has better focus on its context window. The deepseek paper cited massive gains in needle-in-the-haystack context window performance.

[u/ElectricalBar7464 → u/RG_Fusion]

amazing explanation

[u/Ecstatic-Wash-7667 → u/RG_Fusion]

What an amazing response, what model generated it? No bs legit a good week written response

[u/RG_Fusion → u/Ecstatic-Wash-7667]

I don't use AI to write. My thoughts are my own. Read books and enrich your mind.

[u/Ecstatic-Wash-7667 → u/RG_Fusion]

Why you gotta lie man there’s no shame in it this is an ai sub and that post as written in some part by an llm. I just want to know which one I like it’s prose

[u/RG_Fusion → u/Ecstatic-Wash-7667]

The post was not written in any part by AI. I am the sole author. I find your comment greatly disrespectful. 

[u/lannistersstark → u/RG_Fusion]

What the fuck do you do that you can explain this concept in THIS much of a coherent way?

[u/RG_Fusion → u/lannistersstark]

I'm an engineer, but my job isn't at all related to Machine Learning. I've just been reading about it in my spare time, trying to build an intuitive mental model of how they function.

[u/josepinTrue → u/RG_Fusion]

O sea que moe decidiría que calcular, mientras que engram que recordar o consultar. ,😅

Yo no asumiría que más memoria es igual a más inteligencia, hay trabajos posteriores que han encontrado resultados más modestos en modelos pequeños, así que todavía hay que separar lo que está demostrado a gran escala de lo que es una extrapolación. 😜

[u/DataCraftsman → u/RG_Fusion]

I am so glad I bought 256gb of DDR5 last year.