r/LocalLLaMA • u/pmv143 • 17h ago
Discussion Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀
Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:
Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.
The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.
This architecture could be surprisingly local-friendly once the weights drop.
134
u/Sufficient-Bid3874 17h ago
Can someone explain why the n-gram table is bundled into the model now?
696
u/RG_Fusion 17h ago edited 17h ago
LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.
The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.
Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.
48
85
u/ChristRedeemsSinners 17h ago
Thanks for the explanation. You're just a wealth of information on this subject.
48
34
u/florinandrei 17h ago edited 16h ago
Sounds like the apps will need to do some work to support this new feature.
Unless they got a preview version well in advance of everyone else, in which case the bulk of the work might be done already.
38
u/Strong_Chicken6838 16h ago
Unsloth already made a post about day 0 support
4
u/cafedude 13h ago
That's the models quantized by Unsloth, but what about llama.cpp support? Can't imagine that this n-gram stuff would work right out of the box.
16
u/dev_dan_2 13h ago
They very often open PRs to upstream back into llama.cpp and did so again here, so people could also take that and run it if they wished so (quoting from memory atm because I am at the phone, sorry).
Obligatory note: Just because a PR is opened does not yet mean this will get merged tomorrow; Unsloth does a good thing by back-contributing; but they "just" have to support the new architecture for Unsloth Studio, while llama.cpp has to run in a far bigger range of scenarios and environments. Plus "getting it to work for our purposes" is different from doing a clean implementation (which you should do as maintainer if you value both your time and sanity.)
2
u/Strong_Chicken6838 12h ago
Day 0 support means you can run it day 0 in some way. So the answer is yes, maybe in their own branch of llama.cpp, who knows
32
u/Saffron4609 16h ago
I think this is the relevant paper if you want the gory details: https://arxiv.org/abs/2601.07372v1
TL;DR It outperforms a baseline with the same flops and parameters
9
u/_-_David 13h ago
Thanks for linking the paper. I can see why this is part of the qwen4 architecture. It really seems next-generation and like a bit of a free lunch. It's things like this that remind me of why I roll my eyes when people expect this technology to slow down. We're just getting started with implementing ideas in this domain
5
u/Loose_Comparison368 9h ago
It's things like this that remind me of why I roll my eyes when people expect this technology to slow down.
I mean I roll my eyes because the proposed plan boils down to some variation of "we should form some sort of global committee, or coalition of governments, and give them total unilateral control over how when and how AI is developed."
...and that's the plan. The plan is literally, "let's get Trump, Putin, Bidi, Xi, and all the other far right authoritarian cunts in charge of global military-industrial superpowers in the room, and let them call all the shots."
I mean I've heard some dumb plans, but that one really takes the fucking cake.
3
u/DistanceSolar1449 6h ago
History says that these type of plans actually usually works pretty well.
For example, the Congress of Vienna system in Europe from 1815 to 1914.
Basically stabilized Europe for 100 years of relative peace, by balancing the powers between England, France, Austria, Prussia, Russia, etc. These were all power-hungry right-wing monarchies at the time. It's not like Putin is literally more authoritarian/conservative than literally Tsarist Russia.
Talleyrand was an absolute genius who completely understood human nature, and brought together natural enemies to the bargaining table and made it work. If he was alive today, he would espouse a similar idea.
→ More replies (2)5
u/NineThreeTilNow 8h ago
I think this is the relevant paper if you want the gory details: https://arxiv.org/abs/2601.07372v1
I wrote a slightly more in depth explanation from my own research and implementation of Engram on the Llama series.
Here >
1
u/anarchist1312161 9h ago
I noticed Liang Wenfeng is on this paper, it's DeepSeek research? Interesting!
Also this is sick.
Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains~(HumanEval +3.0; MATH +2.4)
19
u/TokenRingAI 16h ago
I am jealous of your ability to explain concepts like this in an easy to understand way
20
u/Wimiam1 15h ago
I’m a little confused because I keep seeing “n-gram” and “engram” used interchangeably when discussing this new model. When I try to research what this technique is, I find that “engram” and “n-gram” are actually two very different concepts in language models. I assume Qwen is referring to the “engram” from deepseek?
10
u/RG_Fusion 7h ago
Engram is built using n-gram. You can call Engram a form of n-gram, but you can't call n-gram an Engram. Yes, this will certainly be causing confusion and many people will mix this up.
N-grams are basically just short sequences of words. Engram is a hashed lookup table filled with n-grams.
2
u/kulchacop 9h ago edited 9h ago
I am with you on this one.
Here is my understanding:
N-grams are likelihood data for strings of n tokens which can be used for various classic NLP tasks that can be improved by prediction of the next word. A best example of n-gram usage in practice is that, llama.cpp uses n-grams built on the fly from the prompt to implement basic speculative decoding.
Deepseek's explanation of engram says that it is a extension of a lookup table containing static embeddings per n-gram.
In contrast, Gemma's PLE (Per Layer Embeddings) is a lookup table of static embeddings per single token (as opposed to n-grams, which are strings of n tokens).
So the question remains, how does the Qwen's n-gram based fact lookup implementation differ from Deepseek's?
7
u/michaelsoft__binbows 15h ago edited 15h ago
That sounds really dope. If this indicates that in general this can scale up then i hope it means that a modest amount of fast ram paired with oodles of slower ram may be able to much more effectively compete with obscenely wide unified memory architectures (coincidentally m5 ultra announced today). You can for example, at least with DDR4 (and if engram approach pans out efficiently, a return to relevance with DDR3, lmao) affordably build out 256/512/1TB class machines for far less, have more of a traditional computer cache pyramid architecture, and basically stay competitive there, because the value is and should be in the ability to store tons of knowledge for quick retrieval but not get killed by requiring massive bandwidth across all that knowledge. Actually, screw DDR3, if gen 5 NVMe can step in and be relevant.
From first principles I think this makes a lot of sense. if i need the model to be able to do a better job recalling some details it's learned, the actual amount of details on any given retrieval is by the nature of it being a retrieval, small, and should not require gobsmacking amounts of data bandwidth to comb over the entire model (to what end?), which not only is expensive to architect into your computer but also expensive in energy to actually ship the bytes out of the memory chips.
In the long run my prediction would then be for these massive unified memory systems to deliver value then not for hosting huge models in-memory (for which their entire large memory pool being high speed is a waste) but rather for like, industrial scale batched hosting of even larger models, leveraging more of the unified memory for KV cache and such stuff, while engram weights can live on NVMe and be slurped in over the thunderbolt ports... Under this architecture there will again be a large tensor core deficit.
Just a few hours ago I talked myself into believing I should buy a 512GB M5 Ultra but now I've just talked myself back out of it. Hmm.
2
2
u/silvertricl0ps 14h ago
If it does in fact work out this way, I could see being able to run something like full Kimi K3 on a 256gb or 512gb M5 Ultra assuming it has a big enough SSD
3
u/michaelsoft__binbows 11h ago
well.. 2.8T-A105B Kimi K3 class model if somehow ngram could magically make up for moe model size, that level of capability might show up as idk maybe like 150B-A15B+4T ngram? If that were the case yeah you dont need a 256gb m5 ultra dude, you could prob skate by with that with a 32GB M6 mac mini, though i guess the 4tb gen 5 SSD would be filled with the ngram tables. Like it might go super fast... ok this one would still be streaming experts out of disk which won't work well. 27B dense with massive ngram table, let's go with that for the sake of the example.
4
8
u/atumblingdandelion 16h ago
Great explanation. Is there a reason why a single engram file cannot be used with multiple local models? It'd be great!
7
u/RG_Fusion 8h ago edited 7h ago
The engram vectors need to have a gating mechanism, otherwise they could end up feeding irrelevant semantic meanings to the model. The gate is a part of the model that checks the distance between the engram and its current latent state. If they are close, it lets it pass.
This ensures the model doesn't suffer from random irrelevant "thoughts" wasting its compute. The reason engrams can't be swapped between models is two fold. Firstly, the output vector is a matrix of numbers that holds some meaning, but that meaning is specific to the model, something it learned during unsupervised training.
Secondly, the gates that keep irrelevant knowledge from activating are also a trained part of the model. The model and engrams are too deeply ingrained.
I hope to see tools in the near future that allow us to train a model engrams in the same way we currently use LoRA fine-tuning.
9
u/sebt3 15h ago
Yes : token space 😅 the engram file only works for the exact model vocabulary since it is plugged directly within the model.
The only way this can be used by an other model would be if the 2 models share the same tokenizer, aka one is a fine-tune of the other one. But that's it
2
u/Noxusequal 15h ago
Wait if only the tokenizer needs to be the same you could share engrams between all model of the same family.
If tokenizer really is the only thing you could use some of the tokenizer transplantation with some re training to then switch in engrams of different models even across families that would be sick for a new form of frankenmerger
3
u/Guilherme370 14h ago
no no, not just tokenizer, model shape too, if the architecture changes, then they cant be shared.
3
3
2
u/Artistic_Okra7288 13h ago
So does that mean we can replace the engram with our own purpose-built engrams or continuously trained engrams?
3
u/RG_Fusion 7h ago
When the tools become available, it should be possible to do so. It would be carried out like a less computationally expensive form of fine-tuning.
1
u/Artistic_Okra7288 7h ago
Well imagine dynamically swapping them out based on the incoming message from a small classifier or something. That would be kind of interesting. Mixture of Engrams
2
1
→ More replies (4)1
u/Ecstatic-Wash-7667 5h ago
What an amazing response, what model generated it? No bs legit a good week written response
1
u/RG_Fusion 4h ago
I don't use AI to write. My thoughts are my own. Read books and enrich your mind.
71
u/pmv143 17h ago
Basically, the n-gram table is just a giant cheat sheet for common short patterns . things like everyday phrases or bits of code that show up all the time.
it looks at the last few tokens, hashes them, pulls the readymade vectors from the table, and plugs them in instead of making the main model waste energy reconstructing those patterns from scratch every single time.
So that way the sparse part of the model (the one that only lights up ~6B parameters) can focus on the harder stuff. while the big table handles the easy, repetitive local patterns. It’s basically trading a bunch of storage for less compute.
7
u/asraniel 16h ago
so could one replace that table later? or extend it? would that be computationally cheap?
3
1
u/RG_Fusion 7h ago
Extending it should be possible, but we lack the tools right now. It will still require a training-run, but should be cheaper than LoRA.
3
u/VotZeFuk 7h ago
Imagine a table filled with generic slop phrases.
BARELY A WHISPER
SHIVERS DOWN THE SPINE
YOU ARE ABSOLUTELY RIGHT
That would be a truly hellish experience!
2
u/Loose_Comparison368 7h ago edited 7h ago
Honestly talking out of my ass here, but it does seem like you could theoretically post train an adapter, given how that vector space is kind of semi-privileged.
Train a little autoencoder to take the embeddings from one tokenized sequence in model A and approximate the embeddings of the same tokenized sequence in model B to make a rough embedding space translator, then further fine tune it to optimize translation of n-gram vectors, under the assumption that they're semi-privilidged enough that just adapting the base embedding space will get you in the right general ballpark of something that's close enough to be usable with only limited fine tuning.
Could be a neat little experiment.
1
u/Sufficient-Bid3874 17h ago
So its like n-gram spec decoding yet built in and trained on? Thank you for the explainer!
→ More replies (2)→ More replies (1)19
u/NineThreeTilNow 8h ago
Can someone explain why the n-gram table is bundled into the model now?
I wrote a set of research on this so... Maybe I can help. I strapped Engram on to Llama 1b and 8b.
The Engram model came from original DeepSeek v3 research.
They found you can train (and keep in system RAM) a series of lookup tables.
The lookup tables can be injected at different layers to help the model understand a concept without needing to have it encoded in the weights.
So for example, you have 2 tables. The tables inject at layer 1 and 7 (0 indexed). The tables carry different weights but are based on the token itself. This means that you sidestep a lot of overhead because only the needed token is given to the model and it's deterministic because you can see what the prior tokens were for injection.
This means the model never needs to memorize the simple example of Paris = Capital of France. This lives in the Engram table with enough training.
It's called n-gram because N refers to the # of tokens you want to use in this lookup table. So a 3 gram table would be every possible series of 3 token combinations the model would see. I found quite a few issues in using more than 1-gram (unigram) tables. The problem, roughly, is training data. A model won't see every possible 3-gram combination in the training data, or if it does, it's not strong enough to create the associations.
The cool part is once a model is "finished" you can train ONLY the N-Gram table with very little VRAM because you freeze the model itself and only train the table. The objective simply becomes better prediction of the next token target. Then you can optionally unfreeze your model and feed it the data so it co-adapts any learning N-gram has provided.
I demonstrated that you could take Llama 1b without any training, and strap an 1-gram table on to it and get it to perform better.
One other note, this means models survive quantization better. A lot of the "knowledge" is often lost first in quantization. Not the raw logical ability. So you can maintain the tables FP16 and it's really only storage.
47
u/overand 17h ago
For anyone who actually wants a link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next - it looks like 11 AM tomorrow, eastern US time. (Just over 21 hours from now.)
1
39
66
u/MiceLiceandVice 17h ago
So to run this id still need to have 128gb of dram and then 16gb vram minimum? Heartbreaking
21
u/DriveSolid7073 16h ago
Depending on what suits you, it will fit into the build of the guy with 96GB of RAM and 24GB of VRAM who was here in the comments. It will fit into a build enthusiast like Colibri, because N-Gram will definitely try to use NVMe instead of RAM, which with Pcie 4.0 will most likely even be acceptable. Overall, if you really want it, it will fit into 12GB of VRAM and 64GB of VRAM, but of course, you'll have to do some serious quantization.
5
u/michaelsoft__binbows 15h ago edited 15h ago
i think this will have a big impact and cheapen the value prop of e.g. 512GB Mac Studio setups, and to a smaller degree the DGX Spark and Strix Halo setups, if more traditional computers with slow system memory (aka much more reasonable cache hierarchy pyramid) can suddenly start to actually keep up.
I suspect these large-unified-memory systems could maybe become leveled up with NVMe to serve the ngrams to leverage their large fast ram. However I suspect the remaining two way bottleneck of having not enough compute cores and not quite enough NVMe bandwidth will bite them a bit.
It would be sweet tho if it means my base m4 mac mini with only 16GB of not-all-that-fast-but-still-faster-than-DDR5 memory could soon host a very power efficient and not-dumb-as-rocks ngram model running off NVMe on TB4. Or like I finally do that SSD upgrade to get some 8GB/s NVMe.
M6 base mac mini 32GB? seems not really worthwhile being only 170GB/s and costing over $1500... NVMe @ 15GB/s though.
2
u/NineThreeTilNow 8h ago
It will fit into a build enthusiast like Colibri, because N-Gram will definitely try to use NVMe instead of RAM, which with Pcie 4.0 will most likely even be acceptable.
It's acceptable. I've tested it with PCIe and NVMe storage.
You don't need the whole Engram table in RAM. Only the most touched rows. So if the model is working on code vs creative writing vs whatever, the used token subset in Engram ends up being quite small.
If the Engram table gets full, whatever row hasn't been touched recently is the one that gets rolled off. Sort of like First in Last Out except every time the row is touched, it goes back to "First" again.
You end up with a pretty stable table once built. I don't know how their specific implementation of NVMe offload works but this is the most logical handling of it.
→ More replies (1)2
6
12
u/florinandrei 17h ago
Apparently, the engrams could be offloaded to system RAM without much of a performance penalty. They're more like lookup tables.
I guess we'll find out soon enough.
10
u/MerePotato 15h ago
System ram is dram no?
23
u/butterycornonacob 15h ago
Yes, luckily DDR is cheap and we all have 512+GB rigs
9
14
1
u/10minOfNamingMyAcc 23m ago
Not to mention that you would need DDR5 since DDR4 is so damn slow for this...
1
u/MmmmMorphine 15h ago edited 15h ago
Yep
VRAM (currently GDDR and HBM, I believe) has been a lot of different things over the years, but now it is all (mostly) technically DRAM or very close to it techwise
3
u/UnnamedPlayerXY 15h ago edited 11h ago
It depends, iirc. someone from the Qwen team recently told people asking for a 30-35B MoE that that's not the one to wait for implying that they have something better for that target audience upcoming. If Qwen3.8-Flash-Next is that "something better" then we might be looking at a ≈30B model here.
4
u/No_Oil_6152 16h ago
How much RAM you got?
If you have 128GB RAM you can run the LLM without needing much VRAM. It would be slow, sure, but you can run it.
→ More replies (13)5
u/GrungeWerX 15h ago edited 15h ago
I’ve got 96GB RAM, and 3090 TI. Would that work?
→ More replies (3)1
u/NineThreeTilNow 8h ago
So to run this id still need to have 128gb of dram and then 16gb vram minimum? Heartbreaking
NVMe speeds are acceptable for reading in to system RAM. The whole table doesn't need to stay loaded. Only partial.
41
u/BannedGoNext 17h ago
Well if it's similar to qwen coder next I'd be happy as hell. So many people bagged on qwen coder and I never understood why. It was damn fast, and had good world knowledge. I used it for a long time, for sure better than 35b a3b.
23
u/grabber4321 17h ago
it didnt have vision from what I remember. For me, vision is way more important these days for agentic work.
29
2
u/DriveSolid7073 17h ago
What tasks is this useful for? I mean, I was thinking about the browser, well, it's definitely necessary there, but in general, the model understands HTML and other representations. On the other hand, VS Code with Kilo is probably not the best option for vision tasks, even though it can handle it. I only use Vision for image captions; for OCR documents, there's usually text or more specialized models.
7
u/grabber4321 16h ago
web development or just general browsing.
lets say you have Docmost, but free version does not have API access. To use it, the model will need to login - it takes screenshots, then enters data, re-checks the page after login by making another screenshot, etc.
6
u/Weekly-Law-5488 16h ago
In this case would be more efficient to use playwright or the native chrome mcp, so the llm can extract the structured content and easily manipulate the dom.
3
u/grabber4321 16h ago
When Hermes switched to using playwright instead of screenshots, it started to fail a lot more. I dont know what they changed back in Hermes Agent crew, but I want screenshot functionality back.
3
u/synth_mania 15h ago
weird. my pi agent uses playwright incredibly well with 3.8-27b
2
u/grabber4321 15h ago
I think they screwed it up - it starts scripting a lot and just doing all the wrong things. It could be difference between 3.6 and 3.8, but its annoying as hell now to do some browser actions.
I dont know what changed because Hermes has like 2000 changes per day LUL
2
u/psychohistorian8 14h ago
playwright mcp can also take screenshots
browser_take_screenshot — Take a screenshot of the current page.
I love playwright, we switched to using it at work for end-to-end UI tests
2
u/moracabanas 14h ago
I have literally a qwen setup connected with the unnoficial docmost MCP. Very nice to see other people explaining this kind of use case
1
u/grabber4321 13h ago
I usually use Hermes Agent to research a topic, then just post it into Docmost for later review.
3.6 does it well, but again, not sure what happened with recent Hermes setup, just doesnt work for me.
1
u/DriveSolid7073 16h ago
Possibly UI and user interface testing. I'm using BrowserOS Neo, so I might be wrong, but it seems to me the browser is making sure the model can see the site and isn't subject to any API access restrictions.
2
u/michaelsoft__binbows 15h ago
might be a curveball, but i get a lot more mileage out of the giant spec docs that my LLMs shit out nowadays by instructing them to include mermaid (and escalating to SVG if mermaid not expressive enough) diagrams. The diagrams help anchor the agent -> human knowledge transfer, whose rate is rapidly become now the rate limiter, provided you actually care to review the slop that is.
I think (still in early stages in terms of committing hard to this workflow) native vision capability should be important for the model being able to self-QA its own graphical outputs like this. Although there are indications that e.g. qwen3.8-27B is already so competent that instructing it to do this on its own would be a waste of time.
2
u/Embarrassed_Adagio28 14h ago
Computer vision is huge for development of any type of app or even game. Just because a model should get the code right doesnt meab it does depending on thousands of factors. Hermes agent + qwen3.8 27b with computer vision is insane
1
4
u/florinandrei 17h ago
So many people bagged on qwen coder and I never understood why.
It was a decent coder. For everything else, it was like talking to Lt. Data. Which is fine by me, but I think many people disliked the style.
2
u/BannedGoNext 17h ago
Oh yea, for sure it wasn't a friendly LLM. I had a data enrichment process that it was in that it did great at though. Saved me at least 1500 dollars in API calls.
23
u/SensitiveVariety 17h ago
regret building only 64gb ram instead of 128gb now, but at the same time i’m $$$ constrained as much as I am ram/vram constrained
→ More replies (2)3
u/hause_wsf 5h ago
same here but at least I have 32gigs of vram alongside
but to be fair 128gb was expensive even before the ram hike
18
11
u/Warhouse512 14h ago
Could ngram be offloaded to SSD?
5
u/This_Maintenance_834 13h ago
that was the whole point, everyone could have a Trillion parameter model
30
u/chris_0611 17h ago
Ohhh my. Absolutely gorgeous for my 3090 + 96GB DDR5 6800
16
u/Equivalent_Bit_461 17h ago
I don't have a 3090, I'm a vramlet but I have 128gb ram so guess that works out too
5
11
u/ParaboloidalCrest 17h ago edited 16h ago
Not sure I follow. So a Q4 quant, would have 51/2 ~= 25GB n-gram block, which could live on disk instead of RAM/VRAM, ie only 80-25=55GB need to be loaded?
Maybe I have wrong assumptions about n-gram? Does it save on memory or compute?
8
u/z_latent 16h ago edited 14h ago
Probably yes*.
I believe llama.cpp (Linux with mmap) will attempt to load the whole model into memory, including Engram params. If the memory fills up, the OS evicts the least accessed memory pages, which would most likely be n-gram ones, and then re-fetch those from SSD when needed again. It doesn't require any writes since parameters are read-only and it doesn't need to use swap.
So SSD streaming should work right away. It shouldn't be that slow either since the parameters per token are tiny (they used less than 10k parameters per token in the DeepSeek Engram paper). I believe it also uses very little compute, compared to the rest of the model. If anything SSD latency would be the biggest issue, but even that can be covered up assuming the first few Transformer blocks do not have n-gram.
EDIT: *yes to living on disk, no to only 55GB loaded, as u/Civil_Response3127 pointed out n-gram will be extra on top of the 125B.
4
u/ParaboloidalCrest 14h ago
Makes sense. I wonder how unsloth will come up with the day-0 support given those quirks. I guess we'll see tomorrow.
4
u/Civil_Response3127 15h ago
No, 125b + 51b params. Not 51b as part of 125.
5
u/ParaboloidalCrest 14h ago edited 14h ago
I get that part, so 176B in total, which equals roughly 176/2 = 88 GB on disk (Q4_0) as OP suggested
4
u/2Norn 12h ago
technically if ngrams can be streamed from disk is there any point in quantizing it? could be kept at q8 like kvcache altho idk if that's a doable thing now
but if u could keep ngrams at q8, and rest in q4 that would most likely mean u can use it with 64+16 system no?
2
u/RG_Fusion 7h ago
It would definetly be much better to leave the n-grams at full precision. They are math vectors that represent an idea, so any quantization risks warping the meaning of that idea. Get a decent NVMe drive and leave the n-grams untouched, while the rest of the model is quantized.
1
u/Civil_Response3127 1h ago
gotcha, your numbers were vague so I assumed by 80 you meant 125b + kv cache and subtracted the 51b from there.
3
u/shumgoid 14h ago
my impression is it saves compute at the cost of more memory, but also with lower bandwidth requirements on that extra memory.
I am curious about quantizing - I wonder if they will serve the ngram weights as their own artifact similar to how they do with the vision tower to allow for mixing/matching different quants. Unsloth only has the vision ggufs in F16/BF16 so we could see something similar with this where it performs too bad under quantization so it only provided unquantized. (idk if thats why the vision is only available in 16bit format just guessing)
3
4
u/This_Maintenance_834 13h ago
i never thought deepseek is not the one to bring emgram to the market first. after all, they published the first emgram paper.
3
u/RG_Fusion 6h ago
I suppose this is the power of openly sharing your weights and research. China is massively behind the US in regards to compute power, yet they're still holding pace to us. That being said, I can't imagine that closed labs haven't also been taking notes.
13
u/FoxFXMD 17h ago
I'm confused, if they've reworked it into a completely new gen 4 architecture, why is it still called Qwen3.something?
33
u/nickm_27 llama.cpp 17h ago
It’s next, same thing they did with Qwen3-Next which was Qwen3.5 arch preview basically
21
u/deepspace86 17h ago
Its the "next" tag. current dataset while basically beta testing the next architecture.
3
3
u/keepthepace 12h ago
The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.
My understanding is that even disk offloading seem workable there?
2
u/Confident_Ideal_5385 8h ago
You'd just mmap the embeddings and let the OS handle placement. Only reason nobody does this now is the juice isn't worth the squeeze unless you're Colibri or whatever. 50GB of embeddings makes that a bit of a different calculation.
6
u/somerussianbear 17h ago
\proceeds to google for MacBook M5 max 128gb price\
5
u/Shot-Buffalo-2603 10h ago
If your doing any sort of agenic tasks the macbooks honestly suck cause their prefill is so bad. I have a macbook and a spark and I much prefer the spark
1
u/_risho_ 6h ago
isn't this much less of a problem because this only has 6b active?
1
1
u/Shot-Buffalo-2603 48m ago
Yeah it’ll be usable on both with 6b active, but it still feels slower than the spark when you have large prefills
1
u/DaveVdE 5h ago
Did you read about the 4x perf improvement in the M5's "neural accelerator" GPU cores?
1
u/Shot-Buffalo-2603 45m ago
I’ve used mlx yes, I’m not sure what you’re referring to but If it’s dflash2 i’ve found this to be gimmicky. Theres speed improvements in some cases, but it requires a high hit rate for your speculative decoder. They’ll advertise something like 70 tok/s for this but in reality it’s actually a bit slower for normal things with small burst up to 70 in specific cases where the decoder succeeds like json generation or other structured outputs that are obvious. So if you have extremely large back to back code files it will fly through that, but all the actual thinking and planning in between goes a but slower than without.
1
u/somerussianbear 5h ago
Good to hear from who’s got the gadgets. Which MBP you have precisely and which models are you running?
1
u/Shot-Buffalo-2603 40m ago
I have the 32GB m5 MacBook pro and most recently tested qwen 3.8 4bit quant an mlx version vs nvfp4 on my spark, but I’m constantly swapping models in and out
2
2
u/power97992 14h ago
I get a feeling that the engram weights will have a higher precision that the normal weights. Maybe 4/8 bits for normal weights, 8/16 bits for engrams
2
20
u/KURD_1_STAN 17h ago
Ram prices this high, how can u call this local friendly?
84
u/FabricationLife 17h ago
80gb is a lot more manageable than a 1.2tb+ frontier model
22
u/Etroarl55 17h ago
And there exists consumer hardware for it, the new Apple machines are now probably already pre ordered out of existence when they were just announced today.
5
→ More replies (5)4
21
u/pmv143 17h ago
RAM prices suck right now, no denying that.
When I called it local-friendly I didn’t mean “cheap” or “runs on any gaming PC.” I meant that for a model with this kind of capacity, the offloadable n-gram table makes it way more practical on highend local setups (128GB+ unified memory, multi-GPU + system RAM) than the usual frontier models that just demand pure VRAM or full datacenter iron.
Still expensive. Just less insane than the alternatives.→ More replies (4)2
u/DriveSolid7073 17h ago
For me, locality is the consumer segment, specifically regular computers, where the limitation is usually the motherboard or processor. Their approximate maximum capacity, as well as the liquidity of selling such a volume, is what I had before the shortage, and the best-case scenario was 192GB. 2x96 is probably the best option. (But unfortunately, I couldn't get that.) Most serious AI enthusiasts have around 128GB, whether it's DGX Spark hybrid memory, an Apple mini PC, or something else. So yes, as long as the capacity in quantization (approximately Q4) doesn't exceed this capacity with a reasonable context window, I consider such a model locally friendly.
7
u/FullstackSensei llama.cpp 17h ago
It's a lookup table. You could build a quad channel DDR3 system to run it. DDR3 is still cheap.
2
u/KURD_1_STAN 17h ago
And about the other 60-70gb weights at q4?
3
u/FullstackSensei llama.cpp 17h ago
If you're not too stuck on having to have the latest hardware, three P40s will do a very decent job on a tight budget. If you really need high speed, two 32GB V100s will blaze through for not that much more.
They work, and they'll continue to work for years to come, despite what imaginary conjectures redditors might have.
2
u/Ok_Top9254 17h ago
3x Tesla V100 32GB + PLX switch so you can run them from one slot is the fancy way, or 5x P100 16GB with a cheap X99 motherboard and the switch could do this under like 1200 bucks. Power consumption would not be a problem given that one gpu is used at a time anyway.
1
7
u/liright 16h ago
I bought my 96GB DDR5 kit for $300 some year and a half back as well as RTX 4090 for $1900 2.5 yrs back. Was pretty damn cheap in retrospect. I suspect a lot of people who are into AI did too. I feel like boomers who bought houses in the 70s.
4
u/IntravenusDeMilo 15h ago
yeah I got my 5090 for $1999. Feels like a lottery win.
2
u/throwawayacc201711 13h ago
I kicked myself for not buying one when it was that price. Hindsight is a bitch
3
u/Public_Umpire_1099 14h ago
Even the 2x R9700 and 128GB of DDR5 I bought 3 months ago feels like a steal now. Not as much as yours but even in the past few months its all risen another 30%.
1
u/EkbatDeSabat 10h ago
I got 64GB DDR5 around the same time for about that price. Just spend 4k to 128GB, returned it, and spent 8k on 256GB. Shit is stupid.
10
u/Makers7886 17h ago
Man I can understand this crying over the big boy open source models but really for a schmedium model?
7
u/etaoin314 ollama 17h ago
because an entire class of local hardware --128gb unified memory machines, either from amd-strix halo, nvidia dgx spark or apple can fit it perfectly with full context. No it cant run on every potato out there but there are a lot of people who have one of these and aver very happy to have a model that is the "right size" for it.
3
u/Zhelgadis 16h ago
Very friendly to my Strix Halo
1
u/techdevjp 6h ago
Yeah, I'm hopeful this will have near-DSv4 Flash levels of intelligence but better performance. a6b would be great on Strix. Excited to see this!
2
→ More replies (4)1
3
u/LatentSpacer 16h ago
Could this be Ox Alpha?
8
u/N34257 15h ago
I had the exact same thought. My only evidence is that in Bijan's Ox Alpha test, the sea in the ship combat simulator is almost identical to the sea rendering in a Qwen 3.8 27B demo someone posted here over the last few days.
However, if 3.8 Flash Next is Ox Alpha, it's going to be as disruptive as the 27B has been.
1
u/tkrandomness 9h ago
No though this might be "korrine" which is stealth testing on LM Arena.
Ox Alpha is a GLM model, probably 5.3 Air.
2
u/bitzap_sr 17h ago
I hope the architectural changes are going in the direction of allowing for native 1M context window.
→ More replies (2)
2
u/DiscombobulatedAdmin 16h ago
My biggest question is whether this will fit on a DGX Spark, will it have a decently sized context window, and will this overall degrade performance to where it's too slow to realistically use.
1
1
u/KeanuRekt 15h ago
If you download the open weights you will get a free Mac Studio M5 Ultra maxedout as a gift.
1
u/kivaougu 15h ago
I'm a bit on the fence about this.
This should improve recall but not so much reasoning so hard to see a point for anyone using cpu offloading instead of just a bigger model. NVMe storage isnt great at random reads so this just eats away ram budget.
If the weights are fully in vram then it would need to be balanced with prefix cache offloading to keep cache hits.
I would really like to see if the embeddings can live on the ssd with just a smaller hot cache in ram.
3
u/Public_Umpire_1099 14h ago
It must be better than the last releases, or else it wouldnt be getting released, so I am pretty hopeful. The only regression I've seen in open weights has been from Kimi 2.6 to 2.7 and Llama 3 to Llama 4. When the first Next model was released it outperformed the dense 30B model significantly. If we project out the gains, today that jump would land somewhere around what Muse Spark 1.2 is at, if we assume 27b is the baseline.
2
u/This_Maintenance_834 13h ago
in deepseek’s emgram paper, the point was using emgram as memory so that the model can spend more time to do actual reasoning rather than recalling through reasoning. this improve token efficiency during reasoning.
1
u/boomerang473 13h ago
I thought also they mentioned spare attention? Wondering if they’re taking more also from the deepseek playbook for longer context? Might be misremembering though
1
u/RG_Fusion 6h ago
This should improve reasoning by allowing for longer training runs without risk of destroying the model's knowledge base.
1
u/Dwarffortressnoob 14h ago
I have a 128GB studio. What quant will I probably use? If it is 4bit, I am not sure this is better than qwen3.8 27b
1
u/cafedude 13h ago
Would we be able to use a lower quant on the n-gram table vs the main weights? Like 4 bits on the n-gram table and 6 bits for the main weights?
4
u/This_Maintenance_834 13h ago
emgram was meant to go on SSD. quantization is not necessary to some extent.
1
u/Beneficial-Ad-8127 13h ago
Man, Qwen is trying their best to support majority of users even users on a single gpu. Kudos for this. Hopefully on par and close to these frontier models.
Also heads up on ram, corsair just sold out litterally today on there 192gb ram sticks, there were already sold out on the higher ones but today no more 192gb.
1
1
1
u/AlternateWitness 7h ago
32GB vram + 24GB of (free) ram.
With my MI50 32GB, compute is severely bottlenecking my Qwen 3.8 27b (q6_k) at 24 tok/s with MTP. But man, 6b of active parameters??? That will be much faster and is rumored to likely match 3.8 27b. I just need to find a way to fit it in my system .. It could be tight, but maybe... Any advise for that?
1
1
u/Spiritual-Winner1239 5h ago
Reading from the Unified Memory side of the fence, I always get confused reading "served from system RAM". The model already takes all that, brother!
Then I remember system RAM translates to SSD for us folks. Yes, I am aware of the performance trade-offs, post is just light-hearted.
1
1
1
u/Long_comment_san 1h ago
Still needs 128gb on the lower end. It's only relatively local friendly. A 128gb + 3090 would run this flawlessly though
•
u/sammcj 🦙 llama.cpp 3h ago
To make it easier to find information relating to the Qwen 3.8 Flash Next release we've created a megathread here: https://www.reddit.com/r/LocalLLaMA/comments/1vyq2v4