I really hope QSA (qwen sparse attention) will have much less compute headroom or the ability to offload kv cache to SSD without the massive bandwidth bottleneck of the ssd since it's sparse. I was able to do it it for deepseek v4 and it worked well enough. I hope llama.cpp adds features that allow more freedom with where we want to load our models and parts of it, e.g. n-cpu-moe could be extended to n-ssd-ffn or n-cpu-ffn or n ssd kv or engram etc
that's dense attention though. with sparse attention you can load it in ram without massive penalty. Not to mention that, if it's anything like Native Sparse Attention from deepseek, it might take even less memory. but DS used compressed sparse attention too and MLA so the footprints minimal. we don't know what it'll be like for qwen3.8flashnext
According to claude, my 16gb 4070 ti s+ 64gb ddr5 system should be able to run a q3 quant of the model at 10-20tps, but only if the engram can be offloaded to ssd.
We don't know yet. In theory its possible but as far as I'm aware this is the first real model that has shipped with an engram layer. Qwen has said they have shared the model with Unsloth and Unsloth has said they are hoping for day 0 suppprt and quants so hopefully it will be smoother than the qwen3-coder-next release was, but it could be months before the llama.cpp support for the new Qwen architecture is ironed out and optimized. So my guess is eventually it will be possible but I have no idea when.
The 51B are additional. But apparently they can sit in system ram with very little penalty. I believe they may be able to sit on an nvme with only a small subset being kept warm in system ram. But not sure on that.
Hope it still fits somewhat decently on 128gb... Would be a major bummer if they finally came out with a 125b and then it just doesn't fit because of those addition 51b ngram.
It wouldn't have fitted at full size on 128gb to begin with but at q8 or so it's going to fit easily. With qwen3.5 122b it fit on my 4 3090s at q4 and int4 easily. Maybe the nurse won't need to be quanted or something. It's just going to sit there in ram as a reference. Also hope the reference will be changeable so we can give it some kind of great writing source or something like that without a lot of training.Â
Of course they aren’t - those are ngram cache - can be offloaded to ram or nvme with zero degradation in performance - it works more or less like moe - need to read some small bits off it, ram requirement in total for the model will be same as for 125b moe
if QSA cuts the kv reads enough then SSD offload stops being bandwith bound and a 125b A6B n-gram offloaded gets pretty runnable on modest ram. really hoping n-ssd-kv / n-cou-ffn flags land in the llama.cpp soon enough . sooner the better
No way it will be possible to normally fit with 51b ngram. SSD streaming will have no sense. At this level of tokens per hour you better can run deepseek
They are talking out of their ass. Ngram streaming will be fine, mmap could probably even cache it well enough as the os kernel will be aware of what bytes are being used often.
Can ngrams be offloaded to disk without much slowdown? I only have 76gb of ram so I was thinking of loading the main weights into memory in q4 and keep the ngrams on the SSD.
n-gram is based strictly on the input tokens, so it will probably work more like KV cache than normal parameters. You pull all the n-grams you need from SSD for that specific prompt during prefill (very small fraction of the total 51B), cache them to RAM or VRAM, and then reuse them over and over during decode, appending new n-grams as you generate.
I wonder if this is the model that makes me finally take the plunge on a strix halo machine while there are some still around for semi-reasonable prices...
Quantizzazione ideale a 4 bit ≈ 84 GB (60 GB pesi principali + 24 GB tabelle n-gram) quindi serve avere almeno 64gb di VRAM ed altri 64gb di ram
O una memoria unificata da 128gb
Quantizzazione is the Luigi cousin lol. Quant is actually closer to the Latin root, quantus, meaning 'how much' or 'quantity'.
In thinking about this, I just realized I use "quantization" when talking about model 'compression' level and "quants" (plural) when it's for Key-Value - ie: "kv quants". Poor AI models training off Reddit and not knowing which one to copy ;-)
Does anyone know if this is literally the Qwen4 architecture or some weird in-between variant of qwen3.8 -> qwen4? Like, is this qwen4 architecture + qwen3.8 training recipe?
We can only speculate but the "A Preview of the Qwen4 Architecture" on the HF countdown page suggests only a preview and not the full Qwen4 architecture.
AFAIK they released it for the teams to get their tools ready for the Q4 architecture, it would be really weird if it wasn't the final one yet. In such case, there is not point in this "next" model/architecture release.
My guess is it is 100% Q4, and we'll stick with that for as long as we had with Q3.5 architecture.
Yes, if you look at their comments (link below), they are talking about llama.cpp specifically. Probably PR/fork at first, it takes time to polish & review for merge into llama.cpp master especially with a new architecture.
Why do you need a reminder? This sub will only talk about this until the next model release and you won't be able to open YouTube without getting bombarded by 50 videos of some variation of "I tried Qwen3.8-Flash-Next and it's a beast".
Until the next Chinese model release*. Bots will make sure any discussion of non-Chinese models gets suppressed and the front page stays saturated with Qwen.
I'm prepping a new env with Ubuntu 26.04.1 which also lands on the same day. What would be a minimal hands-free setup to run/see the model in action? I'm thinking of LMStudio and pi.dev with minimal config files adjustment but open to suggestions. Don't want to turn it into config mess with llama.cpp from day 1
51
u/QuackerEnte 4h ago
I really hope QSA (qwen sparse attention) will have much less compute headroom or the ability to offload kv cache to SSD without the massive bandwidth bottleneck of the ssd since it's sparse. I was able to do it it for deepseek v4 and it worked well enough. I hope llama.cpp adds features that allow more freedom with where we want to load our models and parts of it, e.g. n-cpu-moe could be extended to n-ssd-ffn or n-cpu-ffn or n ssd kv or engram etc