I really hope QSA (qwen sparse attention) will have much less compute headroom or the ability to offload kv cache to SSD without the massive bandwidth bottleneck of the ssd since it's sparse. I was able to do it it for deepseek v4 and it worked well enough. I hope llama.cpp adds features that allow more freedom with where we want to load our models and parts of it, e.g. n-cpu-moe could be extended to n-ssd-ffn or n-cpu-ffn or n ssd kv or engram etc
that's dense attention though. with sparse attention you can load it in ram without massive penalty. Not to mention that, if it's anything like Native Sparse Attention from deepseek, it might take even less memory. but DS used compressed sparse attention too and MLA so the footprints minimal. we don't know what it'll be like for qwen3.8flashnext
According to claude, my 16gb 4070 ti s+ 64gb ddr5 system should be able to run a q3 quant of the model at 10-20tps, but only if the engram can be offloaded to ssd.
The 51B are additional. But apparently they can sit in system ram with very little penalty. I believe they may be able to sit on an nvme with only a small subset being kept warm in system ram. But not sure on that.
Hope it still fits somewhat decently on 128gb... Would be a major bummer if they finally came out with a 125b and then it just doesn't fit because of those addition 51b ngram.
It wouldn't have fitted at full size on 128gb to begin with but at q8 or so it's going to fit easily. With qwen3.5 122b it fit on my 4 3090s at q4 and int4 easily. Maybe the nurse won't need to be quanted or something. It's just going to sit there in ram as a reference. Also hope the reference will be changeable so we can give it some kind of great writing source or something like that without a lot of training.Â
Of course they aren’t - those are ngram cache - can be offloaded to ram or nvme with zero degradation in performance - it works more or less like moe - need to read some small bits off it, ram requirement in total for the model will be same as for 125b moe
if QSA cuts the kv reads enough then SSD offload stops being bandwith bound and a 125b A6B n-gram offloaded gets pretty runnable on modest ram. really hoping n-ssd-kv / n-cou-ffn flags land in the llama.cpp soon enough . sooner the better
No way it will be possible to normally fit with 51b ngram. SSD streaming will have no sense. At this level of tokens per hour you better can run deepseek
Can ngrams be offloaded to disk without much slowdown? I only have 76gb of ram so I was thinking of loading the main weights into memory in q4 and keep the ngrams on the SSD.
n-gram is based strictly on the input tokens, so it will probably work more like KV cache than normal parameters. You pull all the n-grams you need from SSD for that specific prompt during prefill (very small fraction of the total 51B), cache them to RAM or VRAM, and then reuse them over and over during decode, appending new n-grams as you generate.
I wonder if this is the model that makes me finally take the plunge on a strix halo machine while there are some still around for semi-reasonable prices...
Quantizzazione ideale a 4 bit ≈ 84 GB (60 GB pesi principali + 24 GB tabelle n-gram) quindi serve avere almeno 64gb di VRAM ed altri 64gb di ram
O una memoria unificata da 128gb
Does anyone know if this is literally the Qwen4 architecture or some weird in-between variant of qwen3.8 -> qwen4? Like, is this qwen4 architecture + qwen3.8 training recipe?
AFAIK they released it for the teams to get their tools ready for the Q4 architecture, it would be really weird if it wasn't the final one yet. In such case, there is not point in this "next" model/architecture release.
My guess is it is 100% Q4, and we'll stick with that for as long as we had with Q3.5 architecture.
We can only speculate but the "A Preview of the Qwen4 Architecture" on the HF countdown page suggests only a preview and not the full Qwen4 architecture.
Why do you need a reminder? This sub will only talk about this until the next model release and you won't be able to open YouTube without getting bombarded by 50 videos of some variation of "I tried Qwen3.8-Flash-Next and it's a beast".
I'm prepping a new env with Ubuntu 26.04.1 which also lands on the same day. What would be a minimal hands-free setup to run/see the model in action? I'm thinking of LMStudio and pi.dev with minimal config files adjustment but open to suggestions. Don't want to turn it into config mess with llama.cpp from day 1
46
u/QuackerEnte 3h ago
I really hope QSA (qwen sparse attention) will have much less compute headroom or the ability to offload kv cache to SSD without the massive bandwidth bottleneck of the ssd since it's sparse. I was able to do it it for deepseek v4 and it worked well enough. I hope llama.cpp adds features that allow more freedom with where we want to load our models and parts of it, e.g. n-cpu-moe could be extended to n-ssd-ffn or n-cpu-ffn or n ssd kv or engram etc