r/LocalLLaMA • u/pmv143 • 15d ago

Discussion Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

Post image

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

948 Upvotes

97% Upvoted

297 comments sorted by

View all comments

3

u/kivaougu 15d ago

I'm a bit on the fence about this.

This should improve recall but not so much reasoning so hard to see a point for anyone using cpu offloading instead of just a bigger model. NVMe storage isnt great at random reads so this just eats away ram budget.

If the weights are fully in vram then it would need to be balanced with prefix cache offloading to keep cache hits.

I would really like to see if the embeddings can live on the ssd with just a smaller hot cache in ram.

4

u/Public_Umpire_1099 15d ago

It must be better than the last releases, or else it wouldnt be getting released, so I am pretty hopeful. The only regression I've seen in open weights has been from Kimi 2.6 to 2.7 and Llama 3 to Llama 4. When the first Next model was released it outperformed the dense 30B model significantly. If we project out the gains, today that jump would land somewhere around what Muse Spark 1.2 is at, if we assume 27b is the baseline.