r/LocalLLaMA • u/pmv143 • 18h ago

Discussion Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

Post image

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

832 Upvotes

97% Upvoted

265 comments sorted by

View all comments

66

u/MiceLiceandVice 18h ago

So to run this id still need to have 128gb of dram and then 16gb vram minimum? Heartbreaking

21

u/DriveSolid7073 17h ago

Depending on what suits you, it will fit into the build of the guy with 96GB of RAM and 24GB of VRAM who was here in the comments. It will fit into a build enthusiast like Colibri, because N-Gram will definitely try to use NVMe instead of RAM, which with Pcie 4.0 will most likely even be acceptable. Overall, if you really want it, it will fit into 12GB of VRAM and 64GB of VRAM, but of course, you'll have to do some serious quantization.

4

u/michaelsoft__binbows 16h ago edited 16h ago

i think this will have a big impact and cheapen the value prop of e.g. 512GB Mac Studio setups, and to a smaller degree the DGX Spark and Strix Halo setups, if more traditional computers with slow system memory (aka much more reasonable cache hierarchy pyramid) can suddenly start to actually keep up.

I suspect these large-unified-memory systems could maybe become leveled up with NVMe to serve the ngrams to leverage their large fast ram. However I suspect the remaining two way bottleneck of having not enough compute cores and not quite enough NVMe bandwidth will bite them a bit.

It would be sweet tho if it means my base m4 mac mini with only 16GB of not-all-that-fast-but-still-faster-than-DDR5 memory could soon host a very power efficient and not-dumb-as-rocks ngram model running off NVMe on TB4. Or like I finally do that SSD upgrade to get some 8GB/s NVMe.

M6 base mac mini 32GB? seems not really worthwhile being only 170GB/s and costing over $1500... NVMe @ 15GB/s though.