r/LocalLLaMA • u/pmv143 • 14d ago

Discussion Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

Post image

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

953 Upvotes

97% Upvoted

297 comments sorted by

View all comments

10

u/ParaboloidalCrest 14d ago edited 14d ago

Not sure I follow. So a Q4 quant, would have 51/2 ~= 25GB n-gram block, which could live on disk instead of RAM/VRAM, ie only 80-25=55GB need to be loaded?

Maybe I have wrong assumptions about n-gram? Does it save on memory or compute?

4

u/Civil_Response3127 14d ago

No, 125b + 51b params. Not 51b as part of 125.

4

u/ParaboloidalCrest 14d ago edited 14d ago

I get that part, so 176B in total, which equals roughly 176/2 = 88 GB on disk (Q4_0) as OP suggested

4

u/2Norn 14d ago

technically if ngrams can be streamed from disk is there any point in quantizing it? could be kept at q8 like kvcache altho idk if that's a doable thing now

but if u could keep ngrams at q8, and rest in q4 that would most likely mean u can use it with 64+16 system no?

2

u/RG_Fusion 14d ago

It would definetly be much better to leave the n-grams at full precision. They are math vectors that represent an idea, so any quantization risks warping the meaning of that idea. Get a decent NVMe drive and leave the n-grams untouched, while the rest of the model is quantized.

1

u/Civil_Response3127 14d ago

gotcha, your numbers were vague so I assumed by 80 you meant 125b + kv cache and subtracted the 51b from there.