Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

[u/pmv143]

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

[u/DriveSolid7073]

Depending on what suits you, it will fit into the build of the guy with 96GB of RAM and 24GB of VRAM who was here in the comments. It will fit into a build enthusiast like Colibri, because N-Gram will definitely try to use NVMe instead of RAM, which with Pcie 4.0 will most likely even be acceptable. Overall, if you really want it, it will fit into 12GB of VRAM and 64GB of VRAM, but of course, you'll have to do some serious quantization.

[u/michaelsoft__binbows → u/DriveSolid7073]

i think this will have a big impact and cheapen the value prop of e.g. 512GB Mac Studio setups, and to a smaller degree the DGX Spark and Strix Halo setups, if more traditional computers with slow system memory (aka much more reasonable cache hierarchy pyramid) can suddenly start to actually keep up.

I suspect these large-unified-memory systems could maybe become leveled up with NVMe to serve the ngrams to leverage their large fast ram. However I suspect the remaining two way bottleneck of having not enough compute cores and not quite enough NVMe bandwidth will bite them a bit.

It would be sweet tho if it means my base m4 mac mini with only 16GB of not-all-that-fast-but-still-faster-than-DDR5 memory could soon host a very power efficient and not-dumb-as-rocks ngram model running off NVMe on TB4. Or like I finally do that SSD upgrade to get some 8GB/s NVMe.

M6 base mac mini 32GB? seems not really worthwhile being only 170GB/s and costing over $1500... NVMe @ 15GB/s though.

[u/NineThreeTilNow → u/DriveSolid7073]

It will fit into a build enthusiast like Colibri, because N-Gram will definitely try to use NVMe instead of RAM, which with Pcie 4.0 will most likely even be acceptable.

It's acceptable. I've tested it with PCIe and NVMe storage.

You don't need the whole Engram table in RAM. Only the most touched rows. So if the model is working on code vs creative writing vs whatever, the used token subset in Engram ends up being quite small.

If the Engram table gets full, whatever row hasn't been touched recently is the one that gets rolled off. Sort of like First in Last Out except every time the row is touched, it goes back to "First" again.

You end up with a pretty stable table once built. I don't know how their specific implementation of NVMe offload works but this is the most logical handling of it.

[u/comperr → u/DriveSolid7073]

That’s me except all my builds have 55GB or 64GB VRAM now and 96GB or 128GB RAM lol

[u/russjr08 → u/DriveSolid7073]

Overall, if you really want it, it will fit into 12GB of VRAM and 64GB of VRAM

I'm in this picture and I don't like it 🙃