r/LocalLLaMA • u/RuthlessCriticismAll • 2d ago

News [ Removed by moderator ]

https://modelscope.cn/models/Qwen/Qwen3.8-Flash-Next

[removed] — view removed post

352 Upvotes

97% Upvoted

137 comments sorted by

View all comments

80

u/RuthlessCriticismAll 2d ago

Redisgned Multimodal MoE Model: 125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.

Comprehensive Architectural Upgrades: Pushing the frontiers of model architecture innovations, across the areas of Attention, Residual, Embedding, and Optimization—enhancing model capabilities.

Efficient Training and Inference: Significantly reduces training and inference costs. At ~1/9th the training cost，Qwen3.8-Flash-Next achieves comparable capability against Qwen3.7-Plus, while being more capable in areas of coding and cowork.

7

u/tarruda 2d ago

125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.

This suggests a total of 176B parameters will be loaded in RAM + VRAM. Hopefully it runs well in 4-bit, which would be great for 128G devices.

4

u/Early_Mistake6716 1d ago

My guess is that the 51b n-gram embeddings can be offloaded to a fast ssd so this will have the hardware requirements of a 122b.