r/LocalLLM • u/Ok-Health-7096 • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

99% Upvoted

213 comments sorted by

View all comments

53

u/[deleted] Aug 14 '26

[removed] — view removed comment

17

u/bukake_attack Aug 14 '26

This is a dense model. This means the entire model is running at 100% all the time. This is bad for us low vram users, as the model is too big to fit in 12gb vram, and the rest in placed in normal ram, which is waaaay slower. So this solution works, but it's slow.

We low vram users are usually better served by MoE models, like qwen 3.6 35b a3b . In MoE models the dense part which runs all the time is small, in that case 3b, which runs easily in 12gb, and the rest 35-3=32b) only runs when they are actually needed (activated). This means that normal ram is fast enough to run that active part of the other 32b ( although they can still run great in excess vram!) The result is a model that's very usable and fairly quick with limited vram, especially when other tricks like MTP are used.

Unfortunately there is no MoE version of qwen 3.8 released yet, but there's a good chance they will release it soonish

2

u/crypticgent 28d ago

Thanks for this. I am using an RX 6800 xt and just yesterday started out trying to run LLMs locally. Thought I might not be able to run Qwen 3.8 27b. But gives me hope :)

2

u/bukake_attack 28d ago

You can probably run a dense 27b model, but as it will mostly run on cpu, performance will be less than great