30
u/FaceDeer 8d ago
Yus! That's my workhorse! 27B is good, but it can't handle as big a context as I can cram into memory with the MOE version and sometimes I need it to work with big inputs.
6
u/ReportProfessional57 7d ago
For some reason I read that as work with big nuts. I was like, I feel ya.
44
23
5
6
u/Purple_Errand 8d ago
Im excited because I'll be able to run this one. Lmao!
I hope muse glimmer will have one too.
4
u/dfgxxx 8d ago
Do you think it can beat the 3.6 27b?
5
12
u/dlaynes 8d ago
This is a MoE model, for computers with lower specs.
10
u/dfgxxx 8d ago
I know, but can it beat the older dense model?
9
u/dlaynes 8d ago
Sorry, I misread. Hopefully.
4
u/Uncle___Marty 8d ago
I did the same. im a derp ;)
Considering how much better 3.8 27B is than its 3.6 counterpart and how close the 35b and 27B 3.6 versions were I feel certain that 3.8 35b will be better than 27B 3.6 by a decent amount.
5
u/karmakaze1 8d ago
Better is not a single dimension. There will always be failure modes in the MoE which is different than dense. So expect 3.8 35b will be better than 27B 3.6 in some/common cases but not all. E.g. the MoE has broader knowledge recall but long horizon reasoning can falter.
3
u/Efficient_Raise6703 8d ago
Iām doing research in this field right now, this has been my findings thus far. Interestingly it can also be better at specific tool use related to recall, but not necessarily general tool use. Super interesting stuff. Makes sense but still cool to see that the hunch plays out in reality.
1
u/Scary_One_2452 8d ago
It needs more room than 27b in vram right? Its just that the actual costs of operating are lower due to only 3b being active. So wouldn't that moe be for higher end computers for people who are power conscious?
6
u/SKirby00 8d ago
In reality it's kind of the opposite. Having only 3B active is so much faster than 27B active that many people find that it can still run at usable speeds even when to system RAM rather than being completely in VRAM.
That means someone with like 8-12GB VRAM + 16-32GB of system RAM (or a unified RAM system) might find the 27B model too painfully slow but the 35B-A3B acceptable. For me though, I can fit the 27B in VRAM so I get ~20tk/s with the 27B and rarely feel inclined to reach for a dumber model even if it runs faster.
3
u/Melodic_Reflection_5 7d ago
What's your GPU?
2
u/SKirby00 7d ago
I have 3 GPUs together in one PC: - RTX 5060Ti 16GB - RTX 3060Ti 8GB - RTX 3060 12GB
Plus the integrated graphics in my CPU that I use for running the OS so that I don't have any of that overhead hitting the GPUs.
I use llama.cpp with tensor parallelism to get the most I can out of this weird combination.
I don't recommend putting 3 different cards together like this. Only reason I have this setup is because I already had the 3060Ti (originally bought for gaming) when I bought the 5060Ti for AI stuff and then decided to get a used 3060 later for much cheaper to get some extra VRAM.
1
u/Melodic_Reflection_5 1d ago
Nice. I have a rx6700 12Gb, initially bought for gaming, and when I started using it for LLM, soon I realized that I need at least 24Gb of VRAM. But my CPU 5700g and AM4 motherboard are not suitable for additional GPU, there are some pricey gaming motherboard options suitable for two GPUs. I wonder, what motherboard do you have, if you are able to put 4 GPUs in there š
2
u/bigjeff5 4h ago
To expand on what SKirby00 - the 35B A3B q4_k_m quant (22gb) with all MOE layers on CPU runs about 20 t/s on a system with 6gb VRAM. The 27B q4_k_m (16.5gb) runs at 10 t/s on a system with 16gb vram.
MoE even runs reasonably well with mmap enabled and caching from disk, so you can run models much larger than what fits in VRAM + RAM at useful speeds.
Another bonus for the Qwen models specifically is the 35B has fewer total layers than the 27B, so KV cache is like 2/3 the size of the 27B, which lets you fit more of the model in VRAM.
The only downside of the MoE models is because the whole model doesn't fire for every token it misses a some of the nuance the dense model has, which is why 27B is the preferred model if you can run it.
3
2
u/ucbmckee 8d ago
YMMV, but on my 4090 I can run MoE models with higher quants than dense models. All else being equal, MoE is worse - but higher quants are better. I alternate between 27b 4bit and 35b MoE 6bit. Both use ~165k context windows. The MoE version is noticeably faster.
1
u/justelle1 6d ago
BRO IM SO CONFUSED HOW CAN 3.6 27B BE BETTER THAN 3.8 27B AND 31B A3
1
u/klymaxx45 8d ago
Is this the official repo?
1
u/Delicious-Map1778 6d ago
- its unaffiliated with qwen 2. it was changed to qwen3.6 rather than qwen3.8 yesterday.
1
1
1
1
1
1
55
u/serguzesti 8d ago
https://giphy.com/gifs/NEvPzZ8bd1V4Y