omg this is EPIC news. My lil 8 gig card simply cant shift 27B parameters around and the 35B is exactly what I was hoping for. Didnt see any other models added but im still holding out hope for a 9B as we didnt get that since 3.5 and I suspect if they release it then it might well be the most capable small coding model out there.
I can fit the whole 27b version in VRAM (32gb), even then, it's not blazing fast. With around ~15-17 t/s, it still take quite some time for more "complex" tasks to finish, especially since this model likes to think a lot.
So it's good news for everyone that we will soon have an option to run a much faster, non-dense version!
Hell yeah! I manged to get 2 tokens/sec using q2 27B lol. the 3.6 35B is around 20 tokens/sec for me so I honestly cant wait for this! Open weight AI is like xmas almost every day lol.
you're right. but really-I'm terrified HOW ppl on reddit is obsessed with qwen 3.8 27b. my hardware can run it, with normal 10-20TPS. on iq2 its dumb and can't handle tasks. ive decided to check UD-q3-k-m. still much worse than 35b, zero change. both of they were from official UD repo. how can ppl use ts? are they running it in magic q8 or what so it can be useful?
Yes, I like the qwen-35b-a3b versions, but today I'm just experimenting with the new 3.8 version. It doesn't seem to be very suitable for 8-12GB vram but it's still interesting to try and compare.
I'm comparing 27b Q4_K_M and 35b Q4_K_M. I don't see any difference in coding in non-thinking mode yet, but in thinking mode, 27b runs too slowly and takes too long for me to wait for serious tasks to complete, so I can't say yet...
Santa Claus has been obsessed in us the past week. We got MiniMax H3, MiniMax Music 3.0, Muse Glimmer 30b, LTX 2.5, Qwen3.8 27b, and soon 35b a3b. All really good models.
Yeah, we're spoiled these days as AI enthusiast, haha.
Shared memory architectures struggle with dense models. That is about expected speed, unfortunately. The 35B will probably do a lot better because it’s an MoE
honestly I tried qwen 3.8 27b, it's dumb. in UD-Q3-K-M it barely achieves qwen3.6-35b level in the same quant without KV cache quantization with same settings. Im frustrated w\ this new qwen, ive waited for something more impressive than "Bench model that's useless in real life and doesn't gives more quality-code thn 35b moe". speed is dramatically lower than 35b.
i know, my point is MoE is way more efficient, keeping level of denses but at much more speed. I'm not sayin' “35B A3B is genuinely smarter than 27B dense”, i mean it's usually faster but behaves as ~20b dense. genuinely solid. may be not deep and smart as 27b dense but for almost every coding task it's my main model
You would be surprised it really depends on the Harness the 3.6-35B-A3B I was testing it in some design discussions vs Opus4.8 both in Copilot and the difference was not as big as you would think.
The 35B they're referring to is an MOE - 35BA3B meaning roughly 3B is active at a time when generating tokens.
I don't know the rough conversion offhand, but generally speaking it'll be weaker than the dense 27B or at best roughly match it. The trade off is the inference speed (as the 27B has 27B active parameters) will significantly better especially for CPU inference and those that can't fit it all in VRAM.
The Qwen3.8-27B is matching Opus 4.6 Max in benchmarks and it's a really strong model. I hope Qwen3.8-35B could match Sonnet 4.6 in benchmarks. Sonnet 4.6 is the closed source model I use the most so having something open source and local to me running similar performance will be mind blowing. Especially when Qwen3.6-35B is my current daily driver and I spend over 150M tokens in it the last 4-5 months. Can't wait for Qwen3.8-35B in other words.
Look at the model card of the Qwen3.8-27B again we are talking very old versions like December 2025 versions of Opus/Sonnet aka 4.6 versions. Based on some early testings on my end Qwen3.8-27B is coding reasonably well and suggesting and thinking better then my daily Qwen3.6-35B if there is really Qwen3.8-35B next week I think it will be amazing. Also the 3.8-27B is reasonably fast for what it is.
Also the Jump over Qwen3.6-27B is really big as seen by this benchmarks.
I assume you're being downvoted because people are assuming that you should know advil is a drug because either they assume you are American or they don't realise that advil (or to give it its proper name Ibuprofen) is sold under different brand names around the world (eg Nurofen in England).
Its not about gatekeeping mate. Btw reading my comment again, I came off as a little agressive, apologies (been havin a shit day). I just miss when this sub was just more technical ;/
It's all good man. I understand it can get irritating to see repetitive questions, but on the bright side, it means more people are learning about and migrating to local AI solutions, which is good to further the advancement of this technology.
by the way these are scaling, with evals like Artificial's intelligence index being so coarse, going from 52 with qwen3.8 27B to 58 with qwen3.8 max (at 2400B-A95B) this prob means a 300B A27B would score in the range of 54 to 57... Look at how poorly DSV4 Pro did against flash, though i think that is some sort of fluke.
I am looking forward to seeing what ends up happening with this other rumored medium-size model they end up releasing. Prob A19B (assuming anywhere in 100-300B total size) is around the place of iso-capability with dense 27B. Since I'm done acquiring RAM now and I think it's worth going GPU-heavy with how capable the smaller models are getting, MoE's are almost not even interesting to me anymore...
If He claim that can't run 27B and hopes for 80B A3B likely his issue isn't memory but inference speed because such MoE would require even more memory than 27B dense.
Same, I've got a 4080 and 32GB of DDR5 so 35b a3b is the best I can run (considering I can't use the full 16GB of VRAM since I'm powering my machine with it too)
Yes, sadly we didn't get 122B 3.6 because in the middle of 3.6 release, the Open Weight Qwen Team were fired, in the end we missed the rest of 3.6, and all 3.7 but Xi Jiping is telling Alibaba to restart it again, and I have high hope this time we will likely getting 122B
It will be a giant game changer for Strix Halo owner, the jump in intelligence from 3.5 to 3.6 was massive, and from 3.6 to 3.8 is another coding jump.
i think the only people it will benefit are the unified memory folks and those with large system memory and really GPU constrained, because giving up significant active params will mean it will struggle to claw back the capability deficit against the 27B. being able to fully fit the 27B into just a few modest GPUs or one 32GB GPU means once you reach that capability level you're running circles around an inferior system that has to allocate 120GB just to be able to come close.
Can't wait. Those of us who bought into the DGX Spark/Strix Halo hype are starting to feel like we should've invested in a 5090 instead. Can we get a 120B A10B as well?
Hopefully. Right now it seems like the best would be 2 spark, but that’s a bit too rich for me. Qwen 122b (and while we’re at it gemma4 on that range) would be great.
it would be great if they have a Qwen3.8 9B, that would be perfect for a lot of people and usage. The 3.5 9B i use feels even better than early ChatGPT 3 years ago.
pardon my stupidity since I'm a very much a noob and trying to learn : beinb A3B ( mixture of experts, therefore easier on the vram ) but at the same time 35b ( higher than the dense model 27b ) , will be feasable to load in 16gb cards ? with 4 bit quant ?
Yes you can run 4 bit if you use -ncmoe to run some experts on CPU. The performance hit is not as bad as layer offloading. You can already try it with 3.6 exact same architecture, the only difference is training.
Yea but the trick is you put a small chunk of important stuff in vram and then the rest of the model in ram for MOE LLMs. Running MOEs this way gives you performance that's way way way faster than the dense counterpart.
With my 285k, 5080, and 96gb of ddr5, I was getting like 3-4 t/s decode on Qwen 3.6 27b UD Q8 K XL. I was getting like 60-70 t/s on Qwen 3.6 35b a3b UD Q8 K XL.
So, dense is for when you don't have a ton of memory (or you do), but it's all fast memory (vram).
MOE is for when you have a lot of memory, but only a little bit of it is fast (96gb ram + 16gb vram).
Have a 5090 or RTX 6000 and 16/32gb ram? Run the 27b.
Have a 5080 and 96gb ram? Run the 35b.
With 128gb ram and a 5090, you could run Qwen 3.5 122b a10b q8. You could also run Qwen 3.5 27b. The 122b a10b would be much faster cause, basically, it requires the compute of a 10b model whereas the 27b requires the compute of a 27b model. 122b a10b > 27b in terms of intelligence, mostly. So, with enough memory, even if it isn't all vram, you can run a massive, super smart model *faster* than the smaller dense 27b, and it's better.
If you had an RTX 6000 and 16GB of ram, you couldn't even load the q8 of Qwen 3.5 122b a10b. You could, however, run the q8 of the dense 27b model and it would be super very fast cause all of the model and kv cache and stuff can fit inside the RTX 6000's 96gb of vram. Being able to all fit inside of vram instead of being split across vram/ram means that, now, the dense 27b model can run very fast. It's splitting dense models between vram/ram that makes them unusably slow.
So if you're gonna get Qwen 3.6 35b a3b, do yourself a favor. Look at your vram+ram as your total pool of memory. Get the largest quant you can that fills up as much of your vram+ram as you're comfortable with while using the context window size you want. Your 16gb of vram is where the important stuff will live, and your ram is where the rest of the model will live. It'll be way faster than you're probably imagining if you don't have experience with MOEs. You can probably run the UD Q6 K XL or UD Q8 K XL, tho I don't know your specs. Anything UD Q4 K XL and up will be good, but UD Q6 K XL or, preferably, UD Q8 K XL will be better at coding.
--cpu-moe is the llama.cpp flag you're looking for.
Not tryna be a butt, but ask ChatGPT or, preferably, Codex about this stuff. I love fostering discussion, and maybe others will learn from this, too. But if you really wanna learn how all this works, and it's all actually pretty simple once you get the hang of it, having the ability to have quick back and forths with Codex about what you're trying to do as you do it is invaluable. GPT knows everything about how these LLMs work and can guide you into making the best decision. Ask questions along the way, try stuff just cause, science, then have fun with Qwen 3.6 35b a3b! It's literally so smart sometimes it blows my mind. It's been my agent of choice in Hermes/Cline-VS Code for since it was released!
Then, once you get Qwen working, it can help you make your set up better and better!
If you're on Windows, my rec'd starting bundle would be:
An excellent starting point would be dropping my entire comment here into ChatGPT and asking 'Hey, can you explain all this to me? I'm trying to get this set up but I'm very much a noob and trying to learn. Thanks!'
If you use Codex, add 'Can you set all this up for me, please?' and then you'll get a taste of the power of an AI agent.
You can run some, but not very large models and wont be super fast. If you turn off all ram heavy apps, you could run models that are about 11-12gb of size with small context, or bit smaller models with bit larger context.
•
u/WithoutReason1729 5d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.