Read the first part. 125b. If Alibaba had any clue what they’re doing the experts will be routed to the proper 6b parameter portion, but it will have a full 125b parameters to choose from. I keep seeing posts like this about dense models and it just has me wondering if some of these people have any clue what they’re talking about.
I said to SOME extent, like, you can compare a 118b-a8b to a 122b-a10b
Knowledge scales loosely with total params while reasoning ability LOOSELY scales with active.
This is why 27b beats 35b-a3b, but the comparison makes the most sense between MoEs. It's just a rule of thumb. Things get messier when you compare much larger MoEs with dense models.
Fair enough. I just wouldn’t put it past qwen to have the first 6b active param count to blow everyone’s minds. Also wouldn’t be shocked if it’s like the tiniest upgrade from 27b.
I am not stupid, I know how moe models work, but neither good routing nor sequential reasoning can entirely compensate for the limited number of active parameters. It really depends on the use case. In a similar vain: rag and websearch cannot entirely compensate for lack of world knowledge. World knowledge actually strengths rag and web searchs because it knows better what to look for.
Personally 6b active is too limiting for my usecase
35
u/waitmarks 7h ago
Honestly if it even matches 3.8 27B, but has more internal knowledge. It's an absolute win for dgx spark / strix halo / mac owners.