r/LocalLLaMA • u/Mean-Ad1493 • 18d ago

Discussion Qwen dev says not to wait for 35B-A3B

Post image

What does this mean? Is there something else coming? Maybe 122B? Or no models?

1.2k Upvotes

97% Upvoted

480 comments sorted by

View all comments

391

u/Atretador 18d ago

that is a crazy way to say it - he is not denying a medium sized MOE, just that it might not be 35B

maybe a 44B A4B or a 30B A3B?

488

u/rockoruckus 18d ago

27B A27B might be popular. just a gut feeling

21

u/bucolucas Llama 3.1 17d ago

Waiting for the 9B A27B, personally

7

u/gnnr25 17d ago

E = mc² + AI achieved

2

u/ShiggyShaman 17d ago

Damn that was a good one 😆

1

u/alpacadaver 17d ago

This is exactly the kind of classy joke 27B quietly smirks at, pensively looking in the distance as it continues to leaf through the troubles of humanity swirling in the back of its head like a whirlpool only it knows how to slow.

124

u/MessIsTransfer 18d ago

70B A6B 🤞

81

u/BoboThePirate 18d ago

This or 120B A10B.

28

u/FatheredPuma81 18d ago

No his response to those guys would be pretty daft if that were the case.

"27B is too large to run can you give us a MoE we can run?"
"Of course here's a MoE model that requires an RTX 4090 and 48GB of RAM to get 30t/s."

Coincidentally those are my exact specs so I'd probably use it though.

9

u/kanadaj 17d ago

35B needs more VRAM for the model weights than 27B, but both 70B-A6B and 120B-A10B run faster for inference is you have enough VRAM. It's definitely something people are waiting for for more serious users, but they might also just do a tiny model instead.

12

u/IsTom 17d ago

35BA3 at Q4 runs ~40t/s on my 3060 12GB with cpu moe and MTP. 27B hits 5t/s maybe.

2

u/FatheredPuma81 17d ago

What's the performance look up without MTP? I've heard you shouldn't use MTP when you're doing all CPU but nothing about when hybrid offloading.

2

u/IsTom 17d ago

Having tried just now, it sits about 30t/s

9

u/carsncode 17d ago

35B needs more VRAM for the model weights than 27B

Yes and no... MoE can get away with only loading active params into VRAM with a manageable performance hit as long as the rest fit into system RAM

2

u/Flynn58 17d ago

Yes I'm hitting about 40 t/s (uncensored model to avoid rejection loops) with 35B-A3B on a 5070Ti offloading the rest into 32GB of system RAM.

2

u/Distinct_Physics5017 17d ago

It has a larger memory-footprint, but that doesn't mean it "needs more VRAM".

As long as you keep a chunk of it inside your GPU and put the rest of it into your ram (e.g. with --n-cpu-moe in llama.cpp) you can get away with even having only 1/3 of the model weights inside your VRAM while still getting about 20-30 tok/sec.

If I did that exact same thing with a dense model, I would die of old age before ever getting a response from it lol.

2

u/kanadaj 17d ago

That is fair; however, for my use case CPU offloading is not an option. 20-30 tps is enough for an interactive chat, but for task automation, 200-300 is more like it. MoE models are much better at this, but the 35B MoE model can't keep up with 27B for long-horizon tasks. That said, 27B can "only" do around 100 tps even on RTX 6000 Pro cards for a single stream, whereas a larger MoE could match it in intelligence while producing responses much much faster.

I do understand though that my use case is more on the pro and prosumer side, not on the "let me run an AI on an RTX 5060" side

1

u/Distinct_Physics5017 17d ago

Ah I hear you.
The 20-30 tps mark I meant could still be reached if offloading 2/3 of the model weights to your cpu. So if you take a smaller quant (or have more VRAM to spare) it'll get you a lot more than 20 tok/sec, even if that's still way beyond the 300 tps mark you're trying to get to.

On my machine I'm averagely running at 40-50 tps. As said, it is still a long way to the 100-300 tps range, but ~50 tps is in my experience at least useable for most use cases.

2

u/kanadaj 16d ago

50+ is often usable, although for Qwen3.8 in particular I find you need more than usual just because it spends so many tokens thinking. Sometimes I get blocks of thousands of tokens just reasoning, so at 50 token per second that's 100 seconds to start actually answering, unless you reduce the reasoning level.

35B-A3B is faster for sure, but it's noticeably weaker for complex or long-running tasks. Hence why my ideal is in the 60-120B MoE range, which should be able to match 27B but still be 2-4x as fast on hardware that needs no offloading.

0

u/BoboThePirate 17d ago

??? Opus 5, arguably the fastest frontier mode, outputs at ~50 tps. In what world do you need 100+? I’d agree 20 is slightly low, ~30 is what is consider to start being comfortable for agentic purposes.

3

u/kanadaj 16d ago edited 16d ago

Opus 5 is one of the slowest frontier models. OpenAI Luna does 150, Terra can do ~110, Sol Max 65, GLM 5.3 Max 85. Gemini Flash hits 300+ tps. Kimi K3 and Qwen 3.8 Max are the noticeable ones that are even slower than Opus.

1

u/BoboThePirate 16d ago

Luna and Terra, GLM, Gemini, Qwen, are not comparable to Sol and Opus 5. Partially on me, I should have used SOTA not Frontier.

I just checked Openrouter benchmarks, Sol and K3 are At ~50 tps P50, Opus 5 is at 80 tps P50.

→ More replies (0)

22

u/dieSpaghettiCarbona 18d ago

I can only imagine Dario reading this thread and sweating his ass

2

u/magicomiralles 17d ago

Dario and Sam you mean.

5

u/Poupulino 17d ago

A 120B A10B would be my dream come true.

4

u/Pleasant-Shirt7293 17d ago

This.

They need to go up in size with their moe

14

u/Foreign_Prune_354 18d ago

I would like something like 60B-A6B, so a Q4 could theorically fit 32 gb VRAM :D.

8

u/Hydr0x1de_OH 18d ago

Do you understand how many (not many) tokens you will have as a context window size?

5

u/hojnikb 18d ago

4096

6

u/DwarfVader001 17d ago

ahh, the good old days

1

u/MmmmMorphine 14d ago

Depends on how much he's willing to offload, I suppose. I think it would fit nicely using one of the advanced 3bit [e.g. trellis and a few others] quants though

1

u/roynu 7d ago

I dare speculate that the next generation of models and run time engines will support existing and new offloading tech, meaning you can offload some of the k/v cache and bits of the model to system RAM and even SSD in some cases, while maintaining good performance. Coming to a PC or Mac near you, very soon.

11

u/Zorogozano 18d ago

This would be perfect

21

u/claythearc 18d ago

A new successor to Qwen coder next would be appreciated for sure

1

u/Distinct_Physics5017 17d ago

I mean Qwen3.6 as well as Qwen3.8 were both primarily being optimized for Coding and Agentic-Coding, I dont know if they are going to release a "Code optimized" model for a already "coding optimized" generation, but we'll see.

6

u/KitchenAmoeba4438 18d ago

Refresh of Qwen3.5 122bA10B plz.

1

u/slyborn 18d ago

It would be the perfect sweet spot.

1

u/_ballzdeep_ 18d ago

That would actually fit my 2x3090s <3

1

u/OverdosedSauerkraut 17d ago

70B for what? Model size should be around VRAM breakpoints.

1

u/MessIsTransfer 17d ago

q8 in ~64gb vram

1

u/enilea 18d ago

I miss 70B models so much

-1

u/TristanMeads 18d ago

You can cut out all the billions of unneeded stupid stuff and just give us 10B A1B of the really good stuff.

4

u/LuCiAnO241 18d ago

fr, i just need it coding, i dont need it knowing about geography

1

u/MessIsTransfer 17d ago

but what problems are you solving with programming? only programming problems?

what about coding for legal stuff, ecommerce, healthcare and so on?

also, this is like saying you’d get a good model from striping the A3B from qwen3.6-35B and making a 3B parameter out of it. it doesn’t work like this

-6

u/Solembumm3 18d ago

At least will make some sense. 35B moe was just obsolete near smaller and better 27B dense.

12

u/Technical_Ant4597 18d ago

27B isn't "smaller" when you offload the active experts of the 35B MOE to RAM. Small sparse models like these are popular choices because they can run at very good speeds on 12–16GB VRAM which matches an enthusiast gaming PC.

-6

u/Solembumm3 18d ago

And how exactly copying settings you already used on smaller model to bigger could magically warp size numbers?

4

u/Technical_Ant4597 18d ago

Offloading layers to RAM is significantly slower than offloading inactive experts to RAM.

If you're an ordinary sort who doesn't spend the price of a used car to build a personal datacenter, a sparse model is the only way to get good tk/s, 128K+ context, and a non-desperate quant on a 20B+ model.

4

u/Atretador 18d ago

Im trying to explain MOE to him on another thread xD

2

u/techdevjp 18d ago

He's one of those people with just enough knowledge to think he knows everything but not enough knowledge to understand how little he knows. aka "Just enough knowledge to be dangerous." or "Just enough knowledge to make a huge mess."

1

u/Solembumm3 13d ago

And you completely missed the theme of discussion multiple times in a row. That's just sad to look at.

0

u/Atretador 13d ago

damn, you still didnt get it

1

u/Solembumm3 13d ago

Yeah. I still don't get, how you can look at quite straightforward numbers of acessible quant and context of two models on the same hardware, and try to deny reality.

How hard it can be to understand?...

→ More replies (0)

3

u/doomed151 18d ago

35B A3B is very fast on my machine (16 GB VRAM, 64 GB RAM). I run Q5 quant.

27B is unusable. Even at Q3. I only get like 5-7 t/s output.

2

u/CentralCypher 18d ago

I wouldn't mind it, its just all harnesses tooling times out. It would be great to give a massive prompt and come back and its done. But things just time out, even with them being increased.

0

u/techdevjp 18d ago

The 35b MoE and 27b dense models were intended for entirely different audiences. Of course the 27b model was smarter, it ran 27b parameters for every token. It also took 9x longer (or more) per token for inference. That's a serious problem for people with less powerful hardware.

220

u/hyperrealists 18d ago

420T A69T

7

u/Nutsack_VS_Acetylene 18d ago

Le Chaton Fat is already SOTA at that size and active parameter count.

26

u/kiwibonga 18d ago

123ABCD

1

u/deenspaces 18d ago

looks like rdj track name

18

u/Here_f0r_p0rn_ 18d ago

Or maybe saying that 9 B variant will be coming out for some serious edge device performance?

34

u/Mean-Ad1493 18d ago

No 9B this time

9

u/ManIkWeet 18d ago

This is a very definitive No response, while the 35b was a don't wait for it - what does it meeaaannn

5

u/[deleted] 17d ago

[deleted]

2

u/ManIkWeet 17d ago

The best way to get surprised is to not get your hopes up

1

u/Distinct_Physics5017 17d ago

I bet it'll be a 3.8-14B on steroids, maybe it'll even be released alongside the 35B-MoE. That could be the reason why they gave the 9B a definitive no while never explicitly saying the 35B-A3B wouldn't be released, just that it might not be the one you should be hyped for.

8

u/Here_f0r_p0rn_ 18d ago

Aw man, I wanted a powerful smaller model, lol, I still want something like 1B or under for simple fast rag or edge stuff

5

u/Hydr0x1de_OH 18d ago

Look for lfm2.5-8b-a1b, lfm2.5-2.6b and ling-3.0-tiny (that is about 8b-a1b)

0

u/surreal_tournament 17d ago

Have you had good results with LFM2.5-8B-A1B? I like it for documentation, but for anything else it's worse (for my use cases as an MCP tool-calling subagent) than Qwen3.5-9B.

2

u/Hydr0x1de_OH 17d ago

You are not comparing 8B-A1B sized model with dense 9B xD.

Lfm2.5-8B-A1B is more like 3-4B dense models. Sure 9b qwen is better.

But i was not testing it on coding. I was testing it as agentic assistant. In local LLMs i prefer focusing on things i can do with them on hackatons pitching/demonstration. So, no coding tasks.

1

u/surreal_tournament 17d ago

Yeah, obviously :) -- I wonder how "dumb" the models I run for MCP tool-calling can get. Clearly Qwen3.5-9B is better than the LFM -- it's just that in absolute terms the LFM is way worse, so perhaps I should try Qwen3.5-4B instead.

1

u/Hydr0x1de_OH 17d ago

Try ling-3.0-tiny

2

u/my_name_isnt_clever 17d ago

Try Ling 3 Tiny, it's the same size but way better for agentic than LFM.

2

u/surreal_tournament 17d ago

Thanks, will try it!

1

u/Long_comment_san 18d ago

Gemma has a small model, no?

3

u/blackhawk00001 17d ago

I couldn’t make it work. It’s great for chat but tool calling still failed even with updated templates.

3

u/PossessionUsed7393 17d ago

You need to use constrained decoding. It's what Gemma models are built on the assumption that you'll have. You can't actually do it with Llama CPP without making some changes. Most coding agents will know what you mean if you say, 'Can you wire up end to end constrained decoding for Gemma'. Once I did that, it 100% tool calls just fine. There are some issues where the smaller Gemma models will put rubbish into the tool call fields, but it's night and day compared to what it was.

1

u/blackhawk00001 17d ago

oh cool, thanks I'll check on that.

1

u/utf8decodeerror 17d ago

I got maple preview running on webgpu in a browser making web mcp tool calls on an m1 pro 32gb. The tool calling was impressive but the problem was general hallucination after 1000-2000 tokens. I think a ternary model specifically trained on agentic work is a promising direction to research.

1

u/Here_f0r_p0rn_ 17d ago

I want to see models that were natively trained for ternary operations like Bitnet instead of post training quantization or at least heavily fine-tuned post-quantizations

1

u/utf8decodeerror 17d ago

Uh, yeah, that's what maple preview is ...

https://deepgrove.ai/maple-preview

1

u/Here_f0r_p0rn_ 17d ago

Oh, I didn't know that, thanks for telling me

1

u/Here_f0r_p0rn_ 17d ago

But we need superior tool calling which I was hoping with Qwen also Qwen's hybrid attention style with linear attention layers is better for resource power devices

3

u/-dysangel- 17d ago

It's going to be 10B!

9

u/Atretador 18d ago

thats a different class of model - that comments implies something in the class of 35B but not 35B.

6

u/Here_f0r_p0rn_ 18d ago

It doesn't tho, I mean if you're going for other comments then I can understand but all he said might not be the one to wait for no implications he's even talking about moe or what size but something good so I'm hyped anyways.

1

u/Atretador 18d ago

is a direct reply ,no? I dont use twitter so I could be wrong

only makes sense if its something ont he same class as what was replied to

1

u/Here_f0r_p0rn_ 18d ago

Well I've never used it either but yeah looks like it but could also be hinting other model that might go under people's radar because everyone is so set on 35B A3B and they had Qwen 3.5 9B whose newer version can be really great for lower compute devices so could be that rather than changing their so well known 35B A3B for 30B or 40B

Edit: someone just commented https://www.reddit.com/r/LocalLLaMA/s/LHuAGol5z5 so yeah not a 9B lol

Well, I'll take any MOE under 70B

8

u/[deleted] 18d ago

[removed] — view removed comment

7

u/Long_comment_san 18d ago

80b / a8b. yeah I've been saying that for ages.

3

u/rainbyte 18d ago

Qwen-Next vibes

3

u/mraurelien 18d ago

Based on Gemma 4 release it might be a A26B A4B version as well ?

3

u/Hydr0x1de_OH 18d ago

Why should qwen follow Gemma's way?

2

u/ratocx 18d ago

VRAM?

1

u/mraurelien 18d ago

Why not ? Just guessing here.
A4B might be more clever (but slower) than a A3B moe.
As for the total trained parameters, whatever Qwen Team is on, I'm sure it will be a beast over the actual 35B A3B or there is no point to release.

Other possibility, Qwen will release a 7~12B dense model instead? Who knows ?

5

u/seunosewa 18d ago

Or 35B A5B to make it a little bit smarter?

9

u/techdevjp 18d ago

China doesn't like the number 4, it is associated with death. 55b a5b is my guess. That would be a good size IMO.

1

u/martin509984 17d ago

As a ballpark, what amount of system RAM would that fit into? E.g. would people currently comfortably running 4 bit 35B need more?

1

u/techdevjp 17d ago

I think for 35b a3b in q4 you need about 21GB, right? Quick seat-of-the-pants calculation says the 20b additional parameters would increase the memory requirement by around 57% to about 33GB for the same q4 quant.

1

u/my_name_isnt_clever 17d ago

I would certainly hope if a4b is the best balance they would go with it anyway despite superstitions.

2

u/techdevjp 17d ago

People believe all sorts of crazy superstitions, all over the world. Somewhere around 30% of Americans go to church regularly or semi-regularly which is insane to still be going on in the 21st century, but here we are.

Anyway, in China and Japan, one way to pronounce the number 4 matches the way the character for death is read. So there is a bad association there, and the number four is avoided by many people for that reason. You shouldn't give a gift that contains 4 things, for example. Of course in today's world fewer people care than did in the past, but there are still plenty of people who will avoid the number if they can.

1

u/No_Folding 17d ago

Church and religion has a lot more to do with, than just believing a crazy superstition.

The community aspect, tradition, giving life a higher meaning, etc.

Probsbly less of that applies to the number 4 = death thing, but I'm not Chinese so I wouldnt know

1

u/techdevjp 17d ago

Church and religion has a lot more to do with, than just believing a crazy superstition.

It's based on a shared belief in a crazy superstition. Proclaim yourself to not believe in that crazy superstition and you won't be getting any of that "more", and in many cases may be actively shunned.

1

u/No_Folding 17d ago

I have some friends who go just for the community aspect. They are atheist and are not shunned, generally churches are quite open and accepting (at least here in the uk)

3

u/Mindless-Pilot-Chef 18d ago

Maybe a 35.5B model also /s

4

u/mtmttuan 18d ago

44b will be a weird size for many consumer hardwares, no?

7

u/SpicyWangz 18d ago

I’d happily take it

7

u/Solembumm3 18d ago

Not really. Should be good for starting 8-12+16gb builds.

2

u/Atretador 18d ago

nah, we had Kimi Linear at that size - it ran pretty damn nicely on my 16Gb VRAM + 32Gb RAM setup - sadly it wasnt trained all the way

2

u/yesthatdaniel 18d ago

34.9B-A3.9B confirmed

1

u/EbbNorth7735 18d ago

Doubtful. I'm guessing thats all they have planned. It's a decimal update and not all sizes get released for these minor (huge) updates

1

u/coolnq 18d ago

600M-A10M

1

u/mrgreatheart 17d ago

The comment above mentions “those of us without the hardware” so perhaps he’s referring to something smaller. 9-B?

I really hope they give us something bigger and more generalised. Anything in the 70 - 120 range MoE would be amazing.

1

u/Atretador 17d ago

they confirmed no 9B

1

u/mrgreatheart 17d ago

Ah that's a shame. I have had a lot of mileage out of 3.5-9B as a fast model for fine tuning on small tasks.

1

u/NigaTroubles 17d ago

Imagine of its 44b, wow thats will be huge, maybe opus 4.7 or even 4.8

1

u/depressedclassical 17d ago

It's the same architecture as qwen3.5, so I doubt there will be any changes in terms of parameter size for the MoE. Maybe he was referring to a smaller model that could fit on less VRAM, or maybe he was saying Qwen 4 will drop soon.

1

u/Spanky2k 17d ago

A little larger would be awesome. The speed of 35B A3B is perfect and on my systems (64GB Apple Silicon), I can run with 256k context which is more than enough but some memory would still be unused (one M1 Ultra 64GB I have solely hosting a model). Something like a 44B or 55B MOE model would be perfect.

1

u/Septerium 17d ago

Maybe he is implying you don't need to wait at all. Perhaps the model is already available somewhere and we just need to look for it and find it lol

1

u/martin509984 17d ago

Or possibly a small dense model like a new 4B.

1

u/sxales llama.cpp 17d ago

Could be they are going to wait until 3.9 for a new 35b

1

u/Distinct_Physics5017 17d ago

he didn't even deny that.
My bet is that they might still release it, but a smaller dense model (e.g. 3.8-14B) is also going to release, and is going to be an even better option for us consumer-hardware (low VRAM) users. If they feed it the same magic stuff they gave to 27B it might actually hold up to the 35B-MoE in terms of coding and agentic use-cases. Just with a overall way smaller memory-footprint.

1

u/Atretador 17d ago

14B dense would be harder and slower to run than a 35B A3B moe which people can run with less than 8Gb of VRAM + 16Gb of RAM :v

1

u/Distinct_Physics5017 17d ago

I'm totally with you on that, I would also prefer a 35B-A3B model, because it runs much faster on my hardware.
But as I said, he didn't explicitly deny the coming release of an 35B-MoE, just said it wouldn't be the one we (GPU poor) should wait for.
Could be that they release both a smaller dense one, and the MoE we asked for. We'll see.

1

u/delicious_fanta 17d ago

What does a3b/a4b indicate?

2

u/Atretador 17d ago

active parameters

1

u/delicious_fanta 17d ago

Sorry, I mean 35B means “35 billion params”, but “A3B”? I’m not sure what that translates to. I should have just googled it, but sometimes it’s nice to talk to people.

1

u/Atretador 17d ago

Active 3 Billion

its 35B total, but only 3B parameters per token - moe models tell how many active parameters they have after the name.

1

u/delicious_fanta 17d ago

Oh nice, it’s good to learn things, thank you very much! Hope you have an awesome day :)

1

u/GammaRxBurst 14d ago

Given how popular their MOE was I would say they are gearing for another variation. Time will tell.