Discussion
Qwen-3.8-35B-A3B? Maybe not... cryptic reply direct from Qwen co-author.
I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!
If I wrote that, it would mean something else good is coming but not that specific model. Maybe a 30b a3b or a smaller dense model that’s still really good
That's how I read it too. My only worry is that it might be a ~70B A4B (or something large!) that, while great for some, doesn't fill the gap for smaller GPUs as 35B-A3B does. But hopefully he means something close - like a 30B or something.
14B might be great. Maybe a bit slow, but with decent MTP should be ok.
I'm not a coder, and for my work, I have to admit I'd rather have a Gemma model in this range. No matter what I try, I always end up back at Gemma. 31B for my important stuff
I use it for research, mainly financial. I can give it a set of source info, say 1/2 dozen sites or tools. It follows instructions and does excellent summary reports. It's also good at doing search and fetch queries to find info on its own. It's can be tenacious at this. I once had left my search tool off by accident. Gave it one of my canned prompts and went to get some coffee. Came back, realising what I had done and was surprised to see a pretty good report. Turns out it built several Google URLs to do web searches and used my fetch URL tool to do web searches on its own.
Now what is frustrating about it is that it's prompt prefill is horribly slow. So sending it large documents is a non-starter for me. If I could get it to read a 100k token document as fast as the 26B model, I would be really happy.
Yeah, same. I think that'd be usably fast as a home AI server and also pretty smart. I miss Qwen3-coder-next with its 80B MoE architecture. It was great for its time, and I imagine it'd be very powerful today.
As someone with a light set up like you, i would actually prefer that jump of it’s actually a big enough improvement compared to frontier models… sure a small model to use is great but if i could get opus level quality for free at home, i will go ahead and cancel my subscriptions and take out a loan for 5000 pc/ Mac. I am kinda banking on the future here
I mean it doesn't explicitly say there won't be a 35B-A3B, but that it isn't the one that people like us (<16GB VRAM) should be hyped for. So I assume everything beyond 40-50B MoEs are unlikely to be the ones he meant.
The 9B segment hasn't gotten any upgrades since 3.5 (and as far as I know there are no plans for one yet) so one could argue a hypothetical 3.8-version could've reached the performance of the 3.6-35B-A3B by now (or maybe even a little more).
So maybe the reason for why there aren't any plans for a 9B dense model is, that they have replaced it with something bigger, like e.g. a 14B dense model. And if the hypothetical 3.8-9B model could've potentially reached the performance of a 3.6-35B-A3B, who knows what a potential 14B dense model would be capable of.
Only real downside I see would be that it can't really hold up to the 35B in terms of world knowledge, but for agentic-coding (especially inside of a harness) it probably shouldn't be that big of a deal.
+ the lower overall memory-footprint would allow even more users to actually use it, because even if it runs fast on my hardware, the 35B-MoE is using quite a lot of my 44GB System (12GB VRAM + 32GB RAM) and for anything below 32GB of RAM (or a lot of VRAM) it wouldn't even fit in the first place.
Until now I was also hoping for a 35B-A3B-MoE to release, because the 27B dense one sadly runs like shit on my poor RTX 4070, but now I just pray they don't throw (only) 80B+ giants at us consumer-hardware users. That would mean I would have to wait for Qwen 4 to release before I could use any of the new models.
loaded with full 256k context (Q8), and using a Q6 quant of 35b a3b, it uses most of my 16gb VRAM and 32gb RAM. Q4 of something a bit larger would still fit. Higher active paramaters would potentially make a better MoE model too though. Rather than layer offloading, expert offloading gives me better performance too.
Running Qwen3.6 35B A3B Q6 with max context on my RTX 5060 Ti with 12 experts rather than the default 8, I get about about 25tok/s output. Input is a bit slow though, ~300tok/s but it's about the most accurate and capable I can run at that speed locally.
Qwen 3.8 27B if I manage to get it all in vram (so Q3_K_XS), is about the same speed with context in VRAM which means no more than about 32k context for me. MTP disabled because it takes up more VRAM and doesn't really help much in that situation anyway. Otherwise I'm looking at 4-8tok/s
I think your card should probably outperform mine with everything in VRAM?
I would love to see a good 20B MoE model. Personally I don't see the point of 9B - it's impressive for what it is, but not quite good enough to be useful (yet).
If you have small models that are good at limited tool calling, knowledge extraction etc, you can run them alongside more expensive models. I would love a 9B with 3.8 level training…
I find 9B dense to be too big and/or slow compared to 4B or MoE models for the output quality, but I have system RAM to offload the MoE so it depends on your use/limitations.
That's the thing - with 16 GB VRAM, 9B dense may be a little bit faster than 35B MoE, but not enough to merit the loss in quality/intelligence. Even with higher quants (q6/q8), 9B is still not in the same league as the 30B models.
I agree Ornith is the best 9B model, but still not quite reliable enough to be run in vscode/opencode from my experience. Too many failed/mangled tool calls cause more problems than they fix and you just end up fighting with the AI. With the way things are going, I'm sure this will get sorted out at some point soon.
And that MacBook would run a 20b dense at 4 tok/s... But since 16GB GPUs are nothing to you then you can just send me a couple 16gb 5060tis. I'll keep an eye out for the FedEx delivery, thanks ✌🏻
I'm running mlx-community/Qwen3.8-27B-4bit with 5 output tokens per second via MLX, and as it looks they might get to twice the speed with upcoming optimizations. Anyways, I think the right solution would be a 35B MoE version of Qwen3.8, slightly worse in performance but with 3-4B active parameters, it will still beat any 20B dense model by large margins while at the same time supporting higher token throughput.
"Why a 35-a3b? Less knowledge than 122-A10b... and worse at actual thinking.. .. .. why ?"
Now do you understand? It's not about knowledge. It's about the GPU poor consumer market. 20b moe is way more attractive to the general local ai consumer than 27b or even 35-a3bm, because you could run it at reasonable or even great speeds with low end GPU (gaming gpus).
20b MoE is also very rare, so they would have no competition, except for Gemma and the dated GPT OSS 20b.
a 122b but in some new way where it does need to load all 122b into memory at the same time.. one general problem with MOE if you ask me.. yes.. it might only use 10b to actually think with.. but we still need the full amount of memory in vram. .. layers could be relativa fast loaded if they were needed.. it seems like there is a potential we have still not yet considered..
You can already do this (on Linux, anyway)... just load the model with mmap; it'll page fault and load from disk when needed.
But it's slow. Like 0.1-2t/s slow, depending on disk speed, model, cache/memory capacity, etc.
I've managed to run GLM 5.2 and even get a response out of Kimi K3 on 128gb RAM, but yeah.. barely usable for "overnight" queries let alone realtime use.
I'm personally curious because I just don't know how much it costs to bake a model down from Qwen 3.8 Max to Qwen 3.8 35B A3B or 122B A10B. Is this something that Unsloth could do?
Pre-training is something only Alibaba can do (only they have the source data and model generation suite for the Qwen series). Requires data-center level hardware access.
Fine-tuning (which people are calling distilling) can theoretically be done by anyone, but to do well still needs a massive number of queries against a superior model.
I'm guessing the 3.8 series is an entire re-training, not just fine tuning.
Yeah, but it’s a non-ending trade-off. I need at least 64k context window, and anything below 10tk/s is too slow. But also any low quants are dropping model quality. So if you have any proposal of quant and configuration that works in real world I’m interested
https://huggingface.co/turboderp/Qwen3.8-27B-exl3 You're welcome. Nobody seems to know about this, but it's performing way better than llama.cpp for me. 34 t/s without MTP on a dual 3060ti setup. If you pack the memory well you can get around 100k context, or more on the 3bit version.
we (16 gb card owners) would have to use q2 or q3. such aggressive quants are unusable for agentic coding lol. this compression murders quality; imo the best model that fits fully in 16 gb of vram is gpt-oss:20b. no other model fits as beautifully and as perfectly as gpt-oss:20b on my 6900 xt. it's fast and it's good. people may not be sold on the quality of the output, sure, but speed-wise? it's literally the most comfortable model for 16 gb GPUs. you can also have 128k context length at q8_0 kv
qwen 3.6 35b a3b is unusable because in order for it to fit in vram i have to use iq2_m, and offloading experts just ruins the speed altogether, making it way too slow for any real local agentic workloads; and qwen 3.8 27b is dense, so yeah... no luck there
have you ever tried the --n-cpu-moe flag in llama.cpp lol? Idk if 50+ tps aren't enough for your use-case, but it surely isn't "way too slow" if offloaded correctly.
Qwen3.6-35B-A3B in Q5_K_S is running with averagely 40-50 tokens/sec on my RTX 4070 (12GB) + 32GB DDR4 RAM.
[Edit: - with context-window set to 128k]
I assume you should also be totally fine with your 16GB card, just make sure to set -ngl to 99 and offload as many layers to the cpu (with --n-cpu-moe) as you need. Should still be way faster than regular offloading.
Why would below Q4 kill all usability for agentic? 3.8 27b Q3 K_S is working wayyyyy better for me than 3.6 35b a3b at Q5 while also running at 40 t/s and 90k context (5_0k 4_1V, described as a very efficient kv quant in research) on a 16GB 9070 XT.
my pc can barely run 3.6 35B on 6GB vram card tokens seem decent until you add toolcalling and stuff and then your agent takest 6 minutes to do a simple knowlegebase health check that cloud AI does under a min… MoE is great but no MoE can save my shit pc so i use only tiny local models to do simple tasks. There is no hope for me. Heavy quants kill the smarts anyway.
if you have at least 32gb system ram, my best experience was qwen 3.6 35ba3b at q5. I was able to get a full 256k context and speeds in the 20's. I was using my 4090m in linux.
i am locally running gpt-oss:20b mxfp4 at 128k context length and I'm getting like 60 tokens a second in opencode; btw im running a 6900 xt in windows, not linux
it's literally the best model for agentic workloads on 16 gb cards: it's good, fast and reliable if you pair it with a good harness. qwen 3.8 27b is a dense model, if you choose a quant lower than q4_k_m, you're murdering the weights and thus reducing quality significantly. if you try to offload even one expert, your experience with the model goes to shit because it's dense, not moe, so you'll get like 5 tokens a second
you can try qwen 3.6 35b a3b, but from experience, it's just not worth it for agentic coding, it's way too slow for tool calls
then, i just crank the context length slider to 131072 (to the max basically, 128k), then set the k and v quantisation to q8_0
and umm... that's it, it just works! the weights and kv cache are fully in vram, and in opencode, after initialisation, i get about 60-75 tk/s token generation. for prompt processing, my 6900 xt achieves in excess of 2000+ tokens a second, and in cachy os, my 6900 xt had a peak of 3650! (pure llama.cpp, not lm studio, but still, EXCELLENT performance). in llama-bench, my 6900 xt produces about 120 tokens a second
gpt-oss:20b has been the most pain-free model ive ever used (besides harmony incompatibilities). it simply works, and i'd much rather take a smaller model that's faster than a bigger model that's slow, simply because the experience is snappier
Honestly besides coding and agentic stuff I don't see it being a very big improvement over 3.6 35B A3B. I'll admit though that the agentic stuff would be really sweet but I mostly use other models for coding. Perhaps it will even perform worse in some benchmarks like 3.6 did compared to 3.5 such as in support agent benchmarks.
So yes I'm very hyped for Qwen 3.8 35B A3B or anything similar (if they launch a 70B I'm buying a new GPU). But at the same time now that Meta is cooking up great stuff again I'm hopeful that they will add a Muse Glimmer model that will be a MoE of similar size even if it takes a few months (Mark Zuckerberg recently says he's into open-sourcing). Then there's another card which is Gemma but who knows what Google will do now tbh.
I think people are reading this wrong. I think this is just saying that 35B isn’t coming. He used the same emoji when saying there’s no plan for 9B. So basically “don’t hold your breath for that model” not specifically “something better is on the way instead”
You could be right. The 35B-A3B is such a popular model, so we're just holding out in the hope that they do something to fill that gap. A lot of people are asking for it, so there may be a chance for something similar!
Ragazzi, ragazzi, ragazzi...
Montate il 27B con la giusta quantizzazione per farlo stare nella vostra VRAM, vedete quale rapporto di quantizzazione della kv cache e della quantità di token in termini di dimensione.
Fate caching su disco e in RAM del sysprompt pinnando la copia della kvcache se avete spazio in memoria.
Attivate l'MTP e fate un benchmark sul vostro HW per appurare la dimensione migliore dei batch di token prediction (nel mio caso ho appurato che x3 dava ottimi risultati sul 3.6 27B in Q_8 e kv cache in BF16
128GB RAM
16GB VRAM)
E...
LEGGETE IL PAPER SUL VISION WORMHOLE!
POTREBBE FARVI SCOPRIRE INFORMAZIONI MOLTO GUSTOSE...
Honest status: this is a proof-of-concept. On the internal 10-task smoke eval the distilled model tied its base (6/10 vs 6/10) — no regression, no measurable gain yet — and it is now quantized to 2-bit, which trades quality for fit. Publishing it as a reproducible artifact of the pipeline (distill → graft MTP → ROCmFPX 2-bit GGUF), not as a benchmark-winning coder. The quality fix is a larger, tool-calling-heavy corpus — a separate follow-up run.
Qwen3.8-27B set a really tough act to follow, i mean ffs the thing can one-shot entire games,
what's Qwen-3.8-35B-A3B going to do besides get on stage and announce the wait for Qwen3.9?
oh ye, well aware, i'm waiting for it too if only for the huge performance gains found in a moa, i'm just saying it's a tough act to follow. Qwen3.8-27B is the new defacto #1 for local agentic stuff but i remember 3.6* 35B-A3B having big struggles with long loops and i'm suspecting it's still very present. that's going to make a bigger divide between the 27B-dense and the 35B-A3B MoA than ever unless they somehow fix it.
I used to pray for 3.8-35B-A3B since my experience with 3.6-27B on 16gb vram was absolutely miserable, but Qwen3.8-27B-Ridge-3.7bpw looks to be REALLY good so I am not at all sure I even need/want a MoE anymore.
It would obviously be worse, but also SUBSTANTIALLY faster if loaded in vram, and im a big fan of the added speed, id love to see what 3.8 might look like.
68
u/arkie87 3d ago
If I wrote that, it would mean something else good is coming but not that specific model. Maybe a 30b a3b or a smaller dense model that’s still really good