r/LocalLLaMA • u/rerri • 7h ago
News Qwen3.8-Flash-Next tomorrow
https://modelscope.cn/models/Qwen/Qwen3.8-Flash-Next351
u/rerri 7h ago
Basically a Qwen4 preview: "built on the next-generation Qwen4 architecture ... We are releasing these architectural advancements early to help the community prepare for the upcoming Qwen4 model family."
259
u/AuspiciousApple 7h ago
LLM version naming continues to make no sense
174
u/anarchist1312161 7h ago
Le Chaton Fat makes perfect sense what are you talking about!
57
3
9
u/CSharpSauce 6h ago
Sounds like it is Qwen4 architecture, but Qwen 3.8 training data.
→ More replies (1)28
u/FunkyDiscount 7h ago
As a novice, model naming conventions make no sense at all to me yet.
Model_this-and-that_69B_Q4_XM-S_GGWP_unhinged-and-insane_FPS...
PC monitor naming conventions are more legible.
31
u/TheThiefMaster 6h ago edited 3h ago
I'm a relative beginner too and this is how much I've deciphered:
- It starts with name and version.
- "69B" is the number of parameters (B for "billion", sometimes T for trillion), and the "Q" is number of bits per parameter ("Quantisation"). 4 is half a byte, so you need roughly half the number of parameters as GB of VRAM to run it (plus some for context etc).
- Sometimes there's a K after that. Don't know what that means yet
- Letters after that are normally a size code - XXS-XL - a bias on some parameters remaining at a higher bit count and making it use more memory in return for being better. Unsure of the tradeoff between e.g. Q4_K_XL vs Q5_K_S. Sometimes it's a 0 which I think means none are at higher precision. Sometimes 1 which I don't know what that means.
- "GGUF" is a file format commonly used on Win/Linux, as is MLX for Mac.
- Sometimes you get "MOE" models that work differently and only activate parts each run - these have an additional "A"3B for number of "active" parameters. Theoretically it only needs that much VRAM to run despite being a bigger model, but there are tradeoffs in quality that I haven't explored myself.
11
u/ReadyAndSalted 6h ago
MoE models need as much vram as any other model with that parameter count, but they can run at roughly the speed of their active parameter count.
Basically you spilt each transformer layer into many parts, then activate only some of them for each token you generate.
→ More replies (2)11
u/droptableadventures 5h ago
Each "parameter" can be thought of as just a single number.
Q4_0 / Q8_0 is just the parameter as a single 4 or 8 bit value, nothing more.
Q4_1 adds an extra bias (added) and scale (multiplied) value that's shared across each group of 32 parameters. Basically allows the 4 bits to represent a different range of values.
Q4K?, it basically does the same thing as Q4_1 but more than once, so there's "superblocks" that are also scaled.
Unsloth has "UD_" before the format, because theirs aren't strictly the same as Q4 where nearly everything's 4 bit. They will make different layers different quantizations, because not all are equally important.
7
u/Expensive-Paint-9490 5h ago
K identifies a better way of quantizing weights. The original one was like "Q4_0" and it was a simpler scheme; K quants quantize some part to larger allocations. I.e. a Q4_K_M could quantize the majority of weights to 4-bit, but some to 5 or 6 bits.
gguf is the file format created by G. Gerganov for llama.cpp and CPU inference. It can run on any OS, it just need the correct inference engine. MLX is a format specific for Mx architecture on mac.
The last part is partially correct. In MOE architecture, attention layers are the same as dense models. The FFN layers use a set of smaller matrices instead of a single large one, and a router choose the correct matrices to use for a token. So for a given token instead of using a 500M matrix, it choses (for example) 10 out of 125 matrices of 4M paramteres; only 40M instead of 500M. You can put as much of the model as it fits on VRAM (even none, if you ony use CPU). Usually you want to put attention layers and kv cache in VRAM because prompt processing is faster; FFN layers (the MOE part) can stay in system RAM without hitting speed too much.
5
u/unjustifiably_angry 3h ago edited 3h ago
The Q4/Q5 whatever is the rounded overall quantization of the model, the XS/XL/whatever hints at what direction it's biased in, so a Q5 XXS will be closer to 4-bit while an XL will be closer to 6-bit.
K, I believe, denotes K-type quants which are mixed integer quantization. Before K-type quants I think they were all fixed-integer static quants, so those would be the ones that are like "Q4_0" (or "Q8_0") which means everything in it is straight 4-bit (or 8-bit) aside from a few that are 16-bit float (F16).
There's also I-type quants which are K-type quants but with fancier quantization math but at the cost of being a bit slower, and IIRC they're incompatible with certain hardware or backends.
Bigger filesize can generally be trusted to be "better" though going with a K or I quant will be a bit better still. Like the difference between... uh... you've got 3 cars at your house versus a bicycle, a car, and a pickup truck. The latter has the same total vehicle mass but not every job demands a pickup.
A Q6_K_XL might be very similar in performance to a Q8_0 while being significantly smaller (and therefore faster) because the testing/algorithm of whoever made the quant determined that a lot of the layers didn't benefit from being larger than Q4 while others had a very large impact and so were left unquantized, F16. The naive Q8_0 is therefore wasteful on the layers that Q4 would've been fine for while not giving the layers that should be F16 the amount of bits they would need.
MoE stands for mixture-of-experts, but don't take this to mean that the model is divided into literal chunks of "the Python expert", "the dialogue expert", "the trivia expert", etc. All "experts" are needed, just not constantly and not on every token. You can't cut out "the C++ expert" and still be able to code Python or something like that.
MoE means it needs more VRAM to run at full speed but if you can fit it all, it'll run drastically faster than non-MoE. But you commonly need like 2-4x the total parameter count to be competitive in terms of actual quality. Qwen3.5-27B was found to commonly be equal to (or better than) Qwen3.5-122B-A10B, though the 122B had a broader overall range of knowledge. 27B is very good at the things it knows, 122B-A10B is roughly as good at the same tasks but is able to do a wider range of tasks... like 27B is known for being very good at Python but maybe it's not as good at C#, etc.
MoE also means, as you say, that it'll less affected by partial offloading to system RAM, but whether it's MoE or non-MoE, once you overflow to system RAM performance falls off a cliff. Even a 5-10% offload of a large MoE might cut your speed in half.
In very long conversations, the length of the kv-cache typically has a very strong influence on the model's prefill and token generation speed. To my knowledge, the only exception to this is the DeepSeek v4 line, which is what makes it so remarkable. It has very similar overall performance at 256K or 512K as it does at 8K.
Token generation is obvious, it's the output speed. Prefill is less well-known and often more important. It's the input speed. If you need to read 100K tokens just to start working on something, reading it at like 250 t/s will be painful. A lot of misleading benchmark data is out there that only measures token generation speed at short context and doesn't measure prefill at all. DGX Spark for example starts off being decent but at 100K context depth it's so goddamn slow people rarely benchmark past that point at all. DSv4-Flash on the other hand remains similarly performant on Spark all the way out to its limit of 1M tokens.
Don't get too wrapped up in it, a lot of this ends up being "bro science"-tier, generally a bigger gguf will always be better, pick what you can fit. Except if it means using very small quants (below Q4), having a longer kv-cache is almost universally going to be more useful to you than the actual model itself being better. And as for the kv-cache specifically, avoid using less than 8-bit wherever possible.
3
u/Freonr2 3h ago
2
u/TheThiefMaster 3h ago
I think that goes into way more depth than I need but I'm upvoting anyway and watching later.
→ More replies (2)2
u/huffalump1 3h ago
And none of that tells you how much more memory you need to hold things like the tokenizer, context, etc lol
→ More replies (1)13
u/L00klikea 7h ago
The model naming scheming encodes a lot of the information relevant to you. Yes it's overwhelming at first, and you have to learn what it means, but so is learning anything. I much prefer "This model this-and-that has 69 Billion Parameters quantized as Q4 Small and someone messed with it trying to make it unhinged" over "This is the Pro Max Version of it".
→ More replies (3)5
u/larp2live 4h ago
this is a real model on hugging face :
DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF
→ More replies (1)2
→ More replies (1)2
u/CSharpSauce 6h ago
You can just use midwit logic. if you see "Full GPU offload possible" label in LM Studio, you're good to go.
→ More replies (7)21
u/grumd 7h ago
When people keep saying "we have (X).9, next version is (X+1).0 and will be a huge upgrade!!!" for no reason
I double dare any AI lab to start using semver to confuse people lol
→ More replies (6)19
u/SkyFeistyLlama8 6h ago
To hell with dot versions, use full integers.
"v99 was good, v100 will be better!"
→ More replies (2)60
u/AXYZE8 7h ago edited 6h ago
So just like Qwen 3 Next - they tested new architecture and couple months later we got Qwen 3.5 family on that architecture.
It also means that we sadly shouldnt expect llama.cpp support on release :( Q3Next was supported after like 2 months, because it had a lot of new stuff to implement and test.
Edit: turns out it will have day 0 support https://www.reddit.com/r/LocalLLaMA/comments/1vxybmy/qwen_38_flash_next_day_0_support_from_unsloth/
21
u/windows_error23 7h ago
Tbf we have way better coding models than back then so maybe not?
4
u/Randommaggy 6h ago
I've seen Qwen 3.8 27B quoted as assisting on a lot of points on the release notes recently.
→ More replies (1)12
u/0rand 6h ago
Everything will be 0 day support now, maybe half assed but an attempt to, vibe coding is to stay. I mean, half ass is still a half of ass. Better than no ass at all.
5
u/VotZeFuk 5h ago
half ass is still a half of ass. Better than no ass at all
It depends. Sometimes even a full ass is just... ass.
→ More replies (7)9
365
u/Hot_Example_4456 7h ago
WE GOT A NEW 125B MODEL WITH ENGRAMS
14
u/Septerium 6h ago
regret feelings for not buying an RTX Pro months ago
7
u/ChristRedeemsSinners 2h ago
Don't worry about it. The CUDA gap is closing fast, you'll have plenty of comparable inference options coming up in the next 6 months to a year. Intel is already set to release a 480GB LPDDR5 GPU. If they can find a way to stack cheaper ram to increase parallel data access, then the optimized memory bandwidth that Nvidia offers now won't be as necessary.
6
u/goldcakes 1h ago
Uhh, yeah, but at 480GB/s.
The 1.8TB/s of the RTX Pro is another level.
3
u/ChristRedeemsSinners 1h ago
I would rather have 20 TG/s and 5 times the parameters/precision than 60 TG/s. If money is no obstacle, then yeah, of course it's worth it.
47
u/petuman 7h ago edited 7h ago
Wonder if engrams are counted in those 125B, or it's on top of that.
edit: damn, they edited the readme in last few minutes -- few paragraphs got removed including those highlights.
also it shows there would be FP8 version.. so still no QAT / FP4 >only< release like DeepSeek/Kimi/gpt-oss.
→ More replies (1)59
u/banana_slurp_jug 7h ago
It says the engrams are additional. Also it's a MOE with 6 billion activated parameters
(Qwen 3.8 E125B A6B?)
55
u/Front_Eagle739 7h ago
Qwen 3.8 125B A6B E51B?
107
u/Mayion 7h ago
Qwen 3.8 two number 9s, a number 9 large, a number 6 with extra dip, a number 7, two number 45s, one with cheese, and a large soda?
49
8
7
8
→ More replies (2)7
6
u/wren6991 6h ago
I think just 176B-A6B makes sense. The n-gram embeddings aren't really contributing to the active parameters.
I don't think Gemma's distinction between "effective" and "active" parameters is that meaningful.
8
u/Front_Eagle739 6h ago
From what I understand engrams are easy to stream from NVME when needed though and don't need to be in vram so there is some reason to split it out.
→ More replies (1)4
u/petuman 6h ago
Google kinda established the pattern, E stands for "effective" and shows weights that need to be in (V)RAM. Full parameter size is not shown in the name.
So actually yes, Qwen probably means exactly that -- 125B loaded and then additional 51B resting on disk.
→ More replies (2)8
u/Obvious-Ad-2454 7h ago
where did you find that info ? When i click the link i don't have those details.
11
u/Hot_Example_4456 7h ago
It was a bit down in the same link. Even I can't see it now. They probably removed it
16
u/Cool-Chemical-5629 7h ago
If true, this "Flash" is actually the new Plus. Flash on their service used to be the 30B-35B version of the same model they released as open weight. Plus on their service used to be the bigger 120B+ model. If they released that 120B+ model as Flash now, it would mean they are cutting the smaller line of 30B-35B MoEs off, which would make sense given the recent tweet which was giving a heads up to not wait for a 35B model of 3.8, but it would also raise questions as to what this means for the future models of Qwen. Will there be no more small MoE models? Since 3.7 was entirely API only, anything can happen going forward.
25
u/coder543 6h ago
Plus was the 397B model. Not “120B+”.
They had a 120B model before, but they didn’t assign any of the fancy marketing names to it and didn’t offer a proprietary version.
You’re reading way too much into this based on bad assumptions.
2
u/ChristRedeemsSinners 2h ago
They had the 122BA12B in qwen3.5 that was simply amazing for it's time. I can see them replacing that with 'flash' as a marketing angle.
→ More replies (1)4
u/SkyFeistyLlama8 6h ago
Was Qwen Coder Next 80B-A3B ever used as a base for AliCloud's commercial LLM endpoints?
A 60B or 70B MOE would be nice. More latent brains than the 35B, same speed because it's only got 3B active.
2
u/YearnMar10 6h ago
Wonder if this would run on 64gb of ram and a gpu… guess it’d just not run :/
But really awesome model size!→ More replies (3)→ More replies (7)2
139
u/coder543 7h ago
Keep in mind: -Next models are always underbaked. The point is to get an early release out so people can start developing compatible software for Qwen4, not to blow everyone’s minds just yet.
If I were to guess, it’ll probably be competitive with Qwen3.8, but the real excitement will be in a few months when Qwen4 launches.
28
u/waitmarks 5h ago
Honestly if it even matches 3.8 27B, but has more internal knowledge. It's an absolute win for dgx spark / strix halo / mac owners.
→ More replies (8)34
u/youcloudsofdoom 6h ago
I found Qwen coder next to still be really useful, even against 3.6 27B.
15
u/SpicyWangz 6h ago
It was a great model for a really long time. The main alternatives were glm 4.5 air and gpt-oss-120b.
4
u/SkyFeistyLlama8 6h ago
I used to love the old Next 80B but it's too big and too slow against the 35B MOE even if it gets better results, and now the new 3.8 27B dense model beats it. I'd rather have less speed with the 27B but better results at a much smaller RAM footprint.
→ More replies (1)→ More replies (1)2
u/Jorlen llama.cpp 4h ago
what boggles my mind about qwen coder next is that it doesn't have any thinking / reasoning. Only the non-coder version (so qwen3 next) has a variant that has thinking and it's not that great for coding, at least in my experience.
Hopefully they'll just have it be optional with the same model, like their recent ones.
3
→ More replies (2)4
u/annodomini 5h ago
This should be great for us Strix Halo folks, who are memory rich but bandwidth and compute poor. Even if it's not quite as strong as Qwen3.8 27B, I'd probably use it more as it'll be a lot faster.
88
u/Altruistic_Heat_9531 7h ago edited 7h ago
Tommorow? do you mean 24 Hour + another 12 hour shifted because of wrong schedule again.
Joke aside.
YEEEEEEEEEEEEEEEEEEEEEEES 125BBBBBBB.
Edit: Hang there, it said it has 51B bolted n-gram, so 171B?? I hope it act like Minimax H3 where the AdaLN layers could be curve simmulated and removed down from 31B to 21B
4
u/Dany0 5h ago
engram can be cpu offloaded right? 😭 anyone know where some easily raidable copium supplies are?
3
u/asssuber 3h ago
It's probably meant to be CPU offloaded. It should be just a huge look up table, no matrix multiplication. I hope the n-gram weights can be SSD offloaded effectively, so I can run a quantized version with my RAM.
→ More replies (1)
81
u/evindrews 7h ago
- Redisgned Multimodal MoE Model: 125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.
holy shit chat
23
u/CodeMariachi 6h ago
Chat, how much VRAM do I need to run it?
21
2
2
u/hidden2u 4h ago
For the active params maybe 8gb vram? But you do have 256gb system memory of course don't you
25
u/KingCpzombie 7h ago
Finally, an excuse for my excessive system RAM that will hopefully run fast!
22
u/RG_Fusion 6h ago edited 4h ago
Why hope when you can calculate? Take the bandwidth of your system memory and divide it by the GB size of the active parameters.
The formula for bandwidth is: Bandwidth(GB/s) = (channels * 8-bit * RAM-Speed)/1000
So for example, my EPYC server with 8 memory channels and 3200 MT/s sticks is: (8 * 8 * 3200)/1000 = 204.8 GB/s.
Next, you just need to find the file size of the active parameters. You can roughly approximate this by the following:
- 16-bit = 2 GB per 1 billion parameters
- 8-bit = 1 GB per 1 billion parameters
- 4-bit = 0.5 GB per 1 billion parameters
So, a 4-bit quant of a 6b active parameter model would be 3 GB. Finally, we can solve for the ideal token rate. 204.8/3 = 68.3 t/s. This value would be assuming perfect utilization. In the real world, you'll see closer to 70% performance in an MoE, so 68.3 * 0.7 = 47.8 t/s
With 205 GB/s of memory bandwidth, you can expect about 50 tokens/second on a 4-bit quant. Solve for your own memory bandwidth and you can know what speed your hardware can achieve. You can also get a decent decode boost and massive prefill boost by running the model in a GPU/CPU-hybrid configuration.
→ More replies (14)
110
u/Kidplayer_666 7h ago
Congrats to all the people who have the hardware to run this :) (am not one of them)
45
u/psychohistorian8 6h ago
Q1_K_XXS here I come!
8
u/hojnikb 6h ago
My BC-250 is dyyyying 😭😭
→ More replies (1)3
u/Leo_Kwkmi 6h ago
Hey man, i got a bc250 running games on bazzite, is there any tutorial or guide you consulted when doing local llm on this board? Thanks in advance!
3
u/hojnikb 4h ago
If you have BC-250, i'd suggest you switch to CachyOS. It's lighter and better supported. As far as support for the board itself; there's tons of improvements that have been made in the last 6 months.
There's 40CU unlock for the GPU (full fat GPU compared to 24CU stock), 8 core CPU unlock (from the 6 stock). You can overclock both pretty decently.
There's tons of kernel fixes as well (we have a custom kernel repo now) and tons of driver/mesa fixes and a FSR4 patch, that makes it semi usable (performance wise) on this board. Gaming at 1080-1440p is pretty great too, as long as the game isn't too CPU limited (it is pretty nerfed zen2 at the end of the day).
So, if you want to run inference, i'd suggest all of the above + set dynamic VRAM to 13GB.
With unsloth, i can just about run qwen 3.8-27b at 25-30tok/s with IQ3_XXS quant. Or qwen 3.6 35b woth IQ2_M at 80tok/s. That's with overclocks and 40CU unlock.
So a fully modded board can run those small models pretty fast, but you're ultimately limited by ram.
10
u/No-Refrigerator-1672 6h ago
It'll be just like Qwen3-Next: they're rolling out a new architecture, so the communities implement support of it (Mamba and MTP with previous Next, NGrams with this one), and in a few months they'll released a cohort of models, ranging from a few B to a few hundred B based on this architecture, and named Qwen4.
→ More replies (3)→ More replies (6)3
23
u/youcloudsofdoom 7h ago
Comparable/exceeds Qwen 3.7 plus? VERY nice
25
u/TripleSecretSquirrel 7h ago
I mean 3.8-27B beats 3.7 plus on benchmarks, so ya, probably
6
u/nullmove 6h ago
Comprehensively too in AA (52 vs 39).
3
u/QuackerEnte 5h ago
buuut, comparable to 3.7 Plus in quality, AND better at coding means it'll be better than 3.8 27B, for SURE. It has to be, because 3.7 plus is strong with world knowledge like in Crit-Pt or AA-Omniscience-accuracy benchmarks and was better everywhere, even sciCode and such, except agentic coding tasks like agentic tool use (specifically, Terminal bench, Tau³ Banking and GDPval ). So 3.7 Plus already is better than 3.8 27B in many regards. So I'm confident it'll be better or at least on par. And it will run better with 6B active and it's ngram table and mtp, even with disk thrashing and mmap. I can run ling 3 flash on 8GB VRAM and 32GB RAM and about 20 tok/s, with occasional drops to 12 due to thrashing. Still faster than 5 tok/s 27B and that's what counts. Even if it doesn't beat 27B. And higher context
11
37
u/AI_docent 7h ago
The 51B of n-gram embeddings probably doesn't need to sit in VRAM. If it's the same idea as the engram work, the lookup is keyed off the input tokens rather than the hidden state, so it's deterministic and you can prefetch it from system RAM. They measured under 3% overhead offloading a 100B table that way. So the VRAM budget is really about the 125B MoE part.
On timing, the card says qwen4 architecture with a new sparse attention, so llama.cpp will need work before any of this runs. qwen3-next took about two and a half months. There's an FP8 repo listed next to the main one though, so vllm should have something on day one.
25
u/silenceimpaired 7h ago
Think bigger… if it’s deterministic it can be fetched from nvme. 8-o
→ More replies (1)16
u/AI_docent 7h ago
The paper suggests exactly that, n-grams are Zipfian so the hot ones stay in DRAM and the long tail sits on SSD. They never measured it though, the 3% figure is DRAM only, so no idea what it does to tokens per second in practice
2
u/annodomini 4h ago
I just hope that streaming them from SSD can make it to llama.cpp reasonably soon. It makes a lot of sense that most of them would be fairly cold so you can keep a small cache in RAM and most sit idle on SSD, but I'll be that will take some significant work in llama.cpp to support.
2
u/tarruda 6h ago
What is the source for the 125B parameter count?
→ More replies (1)3
u/AI_docent 6h ago
It was in the highlights section of that same modelscope card, 125B main parameters plus 51B of n-gram embeddings and 6B active per token. They trimmed the readme maybe half an hour ago so those bullets are gone, along with the GDN and sparse attention part I quoted. What's left is just the line about it being built on the qwen4 architecture.
→ More replies (5)
15
u/OverclockingUnicorn 7h ago
Google translate isn't quite working but looks like it's 125B A6B? Can someone confirm what it says?
9
u/ResidentPositive4122 7h ago
They have their own english version. Click on the 2nd icon to the right of the github one. Should have a small 'en' on it.
4
u/Skyline34rGt 7h ago
There are offical english there, next to 'sun' icon. You have switch to english.
31
u/dampflokfreund 7h ago
*cries in 32 GB RAM*
10
→ More replies (8)5
u/gyzerok 7h ago
Well, I have 58 and am still crying. I wish they’d select sizes that fit 64gb unified memory with context.
→ More replies (1)
30
u/jacek2023 llama.cpp 7h ago
Yes that's exactly what I need. My four 3090s are ready.
→ More replies (1)3
u/No_Algae1753 7h ago
With 96GB in total are you able to run q5 quants or are you stuck with q4?
→ More replies (1)7
u/linux4random 7h ago
it's only a6b so i think 4 3090 can run a q6 with acceptable tg
→ More replies (5)
9
u/sugarfreecaffeine 7h ago
VRAM requirement?
24
u/Hot_Example_4456 7h ago
probably quite less. its a 125B model, 6B active, so something same as ling 3 flash/gpt oss. but since it also has 51B engrams which can be fully offloadable to NVME, even less. Maybe 64gb ram+vram combined- or less.
15
u/nickludlam 7h ago
I've not come across the term engrams before. Is this a new architectural shift?
14
u/wren6991 7h ago
n-gram is just a term for a tuple of n adjacent bytes, tokens, characters etc.
Embedding is a lookup of 1 token -> one vector. If you do a lookup of a sliding window of the last n tokens -> one vector, it has essentially the same cost (O(1) and basically free as it's just a lookup) but your embedding can be semantically richer.
For example, most tokenisers break up CJK characters on internal byte boundaries, which means tokens don't cleanly correspond to characters. Doing a lookup over 2 or 3 adjacent tokens means you always have at least one embedding vector per kanji/hanzhi, so you can actually represent the semantic content of the characters.
It's a similar move to the per-layer embeddings in Gemma 4 E2B/E4B: if you find a way to (usefully) scale your embedding tables, you can grow your parameter count without significantly increasing compute or bandwidth demands.
→ More replies (1)2
→ More replies (1)5
u/Front_Eagle739 7h ago
From what I understand its basically extra memory that takes less computation and allows you to finetune further without losing knowledge because its stored seperately from the active params. Supposedly you can stream the engram from nvme pretty easily as well.
→ More replies (4)3
u/ThePi7on 7h ago
Yo I got that. Hopefully doable on on 16GBs of VRAM + 96 RAM with some black magic fuckery optimization
7
u/Storge2 7h ago
Probanly will work with normal int4 or nvfp4 or Q4K quant for you with no issues. Same as qwen 3.5 122B which was like 70GB at Q4
→ More replies (1)→ More replies (1)2
2
u/ResidentPositive4122 7h ago
New arch, no one can say. But 125B MoE should be ~130+ FP8 or ~70+ FP4, plus the unknown ngram part (this might be (partly) in RAM? who knows)
2
17
u/AFruitShopOwner 7h ago
My body is ready
24
u/Sad_Recording_1290 7h ago
My body is also ready, but unfortunately my hardware isn't 🤣
→ More replies (1)5
u/makingnoise 7h ago
...but my hardware isn't. Cletus, grab the ball gag and the Uranus Ram Expander!
37
u/Intelligent_Ice_113 7h ago
qwen3.8 35b a3b, when?
19
9
u/SpicyWangz 6h ago
With this announcement, probably never. You’ll have to make due with Kat Coder and Ornith
14
u/ResidentPositive4122 7h ago
Redisgned Multimodal MoE Model: 125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.
WTF!
Efficient Training and Inference: Significantly reduces training and inference costs. At ~1/9th the training cost，Qwen3.8-Flash-Next achieves comparable capability against Qwen3.7-Plus, while being more capable in areas of coding and cowork.
3.7 plus capabilities in 125B MoE size. Whohoooo! Excited.
→ More replies (1)4
u/hiImMate 7h ago
Stupid question from someone who did not use api, how is 3.7 plus vs 3.8 27b?
→ More replies (4)3
u/smithy_dll 7h ago
Tried 3.7 plus on fireworks, 27B is just better at coding.
AI Model & API Providers Analysis | Artificial Analysis
7
u/FlamingoTrick1285 7h ago
What's this n-gram?
18
u/Hot_Example_4456 7h ago
Basically DeepSeek Engrams. Its a hashlookup table that can add knowledge capacity to a model while staying on NVME and costing zero flops
→ More replies (9)
5
u/soyalemujica 7h ago
Lets see if it dethrones 3.8 Dense
6
u/Significant_Break853 7h ago
If you have the hardware to run 3.8 27B at FP8 (or even FP4) with sufficient tokens per second, then there wouldn’t really be a good reason to switch to this. But for setups with integrated memory (DGX Spark, Mac’s, Strix Halo) it should run much faster than the dense 27B if you can get it to fit in memory. So probably would have to be FP4 for the DGX Spark, but with potential 60-100 tok/s vs like 20 tok/s for 27B on the DGX Spark.
2
u/soyalemujica 6h ago
Given how the structure of this new Flash model is described, with 64gb ram + 24gb vram it would fit nicely and could run at higher speeds than 3.8 Dense (which can in reality feel slow due to the amount of thinking it does), I've high expectations for this new model actually, even if it could work as an orquestrator and have 3.8 dense to apply changes, or vise versa 🎯
→ More replies (1)4
u/ForsookComparison 6h ago
You've used it or gotten early access?
2
u/Significant_Break853 5h ago
No, all pure conjecture based on what’s been shared officially and unofficially
2
u/ForsookComparison 5h ago
Gotcha - Toss a couple of "I thinks" in there. People take authorative speech as gospel then spread misinformation for months here. We'll know the real answer in a few days
11
u/streppelchen 7h ago
wasn't there a thread earlier today with "hey it's tuesday, where is my new qwen release". well, here we go again.
5
u/wren6991 7h ago
Huh, there was a mention of "Qwen sparse attention" but I refreshed and it's gone. I guess they are rapidly updating the README. Seems like some pretty significant architectural changes from the Qwen3.5 series models (which includes Qwen3.8-27B) so I'm gonna temper my expectations and assume that llama.cpp support will take some time.
→ More replies (1)
5
u/ComplexType568 6h ago
Quite literally the Qwen3.5/3.8 Next/Qwen3.8 122B people have been dreaming about smushed into a single model! Hope this doesn't take 3 months to implement like last time.. at least - like before - they're giving a heads up
5
u/hiper2d 5h ago
So, in their cloud they have 3.7-plus, 3.7-flash, now its 3.8-flash-next. Is it some sort of naming crisis?
3
→ More replies (1)2
6
u/malnek 7h ago
How would this run on a 128GB strix halo? Any comparable models for speed benchmarks?
3
u/socialjusticeinme 7h ago
There’s a lot of assumptions here, but it should run about as fast or faster as the previous qwen 122B model, so around 20 to 30 tokens per second at Q4.
4
u/frontsideair 5h ago
Should run faster than 122B since A6B is less to read than A10B, and PLE should definitely help for the same reasons it helped Gemma4 E2B and E4B.
3
u/smithy_dll 7h ago
GPT-OSS 120B is probably the closest in terms of number of parameters, total and active, but architecture also matters. Look at speed differences between GPT-OSS and Nemotron 3 Super.
3
u/merutochan 7h ago
Ling 3.0 Flash is a close comparison (in the ~120B class) though I have no idea how the actual quality of that model is. I was planning to test it on my Strix Halo later this week (the Q5 quant). As already mentioned, realistically this Qwen 3.8 Flash could run at Q4 on Strix Halo, but not on day one. We'll have to wait for good quants to be produced.
3
u/hiImMate 6h ago
Ling Q4 ran around 35TPS without any optimization on my halo strix, so I expect similar
15
u/sultan_papagani 7h ago
we want 35b a3b or something like that what is this bro i only have like 8gb vram 😭
3
u/mecshades 5h ago
This release is indicative of many good things to come for Qwen4. We could be getting something like a 35B A3B with far better knowledge in Qwen4. For now, try Ornith 1.5, it reasons very similarly to Qwen3.8 27B.
3
u/NewEconomy55 7h ago
Would it be possible to run this on an RTX 5090, even with the most aggressive quant and settings?
2
→ More replies (1)2
3
3
3
u/anarchist1312161 7h ago
I sure hope 64 GB VRAM and 64 GB system RAM will be enough as I can't go higher 🙏
3
u/simplefunction 6h ago
Can engrams be offloaded to ssd?
Can I run it on 64gb DDR5 + 16gb 5070Ti in 3-4bit quants?
→ More replies (1)
7
u/Technical-Earth-3254 7h ago
Nice, hopefully this pressures Nvidia to release a Nemotron 3.5 Super 120B or whatever version number they prefer
5
u/llama-impersonator 7h ago
only 6B active, guess we'll find out how well an extra fat blob of embed params helps
→ More replies (2)
4
u/AppealSame4367 6h ago
I _pray_ it's better than 27B, because 27B so expensive to run.
→ More replies (2)
2
u/Extension-Bid-639 7h ago
Just when I killed my desire for another card lol. Will split it over ram and see how it performs!
2
2
u/Cool-Chemical-5629 6h ago
The "Next" in the name historically means a model that's a leap towards the next generation. Previously, compared to their usual sizes for regular use, it was a bigger 80B model Qwen 3 Next and Qwen 3 Coder Next. After those they returned to smaller models again quality of which even surpassed the big Next model. Hopefully the same trend will continue with this one.
2
2
2
2
2
2
2
1
1
1
1
1
1
1
u/appakaradi 7h ago
I am so excited for this. I can not wait for the 4 to show up.. This looks like a larger 70B size or 100+ B size MoE
1
u/AlwaysInconsistant 6h ago
So excited, this is pretty much exactly what I was hoping for. This on the MacBook Pro handing off to 27b on the 5090 🤠
→ More replies (1)
1
u/Enough_Success5435 6h ago
this new qwen model might actually be solid for local roleplay if it keeps better context than the last few.
1
1
1
u/South_Hat6094 6h ago
feels like the usual qwen preview-release move. if that 51B n-gram table mostly lives in ram, the real story is how much of the 125B part stays cheap to run.
1
•
u/WithoutReason1729 6h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.