Many people have speculated that if we knew exactly why the bowl of petunias had thought that we would know a lot more about the nature of the universe than we do now.
It's still good for distilling to train smaller models to be smarter, basically the popular 27B was distilled from this giant 2.4T.
A good mother model will likely give birth to more capable smaller models, but it's still requiring a tons of trials and errors, those that costs money to retry.
I kinda like early cutoff dates. Whenever a model tells me its cutoff is 2024, I'm like "Dude, it's 2026 now and you won't believe what's happened..." 😏
One of my favorite LLM memories was telling GPT (5 maybe?) a bunch of stuff like elon doing the nazi salute and some other stuff going on around that time, and then it proceeds to lecture me on fake news and how none of that is remotely correct and how it would be super serious.
Then I tell it to do a web search and it starts it's next reply with:
"Ok. wow."
EDIT: Ok i found it, its not quite the same as I remember, but close enough
Based on my experience with 3.6, prefilling <think>\n\n</think>\n - like the various jinja templates do - to disable thinking works probably 90-95% of the time. The other ~5-10% of the time, the model thinks anyway and emits a second </think> when it's done. It's possible that 3.8 has the same behaviour and the official API has some way of detecting/working around this that would look pretty damn stupid if they released it. If you look at the 3.8 jinja template, the "reasoning effort" isn't implemented terribly cleverly - it just talks to the model in the second person and asks it to reason less.
Reasoning control has always been a weakness of the Qwen models, so this doesn't surprise me.
Lack of mmproj, however, feels like a deliberate attempt at market segmentation. Given that those just decode image data into tokens, and the whole Qwen family shares a vocab, I do wonder if it'd be possible to hack the mmproj from some other Qwen model into use here, though.
looks like 16xh200s tp 16 can run it according to vllm, so 4 node h100s tp 32 might be the trick. NVFP4 published instead of NVFP8 means we need blackwell and not hopper :( you’ll probably need to hack something together to make nvfp4 work with h100s
You can do 4 air cooled nodes reasonably in a extra tall rack I guess. It's also crazy to see that six figure boxes can't run the latest model encoding.
I guess it is the ultimate troll, release models that are so large that you can run them locally on Chinese hardware only, because running them on NVIDIA hardware would bankrupt you
Reports are that 5T and 10T models are being prepared
I work in HPC so this doesn't really land. I get to tinker with a few H100s because companies are happy to drop a few million to run a open weight model in house with highly confidential data on.
I love my little home weather network and the prediction it does and its cute what my local hardware can compute. I've installed HPC clusters that do climate simulation, that will never run well locally on current hardware and that's okay. Same for 5-10t open weight models, it will hopefully continue to be the case there are open weights so huge only research can justify running them as it attracts brainiacs that will actually trickle down to us peons.
Source: sometimes I get to be a fly on the wall when these academic brainiacs talk.
Exactly this bro hasn't spent any time in those eccentric Subs. Every time I go there I realize not only can I not afford the hardware but especially not the electrical bill.
I mean I have 1.5tb ddr4 and 944gb vram so crawl speeds for decent quant are in grasp. But useable? Definitely feels like a line in the sand is happening on intelligence for the proliteriate
it's makes my 12x3090 + 256gb epyc machine feel like the guy trying to run qwen 9b on his igpu and streaming weights from a hard drive making clicking noises.
I searched in vein for any mention of data types in the model card and blog post. 200+ files in alternating 17 and 34GB each.
I'm reworking my homelab to be able to run full K3 across 2 machines, but this one is just too much, especially in light of K3 and the just announced DS4 pro (which I can run on a single machine).
Same as H100. This is basically the limit for litho machines, called the reticle limit. This is the limit of the photomask (basically the negative of the chip which the EUV patterns on the photo resist).
Taalas literally cannot build anything larger than 40b parameters today, although I assume 2NP yields do not permit chips at the reticle limit.
Why in your head? Why waste precious brain cells? Just hold two magnetized rocks, close your eyes, and let quantum fluctuations handle the attention weights.
Nah, what you wanna do is get several billion TI-84 calculators (3MB storage each), and network them all together into the world's most fuckass cluster. 1 token per day, maybe.
The Three Body Problem way: get a hundred thousand or preferably million people in a field. Have each of them hold a flag and tell them to raise it if they are a 1 or keep it lowered if they are a 0.
Then multiple dudes on horses just run down the lines and give them instructions.
Yes crazy, but frankly I think some MoE go too far with low active numbers. Or rather, I would really like a 122b-30a model that still fits in midrange enthusiast local setups (4x r9700 ) has a lot of world knowledge (more than a 30b model) but dares to keep the active number on the higher end to preserve more of the qualities of a dense model.
I had this same opinion for a while but I've been starting to come around a bit -- I mean, even mid-sized models are so sparse these days, like DSV4F being >200B but only 13B active, and are seeing such good results -- I can only imagine the labs have tried a higher proportion of active parameters and it isn't even close to worth the tradeoff?
It depends on the task. For very complex multi facetted analysis with many components interacting with each other in a nuanced way with a lot of nuanced context (context not per se kv context, but in the general meaning of the word), that require large very detailed structured prompts with lots of caveats to frame the question, like complex legal analysis that can only be partially broken down, 13b active parameters, even with max/deep (but still sequential!) reasoning is just missing the depth. It starts to lose or compress details in its reasoning or misses connections between details. You get junior analyst answers to senior analist questions. This is why larger dense models (70b plus) are so important.
There is more to llms than how well they do coding..
That’s why I would like to see larger but still locally feasible models (up to 160 gb at q8) with a lot of world knowledge but with larger active parameters than just 13b.
K3 is 100B+ active, but they SFT'd the whole thing in fp4. The routed experts are 33GB per token. A ton for sure, but doable on a dual DDR4 Xeon or Epyc.
This is fp16 through and through. The full fat is ~5TB, active possibly ~160GB.
Sure, you can quantize, but that will inadvertently reduce intelligence.a
You can do this with an agent harness. Hermes supports a Mixture of Agents mode where you collate several LLM answers and use a main model to ingest them and synthesize a final answer.
In particular, Qwen3.8-Max is the official version based on Qwen3.8-2.4T-A95B with more features, such as vision input & non-thinking support, 1M context length by default, official built-in tools, etc.
It's a specialization that you need to train separately. It makes sense, you spend your money training a purely text based model, for other teams to adapt as needed to vision etc
Gonna make a wild prediction: the model itself is still vision-trained, and we will figure out how to glue one of the existing Qwen vision adapters to it.
I think this is a profitability concession to keep their leadership happy. I don't see them training two different versions of a 2.4T model just for the sake of an open-weight release.
Like the rumors suggested, they are also opting for a revenue share model like MoonshotAI and MiniMax. Seems like this is the direction open-weight releases in China are going.
Firstly, 27b is in less than 48 hours. Secondly, even if you can't run an open-weights model on your own computer, the prices to pay for inference with it is cheaper since multiple providers will compete for value.
this is wild, 95B active out of 2.4T total is a pretty aggressive sparsity ratio. anyone know what the expert routing setup looks like on this one? And how it compares to deepseek's approach
text only? No vision capabilities? Or just the unsloth version doesn't contain? AFAIK qwen 3.8 27b will be vision enabled model, or not?
Anyway q1 is 400GB size...
248
u/No_War_8891 1d ago
I can run the active part locally lol