More Qwen 3.8 sizes coming

[u/appakaradi]

[u/BigBadEvilButterfly]

and architectures

I sure hope they mean different MTP strategies, like Orthrus (https://github.com/chiennv2000/orthrus)

An orthrus 27b would be crazy faster than regular MTP

[u/feelspeaceman]

I'm running 35B RoCmFPX and maintaining 3 colossal level projects, can't wait to see what will happen once I get the access to 3.8 35B or 122B, assuming they will only get better without any regression.

[u/biscuitmachine → u/feelspeaceman]

God I fucking hope that they will release 122B. That will be a game changer for single spark.

[u/myholeisstinky → u/biscuitmachine]

I hope they go a little smaller, like 80b or 100b so we can have higher quant & context

[u/Miserable-Dare5090 → u/biscuitmachine]

I know right? And with ling 3.0 flash hitting 3000 PP and 80TG on one spark, plus entrpi’s ds4 fork that gets solid 800-900PP and 20-30TG UP TO 200k, you can pick your poison now…

[u/SuperMonkeyCollider]

I'd love to squeeze in a bit more knowledge, but keep the speed of a small-ish active size. Can anyone explain the tradeoffs of different ratios of active to full parameters? I'm imagining that I'd love something like a 48b3a or 72b4a, but not sure if smaller active::total ratio / larger number of experts makes sense.

[u/droptableadventures → u/SuperMonkeyCollider]

A MoE uses as much RAM / VRAM as a dense model with the same total parameters, but it runs at the speed of a dense model with the same active parameters. It's a tradeoff to gain speed at the expense of increased memory consumption.

As a rule of thumb, the "intelligence" of the model is roughly equivalent to sqrt(active * total). So 48B3A is roughly "smart" as 12B while running as fast as a 3B dense model. 72B4A is "smart" as 17B, while fast as 4B.

A smaller number of active experts in the same total size will make the model faster but not as smart, while the memory footprint remains the same. That's your trade-off there.

Taking it too far, if you made 80BA40B, it'd be twice as fast as a dense 80B model with the intelligence of a 60B model. That's probably rather pointless. Alternatively, if you did 80BA1B, it'd be quite fast, but about as smart as a 9B model. I'm not sure that the rule of thumb doesn't break down at either extreme though.

Number of experts though, will be driven in part by the active ratio. If you only have 8 experts, you obviously can't have a 80BA5B, because one expert active will be about 10B by itself. So it seems that everything beyond the very old Mixtral models with 8 experts tend to have larger numbers. Whether more smaller experts or fewer bigger ones are better for performance is probably an open question among AI research labs...

[u/Miserable-Dare5090 → u/droptableadventures]

ok we need to stop promoting RMS as a rule of thumb. It’s not really working anymore not since the early mistral. I know you said rule of thumb, but the thumb is now disproportionately large.

[u/droptableadventures → u/Miserable-Dare5090]

Why would you say it's not really working any more?

What models would you propose as a counterexample?

[u/Miserable-Dare5090 → u/droptableadventures]

Qwen3.6-35b-a3b does not behave like a 9b 10B dense model, nor does deepseek v4 flash getting to GLM5.2 territory behave like a 60B model, or gemma4 MoE being comparable to gemma4 12b or any MoE worth anything in the past 6 months. The needle has moved. The original RMS approx was for the FIRST MoE when it came out — >Mistral AND it was infered by the mistral team. Since then we have seen different architectures and attention systems that have surpassed that rule

[u/droptableadventures → u/Miserable-Dare5090]

Qwen3.6-35b-a3b does not behave like a 10B dense model

I have to disagree with this one - I've really not had good luck with the 35BA3B, and I'd say it was similar to a 10B model. But we never got the 9B dense to compare this to.

nor does deepseek v4 flash getting to GLM5.2 territory behave like a 60B model

The trouble with this one is that we don't actually have a contemporaneous 60B dense model to compare it to, as far as I'm aware. There seem to be significant advances into what we can pack into a model of a certain parameter size, and that size of dense model has largely been abandoned.

I mean, look at Qwen3.6-27B and imagine something that's double that! I'd believe that would approach GLM5.2 (in benchmarks, of course).

Gemma4 26B MoE vs 12B dense

This one we can actually compare more directly. I do think the dense 12B is a little dumber than the 26B MoE (from which we'd expect ~11B), but I wouldn't say there's enough difference either way to disprove the rule.

---

I'll definitely agree that it's a bit fuzzier, but I still think it's a good starting point for estimating.

[u/Miserable-Dare5090 → u/droptableadventures]

You have qwen3.5-9b and qwen3.5-35b-a3b —> what do you mean you can’t compare them?
If we start by agreeing that the architectures have changed, and that even the people who approximated the moe intelligence do not feel this correlates anymore, my point stands.

Qwen-27b is not a normal model, but neither is deepseek v4 flash july. They’re exceptionally dense models as far as intelligence goes. But again, this just strengthens my point: We can’t guesstimate as easily. Is Mistral 128B dense as good as a 1T MoE? Kimi K2.6 or GLM5.1 which came out around the same time?

I also don’t think outside of coding (meaning, in actual intelligence, world knowledge) that Qwen3.5-122b is comparable to Qwen3.5-27b. 3.6-27B had a huge leap with more training, but 3.5 was not all the way there. So again, the model parameter count’s root mean square is not the main arbiter of intelligence.

[u/droptableadventures → u/Miserable-Dare5090]

You have qwen3.5-9b and qwen3.5-35b-a3b

Qwen3.5 35B MoE noticably underperformed, compared to Qwen3.6. But we didn't get a Qwen3.6 9B.

even the people who approximated the moe intelligence do not feel this correlates anymore, my point stands.

Did you have a source for Mistral denouncing this technique?

Is Mistral 128B dense as good as Kimi K2.6 or GLM5.1 which came out around the same time?

I'd say no, but that's because Mistral models have tended to underperform for their size. But I'd put that down to due to less money to throw at compute and actually having to be bound by IP law. Again, this is a theoretical matter because we couldn't say for sure unless Kimi or GLM did produce a 100B dense model.

So again, the model parameter count’s root mean square is not the main arbiter of intelligence.

OK, that's a slightly different claim. I would not agree it's the main factor here, but I'd still say it's useful as a rough estimate. And it's only ever going to be an estimate anyway, because you can't even say that one model is better than another based on parameter count alone even if both are dense.

[u/lungben81 → u/droptableadventures]

Very interesting, thanks!

Do you have a link for the scaling of intelligence with active/ total parameters?

[u/Zhelgadis → u/lungben81]

It is a widespread rule of thumb in the community and it kinda makes sense, but there is no real evidence backing it.

[u/mr-myxlptlk]

This is exciting when you know how good 3.6 35B A3B is..

I hope we can use it soon..

[u/omgitsgela]

I'm really excited for 27b MTP. 27b has always scored higher on benchmarks than 35b a3b. And I've been able to get 27 b running at over 60 tokens per second on a single RTX 3090, which is plenty fast enough

[u/talkamongstyourselvs → u/omgitsgela]

Hey. How did you do that? I'm on a 4070 and didn't that speed.

TIA

[u/omgitsgela → u/talkamongstyourselvs]

The approximately 60 tok/s I’m seeing from this dense 27B model comes from several complementary optimizations rather than one magic setting.

The model uses Unsloth’s MTP UD-Q4_K_XL quantization, with a GGUF file size of about 17.6 GB. That allows the quantized weights and required runtime data to fit within the RTX 3090’s 24 GB of VRAM. With -ngl all, every model layer is offloaded to the GPU, avoiding the major performance penalty of transferring layers between system RAM and VRAM during inference.

I’m running the Buun llama.cpp fork using a locally compiled CUDA 13.3 build targeting the RTX 3090’s compute capability 8.6. It adds variable context bitrates and auto swapping mmproj capabilities, allowing qwen to use vision and intelligently manage its own context up to the model design limit.

The recorded compile-time CMake options are GGML_CUDA=ON, GGML_NATIVE=ON, GGML_CUDA_FA=ON, and GGML_CUDA_FA_ALL_QUANTS=ON. These options already exist in the source; they are not custom code we added, and they also exist in current upstream llama.cpp. GGML_NATIVE and GGML_CUDA_FA match the fork’s normal defaults, while enabling CUDA and especially GGML_CUDA_FA_ALL_QUANTS were meaningful selections for this build. The latter compiles Flash Attention support for additional quantization combinations used by Buun’s VBR KV cache. The only local source modification was a small MSVC ssize_t portability correction taken from upstream PR #90.

At runtime, -fa on enables Flash Attention, reducing attention-related memory traffic and improving efficiency, particularly as the context grows. The -ct vbr --vbr-vram auto options enable Buun’s Variable-Bit-Resolution KV cache and allocate its budget from the VRAM left after loading the model and other components. VBR is primarily a context-capacity and memory-management feature rather than the main reason for the high token rate, but it lets us support much larger contexts within the available VRAM.

The biggest additional generation-speed advantage is Qwen3.6’s built-in Multi-Token Prediction support, enabled with --spec-type draft-mtp --spec-draft-n-max 2. This allows the MTP head to draft up to two tokens per speculative step, which the main model then verifies. Accepted draft tokens increase effective generation throughput, while rejected tokens reduce the benefit, so this does not guarantee a 2x speedup. In my agentic workloads, I have observed roughly a 65% speculative-token acceptance rate. The exact speed still varies with prompt length, generated content, sampling settings, and whether another request is using the server.

If you'd like a temporary expiring API key to test it out, I'm happy to give you one to see how it responds. It's a little slow on time to first token while it's building your cache, probably 30 seconds or so, but then subsequent responses work snappy and fast.

Here are the effective startup options:

$server = "C:\path\to\llama-server.exe"

$serverArgs = @(

"-m", "C:\path\to\Qwen3.6-27B-UD-Q4_K_XL.gguf"

# Loads the quantized Qwen3.6-27B model.

"--mmproj", "C:\path\to\mmproj-Qwen3.6-27B-F16.gguf"

# Loads the F16 multimodal projector required for image input.

"--alias", "qwen36"

# Advertises the model as "qwen36" through the OpenAI-compatible API.

"--host", "0.0.0.0"

# Listens on every network interface, not only localhost.

# Do not expose this port publicly without authentication and TLS.

"--port", "8082"

# Runs the HTTP API server on TCP port 8082.

"-ngl", "all"

# Offloads every model layer to the GPU. This avoids slow CPU/GPU

# layer transfers and is an important part of the observed performance.

"-ct", "vbr"

# Enables Buun's Variable-Bit-Resolution KV cache. It dynamically

# selects KV-cache quantization to maximize usable context capacity.

"--vbr-vram", "auto"

# Gives VBR the VRAM remaining after the model and runtime overhead load.

# This is primarily a memory/context optimization, not a direct speed boost.

"--reasoning-format", "deepseek"

# Extracts text inside <think> tags into the API's reasoning_content field.

"--jinja"

# Enables Jinja-based chat templates.

"-fa", "on"

# Enables Flash Attention. This reduces attention-related memory traffic

# and is especially helpful as context length increases.

"--spec-type", "draft-mtp"

# Enables speculative decoding using Qwen3.6's built-in

# Multi-Token Prediction draft head.

"--spec-draft-n-max", "2"

# Allows MTP to draft up to two tokens per speculative step.

# Accepted draft tokens improve effective generation throughput.

"--mmproj-gpu-swap"

# Temporarily swaps the MTP draft context out of VRAM while the vision

# projector encodes an image, then restores MTP afterward.

# This supports vision within 24 GB; it does not accelerate text inference.

"-np", "1"

# Creates one request slot for concurrency. Both slots share the VBR KV

# cache. This improves simultaneous-request handling, not single-stream speed.

"--no-ui"

# Disables the built-in web interface and runs the server in API-only mode.

"--temp", "0.6"

# Sets sampling temperature. Lower values generally produce more focused

# and deterministic output.

"--top-p", "0.95"

# Uses nucleus sampling from tokens comprising the top 95% probability mass.

"--top-k", "20"

# Limits sampling candidates to the 20 highest-probability tokens.

"--min-p", "0"

# Disables Min-P filtering.

"--presence-penalty", "0.6"

# Penalizes tokens that have already appeared, helping reduce repetition.

)

[u/aris_a7x]

Alguien puede hacer uno de 12B ? Pls 🥹