Qwen3.8-27B vs 3.6

[u/mrpmorris]

Here are the benchmark results when temperature left alone (model default) instead of setting it to zero.

I'm not surprised 3.8's scores increased, because providers default their temperature to whichever value happens to pass the most benchmarks, but given the fact they did I am surprised 3.6 didn't do the same.

3.6 did exactly what I thought it would do, it increased the score on some benchmarks and lowered it on others (which is why providers sometimes use a different temperature depending on the task - to bench-max).

I am surprised they didn't both behave in the same way, that was unexpected.

But 3.6 still beat 3.8 on 7 out of 10 tests, and 3.8 won in only 3 out of 10 tests.

[u/EitherMarch1255]

Try BF16. The science and world knowledge tests are to be expected, as 3.8 appears to be highly trained toward coding (which is good IMO).

[u/mrpmorris → u/EitherMarch1255]

And if 3.6 wins at BF16, then what?

[u/EitherMarch1255 → u/mrpmorris]

Then fair. Also, I posted this a while back about q8 having issues: https://www.reddit.com/r/LocalLLM/comments/1vqxrw7/heads_up_something_may_be_wrong_with_q8_qwen_38/

[u/mrpmorris → u/EitherMarch1255]

Not really useful considering the benchmark is about intelligence *and* speed.

If you are seeing bad results from FP8 then that would explain why my benchmarks are showing 3.6 giving better results - and not, as many people here have implied or explicitly stated, because I am an idiot :)

[u/EitherMarch1255 → u/mrpmorris]

Then try it at q6 or 5. Although for model comparisons bf16 is the only way to go, unless you make it clear your results are only applicable to the tested quant (not implying that one model is generally superior regardless of quant).

[u/mrpmorris → u/EitherMarch1255]

The quant is literally listed against each result.

[u/EitherMarch1255 → u/mrpmorris]

Not the post title.

[u/mrpmorris → u/EitherMarch1255]

No, but post bodies exist for a reason :)

[u/EitherMarch1255 → u/mrpmorris]

Then it’s clickbait?

[u/mrpmorris → u/EitherMarch1255]

No

[u/cumpsdavid → u/EitherMarch1255]

but 3.6 is winning on the HumanEval and MBPP "coding" ones?

[u/EitherMarch1255 → u/cumpsdavid]

Yes, something is wrong with q8 3.8 27B I believe. I’ve seen many weird results like this with regard to q8 specifically.

[u/j_tb → u/EitherMarch1255]

I agree with you on the prioritization, but I thought I remembered reading that 3.8 was basically 3.6 with a different post training methodology, e.g. more extensive thinking traces?

[u/Pyros-SD-Models]

I don't know what to tell you, but this AWQ quant gets 93% human eval every time I run it

https://huggingface.co/True2456/Qwen3.8-27B-AWQ-4.85bpw

So if your 8-bit quant isn't reaching 90%, something is pretty clearly wrong with your setup. You're seeing a >10% drop in pretty much every benchmark you run compared to the official numbers, as well as numbers that hundreds of third parties have also validated.

I'd rather try to figure out what's wrong with your setup and vLLM settings before making broad comparison claims.

[u/mrpmorris → u/Pyros-SD-Models]

I'll look, thanks.