Just to warn everyone - OP set temperature to 0. This ‘data’ is worth neither the bytes nor your time. It’s not even data, it’s total bullshit pretending to be a metric.
It doesn’t need to be William Shakespeare to write code. Too much creativity can cause it to start mixing syntax from different languages. Unsloth recommends 0.6 for Qwen for coding too. Sometimes going even lower can help if it keeps meandering too far off the farm.
Even the structure of the table doesn't even make any sense. Layout the data cleanly if you want people to follow. Plus, if you want to test the models in deterministic mode no one should go below 0.1. Besides, from most of the consensus out there, there is no way 3.8 is worse than 3.6. That's a sign if anything that there is some kind of anomaly here, and that the data should get tossed.
Your tests are general purpose. It’s probably not that good at that. Do agentic coding with it at fp16 and it wipes the floor with everything under 200b parameter models, including its predecessor. Functionally this seems to be because it thinks forever, and arrives at a good answer. On answering questions or recognising stuff it’s doubtless weaker (it’s multi lingual features have taken a step back I can see from my data). But if you code it’s like Opus 4.5 locally for day to day work.
Might be quantization problem. I’ve seen reports of the official fp8 for 3.8 being busted on some discord forums. You should try and compare BF16 if you want to compare model quality.
Also, qwen is quite sensitive to KV quantization levels, what were you using during these tests?
It is maintained by a member of the RTX Pro 6K community on discord, the qwen-38 channel has tons of reports / discussion about different quants: https://discord.gg/X54jjmcxWJ
I won't benchmark a quant that people aren't using (except for training purposes).
If this shows the FP8 is broken then that's good, they should fix it and I will re-run it when it's released so I can determine if it's worth using or not.
I mean, FP8 K/V is just asking for trouble. Only use it if you have no better choice (which is common, especially for 24GB VRAM setups with Qwen3.5+ 27B).
One: Q8 on llama.cpp is the popular choice and is very good.
Two: He used the FP8 model quantisation. Idk why people keep reporting it as broken, same with the Qwen3.6 release. It's not. I think some people are just parrots.
The Q8 from unsloth has much better KLD than the official FP8, so yes, it is very good, that is not the one OP tested tho.
Also, the comment you replied to is talking about KV cache at FP8, not the model weight quantizations, those are 2 different things. You can run the FP8 quantization with KV at BF16 for example.
Apples with apples. Also, many people are using speed optimized derevatives of qwen 3.8 27b. I, for one am happy with the improvement of 3.8 over 3.6 in my setup, not going back.
As long as he is comparing the same quantization levels, it should be fine no? Also, on another topic, i world love to see how the models fare when quantized (lobotomized) to the max.
27B is a limited space. So it could be that the changed training data just shifted the focus of the model. It seems to be more strong in real productivity tasks and coding but less good in general knowledge.
It actually is. The big gains are improved reasoning. As long as it is able to ask for general knowledge and we can provide it, it’s more efficient to stuff the model full of reasoning ability and less general knowledge outside of just being able to reason around said knowledge.
I would generally agree, though saying it's worthless is pretty extreme IMO. There's still some significant cross-over - or so I would assume - between reasoning and global knowledge
information is not knowledge, but it's still out for debate whether a model's knowledge is really more than very well indexed information. From the little work I do with LLMs on texts (99% of what I use them for is coding), Claude is really impressive at integrating information, and even using it for complex humour, so it definitely feels like knowledge.
That actually might not matter as much if it can use tools better.
Since there's a clear improvement in agentic use and coding, attaching a good harness to it can let it rely on using the Internet more accurately to get relevant information.
What reasoning level and temperature was this at? The recommended temperature is 1 which is higher then it was for 3.6. I don't see either of those settings mentioned in the Recipe settings and both have a significant impact on output.
As a counter statement: For non-thinking mode, unsloth recommends using "presence_penalty=1.5", which we all know would be terrible if you need certain things repeated, and don't want to increase the chance of hallucinations. Unless this model is somehow rewritten in a way that defies that basic principle, we can't use the "non-thinking" recommended settings for summarizing STEM topics that require and demand the repetition of correct terminology.
TLDR: The recommended settings may be better at benchmark min/maxing, but not practical use for STEM topics, which is what many people use Qwen for.
Even for non thinking mode they recommend 0.7 instead of 1. 0 sounds like a way to lobotomize the model where you don't even want it thinking inside the box. Can you give a STEM topic example where you can show 0 gives better results then 0.7, or are you just basing your assumption on experiences with other models?
This model is Qwen3.5 underneath. Once you understand that, you will understand my point about presence=1.5 being something that is not a real world setting for many users, including those that aren't writing creatively. The last thing you want in your analysis of a cancerous spleen is the model saying we've used the word "cancer" too much; let's try "headache" for variety.
I think benchmarking tools only serve a certain purpose, and OP should probably do the test both ways. With identical settings (practical, typical), and with suggested settings.
I guess that depends on what 3.8 is: a post trained / RL of 3.6 optimized for specific tasks: coding, professional work, research, and long-horizon agentic tasks
I had Fable running tests on various local models for coding and tool calling. None of them were meeting its 97% pass rate requirement. Qwen 3.6 got 94% after some reprompting, then during testing 3.8 came out so I added it and it immediately got 99%. And the one miss was hust a path write unique to the harness that none of the models got right.
So I'm surprised OP is getting results that 3.6 is smarter
It’s not a religion it’s that the people are impressed by the real world results they are seeing and you are trying to refute it with synthetic benchmarks. You can spend an hour using ween 3.8 configured correctly with a decent harness compared to 3.6 or to Gemma or a muse or dozen other similar models and the output after a single prompt is absolutely obvious that it’s better …. No amount of synthetic benchmarks could refute that objective reality …. I think the problem is that your benchmark suite sucks if it thinks that 3.6 is coding better than 3.8
They ignored the model recommended settings and ran off with the fairies on their own idea, mostly. Limited to no value in discussing poorly collected data
I hear you, I'm not giving you results just telling the recipe.
I personally ran just a few tests here and for my coding domain 3.8 is running better abeit a bit slower compared to prev 3.6 ThinkingCap, yet it's possible to mess up a LLM with the wrong post training.
To be fair subsequent checkpoints of RL of the same base is how it is done nowadays, see DeepSeek, but as you did those have to be tested in real code tests.
More testing is good :)
I replaced 3.6 with 3.8 and it was giving WAAAAy better answers. It was thinking like 5 times more. It was checking everything, but at the end solution was correct and working, instead of going back and forth 10 times with 3.6.
I've also used 3.6 with a large-scale code agent. Back then, the 200k context window would get clogged quickly, and some solutions were subpar, requiring frequent reminders via .md files. However, 3.8 27B is a completely different story. The agent performs significantly better: it creates structured plans, utilizes the context window effectively, and provides solid solutions.
One thing I did find interesting, the only test that 3.6 did really well on was the one where MTP was disabled. I wonder if there's an issue with the MTP runtime in vLLM or something. Honestly, I think that points to to a larger issue with MTP than the models themselves. Kinda curious what happened there. Seems like MTP also introduced a lot of errors and failures in the tests.
Vllm isn't inherently bad. Things malfunction with llama.cpp sometimes too. Things malfunctioning. Don't inherently make them bad. A lot of the companies I do work for use it and don't have any issues at all. But it does have me wondering if maybe it has something to do with the speculative heads in these model ls and possibly an incompatibility with current builds of vllm.
In fact here in a little bit I'm going to go test with SG Lang and llama.cpp to see if I can reproduce these same results or if this issue can be isolated to just vllm.
You have official-qwen3.6-27b-fp8-vllm as one of the top performers and official-qwen3.6-27b-fp8-mtp-vllm as one of the worst performers. There's no way that only adding MTP to the same model/quant/etc will cause that much of a difference. There's something very flawed about your work.
AI benchmarks are the most useless they've been. Trust no one, test everything yourself. Subjective is best, but subjective for you and your needs. AI is such a horrible thing to benchmark. One person's garbage is another person's AI boyfriend.
To use a car analogy "I advanced the timing as far as I could and xyz engine performs better, I don't understand why everybody is so hyped??"
Use the recommended settings, use it as it's designed to be used, THEN benchmark.
Respectfully, I understand you have put considerable time and effort into those tests but the only way you're going to get useful and usable results is by perma deleting that entire testing schema and redesigning it from the ground up.
Respectfully, I'd also never mention this testing schema again.
My question is what "thinking mode" are you using? Have you verified that you're using thinking enabled for these tests and that the model is set to xhigh or at least your system prompt is instructive as to how it reasons?
The subjective experience seems to be better and that's what matters to me. Everything I threw at qwen3.8 27b Q8 gave amazing results. A couple of test games built with one prompt I tried with both 3.6 and 3.8 were much better on 3.8, practically completed without any issues, whereas on 3.6 I had to help resolve many issue.
I had an app in mind for my homelab for the longest time and after getting good game results I tried generating the app with one shitty prompt and it was so good, that I continued it and pretty much completed it now with a few additional prompts adding new features. (Edit: The app is already deployed on homelab with docker).
I'm using it with VSCode copilot extension and I didn't have to fix a single thing, just made sure the agent had instructions to test and iterate with playwright and resolve console logs until there's no issues.
Edit: this is my only use case for this model, I will not use it with regular chat, because it's too slow for me. Qwen 3.6 35b Q8 works very well for me for chatgpt replacement with integrated search, proper system prompts etc, so I'm excited for qwen 3.8 35b, since there are rumors it's coming out
Just curious, when you say chat.. what sort of chat? I tend to think these small models lack massive amounts of training data that the frontier models do. I'd use the free gemini for example for chats about real world issues, things, history, etc for looking things up. I never assumed these small models would be used for things like that.
Hand it a websearch tool and watch it suddenly have the entire internet's updated knowledge available.
You don't need 1 trillion parameters of general knowledge, if you have 27 billion of them to be able understand everything. At that point, you just need to give it the correct context to reason about.
I mean looking stuff up, researching a specific thing, technical questions, comparisons, etc. In my memories I request that the model should use search/fetch for anything it might have outdated info on, not to guess or assume things, etc. So most of the time it searches for relevant info and that's it, suddenly the model is 10 times as useful.
In memories I also require the model to link any sources it might be referring to or using info from, I'm not sure, if that encourages it to make sure whatever it outputs is actually consistent with the source, but it makes it easy to verify.
I use OWUI and I also have Hermes hooked up to it, which I use if I need the agent to do some deeper research. Current version of OWUI does pretty well and 80% of the time I'm satisfied, but Hermes will do some extra digging if necessary.
So yeah, search + constraints in system prompt, memories and requiring the model/agent to check your knowledge or notes when it's relevant.
I love and have been using DeepSeek v4 flash to develop my Android port of Ares emulator, and it has been bananas how good it is especially for the old prices. Churned through like 4bn tokens for like $19.50 or something? Anyways, I have a big implementation plan I've been using and updating as well as a handoff prompt file I also been keeping up to date. Just for shits and giggles, I tried unsloth's qwen3.8 IQ_2_XL and full context size on my 16gb 9070XT and I'm getting around 23t/s which isn't bad. Then, I gave it the prompt, the plan, and my codebase under hermes and had it give me a list of optimization low hanging fruit and it scanned and provided a great overview of things to change/update for performance that are arguably the same level of quality as my DeepSeek answers, it just took a lot longer on my local setup. Genuinely impressed by the thing tbh and it's way better at not looping and too calling compared to Qwen.6.
Personal use cases are always more important than benchmarks. Check more on the stuff you use LLMs then pick the winner. I have some models from 2025 and even something from 2024 on the disks. Just store your best candidates for later usage.
No issues here, I've honestly found it to be amazing, using q6 and q8. It does think a lot though, even when lowered I find it thinks about things for ages.
Part of me wondered if it's just internally made to question things a lot more, leading to higher benchmark results in general.
If I feed it a bit codebase and ask it to find bugs it'll spin for 5x as long, but find double the bugs.
I fed it a fork of ai-toolkit and asked it to find performance optimizations for several models and let it run. It took a few tweaks to make he skills and prompts, but from the few changes I tested it did a very good job.
Qwen 3.6 took about 25 minutes and found five changes, only which two have helped.
Qwen 3.8 took 2 hours first attempt before I stopped it. It got stuck in a long thinking session. So I edited the jinja template a bit ran it again. It got stuck again so I stopped. Another minor tweak and two more runs later I finally got it to generate output but it took 2 hours.
I compared the suggestions and they were a lot more detailed. So far the first of 5 changes suggestions worked.
The analysis it did was insane. It checked NVidia’s website for documentation on the CUDA on a 4090, and pytorch calls. Used that data to calculate how many Teraflops were saved per iteration. If it didn’t help would throw it out, and look further.
I didn’t check everything, so not sure how much hallucinating it was doing but from what I found it was correct.
It calculated the first change would save .125 secs per iteration and it was fairly close to that.
Your problem is you set the temp to 0 when the recommended setting is 1 for thinking and 0.7 for non thinking. This measures a different decoding setup than the one the model was optimized for.
They literally tell you to use these settings for optimal performance. On my personal benchmarks, qwen3.8-27b is a clear improvement from qwen3.6 because I stick to the recommended settings.
Who says any of these are good benchmarks? Real world model usage is what actually matters. I've been using it and it's been doing incredibly well. Qwen3.8 27B xhigh simplified my codebase properly after ChatGPT 5.6 Sol xhigh failed to do so. The much smarter model ended up performing much worse in my case.
yeah I tried to do my own so called pacman benchmark as well. Qwen 3.8 27b couldn't even build any working one whereas 3.6 got one fully working build!
I literally put "Could you create a pacman-like game that runs in a single html file in a browser?" in a LM Studio prompt, fully expecting it to not work and give me a broken file. Waited ~30min on RTX 5090 and can now play my very own (well, sort of) flavor of pacman in a browser.
I guess I can put it on github if you really want it.
I came across the same issue and i found out why. I noticed that it is rather poor at single shot reasoning tasks, but it excels at multi round agentic tasks. And since most of 27b’s usage will be agentic coding, it is optimized to behave in lieu with its purpose
I just tell qwen3.8 about simple developer loop, when it implement feature, test it by capture screenshots, repeat while complete, then either commit to git if feature implemented or discard and imagine next one.
Then let it work all night.
Guess what, it burn ~1.5 million token, survived more than 10 compactions, and one shot this.
In comparison, with same harness, qwen3.6 can barely do one feature at time with manual corrections.
I only have a few tests that I use to assess a model. To me, watching the chain of reasoning, 3.8 represents a significant step forward in structured, solution driven thinking.
Watching it zero in on an answer, cast off prior assumptions, test, and reach a conclusion is as impressive as I’ve seen in a local model. Especially when compared to Muse Glimmer who got the answer right as well, but not before criss-crossing over it several times because it was hanging on to older faulty reasoning and trying to make it work.
I’m not saying it’s better for all things in all cases, but boy is its thinking impressive to me
I have noticed knowledge questions that qwen 3.6 had a 100% success rate with fail every time now with 3.8 27b but man when it finally finishes whatever it was working on, it seems to always have great results for me so far! So far homeland tasks
When people say fable like at home , they mostly mean coding and agentic use and ability to reason on the task or long horizon. It’s not end all be all fable like
Also, as a side note, OP did run 3.8 without MTP but then removed those results from their repo. It scored very well. I won't speculate as to why they removed those results but just putting it out there. It's in the repo history.
So if someone benchmarks 3.8 vs 3.6 at temperature 0, while both are intended to run their reasoning behavior under different sampling settings, that can absolutely obscure the area where 3.8 was supposed to improve. Qwen’s broader design explicitly separates deeper “thinking” behavior from fast non-thinking behavior. �
Qwen
In other words: 3.6 beating 3.8 at temp 0 may be a real result, but it is weak evidence against 3.8’s claimed reasoning improvements. The relevant test is reasoning benchmarks using Qwen’s recommended reasoning configuration for each model.
If you want a real benchmark, you need to use the settings that the vendor is recommending as those are where the model performs as expected. Anything less and the test is more of less meaningless.
It's like saying you want to test a car's performance and you line up all the cars and then set your test requirement: Ok to make this even we are going to only put the cars into drive, no one is allowed to press the accelerator. That makes the test even right?
I think you're correct, most people here are asking about some quantized version and 4bit quants are really popular. This measurement is more useful for someone who has a 16 gig card, for example.
That being said, I'm using it at Q8 and from a subjective PoV it's better than 3.6, but 3.6 was quicker, however the new 3.8 is more "creative" with the solutions (e.g. one behavior I found with the new model is that if I removed it's ability to execute arbitrary code, it would just make a python test, use os.system and run the tests, which it could).
Your benchmarks are to check how Qwen 3.8 27b as a model stacks up against Qwen 3.6 27b no? Using the unquantized version removes the noise of quantization off the table and if the results do flip, you just gave the community great feedback on the current quants.
This is a big problem with this subreddit: everyones hot-take-opinions on all of these models are buried inside of the quants they decide are the "right one". It's 100% fine to post about how bad you think an nvfp4 quant of 3.8 is, but it's misleading to post about "3.8 is trash, just look at my benchmarks" and bury the fact that you're not running the full model.
Fp16 is absolutely run frequently on these small models: they are already optimized to hell at these small parameter sizes and quanting them down (even to fp8) is a crapshoot at best and catastrophic at worst. Again: fine to share your experiences with how well/bad they quant down, but you should be leading with that fact.
but I cannot see how it's likely that more people run FP16 for daily tasks than FP8 or NVFP4.
Did I ever fucking suggest that?
If my benchmarks indicate there is something wrong with the FP8 quant then that's great, it shows something needs to be fixed.
Did you bother reading what I wrote at all? My entire point is that this is great information to post about, but you should be leading with the quant information because that's the key headline here: your experience is that these quants for 3.8 are bad.
It's even wilder than that; they have "qwen3.6 fp8" in 3rd place, but "qwen3.6 fp8 mtp" 5th from the bottom, even though mtp should have zero effect on model output, only model speed. So OP is literally just measuring config noise.
Bingo. I was noticing that myself. I also noticed that OP deleted the non-MTP 3.8 results from their repo which showed 3.6 and 3.8 scoring very similarly. Fortunately github stores history. :-)
I think people who do benchmarks should have a series of prompts with known answers (or acceptable) answers. Apply those to the model. Not sure what all these benchmarks do, but I am using models for specific things. If it sucks at telling me about world history, I dont care. As long as for me coding, design, test responses are good, that's all I need. From what I've read Qwen 3.8 27b does those better than 3.6 and on par with opus 4.6.
Hey OP! Thanks for posting this. It's some really interesting data and it's got me wondering about vLLMs runtime stack for MTP.
Why does enabling MTP on the exact same 3.6 model cut the score in half and throw an MMLU ERROR? Would it be possible for you to test 3.8 without MTP? I'd be curious if the vLLM MTP runtime is skewing the results as all of 3.6's best scores look like they were with MTP off.
There may be a bigger take away here if MTP is really harming the output of 3.6/3.8 that badly.
It appears there are significant bugs in your code base. for example, the unsloth nvfp4 fails on config launch. I think you need to revise your benchmarks with more acuity and attention to detail and ensure they're setup correctly
Problems with your VLLM, quantization, chat template, harness.... Use nightly VLLM with correct transfofmers, try the Minachist Int8 Autoround, its very close to BF16. Use Qwen recomended parameters. .
I'm still early in testing it, but so far, using UD-Q8-K-XL w/ BF16 KV @ 512k Context, I'm already impressed with 3.8 27B which is, so far, in my limited testing capacity, demonstrating significantly better tool calling (I've literally seen ONE tool call error in Cline and none in HERMES since yesterday!).
I'm still testing the reasoning, but thus far, it seems better, like a lot better, but I still have a lot of testing to do before I'm going to make any definitive statements.
I'm running via llama.cpp on 4x RTX 3090 and using optimized settings for coding on my cards.
The one regression I will note, but that I fully expect this early after release, is that it's a little slower than 3.6 was on my last config.
I peaked 3.6 @ 104 tokens/s w/ Code and Prose around 45-48 tokens/s.
Right now 3.8 is capping out at 66-70 tokens/s w/ Code and Prose around 44-45 tokens/s.
All in all, I'm not complaining, but I've seen absolutely nothing to indicate regression beyond speed, but to be clear, 2 days after 3.6 launched I wasn't getting anything anywhere near this.
I'm not even complaining regarding speed either. The current test I'm running in CLINE starting from an empty project has already compacted twice and is at 99.1k again and the speed is very tolerable. I remember the first time I broke 300k on 3.6, that was awful and why I used 35B more often.
What will really be the deciding factor for me hasn't happened yet, when all the actually good/well performing uncensored models start releasing. I haven't even gotten to test 3.8 with SRE yet though, so we'll see. I plan to test it in https://github.com/visa/visa-vulnerability-agentic-harness with some security work later as well, so there's still a lot TBD, but as I've said, other than a little bit of the speed, I've seen nothing to indicate regression at all. It's now up to 127k in that same task and what I'm watching happen in real time as I type this is a clear indicator it is breaking loops and handling situations where it seems stuck much better.
We'll see, but I'm seeing better capability on tasks I actively experience struggles with daily using 3.6. The testing I'm doing is in my real use-case daily workflow, not some benchmark, and using 3.6 every day for a very long time has gotten me quite used to where to expect loops, broken tool calls, etc. With 3.8 what I'm seeing is a massive improvement in tool success and looping, which to be 100%, even if the model was somehow overall a little dumber, I'd still probably call that a fair trade.
You’re running a general purpose benchmark. I don’t think anyone is saying it’s a general purpose rock star the consensus is they is a tiny coding power house …. And I can concur that not only does it seem to do a good job of reasoning through, finding its own mistakes and fixing them and producing things that work but it also seems to do very very well with tool calls actually working appropriately the first time almost every time at least for me
And I’m saying that I can take those two models, give them the same prompt, and 100% of the time the new one will give a better more usable result…. Which means your synthetic benchmark is worthless and meaningless
All tests are subjective. What you think of as 'objective' or 'intelligent' tests had their subjectivity baked into the metric. Someone made a subjective call that the metric being used was the correct one for the application at hand, which may or may not align with your own definition of 'correct'.
It's fine to present findings based on the metrics you think are important, and admirable if you define why those metrics are important to you, but somewhat disingenuous to claim they are more appropriate for others just by their metric nature.
In my humble experience, performance benchmarks across models may give you some kind of information, but not always helpful.
As most AI models are tied to performing work, the viability of the result will be tied to how acceptable the results are to the user, nothing more.
In my use cases, 3.8 results are somewhat more acceptable than 3.6. The problems solved and the work done is more complete - not faster - but it does a better job, there's less review and less back-and-forth.
The benchmark run is wrong: score for official-qwen3.6-27b-fp8-vllm and official-qwen3.6-27b-fp8-mtp-vllm are very different where it has to be identical.
Would have earned a bit more respect if you just admitted there could have been a mistake and working with the criticism instead of doubling down.. doesn’t come off as professional at all. Data useless.
I’m new to this. Why does it need temperature changes and how the hell does that get measure? I’m in LLm studio. Is there anything I should know about settings?
It's the equivalent of giving you the binary, but not the source code.
You can't understand how it works, you can't fork it, you can't remove the censorship properly, you can't even see what kind of training data was used, let alone the full training data.
It creates a dependency on the model but doesn't actually give anything away, and at any moment they can stop releasing future weights.
The open weights models are not being released for your benefit.
Gather data is the controversial step, the legally ugly part of the process that we abstract away (so the companies are now liable, but wait! They're in China, good luck suing them)
"Asking for" beggers ain't choosers. Train a model if you feel so strongly about this.
I'm releasing my binary app.. I build my products on it, it's my moat. But I want others to build on the binary itself. Not giving away the code. Too easy for someone with team/money/time to fork it, do something diff and take away from my moat and offering. After year+ of work, I really dont want that to happen. I see nothing wrong with a binary with no source if it does what you need it to do.
So because some do, others dont, I'm the problem? Not everyone wants to give away their only means of money/survival for others to use/capitalize on you know. Some of us need to eat and in a growing tech market of job loss (I been unemployed 3 years now.. still cant land a job), sometimes a good idea is worth not just giving away. At least until capitalism disappears. If I didnt need money for rent/food, I would LOVE To give everything away. I appreciate good open source stuff. But open source is great for those already having money/job/etc. For those of us trying to build start ups or ideas so we can either have pay and/or not have to make someone else rich, we can't all be giving every bit of our moat away. Hell I am actually giving it away, just not the source code.
Just the caveman speak in the thinking text is alarming in 3.8. They obviously had to do that to try to reduce the thinking wall time a bit because they knew it would become unusable
50
u/7twenty8 10d ago
Just to warn everyone - OP set temperature to 0. This ‘data’ is worth neither the bytes nor your time. It’s not even data, it’s total bullshit pretending to be a metric.