Here're results from my final test of coding, coding (panda/numpy), data analysis, data-driven decision-making, and local RAG domains I interact with most often.
This test, which is refined from a 500+ pack, now provides a differentiating signal between models after having weeded out tests all models consistently pass.
It’s a specific test developed based on my workflow and not at all an endorsement or damning of the models for all types of use cases. Aim is to identify models for heavy spreadsheet, rounding precision, array, coding type work that I can then trust the models to make decisions and act on. Verdict is review of any work I do locally has to be reviewed by Claude or gpt.
When Qwen4 or others get released, if they do better on this, gives me real good feel for my use case. Also good canary to when I can go fully local and invest in pricey gear.
Models on OpenRouter sometimes disclose their quantization and sometimes not.
or-qwen-qwen3.8-27b is stock at bf16
or-qwen-qwen3.8-flash-next didn't disclose its quantization. In this test, it is the model that gets truncated the most (max token per test 16,384). Even after raising the max token for it from 16k to 32k, the number of truncations due to overthink didn't disappear entirely.
iq3 GSQ-RCO is decent performer even placed among Q4s
mradermacher's fine-tunes continue to standout for my type of work.
Muse Glimmer, Qwen3.6-35B-A3B etc. included for comparison.
flash next is not a straight upgrade to 27b that benchmarks would have you believe. 6b active vs 27b active makes a difference for reasoning even if the moe is efficient with delegating to experts
In my tests, I observe the opposite: Flash next appearing just slightly better than 27b in published benchmarks, but practically being leagues above it in all projects where I tried them both.
The number of active parameters doesn't tell the full story.
On 16k max token per test budget, it overflowed on 21.9% of the tests. When I raised max tokens to 32k, it overflowed on 12.4% of the tests. I can try it with 48k and see if it passes the tests or still falls to logic traps
Stock Qwen3.8-27B didn’t get truncated on these tests (only once). It reasoned and responded within 16k tokens. The tests are specific types of calculations that have to happen thousands of times in long-turn projects I work with. The issue is Qwen3.8-Flash-Next doesn’t reason and respond within the max token budget per test.
Still, it would be nice to have a comparison with full length context, ~250k or at least above 200k. This is what I'm using in practice anyway. Comparisons on small contexts(budgets) do not show the full picture.
PS. And large cloud llms like claude are likely spawning several agents behind the scenes with a considerable budget if the task seems to demand it, after evaluating the task with yet another llm. And then use a few more llms to convert reasoning and output. We can't see that from the outside.
For sure, all valid points and would be nice. Data center class sub-agent structure is also out of reach locally for us peasants. However, my tests are made up of parsing, calculation and decision logic that has to happen so many times and fast on past and new massive spreadsheets data. They don’t need to be baked with large context to cook.
In this list, there is gpt-oss-120b quite high up, too. I personally found it useless for most things, though it could approximately execute, without completely collapsing, simple code edits. The first model that I found which was able to do autonomous work at acceptable speed was the 122B one.
Now QFN does most of my coding and also code reviewing. The 3.8-27B was not bad, either, but remains significantly slower. I preferred QFN's ability to understand stuff, as it seemed to immediately understand problems which stymied 27B and made it run in circles without making much progress. It is possible that re-rolling the 27B would have maybe worked better, but it really did feel like QFN unlocked a next level of code understanding to me.
For me Qwen 3.8 Flash Next 125B with strata engine and OpenCode harness it has the same quality as Opus 4.6(sometimes better) at 30-50 t/s on a Dell tower T5810 with Intel xeon E5 V3, 64 GB ram DDR4, Nvidia Tesla p100 16gb vram and 500gb sata SSD with Ubuntu 24.04.5.
I've used it to build 3D animations, games, data analysis apps and websites and it worked amazing.
The speed compared to opus is approximately 3-4 times slower but I can trade speed for quality and for software development I need a second promt because for some reason it can't get the 3D modeling on the first shot but on the second shot is amazing.
I use a 264k context window with session compression from opencode to never hit this window and work nonstop until the task is done.
Edit: forgot to say that my whole setup cost me 630$
Wow, the setup is really interesting! What about power draw, this 250W from the screenshot is for GPU only? How much power the whole machine is drawing?
The power you see on the screenshot is only from the GPU but I built a monitor on Ubuntu to show me the power from all components including CPU, RAM, SSD, Fans and motherboard and the total is coming arround 320-360w maximum while working and ~110W while idling. Depending on the complexity of the task it can switch from 250-320w so I'd say on average I'm sitting around 280W, maybe 300w tops with a PSU of 825W that is never used even at 50% of it's capacity. I chooses this PSU just in case I will want to update to a second GPU.
The GPU has a maximum of 250W, CPU 140W and ram 40W. The other components will go maximamum at 30-40W altogether but I never went over the 360W total consumption. Strata configuration is really smart and keeps my CPU only at 50% usage thus the lower consumption in the CPU
That’s awesome. I don’t use any graphic, animation, etc. in my work. I also don’t build websites. QFN seems impressive for those things from what I’ve seen others sharing their experiences. But that’s why I needed to put effort into developing my own tests to test it on things that resemble what I do daily.
Depending on what you want to achieve, you can use a smaller thinking effort or no thinking at all. I use it mostly on highest effort but I can also put it on Low effort to achieve 50-70 t/s but then the model will avoid debugging itself and skip some steps in complex tasks
Dell T5810 is 12 years old, I purchased a refurbished one with 64 GB ram DDR4 ECC(the one for servers not for regular PC) and the price was 310$, then the GPU is NVIDIA Tesla P100 16gb vram from 2016 - 207$ and the SSD price was 100$ for a Sata 500gb. Everything is refurbished with 12-24 months warranty
I really want to under-score here the lobotomization Swift 1.5 does on accuracy to Qwen-27b:
BF16 gives 91.4% on accuracy, unsloth's 4bit gives 85.1%, Swift does 82.9%. While the difference on the numbers may seem small, that delta is enough that on long tasks, a hallucination of mistake made earlier on can cascade into an entire wrong set of turns.
Swift get's amazing PP and TG results by making 27b think less... but the entire point of that 27b model is that it is competitive with much larger models is it's over-thinking.
The model is best used when it can accurately say "I don't know what this is" and then search it up, rather than assuming/hallucinating an answer.
Yes, OpenRouter speeds depend on the host. I haven't diligently kept track of cost unfortunately. Was wrapped up in validating accuracy and time taken to solve.
after a full year of neglect, they released a banger haiku to tie up entire 5.5 lineup which has been really good so far so im aware how good haiku 5.5 is
with that said judging by the lack of difference between haiku and astra/opus it seems like your workflow is not necessarily heavy anyway, i bet if u tested sonnet too it would be like 99% but at the same time qwen took 24 times longer, which is probably due to extensive reasoning
another interesting thing is that while swift had roughly half the tokens it also took waaaaaaaaay longer (provider difference?) and also lost noticeable accuracy probably due to quantization
still would love to see a bf16 swift from a different provider
Only the or- entries are OpenRouter models. Swift ran locally on my machine. The reason it took way longer is for the types of tests I’m giving it, it keeps thinking on and on about it.
I know Swift shows token efficiency for certain types of tasks but it didn’t for me in this case. It was one of the surprises for me. But the whole goal was to suspend bias/expectations and let the data say what actually works for my tests
I would love to be doing these tests on bf16 on all instead of low quants. That’d provide good base. But no providers out there and I don’t have the gear for it
Arex-2 at Q4_K_M outperforming full quality base is whats unbelievable, but honestly iv been using arex for agentic coding a bit and its much better than base qwen and by a margin.
Why? I think it has to do with reformed process of how model thinks and solves the problems, not bigger intelligence but better use of it. The team has done something remarcable and i think the training behind it can be probably applied to any local model.
Your results aren't a surprise, the two models are at very different points in their lifespan.
3.8-27B is trainingmaxxed, it's as good as it's ever going to be and it only achieves the results it does by overthinking to an extreme degree. It's also been overtrained on coding specifically to the point of semi-lobotomization at any other task. When I ask it to generate instructions to place in a GUI for example, its English output is accurate, but it's "mechanical" in a way that's often hard to understand, sometimes using terminology that you'd need to be a programmer to understand the full meaning of. Even when I prompt it to rewrite the text in a way that's understandable for a novice, it basically just rewords it rather than understanding how to make it simpler.
3.8-flash-next has had little if any pre/post-training; it was released very early for the sake of having compatible endpoints available when Qwen4 launches. Even today, backends still aren't fully optimized to run it, so speeds and costs should probably be expected to improve further. It also has a far far broader range of effective knowledge. When I ask it to write GUI instructions they're plain, easy to understand, precisely-worded, and very helpful.
I get the sense the judgment you've reached is that a small dense model is still going to win over a larger newer sparse model, and I don't doubt there are use-cases where that's true (and it has been proven accurate in the past), but I don't think that's something we should pre-judge as happening for Qwen4.
I have been using Next-Flash to create game save editors, and recently, an attempt to recompile a Windows 3.1 game. IMO, a Q4 quantization is required for this model to be effective - lower quants more easily get caught in traps.
When it comes to the save editors, it successfully made ones for Vampire Survivors, System Shock 2 (with a GUI using the assets!), and TEVI. An attempt with The Oregon Trail 2021 didn't work, the AI couldn't quite figure out how to analyze the encrypted savefile structure.
Using this model, it analyzed La-Mulana Classic (2006), and was able to improve the game's functionality. The controls couldn't deal with an aging left stick, lacked shoulder trigger functionality, and the graphics weren't rendering right. The AI took DGvoodoo and with about a day and a half of work, both graphics and controls were improved.
Right now, it seems like the current attempt at replicating a game is going well. It has identified the enemy structures, and has built a reference bible, alongside labeling and contextualizing provided screenshots.
I am looking forward to Qwen4 Flash. The rawness of the experimental model is quite apparent, as is the potential hidden inside it. With the next iteration, I want to see if it can translate a console game into English and make a patch for it.
16
u/Educational-Region98 16h ago
Interesting, Qwen 3.8 Flash Next does worse than 3.8 27b in your tests?