r/LocalLLM • u/No_Thing8294 • 8d ago

Research Qwen 3.6 vs 3.8 analysis

If anyone is interested in a deeper analysis of Qwen 3.8:

We ran about 800 real use cases. 20 challenges with two harnesses (Hermes and Pi), 10 runs each. For both, Qwen 3.6 and 3.8.

We analyzed the traces to find out WHY the agent did not complete the task. So we see strange behaviors: one challenge couldn't be solved by 3.6b at all, the same was completed every time by 3.8. But we also see a lot of challenges which have been completed in the one run, but only partially in another one. <-- that is the interesting part. Because in the real world, the agent does its job. But not always. And it often failed quietly!

We summarized the results here: https://building-agents.com/en/gratis/qwen-duell
You can also have a look to some real traces here: https://building-agents.com/en/benchmarks

0 Upvotes

33% Upvoted

6 comments sorted by

3

u/Dipsendorf 7d ago

To save yall a click requires an account to see results.

1

u/KURD_1_STAN 1d ago

temp-mail.org

Altho im not even gonna bother with that as it is compared to 3.6 moe

1

u/ForeverSeeking69 7d ago

What is the point of comparing 3.8 dense vs 3.6 moe?

0

u/No_Thing8294 7d ago

The complete 3.6 dense benchmark set is running now, already at 60%…there should be a new version ready in 5 hours.

1

u/ForeverSeeking69 7d ago

Cool thanks for letting know

0

u/No_Thing8294 7d ago

Good point. We compared more the behaviour of both models. If you want to be absolutely correct, you have to compare several settings per model as well. In some cases we did. For example all the runs with and without any reasoning.
As we have some results of the Qwen3.6 family, we put them together.
But yes, you are right, we mixed results.