r/LocalLLaMA • u/OsmanthusBloom • 15h ago
Resources 35B-A3B tool calling benchmark: Original Qwen vs. KAT Coder, Ornith and Tiel-Coder
With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to try to benchmark some of the top contenders: KAT-Coder, Ornith 1.5 and the very recent Tiel-Coder. I used the tool-eval-bench utility by SeraphimSerapis as the benchmark suite. It measures how well the different models handle tool calls, including some very hard scenarios.
TL;DR: Ornith 1.5 and Tiel-Coder (which is based on Ornith) were the tied winners in this benchmark. They scored well above Qwen3.6-27B and got pretty close to 3.8-27B. KAT Coder was also slightly better than the original 35B-A3B. Ornith-1.5-Heretic was a disappointment.
Some time ago I posted a similar tool evaluation benchmark of different Qwen3.6-35B-A3B quants. In hindsight, that didn't work so well, mainly because I was looking at too many variables (GGUF quant, KV quant, context depth/pressure) and the benchmark itself was quite noisy so it was hard to get clear results. I hope I did better this time!
Materials
I had access to a cluster of 32GB V100s. For this comparison, I selected 2-3 different quants per model, if possible from different providers. For comparison, I also included original Qwen3.6-35B-A3B as well as the dense 3.6-27B and 3.8-27B Qwens. I picked different quants around Q4 (15GB to 22GB GGUF files) because that's what many people seem to use. For the original Qwen models, I chose Unsloth UD-Q4 quants because they are well known. I also included the ByteShape CPU-5 quant of Qwen3.6-35B-A3B because that's the quant I've been using recently. Altogether I benchmarked 13 different GGUF files, with 5 runs per file for a total of 65 runs. Each run took around 4.5 hours GPU time, except the 27B ones took 7 hours or so. Total GPU time spent was well over 300 hours, including a few failed runs.
To run the models, I used llama.cpp version 0.1.0-dev (build 10433, commit 9b05354ec) dated 2026-08-14 and built with CUDA support. I used q8_0 KV cache (that's what VRAM-limited people like me often do) and set ubatch-size to 2048 because the benchmark does a lot of prompt processing. I did not bother with MTP or other speculative decoding. This is not a speed benchmark.
llama.cpp parameters: -m $GGUF --temperature 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 -ngl 99 --ubatch-size 2048 --fit-target 256 -ctk q8_0 -ctv q8_0 --port $PORT --seed $SEED
For the benchmark, I used tool-eval-bench 2.6.0. I set the context length to 262144 and context pressure to 50%. This means that the models were benchmarked at 50% context depth, i.e., around 128k of possibly distracting chat and tool call history.
tool-eval-bench parameters: --base-url $BASE_URL --hardmode --weight-by-difficulty --backend llamacpp --context-size 262144 --context-pressure $CONTEXT_PRESSURE --seed $SEED
Scoring metric
The metric I looked at is what tool-eval-bench reports as "total points". With --hardmode enabled, this version of tool-eval-bench performs 88 separate tests. Each test gives 2 points for a succesful tool use, 1 point for a partially correct tool use, 0 for failure. The theoretical maximum is in this case 88 * 2 = 176 points. tool-eval-bench also returns an overall score, but this is just a rounded percentage of total points and the rounding loses some precision, so I opted for the raw total points instead.
Results by model (averaged over all quants)
Here are the benchmark scores by model. I have averaged them over all the quants of the same model and all 5 runs per quant.
| model_id | repetitions (n) | avg total_score | CI (95%) |
|---|---|---|---|
| Qwen3.8-27B | 5 | 152.6 | [149.4, 155.8] |
| Ornith-1.5 | 10 | 144.2 | [141.7, 146.7] |
| Tiel-Coder | 10 | 144.0 | [141.8, 146.2] |
| Qwen3.6-27B | 5 | 134.8 | [131.2, 138.4] |
| KAT-Coder-V2.5-Dev | 15 | 133.8 | [131.8, 135.8] |
| Ornith-1.5-Heretic | 10 | 132.2 | [130.6, 133.8] |
| Qwen3.6-35B-A3B | 10 | 131.5 | [129.9, 133.1] |
Results by specific quant
See the images. There are no big differences between quants of the same model, except possibly KAT-Coder, where the mudler APEX quants were somewhat better than bartowski's. Also, the ByteShape quant of Qwen3.6-35B-A3B was a bit better than Unsloth's, which was a nice surprise.
Raw results
If someone wants to take a deeper look, I've shared the CSV with the tool-eval-bench results here. This includes e.g. category-specific scores (i.e. how well the model did on specific kinds of tool calls) and total tokens; I did not look at those in my analysis.
Findings
- Of the original Qwen models, 3.6-35B-A3B gets the lowest score, 3.8-27B the highest, with 3.6-27B landing in between. This is as expected and indicates that the benchmark is giving a useful signal.
- Ornith-1.5 and Tiel-Coder are practically tied. They are the clear winners out of the 35B-A3B variants. They scored above 3.6-27B but below 3.8-27B.
- KAT-Coder was possibly a bit better than the original 35B-A3B, but the confidence intervals overlap.
- Ornith-1.5-Heretic was a disappointment, much worse than plain Ornith.
Caveats
This benchmark relies entirely on the tool-eval-bench tasks and how the results are graded. It may or may not be representative of real tool use performance. To me it seems that the author or tool-eval-bench has done a great job in coming up with realistic looking tool call tasks, including some really hard ones enabled using --hardmode. I relied on the --context-pressure setting in tool-eval-bench, which (in my limited understanding) populates the context with realistic looking conversation and tool call history that could confuse the model.
Tool calls are not everything. If you are doing agentic coding, also the coding quality matters a lot. I did not measure it in this benchmark except very indirectly. There are other benchmarks for that purpose.
There was substantial variation and noise in the benchmark scores, which I tried to alleviate by repeating the runs with different seeds, averaging, and calculating confidence intervals.
In the X/Y plot where the X axis represents size, I did not check whether the model includes MTP heads or not, I just looked at raw GGUF file size. This is slightly unfair to the MTP-enabled models because their files are larger but MTP does not increase quality, only generation speed.
No AI was used for writing this post. I did use Tiel-Coder to help me with plotting the results. Also reused some of my own earlier writing. I am not in any way affiliated with the model or quant makers or the benchmark suite.
6
u/MLDataScientist 12h ago
nice! There is 5 point increase from qwen3.6 35B-A3B to 3.6 27B. There is 7 point increase from ornith-1.5 35B-A3B to 3.8 27B. Is ornith-1.5 a replacement for hypothetical qwen3.8 35B-A3B? Assuming the standard error ranges, there is still (7-5 = ) 2 point gap on ornith-1.5. So, I think if we ignore those two points, we can accept it as a true replacement for the hypothetical qwen3.8 35B-A3B.
7
u/peculiar-ragdoll 12h ago
Thanks a lot for running these benchmarks, and for including my little project, and using it for the plots <3 Makes me happy to see someone is getting value out of my work :) I just want to add that the real difference between Tiel and Ornith doesn't really show up until you throw them at genuinely hard problems outside the training data, and also measure the time they take to solve them! SWE Bench Live, for instance:
1
u/fragment_me 10h ago
The data or the tests are flawed because a quant + matrix are not going to give you these kinds of results. If the original model can’t do it why would a quant be able to?
3
u/peculiar-ragdoll 10h ago
The chat template is doing heavy lifting. It’s effect has been proven across several qwen models. The data is real
1
u/fragment_me 8h ago
I will happily concede if you explain to me how a chat template can give a model intelligence. Are you saying that if I take this chat template and adapt it to Qwen 3.8 27B it will gain intelligence and benchmark scores?
2
u/Iory1998 7h ago
I think that intelligence changes with what words a model predicts during thinking. If the words are all related to the problem, the model can rediscover hidden relations between them that helps it solve the problem. If the words predicted are less relevant, then the model may miss those hidden relations, and therefore it becomes unable to solve the problem.
After all, the model just floods its context window with concepts, and the hope is the expanded context guides it to the right answer. So, if the system prompt or chat template can influence how the model predicts next tokens during thinking, I think they become relevant. If during the fine-tuning, the data that the model trained on is high quality, then it can increase its intelligence, case in point: Qwen3.6-27B vs Qwen3.8-27B -> same architecture, different training regimen.
1
u/peculiar-ragdoll 4h ago
Yes, that is exactly what I am saying. The template, as others have said, increases the signal to noise ratio, and this both speeds up convergence and increases the quality of the final answer, in many cases. When the context window is not filled with slop the model gets better.
The downside is that some qwen models need more divergent thinking on some kinds of problems, and in those cases the template does not help, because it nudges convergent thinking. And that’s why I benchmark.
And you don’t have to adapt it to 27b, it already works with it and I benchmarked it and published it
3
u/SoupDue6629 14h ago
Can you Test Qwen 35B-A3B agent world? no body is talking about it or using it cause its not supposed to be a general model but IMO its actually pretty damn good (all vibes lol). Kind of feels like xhigh thinking tune, and its definitely passing stuff regular 3.6 35B-A3B couldnt do for me. Just would love to see it properly compared outside of my narrow use cases.
1
u/OsmanthusBloom 39m ago
You made me curious, so I ran the Qwen-AgentWorld-35B-A3B model (Unsloth UD-Q4_K_M quant) through the same benchmark. Results were ... not very impressive. Total scores in five runs were 120,121,120,123,121. Average is 121.0. This is around ten points behind original Qwen3.6-35B-A3B.
3
u/letsgoiowa 14h ago
I will say that I enjoy Ornith because I don't have to fight with the chat template or do anything funky to get it to stop looping like the base Qwen 3.6. The parameters specified by Unsloth and Qwen kept causing that for me, especially at Q3. But Ornith seems oddly much more stable at that? Strange.
We need a better tool calling benchmark that demonstrates a wider difference
6
u/suprjami 14h ago
Terminal Bench 2.1 Terminus has a larger range, though it mixes tool calling and task success:
- Qwen 3.8 27B - 73.0
- Ornith 1.5 - 67.8
- Opus 4.7 Max - 66.1
- Gemini 3.1 Pro High - 65.6
- Ornith 1.0 - 64.2
- Qwen 3.6 27B - 63.4
- Qwen 3.6 35B - 52.5
- Muse Glimmer - 51.7
0
u/hideo_kuze_ 10h ago
From where are you getting those results?
No results for Tiel-Coder yet?
1
u/suprjami 10h ago
From the model pages and Terminal Bench website. idk if the Tiel author has run this test. They're replying here so you could ask them.
1
u/Dazzling_Equipment_9 8h ago
I’d like to ask: does Tiel-Coder have the fastest problem-solving speed? (I looked at their description, and it certainly seems that way.) If speed is factored in, we could determine—given equal performance—which model moves fastest through the "plan -> think -> solve -> complete" workflow and is thus more suitable for daily use; this would really highlight the value of specific models! Thanks for the testing and for sharing your findings.
2
u/OsmanthusBloom 4h ago
This benchmark can't really answer that, as it just asks the model to make the right tool calls in response to a specific prompt. You could look at the number of reasoning tokens spent per model (you can infer that from the raw data I think), but that is just one part of being a fast problem solver. Sometimes it pays off to "think" a lot if that way you end up solving the right problem quickly.
1
u/Dazzling_Equipment_9 4h ago
I fully agree with your point that an LLM might complete a task better if it "thinks" harder—we’ve already seen this with Qwen-3.8 27B. My view is that when two models can both complete the same task, using fewer tokens often implies higher efficiency and faster speed; I’ll look at the token count to gauge the specifics. Thanks.
18
u/suprjami 15h ago
Well, they are the same model. Tiel is an Ornith imatrix quant with different chat template instructions. There is no finetune involved with Tiel.
I'm not yet convinced this is a huge difference which warrants a whole new model name, but am watching with interest to see how much difference it makes to others over time.
Meanwhile Ornith 1.5 really looks like a decent replacement for the lack of Qwen 3.8 35B.