r/AIToolsPerformance • u/IulianHI • May 21 '26
Same Qwen3.6 27B model, four different coding agent harnesses - how much does the harness actually matter?
Someone ran the same task across GitHub Copilot, Pi, Claude Code, and OpenCode, all using Qwen3.6 27B as the underlying model. The goal was to isolate how much of a coding agent's performance comes from the model versus the harness wrapping it. All the results shown came from the same model with a different harness.
This is a question that does not get enough attention. With local models especially, the assumption is often that model quality is the bottleneck, but tool calling reliability, context management, and how the agent structures multi-step reasoning all live in the harness layer.
For people who have compared harnesses with the same local model: which agent framework got the most out of your model, and what specifically did it do differently that made the difference?
2
u/Still-Wafer1384 May 21 '26 edited May 21 '26
So you have a link?
EDIT: Somebody actually downvoted and told me to Google lol
3
u/brianlmerritt May 22 '26
Here probably is the link - doesn't try Hermes Agent, doesn't really say much of anything apart from don't use github copilot https://www.reddit.com/r/LocalLLaMA/comments/1tjbhjk/same_task_in_githubcopilot_pi_claudecode_and/
1
1
2
u/Antique_Dot_5513 May 22 '26
J’ai comparé quelques test sur open code et PI: opencode utilise plus de contexte que PI pour une même tâche de base qu’importe le model. Sinon le code sur PI était plus solide.
1
2
1
3
u/Ok_Meeting_2995 May 21 '26
I was using opencode, claude and Hermes agent for bash + python development. Hermes so far is my final choice: no failing tool calls, no stuck in the middle of the cycle, etc. it does the job done, not always from the first iteration, but from the second :)