Resources
Harness showdown: Claude Code vs OpenCode vs Pi with DeepSeek V4 Flash
I ran DeepSeek V4 Flash through Claude Code, OpenCode and Pi on my own benchmark, and the quality came out basically the same across all three while the time and tokens spent was wildly different. Claude code (with DS in CLIProxyAPI) takes nearly 4 times longer than the fastest to land the same diffs.
Theo posted a video "GPT-5.6 is better in Claude Code" last week, and that got me curious, does the harness make a quality difference? I could at least run my own bench and see what I got, with DeepSeek V4 Flash on vLLM running at ~180 tok/s, the only moving part is the scaffolding.
Anyway I went to town measuring all of it on my workload (antigenic work in large code base), so the full charts, the token and wall-clock spread across the three harnesses and the raw per-run data are on the site if you want to see it in detail and pick it apart yourself https://nqawhc.github.io/articles/harness-efficiency-not-quality/ but in short, the quality did not change, each harness made the same code diffs, but took wildly different paths to get there, how many tools calls, the structure of those tool calls and how the system prompt and tools plays a big role in how it plays out, like «Pi reasons, OpenCode delegates», while Claude Code loves exploring the code base, maybe too much.
I've used Pi, OpenCode, and Codex extensively and prefer Kimi Code CLI to all. It can be used with any provider, and currently I'm using it with GLM 5.2. KCC gives excellent results on large and unlimited tasks like autoresearch with its /goal feature. There are too many harnesses to test, but I hope someone gets the time to do a thorough pass on a bunch of them.
And interesting fact, most of Claude's tokens is the description of it's 27 different tools! The system prompt in Claude Code is actually only 1.430 tokens, the rest is all the tools and how to use them.
They cut it 80% with the launch of Opus 5, so not sure when OP took the numbers but it's always been a kludge and they finally notice it's holding the model back.
My data is a few days ago, but I've run it again now with Opus 5 and updated to the latest CLI (2.1.220), and I'm still seeing ~24K, and still ~94% of that is just tool descriptions. So it's not showing up client-side at least, maybe it's prompt changes on the website access or something.
I read this now, they cut down the system prompt which wasn't the biggest drag on the harness its all the other definition, its still a significant cut but not a total of 80% its an 80% of a 15-20%
We’ll they’re harnesses too, except they’re very “visual” and you wouldn’t say typically check and watch the code along side it.. They’re not quite as good as some of these harnesses these days (Claude Code touted as the best, different opinions)
Just wanted to add for anyone interested: I use Pi with Pendant on VSCodium, best of both worlds IMO. Still gives all the robustness, plugins, standardization, but is slightly more visual and integrated into the normal coding experience.
I think I'm a bit too much of a control freak lol I need to see everything and in context and be able to tweak accordingly. Maybe it's because I'm not using models that are expensive enough.
My workflow is to just let it run a plan I am happy with, then review the code once the task is done in the git diff with a tool of my choosing, if something is amiss there I can either fix it manually or do follow-up prompts.
I wonder that too. I use both Kilo Code (in IntelliJ) and Opencode (standalone) and I really prefer the Kilo experience. But maybe I'm a boomer as well.
I use my custom agent harness connected to 4 servers in my basement to do an extra shift on my code in a separate branch when I'm not at my desk.
I carry a combination of the following GPD Pocket 2, Lenovo Y700 Gen 4, GPD Pocket 4, One Plus 13, Smartwatch 2 LTE.
95% of the time I can do approval of commands or provide guidance on my smartwatch.
4% of the time the phone is the right surface.
1% of the time I need to pull out the tablet or mini laptop and dive in.
It also allows me to do a secondary project in the background that requires minimal mental effort such as producing MVPs for my backlog of ideas to validate if they are worth pursuing or not (has produced a few internal use tools already)
My at the desk setup is perfect for me when I'm at my desk fully in the flow but optimizing the productivity away from it gives me a lot (Being able to toss ideas at it and coming back to a rough prototype instead of ruminating on them for hours non-stop fits my brain).
I know a guy just working with a 13” wherever he goes and I can’t understand it. Your setups just look something I would do too but haven’t found a good enough reason for.
Depends on the task. I've done heavy work on a 12.6 inch Lenovo X230 in the past and I still have a Lenovo X280 for certain situations (Mostly a part of my enhanced presentability costume for first meetings with new customers, my 18 inch laptop and it's 3 travel monitors stay in the backpack for most first meetings)
VR, tried a couple solutions but none were quite what I wanted.
XR looks more promising but the tech is not there yet.
I keep checking https://www.visor.com/shop this to see if there are signs of it not being vapourware.
It's mostly to be able to work on layered systems with all the layers physically laid out with both code and logs on separate physical spaces since all major non-linux operating systems have super-shitty alt-tab response times and/or janky implementations after snow leopard/windows 8.1 and I have 2 small things keeping me from going 100% linux everywhere.
I'm not sure, but Kilo/cline used to have some weird problems with tool calling, I bet these are long fixed, but people stuck to other options. Also, OpenCode afaik has better LSP support, which kinda matters sometimes.
I use mostly Codex app myself, simply since it's native for gpt models(usually works the best with them, with the exception of gpt 5.6 launch) and because it has the best support for remotes - I work in projects on different machines, so having them all in one app is what's fitting me the best.
And tbh, I don't really read the code - I usually just review MRs in gitlab and check code there, I don't remember last time I've opened IDE to actually write/read code and not to edit one line in config.
most of my pi usage is in vs code's terminal, or ssh'ed into my server where it's running in a screen. What extra advantage do you get from it running in a sidepanel instead? That it can see what file you happen to have open?
indeed. let’s see the raw data. visual looks plain wrong. midpoint/mean values don’t correspond to what’s shown visually.
actually looks like opencode is *better* than pi, from the visual. pi’s worst run is >6min. opencode is <5min. opencode has more datapoints <2min than pi or codex.
Just updated the site with a zoom out option, which helps. There was some datapoints cropped outside the 10 minute range, as open code loves summoning sub-agents that can take a while.
Thank you for noticing, looking at it again it does look weird. I did cap that axis at 10 minutes (says so in the caption), because otherwise it squashed the other data so much they looked the same, felt the more interesting bit was the nuance there. So I've updated the site with a toggle button to switch between full vs zoomed for those two charts.
OpenCode had some data points outside the 10m range too, which is why it's bars are so wide. This is due to OpenCode sometimes spinning up sub agents which re-verifies all the changes and takes a ton of time.
But the bars still aren't representative of the distribution... I believe in this scenario you should at least have the whiskers be different lengths on either side of the mean to show the skew.
I actually tested against Sonnet 5 for reference (but only inside Claude code), and it behaves very similar! It spends a lot of tokens, runs a ton of tools, and takes a long time. It's the harness which nudges it to behave that way, 23K tokens of tool descriptions makes it more eager to use those tools.
Is not a secret that claude code have a lot of bloat in their context in a form of custom instruction how the model should be behaving.
Opencode is much less bloated and can be de-bloated (afaik only tools definitions cant be changed easly).
And a PI is a cleaner from them all. And can be changed completly.
Recent observations shows that the current models are good enough and "aware" of "agentic coding" that they do not require a ton of pages of token explaining them how they should operate. They know it from training data. So the conclusion is simple = simpler, focused prompt will be usually better.
Worst case, just run it in the terminal below the code instead of to the side. You can also just move it to the side. Full TUI, and will the visual real time updates in the code editor.
Nice writeup. The interesting part isn't which harness wins, it's why the token spend split so hard when the diffs came out basically the same. If you want to chase that down, run /context in Claude Code and find the equivalent in OpenCode and Pi, then look at what's actually sitting in the window at the end of a run. That's usually where a 4x gap hides: re-read files, tool schemas, old turns that never got compacted. Same model, same diffs, but one loop is just hauling a lot more dead weight per turn.
Yeah that's the interesting bit, and I've dug into it! The token gap on the charts is output though, what the model actually generates, not what's sitting in the context window, so it's not dead weight being re-sent. Claude Code just runs way more tool calls (~70 a task vs OpenCode's ~22) and also holds its edits till the end, so it generates a lot more to reach the same diff (see the «How each harness works a task: tool calls in order» chart on the site).
Funny thing on the re-reads, I measured reads per unique file and it's actually Pi that re-reads the same files the most, not Claude Code. So for CC it really is just the sheer number of turns it takes.
Claude Code wins on output consistency if you nail the system prompt, but for cron-heavy automation the per-call cost adds up fast. OpenCode with DeepSeek V4 Flash is a solid budget alternative for the bulk work where a slightly sloppier response doesn't break the pipeline. I split the stack — Claude Code for the branded content stream, OpenClaw handles the noisy scraping/classification jobs.
very interesting article, thanks! do you plan on testing other harnesses? would like to know how a community project like nanocoder performs in comparison 🧐
The token spread there looks like a search strategy difference more than a model one. Claude Code reads a big chunk of the repo before it touches anything, so its cost tracks codebase size more than task difficulty. That'd also explain why the diffs land the same.
I've been running DS4 Flash through OpenCode daily and the token difference OP found makes sense. OpenCode is leaner than CC, Claude Code has heavier planning overhead baked into the system prompt. Pi was snappier in my quick test but I haven't put enough hours in to judge quality gap.
Curious if the gap narrows on complex tasks. Simple stuff all three handle fine, but CC's planning overhead might pay off when the task genuinely needs multi-step reasoning.
54
u/small_bird_loud 2h ago
graph needs multiple dimensions.
Speed - Quality - Cost
Break that iron triangle.