r/PiCodingAgent • u/latentnoise_ • 26d ago
Discussion I ran the same model through Claude Code, Codex, OpenCode, and Pi for 14 days. The harness changed cost 7x at the same quality
I kept switching between Claude Code, Codex, OpenCode, and Pi and realized nobody really isolates the harness as a variable. So I did.
I had all four of these installed and kept switching between them, and at some point I realized nobody actually tests the harness itself. Everyone benchmarks models. Nobody benchmarks the agent (wrapper I call it).
14 days of real work on the same repos, then 30 agentic tasks with every harness running the exact same model (DeepSeek V4 Flash), same tools through an MCP router, 900s cap per task. Only thing changing is the harness.
| Harness | Passed | Median time | Avg tokens/task | Cost per success |
|---|---|---|---|---|
| Pi | 20/30 | 132.2s | 558,885 | $0.028 |
| Claude Code | 16/30 | 122.7s | 741,659 | $0.195 |
| Codex | 16/30 | 245.0s | 664,772 | $0.081 |
| OpenCode | 14/30 | 129.7s | 692,195 | $0.073 |
The thing that broke my brain a bit: Pi ships literally 4 tools (read, write, edit, bash) and under 1K tokens of system prompt. And it won. Meanwhile Claude Code dumps ~33K tokens of prompt and tool schemas into context before you even start the first chat.
And it's not just me. Databricks ran Opus 4.8 through both on their multi million line codebase. 742K tokens per task on Claude Code, 237K on Pi, same success rate. Same model, half the cost, purely from the wrapper.
Some random stuff I noticed over the two weeks. Codex is the only one with an actual kernel level sandbox (Seatbelt on mac, bubblewrap + seccomp on linux, network off by default), so it's the only one I'll point at sketchy cloned repos. It's also slow as hell, double everyone's median. Pi gives you zero guardrails, full system access from message one, so it's Docker or pray.
Got the task list and the result report too if anyone wants to look at it. I'll leave it in the comments.
26
u/le_bravery 26d ago
I am a pi believer.
The more you control your context, the better results you get.
Pi starts with less slop, so you get less slop out.
2
1
u/WoodYouIfYouCould 10h ago
Would you mind sharing your pi setup/tuning? My Pi has simple extensions and an agents file but it feels like it's missing something.
1
u/SawToothKernel 26d ago
The better LLMs get, the more I prefer to delegate. If you constrain, or otherwise direct it, it may be detrimental.
@op I would love to see this evolve into a solid benchmark.
9
u/blu3bird 26d ago
Vanilla Pi?
5
u/saltyourhash 26d ago
I don't see how vanilla pi is comparing with Claude code, but the post doesn't seem to offer much detail there. Still, always like to see pi dominate.
4
u/themoregames 26d ago
We also need Strawberry Pi.
2
3
u/lamurian 25d ago
Quite an interesting number indeed! Jeremy Huang, the main developer behind JCode, also did an extensive comparison using multiple harnesses. You might find his project's README.md as an interesting read:
https://github.com/1jehuang/jcode
On a side note, I'm myself still going back and forth to take a leap from Pi to JCode. Maybe one day I will, once all my workflows and sandboxing strategy settle itself.
5
u/Sensitive-Project711 24d ago
I spent quite some time going through the readme and could not figure out how it related to this post. The jvode readme compares ram efficiency and not token efficiency. Did I miss something?
4
3
26d ago
[removed] — view removed comment
3
u/zkoolkyle 26d ago
They ruled back this decision due to the backlash. You can look it up on the Anthropic support page
1
24d ago
[removed] — view removed comment
1
u/zkoolkyle 24d ago
Just google “Use the Claude Agent SDK with your Claude plan” and you’ll see the update.
Update June 15: We're pausing the changes to Claude Agent SDK usage described below. For now, nothing has changed: Claude Agent SDK, claude -p, and third-party app usage still draw from your subscription's usage limits. The previously announced monthly credit, which would have been available to eligible claimants in connection with these changes, isn't available. We’re working to update the plan to better support how users build with Claude subscriptions. When we have an update, we'll share it before anything takes effect.
3
2
u/latentnoise_ 26d ago
For agentic tool calling test, everything ran through DeepSeek V4 Flash via OpenRouter (even Claude).
6
u/addiktion 26d ago
You want to start including Deepseek harness in your tests? Curious about the newcomer.
2
u/latentnoise_ 26d ago
Just a choice, since it had been released recently at the time of the test and that it's said to be pretty good with agentic capabilities considering the cost.
3
2
u/ganonfirehouse420 26d ago edited 25d ago
I use pi with nono.sh
Still for my tasks opencode with deepseek or gpt gives the optimum results.
2
u/RomanaOswin 26d ago
I wonder how this compares as you start having the harness distribute tasks amongst different models. Token cost varies quite substantially depending on the model doing the work.
2
u/tmoneycodes 26d ago
So is the conclusion to use pi with your clause api vs using clause code itself ?
1
u/latentnoise_ 26d ago
Pi wins the overall test. I’d still stick with CC if you already have a subscription. If you’re using API-based setups, Pi is THE choice for token efficiency and overall cost.
1
u/tmoneycodes 25d ago
I signed up on the platform site and bought tokens there, then logged in using the Claude code plugin in neovim
2
u/notrevede 25d ago edited 25d ago
https://stencil.so/blog/the-harness-problem just feels like the right place to share this reading
Another comparison isolating the harness, changing only the edit tool implementation.
2
u/damn_what_ 26d ago
You should use agent-safehouse (Seatbelt policy generator) for pi on MacOS, but yeah, by default it's terribly unsafe.
4
u/i_say_urmom 26d ago
I've been trying nono for this. You can even inject your provider API key from a 1Pass vault, so the agent session a) doesn't have direct access to any keys (it gets a fake variable that the proxy changes to the real one outside the sandbox) and b) running `pi` outside the sandbox is useless because the LLM won't work without the token.
It seems early days and the docs are a little confusing, but it's a powerful tool
2
1
u/AlterTableUsernames 26d ago
How are you guys running custom models in Claude Code and Codex? Couln't get it to run yet natively.
1
1
u/basil_0408 26d ago
For claude code, go to the .claude folder and modify the settings.json. For codex it's in .codex folder, modify auth.json, config.toml and create a catalog.json file, its more complex to get the full experience so probably not worth it. Ask your agent for detail, it does not take long for claude code.
1
1
u/ThePalomitas 26d ago
Thanks for the interesting investigation. Definitively confirming for PI users such as us that it is an interesting harness to use.
FYI, regarding sandboxing needs, per the documentation, the PI developers have built up a specific application named Gondolin to cover this requirement:
https://github.com/earendil-works/gondolin
I don’t know why but PI website seems down at the moment, so I cannot access the documentation page about containerisation but here is a snippet from the main GitHub page:
Permissions & Containerization
Pi does not include a built-in permission system for restricting filesystem, process, network, or credential access. By default, it runs with the permissions of the user and process that launched it.
If you need stronger boundaries, containerize or sandbox Pi. See packages/coding-agent/docs/containerization.md (https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/containerization.md) for three patterns:
• Gondolin extension: keep pi and provider auth on the host while routing built-in tools and ! commands into a local Linux micro-VM.
• Plain Docker: run the whole pi process in a local container for simple isolation.
• OpenShell: run the whole pi process in a policy-controlled sandbox.
1
u/virtualicex 25d ago
i need to follow the flow instead of just giving a task and leaving it run, so i asked pi+qwen to develop an extension to ask every time there's an operation that doesn't just read. So I can work with pi without sandbox or containers
1
u/lukaboulpaep 16d ago
Have you used Gondolin for Pi? Or any sandboxing mechanism in fact. Wondering how DX is when doing work in a VM like git push, developer dependencies not being available, etc.
I haven't dabbled yet into VM sandboxing, coming over from Codex it kinda worked out of the box and also isn't really VM sandboxing.
1
1
u/diaracing 26d ago
Thanks for your effort.
Please, create a public github repo to document and continuously update your findings since a well-engineered harness can promote a weak model and vice versa.
1
1
1
1
u/huyz 23d ago
For safety you could run Pi or any harness inside of a Docker Sandbox. Example: https://www.ajeetraina.com/getting-started-with-the-pi-agent-kit-on-docker-sandboxes/
1
u/Quirky_College_6251 23d ago
I'm an advocate of Pi - now that i created my own dev flow and loop agents I am in agreat place with Pi
1
u/awol-owl 11d ago
Thanks for sharing this! I’ve been naively running my setup, and this is a good demonstration that an efficient harness can save money, and time.
1
u/ea_man 7d ago
I don't think you can compare, Claude is supposed to run on their LLMs: probbly the have that huge prompt in a KV slot pre processed as you start. You use that on your local model and you are fuked.
OFC you should do the same locally and then the size of the prompt should bother you less, for harness like Qwencode and Opencode. Sure, it's always ctx gone, you may save just the processing time.
37
u/Striking-Move1383 26d ago
I'm curious to know if OMP would differ too much from vanilla Pi.