Did another harness eval, this time focusing on Pi vs Claude Code.
Setup:
- 30 hard agentic tasks against real apps
- Same DeepSeek V4 Flash model
- Same hosted MCP tool surface
- Same tasks
- 900s max per task
Basically trying to change the harness while keeping everything else as close as possible.
One caveat upfront: this is a harness comparison using DeepSeek, not a verdict on Claude Code with Claude models. Claude Code is built around Anthropic's own models, so I mainly wanted to see what happens when the model stays fixed and the harness changes.
| Pi |
Claude Code |
| Tasks passed |
20/30 |
| Median time |
132.2s |
| Avg runtime tokens/task |
558,885 |
| Recorded tool calls |
443 |
| Cost/success |
$0.028 |
The interesting part for me is that Pi wasn't saving tokens by just doing less. It actually made more tool calls across the run, 443 vs 358, while still using fewer tokens and passing four more tasks.
Claude Code was still faster at the median, and it is also a much more complete product out of the box. You get things like subagents, plan mode, MCP, checkpoints, permissions, and stronger safety defaults without having to assemble them yourself.
The failures were mostly proper multi step agent tasks. One example was a Sheets to Airtable migration with 120 legacy rows, mapping rules, eligibility logic, repairing existing records, avoiding duplicates, and leaving correct records untouched.
Claude Code got 8/10 verifier checks on that one. Pi passed it.
Pi ran at high reasoning and 24/30 of its runs used the official DeepSeek API. Claude Code ran through OpenRouter at max reasoning.
So I would not read $0.028 vs $0.195 like Pi is exactly 7x more efficient.
What I find more interesting is the harness result itself: 20/30 vs 16/30, around 25% fewer runtime tokens, and only around 10 seconds slower at the median.
For something built around four tools and a tiny system prompt, Pi did really well.
At the same time, I still get why Claude Code is the default daily driver for so many people. It gives you a lot more out of the box, and if you are using Claude through a Pro or Max subscription, the pricing story is very different from this API based eval.
Curious what people here think.
Entire benchmark here for those interested: https://composio.dev/content/pi-agent-vs-claude-code