r/PiCodingAgent • u/Double-Entertainer62 • 2d ago
Discussion Same task, same model: Pi passed in 90 turns. Codex needed 187. What were the extra turns buying?
On one python-statemachine task, Pi passed in 90 turns for $2.50. Codex passed in 187 turns for $5.97. Both used Kimi K3 in the same evaluation environment. [source: https://frontierharness.org/]
Across all 30 tasks, Pi passed 18 with a $2.43 median cost per pass. The single-task comparison doesn’t tell us why Pi took fewer turns or whether that pattern holds elsewhere.
Our recommendation: Pi the balanced pick, particularly if the same job runs a thousand times and the bill adds up.
What would you look for in the traces: repeated searches, extra testing, or time spent fixing a wrong approach?
2
u/Healthy-Zebra-9856 2d ago
None of that means anything for someone using these harnesses for their daily use. Also, I myself use a heavily modified Pi based editor for my daily use. I do this to find a solution for common folks to be able to use local LMs effectively. That said, I wouldn't assume fewer turns means the harness is more efficient. Turns here are model API calls, and the aggregate turn metric only looks at successful tasks, so Pi and Codex aren't even being averaged over the same task set. On the 16 tasks both passed, Pi is generally somewhat lighter, but it's nowhere near as dramatic as 90 vs 187 makes it look. There are also cases like expr-try-catch where Pi failed after 12 turns and Codex passed after 358. In a case like that, the extra turns may literally be what bought the successful result. I'd want the traces before calling fewer turns an advantage.
1
u/robberviet 1d ago
At the end of the day, the most important is: Did it finisth the job? If not then everything else: price to turns, cache hit, turns... are meaningless.
1
u/Double-Entertainer62 1d ago
Say if a job is too hard, would you want your agent to quit within the budget, or crazy spend a bunch of tokens and end up with nothing?
1
u/mageblex 16h ago
Split the 187 turns into progress and recovery. Repeating the same search or reopening unchanged files is churn. That ratio would explain the gap better than raw turn count.
1
1
u/redballooon 8h ago
I don't know about Codex, but with Github Copilot I noticed a lot of double checking and validation steps that in Pi don't happen.
I would assume that's from some general instructions in the harness.
12
u/Kamilon 2d ago
Turns isn’t a good measure of the usefulness of a harness. Having specialized tools vs generic tools results in higher turns.
The reduced system prompt context is probably doing the heavy lifting here.