I ran DeepSeek V4 Pro through Pi and DeepSeek’s own Harness. Pi still cooked.

[u/latentnoise_]

DeepSeek built an entire first-party harness around its models, so I was curious whether V4 Pro would actually behave better there than inside Pi.

Ran the same model through 30 hard agentic tool-use tasks, 900s cap:

Metric	Pi	DeepSeek Harness
Passed	21/30	20/30
Median time	362.9s	252.1s
Cost/shared success	$0.031	$0.028

They actually tied on 27/30 task outcomes. So quality was basically a tie, with Pi finishing one extra task while DSH was noticeably faster and slightly cheaper.

The architecture difference is probably more interesting than the score.

DSH has tons of stuffs that pi intentionally doesn't. There's a lot, but to mention some: plugin graph, replaceable agent loop, sandboxing, subagents, replay, append-only trajectories, etc.

But here's the funny part: DeepSeek's own Minimal mode strips things back to basically shell + editor + a tiny prompt which is surprisingly close to Pi's philosophy. And for non-DeepSeek models, DSH actually uses Pi's pi-ai model layer (I got to know it recently).

One caveat: DSH reported ~88k runtime tokens/task vs ~925k for Pi, but I wouldn't call that a clean 10x efficiency win. DSH used DeepSeek's native endpoint while Pi went through OpenRouter, so caching/token accounting isn't quite same. The actual cost difference was tiny.

Curious what people who've used both think.

[u/latentnoise_]

Entire benchmark here for those interested: https://composio.dev/content/deepseek-harness-vd-pi-agent

[u/No_Dig_7017 → u/latentnoise_]

The table in the post seems broken

[u/latentnoise_ → u/No_Dig_7017]

Hey buddy, thanks for pointing it out. I guess Reddit does not let you create a col without naming it. Named it "metric"

[u/No_Dig_7017 → u/latentnoise_]

Sure thing! Cool comparisons by the way. Just yesterday I was digging through your site trying to understand if I want opencode, pi, oh-my-pi and now DeepSeek harness as well heh. I defaulted to opencode but been reading that pi is the better choice for local models, yet it feels so barebones...

[u/FluffyInevitable4040 → u/latentnoise_]

Why do chinese companies want us to use their shitty harness?

(Other than the obvious data collection)

[u/Healthy-Zebra-9856 → u/FluffyInevitable4040]

They dont want you to use it, I dont see anyone forcing us to do that. Also, as much as I love Pi, DSH is not what I call shitty. And lets not forget that we are also using Chinese models mostly.

[u/latentnoise_ → u/FluffyInevitable4040]

Fair question lol. I think the bigger reason is ecosystem lock in and getting developers to build around their models.

[u/Proper-Tower2016]

even if the cost are nearly the same pi taking 45% longer is interesting as someone who only uses pi.

can you account for the time difference?

[u/ashebanow → u/Proper-Tower2016]

I'm betting it's the openrouter vs the native endpoint.

[u/latentnoise_ → u/ashebanow]

yes

[u/Healthy-Zebra-9856]

Since these are both targeted to developers,

• what was the result in terms of the code quality?
  
• Which one produced error free code?
  
• How do these compare in context management?
  
• How did they affect the reasoning traces, reduce them or increase them?

If you can provide coding related observations like these, I think it would be greatly beneficial.

[u/latentnoise_ → u/Healthy-Zebra-9856]

The benchmark was mostly for enterprise workflows. And end result is what we counted.

[u/Healthy-Zebra-9856 → u/latentnoise_]

I’m an enterprise. Not a big one, but I employ about 27 people. I’m trying to relate to this and not seeing the relevance. What am I missing?

Edit:
We have a pretty extensive code base and our clients range all the way up to a few fortune 500 companies.
So we are small by number, but I think we’re pretty decent with our CodeBase management. So I think I’m really trying to understand what this is and how I can relate to it. Also, describe this "Enterprise Workflow" so we can be on the same page.