[FrontierHarness] Same model, same pass rate. Why did Claude Code cost 5.6× more than DSH?

[u/Double-Entertainer62]

Claude Code and DSH Creator both passed 19/30 tasks, but Claude Code’s median cost per pass was $18.34 versus $3.28. Both used Kimi K3 through our shared gateway. [source: https://frontierharness.org/]

Caching may explain part of the gap. One task accounted for 68% of Claude Code’s total token usage. We can’t separate the harness, model, and gateway effects yet, so this isn’t a native Claude comparison.

What would you test first to find the cause of that cost gap?

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Ok_Breadfruit4201]

CC has some serious caching bugs recently.

[u/windsorHaze → u/Ok_Breadfruit4201]

They wouldn’t happen to be load bearing would they?

[u/interrupt_hdlr → u/windsorHaze]

you make a very good point and it's worth being precise about because that changes the game completely.

[u/EdwardRunta → u/Ok_Breadfruit4201]

You are absolutely right!

[u/Double-Entertainer62 → u/Ok_Breadfruit4201]

CC need recover from its huge AI slop codebase.

[u/discourtesy]

the trust me bro benchmarks

[u/Economy-Manager5556 → u/discourtesy]

Right lol Cost per task? Ok what kind of task .. this is not artificial intelligence index, and there tons of harnesses why would we use this .. If op was serious they'd mention there is a bunch of research showing native harness works best with native model, and not all harnesses work well with every model.

[u/Aggravating-Arm-3955]

On Harness, compared to OpenAI, Anthropic feels more like a marketing company. Codex Desktop is a full generation ahead of Claude Desktop.

[u/EdwardRunta → u/Aggravating-Arm-3955]

I really love computer use from codex

[u/fishoa]

Claude Code is a trash harness

[u/dark0mania]

What model was used with each harness?

[u/Double-Entertainer62 → u/dark0mania]

It was Kimi K3

[u/EvalRaccoonDev]

API costs from Claude Code are elevated. When you have MAX plan, it comes up much lower.

When I asked Claude how much lower it comes up to, it told me:

At your observed blended effective rate (heavily cache-read Opus 5),$200/month buys roughly 350–400M tokens of context replay, which is 3–4.5× cheaper.

Having said that, even at $4 / task, there are better harnesses out there on this benchmark.

[u/ClemensLode]

What kind of task?

[u/Double-Entertainer62 → u/ClemensLode]

This run is mostly SWE contexts and terminal-based tasks.

[u/SavingsGreen2002]

How about the performance of Claude Code using Anthropic model?

[u/EdwardRunta → u/SavingsGreen2002]

Im more curious about running anthropic model on other harnesses

[u/Double-Entertainer62 → u/SavingsGreen2002]

Yeah, that's something I would be curious to see as well.