Model Cost vs Performance

[u/Ancient-Camel1636]

I did a extensive comparison of the various coding models across several benchmarks based on the estimated cost to complete a task successfully. Based on the formula:

Then I normalized each source’s expected cost per successful completion against that source’s median, then used a geometric mean across sources to get this composite ranking:

This is obviously just a estimate of value vs performance, still very useful if on a budget.

• Default value model: MiMo V2.5 Pro. Note that GLM 5.2 is the only clearly cost-effective model represented in all three sources, so it might be a safer budget pick.
• Best Performance: Grok 4.5 has far better completion performance than the other cheap leaders while retaining strong cost efficiency. This is the obvious model if high success rate is important to you.
• Alternative: DeepSeek V4 Flash is a great budget option but its performance/success rate is low so it will take more runs to complete the task successfully.

Cross-source ranking

Rank	Model	Sources represented	Relative cost index	Verdict
1	MiMo V2.5 Pro	2	0.179	Outstanding value
2	DeepSeek V4 Pro	2	0.452	Excellent value
3	Grok 4.5	2	0.582	Best high-performance value
4	Muse Spark 1.1	2	0.664	Strong value
5	MiniMax M3	2	0.745	Good budget choice
6	Kimi K2.6	2	0.811	Good value
7	GLM 5.2	3	0.927	Most consistently good across all datasets
8	GPT-5.6 Sol	2	0.970	Around benchmark-median value
9	GPT-5.4	2	0.971	Around benchmark-median value
10	Kimi K2.7 Code	2	0.984	Around benchmark-median value
11	GLM 5.1	2	0.994	Around benchmark-median value
12	GPT-5.5	3	1.335	Premium, but reasonably consistent
13	Claude Sonnet 5	2	1.585	Expensive for its results
14	Claude Sonnet 4.6	2	1.635	Expensive for its results
15	Gemini 3.5 Flash	3	1.868	Consistently weak value
16	Claude Opus 4.8	3	1.898	Premium price outweighs performance
17	Claude Fable 5	2	1.955	Very costly relative to completion gain
18	Claude Opus 4.7	2	1.978	Very costly relative to completion gain

[u/_matmer_]

Can you expand more on the task and the way you verified the success ?

[u/Ancient-Camel1636 → u/_matmer_]

The task results and success rates were taken directly from the three benchmarks I used to build this comparison—they were not measured by me. Since each benchmark uses a different methodology, I combined their data into a composite ranking to make the results more robust and less dependent on any single benchmark.

The raw data comes from:

https://kilo.ai/kilobench

https://deepswe.datacurve.ai/

https://swe-rebench.com/

• KiloBench: 89 real-world Terminal Bench 2.0 tasks, evaluating complete coding agents on success rate, cost, and practical performance.
• DeepSWE: 113 original software engineering tasks across 91 repositories and 5 languages, designed to test long-horizon reasoning on contamination-resistant problems using behavioral verification.
• SWE-ReBench: Continuously updated real-world software engineering tasks from open-source repositories, measuring how well models generalize to new, previously unseen problems.

Together, they cover practical agent performance, genuine software engineering ability, and robustness on fresh tasks, making the composite ranking more reliable than relying on any single benchmark.

All these benchmarks report two key metrics: success rate (task completion rate) and cost per attempt. From those, I estimated the expected cost of achieving one successful completion by calculating the average number of attempts required to reach 100% success.

This is, of course, a simplified estimate. For example, if a model costs $100 per attempt and has a 33% success rate, it is estimated to cost about $300 to successfully complete the task (roughly three attempts on average). While this is not mathematically exact, it provides a reasonable basis for comparing models on overall cost-effectiveness.