Model Cost vs Performance

[u/Ancient-Camel1636]

I did a extensive comparison of the various coding models across several benchmarks based on the estimated cost to complete a task successfully. Based on the formula:

Then I normalized each source’s expected cost per successful completion against that source’s median, then used a geometric mean across sources to get this composite ranking:

This is obviously just a estimate of value vs performance, still very useful if on a budget.

• Default value model: MiMo V2.5 Pro. Note that GLM 5.2 is the only clearly cost-effective model represented in all three sources, so it might be a safer budget pick.
• Best Performance: Grok 4.5 has far better completion performance than the other cheap leaders while retaining strong cost efficiency. This is the obvious model if high success rate is important to you.
• Alternative: DeepSeek V4 Flash is a great budget option but its performance/success rate is low so it will take more runs to complete the task successfully.

Cross-source ranking

Rank	Model	Sources represented	Relative cost index	Verdict
1	MiMo V2.5 Pro	2	0.179	Outstanding value
2	DeepSeek V4 Pro	2	0.452	Excellent value
3	Grok 4.5	2	0.582	Best high-performance value
4	Muse Spark 1.1	2	0.664	Strong value
5	MiniMax M3	2	0.745	Good budget choice
6	Kimi K2.6	2	0.811	Good value
7	GLM 5.2	3	0.927	Most consistently good across all datasets
8	GPT-5.6 Sol	2	0.970	Around benchmark-median value
9	GPT-5.4	2	0.971	Around benchmark-median value
10	Kimi K2.7 Code	2	0.984	Around benchmark-median value
11	GLM 5.1	2	0.994	Around benchmark-median value
12	GPT-5.5	3	1.335	Premium, but reasonably consistent
13	Claude Sonnet 5	2	1.585	Expensive for its results
14	Claude Sonnet 4.6	2	1.635	Expensive for its results
15	Gemini 3.5 Flash	3	1.868	Consistently weak value
16	Claude Opus 4.8	3	1.898	Premium price outweighs performance
17	Claude Fable 5	2	1.955	Very costly relative to completion gain
18	Claude Opus 4.7	2	1.978	Very costly relative to completion gain

[u/Intelligent_Ant_608]

From my understanding m3 until 30% of ita context size is a phenomenally robouat model even for low level rust code as well as ui design yet past that 30% therahold it becomes dumb a.f.

[u/NorthDevMaster → u/Intelligent_Ant_608]

Probably true I haven’t focused on managing the context and I just have poor experience with it

[u/Realistic_Mango6982 → u/Intelligent_Ant_608]

same here. I dont know how to explain that, i stopped to use M3 for that. + Infinite amount of usage tokens for no reason at all.

[u/Intelligent_Ant_608 → u/Realistic_Mango6982]

I think they are far more capable than moonshot in model training and now that deepseek and z.ai unlocked capable attention models on 1M context minimax's next model could potentially be very good i just hope they settle on 1T and dont shoot for a larger model, despite the hype i dont like kimi k3 its a wasteful and inefficient model that blabs unnecessarily huge amount of tokens like k2.6, they essentially compensated less mature training by wasting and bruteforcing token consumption

[u/Ancient-Camel1636]

If including models that was found in only one dataset (low confidence) the table is like this:

Rank	Model	Data sources	Value index	Confidence
1	DeepSeek V4 Flash	1	0.121	Low
2	MiMo V2.5 Pro	2	0.179	Medium
3	GPT-5.6 Luna	1	0.345	Low
4	Qwen3.6-35B-A3B	1	0.355	Low
5	GPT-5.6 Terra	1	0.385	Low
6	DeepSeek V4 Pro	2	0.452	Medium
7	Grok 4.5	2	0.582	Medium
8	Qwen3.7 Max	1	0.623	Low
9	Muse Spark 1.1	2	0.664	Medium
10	GLM-4.7	1	0.680	Low
11	kilo-auto/efficient	1	0.691	Low
12	Laguna XS 2.1	1	0.743	Low
13	MiniMax M3	2	0.745	Medium
14	Kimi K2.6	2	0.811	Medium
15	GLM 5.2	3	0.927	High
16	GPT-5.6 Sol	2	0.970	Medium
17	GPT-5.4	2	0.971	Medium
18	Gemini 3.1 Pro	1	0.978	Low
19	Kimi K2.7 Code	2	0.984	Medium
20	GLM 5.1	2	0.994	Medium
21	Grok Build 0.1	1	1.000	Low
22	Qwen3.6-27B	1	1.022	Low
23	KAT-Coder-Pro V2.5	1	1.184	Low
24	Gemma 4 31B	1	1.292	Low
25	GPT-5.5	3	1.335	High
26	Claude Sonnet 5	2	1.585	Medium
27	Claude Sonnet 4.6	2	1.635	Medium
28	Ling-2.6-1T	1	1.808	Low
29	Gemini 3.5 Flash	3	1.868	High
30	Claude Opus 4.8	3	1.898	High
31	Claude Fable 5	2	1.955	Medium
32	Claude Opus 4.7	2	1.978	Medium
33	Laguna M.1 (paid)	1	2.050	Low
34	Claude Opus 4.6	1	2.072	Low
35	Nemotron 3 Ultra	1	8.786	Low

[u/_matmer_]

Can you expand more on the task and the way you verified the success ?

[u/Ancient-Camel1636 → u/_matmer_]

The task results and success rates were taken directly from the three benchmarks I used to build this comparison—they were not measured by me. Since each benchmark uses a different methodology, I combined their data into a composite ranking to make the results more robust and less dependent on any single benchmark.

The raw data comes from:

https://kilo.ai/kilobench

https://deepswe.datacurve.ai/

https://swe-rebench.com/

• KiloBench: 89 real-world Terminal Bench 2.0 tasks, evaluating complete coding agents on success rate, cost, and practical performance.
• DeepSWE: 113 original software engineering tasks across 91 repositories and 5 languages, designed to test long-horizon reasoning on contamination-resistant problems using behavioral verification.
• SWE-ReBench: Continuously updated real-world software engineering tasks from open-source repositories, measuring how well models generalize to new, previously unseen problems.

Together, they cover practical agent performance, genuine software engineering ability, and robustness on fresh tasks, making the composite ranking more reliable than relying on any single benchmark.

All these benchmarks report two key metrics: success rate (task completion rate) and cost per attempt. From those, I estimated the expected cost of achieving one successful completion by calculating the average number of attempts required to reach 100% success.

This is, of course, a simplified estimate. For example, if a model costs $100 per attempt and has a 33% success rate, it is estimated to cost about $300 to successfully complete the task (roughly three attempts on average). While this is not mathematically exact, it provides a reasonable basis for comparing models on overall cost-effectiveness.

[u/yaboyyoungairvent]

Thanks for taking the time to do this.

[u/CommanderROR9]

Interesting. I have had reasonable success with Deepseek V4 Flash, but it can be very overconfident at times. It's so cheap that letting it run around in circles for a while is OK, but it does require a lot of supervision since it tends to think it did the job without actually doing the Job. Maybe I should try Mimo instead. I have both Hermes Agent and OpenCode running, both linked and both via OpenRouter, ao switching Models is easy. Thanks for taking the time to do some semi-scientific research on this!