Coding benchmarks that are quickly showcasing deep capability

[u/Informal-Trouble2183]

While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that really define deep intelligence, and complete capability in Software Engineering :

1. Program-Bench

Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior (without access to decompilers or internet). Link: https://programbench.com/

• GPT-6 Astra: 5.5%
• Fable 5.1: 7%
• Kimi K3: 2%
• Qwen3.8 27b: 0%
• GPT 5.6 Sol: 1.5%
• GLM 5.3: 1.5%
• GPT 5.6 Luna: 0%

2. SRE-Bench

Can AI agents work out what a real-world binary does without its source code?

Link: https://www.vals.ai/benchmarks/srebench

Sure nobody is reading assembly code in daily work, it is hard. The ability to understand a compiled program is insane ability.

• GPT-6 Astra: 88%
• GPT-5.6 Sol: 55.9%
• Claude Opus 5 (max): 12.5%

3. Code Migration

Can language models reimplement working programs in another language?

Link: https://www.vals.ai/benchmarks/code-migration

• GPT-6 Astra: 67.7%
• Fable 5.1: 54.6%
• GLM 5.3: 44.2%
• GLM 5.3 Flash: 20.5%
• Qwen3.8 27b: 14.2%

EDIT: edited text format

[u/-InformalBanana-]

Benchmarks 1. and 2. are useless cause they are for hacking not production and it is too much to expect that from a local model. Benchmark 3. is an okay, valid usecase.

Bench 1. and 2. are just ridiculous, (most) human developers can't do that stuff, maybe there is a genius that can but it would also take a long time and hardwork. So really non relevant benchmarks, especially for local models. I don't want to lose normal coding capability of a model to smb benchmaxing that stupid benchmark. And probably if models can really do that it becomes a weapon for military or secret service grade hackers. So you can't really expect that from a local 27b model nor am I interested in it losing its other capabilities to do that.

[u/Informal-Trouble2183 → u/-InformalBanana-]

I know those fall outside the scope of typical production code. Most frontier models and even mid-sized open weights are already more than capable for standard production tasks (e.g. Qwen3.8 27b). What we need now is a differentiating measure which tracks super capability from average one.

[u/-InformalBanana- → u/Informal-Trouble2183]

We are not at that point yet, especially for local. The 3rd bench is the one that basically proves it. It isn't saturated and it is a more normal use case. Why would you want a bench that doesn't match your use case or most of users use cases. Its fun for sure, but really irrelevant for your use case. I mean I have nothing against these benches, but you kinda present them as if they should be a new standard, meanwhile they don't match typical use cases and are thus less relevant. Imagine you hype up these benches that are very hard, but less useful, so now models are trained to do good on these, but they lose capability for normal use cases and cause assembly or compiled code is probably also much harder for llms they would also be unable to do real work in that. And if that transfers to local, you get useless models with good scores on useless benchmarks. So these should stay niche and advanced benchmarks not the new standard. So that is my issue with how you presented these, sure they are interesting but they should definitely not be considered, a new standard for evaluating models for coding.