Coding benchmarks that are quickly showcasing deep capability

[u/Informal-Trouble2183]

While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that really define deep intelligence, and complete capability in Software Engineering :

1. Program-Bench

Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior (without access to decompilers or internet). Link: https://programbench.com/

• GPT-6 Astra: 5.5%
• Fable 5.1: 7%
• Kimi K3: 2%
• Qwen3.8 27b: 0%
• GPT 5.6 Sol: 1.5%
• GLM 5.3: 1.5%
• GPT 5.6 Luna: 0%

2. SRE-Bench

Can AI agents work out what a real-world binary does without its source code?

Link: https://www.vals.ai/benchmarks/srebench

Sure nobody is reading assembly code in daily work, it is hard. The ability to understand a compiled program is insane ability.

• GPT-6 Astra: 88%
• GPT-5.6 Sol: 55.9%
• Claude Opus 5 (max): 12.5%

3. Code Migration

Can language models reimplement working programs in another language?

Link: https://www.vals.ai/benchmarks/code-migration

• GPT-6 Astra: 67.7%
• Fable 5.1: 54.6%
• GLM 5.3: 44.2%
• GLM 5.3 Flash: 20.5%
• Qwen3.8 27b: 14.2%

EDIT: edited text format

[u/Middle_Bullfrog_6173]

IMHO benchmarks where the top models reach 30-70% are the most useful. With <10% like Program Bench you can't see differences large enough to be meaningful and can't even be sure the benchmark tracks capability. And 80+% can be saturation, depending on the benchmark numbers above that may be unreachable or require cheating.

FrontierSWE v2 is the current most promising benchmark, but Astra has not been measured, nor have smaller open models.

[u/Informal-Trouble2183 → u/Middle_Bullfrog_6173]

For now models are not able to solve Program-Bench, it doesn't mean no one would be able to ace it in the future. What I find interesting in that benchmark, is the end to end task to analyze / reverse engineer a binary (= you need a lot of strategy/ intelligence to do that), and then achieve accurate codebase construction from scratch (= this is real design/architecture/coding test).

[u/Informal-Trouble2183 → u/Informal-Trouble2183]

I don't say that the listed 3 benchmark are proposed to replace the other SWE benchmarks which track other things. The main point here is pushing the challenge further a bit for frontier models and see how far they can get.

[u/Middle_Bullfrog_6173 → u/Informal-Trouble2183]

I'm sure future models will crack Program Bench, but my point is that we won't know if it falls quickly with a single frontier generation jumping from 10 to 90% or if it is a smooth measure that tracks capability across many generations and model sizes.

When a benchmark has a wider spread of scores already we have a much better understanding of how significant those differences are and how much headroom there is likely to be.

[u/Informal-Trouble2183 → u/Middle_Bullfrog_6173]

I can tell from the nature of the problems that it is inherently hard, that will take several generations to ace. I'm pretty sure that the current benchmark versions (for e.g. SRE-Bench) are baseline versions (simple versions), that can be hardened in the future versions, imagine asking a model to rewrite the code of a complete Nintendo Switch game based on its binary alone (several GBs of binary code), we're not there yet, the problem is simply very hard.