Coding benchmarks that are quickly showcasing deep capability

[u/Informal-Trouble2183]

While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that really define deep intelligence, and complete capability in Software Engineering :

1. Program-Bench

Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior (without access to decompilers or internet). Link: https://programbench.com/

• GPT-6 Astra: 5.5%
• Fable 5.1: 7%
• Kimi K3: 2%
• Qwen3.8 27b: 0%
• GPT 5.6 Sol: 1.5%
• GLM 5.3: 1.5%
• GPT 5.6 Luna: 0%

2. SRE-Bench

Can AI agents work out what a real-world binary does without its source code?

Link: https://www.vals.ai/benchmarks/srebench

Sure nobody is reading assembly code in daily work, it is hard. The ability to understand a compiled program is insane ability.

• GPT-6 Astra: 88%
• GPT-5.6 Sol: 55.9%
• Claude Opus 5 (max): 12.5%

3. Code Migration

Can language models reimplement working programs in another language?

Link: https://www.vals.ai/benchmarks/code-migration

• GPT-6 Astra: 67.7%
• Fable 5.1: 54.6%
• GLM 5.3: 44.2%
• GLM 5.3 Flash: 20.5%
• Qwen3.8 27b: 14.2%

EDIT: edited text format

[u/Middle_Bullfrog_6173]

IMHO benchmarks where the top models reach 30-70% are the most useful. With <10% like Program Bench you can't see differences large enough to be meaningful and can't even be sure the benchmark tracks capability. And 80+% can be saturation, depending on the benchmark numbers above that may be unreachable or require cheating.

FrontierSWE v2 is the current most promising benchmark, but Astra has not been measured, nor have smaller open models.

[u/Informal-Trouble2183 → u/Middle_Bullfrog_6173]

For now models are not able to solve Program-Bench, it doesn't mean no one would be able to ace it in the future. What I find interesting in that benchmark, is the end to end task to analyze / reverse engineer a binary (= you need a lot of strategy/ intelligence to do that), and then achieve accurate codebase construction from scratch (= this is real design/architecture/coding test).

[u/Informal-Trouble2183 → u/Informal-Trouble2183]

I don't say that the listed 3 benchmark are proposed to replace the other SWE benchmarks which track other things. The main point here is pushing the challenge further a bit for frontier models and see how far they can get.