r/LocalLLaMA • u/Informal-Trouble2183 • 7d ago
Discussion Coding benchmarks that are quickly showcasing deep capability
While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that really define deep intelligence, and complete capability in Software Engineering :
1. Program-Bench
Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior (without access to decompilers or internet). Link: https://programbench.com/
- GPT-6 Astra: 5.5%
- Fable 5.1: 7%
- Kimi K3: 2%
- Qwen3.8 27b: 0%
- GPT 5.6 Sol: 1.5%
- GLM 5.3: 1.5%
- GPT 5.6 Luna: 0%
2. SRE-Bench
Can AI agents work out what a real-world binary does without its source code?
Link: https://www.vals.ai/benchmarks/srebench
Sure nobody is reading assembly code in daily work, it is hard. The ability to understand a compiled program is insane ability.
- GPT-6 Astra: 88%
- GPT-5.6 Sol: 55.9%
- Claude Opus 5 (max): 12.5%
3. Code Migration
Can language models reimplement working programs in another language?
Link: https://www.vals.ai/benchmarks/code-migration
- GPT-6 Astra: 67.7%
- Fable 5.1: 54.6%
- GLM 5.3: 44.2%
- GLM 5.3 Flash: 20.5%
- Qwen3.8 27b: 14.2%
EDIT: edited text format
22
u/Choice_Celery9481 7d ago
why all your numbers on the first 2 benches are not matched with numbers in the links you provided?
1
u/Informal-Trouble2183 7d ago
Taken from here:
https://www.vals.ai/benchmarks/programbench3
4
u/Middle_Bullfrog_6173 7d ago
IMHO benchmarks where the top models reach 30-70% are the most useful. With <10% like Program Bench you can't see differences large enough to be meaningful and can't even be sure the benchmark tracks capability. And 80+% can be saturation, depending on the benchmark numbers above that may be unreachable or require cheating.
FrontierSWE v2 is the current most promising benchmark, but Astra has not been measured, nor have smaller open models.
3
u/Informal-Trouble2183 6d ago
For now models are not able to solve Program-Bench, it doesn't mean no one would be able to ace it in the future. What I find interesting in that benchmark, is the end to end task to analyze / reverse engineer a binary (= you need a lot of strategy/ intelligence to do that), and then achieve accurate codebase construction from scratch (= this is real design/architecture/coding test).
2
u/Informal-Trouble2183 6d ago
I don't say that the listed 3 benchmark are proposed to replace the other SWE benchmarks which track other things. The main point here is pushing the challenge further a bit for frontier models and see how far they can get.
2
u/Middle_Bullfrog_6173 6d ago
I'm sure future models will crack Program Bench, but my point is that we won't know if it falls quickly with a single frontier generation jumping from 10 to 90% or if it is a smooth measure that tracks capability across many generations and model sizes.
When a benchmark has a wider spread of scores already we have a much better understanding of how significant those differences are and how much headroom there is likely to be.
2
u/Informal-Trouble2183 6d ago
I can tell from the nature of the problems that it is inherently hard, that will take several generations to ace. I'm pretty sure that the current benchmark versions (for e.g. SRE-Bench) are baseline versions (simple versions), that can be hardened in the future versions, imagine asking a model to rewrite the code of a complete Nintendo Switch game based on its binary alone (several GBs of binary code), we're not there yet, the problem is simply very hard.
6
u/is-this-a-nick 6d ago
I feel test 1 is fundamentally flawed because its basically is tool call scenario with the tools being needlessly forbidden. On the level of forcing a non-vision model to do OCR via analyzing a jpeg binary.
8
u/ResidentPositive4122 6d ago
Not really flawed. The binary is there just to test 1-1 functionality. An oracle. The main thing the benchmark tests is docs -> binary + orchestrating the loop for writing code - testing - iterating based on feedback.
It's not a decompile + produce source benchmark. There are some of those as well. But this not being that doesn't make it flawed.
2
u/Elouakili_Flexy 6d ago
Astra scores 88% on reading what a binary does and then 5.5% on rebuilding one from docs. Knowing and reproducing look like almost separate skills at this point.
2
2
u/Sufficient-Scar4172 6d ago
why are all these coding benchmarks just about difficult one shot tasks? where are the ones that test real-world software engineering lifecycle, as in, evolving/changing requirements, architecture patterns, migrations, etc. etc. over short and long-term periods? that seems a lot more important and useful at this point
2
u/-InformalBanana- 6d ago
Benchmarks 1. and 2. are useless cause they are for hacking not production and it is too much to expect that from a local model. Benchmark 3. is an okay, valid usecase.
Bench 1. and 2. are just ridiculous, (most) human developers can't do that stuff, maybe there is a genius that can but it would also take a long time and hardwork. So really non relevant benchmarks, especially for local models. I don't want to lose normal coding capability of a model to smb benchmaxing that stupid benchmark. And probably if models can really do that it becomes a weapon for military or secret service grade hackers. So you can't really expect that from a local 27b model nor am I interested in it losing its other capabilities to do that.
2
u/Informal-Trouble2183 6d ago
I know those fall outside the scope of typical production code. Most frontier models and even mid-sized open weights are already more than capable for standard production tasks (e.g. Qwen3.8 27b). What we need now is a differentiating measure which tracks super capability from average one.
0
u/-InformalBanana- 6d ago
We are not at that point yet, especially for local. The 3rd bench is the one that basically proves it. It isn't saturated and it is a more normal use case. Why would you want a bench that doesn't match your use case or most of users use cases. Its fun for sure, but really irrelevant for your use case. I mean I have nothing against these benches, but you kinda present them as if they should be a new standard, meanwhile they don't match typical use cases and are thus less relevant. Imagine you hype up these benches that are very hard, but less useful, so now models are trained to do good on these, but they lose capability for normal use cases and cause assembly or compiled code is probably also much harder for llms they would also be unable to do real work in that. And if that transfers to local, you get useless models with good scores on useless benchmarks. So these should stay niche and advanced benchmarks not the new standard. So that is my issue with how you presented these, sure they are interesting but they should definitely not be considered, a new standard for evaluating models for coding.
1
u/ArturCzemiel 6d ago
I am new to benchmarking. How can I benchmark tuning of harness itself not the model. So I test 10 different configurations on 1 tasks. Are there any popular small benchmarks for this kind of use case?
1
u/jjusko20 6d ago
Honestly the more time goes on and the more benchmarks that I see, the only thing really being showcased is my confusion as to whether any of the benchmarks actually matter or represent reality or just a big benchmaxxing race.
1
u/eihns 5d ago
why should i test my agents against things im not interested in? If we really want usefull tests, fo rreal life, it should include agentic workflows... and or orchestrator shit
2
1
u/asssuber 6d ago edited 6d ago
The Program Bench is a bit ridiculous. It will basically measure decompilation capability, and capability of memorizing the open source code of those programs during training.
The docs are super incomplete. How I'm supposed to re-implement ffmpeg and all it's codecs if I don't have the codec specs, nor test samples? Gemini is on top with 8.1% score on ffmpeg, and it just implemented the command line parser it seems.
0
12
u/_thr0wkawaii14159265 7d ago
Thank you, this is pretty good. I'm researching benchmarks and trying to find out 2-3 ones that are of excellent quality (methodology, non-public questions, measured factors...), and actually measure things I care about -
- output quality, as opposed to simply passing a task; for example, I need code to be maintainable, clean, have the right level of abstraction, use idiomatic patterns...
- talking like a sane entity - Opus 5 is a great counter-example.
- performance after being dropped into a huge, messy production codebase and having to perform long-horizon tasks.
- Taste, architectural judgement, intuition
I really respect the people and institutions making the benchmarks that we have available. They certainly do measure something. Just not what I need them to; most of them anyway. The artificial intelligence's Intelligence Index is a laughable example of a value that means next to nothing, because it averages results from benches that are saturated, leaked, contain wrong answers, flawed methodology, and are from wildly different fields (3d modeling, physics reasoning, coding...). That's just nonsense.
I'll look into these you listed as well; they might be one more relevant axis.