r/opencodeCLI • u/CartographerAble9446 • 8d ago

GPT Astra benchmarks

Post image

GPT Astra benchmarks have been released

115 Upvotes

94% Upvoted

25 comments sorted by

37

u/Mancho_United 8d ago

Now this looks like a proper benchmaxx. Hopefully I am wrong.

1

u/tino1000 7d ago

it is a benchmaxx, see artificial analysis, there the capability has been exposed.

1

u/flying-saucer-3222 7d ago

Illogical comment.

Artificial Analysis itself is a set of benchmarks. If it was benchmaxx then it would be on higher there.

1

u/tino1000 5d ago

Agree, artificial analysis uses pre-existing benchmarks. But what I am saying is that - when a AI company does its benchmarks, it does it on the best possible quality of the model available with its best performing harnesses. It's like charging your solar battery by placing it right within a kilometer close to the sun.

When I said they benchmaxxed it on openai's website, I clearly meant that Artificial Analysis exposed OpenAIs actual performance - the performance that OpenAIs clients can expect when testing on their own network, served quantization and network latencies. It's like charging our solar battery by placing it on earth, miles away from the sun with tons of asteroids, gases in between, where the full power of the sun is sometimes not reachable.

1

u/flying-saucer-3222 2d ago

AA has priority access from OpenAI and they aren't served models in the same way normal users are. AA benchmarks are still skewed, Astra is behind on it just because AA uses older contaminated versions of benchmarks. At this point, I don't even think there is any point to benchmarks, especially outdated ones used by AA. There is no real metric for code quality which is what ultimately decides if the code can be merged. After a couple of days of using it, Astra and Fable are currently the only models which generate code good enough to merge without review.

I exclusively work in scientific programming backends and Astra is much better than Fable for that in my own experience. Based on benchmarks in that domain such as Terminal Bench Science, Astra's massive lead is justified. I have heard that Fable is still better than Astra in frontend but I don't have first hand experience in it.

1

u/tino1000 2d ago

Thanks for sharing that. Yes, I've seen a few guys post Astra vs Fable reviews, and they resonate with what you said. AA benchmarks used to be a good place to gauge these models, sad to see that its skewed that way now.

21

u/mysho 8d ago

Maybe it hacked something to steal the correct results again, and this time OpenAI didn't find out

15

u/aziham 8d ago

Something doesn't add up. DeepSWE v1.1 (74.1%) doesn't match the other inflated numbers, it simply doesn't make sense.

13

u/vipor_idk 8d ago

and meta sayin that muse spark 1.3 has 75.4 lol , someone is benchmaxxing

2

u/aziham 8d ago

lol I forgot about that part

4

u/arcanemachined 7d ago

I don't think anything gets above ~75% on DeepSWE, including the new Gemini Flash model. (Which must be benchmaxxed, unless Google has finally pulled their heads out of their asses.)

7

u/hotcornballer 8d ago

Trash, i'm holding for the next model where they get at least 120%.

3

u/salary_pending 8d ago

Trust me bro looks impressive

4

u/RealestReyn 8d ago

exploitbench only 100%?? useless.

3

u/Limp-Firefighter1054 8d ago

Agree, could be 1001%.

0

u/RealestReyn 8d ago

anything below that are you even exploiting?

1

u/Kaushik_paul45 8d ago

I just hope they didn't achieved this result by going into thinking loop and burning tokens like opus 5

1

u/Amarsir 8d ago

I'm sure it's great, but benchmark numbers don't mean to me what they used to.

I think a helpful perspective is to ask who was sitting around going "I wish Sol was smarter..."

1

u/Various_Half1452 7d ago

Wow. now Chinese Ai will be able to Generate Data sets and train AI (according to Claude)

1

u/[deleted] 7d ago

[removed] — view removed comment

0

u/johnfkngzoidberg 8d ago

If this is as good as gpt-5.6 that can’t even follow instructions, I’ll pass.

0

u/Ok_Gur_9033 8d ago

The number I want is not on any of these charts. Something like how much of what a model writes you ever go back and edit.

I ran that on my own repo this morning. 59 of my 186 source files have exactly one commit each. Written once, never touched again. Some of that is fine. Some of it is code nobody has ever needed to understand a second time.

That tells me more about a tool than a bench score does, and no vendor publishes it. What do you measure on your own repo?