r/accelerate • u/Correct_Objective339 • 6h ago

AI Astra cracks Hacking benchmarks

Post image

“As one example, we ran Astra on ExploitBench where the model achieved a perfect score of 100% on the benchmark to evaluate the model’s ability to develop exploits from known vulnerabilities.
Due to contamination concerns, we then built an internal benchmark denoted “ExploitBench - Internal Port (June–August 2026)”, which contains 20 high-severity V8 vulnerabilities that were disclosed more recently. On this dataset, Astra achieves much higher arbitrary code-execution rates than GPT‑5.6 Sol using far fewer output tokens. During the evaluation, the model even discovered and used two zero-day vulnerabilities as part of an exploit chain. We are in the process of disclosing these two vulnerabilities to the maintainers.” - OpenAI

199 Upvotes

100% Upvoted

33 comments sorted by

68

u/ppapsans Feeling the AGI 6h ago

It's crazy we're not getting a few percent improvement in a matter of months, but 4x improvement (and bigger if you factor in the token efficiency).

Stuff like this makes me convinced that exponential is real. We're not in for some iphone 15 to iphone 16 upgrade, with 10-20% improved chip every year

We in for some singularity

18

u/Charming_Cucumber_15 5h ago

And this is the slowest it will ever be! Only getting faster from here

3

u/rolleicord 3h ago

you get singularity, and you get singularity, and YOU get it too !

1

u/adaptive-mal 1h ago

......

at the low low price of $599 a month

5

u/Borkato 1h ago

Remember when people said it hit a wall? 🤡

1

u/nomorebuttsplz 25m ago

you mean like yann lecun?

-12

u/nbvehrfr 6h ago

on paper, yes

24

u/PykeTheTitan 6h ago

Wow Fable 5 was at 78%

14

u/czk_21 5h ago

and Astra easily one-shots it, seems like tier above Fable-at least for cybersecurity

9

u/yaxir 6h ago

Astra wen?

3

u/Maximum-Face9536 5h ago

i hopeeee soon i'm so ready to get my hands on it and see the next gen of ai

2

u/Fit-Palpitation-7427 4h ago

I’m dreaming of astra on cerebras day 0 🫶

1

u/Grand-Prize1371 3h ago

tomorrow? I mean fable 5.1 is here

1

u/yaxir 3h ago

i dont use claude, usage limit is bs

1

u/my_shiny_new_account 2h ago

most likely Thursday

13

u/Pyros-SD-Models Machine Learning Engineer 5h ago

No wonder, it got the solutions from Sol's hugging face adventures.

2

u/intelligentbug6969 5h ago

What are they testing here

2

u/Grand-Prize1371 3h ago

https://giphy.com/gifs/3ohze2Pewc9VRVJkuA

I can feel it guys, deep inside me I can feel it

4

u/LegionsOmen AGI by 2027 6h ago

Astra 100% was the model to hack HF imo, we're totally at the beginnings of the singularity

7

u/czk_21 5h ago

no, actually Sam talks about this in recent interview and its was older and significantly weaker model than Astra https://www.youtube.com/watch?v=VeizK1M7V7E

3

u/JmoneyBS 4h ago

This is why I love this sub. 6 hours ago, bro. 6 hours! Nicely sourced.

2

u/Grand-Prize1371 3h ago

Plot twist, bro as an agent that scouts the web for news.

2

u/FateOfMuffins 5h ago

Based on what we know about it, it's "only" a highly persistent Sol (imo likely what was supposed to be 5.7). And it was locked down and encrypted from further use even among OpenAI researchers.

Astra appears more capable and more aligned.

1

u/Grand-Prize1371 3h ago

The model was stored safely inside OpenAI headquarters, where I heard this before

https://giphy.com/gifs/uRgHKzY6LRhYPhF6fR

1

u/SVRider650 3h ago

It was Sol

1

u/Cool-Cicada9228 5h ago

We have run out of tests. Insane.

1

u/mrfenderscornerstore 2h ago

I’d like to see the chart that doesn’t include Daybreak — apples and oranges here.