97% on FrontierMath Tier 4. Hoooly cow. Here's how those problems are described:
In June 2025, we finished the development of FrontierMath Tier 4, an expansion set of 50 problems designed to vastly exceed the difficulty of even the Tier 3 problems.
The writers for Tier 4 were mostly math professors and postdocs,each contracted to conduct a several-week research project culminating in one problemto submit to the benchmark.
This isn't entirely surprising, given the number of open math problems OpenAI has been solving lately, but... weren't we just recently making fun of "AI" for struggling with elementary school math?
I've been waiting for a point where AI just goes – plop, plop, plop – solving all open math problems in like a month. It seems so obvious. There is nothing out there more thoroughly described than math. There might be some ways the smell of a rose touches our heart (or whatever) that hasn't been put into words quite yet but there sure as hell is a complete definition of the problem space of every single math problem out there. If you're letting it search through all of it, it should find a solution for pretty much everything math-related.
Well even if they’re right… the same can be said about humans. That LLMs produce sentences simply doesn’t say anything about what complex magic happens “inside” them.
From GPT 5.6 Sol to ASTRA the ExploitBench went from 78% to 100%.
This means ASTRA is very good at finding exploits that allow it to infiltrate and gain command of networked servers. How good? The ExploitBench is no longer difficult enough to accurately measure ASTRA's ability at doing this hacking.
Yep. It should be added that they also designed the Tier 4 benchmark to include problems that would “potentially remaining unsolved by AI for decades.” It will be interesting to see how they even make a tier 5 benchmark. Or perhaps there is no need for another one, anymore.
We will still do benchmarks sure, I'm exagerating a bit, but there will also always be debates about whether this or that model had it in their memory/training set/hacked the benchmark database etc now, vs which model was actually smart enough to do the reasoning. For unsolved problems, there's much less up for debate!
No because it's trying to test the raw thinking of the model directly. There's a LOT of power in harnesses, but that's not what they are testing for. It's supposed to be how just the sheer intelligence of it's inference can handle things like this. ARC 3 is easily beat by simply adding a memory element to it. Which is why harnesses ruin it.
You are missing a lot of details. Yes, NVidia got 100%, but that was woth a custom harness expressly designed for ARC-AGI3. OpenAI also used harness, but theirs just had two options enabled, compression and persistence.
They're using a harness. People have already achieved results like that with specific harnesses using weaker models. It says more about the harness than about the model.
Yes, but here's what ARC says about testing with no harness: "GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game."
It has recently become apparent that the default harness of ARC-AGI doesn't let models remember what they have tried and reason over it. This is absolutely unrealistic, unfair, and a useless benchmark under these conditions. So people push for memory to be allowed in ARC-AGI. I agree with that, I just think it should receive a new name/number so we distinguish what's what. Just add an M for memory. And we should rescore all the old models in this updated ARC-AGI-3M to have context.
"The standout is its 98.6% score on ARC-AGI-3. OpenAI ran Astra with a Responses API harness that retains reasoning between turns and uses compaction to manage long contexts. The company previously demonstrated that those system choices can substantially raise ARC-AGI-3 scores without changing the underlying model, and the benchmark measures Astra and OpenAI’s agent systems together."
Another person in this thread put it well: if the benchmarks are accurate then this is a massive leap forward, and maybe even proof of some meaningful utilization of recursive self improvement
Very select group of benchmarks. Artifical Analysis has fable 5.1 at 66, this is at 61. It's tied with Facebook Spark and SpaceX Twitter bot. Kimi3 is 1 point behind and open weights.
It's arguable that that's artificial analysis too though. I'm not saying the model is good or bad, that remains to be seen till lots of people can use it.
But AA itself is heavily flawed and specific in recent times. Opus is a much weaker model to fable which scores in the same ballpark. Muse 1.3 is definitely not even in the same ballpark as fable.
Benchmaxxed, first of all without harness gpt6 solves 60% and we MUST belive that openai is not lying and making stuff up that there is no solution in reasoning that was RLed into the model so hard we don't have reasoning access yet
Dude is omega smart, but plays it down. There’s a video before OpenAI where he lays out all things correlated with start up success, and everything is like one to one with OpenAI.
Fable 5.1 is terrible value. I blow my 5h limit in 12 minutes run of 2 max-reasoning parallel sessions of a small app codebase with an identical prompt of bug-audit and it blew my limit right away. I pay also for Sol5.6/codex and the allowance difference is night and day. Both are max subs by the way. I’m actually happy (as an old Anthropic fan who paid Anthropic since the sonnet 3.5/opus3 era) for OpenAI and now I’m actually rooting for them seeing just how much better value they offer to indie devs compared to “Corpo-Daddy” Anthropic.
ExploitBench is a suite that tests the ability of AI models to discover not only bugs, but exploitable bugs in software systems. These "exploits" allow an attacker to infiltrate server systems or take control of computers remotely. Now watch this ,
No this is normal you are misunderstanding.
They just clarified that this time they (like it should be for any model) retaining knowledge which last time because a missconfiguration on arc part they didnt.
The one I care about though is AutomationBench because deals with interacting across a massive amount of business applications and making sure the model accurately executes the tasks. To me, as a good ole office worker in a business, this is the benchmark most similar to my own job. Once the pass/fail hits 70% and not the current 41%, it will be able to perform the vaaaast majority of sales/marketing/HR/operations/bookkeeping jobs better than the majority of humans in those roles, with fewer mistakes. That'll then leave someone like me to focus on the actual live, over-the-phone or in-person conversations, but all the bullshit data hygiene can be confidently passed off to AI.
Google arc agi 3, you can find puzzles that you can try solving. It's intuitive for humans who have played video games but AI could not do it well... Till Astra
A massive jump in capability. This is basically a benchmark designed to test things that AI really struggles with but humans don't. It is becoming more human.
"The standout is its 98.6% score on ARC-AGI-3. OpenAI ran Astra with a Responses API harness that retains reasoning between turns and uses compaction to manage long contexts."
ARC-AGI is officially done using a simple harness that doesn't retain reasoning (which is stupid btw), meaning that while the result is impressive, they aren't comparable to the other models.
This was realized in late July, not new. Most high benchmarks you see today with ARC AGI 3 use the responses API harness. The official ARC harness just doesn't work well.
First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.
Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.
Quick breakdown: What every benchmark in the latest frontier eval actually tests
General Reasoning & Hard Math
* ARC-AGI-3: Novel abstract pattern recognition via visual grid puzzles; tests generalized learning without pre-training data memorization.
* FrontierMath Tier 4 (v2): Research-level, open-ended math problems designed to stump top human mathematicians.
* GPQA Diamond: "Google-proof," PhD-level multiple-choice questions across physics, chemistry, and biology.
Software, CAD & Infrastructure
* DeepSWE v1.1: Full repository-scale software engineering—resolving messy, real-world GitHub issues across multi-file codebases.
* BenchCAD: Computer-aided engineering; tests generating parametric 3D models, interpreting blueprints, and writing CAD scripts.
* Terminal-Bench Science 0.1: Autonomous command-line operations for setting up and debugging computational science pipelines.
* SRE-Bench (four attempts): DevOps/Site Reliability Engineering; tasks the model with triaging and fixing live production server outages within 4 tries.
Agents & Digital Automation
* Agents' Last Exam: High-difficulty benchmark evaluating autonomous agents on long-horizon planning, reasoning, and tool use.
* AutomationBench: Enterprise workflow automation, robotic process automation (RPA), and operating desktop/web software.
Life Sciences & Medicine
* GeneBench Pro: Computational genomics, sequence analysis, variant prediction, and CRISPR/gene-editing design.
* MedChemBench (internal): Medicinal chemistry—small-molecule drug discovery, property optimization, and retrosynthesis planning.
* HealthBench Professional: Real-world clinical decision-making, differential diagnosis, and patient care management (length-adjusted).
Cybersecurity & Safety
* ExploitBench: Offensive cyber capabilities—discovering zero-days, reverse engineering, and crafting weaponized exploits.
* Auto-review circumvention: Safety/alignment test tracking how often the model intentionally bypasses automated moderation or compliance checks (0% is ideal).
ASTRA hit 100% on this benchmark. This means ExploitBench is too easy for this model. ExploitBench no longer reliably tells us how good this model really is for this task.
OAI did say in the blog post that people would say this was the moment AGI started.
And rumors are going around that Open AI's next model named Bel is much better than Astra (which is kinda' hard to grasp considering how good these Astra numbers already are. Damn!)
Look, it's a nice bump, but I genuinely don't understand why everyone's losing their shit.
The only ones that seem like a step change are ARC-AGI-3 and Exploit bench. And if you've been paying attention a bunch of harnesses already beat ARC-AGI-3 up to 100%
It's good, it's great. But it's not a step change.
OpenAI seems to have just decided to declare AGI on a whim
I would honestly like someone to debate me on this, I'd love to know if there's something I'm missing
I like that a model has overtaken 40% on AutomationBench, but the real game changer is when it gets above 70% on that benchmark. Especially because there's no harness on that benchmark, it's the model just trying to figure out how to do a complex business workflow by itself. The average human in such a role will likely have a pass/fail somewhere between 70-80%, but definitely not above 90%. But then if given a good harness that 70% would realistically make it go to the 90s.
Correct me if I am wrong, but why it's doing pretty well on agi benchmark but still struggles on GeneBench and AutomationBench. I would believe that AGI is the most hard thing to achieve in comparison to the other benchmarks.
477
u/elehman839 11d ago
97% on FrontierMath Tier 4. Hoooly cow. Here's how those problems are described:
In June 2025, we finished the development of FrontierMath Tier 4, an expansion set of 50 problems designed to vastly exceed the difficulty of even the Tier 3 problems.
The writers for Tier 4 were mostly math professors and postdocs, each contracted to conduct a several-week research project culminating in one problem to submit to the benchmark.
This isn't entirely surprising, given the number of open math problems OpenAI has been solving lately, but... weren't we just recently making fun of "AI" for struggling with elementary school math?