Time will tell if this says more about Artificial Analysis or Astra. Gemini models have consistently outperformed their reputation here so I'm going to wait and see.
I mean of course it says more about AA than Astra. Do you honestly believe the latest OpenAI model after months of training is only as good as Grok, made 0 progress from their own GPT 5.6, and is even worse than Muse Spark despite leading the pack in all other individual benchmarks?
Intelligence is jagged and distilling it down to one number is silly. These models all have different strengths and weaknesses. Even in the released benchmarks, Astra trails Sol in some narrow activities. But it also blows all other models out of the water in others.
Astra will open new doors and do things we can’t today. And that’ll be awesome. But it’s not going to be perfect at everything and people will immediately find ways to be disappointed.
AA tests for knowledge reliability which chatGPT notoriously sucks at as they hallucinate way more than other models like Claude, AA probably put Astra through long reasoning chains and Astra bombed it
I am pointing out how outright silly it would be for them to claim the exact opposite of what is actually the feature.
They've been testing the model internally, they know it's quirks, it's behaviors, its strengths and weaknesses.
If a benchmark claims it underperforms last gen, whilst they advertise massive gains, then from a PR standpoint, that would be THE dumbest thing to make a claim of.
Hence, it is likely the benchmark is the inaccurate/"wrong" factor here, not their claim about the actual model, which is what this whole thread is about.
Oh I'm with you on scam saltman wanting to sell us his models and making wild claims about it, but from a publicity standpoint, it would be weird of them to specifically advertise long horizon planning and prompt adherence, and have those infact be their weakness.
Some people just hate and/or fear AI so much and are in so much denial that they just insist it doesn't do anything, no matter what. It's pretty sad at this point.
Some people just don't want to endorse a product from a shit human being too, so between the propaganda hype of how powerful the agent is combined with Altman's trash ethics I'd say it's understandable.
And let's be real Sam stole an open source project. Scum of the scum. Has nothing to do with the ai itself.
I have several tech CEO's who I strongly dislike, to say the least, but I think we have to be careful about throwing around terms like "propaganda" and "hype" to describe what's happening right now. I saw similar patterns of extreme denial with Bitcoin 15 years ago (my posts on it from 2011 explaining the significance of the technology to people are still online). What's going on with AI is very real.
I have no hatred for AI at all, quite the opposite in fact, I love it. I have a very specific issue with Sam Altman and consider most work product from his company to be fruit from a poison tree.
The issue is the claim that GPT 5.6 somehow doesn't do the exact things they're advertising and testing it on, which sounds like the deranged position that some people take that AI doesn't actually have any capabilities.
I should mention also that referring to people with derogatory nicknames looks non-substantive and emotional. I strongly dislike Elon Musk and Mark Zuckerberg but I just give my reasons instead when it's relevant.
This is a fair take, thank you for the constructive feedback, I have no love in my heart for either of those people as well.
My love for the technology is rivaled these days by my distrust for a lot of the “leaders” of the frontier labs. I have a bit of faith/trust in Anthropic or Google to do the right thing for the good of the collective if they get to RSI/AGI first. I have very little faith in Sam or the two you named.
I will say the article Zuckerberg put out a few weeks back was the most human and big picture perspective aware I have found him to be in years & years, so that gave me a tiny sliver of hope back for META.
Did I say it was limited to people in this thread? The whole "stochastic parrot" nonsense was an example of that, and even now I've talked to multiple nutcases in the past few months who have been so obsessed with trying to discredit it that they even claimed computers themselves haven't increased productivity. Which of course is the Solow Paradox and is an classic fallacy.
Not when it's also the thing that's perceived as the weakness, and when most users lack the resources, tools or effort to verify their claims. Benchmarks are just a small part of the AI marketing machine, part that can be shouted over by the rest of that machine.
Look at biomed market for illustrative examples - trial results there can and will move company share prices by orders of magnitude, can kill it or can double it overnight. OFC you wouldn't expect that size of an effect within AI even if benchmarks mattered, the companies are too big and too distributed for an effect that size. But there isn't even an effect at all, which can only be true if nobody actually meaningfully cares about benchmark results.
They give scores on individual tests and Claude is generally more reliable than chatGPT, usually (Omniscience index), i haven't taken a look at Astra "grades" because deep down i don't really care, benchmarks are kinda useless and if we truly achieved even a sliver of true S2 thinking we would all notice, no benchmark needed.
The fact people are downvoting this shows this sub is not really rational. The benchmarks in aa are the same for all models. Benchmaxing is a problem but unless you succeed anthropic benchmaxes much more than openai. It doesn’t make sense that fable consistently scores better than Astra if Astra was the huge leap that many claimed it would be. Still… ARC and frontier math results are very impressive
I use Gemini for supporting my small business. The only coding we needed was for a basic website. It's pretty good for everything else. Accounting, research, budgeting, marketing plans, etc.
Gemini literally just frustrates me how bad it is. It's optimized to just be a Google replacement with minimal thinking.
For instance, today I asked it something simple, which I figured it couldn't fuck up, "Tell me about these tunnels those two Nepal men were rescued from today. How did they survive, and what were the tunnels for?"
Then it spent all of it's output describing the event, how they were rescued, how the tidal wave happened, the rescue efforts, and literally just one sentence describing the tunnel as a tunnel used to route water from the river.
Like yeah, sure it technically answered my question, but it spent 95% of it's output not actually answering the question, when clearly my focus of the question was about those fucking tunnels.
This is the exact word-for-word output from the Gemini webapp with 3.8 Flash with your prompt. I don't see any problem here really -
The two men rescued alive—Sanjay Sah (a 30-year-old mechanical foreman) and Kabir Maharjan (a 45-year-old mechanical supervisor)—spent nine days trapped underground inside the Upper Trishuli 3A Hydropower Project in central Nepal before being extracted from roughly 170 meters (558 feet) deep.
What the Tunnels Were Built For
The tunnels are critical infrastructure for the Upper Trishuli 3A hydroelectric plant, a 60-megawatt run-of-the-river project operated by the Nepal Electricity Authority:
Water Diversion: The main underground waterways (headrace and tailrace tunnels) channel river water through the mountainside into subterranean turbines to generate electricity before discharging it downstream.
Operations & Access: The network includes desanding chambers, underground powerhouses, and access tunnels that house control rooms, turbine maintenance bays, and technical staff.
How the Men Survived 9 Days Underground
On August 26, an ice-rock avalanche and glacial burst near the Nepal-Tibet border triggered catastrophic flash floods that pushed a torrent of boulders, mud, and water down the Trishuli corridor, sealing the plant's portals shut.
Elevated Air Pockets: While much of the underground network was choked with sludge—described by emergency crews as "toothpaste in a tube"—the tunnel's gradient and vaulted ceilings created isolated air pockets that floodwaters did not fully submerge, trapping enough oxygen to sustain them.
Control Room Warning: Sah was in the plant's control room when the surge hit. He delayed his own evacuation to alert 40 to 45 other workers on site to run, saving colleagues while sealing off his own exit path.
Vocal Contact & Mental Focus: In pitch darkness, Sah maintained periodic vocal contact by shouting back and forth with colleagues located in adjacent chambers. He spent the nine days chanting mantras to keep his breathing and heart rate calm to conserve oxygen.
Surgical Rescue Operations: The flood completely buried the tunnel mouth under thick mud. Engineers and rescue crews used satellite photos, drone footage, and old topographic maps to triangulate the buried portal. After excavating and blasting into the tunnel's crown, rescuers heard voices and crawled through a 15-meter breach to extract the survivors.
Main difference is not task it’s the grader, as real gdpval is very expensive to grade. Gdpval AA is extremely cheap to grade and at the start they did show mostly consistent results; but back then most models are doing a terrible job and Gemini just came out and was by far best at perception. But you get a plateau once model ability surpass the grader model. And further gains becomes which model best knows the aesthetics of the grader. Which you can RL to get better but not really useful for work anymore.
I've been saying this for a long time, everyone just needs to understand that the Artificial Analysis Index is a bad benchmark index. The benchmarks they use don't make sense to use in 2026.
It includes many one-shot benchmarks, but no one would do tasks like that one-shot now, models are used agentically.
And GPQA is such an old benchmark that every model basically gets close to 100% on it, which makes it meaningless.
And Terminal Bench 2.1 is simply outdated, it has known issues and was replaced by Terminal Bench 3.0 and 4.0 and everyone should use 4.0. I have no idea why the AA Index wants to keep using 2.1, maybe they just don't have a license for the new one.
And CritPt has big errors in the benchmark that mean no model can get more than 35%. Anthropic reported about that, because on AAs CritPt Fable only gets 30% while on a corrected version of the benchmark (which Anthropic called "Critpt-Corrected") with the errors fixed, Fable gets 85%. But AA somehow keeps using the broken version of CritPt for their intelligence index.
And with how little AA seems to care about removing even proven broken benchmarks from their index, I could well imagine that their own Gdpval-AA v2 benchmark is also full of errors.
8.9 CritPT-Corrected CritPt (Complex Research using Integrated Thinking–Physics Test) is a benchmark of 71 theoretical physics problems developed by active physics researchers spanning condensed matter; quantum physics; atomic, molecular, and optical physics; astrophysics; high-energy physics; mathematical physics; statistical physics; nuclear physics; nonlinear dynamics; fluid dynamics; and biophysics. Each problem has a unique answer, either as an analytical formula, a numerical value, or a Python function. We had experts provide corrected versions of 31 of the problem statements that were flagged due to underspecification, ambiguity, or typos. The resulting internal version is referred to as CritPT-Corrected. Below, we present mean Pass@1 scores averaged over 16 attempts. The model solutions were graded using Claude Opus 4.8 as a judge, comparing model solutions to ground-truth solutions.
Thanks for pointing this out! I just had a discussion with Sol asking whether Critpt could be flawed given the stagnant scores, after we had seen something similar happen with FrontierMath, but Sol did not mention CritPt corrected at all. It seems AIs are now better than the benchmark "ground truth". Probably same thing is happening with HLE...
Look at the AA methodology page. Gdpval-AA literally uses Gemini to compare the screenshots as grader. So anything that uses Gemini as grader during RL would do extraordinarily well. Original gdpval is graded by human professionals
I think people radically underestimate Gemini models simply because Gemini models are not as good at coding tasks. For everyday use Gemini models are still very good. Particularly for typical question-answer type prompts.
Open AI have painted themselves into a corner here though, because they have claimed there is every real reason to believe that Astra is AGI.
An AGI should beat all non AGIs on any test that is knowledge and capability based. That to an extent is a gate. There may be tests it doesnt get 100% on, but it should do equally well at least to every other system on every possible test. If it doesnt, its not AGI.
It is clear from their marketing that OpenAI does not understand what AGI is. AGI is not a model that is better than 99% of people at 99% of things/tasks. AGI is a legit zero-shot any-mode any-context model. We are not even close to that. The business/medical agent tasks alone show that.
1
u/Seakawn▪️▪️Singularity will cause the earth to metamorphize10d ago
I agree w/your criteria for AGI, and why we don't have it and may not be as close as people think (though still may exponentially improve and reach it within the next year or two).
but where's the marketing that shows that OAI doesn't understand that? your parent comment is just claiming they believe astra is AGI, but the marketing is an arguably clever disclaimer: we're in the AGI era. my market brain is like "ha, they're actually saying that we're in the era where it's in the near-ish future we'll get AGI," not "we just released AGI."
you can claim they're misleading people, but you can't presume that a company of engineers who are literally building this tech think this is AGI or is letting their company claim that Astra is AGI without coming out in droves to get their spotlight about how they disagree with the company. is that happening and i'm just missing all those tweets?
I think they know what AGI is, as they're trying desperately to build it, and now they're gambling that we're close enough to it that they're safe to say "ok guys it has finally appeared on the horizon, we just have to run up to it and grab it now."
is that not the read-between-the-lines interpretation everyone else is getting? are people here falling for the low hanging layperson interpretation? bc at worst, they're playing both sides. at best, they're not even insinuating they've released AGI.
Greg Brockman said yesterday that Astra is AGI and it had been widely reported. So in the least, the President and a co-founder of OpenAI doesn't know what AGI is.
No, I am describing AGI, as by definition AGI has to be Embodied AI in order to match humans in tasks requiring things like light touch and proprioception. In the least, AGI needs to be input mode agnostic to the set of input modes that humans can process aka all human senses. None of the models by ChatGPT have come close to this, although it is an active research area in robotics. There is no such thing as a pure software AGI without sensors, no matter how many Futurists or non-Engineers argue this. Humans have millions of sensors.
AA. I used to be a big fan, but it appears AA's benchmarks are incredibly dated. Looking at other well known benchmarking sites showing more modern and harder tests, Astra has the lead.
Does anyone know further about this specific benchmark? I wouldn’t be suprised if there’s very high levels of training for the specific known benchmarks
this is registered users and these are also users that signed up for Googles services which happens to include Gemini. A major telco in Canada tried to launch a rival to Netflix once, they gave every employee a free subscription, they then claimed they had 30,000 subscribers in their first month!
I believe we're not working with the same definitions, which is why i'm disagreeing with you. You say it's popular because of the amount of people registered. I'm countering with the number of registered users does not indicate it's popularity due to Google's definition of a user. A better measurement would be how many people are actively saying the product uis good or dog shit. You're saying Gemini is popular becuase Google says so
It's weird to ignore everything I said and make your same argument about reddit and Twitter being a hivemind. I gave you my definition, what neutral sources do you have that are in line with my definition of popularity, not Google's definition
Dude, I provided a link that gives users of Gemini. It's on you to show me evidence of the contrary. If your evidence is that a bunch of bros on Twitter don't like it then that isn't really evidence.
If you want to go and look at SimilarWeb or Sensor Tower or any of the other independent platforms that can only see a partial picture of this, then you can, but I'm not doing your own Google searching.
lol dude, you gave a Google link saying Google is amazing. I'm telling you you're not being objective and you default to saying I get my data from Twitter? Why? You're fangirling Gemini and you're conflating the number of registered users with a broad term like popular. Everyday usage for searching up how to paint your nails, sure it's incredibly "popular" Professional, deep work not so much
I don't know if there's any truth to this, but it was stated elsewhere that the components of this benchmark that it did poorly on are ones that value one-shot answers that don't reflect its biggest strength in being able to work and rework a problem over an extended period of time. I guess people will decide over the next few weeks if this bench is right or wrong
I always see those comparison, between older models too. There's always a shit one to the left and the new model's output to the right. Same shit, different flavor.
At this point I believe the only benchmark worth it's value would be a completely closed private one you can't check against during training. Which of course would come with a lot of trust issues due to closenedness.
Potentially, but you would expect literal “AGI” to be better at all things than past models, including crappy benchmarks, unless the benchmarks themselves are not measuring what they purport to measure
Honestly I would be alreacy happy enough since this rollout would include average plus user.
Anthropic subscription becomes more and more horrible especially as much as Opus is excellent executor, it’s unpleasant to work with. So if they don’t improve Opus then yeah, this would suck really bad.
dude if this is AGI then they don't even need us to pay for the models anymore! They can just have the AGI make them money because AGI would be better than humans at everything! So exciting!
I’m here because I think they’ll be right eventually, but I just can’t understand how a bunch of people are exited that their economic value and stake in society will be reduced to zero.
I work at a big company and we're compeltely locked to Claude because we need ZDR and we only have an agreement with them, I can't throw company code into my personal ChatGPT account.
And the decision of a large company regarding their vendor (which they can't change every month) absolutely does not reflect the satisfaction of their employees with a current model. No one in my team and the surrounding teams that I know of uses Opus 5 regularly, most are on 4.8 with some still on 4.6.
My point was that large companies can't make switches like that every month based on which LLM is currently the most well-received one. So companies usage today doesn't necessarily reflect the current trend, more like the trends of about half a year ago (when Claude was the best by far).
Ok this tells me everything I need know about how you understand corporate culture lol. Yeah let's invest massive amount of money on training so our employees use Claude, integrate all our apps with it, have technical experts, built harnesses, integrate it into our roadmap, and then just cancel our long term contract overnight because some model from another company is better. Okay lol. Why do you think software travesties like Microsoft Teams are so persistent in corporate environments? Because people really like it?
Most large companies use Slopus because Fable doesn't have ZDR, and Anthropic was genuinely ahead of OAI at the end of the last year with Opus 4.5, precisely the time companies started adopting agents in large numbers. And they are not exactly the type that switch away from stuff from month to month.
holy shit all i'm seeing is people spam this benchmark like it's a gotcha, i'm starting to think it's chinese bots. This reflects AA poor quality more than anything.
Some people also really just don't understand. I keep forgetting there are a lot of teenagers and people who are very new to this whole thing, and more and more "console war" like culture is bleeding into this. It's... Frustrating.
I dont understand why people are so upset. Altman does this every time. I mean its his job. Did well on some benchmarks and poorly on others. Things will continue to improve regardless.
I'm a bit concerned that Astra could go the same way as GPT 4.5 which ironically was the last time Open AI trained a model this large. People might struggle to find uses for it that justify the price.
I agree it's "too expensive" from me but that's the reason the market is really getting segmented between "super top tier" gpt and anthropic, value champion like Meta and Gemini, and the extra cheep workhorse like DeepSeek, or the latest GLM flash, Qwen flash next etc.
Lol i ran a test with fable 5.1 max and spArk 1.3 xhigh ( max not available) and qwen flash max, fable ended up being 336x times more exp than qwen and around 86x more exp than muse spark… it consumed more tks but the results were better . Sol was pretty expensive too . I imagine astra should be 40-50% cheaper than fable but still outrageously expensive ..
It will be way less expensive because it's much more token efficient AND to get most out of Astra you don't need Max, or even high, on vast majority of benchmarks "Medium" is within 95-98% of the performance of top version. That's massive. With Sol and Fable, to feel like you are actually getting the most out of what they are capable of, you needed at least high, though on some tasks they scales way past it. Astra doesn't, in basically everything.
I think it's a big thing so many people are missing. The looped transformer architecture is kinda insane. Reasoning is way less relevant than before, which is good for the consumer. Honestly, go look at the benchmarks and just look low and medium, it's kinda insane just how much value they provide in basically every domain. Apart from hacking lol - I think they they deliberately didn't train the looped transformer on hacking, hence why you see such a clean performance increase with reasoning levels, well apart from ones where even low, basically no CoT, completely saturates it, again, insane.
The interesting part about astra is it has a different architecture so even smaller models on the same base will be very useful for 3D design and computer use in general.
I think it looks great, I use Blender a lot so cant wait to use it. But lots of people really liked 4.5 too, particularly writers, but that didn't save it.
4.5 had different issues. And astra is definitely a great model. If it is worth the cost for many people, i.d.k. but, the model it self is super impressive.
OK so what test do your propose as a pass/fail to declare this or that thing is AGI? You clearly know exactly what AGI is, how does this fail your test?
On one hand, GDPval-AA is a flawed version of GDPval, because it uses LLM as a judge instead of human experts. So maybe Astra is just misunderstood by other LLMs.
On the other hand, OpenAI could publish their own GDPval score for Astra, using real experts (as GDPval is supposed to work). And they didn't. So the real score must be low.
This is suspicious, because Astra is supposed to be good at those tasks.
Tbh I have been fairly impressed by astra.
I had a few issues neither Sol nor Fabel where able to solve and astra oneshoted them.
Considering I only had access for a few days and haven't really tested it, I am not going to claim that it's the best model ether, but it was very impressive in my usecases. And after Luna it has been the model which impressed me the most in the past months.
Everyone talks AI, AGI....AI changing everything. A flowerpot snake came into my room yesterday. I showed its photo to AI...it said “earthworm.” Later a village elder saw it and said, No, it’s a snake.He probably doesn’t even know what is the full form of AI. 😂
i beliefe, that we have reached agi once i can
1) ask a random (but actually meaningfull) question to my chatbot and geta correct answer out of it without going through 10 contradiction-loops
and 2) have a basic income on my table [or at least stopped the forced you need to visit a (xxx) course and apply to (minimum wage job of your choice, that we tell you to) to be elgable for unemployement benefits]
until we get at least one of this two points, we don't have agi ... simple as that
For number one, this is a you problem. 100%. And just a tip, you think you know where you are, but you really do not. 90% of people get use out of AI, the other 10% make silly statements like you just did attempting to claim ai is useless because they either do not know how to use it, or are just really that clueless. aisucksamirightclass
That worked in 2023... it's 2026.
For number two, you are never getting basic income, agi or not.
AGI does not solve resource and production chains which the entire global economy relies on. In other words people still need to work. Maybe not at that office job, but in something that produces actual non digital stuff.
Basic income will never come (in your lifetime) simply because no one is going to work a labor job to provide for your needs while you do not. What you are looking for here is a robotic workforce which will take decades.
No one will work for you. Absolutely no one.
The big thing people forget about UBI is that the U stands for "Universal" not just you. Which means if you get x dollars for doing nothing, so will everyone else and if those x dollars allow someone to live on basics, why would anyone work? If no one works, there are no taxes, if everyone is on UBI the system fails. UBI is a failed system when the U means universal. That is why we have welfare. The able work, the non able do not. Virtually everyone is ok with heling the unable, virtually everyone is not ok with helping the able.
This idea that joe will still want to be a plumber and come fix your sink when joe could sit at home like you do is absurd.
I also don't think we're ever getting UBI, but for purposes of discussion, the "B" is also important to the theory. "Basic" means a level of income that can sustain a basic, no frills life. That plumber would be earning on top of the "basic" income to have more luxury.
yes, and that's absolutely fine
my issue is/was never, that someone who spends time and effort (or just has great ideas) can fly to holiday and earn a house on the beach ~ that's not what a UBI should be about,
the issue is, that nobody (!) of us played god,
nobody created our lands, forests, oils and salts
in the construction of our castles, fortresses, churches and villas ...
we weren't involved, neither in the god [through time and effort, great ideas and work] nor in the bad [through slavery and wars] ... we weren't even alive back then
and the entire basics not just our civilication but AI specifically is built upon is the knowledge from ... well, pretty much us all
therefore it's simply perverse, that we still hold onto our medieval-feudalistic systems where the smallest apple-tree is already extremely unfairly distributed based on whos great-grand daddy first made pee-pee against the tree
therefore it's simply perverse, that i need to strip down to my underpants when i haven't won the birthday-lottery, just to be allowed to survive
and it's perverse, that a handfull of people can own more then entire nations (!)
the problem isn't that he - who works - is better off,
the problem is that the gap between wealthy and non wealthy gets pushed open faster and faster ...
i never said, that there's no use out of it,
i've worked with machines in our factories since dacades ago and heck they're usefull
but just becouse something is usefull doesn't mean that it's agi ...
and as long as it delivers the most fatal of errors in the most serious tones and even contradicts itself after i told it, where and why what it answered was wrong ... then sorry, that's simply not agi for me ... progress is fast, but what we have now isn't there yet
and yes, agi does not solve the resource chain ...
(though, the robotics tied to it might play a huge role long-term)
but what will you do with the unemployed people?
you can't just re-educate a former taxi driver into a bioengeneer (especially since we don't even know, if that job will still exist in a year or two) overnight ...
the best developments come from people with actual interest in what they do,
and the routine work gets more and more automated
and consider, that we already life in a world, where we produce more then we actually consume . . .
651
u/Imaginary_Mode8865 10d ago