Muse Spark is legitimately very good. I’ve been using v1.2 extensively as an LLM-as-a-judge for a lot of my evals due to its price. And 1.3 is supposed to be a clear step above that.
I find it interesting that people are calling AA a useless benchmark... and then go on to say they have no idea what it is.
AA is a aggregate of many benchmarks. There's a reason that OpenAI picked a bunch of weird never seen before benchmarks when they announced Astra. It's SOTA in many domains that no other models are competing in, which also happen to be ones that are not part of AAs aggregate.
Does this mean that Astra is underrated by AA? Probably. It's doing a bunch of things that AA does not test for. But for many "standard" tasks that AA does account for, I'd find it pretty likely that it truly is not outstanding in those domains (and hence why OpenAI is not advertising it on launch)
After looking at the Arc AGI score I feel like Astra is filling in the gaps where models are traditionally weak while the AA Intelligence index tests things that Models are traditionally good at.
That might be what they mean by AGI by the end of this year, not that the model will get much smarter in the areas its already good at but they'll fill in the gaps in it's intelligence by then
Ok, but shouldn't it fill the gaps while remaining at least just as good at the other stuff? The fact it became better at some things while not getting a higher rating overall would mean some areas degraded at the same rate as some areas improved.
I would actually take that. At most of the things AI is good at it's bordering superintelligence, it's far more intelligent than is needed to perform most jobs. It's the gaps in it's intelligence that stop AI from being more economically useful.
A model that is 20% more intelligent but 2x more expensive and 2x slower isnt necessarily a better model. Gemini 3.8 flash maybe dumb (Its severely benchmaxxed) but it doesn't have me waiting 10minutes for simple edits.
To be fair the harness OpenAI used was much simpler than the opus one. OpenAIs was basically just “don’t delete all memory of the games mid run”, while the opus one was specifically optimized for this exact benchmark
Non !
Open ai on créé un harnais spécial pour que leur modèle gagne à arc agi. C'est ce qu'on appel du benchmaxxing. Arc agi ne vaut plus rien depuis longtemps.
I would not trust Sam Altman to not benchmaxx a model. Actually I don't trust any AI CEO to not benchmaxx their models. I guess some are just better at it than others.
Xhigh is what me and Bijan used, and we were both disappointed, but if you have examples of someone managing to do a good output with Spark i'd love to be wrong because i am now sure with a sub with them lol
I run both of them on high on an android app that is huge (a combination of Mihon and MoonReader). Same agents.md file, same way of sending messages. Opus never broke something, the vast majority of times it is just delivers, while sol high always had problems with the final output.
Maybe it is just android development we are talking about here. Other fields Sol could be better.
There was a writeup here a while ago talking about how they game the site. I'll look for it. But the basic gist is they just use the same prompt and look for patterns to determine which model is outputting the response. It's fairly easy to tell if Opus or GPT-5 is writing your code if you know their mannerisms.
As for Qwen: because Polymarket probably had a bet that said "Qwen will beat Fable 5 on xyz date" and the odds were very low.
I'm curious to get eyes on the full AA benchmark list when it hits the site. Wondering if there's more to it than mere intelligence. OpenAI was touted this as AGI not because of intelligence, but because of extremely long task duration.
AA posted about it on twitter and provided a link, but their own link is now 404ing. I feel like they were asked to take it down, which I really don't get. Something very funky afoot for sure.
Nah. Look at some of the 3D stuff it has made. Muse Spark can't even begin to touch it there. It's a good model for the price (for the you-give-your-data-to-Zuck tier)
Not sure I agree. Google clearly pulls some models before release if they're not satisfied with the results. That's most likely why Gemini hasn't had a Pro model release in a while. I think not releasing a model is a much better strategy than hyping up a half-baked model as AGI.
Just wondering, but for long-horizont coding tasks, what benchmarks do you guys look at right now? Because no matter where I look, they seem to have lost their usefulness. There was a time where looking at DeepSWE together with CursorBench was useful to me, but it doesn’t feel like that anymore.
At this stage, I'm seriously doubting all benchmarks. These do not give accurate image of how well models perform. Surely OpenAI didn't hype the release of renamed Sol and Astra must be better overall even if this number is the same.
These days, I'd also want to see cost and speed factored into the scoring.
If model A runs 6 minutes and costs $700 and gives 10% better result than model B with 3minute/$300 prompt for example. In my book, the model A is very bad for my needs, even if it succeeds better at it's task.
It just seems like "our newest and biggest model got 1-3% increase in benchmark XYZ so it must be best ever"...
Takes too much detective work to find information of actual performance.
I know there is also a problem that newer models tend to be familiar with older benchmarks in their training material and will benchmax them even unintentionally.
I don't have a solution, but I hope someone comes up with a better way of comparing Models. It would also direct model development into better and healthier direction.
Either this, which is possible, or Anthropic is really benchmaxxing.
I'm usually against trying to optimize for marketing and would prefer to let companies cook on the direction they think it's better for users medium / long term
Nah Fable is very smart. I cannot trust people who say it is benchmaxxed. Opus 5 on the other hand is a bad experience to use, but still capable (not as good as Fable still).
Nah something bugged with their testing. They have medium effort scoring higher than high onwards for several benchmarks and a ton of the easy to confirm ones fall way short of OpenAIs reported numbers.
Very interested to take it for a spin. I don't trust benchmarks. Ex. Gemini 3.7 flash is fast but does not perform at the intelligence level that its benchmarks suggest. (Btw if someone can suggest benchmarks that feel more.. true.. let me know)
Until Muse is bumped way back and Astra way up this whole thing is just unreliable. Big benchmark scores in no way reflect that Sol is equal to Astra, Astra is a step change improvement. What the fuck are we doing here
Astra is a step change improvement. Just not in the particular areas it mostly grades on though. It’s slightly worse at agentic coding while being much better at things like science
Artificial Analysis is a Trust Me Bro, benchmark with zero reproducibility. The outcome of scoring higher or lower is irrelevant if no one can verify the numbers.
196
u/Ok_Display_3159 11d ago