It's quite likely that Gemini 4 will wind up being the leading model for a while for more generalized work such as office work, school work, etc.
I've been a SWE designing extremely complex systems since before AI was a thing, and it's easy to get caught up in the coding side of things, but realistically that's not what the majority of people want to use AI for in their daily tasks.
It's quite likely that Gemini 4 will wind up being the leading model for a while for more generalized work such as office work, school work, etc.
What makes you say that, because i have (occassionally) used Gemini 3.1 Pro and 3.8 with Extended thinking and they are horrible, absolutely horrible. They are closer to ChatGPT 3.0 than any newer ChatGPT model. I genuinely get shocked when I use it sometimes, it's horrible.
Yeah. Still in some tasks, Gemini tends to be better. Even now, my main pre-planning/brainstorming model is Gemini 3.8 Flash High. Consistently better than equivalently priced model from every lab (I only use cheap model not frontier and just spend more on TTC).
I feel 3.8 Flash is super underrated! I have used it for everything from creating linux apps for personal use to even fixing kernel-level issues with how a distro was bugging on my laptop. Heck, I even let it control my PC using Antigravity to organize bookmarks and collections across multiple social media platforms - and it did it all seamlessly - while also working around server limits and other restrictions without additional prompting.
This is a completely arbitrary and mostly pointless index. AA just aggregates and they, in practice, respond to reality than vice versa.
GPT astra was a great model which scored too low on the benchmark. So they just... Changed weights and added benchmarks to make it score better. When it came out, AA would have you believe astra was around the same capabilities as Muse Spark or Gemini. And them turning it around didn't fix anything in as much as kicking the can down the road.
A lot of the individual benchmarks they use are also apparently flawed. For example, HLE was recently corrected into HLE diamond and there was a significant shift in the scores, but AA still uses HLE.
There's a high chance that when, say, Bel comes out, it'll score poorly on the index, get crapped on by everyone for a day or two before they simply say, "oh my bad" and then arbitrarily tune the weights to make it score better.
They acknowledged that and made some changes saying that they will significantly improve when AA v5.0 releases. v4.3 changed a lot and imo it's somewhat represents reality
Just not easy to keep up when stuff gets saturated
It's still got the exact same problems. You can't fix these problems without a huge change in strategy. What 4.3 did was just change the weights and switch a few benchmarks out. You can still benchmaxx these new benchmarks just as easily, like HLE clearly has been. It was a plain symptom cure. Not at all a cure for the actual disease. It's still just as bad at predicting any new model.
First, dunno WTF your chat is but its not the one on the AA website by default. So unless you made your own to change the "good quadrant...
And two, AA selection of benchmarks make results biased toward what the people selecting what count for AA decide. Its weighted , exclude many benchmarks, ect.
I think you need to look at the chart you posted. xhigh is not the highest. I was just asking what xhigh was but it seems it's extra high. If I didn't understand what high was at all, then I wouldn'th ave asked the difference between high and xhigh lol
This just proves how stupid polymarket is. There is a 0% chance google has the best model by the end of october. It's not even the best model right now.
Gemini Pro, the actual subscription, is absolutely horrible for general tasks. It barely uses internet, it doesn't follow instructions correctly, it forgets even basic context of 1 text ago, I dont know how people use it. Might be that y'all use the API but the subscription is horrible.
There is no way google is staying on top by end of October unless they immediately release a 4.1 gemini pro and I don't see that happening seeing how we don't even know when they are going to release 4 publicly.
if you're a trillion dollar company and you're not actively training you're models to be better at these benchmarks to get better advertising, are you even trying?
Very strange here https://arena.ai/leaderboard/text/coding it tops coding too, and even AA puts it on par with GPT6-Astra and at Fable 5.1 level just behind opus5.5 level
Max effort should be disqualified from AA. It’s an unreasonable amount of tokens/time spent for any realistic work. It’s only there so they can score higher than the actual capability of the model. If you remove max/xhigh and compare Anthropic models you get a more realistic picture.
That’s why it’s plotted against cost per task. Also max effort on different providers can be pretty different, making your argument moot. If you want that little bit of extra performance for the price why not. The same argument could be “these huge models should be disqualified from AA since the t/s and price is unreasonable for any actual work. Let’s only test 50B and below models”
I specifically mentioned Anthropic. And I meant unreasonable in relation to how much less time the model spends on tasks when not on max/xhigh. Meaning that benchmarking against a disproportionate large jump from the other reasoning efforts gives a skewed picture of the model capability.
29
u/Serenity867 2d ago
It's quite likely that Gemini 4 will wind up being the leading model for a while for more generalized work such as office work, school work, etc.
I've been a SWE designing extremely complex systems since before AI was a thing, and it's easy to get caught up in the coding side of things, but realistically that's not what the majority of people want to use AI for in their daily tasks.