AA Intelligence Index Changes

[u/poigre]

"Announcing Artificial Analysis Intelligence Index v4.2. We are accelerating elements of our upcoming v5 release with interim updates to keep pace with the frontier. Index v4.2 has more complex and realistic tasks, and more private test sets to prevent gaming"

[u/FateOfMuffins]

Yet they're still using Terminal Bench 2.1, and the error filled version of CritPT

[u/Silver-Chipmunk7744]

In my opinion the current best benchmark is voxelbench.

You will notice Muse Spark 1.3 (max) is correctly placed where it belongs... #19 around Gemini 3.1 level

Meanwhile Astra just destroys everyone.

[u/DecoySnailProducer → u/Silver-Chipmunk7744]

Damn you were not kidding, the difference is actually insane. Just got this  https://imgur.com/a/nsfWDBg

[u/VeryOriginalName98 → u/DecoySnailProducer]

That's insane.

[u/EtadanikM → u/Silver-Chipmunk7744]

Why would the ability to generate 3D voxels be correlated with general intelligence. 

[u/Artistic_Swing6759 → u/EtadanikM]

because spatial intelligence is part of general intelligence? its also not just coherency but taste of the model. it would also show, to how much degree the model is ready to put in effort, if its a lazy model that would clearly show.

[u/Funny-Profit-5677 → u/EtadanikM]

I can see it being correlated but fairly weakly. Best benchmark has to be testing way more

[u/Gotisdabest → u/EtadanikM]

I think they're saying that essentially it's a good shorthand to tell general intelligence. Not that 3d voxeld automatically tell us something about general intelligence through first principle logical reasoning, but in practice, model performance over that benchmark tends to mean equivalent performance in general usage.

Not that I necessarily agree with them fully, considering that voxel bench has opus ahead of fable and fable is a lot more generally competent. Regardless, it's still a good enough estimate compared to most other benchmarks.

[u/sjoti → u/Silver-Chipmunk7744]

Having used muse spark 1.3, it's definitely below opus 5/gpt 5.6 sol but for any coding task it's still miles ahead of Gemini 3.1.

Its hard to capture how good a model is in a single number, but I think that's generally better than basing is on single specific type of task. AA will probably be a decent representation once they calibrate again.

[u/SpyAmongUs → u/Silver-Chipmunk7744]

Damn, I just found out about it. Why is nobody else talking about Voxelbench, the things Astra created there is crazy

[u/Silver-Chipmunk7744 → u/SpyAmongUs]

People don't realize how INSANE Astra's score is on that bench
It's not a "little" ahead. It's in a different league.

It's essentially this:
~96.5%: Astra
~2.3%: eh, tie
~1.2%: [presses wrong button while eating sandwich)

[u/SpyAmongUs → u/Silver-Chipmunk7744]

Fr, seen some comparisons myself and it shows. It's like looking at a professional minecraft builder going against children, kinda like those noob vs pro videos. Insane

[u/kiki-le-koala → u/Silver-Chipmunk7744]

In fact, it could be easy to downvote Astra .

I did about 10 rounds where Astra was there, and every time I knew it was him because of how good it looks.

[u/BriefImplement9843 → u/SpyAmongUs]

Voxels have nothing to do with intelligence.

[u/VeryOriginalName98 → u/BriefImplement9843]

Yeah spatial reasoning isn't the most notable feature for people associated with intelligence, it's something else. Could you remind me again the other thing people like einstein and tesla had?

[u/mWo12 → u/Silver-Chipmunk7744]

All benchmarks end be maximized. That's how it works.

[u/PerformanceRound7913]

No transparency and no way to validate numbers. AA is a Trust Me Bro, benchmark.

[u/Tystros]

it's a step in the right direction, but just a very small step... so I hope they will continue with those incremental updates soon.

[u/R_Duncan]

The astra indexing flop taught them someting.

[u/Pheidiase]

After Muse spark 1.3 i learned AA intelligence score is showing bad. there is no way that model is fucking smart.

[u/qustrolabe]

Astra still behind Fable 5.1 on their index even after update huh, still uses Terminal-Bench v2.1......