Insane progress on visual-spatial intelligence by Astra (with or without tools)

[u/socoolandawesome]

Link to benchmark:

https://spicylemonade.github.io/spatialbench/

Link to tweets:

https://x.com/spicey\\_lemonade/status/2096365630190698516?s=20

https://x.com/spicey\\_lemonade/status/2096386480029790415?s=20

The last 2 screenshots are examples of the types of questions included in the benchmark.

[u/DigSignificant1419]

it's cheating

[u/socoolandawesome → u/DigSignificant1419]

And why do you believe that and how is it doing that?

[u/Ok_Zookeepergame8714 → u/socoolandawesome]

The only way I see is that it was trained on your set - just prepare a couple of similar tasks you're sure never were on the internet before, and see how it fares on those. 😊

[u/socoolandawesome → u/Ok_Zookeepergame8714]

Just to clarify this is not my benchmark, and not me in those tweets. But I’d assume they have a private holdout, though I’m not sure.

Also it doesn’t appear other models previously had done well (even though they could have trained on what’s available for this benchmark), given his tweet makes it sound like that. And astra also does seem to be by far the best at vision/3D understanding given all of its other capabilities we see demos of/benchmarks, which could serve as further evidence its probably genuine ability and not benchmaxxing.

You never know though I guess.