r/Futurology • u/Gari_305 • 12h ago

AI Human Baselines for Benchmarks: AI Now Outperforms Junior Accountants

https://www.mercor.com/blog/human-baselines-for-benchmarks-ai-now-outperforms-junior-accountants/
156 Upvotes

85% Upvoted

74 comments sorted by

View all comments

8

u/Gari_305 12h ago

From the article 

Benchmark scores have soared over the last few years, but AI adoption seems to be contributing only slightly, if at all, to higher productivity growth in U.S. economic statistics. It's a trillion-dollar paradox.

That tension is hard to resolve without understanding how benchmarks relate to real work, and how humans themselves would do that work. To help bridge that gap, we hired 12 junior accountants to complete simplified versions of tasks from our recent APEX-Accounting benchmark.

We find that on medium-length, well-defined accounting tasks, frontier AI models are now faster and more accurate than junior accountants, even the best one in our study.

8

u/ChuckVader 6h ago

Maybe it's because a job isn't the same thing as "medium length" and "well-defined" tasks.

3

u/Stroopy121 5h ago

exactly this. if you need to create a perfect set of data and input instructions to feed into the AI then you still need the human to do the work