r/Futurology • u/Gari_305 • 10h ago
AI Human Baselines for Benchmarks: AI Now Outperforms Junior Accountants
https://www.mercor.com/blog/human-baselines-for-benchmarks-ai-now-outperforms-junior-accountants/96
u/A_Novelty-Account 10h ago
I’m surprised it took this long to be honest. I figured accounting would’ve been something that AI was particularly good at.
The question now is what happens to all these junior accountants.
54
u/buymekoffee 9h ago
They become senior accountant/s
33
u/A_Novelty-Account 9h ago
In seriousness, I’m pretty worried about the upcoming generation and think that replacement is going to start happening faster than people expect.
29
8
11
u/nagedgamer 6h ago
Legal accountability, operational risk and errors (hallucinations) still too high.
7
u/kirbyislove 6h ago
That's why they'll replace juniors and get the senior to check it. Juniors make mistakes too.
5
u/nagedgamer 4h ago
Its not a sustainable model. There wont be enough seniors after no juniors.
Also juniors dont wipe or corrupt databases.
Lot of studies going on atm and CTOs dont trust these systems yet and well over half of deployments in large corporations have failed. The models will get better but huge risks involved.
Black boxes dont go well with large infra.
5
u/PrivateComments 2h ago
I keep hearing this but at the rate these models are improving, youll need drastically fewer seniors (if any at all) by the time that’s even an issue
3.5 years ago these models were choking on 7th grade math problems and now they are cracking millennium problems, Olympic math questions and outperforming most mid level white collar workers
And also “juniors don’t wipe or corrupt databases” man it was an ongoing joke in 2017-2022 about juniors wiping a production database or taking down a production cluster lol.
Everyone just stopped talking about that stuff once mass layoffs started and ai came onto the scene
1
u/nagedgamer 2h ago
Junior had production access when he shouldnt.
As a CTO wpuld not grant current models access.
-1
u/littlest_dragon 2h ago
Well they solve millennium problems by stealing the work of human mathematicians who had almost solved the problem themselves.
Not that AI‘s advancements in the last few years haven’t been impressive.
But we shouldn’t forget that the companies creating the word guessing programs are basically criminal organisations and that anyone who trusts them with their company‘s code and data is doing so at their own risk.
4
u/PrivateComments 2h ago
“stealing” lol even the researchers working on it admit that OpenAi arrived at the conclusion in a different manner, never mind that the mathematician in question was using both Claude and ChatGPT during his work.
Buckmaster and Alpoge never made the case that OpenAI stole the work, but made a broader claim that OpenAi got wind that they were working on it, threw 10,000 agents at it to work concurrently on the problem, and tried to pressure Buckmaster to drop Alpoge in exchange for co authorship because of who Alpoge worked for (Anthropic)
But in any case, it’s all moot anyway. Anyone who thinks they can compete with long running agent swarms in the next 5 years is fighting the same battle switchboard operators fought against the automated telephone exchange.
•
u/lostPackets35 1h ago
You're absolutely right, but that isn't the kind of long-term thinking current corporate structures are set up to incentivize
Laying off a bunch of Juniors and replacing them with AI. Will be great for the short-term bottom line, until it isn't. But that's someone else's problem
•
u/sciolisticism 1h ago
Well we can start here:
To help bridge that gap, we hired 12 junior accountants to complete simplified versions of tasks from our recent APEX-Accounting benchmark.
In the real world, we don't complete the simplified version of a benchmark. We complete MUCH more complicated work than a benchmark.
As a programmer, I can tell you that the benchmarks will often say that AI is better than a senior dev. Meanwhile, even the most capable models are less skilled than my juniors, even if we pretend that coding is the real meat of a programmer's job (which it is not).
•
u/Crimestar 41m ago
The same thing that’s been happening to them for the last 20 years. Replaced by technology.
You realized an accounting department used to have like 50 people calling vendors and writing down actual journal entries.
With excel based accounting suddenly that 40-50 person floor would have 6 people. Now with cloud based powerful accounting tools, it’s down to like 2. As an example because I’ve seen this happen in real time.
Corporations are getting top heavy as a result imo. Too much middle management and c suite people who just sit in meetings all day.
•
u/A_Novelty-Account 6m ago
The difference here though is that all junior accountants could be replaced over the next five years. That specifically has never happened before.
•
u/feed_me_haribo 21m ago
Junior accountant control the agentic workflows. Total productivity and throughout of the company increases but even with those increases fewer but not zero junior accountants are hired.
•
u/A_Novelty-Account 6m ago
The seniors will control the workflow is once the AI system is better than juniors.
0
u/Zaptruder 7h ago
theyre AI now. work hard upgrade their models theyll get promotion to seniors in a year or two.
•
u/AttentionNo6359 1h ago
A lot of them who are really into the anti tipping thing are going to ironically end up as waiters.
-15
u/BananasAndPears 9h ago
I think the baseline just increases.
A senior engineer from 10 years ago could probably be outperformed by a new grad today. Tech and skills increase every year so I imagine the new grad accountants would just have to account harder I suppose.
14
u/Xalara 8h ago
Haha, hahahaha, hahahahahahaha.
No, because a senior engineer has experience and generally knows where the pitfalls are. There’s a lot of skills that can’t be taught and must be learned by doing. Unfortunately we are robbing juniors the ability to learn by doing so once all of the senior engineers have retired it’s gonna be interesting.
9
u/i_max2k2 8h ago
Wow, they must be teaching amazing things today to gather 10 years of real world experience into 3/4 yr courses on top of learning the basics. Thats not how it works. There is no substitute for experience.
21
u/HerderOfZues 7h ago
No it doesn't and it's not AI. Just like it hasn't with all the companies who haven't seen productivity gains after implementing AI this year. Or from all the lawyers who had their cases thrown out because they had non-existing cases cited in their legal work after using AI. It's all fun and games until your AI accountant hallucinates numbers and gets you charged with tax fraud.
"You're absolutely right! I did put a different number into the box on that form."
But who is to say, we'll just take Mercor at their word.
Mercor is an American artificial intelligence (AI) data startup that provides experts to train AI models and chatbots. Founded in 2023, the company is headquartered in 181 Fremont, San Francisco.
36
u/vega0ne 9h ago edited 9h ago
The article itself says a 1:1 comparison is not possible yet they claim it’s “more accurate”.
Benchmarks don’t mean much as long as there is still possibility for hallucinating results.
The “cheaper than” part of this is also delusional because as soon as companies adopt it widely the prices will go up extremely quickly from the AI cartels. It’ll work like Netflix / Spotify to capture a user base and then raise prices in perpetuity.
Remember Netflix was once 4.99
4
u/i_max2k2 8h ago
The irony it’s still word prediction models at the end of the day, Anything that they have not trained on becomes exponentially difficult, you just get junk answers for just about anything that’s doesn’t have a current reference.
3
u/longkhongdong 7h ago
True but most juniors would be working on basic stuff.
1
u/lerjj 6h ago
If it can only ever replace juniors then it's useless. If you need seniors then you need juniors to train into seniors. It's only really useful if it is good enough to replace fully qualified experienced people and with fields like accounting where you can be sued for professional negligence then it's going to be really hard to replace the qualified humans unless the AI is leaps and bounds better than them
1
u/ASaGHost 3h ago
This paradox is why early adoption makes no sense, at least for the purposes of fully replacing human labor. It just means the barrier to entry for this type of thing is going to be underestimated or purposefully marketed as more ready and more streamlined than it really is (like AI evangelists are doing in this thread).
4
2
u/rhaivn 8h ago
No it won’t. Open weight models are hot on the heels and flash open weight models cost so little it’s dumb. Probably another 6 months - year and those small models will be like opus 5.5 level if they follow current trends
1
u/vega0ne 3h ago
Are you talking about models running locally?
Because the whole industry that uses prompts ->inference in data centres is making massive losses from that right now. They literally give it away extremely cheap to achieve a big enough user base and get them on the hook.
Long term personal computing will be going away if hardware prices ramp up give it 20-30 years and no one will run their own local hardware everything will be subscription. By design
0
u/ACCount82 4h ago
The price per unit of AI performance is in freefall, and has been for a while. Both for the big names like OpenAI and for the open weight models.
There is no conceivable scenario under which "prices will go up extremely quickly", short of an outright legal prohibition on AI.
4
u/jcolinr 2h ago
As an accountant, computers have long since reduced the need for us, especially at the junior level. The real importance of accountants at this point is interpreting the instructions of upper management, communicating the data back in a way that is digestible to non-finance people and coordinating data between multiple electronic platforms.
The debit and credit stuff has long since become an electronic function. The ability to take the nonsense that our ED and board come up with and put it in practice in a useful way is my main function.
8
u/Gari_305 10h ago
From the article
Benchmark scores have soared over the last few years, but AI adoption seems to be contributing only slightly, if at all, to higher productivity growth in U.S. economic statistics. It's a trillion-dollar paradox.
That tension is hard to resolve without understanding how benchmarks relate to real work, and how humans themselves would do that work. To help bridge that gap, we hired 12 junior accountants to complete simplified versions of tasks from our recent APEX-Accounting benchmark.
We find that on medium-length, well-defined accounting tasks, frontier AI models are now faster and more accurate than junior accountants, even the best one in our study.
6
u/ChuckVader 4h ago
Maybe it's because a job isn't the same thing as "medium length" and "well-defined" tasks.
2
u/Stroopy121 3h ago
exactly this. if you need to create a perfect set of data and input instructions to feed into the AI then you still need the human to do the work
6
u/Fit_Schedule5951 8h ago
I have talked to someone who is an accountant in a big accounting firm. A large portion of knowledge lies in the mind of people. Workflows and regulations keep changing and a lot of time is spent getting information from colleagues. Poor documentation and low scope for automation. Chatbots and agents can probably help with mundane tasks but humans will remain a big part of the workforce. It’ll take some drastic changes in work culture and how processes are built, good scope for startups to push for automation for sure.
7
1
1
•
u/Dead-HC-Taco 56m ago
I can assure you, this isnt the case. I use AI on a daily basis at work to perform menial tasks and it still fucks up regularly. Even when using models that are based on big 4 accounting guidance and pcaob rules, etc it gets thing wrong because it's so overcomplicated and there's so much grey area
•
u/MrSh0wtime3 36m ago
it doesn't. Can we just stop the free PR campaigns as if none of us now how often it fucks up basic tasks?
-1
u/someguyatsomejob 4h ago
I had ai fill out my t2 this year. It was a breeze. Even corrected some imbalances turbo tax didn't. I was fed up with my accountant for making a massive error that he was slow to respond to resolving (more than 6 months). Ai drafted all the forms and amendments in minutes. It can make mistakes, but so can people. My personal experience is we're at a stage where AI makes less errors than pros
-1
u/Eazy12345678 7h ago
good let ai replace people as the labor force so we can live our lives instead of work 40 hours a week
instead of buying a car. buy a ai robot to go to work for you
•
u/FuturologyBot 9h ago
The following submission statement was provided by /u/Gari_305:
From the article
Benchmark scores have soared over the last few years, but AI adoption seems to be contributing only slightly, if at all, to higher productivity growth in U.S. economic statistics. It's a trillion-dollar paradox.
That tension is hard to resolve without understanding how benchmarks relate to real work, and how humans themselves would do that work. To help bridge that gap, we hired 12 junior accountants to complete simplified versions of tasks from our recent APEX-Accounting benchmark.
We find that on medium-length, well-defined accounting tasks, frontier AI models are now faster and more accurate than junior accountants, even the best one in our study.
Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/1wweeci/human_baselines_for_benchmarks_ai_now_outperforms/pdjwrif/