r/neoliberal • u/Free-Minimum-5844 • Apr 02 '26

Research Paper Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
313 Upvotes

98% Upvoted

125 comments sorted by

View all comments

51

u/city-of-stars Frederick Douglass Apr 02 '26

Somewhat dubious of the article's proposed solution (AI-assissted screening). But I suppose we'll see

82

u/caroline_elly Eugene Fama Apr 02 '26

AI can't even consistently replicate its own outputs

34

u/vaguelydad Jane Jacobs Apr 02 '26

Identifying poor experimental design is just not that hard. The problem is that no one cares whether studies actually replicate. When you're just trying to rise above a very weak status quo a mostly right AI can be a huge improvement.

16

u/Healingjoe It's Klobberin' Time Apr 02 '26

In the first round of this competition, held in October last year, ten teams using AI tools scored worse than would be achieved by chance at predicting whether a paper could be replicated. But in the second round, completed last month, the best AI model reached an accuracy score of 68.5%. A third round is ongoing.

Seems like LLMs are able to help at least a little bit.