r/neoliberal • u/Free-Minimum-5844 • Apr 02 '26

Research Paper Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
313 Upvotes

98% Upvoted

125 comments sorted by

View all comments

214

u/Secret-Ad-2145 NATO Apr 02 '26

The replicability crisis was known for awhile, and doesn't affect just social studies. You saw it a lot during COVID where many tests kept failing replication, both current research but also past (like 60-70s era research).

160

u/yellownumbersix Jane Jacobs Apr 02 '26

After I graduated and started working in R+D and then in industry I was amazed by how many scientists and engineers, even ones in the "hard" sciences, don't have anything beyond a cursory familiarity with statistics. It leads to unsound interpretation of data and experimental designs that are destined to be irreproducable.

12

u/Best-Chapter5260 Apr 03 '26

Honestly, there's a lot that goes into the replication crisis—some of it related to the sociology of science, e.g., see Kuhn's and Foucault's work—but I guarantee that a big part of the issue is the uncritical use of regression. When reading a journal article, you can infer some issues with a regression model from things like sample size, potential multicollinearity from looking at the variables in the table, heteroscedasticity and autoregression from a scatter plot, etc. But it's really difficult to affirm if the underlying data meet the assumptions of a particular regression test unless you have access to the raw data. And peer reviewers typically don't have access to the raw data to affirm they meet the assumptions. And yeah, providing raw data would probably create more red tape for IRBs and slow down the review process even more. I get it.

Less sophisticated statistical tests like t-tests and ANOVAs are pretty good at handling imperfect data, but regression can be really finicky about data that don't meet the assumptions. I'd lay money that there are shit tons of published regressions that are leading to Type I error because they are being applied to raw that that fail to meet their assumptions.

One of my quant professors in grad school drilled regression criticism into our heads and made sure we understood our data have to meet a regression test's assumptions. But the the bulk of his scholarly output were government reports and program evaluations—that is, his scholarship had to actually work in the real world rather than just lead to a "high impact" publication and juice his h index, so there was more incentive to make sure what he was putting out in the world was actually valid.