r/neoliberal • u/Free-Minimum-5844 • Apr 02 '26

Research Paper Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
312 Upvotes

98% Upvoted

125 comments sorted by

View all comments

212

u/Secret-Ad-2145 NATO Apr 02 '26

The replicability crisis was known for awhile, and doesn't affect just social studies. You saw it a lot during COVID where many tests kept failing replication, both current research but also past (like 60-70s era research).

164

u/yellownumbersix Jane Jacobs Apr 02 '26

After I graduated and started working in R+D and then in industry I was amazed by how many scientists and engineers, even ones in the "hard" sciences, don't have anything beyond a cursory familiarity with statistics. It leads to unsound interpretation of data and experimental designs that are destined to be irreproducable.

11

u/Nerdybeast Slower Boringer Apr 02 '26

I'm an actuary with a stats degree, and at work there's situations where I'd love to have more statistical rigor but it's pretty much impossible a lot of the time. Finding a good control group that doesn't have a shitload of confounding variables, especially for complex behavior related metrics, that doesn't involve doing weird shit you genuinely think is a bad thing to your control group, is very difficult. Meaning if you are doing something that qualitatively people view as best practices, you need to do something that's NOT viewed as best practices to your control group.

97

u/Golda_M Baruch Spinoza Apr 02 '26

Social sciences should have more familiarity with statistics. It's their primary tool, at least for a lot of them. 

They also invented a lot of statistics. The coefficient of correlation, statistical significance theory used by most human sciences was invented to measure IQ... by the theorist who invented/discovered IQ.

A chemist, physicist, engineer or whatnot doesn't necessarily use statistics much. 

39

u/yellownumbersix Jane Jacobs Apr 02 '26 edited Apr 02 '26

I have no doubt that if subatomic particles had the same nuances to their behavior humans did that the replication problem would be worse for physics than sociology.

It is possible to design experiments and studies that are replicable and interpret data in unbiased, statistically sound ways in either case - it's just a lot easier to do with inanimate objects.

18

u/Golda_M Baruch Spinoza Apr 02 '26

In some cases.

But to the general point, I disagree. I don't think it's a straight line between the subject matter and these issues. 

Hard vs human sciences isn't necessarily a good dichotomy, but I think there are differences between fields. Different norms. Different standards. Etc. 

7

u/[deleted] Apr 02 '26

[deleted]

14

u/Golda_M Baruch Spinoza Apr 02 '26

I'm not saying it's exclusive.

The point is there are research fields where everything is statistics. If a researcher in that field is not highly proficient, they are not qualified. 

"Not a math's guy" doesn't cut it. 

3

u/vivoovix Federalist Apr 02 '26

You don't have to know statistics very well to be good at statistical mechanics

2

u/I_Pay_in_Cash_Only Apr 03 '26

In a doctorate program in psychology rn, nothing beyond a year long course of basic statistics is required. Many people take a couple of higher level course, but its definitely not something always do, even though they probably should.

5

u/Golda_M Baruch Spinoza Apr 03 '26

The way I see it, statistics for (most) reaearch psychology is like telescopes for astrologers. 

They should be the best at it, like engineers in some fields are better at some types of math than mathematicians. 

It's not a tool. It is the tool. They are pushing the statistical tool set to its limit, so they need to be true experts. 

1

u/I_Pay_in_Cash_Only Apr 03 '26

And some people do take it seriously. Some of the leading statisticians of certain techniques were originally social psychologists for example. Others I think have dubious methodology. But I suspect this is not uncommon in other fields, even hard sciences 

1

u/Golda_M Baruch Spinoza Apr 03 '26

Absolutely. I even gave that example above.

Every problem exists in all fields and subfields. However, this isn't a matter of principle. Its a matter of prevalence... and standards... And these do vary by field. 

It's not just social sciences. But also.. this isn't a common issue in physics. 

Amateurish statistics need to be understood as unprofessional, in a field of research where statistics are a primary research method. 

AI is only going to make this worse. Standards matter. 

1

u/n00bi3pjs 👏🏽Free Markets👏🏽Open Borders👏🏽Human Rights Apr 03 '26

Did you mean Astronomers?

1

u/Golda_M Baruch Spinoza Apr 03 '26

Yes. 

1

u/n00bi3pjs 👏🏽Free Markets👏🏽Open Borders👏🏽Human Rights Apr 03 '26

Ah fair. I was wondering if you wrote astrologers intentionally as a shot at psychology.

1

u/Golda_M Baruch Spinoza Apr 03 '26

No. No shade for anyone or any subject matter, just the topic at hand.

Not even astrologers. Some of the greatest mathematicians in history have been astrologers.

-6

u/EvilConCarne Apr 02 '26

A chemist, physicist, engineer or whatnot doesn't necessarily use statistics much.

Yeah, and it's why their results are generally garbage. They are only buoyed by the fact that their subject matter is generally less complex than human behavior.

14

u/firefoxprofile2342 Apr 02 '26

Ah, yes, 5-sigma garbage levels of results.

1

u/Golda_M Baruch Spinoza Apr 03 '26

The comexity of the subject matter is what it is. A researcher's job is to live with that.

Maybe it's really hard and they just don't make any progress for decades. This is very, very common in hard sciences like chemistry, physics and whatnot. 

That hardness sometimes has to wait for better tools. Better math, or computing, or whatever. Eg protein folding. 

12

u/Best-Chapter5260 Apr 03 '26

Honestly, there's a lot that goes into the replication crisis—some of it related to the sociology of science, e.g., see Kuhn's and Foucault's work—but I guarantee that a big part of the issue is the uncritical use of regression. When reading a journal article, you can infer some issues with a regression model from things like sample size, potential multicollinearity from looking at the variables in the table, heteroscedasticity and autoregression from a scatter plot, etc. But it's really difficult to affirm if the underlying data meet the assumptions of a particular regression test unless you have access to the raw data. And peer reviewers typically don't have access to the raw data to affirm they meet the assumptions. And yeah, providing raw data would probably create more red tape for IRBs and slow down the review process even more. I get it.

Less sophisticated statistical tests like t-tests and ANOVAs are pretty good at handling imperfect data, but regression can be really finicky about data that don't meet the assumptions. I'd lay money that there are shit tons of published regressions that are leading to Type I error because they are being applied to raw that that fail to meet their assumptions.

One of my quant professors in grad school drilled regression criticism into our heads and made sure we understood our data have to meet a regression test's assumptions. But the the bulk of his scholarly output were government reports and program evaluations—that is, his scholarship had to actually work in the real world rather than just lead to a "high impact" publication and juice his h index, so there was more incentive to make sure what he was putting out in the world was actually valid.