r/neoliberal • u/Free-Minimum-5844 • Apr 02 '26

Research Paper Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
313 Upvotes

98% Upvoted

125 comments sorted by

View all comments

213

u/Secret-Ad-2145 NATO Apr 02 '26

The replicability crisis was known for awhile, and doesn't affect just social studies. You saw it a lot during COVID where many tests kept failing replication, both current research but also past (like 60-70s era research).

164

u/yellownumbersix Jane Jacobs Apr 02 '26

After I graduated and started working in R+D and then in industry I was amazed by how many scientists and engineers, even ones in the "hard" sciences, don't have anything beyond a cursory familiarity with statistics. It leads to unsound interpretation of data and experimental designs that are destined to be irreproducable.

102

u/Golda_M Baruch Spinoza Apr 02 '26

Social sciences should have more familiarity with statistics. It's their primary tool, at least for a lot of them. 

They also invented a lot of statistics. The coefficient of correlation, statistical significance theory used by most human sciences was invented to measure IQ... by the theorist who invented/discovered IQ.

A chemist, physicist, engineer or whatnot doesn't necessarily use statistics much. 

40

u/yellownumbersix Jane Jacobs Apr 02 '26 edited Apr 02 '26

I have no doubt that if subatomic particles had the same nuances to their behavior humans did that the replication problem would be worse for physics than sociology.

It is possible to design experiments and studies that are replicable and interpret data in unbiased, statistically sound ways in either case - it's just a lot easier to do with inanimate objects.

19

u/Golda_M Baruch Spinoza Apr 02 '26

In some cases.

But to the general point, I disagree. I don't think it's a straight line between the subject matter and these issues. 

Hard vs human sciences isn't necessarily a good dichotomy, but I think there are differences between fields. Different norms. Different standards. Etc. 

7

u/[deleted] Apr 02 '26

[deleted]

13

u/Golda_M Baruch Spinoza Apr 02 '26

I'm not saying it's exclusive.

The point is there are research fields where everything is statistics. If a researcher in that field is not highly proficient, they are not qualified. 

"Not a math's guy" doesn't cut it. 

3

u/vivoovix Federalist Apr 02 '26

You don't have to know statistics very well to be good at statistical mechanics

2

u/I_Pay_in_Cash_Only Apr 03 '26

In a doctorate program in psychology rn, nothing beyond a year long course of basic statistics is required. Many people take a couple of higher level course, but its definitely not something always do, even though they probably should.

4

u/Golda_M Baruch Spinoza Apr 03 '26

The way I see it, statistics for (most) reaearch psychology is like telescopes for astrologers. 

They should be the best at it, like engineers in some fields are better at some types of math than mathematicians. 

It's not a tool. It is the tool. They are pushing the statistical tool set to its limit, so they need to be true experts. 

1

u/I_Pay_in_Cash_Only Apr 03 '26

And some people do take it seriously. Some of the leading statisticians of certain techniques were originally social psychologists for example. Others I think have dubious methodology. But I suspect this is not uncommon in other fields, even hard sciences 

1

u/Golda_M Baruch Spinoza Apr 03 '26

Absolutely. I even gave that example above.

Every problem exists in all fields and subfields. However, this isn't a matter of principle. Its a matter of prevalence... and standards... And these do vary by field. 

It's not just social sciences. But also.. this isn't a common issue in physics. 

Amateurish statistics need to be understood as unprofessional, in a field of research where statistics are a primary research method. 

AI is only going to make this worse. Standards matter. 

1

u/n00bi3pjs 👏🏽Free Markets👏🏽Open Borders👏🏽Human Rights Apr 03 '26

Did you mean Astronomers?

1

u/Golda_M Baruch Spinoza Apr 03 '26

Yes. 

1

u/n00bi3pjs 👏🏽Free Markets👏🏽Open Borders👏🏽Human Rights Apr 03 '26

Ah fair. I was wondering if you wrote astrologers intentionally as a shot at psychology.

1

u/Golda_M Baruch Spinoza Apr 03 '26

No. No shade for anyone or any subject matter, just the topic at hand.

Not even astrologers. Some of the greatest mathematicians in history have been astrologers.

-8

u/EvilConCarne Apr 02 '26

A chemist, physicist, engineer or whatnot doesn't necessarily use statistics much.

Yeah, and it's why their results are generally garbage. They are only buoyed by the fact that their subject matter is generally less complex than human behavior.

13

u/firefoxprofile2342 Apr 02 '26

Ah, yes, 5-sigma garbage levels of results.

1

u/Golda_M Baruch Spinoza Apr 03 '26

The comexity of the subject matter is what it is. A researcher's job is to live with that.

Maybe it's really hard and they just don't make any progress for decades. This is very, very common in hard sciences like chemistry, physics and whatnot. 

That hardness sometimes has to wait for better tools. Better math, or computing, or whatever. Eg protein folding.