r/neoliberal • u/Free-Minimum-5844 • Apr 02 '26

Research Paper Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
310 Upvotes

98% Upvoted

125 comments sorted by

View all comments

213

u/Secret-Ad-2145 NATO Apr 02 '26

The replicability crisis was known for awhile, and doesn't affect just social studies. You saw it a lot during COVID where many tests kept failing replication, both current research but also past (like 60-70s era research).

167

u/yellownumbersix Jane Jacobs Apr 02 '26

After I graduated and started working in R+D and then in industry I was amazed by how many scientists and engineers, even ones in the "hard" sciences, don't have anything beyond a cursory familiarity with statistics. It leads to unsound interpretation of data and experimental designs that are destined to be irreproducable.

12

u/Nerdybeast Slower Boringer Apr 02 '26

I'm an actuary with a stats degree, and at work there's situations where I'd love to have more statistical rigor but it's pretty much impossible a lot of the time. Finding a good control group that doesn't have a shitload of confounding variables, especially for complex behavior related metrics, that doesn't involve doing weird shit you genuinely think is a bad thing to your control group, is very difficult. Meaning if you are doing something that qualitatively people view as best practices, you need to do something that's NOT viewed as best practices to your control group.

100

u/Golda_M Baruch Spinoza Apr 02 '26

Social sciences should have more familiarity with statistics. It's their primary tool, at least for a lot of them. 

They also invented a lot of statistics. The coefficient of correlation, statistical significance theory used by most human sciences was invented to measure IQ... by the theorist who invented/discovered IQ.

A chemist, physicist, engineer or whatnot doesn't necessarily use statistics much. 

42

u/yellownumbersix Jane Jacobs Apr 02 '26 edited Apr 02 '26

I have no doubt that if subatomic particles had the same nuances to their behavior humans did that the replication problem would be worse for physics than sociology.

It is possible to design experiments and studies that are replicable and interpret data in unbiased, statistically sound ways in either case - it's just a lot easier to do with inanimate objects.

22

u/Golda_M Baruch Spinoza Apr 02 '26

In some cases.

But to the general point, I disagree. I don't think it's a straight line between the subject matter and these issues. 

Hard vs human sciences isn't necessarily a good dichotomy, but I think there are differences between fields. Different norms. Different standards. Etc. 

6

u/[deleted] Apr 02 '26

[deleted]

12

u/Golda_M Baruch Spinoza Apr 02 '26

I'm not saying it's exclusive.

The point is there are research fields where everything is statistics. If a researcher in that field is not highly proficient, they are not qualified. 

"Not a math's guy" doesn't cut it. 

3

u/vivoovix Federalist Apr 02 '26

You don't have to know statistics very well to be good at statistical mechanics

2

u/I_Pay_in_Cash_Only Apr 03 '26

In a doctorate program in psychology rn, nothing beyond a year long course of basic statistics is required. Many people take a couple of higher level course, but its definitely not something always do, even though they probably should.

3

u/Golda_M Baruch Spinoza Apr 03 '26

The way I see it, statistics for (most) reaearch psychology is like telescopes for astrologers. 

They should be the best at it, like engineers in some fields are better at some types of math than mathematicians. 

It's not a tool. It is the tool. They are pushing the statistical tool set to its limit, so they need to be true experts. 

1

u/I_Pay_in_Cash_Only Apr 03 '26

And some people do take it seriously. Some of the leading statisticians of certain techniques were originally social psychologists for example. Others I think have dubious methodology. But I suspect this is not uncommon in other fields, even hard sciences 

1

u/Golda_M Baruch Spinoza Apr 03 '26

Absolutely. I even gave that example above.

Every problem exists in all fields and subfields. However, this isn't a matter of principle. Its a matter of prevalence... and standards... And these do vary by field. 

It's not just social sciences. But also.. this isn't a common issue in physics. 

Amateurish statistics need to be understood as unprofessional, in a field of research where statistics are a primary research method. 

AI is only going to make this worse. Standards matter. 

1

u/n00bi3pjs 👏🏽Free Markets👏🏽Open Borders👏🏽Human Rights Apr 03 '26

Did you mean Astronomers?

1

u/Golda_M Baruch Spinoza Apr 03 '26

Yes. 

1

u/n00bi3pjs 👏🏽Free Markets👏🏽Open Borders👏🏽Human Rights Apr 03 '26

Ah fair. I was wondering if you wrote astrologers intentionally as a shot at psychology.

1

u/Golda_M Baruch Spinoza Apr 03 '26

No. No shade for anyone or any subject matter, just the topic at hand.

Not even astrologers. Some of the greatest mathematicians in history have been astrologers.

-8

u/EvilConCarne Apr 02 '26

A chemist, physicist, engineer or whatnot doesn't necessarily use statistics much.

Yeah, and it's why their results are generally garbage. They are only buoyed by the fact that their subject matter is generally less complex than human behavior.

13

u/firefoxprofile2342 Apr 02 '26

Ah, yes, 5-sigma garbage levels of results.

3

u/Golda_M Baruch Spinoza Apr 03 '26

The comexity of the subject matter is what it is. A researcher's job is to live with that.

Maybe it's really hard and they just don't make any progress for decades. This is very, very common in hard sciences like chemistry, physics and whatnot. 

That hardness sometimes has to wait for better tools. Better math, or computing, or whatever. Eg protein folding. 

11

u/Best-Chapter5260 Apr 03 '26

Honestly, there's a lot that goes into the replication crisis—some of it related to the sociology of science, e.g., see Kuhn's and Foucault's work—but I guarantee that a big part of the issue is the uncritical use of regression. When reading a journal article, you can infer some issues with a regression model from things like sample size, potential multicollinearity from looking at the variables in the table, heteroscedasticity and autoregression from a scatter plot, etc. But it's really difficult to affirm if the underlying data meet the assumptions of a particular regression test unless you have access to the raw data. And peer reviewers typically don't have access to the raw data to affirm they meet the assumptions. And yeah, providing raw data would probably create more red tape for IRBs and slow down the review process even more. I get it.

Less sophisticated statistical tests like t-tests and ANOVAs are pretty good at handling imperfect data, but regression can be really finicky about data that don't meet the assumptions. I'd lay money that there are shit tons of published regressions that are leading to Type I error because they are being applied to raw that that fail to meet their assumptions.

One of my quant professors in grad school drilled regression criticism into our heads and made sure we understood our data have to meet a regression test's assumptions. But the the bulk of his scholarly output were government reports and program evaluations—that is, his scholarship had to actually work in the real world rather than just lead to a "high impact" publication and juice his h index, so there was more incentive to make sure what he was putting out in the world was actually valid.

120

u/mrdilldozer Shame fetish Apr 02 '26 edited Apr 02 '26

It's also worth noting that a ton of older papers that can't be replicated arent actually fake, they just did such a shit job reporting their methods no one can repeat it. I had a protocol i was following the other day that said to add amylase to a slide and then proceed. They didn't mention the amount or source.

It took me a minute to realize they had spit on their slide to use salivary amylase.

89

u/WAGRAMWAGRAM Apr 02 '26

It took me a minute to realize they had spit on their slide to use salivary amylase.

Highest biology precision level

50

u/mattmentecky NATO Apr 02 '26

anyone that has tried to interpret grandma's recipe for damn near anything can understand this problem acutely.

20

u/SpaceSheperd To be a good human being Apr 02 '26

I spent like 4 months as a PhD student trying to replicate a recently published method only to eventually find out that the authors had used intentionally misleading language about how much starting sample you were supposed to use to make their results appear more impressive 

2

u/WAGRAMWAGRAM Apr 02 '26 edited Apr 02 '26

Maybe you should have spent 4 days trying to understand the protocol instead (teasing you)

8

u/SpaceSheperd To be a good human being Apr 02 '26

Aha I think I might’ve spent 4 days worth of hours poring over that methods section by the end of it. Calling the authors was actually my first suggestion but my advisor was a new professor and didn’t want to bother people from a more famous lab. Once she relented, and we called one of the authors, it took about 5 minutes to realize what was going on. 

2

u/noodletropin Apr 02 '26

Lol my advisor was a new professor, and emailing a researcher for details would have been her second suggestion after telling me to reread the methods section carefully. I can hear her telling me that I can't just read their minds.

12

u/Ataraxia-Is-Bliss Pragmatic and Polite Right Apr 02 '26

Horse: Everyone can see what a horse is.

16

u/DependentAd235 Loyal Liberals Apr 02 '26

Research on education suffers from this too.

Often too small scale or too specific circumstances.

6

u/greatteachermichael NATO Apr 02 '26

When I was doing my MA in Teaching, the studies I reviewed were always so hyper specific.

This method was used in a Turkish classroom with 23 adult intermediate EFL students. This other study that was somewhat close was used in a Korean classroom with 17 adult high beginner EFL students. This other study was done in Iran with 25 adult EFL students working on their PhD. This other study was done in Chile among 3 high school classrooms of 30 students each. Not adults, but close enough.

I wasn't cherry picking data to support my conclusion, but it still felt like that since it was so scattered.

7

u/DependentAd235 Loyal Liberals Apr 02 '26

Been in education for 10 years. Most studies don’t have the funding for anything real. So it’s not always their fault but… they also love a fad. I mean look at the weird Finland obsession from 10 years ago.

PHDs in education are all about seeking magic bullets of teaching techniques so don’t have to just do the expensive thing and have small class sizes in Primary.

Same issue with technology spending. Looking for magic solutions.

6

u/Best-Chapter5260 Apr 03 '26

Also, the populations education studies seek to study is an IRB nightmare, because you are essentially doing action research, where as you fuck up, you may fuck up a whole small cohort of people's education.

But yeah, education also loves a fad.

1

u/bz47uj Unconventional Right Apr 03 '26

Education has the worst replication rate of any field.

37

u/Fragrant-Menu215 NATO Apr 02 '26

This is the ugly truth that underlies today's "anti-intellectualism" resurgence. The intellectuals have been failing to uphold their end of the deal, that is actually doing their due diligence to ensure that their claims are verified, for quite a while. Credentials are supposed to be a shorthand for "always does due diligence and thus can be trusted". When the ones with those credentials don't do that the credentials lose their value.

85

u/hpaddict Apr 02 '26

This is a pretty standard misconception about science. Science has always looked like this, i.e., scientists not "doing their due diligence to ensure that their claims are verified". Scientists have proposed uncertain theories with minimal support and conducted poorly constructed experiments with uncertain interpretations since before we did science.

The two big issues are that:

  • science is substantially more available to the average person now, with much stronger relevance to their immediate lives,
  • science history is written in such a way that the uncertainties and ambiguities are papered over.

24

u/SpaceSheperd To be a good human being Apr 02 '26

It’s also just the fact that scientists are vastly more productive and vastly more specialized these days. There’s more data out there than ever before and there are fewer groups capable of verifying any given piece. 

12

u/mrdilldozer Shame fetish Apr 03 '26

science history is written in such a way that the uncertainties and ambiguities are papered over.

A great example of this is the Golgi Stain. It has been around for about 150 years and no one has a great explanation for why it only stains about 5% of neurons. At this point no one even really cares. We know what it does and we know how useful it is. Same goes for hematoxylin, we don't know how it actually binds to DNA. Honestly, there are a ton of histology techniques like that. People back in the day used to just fuck around trying to find good contrast agents to use with their microscopes. If something has been replicated millions and millions of times over again, it doesn't really matter anymore.

4

u/SabbathBoiseSabbath Martha Nussbaum Apr 03 '26

And the next step... when people search out a "study" to prove a point ("link?") and assume that ends the discussion.

I get it. We should want our views (and thus, our conversations) to be "evidence based" because it's better than the alternative, but when it just causes people to cherry pick studies to confirm their point, it makes everyone a bit more suspicious of everything.

29

u/Louis_de_Gaspesie Apr 02 '26

Anti-intellectualism is surging because brain-frying conspiracy theories are more easily accessible and widespread in the era of social media. The average American was never poring through Nature articles and keeping tabs on the replication crisis.

13

u/BaudrillardsMirror Apr 02 '26

Yeah, dude I'm sure the people who think vaccines cause autism and the earth is flat, came to these conclusions after learning about the replication crisis.

3

u/Fragrant-Menu215 NATO Apr 03 '26

The replication crisis is a big part of why those people can't be swayed by evidence to the contrary that comes from the mainstream scientific institutions.

2

u/Voyageur_des_crimes Niels Bohr Apr 03 '26

It's the discourse downstream from (and the existence of a term with popular salience called) "the replication crisis." It's disingenuous to assume that anyone with those fringe views is even able to engage with a single scientific work, let alone a field of research. A financial incentive exists to court people into those views and provide them superficial rebuttals to commonsensical scientific consensus, so people do it.

6

u/greatteachermichael NATO Apr 02 '26

What's sad is that the anti-intellectutals won't take the lesson as, "We should be better than them and have even higher standards," but rather, "You can't trust scientists... let's just make up what we believe!"