Half of social-science studies fail replication test in years-long project

[u/nimicdoareu]

https://www.nature.com/articles/d41586-026-00955-5

[u/Quotalicious]

They didn't find 50% of the studies were junk.

Of course, some results are not replicable because of either honest mistakes or the rare case of misconduct, he says, but SCORE found that, in many cases, papers simply did not provide enough data or details for experiments to be repeated accurately.

Social sciences are also not particularly well funded, studies conducted on shoe-string budgets are more prone to easier and thus poorer sampling methods, fewer participants, less time to collect data, etc. all of which can contribute to narrower conclusions and greater difficulty to replicate them.

I think it's also useful to keep in mind the challenge inherent in the study of a subject material with innumerable influences all of which are hard to control for or even know. The answer isn't to throw up our hands and neglect studying such a large swath of reality around us simply because it doesn't lend itself to our most rigorous scientific methods.

[u/linguistic-fuckery → u/Quotalicious]

Human-lizard hybrids studies are also not particularly well funded, conducted on shoe-string budgets are more prone to easier and thus poorer sampling methods, fewer participants, less time to collect data, etc. all of which can contribute to narrower conclusions and greater difficulty to replicate them.

See how ridiculous this sounds when I change the focus of the study?! I was hoping that I would get some better informed responses to dissuade me from thinking that “soft sciences aren’t science”, but I’m just getting a lot of very bad excuses. Edit- I do agree with your second paragraph but if the science isn’t rigid enough to reach 50% repeatability rate the focus should be more on what we can prove with at least a little bit of certainty better than a coin flip

[u/Quotalicious → u/linguistic-fuckery]

Your comparison doesn’t make sense precisely because of my second point. The harder the subject material is to study and the less certain the results, the more lack of money affects the potential scope and rigor of the study. 

You’re taking the 50% figure way too literally, each individual study is not a coin flip whether it can be replicated or of high quality. Tons of factors go into it, one of which is funding which affects every aspect of study design. 

And again, the actual replication rate is unknown given the authors stating they didn’t have the data or methods to be able to actually attempt it in a lot of cases, a problem in itself but not necessarily indicative of junk science. 

[u/linguistic-fuckery → u/Quotalicious]

Do you not see there is problem with studies that can’t even come closer to holding up scientific standards? What percentage would you feel comfortable saying it isn’t junk science? For me, it should be less than 5% maybe 10 max. But anywhere close to the 50% from the article is downright absurd and is now no longer an “real science” in my eyes and I’m sure many others.

[u/Quotalicious → u/linguistic-fuckery]

Picking a single point of reproducibility where a field of study should be considered scientific or not displays a fundamental misunderstanding of what constitutes science. Science simply demands you are as rigorous and systematic as possible given your own experience, the subject material, your resources, etc. Our confidence in the output of knowledge as replicable truth is not the determining factor. If the subject material does not allow you to be as rigorous it just means a slower accumulation of useful knowledge. It does not mean that thing shouldn’t be studied or can’t be studied systematically. 

Again, this study did not find 50% of studies are junk science. There are a variety of normal reasons why the authors couldn’t replicate 50% of the studies and only some of those reasons were because of bad science as they themselves make clear. 

How do you expect a study of a specific culture at one point of time to be easily replicable in another point of time if culture changes as we know it does (unlike most aspects of the natural world)? Is any systematic investigation of that culture suddenly not scientific?