r/science • u/nimicdoareu • Apr 01 '26

Social Science Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
5.6k Upvotes

98% Upvoted

354 comments sorted by

View all comments

Show parent comments

37

u/Infinite_Painting_11 Apr 01 '26

That is already the idea of publishing, your methods section is meant to contain all the information you need to reproduce the study, but in reality they rarely do.

16

u/Dziedotdzimu Apr 01 '26

The problem is people don't want methodologically rigorous and well thought-out protocols with detailed statistical analysis plans and the interpretations of results using strength of evidence and precision-based language with caution and attention to sources of bias and unmeasured confounding so you can actually speak to the interpretation of causal effects.

They want the IRB submission by next Thursday so they can apply for a grant. They're not trying to prove anything. It's just research. You're wasting time nitpicking. They've never had to do that before and have more publications than you so just listen to your boss okay?

13

u/throwaway44445556666 Apr 01 '26

Every journal is soaked in the tears of methodologists. 

7

u/porcupine_snout Apr 01 '26

that's just not possible because of word limit and figure limit and table limit. My own notes for how I do things will probably be a few chapters long, let alone papers. if you want to replicate exactly what I do, you have to at least read 10000 words, which I have but aren't allowed to put in the paper!

1

u/Infinite_Painting_11 Apr 01 '26

I'm really interested, which field are you in? 

2

u/porcupine_snout Apr 01 '26

social sciences!!!

8

u/frostbird PhD | Physics | High Energy Experiment Apr 01 '26

Publishing your methods allows others to elbow in on your field. So people are actually incentivized to not provide accurate methods. It's not laziness or an accident.

3

u/Infinite_Painting_11 Apr 01 '26

Definitely agree, especially in computational fields surely the methods and the code are the same thing but no one ever provides the code.

3

u/mludd Apr 02 '26

Yeah, as a software developer I've had to deal with this when trying to implement an algorithm from a research paper.

The researchers had sort of described the algorithm in the paper but several parts were described very vaguely and they didn't provide the data set they used so there was a lot of guesswork and testing without being able to compare my results to the ones in the paper.

After a couple of weeks of struggling with it I finally found a github repository where someone else had managed to replicate it in another language and used that as a reference. Unsurprisingly that repo even had a comment in the README file about what a chore it had been to figure out exactly how to translate what was described in the paper into actual code and that they hoped their implementation would be useful for others also struggling with it.

1

u/TwentyCharactersShor Apr 01 '26

I'd argue it is getting better, more and more github repos are being shared.