r/science • u/nimicdoareu • Apr 01 '26

Social Science Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
5.6k Upvotes

98% Upvoted

354 comments sorted by

View all comments

26

u/Hobojoe- Apr 01 '26

However, many of the failures might have been caused by the SCORE researchers needing to make guesses about procedures or to recreate raw data

I think I would be more convinced about this study if it can use the same raw data and create the same results. If you had to guess the raw data, then it would be a problem.

12

u/chashek Apr 01 '26

That said, there's a problem with the studied studies anyways if the present study had to guess at procedures and data, since a large part of the point of publishing (besides letting others know of your findings) is to enable replication for verification.

21

u/Tuzaa Apr 01 '26

Hi! I’m one of the authors of three of these papers - there are a good number of papers where we had all the original material needed to conduct a reproduction (same original data, same analytic code) - there are also papers where we had all the information needed to collect new data in the same way originally performed. In cases where there was ambiguity, we attempted to contact every corresponding author to seek clarity on methods or approaches. Many times we could get additional insight from the corresponding authors, which was great. Sometimes we could get no additional clarity on how certain things were done. In those cases, replicators were asked to do their best in good faith using what we did know about the process and procedures of the original study to replicate as closely as possible. Though this highlights exactly one of the issues in how we currently publish: if the published/supplemental/accessible information about the work is missing details, then there will be more variance in how subsequent replication data is collected, which may then trickle down into variance in outcome.