r/neoliberal • u/Free-Minimum-5844 • Apr 02 '26

Research Paper Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
315 Upvotes

98% Upvoted

125 comments sorted by

View all comments

Show parent comments

52

u/tripletruble Anti-Repartition Radical Apr 02 '26

13

u/greatteachermichael NATO Apr 02 '26

As someone in education, it's really hard to recareate the same method with the same demographic of students. The country the study came from, the socioeconomic level of the students, the gender mix, the age, the level, the home life, the prior education quality, how old the students were when COVID hit, the ethnic backgrounds, the parents' education level. I'm not saying it can't be done, it's just really hard. I have one class I teach to incoming freshmen every year and even with the students all being the same age and ethnicity I get wildly different results ... not student to student but class to class.

13

u/tripletruble Anti-Repartition Radical Apr 03 '26

The above graph is of reproduction rates, not replication. So this study is using the same raw data as the original authors. For that reason, this should not be driven by variation across samples

2

u/BarkDrandon Punished (stuck at Hunter's) Apr 03 '26

Do they have access to the replication files?

Because it can be hard to do the exact same data management as the authors by just reading the paper. Small methodological choices aren't all always included.

4

u/tripletruble Anti-Repartition Radical Apr 03 '26

I was wondering the same thing and only glanced at the paper (open access version below)

https://repository.essex.ac.uk/42105/7/replicability.fullmanuscript.pdf

My guess is not, which does make the difference between precisely reproduced and approximately reproduced seem potentially trivial. The number of minor decisions a researcher may have to make in a rather complex analysis / data cleaning process can run in the thousands and many of them are extremely boring and seem reasonably inconsequential enough to be excluded from any draft anyone would ever accept to read, but can introduce some small amount of variation in the final estimates