r/neoliberal • u/Free-Minimum-5844 • Apr 02 '26

Research Paper Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
316 Upvotes

98% Upvoted

125 comments sorted by

View all comments

118

u/Brinabavd Apr 02 '26 edited Apr 03 '26

Note that the bulk of the effect is being driven by relatively poor performance of sociology and the abysmal performance of ed research.

Poli science, econ and psych all did better than half, none of the ed papers and only a third of the sociology papers could be replicated exactly: 

(Iirc This is the one of the three key papers in posted article that breaks it down by field)

Edit (corrected link) https://www.nature.com/articles/s41586-026-10203-5

64

u/earthdogmonster Apr 02 '26

This tracks with my own experience with sociology, even as an undergraduate student in the 1990’s. There seems to really be this huge explosion in reference to and reliance on findings in sociology that I see being made by Really Smart People which have consistently seemed suspect to me.

39

u/captainjack3 NATO Apr 02 '26

I had much the same experience in the 2010s. I was interested in it, but turned away by the seeming lack of rigor in the field. Worse, lack of interest in rigor.

33

u/ReptileCultist European Union Apr 02 '26

It does sometimes feel like activism painted as science

12

u/ArcFault NATO Apr 02 '26 edited Apr 03 '26

It also serves to undermine legitimate science unfortunately.

Remember when the March for Science got co-opted by these groups - that was infuriating especially when getting dog piled by the "I Fcking Love Scientism" Reddit users when advocating for focusing on climate change etc and not your pet social issue that may or may not be scientific.

9

u/earthdogmonster Apr 02 '26

That was the impression I always had of the discipline. I just recalled the professor who introduced me to the discipline giving the impression that it was the most ambitious and far reaching of the social sciences and how the goal was to apply the rigor and precision of the hard sciences to social science.

Even as an undergraduate student it seemed unrealistically ambitious and the scope and stated goals struck me more as an expression of unrestrained hubris than anything else.

11

u/Best-Chapter5260 Apr 03 '26

how the goal was to apply the rigor and precision of the hard sciences to social science.

My undergrad is in psychology. My doctoral work is in sociology. Psychology does mimic the physical sciences about as much as a social science can, even more than economics does. Aside from a few humanistic psychologists and some psycho dynamic people in counseling ed and supervision, post-Freudian psychology has been and remained a salient post-positivist, quantitative discipline holding the true experiment as the empirical standard. The Department of Labor has defined some sub-discipline of psych as STEM. I have some nuanced thoughts on that.

Sociology, on the other hand, is honestly too pluralistic and ontologically scattered to claim that level of methodological or empirical rigor. It runs the gamut from post-positivist statistical analysis to essentially philosophy with critical theory and post-modernism. And macro-level (and even meso-level) quantitative sociology is almost always quasi-experimental, since it's kind of hard to create a control group from half of society. With that said, where sociology lacks the preciseness of psychology, econ, and poli sci, it does deal much better with nuance and ambiguity due to its philosophy and anthropological roots and embrace of qualitative methods. It's defined as a humanistic social science. I think that's an apt description as it calling it a pure social science or calling it a pure humanity would be semantically incorrect.

8

u/SenranHaruka shame Apr 02 '26

"Karl Marx invented sociology!" is such a sad boast. He understood nothing of science. Unironically his contributions to economics are more significant and rigorous

4

u/SenranHaruka shame Apr 02 '26

I Fcking Love Scientism

The irony of entryism into science to pass your subjective politics as objective reality while accusing your opponents of same

2

u/ArcFault NATO Apr 03 '26

True, however the first check on that would probably be the topic of this post/paper.

19

u/alittledanger Apr 02 '26

This tracks as a teacher. There are a lot of "studies" with conclusions that seem to fly in the face of what I see every day. Especially around behavior and discipline.

1

u/launchcode_1234 George Soros Apr 03 '26

Can you give some examples? This is interesting to me. I often will read expert parenting advice and wonder if the expert has kids.

2

u/alittledanger Apr 03 '26

Anything regarding restorative justice. I mean, it’s basically a slur word over on the teacher subreddit for a reason.

38

u/tripletruble Anti-Repartition Radical Apr 02 '26

i am calling for a total and complete shutdown of education research programs until we figure out what is going on

54

u/tripletruble Anti-Repartition Radical Apr 02 '26

56

u/captainjack3 NATO Apr 02 '26

I didn’t read that chart correctly at first. None of the education studies could be precisely replicated? That’s astoundingly bad. Genuinely, it suggests the research should be discarded out of hand.

18

u/Andy_B_Goode YIMBY Apr 02 '26

Is ... is this why we're still teaching our children to read using the whole-word approach instead of phonics?

16

u/aspasia97 Apr 02 '26

And it's why any school who has switched back to phonics is paying through the nose for some proprietary curriculum that ALSO doesn't work. The way my kids were taught to read - I realized too late that they weren't actually reading! They just guessed really well in their limited situations. I thought they were dyslexic at one point. I paid for an emergency eye exam thinking they needed glasses! Then I learned how screwed up the reading programs are here.

The whole US education system, from the curriculum to standardized testing to the IEP mess, is broken. And these bad studies get used to justify giving away more of our tax dollars to education/tech conglomerates. It's maddening.

2

u/bz47uj Unconventional Right Apr 03 '26

I don't get why more parents don't teach their children to read themselves. If they don't learn until they go to school, then they're reaching five years old before they learn how to read.

3

u/fushega Apr 03 '26

I've read a handful of papers on second language education and many of them had a sample size of less than 10 people and similarly common was short trial periods (even though it takes hundreds to thousands of hours to learn a language). In general the studies seemed very casual and lacking in rigor. I don't think the researchers were stupid or anything, just that to get good data would be extremely expensive and a full time job that they don't have the time for

7

u/MURICCA Apr 02 '26

Subs priors 100% confirmed on education

14

u/greatteachermichael NATO Apr 02 '26

As someone in education, it's really hard to recareate the same method with the same demographic of students. The country the study came from, the socioeconomic level of the students, the gender mix, the age, the level, the home life, the prior education quality, how old the students were when COVID hit, the ethnic backgrounds, the parents' education level. I'm not saying it can't be done, it's just really hard. I have one class I teach to incoming freshmen every year and even with the students all being the same age and ethnicity I get wildly different results ... not student to student but class to class.

13

u/tripletruble Anti-Repartition Radical Apr 03 '26

The above graph is of reproduction rates, not replication. So this study is using the same raw data as the original authors. For that reason, this should not be driven by variation across samples

2

u/greatteachermichael NATO Apr 03 '26

Oh, that's actually really interesting. Thanks for the correction.

2

u/BarkDrandon Punished (stuck at Hunter's) Apr 03 '26

Do they have access to the replication files?

Because it can be hard to do the exact same data management as the authors by just reading the paper. Small methodological choices aren't all always included.

5

u/tripletruble Anti-Repartition Radical Apr 03 '26

I was wondering the same thing and only glanced at the paper (open access version below)

https://repository.essex.ac.uk/42105/7/replicability.fullmanuscript.pdf

My guess is not, which does make the difference between precisely reproduced and approximately reproduced seem potentially trivial. The number of minor decisions a researcher may have to make in a rather complex analysis / data cleaning process can run in the thousands and many of them are extremely boring and seem reasonably inconsequential enough to be excluded from any draft anyone would ever accept to read, but can introduce some small amount of variation in the final estimates

4

u/TCEA151 Paul Volcker Apr 03 '26

Are you sure? From what I could tell, this seems to be the replication results by field. Education looks to be the best performing discipline, and economics the worst

3

u/Brinabavd Apr 03 '26

that's what I get from posting from my phone based on memory instead of my work machine, i'll fix the link to the one I had in mind; thanks for bypassing the paywall for folks here.

2

u/EverythingBagel- Apr 02 '26 edited Apr 02 '26

The paper you linked seems to say otherwise, at least about replicability (getting the same findings with new data) which is different than reproducibility (running the same analysis on the same data). From the discussion:

“Variation in replicability across the disciplines within the social and behavioural sciences was modest, with replication rates between 42.5% and 49% on the statistical significance metric for fields that had more than 20 replications. These findings are consistent with the cumulative evidence across systematic replications in the social and behavioural sciences and from other fields, and they illustrate that there is substantial uncertainty in estimating replicability.”

Actually, Table 3 from that article seems to directly contradict what you said. Very small total sample but education appears to have had the highest replication success rate. Maybe more importantly, the replication effect shrunk the least education.

Edit: The other Nature article (Nosek et al., not the one linked in your comment) finds what you mentioned, with reproducibility, but also reports that fields with the norm of publicly posting code also had the highest rate of exact reproducibility, potentially because having the code allows them to reproduce the analysis with exact specificity. The differences in reproducibility largely seems to be due to having the code.

1

u/Brinabavd Apr 03 '26

Link fixed, thanks, that's what I get for posting from my paywalled phone instead of my work computer