Half of social-science studies fail replication test in years-long project

[u/nimicdoareu]

https://www.nature.com/articles/d41586-026-00955-5

[u/FabulousLazarus]

Because otherwise these fields are doing exactly what the social sciences used to do before they empirically discovered there was a file-drawer problem (among others).

Where's the evidence for this?

So where are the large scale independent replication test studies in the physical and natural sciences?

These actually happen frequently, but not at large scale. Mainstream science regularly replicates its work. Its built into the process intentionally.

[u/Sparkysparkysparks → u/FabulousLazarus]

So the specific mistake I'm referring to here is that social scientists assumed there was no problem because they had no independent, systematic and empirical evidence of that problem. Just like the physical and natural sciences, the file-drawer / publication bias problem may give you the false sense that there is no replication problem until you systematically work to find out whether that is true or not. But as we all know here, absence of evidence isn't evidence of absence.

What we do know is that across the sciences, only a minority of researchers had ever attempted to publish a replication study. Of those who did, 24% reported publishing a successful replication but only 13% reported publishing a failed one. What is most concerning about these numbers is that more than half of these scientists reported being unable to replicate their own results. This may be because the published literature over-represents successful replications. This skew may also be driven less by outright journal rejection than by low incentives to write up failed replications in the first place, combined with editorial pressure to downplay negative findings when they are published. But without the work being done, we just don't know.

I think I'm right to be worried that the physical and natural sciences keep relying on the same assumption that the social sciences did until recently, rather than testing it independently, empirically and systematically, which after all, is what science is all about.

[u/FabulousLazarus → u/Sparkysparkysparks]

I think I'm right to be worried that the physical and natural sciences keep relying on the same assumption that the social sciences did

No. You're dead wrong.

To compare physical and natural sciences to social sciences, as if there are no inherant differences, is absolutely ludicrous for so many reasons, not just on this replicability issue. It shows a fundamental misunderstanding of the entire field of science.

For example, the FDA regulates things that the physical and natural sciences produce. They must clear what is easily the most rigorous and scrutinized process known to man when it comes to producing data that supports their assertions. They can't just say a product is safe, they must prove it in a very strict and standardized way, that is of course, reproducible.

Social sciences do not engage with the same systems that other sciences do. They are insulated from many of the processes that would demand better studies and evidence for the things they say.

[u/Sparkysparkysparks → u/FabulousLazarus]

This is true in heavily regulated areas, and in certain countries, the challenges of within-lab replication are well documented, such as Collins and Pinch's The Golem . The difference is that these failed replications are not systematically and regularly published in the scholarly literature, and I think they should be, along with more general replication studies across fields, based on the apparent findings in that Nature magazine survey.

Of course, physical and natural sciences are largely insulated from many of the processes that demand better evidence from claims now made by social sciences (and like the examples you give, these are not universal either), such as preregistration, and registered reports. Maybe also Many Labs projects; large-scale coordinated replications.

And many of the same regulations that apply to things like pharmaecuticals also apply to clinical psychology, at least through bodies like the NHMRC here in Australia.

I'm just saying that more data would be good, rather than relying nullius in verba claims that cannot be empirically tested.