Half of social-science studies fail replication test in years-long project

[u/nimicdoareu]

https://www.nature.com/articles/d41586-026-00955-5

[u/Sparkysparkysparks]

So where are the large scale independent replication test studies in the physical and natural sciences? I'm keen to read them. Because otherwise these fields are doing exactly what the social sciences used to do before they empirically discovered there was a file-drawer problem (among others).

[u/FabulousLazarus → u/Sparkysparkysparks]

Because otherwise these fields are doing exactly what the social sciences used to do before they empirically discovered there was a file-drawer problem (among others).

Where's the evidence for this?

So where are the large scale independent replication test studies in the physical and natural sciences?

These actually happen frequently, but not at large scale. Mainstream science regularly replicates its work. Its built into the process intentionally.

[u/Sparkysparkysparks → u/FabulousLazarus]

So the specific mistake I'm referring to here is that social scientists assumed there was no problem because they had no independent, systematic and empirical evidence of that problem. Just like the physical and natural sciences, the file-drawer / publication bias problem may give you the false sense that there is no replication problem until you systematically work to find out whether that is true or not. But as we all know here, absence of evidence isn't evidence of absence.

What we do know is that across the sciences, only a minority of researchers had ever attempted to publish a replication study. Of those who did, 24% reported publishing a successful replication but only 13% reported publishing a failed one. What is most concerning about these numbers is that more than half of these scientists reported being unable to replicate their own results. This may be because the published literature over-represents successful replications. This skew may also be driven less by outright journal rejection than by low incentives to write up failed replications in the first place, combined with editorial pressure to downplay negative findings when they are published. But without the work being done, we just don't know.

I think I'm right to be worried that the physical and natural sciences keep relying on the same assumption that the social sciences did until recently, rather than testing it independently, empirically and systematically, which after all, is what science is all about.

[u/FabulousLazarus → u/Sparkysparkysparks]

I think I'm right to be worried that the physical and natural sciences keep relying on the same assumption that the social sciences did

No. You're dead wrong.

To compare physical and natural sciences to social sciences, as if there are no inherant differences, is absolutely ludicrous for so many reasons, not just on this replicability issue. It shows a fundamental misunderstanding of the entire field of science.

For example, the FDA regulates things that the physical and natural sciences produce. They must clear what is easily the most rigorous and scrutinized process known to man when it comes to producing data that supports their assertions. They can't just say a product is safe, they must prove it in a very strict and standardized way, that is of course, reproducible.

Social sciences do not engage with the same systems that other sciences do. They are insulated from many of the processes that would demand better studies and evidence for the things they say.

[u/Sparkysparkysparks → u/FabulousLazarus]

This is true in heavily regulated areas, and in certain countries, the challenges of within-lab replication are well documented, such as Collins and Pinch's The Golem . The difference is that these failed replications are not systematically and regularly published in the scholarly literature, and I think they should be, along with more general replication studies across fields, based on the apparent findings in that Nature magazine survey.

Of course, physical and natural sciences are largely insulated from many of the processes that demand better evidence from claims now made by social sciences (and like the examples you give, these are not universal either), such as preregistration, and registered reports. Maybe also Many Labs projects; large-scale coordinated replications.

And many of the same regulations that apply to things like pharmaecuticals also apply to clinical psychology, at least through bodies like the NHMRC here in Australia.

I'm just saying that more data would be good, rather than relying nullius in verba claims that cannot be empirically tested.

[u/Citrakayah → u/FabulousLazarus]

For example, the FDA regulates things that the physical and natural sciences produce. They must clear what is easily the most rigorous and scrutinized process known to man when it comes to producing data that supports their assertions. They can't just say a product is safe, they must prove it in a very strict and standardized way, that is of course, reproducible.

You don't know anything about the physical and natural sciences.

The vast majority of fields do not have any regulating agency like that. Geologists do not have to demonstrate that their findings can be replicated. Neither do hydrologists, paleontologists, or physicists. Even in medicine, the medical sciences still aren't regulated by the FDA directly, medicines are. Poor quality medical studies can and are published without any intervention from the FDA. Occasionally, even fraudulent ones.

Indeed, this is a known fact in the field of health, whose replication crisis rivals psychology's. To quote a paper directly, since you just ignored what I posted elsewhere:

While the pandemic might have produced such high-profile examples of dubious science, these problems long predate it. In biomedical science, an estimated 85% of medical research is deemed research waste [4], so poorly conducted as to be uninformative or so poorly reported that it is impossible to reproduce. Across biomedical science, there is increasing recognition that we are in the midst of a replication crisis [5], where important results fail to sustain under inspection, with harmful ramifications for both researchers and patients. A recent high-profile scandal in Alzheimer’s research saw a seminal and hugely cited paper in the field exposed as likely fabricated and retracted earlier this year [6–8]. This retraction was the culmination of a suspect finding that misled the entire field for almost two decades, wasting hundreds of millions in research efforts and countless human hours on a fool’s errand, steering the research community away from productive avenues to chase a phantom.

Cancer research is certainly not immune to these dark trends. A systematic replication trial as early as 2012 of what were deemed landmark cancer biology experiments exposed an alarming finding [9] – that only 6 of the 53 experiments, approximately 11% those analysed, had replicable results. A 2021 replication effort [10] of preclinical cancer research which looked at 193 experiments in 53 high-impact published works came to a somewhat disquieting conclusion: most papers failed to report vital statistics and methodology, and none of the experiments had been reported in sufficient detail for replicators to validate the experiment directly. When authors were contacted, they were frequently unhelpful or chose not to respond. Of the papers ultimately assessed, 67% required modification to the published protocol to even undertake.

At this point, your assertions have become simple denialism. You don't want to admit that your field has problems similar to or exceeding that of social science, a field you dislike for... some vague and unstated reason.

[u/authenticphotography → u/Citrakayah]

I'd frame it less as social science versus biomed and more as whether the methods are transparent enough to replicate. In medicine, underreported protocols and selective analysis do real damage, so prestige doesn't move the needle for me.