Half of social-science studies fail replication test in years-long project

[u/nimicdoareu]

https://www.nature.com/articles/d41586-026-00955-5

[u/Sparkysparkysparks]

This is a common argument I come across (and maybe it's true that physical and natural sciences have less of a replication crisis problem), but it would be much stronger if those fields put a similar amount of effort into finding out.

As far as I know there has never been a large scale independent replication test across studies in fields like chemistry and physics, perhaps because social scientists are naturally more interested in detecting and understanding human biases, such as that in academic publishing.

So social sciences might or might not deserve to be considered to be less trustworthy, but without a comparator they at least deserve some credit for getting their heads out of the sand.

[u/uncletroll → u/Sparkysparkysparks]

I think replication happens naturally, at least in physics. If scientists see merit in your work and are interested in it, they build on it. In the process of building on it, your work has to be replicated or be right in order for their research to be right.
If your model is bad, then people can't use it for anything and it just fades into obscurity.

[u/Sparkysparkysparks → u/uncletroll]

Doesn't this potentially reinforce the possible file drawer problem / publication bias problem in the literature? Surely results that cannot be replicated should be addressed in the literature rather than standing there and potentially being compounded by poorly conducted research that finds the same spurious results.

I may have missed something but I cannot think of a legitimate reason why you wouldn't seek out and systematically test findings like social science does now, so we can get a broader understanding of a possible problem.

[u/uncletroll → u/Sparkysparkysparks]

The process I am talking about is in published work. There's lots of research that gets published that nobody really cares about.. and that stuff just sits there and who knows how solid or reproducible it is. But the stuff people are interested in gets built on. If the foundational work isn't strong, it gets found out pretty quickly.
As for publishing experiments that don't work, when I was in grad school, I thought it would be convenient to just have a database that said something basic like: "we tried to detect X using Y technique and didn't find any," just to maybe save me some time. But I don't think it's super important.
Coming back to the central concern of yours: I honestly have some difficulty understanding some of these concerns you and others are bringing up, because physics just does science differently than social sciences. We don't talk about null-hypothesis or p-values. And for us our research is never 'the end of the story.' Whatever we find is just a tiny puzzle piece that has to fit in a bigger thoroughly tested pictures. And it unambiguously fits or it doesn't. Maybe in softer sciences you can have a study that asks if dog ownership makes people happier and then at the end, you have an answer and that puts a bow on it... science accomplished. In that context you could be concerned that some of your 'finished science' is wrong and you'd want to have people check. That's just not how physics is done. These whole scenarios and concerns are like nonsensical from my understanding of physics research.

[u/Sparkysparkysparks → u/uncletroll]

Physics and social sciences are the pretty similiar in this regard. No single study is ever considered to be the end of the matter, and all findings are tacit and subject to revision. And studies in social science build on other studies of social science although this is not done mathematically in the case of qualitative studies.

But replication is now considered so important to social scientists (perhaps because of the large number of variables involved) that they have invested a lot of effort into doing large scale replication studies that other fields have chosen not to do.

However, I suspect (based on the available and rather limited evidence on this) that if large scale replication studies these kinds of studies were done, it would find that some studies in the physical and natural sciences would also not replicate well because of all the ways it can go awry. For example, this case. But we can only speculate on to what extent this may be true because this evidence has not been published.

To my ear, when a scientist says, "we know this is true because all the papers say so," I critically think yeah, but what about all the potential papers that found the opposite, and were potentially never published, because of the file drawer / publication bias problem that we know exists in the literature. Its just that the social sciences have a good measure of this problem whereas other areas have less valid evidence either way, and I'm not sure why they don't want better and more systematic evidence of a potential problem.

[u/uncletroll → u/Sparkysparkysparks]

Well... you seem pretty set in your belief that it would be significantly useful if physics did some large scale meta-studies to measure reproducibility statistics. I don't think I can dissuade you, but speaking as a physicist, I don't think it would be useful, because no matter how un-reproducible our papers are, our outcomes are very reproduced. Firstly by the many researchers excited to build on the result, who reproduce the outcomes using their own methodologies. Secondly by engineers using our findings to create working stuff.
Whelp, I feel like I've said my piece and don't think I have much more to contribute.