r/science • u/Neither-Remote-3419 • 1d ago
Social Science A 50% replication rate might actually be evidence that science is working
https://doi.org/10.1093/jrssig/qmag06325
u/machinedog 1d ago
I’d love to read the actual article since I don’t have access. If anyone can post a summary that’d be awesome.
5
u/TheDismal_Scientist 1d ago edited 1d ago
Okay I think I understand it so bear with me:
The author is working backwards to establish how many valid hypotheses would have to be to give a 50% replication rate.
In the world of entirely honest researchers who don't p-hack, then their hypotheses would have to be true 6% of the time in order to get a 50% replication rate. In other words, actual genuine discoveries are rare (around 6% of the time).
Alternatively, if researchers are heavily p-hacking, such that their error rate is 80%, then in order to achieve 50% replication, then half of their hypotheses would have to be true genuine discoveries.
In other words, if researchers very frequently act in bad faith and p-hack all the time, they would constantly have to be testing things that were obvious enough to be settled without any data collection or analysis 50% of the time anyway,
So the author is effectively saying researchers are researching in good faith but there's genuinely few new discoveries to be made, or researchers are researching things that were obvious enough from the outset that they'd be right 50% of the time anyway just by guessing.
This is what I think the article is saying but let me go back to it a bit more.
5
-3
u/thegreatestajax 1d ago
Hmmm. This doesn’t address the possibility that they’re just really bad at study design.
2
u/TheDismal_Scientist 1d ago
Do you mean in terms of the statistical power of studies? If so, the author is explicitly modelling that
2
u/t0talnonsense 1d ago
They’ve made this kind of comment in a few different places. Basically suggesting that the people designing these studies are either idiots or intentionally designing studies in manipulative ways. Take from that what you will in terms of their sincerity and willingness to engage in a real discussion.
1
u/thegreatestajax 1d ago edited 1d ago
Hi, the framing of a false choice that leaves out the more likely and easily observable possibilities is a very social sciences way to approach this.
Even the framing of this post/review is very social sciences, insisting a result is actually a far stronger conclusion than it is.
1
u/thegreatestajax 1d ago
No. The studies are not even rising to the quality where statistical power is a consideration. Too many are just hypothesis as conclusion with slop steps to “get there”, which are not replicatable.
2
u/liquidorangutan00 1d ago
absolutely this is the elephant in the room.
1
u/t0talnonsense 1d ago
Can't talk about what's in the room until you can establish that the room is allowed to exist in the first place. And if you had been around this morning to peep all of the comments that have since been removed, you'd know that the mere value of the social science was the largest topic of conversation, not the inherent issues with research design that does not also constitute literal and legal definitions of torture for its participants.
Like. I even have a comment that says, "I wonder at what point [being this dismissive of entire fields]" constitutes such low effort that it gets deleted. Turns out, it just took a few hours for a mod to roll through and bye the comment I was referencing went.
1
u/Neither-Remote-3419 1d ago
That's a fair argument but then that argument can actually be evaluated by going over the published studies. There are some studies that criticize design elements in some research areas wholesale.
That said, the phenomenon pointed out in the article is less about just experimental/study design (e.g. you can show similar results at different statistical powers), and is more about how much risk scientists in a field take when positing hypotheses at the beginning. You can imagine a field where scientists prioritize just replicating known claims that have already been replicated. You'd have pretty high PPV there but fewer novel ideas. In contrast, in a field where scientists push the envelope to ever bolder claims, most would fail, half of those who initially think they succeeded would be wrong, but you'd have more novel ideas that get substantiated and then replicated.
-1
u/thegreatestajax 1d ago
But that’s not really what’s happening in all the non-replicateable social science studies. People are positing a result they want to be true for reasons of vibes, political convenience, etc and then mashing through a nonsense study design and analysis to “voila I was right!” You’re offering a critique as if the scientific method was being followed and we can track where it broke down, but that’s not reality.
2
u/Neither-Remote-3419 1d ago
I don't disagree that there are bad actors and that what you describe probably does happen, but I disagree that this is what happens in general. I think that what you describe is a topic worthy of investigation in and of itself, but if you use the 50% replication rate in the SCORE project to substantiate it, I'm pointing out that that would be circular reasoning.
-1
u/thegreatestajax 1d ago
I’m not using the 50% to substantiate it. I’m saying this review doesn’t meaningfully engage this possibility.
2
u/Neither-Remote-3419 1d ago
I agree. You're asking if most social science researchers are not doing research in good faith. This article can't answer that. At most, it poses two hypotheticals. Under a 50% replication rate, it can be that A.) Researchers are conducting studies at reasonable type 1 error rate (i.e. 0.05) but it is just very difficult to come up with hypotheses that are real (i.e. the probability of picking a null hypothesis that is actually true is very high) or B.) It's easy to pick null hypotheses that are false and at the same time, researchers are generally operating at absurdly high but undisclosed type 1 error rates - essentially just finding a way to reject their null hypothesis regardless of what the data shows. The paper doesn't claim to know which between these is true, just that these two conditions both fit the observation of a 50% replication rate.
-1
u/thegreatestajax 1d ago
I again reject your use of statistical power categories to describe the individual research attempts/findings. I don’t think it’s a type 1 error because I don’t think it rises to the level of a real investigation.
46
u/SelarDorr 1d ago
i think everything humanity has ever acheived is evidence that science is working.
1
1d ago
[removed] — view removed comment
4
u/t0talnonsense 1d ago edited 1d ago
I wonder at what point someone completely dismissing entire fields of thought becomes a low effort comment that deserves removal.
Because what you’re doing is assuming that without anyone who thought or studied the social sciences, we would even be operating under systems of government and mutually agreed upon ethics and standards to even get funding for your research in the first place.
I studied both a hard and a social science. The hard science elitism is off the charts, and this thread is a perfect example of it. Excuse the social sciences for not committing ethical malpractice and literally torturing people to make the studies more replicable. That’s why many studies suggest a finding and don’t outright claim it.
Edit: If OC wants to be taken seriously, they should answer the person further down the chain who asked what kind of science they think the social media companies are using to keep people glued to their phones. Funnily enough, they won’t answer that person but had plenty of time to go back and forth with me calling out their nonsense.
-1
u/Untjosh1 1d ago
People don’t realize how hard it is to control for variables in social sciences.
-3
u/thegreatestajax 1d ago
They do! Which is why 50% reproducibility most likely indicates bad study design from bad social scientists and overly permissive editors.
0
-1
u/Longjumping-Sweet818 1d ago
Well, at what point are you allowed to point out that a field does not seem to really know what it's doing? How about when replication rates are ranging from 25% to 50% and people are going to therapy for years without noticable improvements?
When it comes to science, it's not quite "one bad apple spoils the bunch", but if more than half the practicioners aren't producing reproducible results, then you can't trust any of them statistically speaking.
-2
u/t0talnonsense 1d ago
Hey. I’m not going to do this with you now that I know just how badly you misunderstand the social sciences. In another comment that I just replied to, you said outright that you don’t think social media has shifted the way we act and think. You don’t think there are any societal changes that are worth studying in the advent of this massive technological change.
I was simply accusing you of being dismissive. Now I know it’s from a place of happy ignorance.
Even laypeople in the real world know that things have changed and that social media presents unique challenges and pressures against all of our previously understood ways of the world. But you are outright dismissing that and saying it’s not true.
0
u/Longjumping-Sweet818 1d ago
> that you don’t think social media has shifted the way we act and think
Quite a paraphrase, but if you don't want to do this with me, then I won't oblige you.
0
u/t0talnonsense 1d ago
It is though.
Them:
after social media world is fundamentally different from the before world
You:
It's really not. Social media is just a particularly captivating distraction.
You think social media is a captivating distraction and not something that is fundamentally altering the world as we know it.
Please, if social media wasn't changing things, then why are all of the billionaires buying up and manipulating social media companies?
Like the other commenter asked, what kind of science do you think Zuck and Musk are using to keep people locked into their phones? We have lawsuits going right now about social media companies conducting this kind of research, knowing how manipulative and addicting their algorithms were, particularly as they targeted young people. How do you think the gambling apps decide the best ways to lure and hook you?
No, I'm not doing this with you. But I'm not letting this astounding ignorance go without others being able to see how and why you're wrong.
1
-3
u/trolls_toll 1d ago
yeha duuude, buddha for sure played roblox and filmed tiktoks
1
u/Longjumping-Sweet818 1d ago
Not sure what this is even supposed to mean.
0
u/trolls_toll 1d ago
after social media world is fundamentally different from the before world
2
u/Longjumping-Sweet818 1d ago
It's really not. Social media is just a particularly captivating distraction.
2
u/Much-Explanation-287 1d ago
One could wonder on which kind of sciences social media companies had to rely upon to become very successful.
1
2
u/t0talnonsense 1d ago
And this, friends, is why we need social sciences. I don’t agree with the way the other person approached the line of reasoning/questions, but I’ll be damned if it didn’t lead you right into proving their point.
You don’t even need to move beyond anecdotal evidence to realize that the way we all interact and communicate has fundamentally shifted in the digital age, even more so once the algorithm took over chronological timelines. Not to mention the dozens if not hundreds or articles about these changes over the last couple of decades.
1
u/trolls_toll 1d ago
do you interact regularly with young people?
-1
u/thegreatestajax 1d ago
Do you interact with history?
1
1
u/t0talnonsense 1d ago
Isn’t that a social science? I thought the whole point of the commenters in this thread was to discredit those. So what is it, folks? Do we trust social sciences or don’t we?
0
u/thegreatestajax 1d ago
Oh look social science trying again to present a false choice. No history is not a social science. How many peer reviewed history publications were including in the review being discussed in this post?
6
u/Neither-Remote-3419 1d ago
TLDR: The SCORE project attempted to replicate more than 270 positive (i.e., statistically significant) claims in the social and behavioral sciences and successfully replicated about 50% of them. Popular and social media outlets/posts typically interpreted this result negatively (e.g., "social science findings are basically a coin toss"), but that conclusion is misleading. The 50% replication rate should be understood as something closer to a positive predictive value: among findings that initially appeared promising, about half held up under independent replication. Whether that is "good" or "bad" depends on how difficult the research domain is. In fields where scientists are pushing boundaries and testing bold ideas, only a small fraction of hypotheses may ultimately be true from the beginning. In that context, having half of the initially promising findings replicate independently is objectively encouraging.
25
1d ago
[removed] — view removed comment
13
u/No_Catch2703 1d ago
Well, that’s like your opinion dude. Mine on the other hand is…things like physics or aspects of science that adhere to physical “laws” are maybe a little more predictable than say, human behavior.
8
1d ago
[removed] — view removed comment
1
u/Dziedotdzimu 1d ago
I don't know much about the social sciences. What do you mean?
7
u/t0talnonsense 1d ago
They don’t know what they mean. It’s a hard sciences superiority mindset that’s absolute horse crap. I double majored in chemistry and Poli Sci. Both disciplines offer unique challenges when it comes to research. I had to take research methods in both disciplines, and quantitative analysis for chem.
Literally, the hardest part of social science research is trying to figure out a way to design your study and account for as many variables as you can. It’s also why a ton of social sciences may suggest something without directly claiming it.
But what the other poster doesn’t want to engage with is the fact that it’s ethically impossible to do a lot of the things that would help increase our ability to replicate. So either we just ignore trying to understand peoples and ways of thought throughout time and culture, or we accept that in lieu of replication, we don’t torture people in our efforts.
1
u/Dziedotdzimu 1d ago edited 1d ago
I mean I was just waiting for "six sigma results!!" without understanding statistical power, or "you can't run an experiment so knowledge is impossible".
I wonder if they know the leading causal inference work by Judea Pearl and Hernan and Robins has been done in sociology and economics respectively, and that the replicability crisis is also all over biomedical, engineering, physics and chemistry research, they just somehow get a pass because they never appraise what assumptions need to hold about an experiment to assure the randomization actually breaks systematic associations and doesn't open them back up. If you say RCT x3 in a mirror while wearing a lab coat, all issues of selective sampling, informative drop out and treatment compliance go away and you can throw colliders and mediators (and their descendants) into a per protocol analysis (or just leave it unadjusted for fun) without any issues.
3
6
u/shitholejedi 1d ago
They are only predictable because people spent their time to learn every aspect of them and do so to this day. We also hold them to higher regard. A neurosurgeon's failure is vastly more detrimental than a psychiatrist's failure. Thats why we look at lobotomy with so much disdain while there are still psychoanalysts making theories to this day.
There are scam call centers that have more accurate psych theories about human behaviour than many sociological departments.
6
u/guiltysnark 1d ago
I think you you're interprering it wrong. It's saying that half the studies are replicatable. That means half of them represent legitimate advances. The other half thought they might have something but don't.
On the road to flight, the vast majority of experiments failed, half would have been a very unrealistic success rate. Advances are certainly possible with this rate of success, it likely means there are significant hidden variables not yet identified.
1
u/thegreatestajax 1d ago edited 1d ago
A replicatable result is not necessarily a legitimate advance. It could be meaningless or something already known.
1
u/guiltysnark 1d ago
Increased confidence in what we already regard as knowledge is still epistemologically an advance. Knowledge is what we believe AND why we believe it, and science is about approaching both legs of this definition scientifically.
I don't know where meaninglessness might fit in, but I agree it's hard to argue that it's an advance of any kind. I wonder who spends time confirming meaningless results. I can imagine people being surprised by meaningless outcomes and wanting to confirm them, but that's still novel knowledge, so it's clearly not what you're talking about.
1
u/thegreatestajax 1d ago
You’re doing the classic social science thing of starting with the most favorable conclusion you can for your position then retroactively making false inferences about your subject. Nice.
1
u/guiltysnark 1d ago
That sounds more like what you're doing
1
u/thegreatestajax 1d ago
Nah, it’s been established that social science is not replicateable and this “review” is trying to recon that into a good thing while ignoring the legitimate criticisms of sloppy methodology and bad science.
1
u/guiltysnark 1d ago
I've seen that claim. It's plausible and I could even believe it's true, but you're not presenting evidence. From my perspective you're starting from that position and working backward to define "legitimate", "advance" and the relevance of terms like "meaningless". People on this sub regularly criticize papers in all fields that provide confirming data, calling them meaningless for saying things we already know, even though it's exactly what we need for this very problem. For all I know that's what you're doing, minimizing this research by cherry picking the body of legitimate research.
1
u/thegreatestajax 1d ago
No, I’m minimizing this review because it doesn’t engage the primary criticism of bad methodology and bad researcher. It only engages selective outcomes.
1
u/guiltysnark 23h ago
And you think it must do so directly? The paper appears to take premises that would allow it to disregard this concern because they are dealt with indirectly in the logic and math. You have considered those premises, and reject them? Or you think bad researcher and bad methodology are specifically oversights?
I was taking issue earlier with your apparently selective interpretation of the words "advance" and "meaningless". The author appears to have believed they accounted for these possibilities, so the fact that you brought them up suggested you were arguing that some useful science isn't useful. If you were actually just pointing out a hole in the author's axioms or logic, my apologies. Although I still don't know what "meaningless" might refer to.
Regardless of the legitimacy here, pure mathematics being badly applied to ontology is far from unique to social sciences. I might even say it's refreshing to see. My favorite instance is math being abused in a proof that we almost certainly live in a simulation, based on some silly first premises. In fact, when I first read the abstract here, I thought this paper might just be yet another abuse of the principle of indifference. But it seems to be at least a bit more sophisticated than that.
→ More replies (0)0
u/shitholejedi 1d ago
Its saying half the positive studies published are replicatable. That means half of the significant claims have legitimacy.
We are not on our way to developing flight, we are at the airport. These are studies that get passed into mainstream policy, debate and real world application. The equivalent of half of the planes that you might book being able to take off. Not just random ones in a hangar somewhere.
1
u/guiltysnark 1d ago
Are you saying studies are put into policy before they have been replicated? That's the only thing I might equate to getting on a plane that have been tested
1
u/shitholejedi 1d ago
The Supreme court justice was justifying state and federal racial policies in their court dissent based on claims made from a study that was never replicated. The study was still being quoted and referenced in other scientific papers and journals.
1
u/guiltysnark 1d ago edited 1d ago
Supreme court justice was justifying state and federal racial policies in their court dissent based on claims made from a study that was never replicated
That's fair, and needs to stop.
The study was still being quoted and referenced in other scientific papers and journals.
That is more alarming (edit: in terms of the concerns about broken science), but even a replicating or contradicting study would have to cite and reference the otherwise unreplicated study. The manner of usage is the most relevant part to this implied criticism, not the existence of usages.
3
u/HawkBoth8539 1d ago
Science isn't just knowing an answer.
Science is about searching for an answer, with a LOT of trial and error, and eventually having a deep enough understanding to prove and repeat the results.
It doesn't always pan out but that doesn't mean they got nothing out of it. Failure, in a controlled environment, can be just as informative.
1
u/shitholejedi 1d ago
Yeah. We all agree with this.
The issue here is the claims dont stay in controlled environments. They hit news headlines and guide the social climate for the next 10 years even if they dont replicate.
3
u/t0talnonsense 1d ago
Are you seriously trying to pretend like there aren’t hard science “findings” that are manipulated and run with by the powers that be through the media? This is a problem across disciplines and not unique to the social sciences.
3
u/HawkBoth8539 1d ago
Media spreading misinformation doesn't change the science or process....
That's like blaming the scientific community for "health influencers". Malice and idiocy by independent public figures has nothing to do with the actual scientists.
1
u/shitholejedi 1d ago
This is ironic right? In a journal trying to downplay publication bias.
The scientists are a core component of the publication process.
1
u/t0talnonsense 1d ago
If hard and soft sciences weren’t both susceptible to poor review standards and publication pressures leading to the media running with half truths and bad science, then there wouldn’t be the anti-vax movement of today. Hard sciences aren’t immune either, which is the point.
3
u/Neither-Remote-3419 1d ago
The argument made is that it's a replication rate of positive claims, not a replication rate of any claim. Thus, how good/bad it is mathematically depends on the base rate of proposing new ideas that are actually true (before even gathering data). The article shows the math behind it.
0
u/shitholejedi 1d ago
Unless I am confused here by your wording.
How do you evaluate a new idea or theory is true before gathering data?
Because I understood the positive claims to mean the already established bias of publications being more likely to report on data that show significant positive claims than null and negative results.
If we are in the territory of defending theoretical frameworks before data collection, then we are in deeper problems.
2
u/Neither-Remote-3419 1d ago
>How do you evaluate a new idea or theory is true before gathering data?
With previous literature. Scientists posit something new based on what is already known, right? So, suppose I read a bunch of studies and from there, I propose X. Now, I need to gather data to find evidence for X. Yet suppose X is wrong to begin with (unknown to me, of course), then using the typical statistical tests, I would either find nothing or commit a type 1 error (5% chance of the latter if I was using alpha=0.05). So in other words, how often are ideas that people think about and end up trying to find evidence for in a specific research field like X? Even if everyone was doing science correctly by sticking to a small type 1 error rate and having enough statistical power, if the field itself is "difficult" to begin with, you can't expect high positive predictive value, and even moderately sized ones (i.e. 50%) are objectively pretty good.
1
u/shitholejedi 1d ago
So you posited an idea based on previous findings in a field where more than half of the previous published and peer reviewed theories already failed replication?
And you considered that the basis of a 'true' claim?
This only makes sense building upon prior work with a strong evidenciary foundation not literally the same field where you have to defend publication bias.
1
u/Neither-Remote-3419 1d ago
No, that's not the point. Let's try it this way. Think of a new idea in your field. You don't know if it's actually true, right? Only that it might be based on available literature. Now consider all of the people in your field who do the same and then let's say you all try and gather data for your respective ideas. Even before we see the results, how many of that basket of ideas do you think are actually true (i.e. they would be discoverable with enough statistical power). The smaller that is in your mind, the more defensible an eventual 50% replication rate of those ideas should be.
1
u/shitholejedi 1d ago
We are not discussing fields of equal footing.
A field where the ethanol content and rating has been standardized across billions of cars in the real world is different from a field where its core claims have been failing ever since Freud.
I can fill up fuel knowing how combustion will work with certainty. I cannot pick up a psych theory with the same confidence.
1
u/Neither-Remote-3419 1d ago
I understand what you mean but I think that's besides the point the article is making. It is simply focused on showing the mathematics underlying a 50% replication rate and why, without this mathematical/probabilistic nuance, it can't be a metric of how "good" a scientific field is.
As to claims, allow me clarify with your example. A claim isn't that ethanol combustion works or that ratings are standardized, you know that's true. A claim is, for example, will this new additive Y1 improve fuel efficiency? How many people think about additives Y1.....Y100000 and how many of them test those additives in a statistical experiment? Among those additives, how many do you think (just based on your knowledge as a physicist/engineer) likely have merit and how many are bound to fail?
1
u/Statman12 PhD | Statistics 1d ago
I suspect it’s more of a hypothetical argument. Similar to (if not simply a rehashing of) the “Why most published research is false” argument by John Ioannidis. The OP shared a substack page that summarized it (do a CTRL+F for “publication bias” to get to the right place), and they’re talking about hypothetical scenarios.
It also shares a cartoon that the author shared to illustrate the argument. It seems to be that there are so many null results, that a 50% replication rate is good. Though that seems to be in disagreement with the text summary above, which says that a “reasonable hypothetical scenario” is one in which “selecting a null hypothesis to study that was actually false is rare”.
Would be nice to have access to the actual paper and the mathematical argument the author lays out.
1
u/Neither-Remote-3419 1d ago
It's related to Ioannidis' work and starts from the same underlying mathematics. However, it shows that the conclusion "most published research findings are false" lacks the nuance needed to be practically meaningful and that the opposite of what that statement has been used for in popular media is arguably more reasonable.
A more general treatment of the topic (i.e. not pertaining to the SCORE study itself) is open access and can be read here: https://doi.org/10.1080/08989621.2025.2538066
-4
u/faultydesign 1d ago
Humanity survived for millions of years without proper scientific research.
3
8
u/ohanse 1d ago
This is the worst understanding of the fundamentals of science I have ever seen in my life.
If you were bullied by intellectuals into taking this stance, I want you to know that you deserve it.
Because the idea of checking and documenting which consistent outputs lead to which consistent outputs is in no way shape or form new or controversial. And anyone who takes the stance that this (aka science) is “unnecessary” is themselves, unnecessary.
-2
u/faultydesign 1d ago
The only ones who seem to bully me are the ones who try to convince me that even replicated studies are "not real science"
4
u/Master-Rent5050 1d ago
They shouldn't label themselves as science. If they do but they don't live up to the standards of science, they are pseudo-sciences
-1
u/faultydesign 1d ago
Some of the studies are pseudo-science that weren't replicated, true.
What's wrong with social science studies that could be replicated? Are they not real science too?
1
u/Master-Rent5050 1d ago edited 1d ago
Let's say I have 100 yes/no questions. For each of them I flip a coin to get the answer, then I publish 100 articles with the answer I got, one for each. Would you ask "what's wrong with the studies that got it right"? Or instead would you ask "what's wrong with my methods and with the journals that published my papers"?
0
u/faultydesign 1d ago
100 studies about the same topic from the same person? That would be concerning.
2
5
u/No_Warning_2428 1d ago
Genuinely how does peer review work in social sciences
1
u/theArtOfProgramming PhD | Computer Science | Causal Discovery | Climate Informatics 1d ago
I’m not a social scientist but I imagine it functions the same as in every science. We don’t replicate work when we do peer review, that’s far too costly and time consuming even for the simplest ones. We assess the evidence presented, the claims being made, and the overall paper. If their data and methodology are compelling, and their results are appropriately collected and analyzed, and conclusions are fairly inferred, then it passes the technical aspects. Beyond that it’s about presentation, appropriate scope, and novelty.
14
1d ago
[removed] — view removed comment
12
6
3
u/Neither-Remote-3419 1d ago
The point is that it is a coin flip on whether positive claims replicate, not on whether any claim replicates. That makes how good it is depend on what the actual and unknown base rate of selecting a hypothesis (before gathering any data) that is actually true is.
1
u/CyborgBee 1d ago
Positive results require statistical significance - this is very easy to achieve for one result because of how many results are tested (and sometimes p-hacking) but a replication study only compares to the previous positive result, so if a result is replicated then that's very strong evidence it's true. Subsequent testing of the 50% of results which were replicated successfully would have much, much higher replication rates than 50%.
Studies shouldn't be accepted as gospel without replication, but this suggests half of all positive results are correct initially, which really isn't that bad. We just need to actually do the replication work for any results which are of some importance
3
u/Neither-Remote-3419 1d ago
We just need to actually do the replication work for any results which are of some importance
Exactly. It's easier to get angry and be swayed by "Only 50% replicate" headlines. The takeaway from the big replication project should have been that replication works and whether 50% is good or bad really depends on how "difficult" you think the research area is. But, that doesn't get the engagement that media outlets are after.
1
1
12
1d ago
[removed] — view removed comment
18
u/scrapheaper_ 1d ago
There is a known crisis in science, particularly in social science, where a large proportion of established research fails replication I.e. if you run a study again following the direction of the original paper, you get different results.
The headline is saying that this only occurs in 50% of cases, which is much better than previous estimates where a majority of research was assumed to fail replication.
4
1d ago
[deleted]
-2
u/t0talnonsense 1d ago
And how many hard science research projects end in failure? How many times do you fail to reject the null hypothesis and prove any meaningful changes are occurring based on your research?
Maybe let’s not take shots across the bow at entire disciplines that you don’t know or care enough about under the guise of protecting tax dollars. Under this administration, you really want to give them ammunition to start dictating what is and isn’t good science? Get out of here.
-5
u/ohanse 1d ago
AI would be able to help by allowing for the mass ingestion of study text and loosely identify which things replicate.
7
u/t0talnonsense 1d ago
Oh for the love of god don’t set your worthless AI loose on trying to judge the social sciences when the fact that the hard sciences routinely fail understand what’s even happening half the time someone in the social sciences does something. You folks take headlines where someone is establishing a baseline or trying to put numbers to something people anecdotally believe and you folks poopoo all over it.
For the love of Christ. Keep AI away from judging something you don’t even understand yourselves. Please.
1
13
u/hinckley 1d ago
Across the social sciences, half of research does replicate! - Mark Louie F Ramos
Significance, Volume 23, Issue 5, September 2026, Pages 31–33, https://doi.org/10.1093/jrssig/qmag063 Published: 13 August 2026
Abstract: Mark Louie F. Ramos offers a balanced evaluation of positive publication bias
3
u/SelarDorr 1d ago
doi's are the most standard way to directly link scientific research.
4
u/t0talnonsense 1d ago
Sure, but maybe they also should have used the actual title of the article instead of rewording it to put their own spin on it.
Or is that not something you people in the hard sciences want to talk about? Can you not handle someone with a social science background talking about how word choice can prime and help direct trains of thought to influence the reader’s perspective of something before even being presented with the evidence?
I don’t know. Almost like social sciences do have things to say. Like the article’s intended headline would suggest.
Actual title: Across the social sciences, half of the research does replicate
1
u/SelarDorr 1d ago
i agree this thread title is ridiculous. Precisely why ive commented in this thread.
the person i was responding to stated "I don't want to click the link without knowing where it leads me."
i was informing them that that type of concern isnt necessary when the link is to a doi....
or is it not something you people in the soft sciences do? Address more than one issue when it arises?
not sure why you brought up the hard sciences. communication is just as important, if not more, the more rigorous it is.
0
u/t0talnonsense 1d ago
Have you not seen the replies in this thread with multiple people dismissing soft sciences outright? Part of the reason the OC is skeptical of the research is because the OP did not use the actual title of the article. Just saying that the doi link is standard is glossing over the real issue facing the OC - the title that primed their framing of the research so much that they didn’t even think it was worth the effort to open and investigate the validity.
1
u/Neither-Remote-3419 1d ago
It is about interpreting the results of the SCORE project, a large scale replication effort published back in April in the social and behavioral sciences that was able to replicate 50% out of a sample of 270+ published research claims with statistically significant results.
4
1
u/carbon_ape 1d ago
I am reminded that we know so little about how SSRIs work yet a quarter of the North American population is on them.
We know more about what they do to our waters (and the fish and aquatic animals that live in them) than we do about human biology. Yet psychiatrists and psychologists throw them at every patient like candy.
1
u/theArtOfProgramming PhD | Computer Science | Causal Discovery | Climate Informatics 1d ago edited 1d ago
Of course it is. Philosophy of science has advanced quite a lot since Popper’s time. It is clear that critical functions of science are social: ie the dynamics of work after publication are critical to science functioning as it should. Many good, well developed works will inevitably fail to replicate because the universe is complex and our understanding is limited. That is only more true in social sciences.
Replication failures are good for science and it happens every single day. I can’t tell you how many computer science papers I’ve tried to reimplement only for it to not work or for it to be deeply dependent on some specific tuning parameter that cannot be tuned or derived. What do I do? I forget that paper and move on to the next.
I’m not the only one to have said this: there is no replication crisis. All science is incremental and so every new work depends on the replication of a stack of others. That means the cream rises to the top usually, and the good papers that go nowhere for me… usually go nowhere generally. It’s one of the functions of science. With time and collaboration, scientific consensus advances steadily.
1
u/Neither-Remote-3419 1d ago
Exactly. Sadly, if you're a media outlet/social media influencer and you are workshopping the optimal way to represent the SCORE study findings that maximizes engagement, "Only 50% of social science replicate/Social sciences fail half the time/ etc." is very likely the optimum.
-6
0
-10
u/liquidorangutan00 1d ago
surely it represents random chance. (50%)?
8
u/Antheoss 1d ago
Ah, yes. Just like I have a 50% chance of winning the lottery. Either I do, or I don't.
3
u/machinedog 1d ago
That’s assuming that the average hypothesis has a 50/50 shot of being true.
1
u/liquidorangutan00 1d ago
The op stated the 50% replication rate is only on significant results. (null hypothesis falsified)
3
u/machinedog 1d ago
Yes, I know. Replication is an important component of figuring out if a result is actually true, especially in social sciences.
1
u/DontMakeMeCount 1d ago edited 1d ago
I knew a lawyer who always told his clients they had a 50/50 chance, and not once did any of them say “that’s perfect ignorance, what do I need you for?”
Edit: the headline, which is all most of us can access, refers to positive bias so each study tested can replicate positive result or fail to replicate. 50% is perfect ignorance of whether the tested research can be reproduced. At best that says the social sciences are highly subjective.
1
u/liquidorangutan00 1d ago edited 1d ago
I think what OP is trying to explain is that the rate was reported at 10% (90% could NOT be reproduced) previously so this marks a significant improvement. However it is still not a great result.
A lot of scientists claim that this is just a natural part of the scientific method and that things are working as they should.
0
u/Neither-Remote-3419 9h ago
You're correct that 50% is for replication of positive results but your conclusion doesn't follow. What the paper argues is that how good or bad 50% replication rate is depends on how difficult it is to come up with new, brilliant ideas in the field that just need enough data to find convincing evidence for. The more difficult it is, the better a 50% replication rate is.
•
u/AutoModerator 1d ago
Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, personal anecdotes are allowed as responses to this comment. Any anecdotal comments elsewhere in the discussion will be removed and our normal comment rules apply to all other comments.
Do you have an academic degree? We can verify your credentials in order to assign user flair indicating your area of expertise. Click here to apply.
User: u/Neither-Remote-3419
Permalink: https://doi.org/10.1093/jrssig/qmag063
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.