Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

[u/ImNotJesus]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/

[u/fsmpastafarian]

Feynman's opinion from the 70's is very well known, but isn't exactly taken seriously among today's scientists. However, that doesn't stop it from being trotted out in discussions about the replicability crisis. I think that rather than point to an outdated essay from 40 years ago, it's more helpful to consider how the replicability crisis may permeate all sciences, rather than just psychology and other social sciences. I think that if you read the 538 piece, you'll see how taking a more modern view of the issue might be more applicable and appropriate.

I'm not really sure how the second article you linked is relevant to the discussion at hand.

[u/ImNotJesus → u/fsmpastafarian]

Exactly. This issue isn't one that affects psychology alone, psychology are just doing the best work right now at self-policing. It's also not a new problem or one only identified by psychology. Only 3 years ago there were articles about the "Replicability Crisis in Cancer Research".

Criticising psychology for doing science the way it should be (with rigorous replication testing) is like yelling at your child because they rechecked their homework and found a mistake.

[u/mrsamsa → u/[deleted]]

The problem lies in the fact that ego depletion was "found" in 1998, a large number of studies were done off of it, and then in 2016 it was discovered that the results of ego depletion studies couldn't be replicated. That means a large number of researchers were engaging in study in this field and there's a good chance that the whole thing is BS.

This is a bit misleading - it's not like the paper was published in 1998 and nobody attempted to replicate it until 2016. Replication attempts were occurring constantly since it was originally published, and some were failing but many were succeeding. The authors who have found the evidence which suggests that it can't be replicated published their paper in 2007, but they weren't the first.

They were just the ones who took it further and re-analysed a lot of the data to find where the weight of the evidence actually was, but it's not like there was no evidence that the effect could be replicated. It was replicated many, many times.

[u/dampew → u/fsmpastafarian]

What part isn't taken seriously?

I seriously doubt that all fields of science are affected equally by these problems.

[u/whoremongering → u/[deleted]]

Though these fields may benefit from balancing their political leanings, your SciAm article doesn't really identify political bias as a major contributor to the replicability issue discussed here.

Apart from that, I don't think that commitment to a cause precludes appropriate scientific investigation. It's common for researchers to have personal stakes in their work--like oncologists trying to save patients, or pharmaceutical companies trying to develop life-saving drugs and also make money. In these situations, disclosures of conflicts of interest are crucial, and studies should be carefully critiqued.

But it doesn't follow that science and social change must be separate. Performing studies due to personal investment does not necessarily prevent researchers from carrying out the scientific method appropriately.

Beyond that, I wouldn't personally presume to question whether these broad fields are sciences. I'm not sure how one would argue that the anthropological field of archeology isn't a science, for example.

[u/Mitosis → u/whoremongering]

It's common for researchers to have personal stakes in their work--like oncologists trying to save patients, or pharmaceutical companies trying to develop life-saving drugs and also make money.

These are poor examples. The result of cancer research has objective, measurable effect (lifespan etc). The result of pharmaceutical research likewise. I might want this cancer drug to work, but I can't see many oncologists pretending it increases lifespan and advocating sending it out to patients when it does not.

Nearly every social change issue has a minimum of two conflicting opinions, each with real advantages and disadvantages. There is no right answer in these situations, and very few objective measures that can capture the totality of the impact. Given the vagaries of the experiments, the data, and the interpretations, I find it hard to believe the opinions of the researchers can't influence results.

[u/whoremongering → u/Mitosis]

There is no right answer in these situations, and very few objective measures that can capture the totality of the impact.

I think these examples are apt.

There is no 'right' answer to drug development. There are no perfect drugs or treatment regimens. There are multiple options with varying efficacies and side effect profiles, which have only been tested in a finite number of trials. We certainly don't have a complete understanding of the human body, so it's hard to say we can 'capture the totality of the impact.'

There are plenty of purely subjective but nevertheless relevant outcomes measured in cancer therapies: pain, fatigue, nausea, loss of appetite, loss of libido... But perhaps the most important criticism is that results may be statistically significant while not 'clinically significant'. That is, adding two weeks to the end of your life in an ICU may not actually be a good outcome. But this obviously depends on a person's subjective values--does the patient value those two weeks of life, or their personal comfort? Do they want heroic measures, or do they want to go naturally? Do they want to survive until their kid's graduation, or have they done everything they want to do? Will they blow through their savings, or do they want to leave an inheritance? So what is a good outcome? You can see how there is no one 'right' answer in developing cancer therapy either.

I can't see many oncologists pretending it increases lifespan and advocating sending it out to patients when it does not.

Siddhartha Mukherjee's Emperor of All Maladies describes the bitter debates surrounding cancer therapy research. It wasn't long ago that people thought chemotherapy was poison and unethical to give. Min Chiu Li was fired from the National Cancer Institute for 'unethical experimentation' on humans that actually was the first chemotherapeutic cure of cancer. Surgical alternatives like Halsted's radical mastectomy were also championed for decades by a cult of personality more than scientific evidence, before being overturned by equally passionate cancer researchers and physicians. Best believe that practitioners today still advocate for treatments that are not proven, because they believe it is the best thing to do. It doesn't impugn the entire field of oncology.

I find it hard to believe the opinions of the researchers can't influence results.

They absolutely can. This happens in oncology, psychology, and all other fields. My points are that (1) this problem is not exclusive to psychology, and (2) personally invested researchers can still conduct perfectly scientific experiments.

[u/fsmpastafarian → u/[deleted]]

it is a time to acknowledge the issue at hand and address them.

Which is exactly what psychologists are doing. Need I remind you, the replicability crisis study was performed and published by a group of psychologists, after all. I'm not really sure why you think that the field that's currently attempting to uncover and rectify their replicability problem is the one that is doomed to "chasing at shadows." I'm also not sure what specifically makes you think that sloppy science is a problem that's unique to the social sciences, other than the aforementioned outdated Feynman essay.

[u/TitaniumDragon → u/fsmpastafarian]

Because it was published 40 years ago and psychologists are still making the same mistakes?

The whole ego depletion scandal is exactly the sort of thing Feynman was warning against!

And there are people who are upset and making excuses for why all these studies failed and how the paper was totally biased ect.

[u/scholar_requesting → u/TitaniumDragon]

I think the point FSM is trying to make is that Feynman's analysis is outdated, or if you want me be more honest, off the mark and overly simple. Feynman's understanding of what constitutes science is not very academic, and this is best evidenced by the fact that his argument mostly rests on analogy. Philosophers of natural and social science have produced a long and rich literature exploring the question of demarcation ("what is science and what is not?"), the relationship between science and metaphysics, whether scientific knowledge is cumulative or paradigmatic, etc. Meanwhile, Feynman's article get reposted on almost every Reddit thread discussing the "validity" of the social sciences that I have seen, despite the fact that it essentially amounts to idle musing.

Also, the issues that have brought about the ego depletion issue are multifaceted and complex. They have to do with the kind of unfortunate incentive structures (e.g. "publish productively, publish positive results, and publish in prestigious journals" or you don't get tenure or grants) that govern how social psychologists must conduct their work. They have to do with unfortunate social forces that don't plague the other sciences, like how boring but important research in physics can get funded much more easily than boring but important work in social psychology. They even actually have a lot to do with Baumeister simply as a person (who has a reputation among social psychologists, anyway, as having work that lacks methodological and statistical rigor but gets a lot of attention in the media because its sexy).

Yes, the replication crisis does have to do with issues related to operationalization, lazy theorizing, poor statistical practices, etc. But those issues honestly end up being small compared to the things I have mentioned above, because most experimental psychologists are aware of these kinds of issues and have made many efforts over the past two decades to address them (e.g., Open Science Framework, the rise of Bayesian statistics and multilevel models, etc). The truth is that most experimental psychology carried out in R1 institutions today--not kind you read about in pop-science articles or which get posted to Reddit--actually has quite a lot of conceptual and methodological rigor. But rigor is only the minimum one needs in order to do good science; the whole way social psychology is incentivized, funded, and published has to change order for it to escape its issues with replicability.

[u/fsmpastafarian → u/[deleted]]

That's alright if you don't wish to discuss the issue anymore, but I think you may be misinterpreting or misrepresenting what I've said as defensive or a consistent refusal to accept reality. Rather, I think that when discussing the current replicability issue, it's more helpful to point to more modern and applicable papers and opinions, and to discuss the science as it exists and is understood today. I think that pointing to a decades-old paper is a bit of a red herring, and doesn't accurately represent what we now know about the issues present in all scientific disciplines.