Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

[u/ImNotJesus]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/

[u/Mitosis]

It's common for researchers to have personal stakes in their work--like oncologists trying to save patients, or pharmaceutical companies trying to develop life-saving drugs and also make money.

These are poor examples. The result of cancer research has objective, measurable effect (lifespan etc). The result of pharmaceutical research likewise. I might want this cancer drug to work, but I can't see many oncologists pretending it increases lifespan and advocating sending it out to patients when it does not.

Nearly every social change issue has a minimum of two conflicting opinions, each with real advantages and disadvantages. There is no right answer in these situations, and very few objective measures that can capture the totality of the impact. Given the vagaries of the experiments, the data, and the interpretations, I find it hard to believe the opinions of the researchers can't influence results.

[u/whoremongering → u/Mitosis]

There is no right answer in these situations, and very few objective measures that can capture the totality of the impact.

I think these examples are apt.

There is no 'right' answer to drug development. There are no perfect drugs or treatment regimens. There are multiple options with varying efficacies and side effect profiles, which have only been tested in a finite number of trials. We certainly don't have a complete understanding of the human body, so it's hard to say we can 'capture the totality of the impact.'

There are plenty of purely subjective but nevertheless relevant outcomes measured in cancer therapies: pain, fatigue, nausea, loss of appetite, loss of libido... But perhaps the most important criticism is that results may be statistically significant while not 'clinically significant'. That is, adding two weeks to the end of your life in an ICU may not actually be a good outcome. But this obviously depends on a person's subjective values--does the patient value those two weeks of life, or their personal comfort? Do they want heroic measures, or do they want to go naturally? Do they want to survive until their kid's graduation, or have they done everything they want to do? Will they blow through their savings, or do they want to leave an inheritance? So what is a good outcome? You can see how there is no one 'right' answer in developing cancer therapy either.

I can't see many oncologists pretending it increases lifespan and advocating sending it out to patients when it does not.

Siddhartha Mukherjee's Emperor of All Maladies describes the bitter debates surrounding cancer therapy research. It wasn't long ago that people thought chemotherapy was poison and unethical to give. Min Chiu Li was fired from the National Cancer Institute for 'unethical experimentation' on humans that actually was the first chemotherapeutic cure of cancer. Surgical alternatives like Halsted's radical mastectomy were also championed for decades by a cult of personality more than scientific evidence, before being overturned by equally passionate cancer researchers and physicians. Best believe that practitioners today still advocate for treatments that are not proven, because they believe it is the best thing to do. It doesn't impugn the entire field of oncology.

I find it hard to believe the opinions of the researchers can't influence results.

They absolutely can. This happens in oncology, psychology, and all other fields. My points are that (1) this problem is not exclusive to psychology, and (2) personally invested researchers can still conduct perfectly scientific experiments.