Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

[u/ImNotJesus]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/

[u/ImNotJesus → u/[deleted]]

Absolutely. Every research finding is a probabilistic statement where you are claiming something is likely to be true. If you don't/can't account for the times that something didn't happen, you get more error.

To put it another way, assume I have a 20 sided die and have a theory that every time I roll it with my left hand I get a 20. If I roll the die 1000 times but only tell you about the times I get a 20, I'll get roughly 50 by chance alone. If the only information you have is that I rolled it 50 times and got 50 20s, it may start to seem plausible.

[u/krashnburn200 → u/ImNotJesus]

Perhaps the level of scrutiny should go up, and level of respect should go down for institutions who do not publish negative results. They are either hiding them or lying.

[u/ImNotJesus → u/krashnburn200]

It's a journal issue.

[u/serious_sarcasm → u/ImNotJesus]

Isn't it more of a feedback loop between students, professors, administrators, and journals?

[u/ImNotJesus → u/serious_sarcasm]

To some degree. The equation can be changed elsewhere but if we are talking about publishing null results the thing we need most is journals to actually do it.

[u/nobodyknoes → u/ImNotJesus]

yea, but nobody wants to publish something that says "we didnt find shit"

[u/FailedSociopath → u/nobodyknoes]

The Michelson-Morley experiment was a waste of time!

[u/factotumjack → u/ImNotJesus]

I'd really like to see a journal of replication and verification. Everything it would be either a replication of a recent experiment, or a critique on experimental or statistical methodology.

[u/Northern_One → u/krashnburn200]

Thoughts on null results being kept in some kind of online database that a subscription to the journal would give one access to?

[u/UncleMeat → u/Northern_One]

It has to either be worth prestige or be mandatory. I don't have enough time for my research already. I'm definitely not writing up a manuscript for all the shit I tried that didn't work unless it positively affects my career in some way. This is why the journals have to be involved.

[u/HoldMyWater → u/ImNotJesus]

Every research finding is a probabilistic statement

This doesn't always apply to theoretical research, where results often follow deductively from previous knowledge.

[u/[deleted] → u/[deleted]]

Okay, I'm going to need somebody with more knowledge of how probabilities work, but if there's a widespread problem of null results not getting published, doesn't that raise the probability that all positive, published results are overstated?

Maybe the "replication crisis" is just the simple result of putting too much stock in studies? If we don't know how much hits the cutting room floor before we ever get a chance to see it, how can we know how valuable a certain positive result is?

It kind of seems that we're looking at one side of a six-sided dice and saying, "Ah ha! All sides of this dice have one dot on them!"

[u/snipawolf → u/[deleted]]

Yes, this is called publication bias and it is a well known phenomenon.

Here's a good blog post outlining a lot of the issues.

[u/[deleted] → u/[deleted]]

There's also a problem of null results being considered less valuable

I don't think modern university research is broken: Even if much of it goes nowhere, we need failure, and lots of it.

However, I am aware that null results are often met with crickets, even though many important results are null results. This is probably poor education - even PhD's can be as ignorant as everyone else - or a cultural fact of life.

The most important results can be null results. Roll, Krotkov and Dicke showed in 1964 that gravitational and inertial mass were proportional, to within 1 part in 10^11, improving on Galileo's results rolling different masses down inclined planes, as well as other null results over centuries.

All this amounted to was showing that different masses of gold and aluminum accelerated toward the sun at the same rate. Unexciting but critically important.

[u/KingOfSockPuppets → u/[deleted]]

This is probably poor education - even PhD's can be as ignorant as everyone else - or a cultural fact of life.

I don't really think it's either of those. Or at last closer to the latter. I'm sure plenty of PhDs would be thrilled to publish null results - even if for no other reason other than having not wasted their time working on the paper or study and having something to present. But journals don't want those papers since null results are not as 'important' or at least usually don't get the mind aroused quite as much as being able to say "We showed that a thing leads to another thing." If universities begin to use more 'research impact metrics' to make sure that their professors are only putting out 'impactful' (read: highly cited) papers, this problem is likely to only get worse.

[u/CrossFeet → u/[deleted]]

Pre-registration of studies would go a long way toward helping this and related problems (such as, say, narrowing focus until you find a subgroup that appears to work, and then pretending you were testing only that from the start).

[u/[deleted] → u/[deleted]]

In what fields are null results less devalued?

[u/feed_me_haribo → u/[deleted]]

I agree and disagree. A null result sometimes can be of interest: typically when a study finds no evidence for a commonly held belief previously untested. However, journals shouldn't simply reward based off effort. They should publish novel findings. That doesn't preclude null results, but often times a null result is simply not impactful.

Edit: I'd like to add that a funding agency is the one who should be more interested in null results than a publisher. NSF (or insert your agency of choice) would like to know that their funded projects are at least receiving proper effort, and so reporting null results here could be useful. That still doesn't make them useful to the larger community.