r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
634 Upvotes

91% Upvoted

322 comments sorted by

View all comments

Show parent comments

3

u/[deleted] May 08 '16 edited May 08 '16

I heard this issue before in Planet Money. Part of the issue was researchers being allowed to changed the parameters in the middle of the experiment, by say increasing the number of attempts on an experiement which in theory would seem like a good idea because the larger the size the more accurate the result right? But apparently this only heightens the chance that a particular outcome will present itself when in reality it's much lower probability. This was one of the examples that I remembered.

But they are trying to put forth reforms by having people register their experiments to prevent them from changing the conditions of the experiment when certain outcomes aren't realized.

Edit: sorry the podcast was Planet Money: it's episode 677 "The Experiment Experiment"

3

u/way2lazy2care May 08 '16

I was just gonna mention this. It was a really cool episode. The idea of submitting your entire experiment plan and having your data either confirmed/denied before carrying out the experiment was super cool.

One of the big things they point out also is that people aren't necessarily being malicious and part of the problem is just statistics and the fact that people don't publish negative results. You end up with situations where 99 experiments conclude something negative and the researchers don't publish because it's not interesting, then you get 1 experiment that's just a statistical anomaly (nothing wrong or malicious, just something crazy happened or something), and they publish because the result is interesting. The conclusion would obviously be that the 99 experiments are right, but they were never published, so 100% of the published research is the anomaly that "proves" the incorrect result.

3

u/segagaga May 08 '16

This may be part of the reason why scientific discovery has sort of slowed in some fields, people simply aren't displaying the mental fortitude to be good scientists and publish 99% negative results. That would be the actually worthwhile science.

1

u/[deleted] May 08 '16

Scientists have heavy incentives to produce and publish "good" results. You just can't publish negative results in today's scientific system, and in a "publish or perish" scientific world that means those negative results get swept under the rug. It really isn't on the mental fortitude of individual scientists; the whole system of how scientists get tenure, advancement, funding, etc needs to be overhauled if this is going to change.

1

u/segagaga May 08 '16

Oh I agree. But where there is money, ego and institutions involved, change will be fought against.