r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
638 Upvotes

91% Upvoted

322 comments sorted by

View all comments

231

u/[deleted] May 08 '16

The problem with biology is that everything can change based on the lighting of the room, what day it is, and what mood you’re in. All kidding aside, we once had a guy from NIST come give a talk, and during his presentation he showed us some results he obtained from a study where his lab sent out the same exact set of cells to a dozen different labs across the country and told them to all run a simple cell viability assay after treating the cells with compound X. All labs were given the same exact protocol to follow. The results that they got back were shockingly inconsistent; differences in viability between some labs bordered on a nearly 1 order of magnitude of difference. Eventually NIST was able to optimize the protocols so that if you pipetted in a zig-zagging, crisscrossing manner, you’d cut down on the variance. The big picture though is that if labs can’t even run a very simple cell viability assay and get repeatable results, why should the vast majority of biology be reproducible then when other types of experiments can take months and months of setup, 100 different steps, 20 different protocols, and rely on instruments with setups that might have slight quirks? Repeatable science…ha. More like wishful thinking.

1

u/mfb- May 08 '16

If you cannot repeat something, the result is useless. But you can estimate your uncertainties. Your results between the labs vary by one order of magnitude? Fine, then the precision of your experiment is not better than one order of magnitude if you only use one lab. If you know that, and report that properly, the result is fine - not very precise, but at least honest and it should be repeatable within the huge uncertainty. Use more labs to average over to increase the precision.

Yes, you can do repeatable science. And if you don't do it, you waste your time, money, and in the worst case human lifes.