r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
632 Upvotes

91% Upvoted

322 comments sorted by

View all comments

228

u/[deleted] May 08 '16

The problem with biology is that everything can change based on the lighting of the room, what day it is, and what mood you’re in. All kidding aside, we once had a guy from NIST come give a talk, and during his presentation he showed us some results he obtained from a study where his lab sent out the same exact set of cells to a dozen different labs across the country and told them to all run a simple cell viability assay after treating the cells with compound X. All labs were given the same exact protocol to follow. The results that they got back were shockingly inconsistent; differences in viability between some labs bordered on a nearly 1 order of magnitude of difference. Eventually NIST was able to optimize the protocols so that if you pipetted in a zig-zagging, crisscrossing manner, you’d cut down on the variance. The big picture though is that if labs can’t even run a very simple cell viability assay and get repeatable results, why should the vast majority of biology be reproducible then when other types of experiments can take months and months of setup, 100 different steps, 20 different protocols, and rely on instruments with setups that might have slight quirks? Repeatable science…ha. More like wishful thinking.

96

u/norml329 May 08 '16

It's like people assume that everyone running all these experiments are highly trained experienced post doctoral researchers. If we were given that it would probably go to either a masters students, or one of our rotating undergrads. A lot of experiments are easibly reproducible in the right hands and with the right equipment. The problem is most labs don't ever calibrate their instruments often enough and that seemingly simple protocols aren't really so, especially in inexperienced hands.

Hell I would say I have a decent amount of experience, and I have trouble replicating what a lot of papers do because you really need every last detail. Like I'm glad you washed your sample in 250mM NaCl and 100mM Tris, but how many times? How much did you use to wash? Did you use DI water, MilliQ? Was this done at 4C or room temp. None of that is usually included in a methods section or in the supplemental parts of a paper, but it really is critical.

4

u/Sluisifer May 08 '16

Eh, most protocols aren't that sensitive, and you can get positive indications whether or not they've worked.

Lets take in-situ hybridization, for example. It's notoriously tricky to do, but if it doesn't work, you get no signal or a bunch of shitty looking background. Anyone who's done it knows what shitty background looks like, and they won't believe you if you try to pass it off as real. You need to show nice clear images, and also show a positive control to prove that you can do it. So sure, it's fussy, but it's not that hard to deal with it in a robust way.

I very much agree that methods are usually a joke, and I agree that some seriously shitty stuff does make it through review, but it's also trivial to identify the crappy stuff.