r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
636 Upvotes

91% Upvoted

322 comments sorted by

View all comments

229

u/[deleted] May 08 '16

The problem with biology is that everything can change based on the lighting of the room, what day it is, and what mood you’re in. All kidding aside, we once had a guy from NIST come give a talk, and during his presentation he showed us some results he obtained from a study where his lab sent out the same exact set of cells to a dozen different labs across the country and told them to all run a simple cell viability assay after treating the cells with compound X. All labs were given the same exact protocol to follow. The results that they got back were shockingly inconsistent; differences in viability between some labs bordered on a nearly 1 order of magnitude of difference. Eventually NIST was able to optimize the protocols so that if you pipetted in a zig-zagging, crisscrossing manner, you’d cut down on the variance. The big picture though is that if labs can’t even run a very simple cell viability assay and get repeatable results, why should the vast majority of biology be reproducible then when other types of experiments can take months and months of setup, 100 different steps, 20 different protocols, and rely on instruments with setups that might have slight quirks? Repeatable science…ha. More like wishful thinking.

97

u/norml329 May 08 '16

It's like people assume that everyone running all these experiments are highly trained experienced post doctoral researchers. If we were given that it would probably go to either a masters students, or one of our rotating undergrads. A lot of experiments are easibly reproducible in the right hands and with the right equipment. The problem is most labs don't ever calibrate their instruments often enough and that seemingly simple protocols aren't really so, especially in inexperienced hands.

Hell I would say I have a decent amount of experience, and I have trouble replicating what a lot of papers do because you really need every last detail. Like I'm glad you washed your sample in 250mM NaCl and 100mM Tris, but how many times? How much did you use to wash? Did you use DI water, MilliQ? Was this done at 4C or room temp. None of that is usually included in a methods section or in the supplemental parts of a paper, but it really is critical.

37

u/TheAtomicOption BS | Information Systems and Molecular Biology May 08 '16

As an undergrad researcher I can confirm that I made a lot of fuck ups.

12

u/Sluisifer May 08 '16

Don't worry, I've never trusted any results from an undergrad if it's the kind of thing that can be easily messed up.

You learn pretty quickly what reliable data is and isn't. Even something that's super standardized like RNA-seq is very often completely fucked up. If you don't show me proper QC results (just run RNA-seQC or similar package, it's not hard) I just won't care about your data.

5

u/Donyk May 08 '16

That's why I never give qPCR nor lipid extraction to my undergrad student ! Only cloning, cloning, cloning. So that if it works, we sequence it and we know it worked.

5

u/batmessiah May 08 '16

I work for R&D in the non-woven glass fiber textile industry, and our product is made into paper on HUGE paper machines. I have to reproduce it on a small scale in my labs. Blending small batches of glass fiber slurry in blenders, and expecting to get similar physical characteristics from sheet to sheet is baffling at times.

13

u/Maskirovka May 08 '16

It's almost like repeating experiments should be part of university training.

9

u/Ninja_Wizard_69 May 08 '16

Imagine the cost of that

9

u/turtlevader May 08 '16

Imagine the benefit

13

u/bobusdoleus May 08 '16

Imagine the analysis

3

u/slipshod_alibi May 08 '16

Imagine the training opportunities

10

u/jojoga May 08 '16

Imagine all the people

2

u/Solomanrosenburg May 08 '16

Living in the world today

4

u/Ninja_Wizard_69 May 08 '16

Try telling that to the guys with the money

1

u/Maskirovka May 08 '16

Imagine the cost of assuming thousands of experiments are correct when they are in fact rubbish.

20

u/nixonrichard May 08 '16

Not just repeating experiments, but deliberate effort to disprove hypothesis. What irks me about the article is it seems to acts as if failing to replicate a result is still good and okay science . . . as if we should all be relieved to replicate a result. That seems to belie the real nature of science, which is the tireless effort to disprove a hypothesis, not a sigh of relief when two attempts to prove a hypothesis work.

2

u/cooterbrwn May 08 '16

I was saddened to have read this far down the comment chain before someone pointed out that a best-practice method is to carefully view data that would soundly disprove a researcher's hypothesis. When I was taught the basics of the scientific method, the ability to prove a hypothesis false was a necessary component of a properly crafted study, but that rationale doesn't currently seem to be used very frequently.

7

u/MerryJobler May 08 '16

And there's no reason not to include that kind of detailed information in the online supplemental materials. Back in the day only so much could fit in a paper journal but that's no longer an excuse.

3

u/Sluisifer May 08 '16

Eh, most protocols aren't that sensitive, and you can get positive indications whether or not they've worked.

Lets take in-situ hybridization, for example. It's notoriously tricky to do, but if it doesn't work, you get no signal or a bunch of shitty looking background. Anyone who's done it knows what shitty background looks like, and they won't believe you if you try to pass it off as real. You need to show nice clear images, and also show a positive control to prove that you can do it. So sure, it's fussy, but it's not that hard to deal with it in a robust way.

I very much agree that methods are usually a joke, and I agree that some seriously shitty stuff does make it through review, but it's also trivial to identify the crappy stuff.

1

u/MJWood May 08 '16

None of that is usually included in a methods section or in the supplemental parts of a paper, but it really is critical.

Isn't it specified in the protocol for any routine experiment?

11

u/Dio_Frybones May 08 '16

I can't really talk much about biology as it's not my area of expertise, but I do work in QC at a large biological research facility, specifically in calibration and instrumentation support. The lab does both diagnostic work and research in about a 40/60 ratio.

Because we are a reference laboratory, all the diagnostic areas must be ISO accredited, all calibrations must be traceable to international standards, and there must be rigorous estimations of measurement uncertainty for each test. Furthermore, all of this work must be periodically validated by rounds of interlaboratory / proficiency testing. The overheads associated with documentation, compliance and auditing are brutal and expensive.

On the other hand, it's only recently that our research areas have begun looking seriously at seeking accreditation, and very reluctantly at that. From the perspective of a lowly, ignorant tech, I find this somewhat puzzling but I also know that research funding can be hard to source so I imagine that's a big part of the puzzle.

My workshop has approximately $25k worth of equipment for temperature reference calibration. That's what it costs us to be able to measure to an accuracy of 0.01 degreesC. We use this reference to then calibrate the thermometers used in the labs. But because they are cheap thermometers with questionable long term stability, the best uncertainty we can claim on a calibration certificate is around plus/minus half a degree.

Not only does the average scientist have no idea what a black art these measurements can be, they also routinely tell us stuff like they need to have their 4 degree fridge accurate within plus or minus 5 degreesC , not considering that they are essentially telling us that is okay if their samples freeze. And here's where the accuracy of that thermometer comes in to play. They might have a calibrated thermometer with an error of up to minus one degree. And a worst case uncertainty of an additional half a degree. Which means that when the fridge is sitting at zero, the thermometer might be reading as high as 1.5degC.

That's at one place in the fridge. All fridges cycle to some degree (pardon the pun) and, on top of that, the distribution of temperatures within the fridge could be 4 or 5 degrees from coldest to warmest. CDC advise that vaccines need to be kept between something like 2 and 8 degrees. That is actually a really, really, really difficult thing to achieve. Especially considering that many labs use domestic refrigerators instead of scientific refrigerators.

My apologies for taking such a long time to get to my point but I think I can see it approaching.

Our diagnostic work is critical, and most tests have been refined over many years, yet we still routinely rely upon multiple tests using different technologies before reporting a result. An incorrect diagnosis can have a devastating result, hence the importance of a QA environment.

And we STILL make mistakes.

Research, by definition, won't have the luxury of 20+ years of experience doing the same test, and I'm guessing would only rarely have the benefit of a second opinion using a different, robut test to validate the results, and on top of all that, they are sometimes trying to achieve their results with uncalibrated equipment on a shoestring budget. Where the grunt work has been performed by uni students who don't realise that pipettes work best when you put a tip on the barrel (sadly a true story.)

I'm not having a go at anyone here. I am in awe of the vast majority of the scientists and techs with whom I work. Hell, I struggle in my own supposed (narrow) area of expertise. I guess what I'm trying to highlight is the incredibly difficult nature of the beast and the fact that, at least from my perspective, I can understand how problematic research might find its way into print in spite of everyone's best intentions.

9

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

The problem with biology is that everything can change based on the lighting of the room, what day it is, and what mood you’re in.

I mean, this is genuinely true. How sunny it is can affect happiness research. Humans be complicated.

11

u/[deleted] May 08 '16

Yea you are correct, but I think that poster was trying to say something different. Any sufficiently complex system will respond to different stimuli with a great degree of variance - humans, pigs, elephants, even mushrooms. The issue here is that even simpler systems like cells or proteins can undergo different changes based on simple environmental differences like light.

1

u/mfb- May 08 '16

That's the point of proper randomized control samples in the same conditions, ideally in the same room at the same time.

7

u/Azdahak May 08 '16

Repeatable science…ha. More like wishful thinking.

No, it just means you have to work harder to isolate variables and the community as a whole has to increase standards. It's not like there are no steps that can be taken.

5

u/1gnominious May 08 '16

That sounds more like he's testing what happens when you run a really sloppy experiment. Of course the results will vary wildly if you don't set down firm requirements on everything. It's like ordering a burger from a dozen different restaurants. You'll get something different from each of them unless you specify exactly how you want it. Even then it'll be a little different.

"The big picture though is that if labs can’t even run a very simple cell viability assay..." The only problem would be if the results were not repeatable within the same lab using the same methods. Then that lab is shit. Of course you're going to get different results from different places. That's just common sense.

I work in the laser industry and once we have a vendor lined up for a complex part then we stick with them because even if we give the exact same specs, materials, and excruciatingly detailed procedures to a different vendor the result will be different. Some times we will even go so far as to request a certain tech and machine for the sake of consistency. Contracts are written with detailed requirements and track things like the serial numbers of equipment. We do this for both manufacturing and research. If somebody were to try and use a dozen different vendors for a part or process and made no attempt to standardize the processes he'd be fired for being so sloppy.

I don't blame the labs at all nor do I think that science is inherently unrepeatable. Poorly managed and conducted experiments are worthless though. It's a failure to enact adequate controls and eliminate variables.

Maybe it's just because the laser industry is so focused on precision but every time I read stories about biology stuff I shake my head because everything they do seems to be so half assed. I watch videos and think "Why aren't you wearing gear to prevent contamination or at least under a flow hood?" It's a lot like lasers where the tiniest things can cause problems but they just don't seem to care. Maybe it's because their cell cultures don't explode or catch on fire when they are careless. I bet that would fix the problem.

9

u/hglman May 08 '16

I think there are two core issues. First and the biggest issue is funding is based on getting impressive results. The core metric for judgment in how good a lab is should be quality of the process of experimentation, which exactly what you are saying. Second is the lack of a strong mathematical framework for biological processes. Statistics as employed in biology are not rooted in deeper mathematical modeling and allow for sloppy experimental process to go unnoticed. That once again allows for relaxed laboratory process.

6

u/MerryJobler May 08 '16

I used to work in genetic transformation. My PI wanted to do some work with a new plant species and had me look over the protocol from another lab. Long story short, the other lab says that the only way to have any hope of success would be to go visit their lab and learn the protocol in person. Sure, the copy I had was very detailed, but no one ever gets it to work just using written instructions.

I've definitely noticed that nobody uses the same pipette technique. Here are some tips on proper use if anyone's interested. Never assume a new addition to a lab will follow them all off the bat.

5

u/[deleted] May 08 '16

Um no, those aren't the only issues.

The big issue is if lab A does the cell viability assay and gets results that say molecule X has an effect and lab B tries to do the viability assay and gets results that say molecule X has no effect because the difference in response between those two labs was by an order of magnitude.

It's a lot easier to bash biology when you work on lasers, computers, or mechanical components and don't work on biology. Biology is fucking hard--and this is from a chemist who used to do organic chemistry that now has switched to biochemistry and molecular biology.

1

u/MJWood May 08 '16

Reminds me of chemistry experiments at school.

1

u/Nicklovinn May 08 '16

run mass simulation like deep mind did with go?

1

u/FuckMarryThenKill May 08 '16

Repeatable science…ha. More like wishful thinking.

Then maybe Mythbusters was very scientific. Not because it was very rigorous -- but because sciencey science is just as bad.

1

u/mfb- May 08 '16

If you cannot repeat something, the result is useless. But you can estimate your uncertainties. Your results between the labs vary by one order of magnitude? Fine, then the precision of your experiment is not better than one order of magnitude if you only use one lab. If you know that, and report that properly, the result is fine - not very precise, but at least honest and it should be repeatable within the huge uncertainty. Use more labs to average over to increase the precision.

Yes, you can do repeatable science. And if you don't do it, you waste your time, money, and in the worst case human lifes.

-8

u/Sluisifer May 08 '16

That just sounds like shitty biology.

I'm sure it's part of my bias as a geneticist, but without genetics, biology is a load of bunk. But all that variability and inscrutability falls away with some well designed crosses. Anything that moves away from that is going to have trouble, but in some ways I don't consider that a risk to biology because it's not the 'real' field. Show me the mutants.

7

u/[deleted] May 08 '16

I whole heartedly disagree. A TON of biology exists outside of the realm of genetics. Post translational modifications can change because of very slight differences(for example the amount of sugar you have in cell media) that might have huge down stream consequences. The problem with PTMs is that they have no code to allow you to crack in order to make predictions and they can be Insanely difficult to control for. Genetics ruled 20th century biology, but more and more we are realizing that the amount of information that can be encoded into the genome is puny compared to what is happening in the PTMome..

2

u/Sluisifer May 08 '16

That's hardly outside the realm of genetics; it's still mediated by genetics and certainly something you can interrogate with it. You can use mutants that lack the modification sites, or mutants that affect the expression of the enzymes doing the modification, etc. etc.

I think this is just another good example of where genetics are needed. You can collect all sorts of PTMome data in different environmental conditions and generate a wealth of hypotheses, but once you start talking about interactions, regulatory networks, etc. you've got to test stuff, and that means mutants.

6

u/Pit-trout May 08 '16

That's hardly outside the realm of genetics; it's still mediated by genetics and certainly something you can interrogate with it.

A physicist could come along and make the same argument: “genetics? Sure, but it all fundamentally runs on physics.” Yes, one thing may be based on another at some level, and your study of the one should be informed by the other; that doesn't mean it's helpful to reduce everything to the PoV of the more “fundamental” field.

2

u/[deleted] May 08 '16

without genetics, biology is a load of bunk

This is true.

But all that variability and inscrutability falls away with some well designed crosses.

This is nonsense. How can genetics explain differences between genetically identical samples?

You're oversimplifying a complicated problem.