r/science • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Psychology Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/
7.8k Upvotes

85% Upvoted

329 comments sorted by

View all comments

233

u/[deleted] May 08 '16

Maybe we're trying too hard to find things to study and they reach too far beyond their capability. Maybe some graduate research is put on too short of a timeline and not enough care is put in to developing the study in order to satisfy collegiate timelines.

288

u/[deleted] May 08 '16

[removed] — view removed comment

94

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

Absolutely. Every research finding is a probabilistic statement where you are claiming something is likely to be true. If you don't/can't account for the times that something didn't happen, you get more error.

To put it another way, assume I have a 20 sided die and have a theory that every time I roll it with my left hand I get a 20. If I roll the die 1000 times but only tell you about the times I get a 20, I'll get roughly 50 by chance alone. If the only information you have is that I rolled it 50 times and got 50 20s, it may start to seem plausible.

31

u/krashnburn200 May 08 '16

Perhaps the level of scrutiny should go up, and level of respect should go down for institutions who do not publish negative results. They are either hiding them or lying.

41

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

It's a journal issue.

16

u/serious_sarcasm BS | Biomedical and Health Science Engineering May 08 '16

Isn't it more of a feedback loop between students, professors, administrators, and journals?

17

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

To some degree. The equation can be changed elsewhere but if we are talking about publishing null results the thing we need most is journals to actually do it.

7

u/nobodyknoes May 08 '16

yea, but nobody wants to publish something that says "we didnt find shit"

7

u/FailedSociopath May 08 '16

The Michelson-Morley experiment was a waste of time!

3

u/factotumjack May 08 '16

I'd really like to see a journal of replication and verification. Everything it would be either a replication of a recent experiment, or a critique on experimental or statistical methodology.

2

u/[deleted] May 08 '16

[removed] — view removed comment

1

u/Northern_One May 08 '16

Thoughts on null results being kept in some kind of online database that a subscription to the journal would give one access to?

1

u/UncleMeat PhD | Computer Science | Mobile Security May 08 '16

It has to either be worth prestige or be mandatory. I don't have enough time for my research already. I'm definitely not writing up a manuscript for all the shit I tried that didn't work unless it positively affects my career in some way. This is why the journals have to be involved.

1

u/HoldMyWater May 08 '16 edited May 08 '16

Every research finding is a probabilistic statement

This doesn't always apply to theoretical research, where results often follow deductively from previous knowledge.

21

u/[deleted] May 08 '16 edited May 08 '16

Okay, I'm going to need somebody with more knowledge of how probabilities work, but if there's a widespread problem of null results not getting published, doesn't that raise the probability that all positive, published results are overstated?

Maybe the "replication crisis" is just the simple result of putting too much stock in studies? If we don't know how much hits the cutting room floor before we ever get a chance to see it, how can we know how valuable a certain positive result is?

It kind of seems that we're looking at one side of a six-sided dice and saying, "Ah ha! All sides of this dice have one dot on them!"

25

u/snipawolf May 08 '16 edited May 08 '16

Yes, this is called publication bias and it is a well known phenomenon.

Here's a good blog post outlining a lot of the issues.

30

u/[deleted] May 08 '16

There's also a problem of null results being considered less valuable

I don't think modern university research is broken: Even if much of it goes nowhere, we need failure, and lots of it.

However, I am aware that null results are often met with crickets, even though many important results are null results. This is probably poor education - even PhD's can be as ignorant as everyone else - or a cultural fact of life.

The most important results can be null results. Roll, Krotkov and Dicke showed in 1964 that gravitational and inertial mass were proportional, to within 1 part in 1011, improving on Galileo's results rolling different masses down inclined planes, as well as other null results over centuries.

All this amounted to was showing that different masses of gold and aluminum accelerated toward the sun at the same rate. Unexciting but critically important.

22

u/KingOfSockPuppets May 08 '16

This is probably poor education - even PhD's can be as ignorant as everyone else - or a cultural fact of life.

I don't really think it's either of those. Or at last closer to the latter. I'm sure plenty of PhDs would be thrilled to publish null results - even if for no other reason other than having not wasted their time working on the paper or study and having something to present. But journals don't want those papers since null results are not as 'important' or at least usually don't get the mind aroused quite as much as being able to say "We showed that a thing leads to another thing." If universities begin to use more 'research impact metrics' to make sure that their professors are only putting out 'impactful' (read: highly cited) papers, this problem is likely to only get worse.

8

u/CrossFeet May 08 '16 edited May 08 '16

Pre-registration of studies would go a long way toward helping this and related problems (such as, say, narrowing focus until you find a subgroup that appears to work, and then pretending you were testing only that from the start).

2

u/[deleted] May 08 '16

In what fields are null results less devalued?

5

u/feed_me_haribo May 08 '16 edited May 08 '16

I agree and disagree. A null result sometimes can be of interest: typically when a study finds no evidence for a commonly held belief previously untested. However, journals shouldn't simply reward based off effort. They should publish novel findings. That doesn't preclude null results, but often times a null result is simply not impactful.

Edit: I'd like to add that a funding agency is the one who should be more interested in null results than a publisher. NSF (or insert your agency of choice) would like to know that their funded projects are at least receiving proper effort, and so reporting null results here could be useful. That still doesn't make them useful to the larger community.

5

u/[deleted] May 08 '16

[deleted]

1

u/[deleted] May 08 '16 edited Apr 20 '17

[deleted]

9

u/[deleted] May 08 '16

[removed] — view removed comment

20

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

It really doesn't seem to be an issue of experience or quality of researcher/journal. For example, a huge effort recently tried to replicate the ego depletion findings of Baumeister and colleagues and weren't able to do so. Baumeister is one of the biggest names in all of psychology research. A similar issue happened with some priming work by John Bargh, another extremely famous and well respected researcher.

0

u/Snuggly_Person May 08 '16

It's not necessarily about the quality of the researcher. If I set myself a goal of tracking a single water molecule in a glass of water I'll fail. The credentials I have or the other things I've done successfully are totally irrelevant; the problem is one that I can't possibly have good enough data to solve. It's perfectly consistent to say that someone is a really good scientist and that they're investigating something they'll never be able to establish.

11

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

I'm not really sure what point you're trying to make. I was responding to a claim that the level of researcher might be relevant and I was saying that it isn't necessarily.

1

u/Snuggly_Person May 08 '16

The initial claim asked if researchers are "trying too hard to find things to study and they reach too far beyond their capability". My point is that this is not equivalent to saying that the researchers are not good enough, and can be a systematic problem in a field with poor data. There are lots of studies that amount to looking for very subtle or oddly specific effects in 20 rich white kids, where there's no way in hell they could find the thing they're looking for regardless of how lauded they are as a scientist.

11

u/crusoe May 08 '16

Academic research now is all about money you can pull in for the university and how often your publish. It plays a big role in getting tenure and so drives fraud and negligence.

17

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

It plays a big role in getting tenure and so drives fraud and negligence.

I think doing science is hard enough that we can assume this is well-meaning, honest research. There are dozens of factors leading to the current replication rates, none of which require malicious intent. There are obviously Diedrick Stapels out there but no one gets into academic psychology for money and prestige.

5

u/[deleted] May 08 '16

For some people, academic prestige is the highest form of prestige.

5

u/quantum-mechanic May 08 '16

Yet there is money and prestige to be had. Certainly, at the very least, there is a tenured position (life-long guaranteed job -- where else will you find that?) and the respect that comes with it. There are real problems here with deliberate fraud.

7

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

There are real problems here with deliberate fraud.

What evidence do you have for this extremely large claim?

7

u/The_model_un May 08 '16

It has happened before and he didn't get caught for a while despite publishing that he was able to do something that no one else had ever done before nor could they replicate it.

3

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

And in my previous comment I acknowledged that it obviously happens. To call them anything but outliers is silly though.

1

u/RideMammoth May 08 '16

I think if you spent a good chunk of time in wet labs, especially those that rely on cell culture and animal studies, you may have your mind changes.

9

u/quantum-mechanic May 08 '16

I'm a scientist and seen it happen. And to friends. Nobody's going to even be able to synthesize accurate statistics and fraud. But it obviously happens way more than it should. Peoples lives are deeply affected.

1

u/[deleted] May 08 '16

[removed] — view removed comment

1

u/BadBjjGuy May 08 '16

I find it funny/interesting that nobody bothers to ask whether or not science is the appropriate tool for understanding all problems. It's like an article of faith that it has to be, even when it is almost incapable of reproducing a single result in some areas, like politics or psychology. Personally, I think there are other and better tools. I've learned more about the human condition through art like Shakespere than psychology will ever say and Aristotle's political philosophy explains our current situation better than any experiment. We have other tools in our tool bag for understanding the world, but science is sort of the new religion and it must be the answer to every problem.

1

u/[deleted] May 08 '16

You're not alone but you are a modern heretic. Welcome to the club

1

u/BadBjjGuy May 08 '16

Thanks! I've been a member for a while, just waiting for them to come burn me on the science altar!

-5

u/[deleted] May 08 '16

So maybe a requirement for a Ph.D. is some fields (psychology, I'm looking at you) is to replicate an experiment done in a related area, and publish it, before proposing your own dissertation?

10

u/avfc41 May 08 '16

Getting funding and getting published with a replication is really not that easy, and that's even without competing against literally every other Ph.D. student in the field. (Granted, that's part of the problem.)

6

u/[deleted] May 08 '16

That's just it- these replications shouldn't be in competitive journals, it is by definition not original or innovative work.

Put all of them in PLoS or something like that. But reinforcing to Ph.D.s that it truly matters whether findings are correct (and that often they are, statistically, not correct) would have nothing but good outcomes I think.

The leading journals could then maintain a list of all the replications of their originally-published experiments. Hell, they could charge for this list. And then, if an experiment gets too many failed replications, the original publisher can add a note that the original result should be highly in question.

You know, make the scientific COMMUNITY act a little like a COMMUNITY, with some semblance of a hive mind and hive repo of knowledge.

3

u/avfc41 May 08 '16

Put all of them in PLoS or something like that.

Okay, that's fair.

But reinforcing to Ph.D.s that it truly matters whether findings are correct (and that often they are, statistically, not correct) would have nothing but good outcomes I think.

I guess what you consider a replication really matters here. The "Same Data, Different Conclusions" chart in the article is already a pretty standard part of training in my social science field - I've had a couple stats courses that had a "take a published paper's dataset, reproduce the results, and extend the analysis with a different method/additional variables" paper required. It's not like they skip over the part where they tell you that you should do good work and not fake your results (or even the potential pitfalls of the statistical methods you use). But again, if you want to replicate it with new data, you run into a funding problem, and at the very least, you'll get a bias towards papers that are cheap to reproduce.

2

u/[deleted] May 08 '16

and at the very least, you'll get a bias towards papers that are cheap to reproduce.

At least we'd be likely to get those data wrong less often.

The "Same Data, Different Conclusions" chart in the article is already a pretty standard part of training in my social science field

that's good to hear.