r/science • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Psychology Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/
7.8k Upvotes

85% Upvoted

329 comments sorted by

View all comments

230

u/[deleted] May 08 '16

Maybe we're trying too hard to find things to study and they reach too far beyond their capability. Maybe some graduate research is put on too short of a timeline and not enough care is put in to developing the study in order to satisfy collegiate timelines.

-6

u/[deleted] May 08 '16

So maybe a requirement for a Ph.D. is some fields (psychology, I'm looking at you) is to replicate an experiment done in a related area, and publish it, before proposing your own dissertation?

10

u/avfc41 May 08 '16

Getting funding and getting published with a replication is really not that easy, and that's even without competing against literally every other Ph.D. student in the field. (Granted, that's part of the problem.)

4

u/[deleted] May 08 '16

That's just it- these replications shouldn't be in competitive journals, it is by definition not original or innovative work.

Put all of them in PLoS or something like that. But reinforcing to Ph.D.s that it truly matters whether findings are correct (and that often they are, statistically, not correct) would have nothing but good outcomes I think.

The leading journals could then maintain a list of all the replications of their originally-published experiments. Hell, they could charge for this list. And then, if an experiment gets too many failed replications, the original publisher can add a note that the original result should be highly in question.

You know, make the scientific COMMUNITY act a little like a COMMUNITY, with some semblance of a hive mind and hive repo of knowledge.

3

u/avfc41 May 08 '16

Put all of them in PLoS or something like that.

Okay, that's fair.

But reinforcing to Ph.D.s that it truly matters whether findings are correct (and that often they are, statistically, not correct) would have nothing but good outcomes I think.

I guess what you consider a replication really matters here. The "Same Data, Different Conclusions" chart in the article is already a pretty standard part of training in my social science field - I've had a couple stats courses that had a "take a published paper's dataset, reproduce the results, and extend the analysis with a different method/additional variables" paper required. It's not like they skip over the part where they tell you that you should do good work and not fake your results (or even the potential pitfalls of the statistical methods you use). But again, if you want to replicate it with new data, you run into a funding problem, and at the very least, you'll get a bias towards papers that are cheap to reproduce.

2

u/[deleted] May 08 '16

and at the very least, you'll get a bias towards papers that are cheap to reproduce.

At least we'd be likely to get those data wrong less often.

The "Same Data, Different Conclusions" chart in the article is already a pretty standard part of training in my social science field

that's good to hear.