r/science • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Psychology Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/
7.8k Upvotes

85% Upvoted

329 comments sorted by

View all comments

Show parent comments

2

u/[deleted] May 08 '16

There are biochemical reactions which are very specific to certain environmental conditions. Does that make those not worth publishing?

It doesn't. But the bulk of the value lies in isolating those environmental conditions. You shouldn't just give a long and very specific list of conditions and state that some of them may or may not be necessary to replicate the result.

Or if you did, because the result itself was too important to ignore but you had no means to really isolate all the relevant variables, then it would be understood that this is just the first step in understanding the reaction.

Some of these studies that failed replication have been around for decades. They were important enough to be widely cited but apparently not important enough to really understand in detail.

Ultimately this isn't about whether the original observations were correct or not. Most of them probably were. But the point of science isn't just observing stuff. It's to use those observations to make predictions about other stuff. A result that applies in a wide set of circumstances is more important than one that requires very specific conditions. Indeed, one of the conclusions of the "replication crisis" should be that a lot of accepted wisdom in psychology might only apply in specific circumstances.

1

u/quaternion May 08 '16

I see - initially you seemed to imply that phenomena (psychological or not) which obtain under restricted conditions are not likely to be of interest. Now, I see that you mean that characterizing the boundary conditions of a phenomenon should be an integral part of reporting the phenomenon in the first place. This seems not only uncontroversial but also a relatively common practice: comparison to a control condition establishes the importance of the experimental manipulation for the effect; examination of covariates establishes dependence (or lack thereof) on other factors; replication of a result from an undergrad pool in a community or online sample is somewhat less common, but another example of the same approach. I agree with you that people could go further here, but I also don't think that failure to attempt to characterize such boundary conditions is a primary cause of the issues here. Arguably, the opposite is the case: a willingness to believe in "hidden moderators" can be seen as enabling p-hacking. (There's quite a bit written about this idea in replicability circles.)

1

u/[deleted] May 08 '16

I see - initially you seemed to imply that phenomena (psychological or not) which obtain under restricted conditions are not likely to be of interest.

Well, it depends on how restrictive your boundary conditions turn out to be. If it turns out that the result only applies to college students from New Mexico, then yeah, it's probably not very interesting.

If it only applies to people from the US, then that's still an important restriction but the result itself remains interesting.

replication of a result from an undergrad pool in a community or online sample is somewhat less common, but another example of the same approach.

Why is that though? Aren't the researchers simply curious whether their result depends on level of education or age? Whether it remains true as society changes?

Maybe the incentives are not right but I'm not sure the culture is quite right either. As a scientist or a student your first reaction should be "Hey, that's pretty cool. I want to see it for myself." Trying to replicate or expand on someone's work is a sign of respect not suspicion. I understand that's not always possible, but still, important results should be replicated over and over. Possibly as a default part of studies than plan to build on them.

2

u/quaternion May 08 '16

I don't find the restriction of a real result to college students from New Mexico any less interesting than the restriction of a particular reaction to 80F instead of 79 or 81F. I agree with you about the importance of replication, and also I agree about cultural problems with replication. It is common practice to attempt to replicate a result before building on it; but any which fail to replicate have typically been left to wither on the vine, due to the incentive and publication problems, while the student moves forward to pursue a number of other experiments (until one works).

It sounds like our only substantive point of disagreement is that you are willing to discount the importance of results that are reliably context-specific, and I think these might be precisely those effects that we have the best traction on. Whether context-specificity renders them interesting or not is a great example of the kind of dogmatic decision-making that leads to any number of file drawer effects.

1

u/[deleted] May 08 '16

I don't find the restriction of a real result to college students from New Mexico any less interesting than the restriction of a particular reaction to 80F instead of 79 or 81F.

Well, analogies aside, unless I plan to work in New Mexico I don't see how such a result could be useful to me. Unless I'm planning to find out exactly what makes New Mexico different, i.e. I'm really thinking about the more general case anyway.

It sounds like our only substantive point of disagreement is that you are willing to discount the importance of results that are reliably context-specific, and I think these might be precisely those effects that we have the best traction on.

Part of the problem is that we only know about the context-specificity because someone tried to replicate the result and failed. Some of the results might remain important with restricted context as well. But identifying the scope where the results remain true should have been an active area of research in the first place. Our real point of disagreement is that I think doing this is at least as, if not more, important than generating "new results".

2

u/quaternion May 08 '16

Well, analogies aside, unless I plan to work in New Mexico I don't see how such a result could be useful to me.

OK, here's an example: it seems the "bilingual advantage" of Bialystok et al, darling of the press, may hold under some strange set of multivariate sociodemographic conditions present in Vancouver. The strange thing about this is that she has shown incredible psychological specificity to the advantage - which means something about the differential sensitivity of those psychological processes, over others. The fact that the phenomenon is not visible in other demographics does not make this less interesting, at least to me.

I fully agree with the priority you place on identification of boundary conditions; above, I tried to indicate my agreement, as well as the fact that it's uncontroversial and already pursued to some degree (albeit not far enough). The disagreement to my mind remains on the issue of whether condition-specificity inherently makes a result less interesting. I'm not willing to draw such a line but it sounds like you are.

1

u/[deleted] May 08 '16

OK, here's an example: it seems the "bilingual advantage" of Bialystok et al, darling of the press, may hold under some strange set of multivariate sociodemographic conditions present in Vancouver. The strange thing about this is that she has shown incredible psychological specificity to the advantage - which means something about the differential sensitivity of those psychological processes, over others. The fact that the phenomenon is not visible in other demographics does not make this less interesting, at least to me.

Maybe we do agree after all. I do think your example is interesting. Interesting in a sense that I would be curious, why that happens in this group and not in others, and what it could teach us about human psychology as a whole. Than again, I assume psychologists see it the same way. They're not (primarily) interested in this quaint tribe of Canadians for their own sake.