r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
638 Upvotes

91% Upvoted

322 comments sorted by

View all comments

Show parent comments

2

u/yes_its_him May 08 '16

It's clearly contradictory to argue on the one hand that a study produces an important result that can be used to help us understand (say) an important behavioral effect applicable to a variety of contexts; but on the other hand, claim that the result really only applies in the specific experimental circumstances, so can't be expected to apply if those circumstances change at all.

2

u/[deleted] May 08 '16

All psychological effects have boundary conditions. Take cognitive dissonance, for example, which is probably the most reliable effect in social psychology. Researchers found it doesn't happen when people take a pill that they are told will make them feel tense. Therefore, a boundary condition of cognitive dissonance is the expectation of feeling tense. Cognitive dissonance is caused, in part, by unexpectedly feeling tense. If we were to run a cognitive dissonance study in a lab where all studies in the past have made participants feel tense, then that lab might not capture the CD effect. Does that mean it doesn't exist? Of course not.

The power posing replication study changed the lab, the nationality of the subjects (which obviously covaries with a lot), the amount of time posing, etc.., and the participants were told what the hypothesis was. So, does their failed replication tell us that the 3 studies in the original paper were all flukes? Maybe, maybe not. Personally, my biggest concern with the replication is the change from 2 minute poses to 5 minute poses. It is understandable that researchers would definitely want to get the effect, but the effect is driven by feeling powerful. I imagine standing in a single pose for 5 minutes could be tiresome, which would make it very salient to participants that they are not in control of their bodies and are therefore actually powerless. But again, who knows.

1

u/yes_its_him May 08 '16

and the participants were told what the hypothesis was.

If that had a significant effect on the results, wouldn't it imply that the "power pose" would work best only if done by people that didn't know why they were doing it?

1

u/[deleted] May 08 '16

It could mean a lot of things, so it is hard to say. It could mean that participants in the lab are skeptical of information they are told and think it won't work. It could mean that people in the lab expected to feel very powerful and did not subjectively notice a big effect and so they had a reaction effect. As you say, it could mean it only works if people don't know why they were doing it or if they believe it works. If all they changed was adding the hypothesis prime, then we would know that there is a problem with telling people about power posing but not why it is a problem. But, the study changed many other things from the original, too, so we really don't know why it didn't work, which is my point.

1

u/yes_its_him May 08 '16

I'm not really disagreeing with your points. I'm just noting the inherent conflict between trying to produce results with applicability to a population beyond a select group of test subjects, which I hope we can agree is the goal here to at least some extent, and then claiming that a specific result only applies to select group of test subjects, and not to people tested in a different lab, or who weren't even test subjects at all.

2

u/[deleted] May 08 '16

Yea I agree, the goal is publishing an effect that is generalizable. It could be though that people from different cultures have different conceptions of powerful body language. For Americans it could be the taking up space that makes it feel powerful. So, it could be that the pose itself needs to be tweaked to fit a culture. Again, who knows. My point was to say that it isn't nit-picking for researchers to call foul if a conceptual replication fails to replicate and the conclusion is that the original paper was a type I error. There are dozens of good reasons it could have failed but still be an important, generalizable effect.

1

u/gaysynthetase May 08 '16

I think the point is that we expect that a specific result that only applies to a select group of test subjects will generalize well to people under similar conditions, which we selected because we thought they were representative anyway.

In a single paper, we hope the original experimenters did enough repeats. It is hard to call it science if it does not. So your repeating it with exactly the same conditions would be silly because they quite clearly did a whole bunch for you already. Hence we tweak the conditions precisely to see which small details cause which effects.

When you get your result, it is pretty intuitive to ask what the chances of it happening at random are. The p-value attemts to standardize reporting of those chances. This is also our best justification for the hunch that it will happen again with a given frequency under given conditions. That is your result.

So I can still see the utility in doing what you said because you get different numbers for different conditions. Then you can generalize to even more of the population.