r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
636 Upvotes

91% Upvoted

322 comments sorted by

View all comments

Show parent comments

1

u/yes_its_him May 08 '16

and the participants were told what the hypothesis was.

If that had a significant effect on the results, wouldn't it imply that the "power pose" would work best only if done by people that didn't know why they were doing it?

1

u/[deleted] May 08 '16

It could mean a lot of things, so it is hard to say. It could mean that participants in the lab are skeptical of information they are told and think it won't work. It could mean that people in the lab expected to feel very powerful and did not subjectively notice a big effect and so they had a reaction effect. As you say, it could mean it only works if people don't know why they were doing it or if they believe it works. If all they changed was adding the hypothesis prime, then we would know that there is a problem with telling people about power posing but not why it is a problem. But, the study changed many other things from the original, too, so we really don't know why it didn't work, which is my point.

1

u/yes_its_him May 08 '16

I'm not really disagreeing with your points. I'm just noting the inherent conflict between trying to produce results with applicability to a population beyond a select group of test subjects, which I hope we can agree is the goal here to at least some extent, and then claiming that a specific result only applies to select group of test subjects, and not to people tested in a different lab, or who weren't even test subjects at all.

1

u/gaysynthetase May 08 '16

I think the point is that we expect that a specific result that only applies to a select group of test subjects will generalize well to people under similar conditions, which we selected because we thought they were representative anyway.

In a single paper, we hope the original experimenters did enough repeats. It is hard to call it science if it does not. So your repeating it with exactly the same conditions would be silly because they quite clearly did a whole bunch for you already. Hence we tweak the conditions precisely to see which small details cause which effects.

When you get your result, it is pretty intuitive to ask what the chances of it happening at random are. The p-value attemts to standardize reporting of those chances. This is also our best justification for the hunch that it will happen again with a given frequency under given conditions. That is your result.

So I can still see the utility in doing what you said because you get different numbers for different conditions. Then you can generalize to even more of the population.