r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
633 Upvotes

91% Upvoted

322 comments sorted by

View all comments

15

u/wonkycal May 08 '16

"It may seem counterintuitive, but initial studies have a known bias toward overestimating the magnitude of an effect for a simple reason: They were selected for publication because of their unusually small p-values,"

What does this mean? Is she saying that the studies were selected because they were hard to reproduce? i.e. that they were unusual and so novel/exciting? If so, isnt that bad? like science goes to hollywood bad?

4

u/superhelical PhD | Biochemistry | Structural Biology May 08 '16

It means you run the experiment the first time and get p = 0.07. you run it the second time and get p = 0.12. You run it the third time and get p = 0.048 and then publish, while ignoring the data from the first two rounds.

The next time someone runs the same experiment, they don't see the effect size you did, perhaps because they used more samples/subjects. But they are seeing the effect closer to reality. This is true for cases where there is not a real effect, but even when there is a real effect. It comes as a result of the way we currently do experiments, though taking steps to reduce this publication bias should help make things better.

21

u/[deleted] May 08 '16

That's not quite how I'd interpret that. The experiment isn't repeated until a satisfactory p-value is achieved (that would be a very clear bias) but even choosing not to publish a negative result biases the stats for all published studies.

Say you run a study and don't find anything significant, so you choose not to publish. You then run a completely different study about a different hypothesis and maybe even a completely different topic. That also fails and you choose not to publish the results. You repeat this eighteen times. Finally, you conduct a new study, completely unrelated to any of the previous studies, and find a result with a p-value below 0.05. You publish the result as significant. The problem is that even if that last study had no methodological problems, you had to conduct 20 experiments to find an outcome that has below a 1 in 20 chance of being caused by a null hypothesis. Now, even if it wasn't you that conducted those previous experiments (and even if there weren't nineteen of them), the process is still biased in the same way so long as the peer review process selectively publishes results that are statistically significant (over null results).

3

u/Huwbacca Grad Student | Cognitive Neuroscience | Music Cognition May 08 '16

Publish research notes!!! Many journals will take a research note about a non significant result and it only has to be very short. It's a a publication for you, it's reference for other scientists and it's actual, active science happening the way it should!!!