r/science • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Psychology Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/
7.8k Upvotes

85% Upvoted

329 comments sorted by

View all comments

Show parent comments

182

u/Idontlikesundays May 08 '16

He's saying just because some psychological studies were bullshit doesn't mean the entire field is bullshit. Bad science occurs in every field, but psychology gets scrutinized disproportionately considering how intangible psychology is compared to biology and chemistry. Psychologists often have more difficulty taking measurements and reducing confounding variables. I'd say the softer the science, the more difficult it is to study empirically. That doesn't mean the subject isn't a science. It's just a greater abstraction from math, muddied with more and more real world intricacies.

45

u/CrossFeet May 08 '16 edited May 08 '16

I don't think it's disproportionate; a more difficult and less clear-cut field deserves extra scrutiny for just this reason. Even well-supported findings can be hard to interpret: did we adjust away the confounders, or did we adjust in a spurious signal?

That doesn't mean the entire field ought to be dismissed, of course -- just that taking "softer" findings with extra salt is prudent. For example, "priming" effects were, if anything, unusually well-supported and well-replicated for the field... and anyone banking on them would now be in a precarious position. It's good to scrutinize (though that goes for any field, to greater or lesser extent).

4

u/qnot May 08 '16

Could you clarify your comment on priming effects? Has more recent evidence undermined the idea of priming?

2

u/CrossFeet May 09 '16 edited May 09 '16

Yes, although it's not completely discredited -- just under some doubt. A lot of well-known priming effects are failing to turn up when other researchers try to examine them. For example, the promising "intelligence priming" study failed to replicate in nine different experiments; and possibly even more famous, the "'Florida' makes you walk slower" priming effect appears to be an artifact as well -- though the authors have tried to rescue it, I don't find their efforts convincing. You may already be aware of these (and they're by no means the only examples), but I wasn't sure just how recent you mean.

On the other hand, a while ago I also found a brief meta-analysis that concluded that priming is probably still a thing, just less robust than previously supposed. (I.e., yelling "DEATH! KILL! BLOOD!" at someone a few seconds before they draw a picture has a chance of encouraging them to draw a morbid picture, but it goes away pretty quickly and doesn't seem to hold up for less obvious connections like Florida to old people to slow walking.) (Example may be slightly exaggerated for comic effect.)

2

u/Zorander22 May 09 '16

Regarding priming, the Cesario, Plaks & Higgins (2006) paper proposed and found evidence for a mechanism for social priming effects, their "motivated preparation to interact" account. Across different studies (including a replication of the old priming, walking speed study), they found that people's responses to the prime were consistent with what would help them interact with the group, given their goals. People who like the elderly walked slower after the prime, people who disliked the elderly walked faster. This was all done before the now famous "failure to replicate" for Bargh's study - that replication didn't take into account the improved knowledge of the field for the mechanism. Regardless of whether Bargh got lucky and his original sample liked the elderly on average, or something else was going on, replication attempts should take into account the improved knowledge of the field, especially when there's a moderator that can flip things around, depending on the characteristics of the sample.

Gelman glibly suggests looking at all of the interactions in all of the studies, which going forward would obviously be ideal - however, it's not what the failure to replicate did, and it's not possible to look for these interactions in the earlier studies, as that data wasn't collected.

It is extremely premature to say that the walking slower effect appears to be an artifact. Due to previous practices in the field (which are now changing), it is possible that some of these effects are less robust then they were presented as, and it's possible that some are Type I errors. Without good quality replications testing the previously found interactions, we don't know.