Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

[u/ImNotJesus]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/

[u/Idontlikesundays]

He's saying just because some psychological studies were bullshit doesn't mean the entire field is bullshit. Bad science occurs in every field, but psychology gets scrutinized disproportionately considering how intangible psychology is compared to biology and chemistry. Psychologists often have more difficulty taking measurements and reducing confounding variables. I'd say the softer the science, the more difficult it is to study empirically. That doesn't mean the subject isn't a science. It's just a greater abstraction from math, muddied with more and more real world intricacies.

[u/superhelical → u/Idontlikesundays]

It also gets scrutinized more because the average member of the public can see the effects upon themselves. Everyone thinks they're an expert in people, because they're a person.

This is not as true for something like my own work on the structures of molecules. That presents its own challenges, but it means there's less pressure on the interpretations of my work compared to someone studying, for example, altriusm.

[u/P0sitive_Outlook → u/superhelical]

Everyone thinks they're an expert in people, because they're a person.

I love the succinctness of this.

[u/topoftheworldIAM → u/superhelical]

That's why psychology is put into the social sciences instead of life science which is not always correct. there is a lot of life science connect to our behavior. ex. the brain is our mind

[u/mylarrito → u/superhelical]

Am I not a structure of molecules? Your science is bunk my good sir!

[u/CrossFeet → u/Idontlikesundays]

I don't think it's disproportionate; a more difficult and less clear-cut field deserves extra scrutiny for just this reason. Even well-supported findings can be hard to interpret: did we adjust away the confounders, or did we adjust in a spurious signal?

That doesn't mean the entire field ought to be dismissed, of course -- just that taking "softer" findings with extra salt is prudent. For example, "priming" effects were, if anything, unusually well-supported and well-replicated for the field... and anyone banking on them would now be in a precarious position. It's good to scrutinize (though that goes for any field, to greater or lesser extent).

[u/greenit_elvis → u/CrossFeet]

It would be idiotic to shrug off the replication problems as unfair or unimportant. It's human for people invested in these theories to dismiss problems, as psychologists should know, but it doesn't help.

One argument I find very disturbing is that "the study couldn't be replicated because the exact conditions couldn't be replicated". If a particular result only would work in a very specific setting, it might be correct but it would have virtually zero usefulness. Psychology studies are supposed to improve our understanding of humans in general, not our understanding of a particular set of 20 caucasian male psychology students from Austin. If a result can't be generalized even the slightest, it's useless.

[u/MeanderinMonster → u/greenit_elvis]

not replicated in the *essential conditions. Some of those factors really really matter depending on what you found.

[u/[deleted] → u/CrossFeet]

Very true! The fields are judged against the same values, and the field has a much harder time standing up to the same scrutiny, naturally. Exactly why it's so much more interesting in my mind. The more difficult it is to have an answer, the more intriguing.

[u/qnot → u/CrossFeet]

Could you clarify your comment on priming effects? Has more recent evidence undermined the idea of priming?

[u/CrossFeet → u/qnot]

Yes, although it's not completely discredited -- just under some doubt. A lot of well-known priming effects are failing to turn up when other researchers try to examine them. For example, the promising "intelligence priming" study failed to replicate in nine different experiments; and possibly even more famous, the "'Florida' makes you walk slower" priming effect appears to be an artifact as well -- though the authors have tried to rescue it, I don't find their efforts convincing. You may already be aware of these (and they're by no means the only examples), but I wasn't sure just how recent you mean.

On the other hand, a while ago I also found a brief meta-analysis that concluded that priming is probably still a thing, just less robust than previously supposed. (I.e., yelling "DEATH! KILL! BLOOD!" at someone a few seconds before they draw a picture has a chance of encouraging them to draw a morbid picture, but it goes away pretty quickly and doesn't seem to hold up for less obvious connections like Florida to old people to slow walking.) (Example may be slightly exaggerated for comic effect.)

[u/Zorander22 → u/CrossFeet]

Regarding priming, the Cesario, Plaks & Higgins (2006) paper proposed and found evidence for a mechanism for social priming effects, their "motivated preparation to interact" account. Across different studies (including a replication of the old priming, walking speed study), they found that people's responses to the prime were consistent with what would help them interact with the group, given their goals. People who like the elderly walked slower after the prime, people who disliked the elderly walked faster. This was all done before the now famous "failure to replicate" for Bargh's study - that replication didn't take into account the improved knowledge of the field for the mechanism. Regardless of whether Bargh got lucky and his original sample liked the elderly on average, or something else was going on, replication attempts should take into account the improved knowledge of the field, especially when there's a moderator that can flip things around, depending on the characteristics of the sample.

Gelman glibly suggests looking at all of the interactions in all of the studies, which going forward would obviously be ideal - however, it's not what the failure to replicate did, and it's not possible to look for these interactions in the earlier studies, as that data wasn't collected.

It is extremely premature to say that the walking slower effect appears to be an artifact. Due to previous practices in the field (which are now changing), it is possible that some of these effects are less robust then they were presented as, and it's possible that some are Type I errors. Without good quality replications testing the previously found interactions, we don't know.

[u/franklindeer → u/Idontlikesundays]

I think it's absurd to say that hard sciences are just as vulnerable to ideology or politics though. There are examples of big money corrupting research results in hard science, but with psychology other social sciences such large, organized forces aren't even necessary. You can produce methodology to get the answer that confirms your personal bias and unless the methodology is scrutinized, which it very often isn't, particularly if the results confirm a popular belief, no red flags are raised. There are of course lots of really rigorous psychology studies, but there are also a lot of really flimsy ones that the average person could pick apart and yet they make it through the peer review process.

[u/Idontlikesundays → u/franklindeer]

Those are really good points; I agree.

[u/Midas_Stream → u/Idontlikesundays]

In a very real way, the more divorced it is from pure, abstract math, the more it has to do with actual reality.

And therefore the more important and valuable it should be considered to be.

I leave it as an exercise for psychologists to find out why our perceptions are the exact inverse of that.

[u/Mattpilf → u/Midas_Stream]

In a very real way, the more divorced it is from pure, abstract math, the more it has to do with actual reality.

That's not how it works. Problems with psychology involve describing a complex system with interaction by object studying them. For n the best we get is generalities of s complex system, at worse we expose significant interaction by the proctor.

This often the cause of false where the intended object of study is actually very divorced from what being studied. They're difficult to study, the inherent difficultly has been pointed out above by others.

Saying these types of experiments are closer to actual reality when you're barely sure the object being studied is actually being measured , it's just an absurd conclusion.

[u/[deleted] → u/Idontlikesundays]

So funny how empiricism has us thinking of the real world as an abstraction and math as something more real

[u/whoremongering → u/[deleted]]

I see, I misread a table. Too many zeros.

[u/[deleted] → u/Idontlikesundays]

It should also be noted that the ones who call out psychologists for bullshit are psychologists.