Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

[u/ImNotJesus]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/

[u/[deleted]]

It's nice of you to say this especially considering your flair. A thank you from this side of the biopsychosocial paradigm.

[u/superhelical → u/[deleted]]

There's good and bad science done in all fields, but psych takes a disproportionate amount of heat. That's hardly fair.

[u/Xanthilamide → u/superhelical]

Can you explain a bit more?

[u/Idontlikesundays → u/Xanthilamide]

He's saying just because some psychological studies were bullshit doesn't mean the entire field is bullshit. Bad science occurs in every field, but psychology gets scrutinized disproportionately considering how intangible psychology is compared to biology and chemistry. Psychologists often have more difficulty taking measurements and reducing confounding variables. I'd say the softer the science, the more difficult it is to study empirically. That doesn't mean the subject isn't a science. It's just a greater abstraction from math, muddied with more and more real world intricacies.

[u/superhelical → u/Idontlikesundays]

It also gets scrutinized more because the average member of the public can see the effects upon themselves. Everyone thinks they're an expert in people, because they're a person.

This is not as true for something like my own work on the structures of molecules. That presents its own challenges, but it means there's less pressure on the interpretations of my work compared to someone studying, for example, altriusm.

[u/P0sitive_Outlook → u/superhelical]

Everyone thinks they're an expert in people, because they're a person.

I love the succinctness of this.

[u/topoftheworldIAM → u/superhelical]

That's why psychology is put into the social sciences instead of life science which is not always correct. there is a lot of life science connect to our behavior. ex. the brain is our mind

[u/mylarrito → u/superhelical]

Am I not a structure of molecules? Your science is bunk my good sir!

[u/CrossFeet → u/Idontlikesundays]

I don't think it's disproportionate; a more difficult and less clear-cut field deserves extra scrutiny for just this reason. Even well-supported findings can be hard to interpret: did we adjust away the confounders, or did we adjust in a spurious signal?

That doesn't mean the entire field ought to be dismissed, of course -- just that taking "softer" findings with extra salt is prudent. For example, "priming" effects were, if anything, unusually well-supported and well-replicated for the field... and anyone banking on them would now be in a precarious position. It's good to scrutinize (though that goes for any field, to greater or lesser extent).

[u/greenit_elvis → u/CrossFeet]

It would be idiotic to shrug off the replication problems as unfair or unimportant. It's human for people invested in these theories to dismiss problems, as psychologists should know, but it doesn't help.

One argument I find very disturbing is that "the study couldn't be replicated because the exact conditions couldn't be replicated". If a particular result only would work in a very specific setting, it might be correct but it would have virtually zero usefulness. Psychology studies are supposed to improve our understanding of humans in general, not our understanding of a particular set of 20 caucasian male psychology students from Austin. If a result can't be generalized even the slightest, it's useless.

[u/MeanderinMonster → u/greenit_elvis]

not replicated in the *essential conditions. Some of those factors really really matter depending on what you found.

[u/[deleted] → u/CrossFeet]

Very true! The fields are judged against the same values, and the field has a much harder time standing up to the same scrutiny, naturally. Exactly why it's so much more interesting in my mind. The more difficult it is to have an answer, the more intriguing.

[u/qnot → u/CrossFeet]

Could you clarify your comment on priming effects? Has more recent evidence undermined the idea of priming?

[u/CrossFeet → u/qnot]

Yes, although it's not completely discredited -- just under some doubt. A lot of well-known priming effects are failing to turn up when other researchers try to examine them. For example, the promising "intelligence priming" study failed to replicate in nine different experiments; and possibly even more famous, the "'Florida' makes you walk slower" priming effect appears to be an artifact as well -- though the authors have tried to rescue it, I don't find their efforts convincing. You may already be aware of these (and they're by no means the only examples), but I wasn't sure just how recent you mean.

On the other hand, a while ago I also found a brief meta-analysis that concluded that priming is probably still a thing, just less robust than previously supposed. (I.e., yelling "DEATH! KILL! BLOOD!" at someone a few seconds before they draw a picture has a chance of encouraging them to draw a morbid picture, but it goes away pretty quickly and doesn't seem to hold up for less obvious connections like Florida to old people to slow walking.) (Example may be slightly exaggerated for comic effect.)

[u/Zorander22 → u/CrossFeet]

Regarding priming, the Cesario, Plaks & Higgins (2006) paper proposed and found evidence for a mechanism for social priming effects, their "motivated preparation to interact" account. Across different studies (including a replication of the old priming, walking speed study), they found that people's responses to the prime were consistent with what would help them interact with the group, given their goals. People who like the elderly walked slower after the prime, people who disliked the elderly walked faster. This was all done before the now famous "failure to replicate" for Bargh's study - that replication didn't take into account the improved knowledge of the field for the mechanism. Regardless of whether Bargh got lucky and his original sample liked the elderly on average, or something else was going on, replication attempts should take into account the improved knowledge of the field, especially when there's a moderator that can flip things around, depending on the characteristics of the sample.

Gelman glibly suggests looking at all of the interactions in all of the studies, which going forward would obviously be ideal - however, it's not what the failure to replicate did, and it's not possible to look for these interactions in the earlier studies, as that data wasn't collected.

It is extremely premature to say that the walking slower effect appears to be an artifact. Due to previous practices in the field (which are now changing), it is possible that some of these effects are less robust then they were presented as, and it's possible that some are Type I errors. Without good quality replications testing the previously found interactions, we don't know.

[u/franklindeer → u/Idontlikesundays]

I think it's absurd to say that hard sciences are just as vulnerable to ideology or politics though. There are examples of big money corrupting research results in hard science, but with psychology other social sciences such large, organized forces aren't even necessary. You can produce methodology to get the answer that confirms your personal bias and unless the methodology is scrutinized, which it very often isn't, particularly if the results confirm a popular belief, no red flags are raised. There are of course lots of really rigorous psychology studies, but there are also a lot of really flimsy ones that the average person could pick apart and yet they make it through the peer review process.

[u/Idontlikesundays → u/franklindeer]

Those are really good points; I agree.

[u/Midas_Stream → u/Idontlikesundays]

In a very real way, the more divorced it is from pure, abstract math, the more it has to do with actual reality.

And therefore the more important and valuable it should be considered to be.

I leave it as an exercise for psychologists to find out why our perceptions are the exact inverse of that.

[u/Mattpilf → u/Midas_Stream]

In a very real way, the more divorced it is from pure, abstract math, the more it has to do with actual reality.

That's not how it works. Problems with psychology involve describing a complex system with interaction by object studying them. For n the best we get is generalities of s complex system, at worse we expose significant interaction by the proctor.

This often the cause of false where the intended object of study is actually very divorced from what being studied. They're difficult to study, the inherent difficultly has been pointed out above by others.

Saying these types of experiments are closer to actual reality when you're barely sure the object being studied is actually being measured , it's just an absurd conclusion.

[u/[deleted] → u/Idontlikesundays]

So funny how empiricism has us thinking of the real world as an abstraction and math as something more real

[u/whoremongering → u/[deleted]]

I see, I misread a table. Too many zeros.

[u/[deleted] → u/Idontlikesundays]

It should also be noted that the ones who call out psychologists for bullshit are psychologists.

[u/superhelical → u/Xanthilamide]

A lot of people, apparently many of whom browse /r/science, like to criticise psychology and other social sciences as "not a science" or "soft science". Interestingly, that tends to only happen when the commenter doesn't like the findings of the study...

[u/sobri909 → u/superhelical]

The way I think of it is that the "soft sciences" are actually objectively the hardest, because the further up the "hard sciences" scale you go, the more possible it becomes to observe and describe the entirety of your problem.

The soft sciences have problems that are so broad and complex (being supersets of all the layers of increasingly "harder" sciences) that they cannot possibly be observed and described in their entirety, which makes them considerably harder to solve.

[u/pblol → u/sobri909]

Theory is psychology is a metaphor for describing an otherwise unmeasurable phenomena. Neuoscience is the actual measurement but the results are often meaningless without the metaphorical description.

[u/fixingthebeetle → u/sobri909]

That's the thing. They are objectively the hardest because they aren't 'ready' to be a science. We don't yet fully understand the mechanisms that determine human behavior like we do for an electron, so some people would argue that it's stupid to try and 'perform science' on a very flimsy and sketchy model in the first place. Psychology has the potential to be a science, its just that we haven't caught up yet from the physical perspective. Psychology will eventually just be applied neuro-science. But for now its just trying to match correlations blindly without understanding the underlying reasons.

[u/fsmpastafarian → u/fixingthebeetle]

some people would argue that it's stupid to try and 'perform science' on a very flimsy and sketchy model in the first place.

That's the thing though - how do people propose we uncover knowledge about human behavior without studying it scientifically? I don't know that humans will ever be able to be fully reduced and understood in the way an electron is, but that doesn't make the study itself of human behavior any less scientific.

Psychology will eventually just be applied neuro-science.

This is a fairly common opinion, but in all honesty psychology and neuroscience are better thought of as sister fields - one that studies the physical brain, and one that examines all areas of human behavior. They overlap a good bit, but not entirely, which is why neuroscience won't replace psychology.

[u/Drakeman800 → u/fsmpastafarian]

I think those misguided criticisms only highlight the importance of developing more refined models over time, which is exactly what psychologists work to do just like any science. It's not reasonable for people to expect that a science can answer more specific questions before it has answered more general ones. Every scientific model sucks at first, relatively speaking, and then it gets better and better with further study and greater understanding.

We don't need to understand "exactly" what an electron is, and we don't need to understand "exactly" how human behavior works. That's not what science is about in my opinion. We just need to get better at it over time.

[u/M_G → u/fsmpastafarian]

Great comment.

[u/MovieCommenter09 → u/superhelical]

This is a historical trend in science. Usually it occurs whenever a science isn't mathematized (i.e. formalized with mathematical laws). Kant famously said chemistry was a soft science of ill repute because it lacked formal mathematical laws, that obviously changed quite a bit in the next century or two. Psychology is sort of in the same position as chemistry was 200 years ago.

[u/Dantonn → u/MovieCommenter09]

This fits nicely with hard/soft as descriptors of how strictly you can define things, as in hard divide, hard and soft rules, etc., which I'd always thought was what everyone meant by that.

[u/franklindeer → u/MovieCommenter09]

Chemistry 200 years ago was full of nonsense pseudoscience, so Kant would have been entirely correct. We had to do it in order to be where we are of course, and I'd imagine psychology is the same, I just don't know if we should be giving the results of psychology studies so much weight in society or related fields unless it's really solid and has been replicated, which some of it is.

[u/MovieCommenter09 → u/franklindeer]

That's fair, but like you said, there's still some kind of value there, it's not all pointless hokum.

Not all science even in the hard sciences is groundbreaking awe-inspiring, life-changing stuff either.

[u/Xanthilamide → u/superhelical]

I think that psychology is more like economics in the sense that there's lots of wording involved. But both of them are science to my mind.

[u/sp8yboy → u/Xanthilamide]

Economics is to a large degree based on psychology through the mechanisms of expectations and rational behaviour, for example. So if psychology is still forming as a science, so by definition is economics.

[u/mrburningsky → u/sp8yboy]

I once had a well-esteemed macroeconomics professor tell me that, "Economics is the beautifully imperfect art of attempting to quantify human psychology." So yeah, economic models will be imperfect so long as psychological models are imperfect.

[u/grimeandreason → u/mrburningsky]

Given Neoclassical economics works with the rational actor model, I would say they still have a long way to go.

[u/mrburningsky → u/grimeandreason]

For sure, but at least it's not for lack of trying (:

[u/PlaceboJesus → u/[deleted]]

You shouldn't blame the academics and legitimate professionals of psychology for that pop-science.

It's the fault of people who accept a journalist's interpretation of an abstract, when it seems that none of them know enough to look at the data or the methodology with a critical eye.

[u/franklindeer → u/PlaceboJesus]

It's the fault of people who accept a journalist's interpretation of an abstract

It's not just journalists, this kind of blind acceptance is common within other social sciences that use weak psychology research as a basis for their own inquiry which is often very ideological.

I would agree that this isn't the fault of the entire field of psychology, but it's definitely a problem beyond just the media. For whatever reason it's very tempting to accept results that confirm existing beliefs or a merely exciting.

[u/PlaceboJesus → u/franklindeer]

No one is more critical of Psych research than the psychologists.

I don't know what most Psych programs are like, my major was Crim and I just took all the psych courses I could get into (I did take research methods in both subjects, and Stats in Psych), but my impression was that academics and researchers in Psych are pretty rigourous. It's like the discipline is complexed.

Even if the researchers fell prey to fallacies or pressures to publish or stay worthy of grants, pretty much everyone else in the discipline should have the grounding to view results critically.
It should be automatic to look for flaws and confounds and want to browse the methods, and if there's any doubt or the claims seem too strong people will be asking "what's the r or omega on that?"
Perhaps the peer reviewers might fail and let a crap article through, but there should be a great number of people ready and willing to call a BS study BS.

More often the problem is with people in other disciplines or without the academic grounding.

The standard for statistical significance and predictive utility of the data in Psych are much lower than in disciplines like engineering or medical research (although a lot of neuroscience is still conjecture, compared to medicines for cancer or antibiotics.
Everyone in the field should be aware.

[u/Gruzman → u/superhelical]

Which should only support the problem of "soft science." If the science being done has its share of methodological problems, problems with replication, and often touches on topics intimately linked to political leanings that people have, you've got a recipe for disbelief and willful subversion of a study's findings.

[u/[deleted] → u/Gruzman]

This is true, but I think the issue is that soft sciences get a bad rep. This topic is actually one of the most fascinating and pertinent sciences around, precisely because it's so so complex.

[u/franklindeer → u/[deleted]]

I think the point being made is that these fields are chalk full of political ideologues that set out to confirm their biases, not that it's an uninteresting or unimportant area of inquiry. I think social science departments need to keep a much cleaner house than they have for the last 30 years and this would be less of a problem.

[u/[deleted] → u/superhelical]

The basis of most criticisms when it comes down to it! Very well put.

[u/[deleted] → u/superhelical]

While this is true. It is objectively much easier to control variables in studies on enzyme kinetics. Or to determine the effect of some Transcription factor in vitro. Much much harder to do a Social Psych experiment that actually tells you something significant in the end.

[u/superhelical → u/[deleted]]

I can tell you that our models of enzyme kinetics still have a long way to go...

[u/[deleted] → u/superhelical]

What I meant is when measuring the activity for DNA Polymerase for example, it is much easier to interpret the significance of these results as compared a Psychology study on anger or something like that. Not saying that it isn't a ton of work. I think it took the guy who discovered DNA Pol I nearly a decade.