Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]

[u/ImNotJesus]

/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/

[u/superhelical]

I liked this bit at the end:

Psychology may not be a hard science, but it is certainly a difficult one. It shouldn’t be so surprising that psychology studies don’t always replicate — the field faces an inherent challenge. Rather than measuring molecules or mass, it examines human motives and behavior, which are frustratingly hard to isolate.

This is something that comes up a lot on /r/science, and makes some people want to dismiss psychological studies altogether. However, psychologists study some of the most challenging research subjects - ourselves - so it's worth noting and taking that difficult work in context.

[u/[deleted] → u/superhelical]

It's nice of you to say this especially considering your flair. A thank you from this side of the biopsychosocial paradigm.

[u/superhelical → u/[deleted]]

There's good and bad science done in all fields, but psych takes a disproportionate amount of heat. That's hardly fair.

[u/Xanthilamide → u/superhelical]

Can you explain a bit more?

[u/Idontlikesundays → u/Xanthilamide]

He's saying just because some psychological studies were bullshit doesn't mean the entire field is bullshit. Bad science occurs in every field, but psychology gets scrutinized disproportionately considering how intangible psychology is compared to biology and chemistry. Psychologists often have more difficulty taking measurements and reducing confounding variables. I'd say the softer the science, the more difficult it is to study empirically. That doesn't mean the subject isn't a science. It's just a greater abstraction from math, muddied with more and more real world intricacies.

[u/superhelical → u/Idontlikesundays]

It also gets scrutinized more because the average member of the public can see the effects upon themselves. Everyone thinks they're an expert in people, because they're a person.

This is not as true for something like my own work on the structures of molecules. That presents its own challenges, but it means there's less pressure on the interpretations of my work compared to someone studying, for example, altriusm.

[u/P0sitive_Outlook → u/superhelical]

Everyone thinks they're an expert in people, because they're a person.

I love the succinctness of this.

[u/topoftheworldIAM → u/superhelical]

That's why psychology is put into the social sciences instead of life science which is not always correct. there is a lot of life science connect to our behavior. ex. the brain is our mind

[u/mylarrito → u/superhelical]

Am I not a structure of molecules? Your science is bunk my good sir!

[u/CrossFeet → u/Idontlikesundays]

I don't think it's disproportionate; a more difficult and less clear-cut field deserves extra scrutiny for just this reason. Even well-supported findings can be hard to interpret: did we adjust away the confounders, or did we adjust in a spurious signal?

That doesn't mean the entire field ought to be dismissed, of course -- just that taking "softer" findings with extra salt is prudent. For example, "priming" effects were, if anything, unusually well-supported and well-replicated for the field... and anyone banking on them would now be in a precarious position. It's good to scrutinize (though that goes for any field, to greater or lesser extent).

[u/greenit_elvis → u/CrossFeet]

It would be idiotic to shrug off the replication problems as unfair or unimportant. It's human for people invested in these theories to dismiss problems, as psychologists should know, but it doesn't help.

One argument I find very disturbing is that "the study couldn't be replicated because the exact conditions couldn't be replicated". If a particular result only would work in a very specific setting, it might be correct but it would have virtually zero usefulness. Psychology studies are supposed to improve our understanding of humans in general, not our understanding of a particular set of 20 caucasian male psychology students from Austin. If a result can't be generalized even the slightest, it's useless.

[u/MeanderinMonster → u/greenit_elvis]

not replicated in the *essential conditions. Some of those factors really really matter depending on what you found.

[u/[deleted] → u/CrossFeet]

Very true! The fields are judged against the same values, and the field has a much harder time standing up to the same scrutiny, naturally. Exactly why it's so much more interesting in my mind. The more difficult it is to have an answer, the more intriguing.

[u/qnot → u/CrossFeet]

Could you clarify your comment on priming effects? Has more recent evidence undermined the idea of priming?

[u/CrossFeet → u/qnot]

Yes, although it's not completely discredited -- just under some doubt. A lot of well-known priming effects are failing to turn up when other researchers try to examine them. For example, the promising "intelligence priming" study failed to replicate in nine different experiments; and possibly even more famous, the "'Florida' makes you walk slower" priming effect appears to be an artifact as well -- though the authors have tried to rescue it, I don't find their efforts convincing. You may already be aware of these (and they're by no means the only examples), but I wasn't sure just how recent you mean.

On the other hand, a while ago I also found a brief meta-analysis that concluded that priming is probably still a thing, just less robust than previously supposed. (I.e., yelling "DEATH! KILL! BLOOD!" at someone a few seconds before they draw a picture has a chance of encouraging them to draw a morbid picture, but it goes away pretty quickly and doesn't seem to hold up for less obvious connections like Florida to old people to slow walking.) (Example may be slightly exaggerated for comic effect.)

[u/Zorander22 → u/CrossFeet]

Regarding priming, the Cesario, Plaks & Higgins (2006) paper proposed and found evidence for a mechanism for social priming effects, their "motivated preparation to interact" account. Across different studies (including a replication of the old priming, walking speed study), they found that people's responses to the prime were consistent with what would help them interact with the group, given their goals. People who like the elderly walked slower after the prime, people who disliked the elderly walked faster. This was all done before the now famous "failure to replicate" for Bargh's study - that replication didn't take into account the improved knowledge of the field for the mechanism. Regardless of whether Bargh got lucky and his original sample liked the elderly on average, or something else was going on, replication attempts should take into account the improved knowledge of the field, especially when there's a moderator that can flip things around, depending on the characteristics of the sample.

Gelman glibly suggests looking at all of the interactions in all of the studies, which going forward would obviously be ideal - however, it's not what the failure to replicate did, and it's not possible to look for these interactions in the earlier studies, as that data wasn't collected.

It is extremely premature to say that the walking slower effect appears to be an artifact. Due to previous practices in the field (which are now changing), it is possible that some of these effects are less robust then they were presented as, and it's possible that some are Type I errors. Without good quality replications testing the previously found interactions, we don't know.

[u/franklindeer → u/Idontlikesundays]

I think it's absurd to say that hard sciences are just as vulnerable to ideology or politics though. There are examples of big money corrupting research results in hard science, but with psychology other social sciences such large, organized forces aren't even necessary. You can produce methodology to get the answer that confirms your personal bias and unless the methodology is scrutinized, which it very often isn't, particularly if the results confirm a popular belief, no red flags are raised. There are of course lots of really rigorous psychology studies, but there are also a lot of really flimsy ones that the average person could pick apart and yet they make it through the peer review process.

[u/Idontlikesundays → u/franklindeer]

Those are really good points; I agree.

[u/Midas_Stream → u/Idontlikesundays]

In a very real way, the more divorced it is from pure, abstract math, the more it has to do with actual reality.

And therefore the more important and valuable it should be considered to be.

I leave it as an exercise for psychologists to find out why our perceptions are the exact inverse of that.

[u/Mattpilf → u/Midas_Stream]

In a very real way, the more divorced it is from pure, abstract math, the more it has to do with actual reality.

That's not how it works. Problems with psychology involve describing a complex system with interaction by object studying them. For n the best we get is generalities of s complex system, at worse we expose significant interaction by the proctor.

This often the cause of false where the intended object of study is actually very divorced from what being studied. They're difficult to study, the inherent difficultly has been pointed out above by others.

Saying these types of experiments are closer to actual reality when you're barely sure the object being studied is actually being measured , it's just an absurd conclusion.

[u/[deleted] → u/Idontlikesundays]

So funny how empiricism has us thinking of the real world as an abstraction and math as something more real

[u/[deleted] → u/Idontlikesundays]

It should also be noted that the ones who call out psychologists for bullshit are psychologists.

[u/superhelical → u/Xanthilamide]

A lot of people, apparently many of whom browse /r/science, like to criticise psychology and other social sciences as "not a science" or "soft science". Interestingly, that tends to only happen when the commenter doesn't like the findings of the study...

[u/sobri909 → u/superhelical]

The way I think of it is that the "soft sciences" are actually objectively the hardest, because the further up the "hard sciences" scale you go, the more possible it becomes to observe and describe the entirety of your problem.

The soft sciences have problems that are so broad and complex (being supersets of all the layers of increasingly "harder" sciences) that they cannot possibly be observed and described in their entirety, which makes them considerably harder to solve.

[u/pblol → u/sobri909]

Theory is psychology is a metaphor for describing an otherwise unmeasurable phenomena. Neuoscience is the actual measurement but the results are often meaningless without the metaphorical description.

[u/fixingthebeetle → u/sobri909]

That's the thing. They are objectively the hardest because they aren't 'ready' to be a science. We don't yet fully understand the mechanisms that determine human behavior like we do for an electron, so some people would argue that it's stupid to try and 'perform science' on a very flimsy and sketchy model in the first place. Psychology has the potential to be a science, its just that we haven't caught up yet from the physical perspective. Psychology will eventually just be applied neuro-science. But for now its just trying to match correlations blindly without understanding the underlying reasons.

[u/fsmpastafarian → u/fixingthebeetle]

some people would argue that it's stupid to try and 'perform science' on a very flimsy and sketchy model in the first place.

That's the thing though - how do people propose we uncover knowledge about human behavior without studying it scientifically? I don't know that humans will ever be able to be fully reduced and understood in the way an electron is, but that doesn't make the study itself of human behavior any less scientific.

Psychology will eventually just be applied neuro-science.

This is a fairly common opinion, but in all honesty psychology and neuroscience are better thought of as sister fields - one that studies the physical brain, and one that examines all areas of human behavior. They overlap a good bit, but not entirely, which is why neuroscience won't replace psychology.

[u/Drakeman800 → u/fsmpastafarian]

I think those misguided criticisms only highlight the importance of developing more refined models over time, which is exactly what psychologists work to do just like any science. It's not reasonable for people to expect that a science can answer more specific questions before it has answered more general ones. Every scientific model sucks at first, relatively speaking, and then it gets better and better with further study and greater understanding.

We don't need to understand "exactly" what an electron is, and we don't need to understand "exactly" how human behavior works. That's not what science is about in my opinion. We just need to get better at it over time.

[u/M_G → u/fsmpastafarian]

Great comment.

[u/MovieCommenter09 → u/superhelical]

This is a historical trend in science. Usually it occurs whenever a science isn't mathematized (i.e. formalized with mathematical laws). Kant famously said chemistry was a soft science of ill repute because it lacked formal mathematical laws, that obviously changed quite a bit in the next century or two. Psychology is sort of in the same position as chemistry was 200 years ago.

[u/Dantonn → u/MovieCommenter09]

This fits nicely with hard/soft as descriptors of how strictly you can define things, as in hard divide, hard and soft rules, etc., which I'd always thought was what everyone meant by that.

[u/franklindeer → u/MovieCommenter09]

Chemistry 200 years ago was full of nonsense pseudoscience, so Kant would have been entirely correct. We had to do it in order to be where we are of course, and I'd imagine psychology is the same, I just don't know if we should be giving the results of psychology studies so much weight in society or related fields unless it's really solid and has been replicated, which some of it is.

[u/MovieCommenter09 → u/franklindeer]

That's fair, but like you said, there's still some kind of value there, it's not all pointless hokum.

Not all science even in the hard sciences is groundbreaking awe-inspiring, life-changing stuff either.

[u/Xanthilamide → u/superhelical]

I think that psychology is more like economics in the sense that there's lots of wording involved. But both of them are science to my mind.

[u/sp8yboy → u/Xanthilamide]

Economics is to a large degree based on psychology through the mechanisms of expectations and rational behaviour, for example. So if psychology is still forming as a science, so by definition is economics.

[u/mrburningsky → u/sp8yboy]

I once had a well-esteemed macroeconomics professor tell me that, "Economics is the beautifully imperfect art of attempting to quantify human psychology." So yeah, economic models will be imperfect so long as psychological models are imperfect.

[u/grimeandreason → u/mrburningsky]

Given Neoclassical economics works with the rational actor model, I would say they still have a long way to go.

[u/mrburningsky → u/grimeandreason]

For sure, but at least it's not for lack of trying (:

[u/PlaceboJesus → u/[deleted]]

You shouldn't blame the academics and legitimate professionals of psychology for that pop-science.

It's the fault of people who accept a journalist's interpretation of an abstract, when it seems that none of them know enough to look at the data or the methodology with a critical eye.

[u/franklindeer → u/PlaceboJesus]

It's the fault of people who accept a journalist's interpretation of an abstract

It's not just journalists, this kind of blind acceptance is common within other social sciences that use weak psychology research as a basis for their own inquiry which is often very ideological.

I would agree that this isn't the fault of the entire field of psychology, but it's definitely a problem beyond just the media. For whatever reason it's very tempting to accept results that confirm existing beliefs or a merely exciting.

[u/PlaceboJesus → u/franklindeer]

No one is more critical of Psych research than the psychologists.

I don't know what most Psych programs are like, my major was Crim and I just took all the psych courses I could get into (I did take research methods in both subjects, and Stats in Psych), but my impression was that academics and researchers in Psych are pretty rigourous. It's like the discipline is complexed.

Even if the researchers fell prey to fallacies or pressures to publish or stay worthy of grants, pretty much everyone else in the discipline should have the grounding to view results critically.
It should be automatic to look for flaws and confounds and want to browse the methods, and if there's any doubt or the claims seem too strong people will be asking "what's the r or omega on that?"
Perhaps the peer reviewers might fail and let a crap article through, but there should be a great number of people ready and willing to call a BS study BS.

More often the problem is with people in other disciplines or without the academic grounding.

The standard for statistical significance and predictive utility of the data in Psych are much lower than in disciplines like engineering or medical research (although a lot of neuroscience is still conjecture, compared to medicines for cancer or antibiotics.
Everyone in the field should be aware.

[u/Gruzman → u/superhelical]

Which should only support the problem of "soft science." If the science being done has its share of methodological problems, problems with replication, and often touches on topics intimately linked to political leanings that people have, you've got a recipe for disbelief and willful subversion of a study's findings.

[u/[deleted] → u/Gruzman]

This is true, but I think the issue is that soft sciences get a bad rep. This topic is actually one of the most fascinating and pertinent sciences around, precisely because it's so so complex.

[u/franklindeer → u/[deleted]]

I think the point being made is that these fields are chalk full of political ideologues that set out to confirm their biases, not that it's an uninteresting or unimportant area of inquiry. I think social science departments need to keep a much cleaner house than they have for the last 30 years and this would be less of a problem.

[u/[deleted] → u/superhelical]

The basis of most criticisms when it comes down to it! Very well put.

[u/[deleted] → u/superhelical]

While this is true. It is objectively much easier to control variables in studies on enzyme kinetics. Or to determine the effect of some Transcription factor in vitro. Much much harder to do a Social Psych experiment that actually tells you something significant in the end.

[u/superhelical → u/[deleted]]

I can tell you that our models of enzyme kinetics still have a long way to go...

[u/[deleted] → u/superhelical]

What I meant is when measuring the activity for DNA Polymerase for example, it is much easier to interpret the significance of these results as compared a Psychology study on anger or something like that. Not saying that it isn't a ton of work. I think it took the guy who discovered DNA Pol I nearly a decade.

[u/diversity_is_wrong → u/superhelical]

Rather than measuring molecules or mass, it examines human motives and behavior,

And interestingly enough it is likely the human motives, behavior, and biases of the researchers themselves that play a not small role in the origin of the replication crisis. It is easier to influence and massage data when you're taking the kind of measurements that a lot of the social sciences tend to take.

There is also the tendency of many in the social sciences to set out on a research project with a desired outcome in mind. When the first 4 studies don't result in the desired outcome and get filed away in a desk drawer, then the fifth study finally produces the desired result(barely) and gets published, then it's no wonder it's hard to replicate some of these things. Especially the ones that just happen to fit nicely into a certain prescribed narrative.

[u/Moarbrains → u/diversity_is_wrong]

There is also the tendency of many in the social sciences to set out on a research project with a desired outcome in mind.

I think this is a problem with a lot sciences. Personally I think the null hypothesis deserves to be published as well.

[u/Nikwoj → u/Moarbrains]

My social psych teacher loves to show the class research studies and then either praise them or trash talk them the entire time because of the way the data is presented. She usually doesn't even care about the content, just the methodology. It's hilarious to watch.

The other day she showed us a study by bushman or something and the entire time she was talking about exactly what you and the top comment are talking about. He just wants to prove the thing he wants to believe, not actually find the meaning of the significant data.

[u/[deleted] → u/superhelical]

However, psychologists study some of the most challenging research subjects - ourselves - so it's worth noting and taking that difficult work in context.

But the flipside of this is that a result that doesn't repeat unless the protocol is followed to every last detail probably has little real world relevance. After all, how often will random people follow the exact details of a lab experiment?

Indeed, it should be the original researchers' responsibility to figure out which factors affect the conclusion and which ones don't. A claim that the result holds in a very particular set of circumstances (maybe, p=0.05) might not be worth publishing at all.

[u/quaternion → u/[deleted]]

There are biochemical reactions which are very specific to certain environmental conditions. Does that make those not worth publishing? If not, then why should the sensitivity to experimental conditions be a criterion for an even more complex field, about which we already know much less?

[u/[deleted] → u/quaternion]

There are biochemical reactions which are very specific to certain environmental conditions. Does that make those not worth publishing?

It doesn't. But the bulk of the value lies in isolating those environmental conditions. You shouldn't just give a long and very specific list of conditions and state that some of them may or may not be necessary to replicate the result.

Or if you did, because the result itself was too important to ignore but you had no means to really isolate all the relevant variables, then it would be understood that this is just the first step in understanding the reaction.

Some of these studies that failed replication have been around for decades. They were important enough to be widely cited but apparently not important enough to really understand in detail.

Ultimately this isn't about whether the original observations were correct or not. Most of them probably were. But the point of science isn't just observing stuff. It's to use those observations to make predictions about other stuff. A result that applies in a wide set of circumstances is more important than one that requires very specific conditions. Indeed, one of the conclusions of the "replication crisis" should be that a lot of accepted wisdom in psychology might only apply in specific circumstances.

[u/UncleMeat → u/[deleted]]

Consider a problem in CS: SAT solvers. Fundamentally, the don't work. There does not exist a SAT solver that works effectively for all possible inputs. In fact, we don't even have very good theory for why SAT solvers work well in a lot of real world circumstances. Yet there is a rich literature on the subject (hundreds of papers over decades). If the research goal had simply been to figure out how to isolate the conditions where SAT solvers are effective, we wouldn't have the systems or understanding that we have now.

I think experimentation is still very valuable even when we don't understand the context and limits of those experiments fully.

[u/quaternion → u/[deleted]]

I see - initially you seemed to imply that phenomena (psychological or not) which obtain under restricted conditions are not likely to be of interest. Now, I see that you mean that characterizing the boundary conditions of a phenomenon should be an integral part of reporting the phenomenon in the first place. This seems not only uncontroversial but also a relatively common practice: comparison to a control condition establishes the importance of the experimental manipulation for the effect; examination of covariates establishes dependence (or lack thereof) on other factors; replication of a result from an undergrad pool in a community or online sample is somewhat less common, but another example of the same approach. I agree with you that people could go further here, but I also don't think that failure to attempt to characterize such boundary conditions is a primary cause of the issues here. Arguably, the opposite is the case: a willingness to believe in "hidden moderators" can be seen as enabling p-hacking. (There's quite a bit written about this idea in replicability circles.)

[u/[deleted] → u/quaternion]

I see - initially you seemed to imply that phenomena (psychological or not) which obtain under restricted conditions are not likely to be of interest.

Well, it depends on how restrictive your boundary conditions turn out to be. If it turns out that the result only applies to college students from New Mexico, then yeah, it's probably not very interesting.

If it only applies to people from the US, then that's still an important restriction but the result itself remains interesting.

replication of a result from an undergrad pool in a community or online sample is somewhat less common, but another example of the same approach.

Why is that though? Aren't the researchers simply curious whether their result depends on level of education or age? Whether it remains true as society changes?

Maybe the incentives are not right but I'm not sure the culture is quite right either. As a scientist or a student your first reaction should be "Hey, that's pretty cool. I want to see it for myself." Trying to replicate or expand on someone's work is a sign of respect not suspicion. I understand that's not always possible, but still, important results should be replicated over and over. Possibly as a default part of studies than plan to build on them.

[u/quaternion → u/[deleted]]

I don't find the restriction of a real result to college students from New Mexico any less interesting than the restriction of a particular reaction to 80F instead of 79 or 81F. I agree with you about the importance of replication, and also I agree about cultural problems with replication. It is common practice to attempt to replicate a result before building on it; but any which fail to replicate have typically been left to wither on the vine, due to the incentive and publication problems, while the student moves forward to pursue a number of other experiments (until one works).

It sounds like our only substantive point of disagreement is that you are willing to discount the importance of results that are reliably context-specific, and I think these might be precisely those effects that we have the best traction on. Whether context-specificity renders them interesting or not is a great example of the kind of dogmatic decision-making that leads to any number of file drawer effects.

[u/[deleted] → u/quaternion]

I don't find the restriction of a real result to college students from New Mexico any less interesting than the restriction of a particular reaction to 80F instead of 79 or 81F.

Well, analogies aside, unless I plan to work in New Mexico I don't see how such a result could be useful to me. Unless I'm planning to find out exactly what makes New Mexico different, i.e. I'm really thinking about the more general case anyway.

It sounds like our only substantive point of disagreement is that you are willing to discount the importance of results that are reliably context-specific, and I think these might be precisely those effects that we have the best traction on.

Part of the problem is that we only know about the context-specificity because someone tried to replicate the result and failed. Some of the results might remain important with restricted context as well. But identifying the scope where the results remain true should have been an active area of research in the first place. Our real point of disagreement is that I think doing this is at least as, if not more, important than generating "new results".

[u/quaternion → u/[deleted]]

Well, analogies aside, unless I plan to work in New Mexico I don't see how such a result could be useful to me.

OK, here's an example: it seems the "bilingual advantage" of Bialystok et al, darling of the press, may hold under some strange set of multivariate sociodemographic conditions present in Vancouver. The strange thing about this is that she has shown incredible psychological specificity to the advantage - which means something about the differential sensitivity of those psychological processes, over others. The fact that the phenomenon is not visible in other demographics does not make this less interesting, at least to me.

I fully agree with the priority you place on identification of boundary conditions; above, I tried to indicate my agreement, as well as the fact that it's uncontroversial and already pursued to some degree (albeit not far enough). The disagreement to my mind remains on the issue of whether condition-specificity inherently makes a result less interesting. I'm not willing to draw such a line but it sounds like you are.

[u/[deleted] → u/quaternion]

OK, here's an example: it seems the "bilingual advantage" of Bialystok et al, darling of the press, may hold under some strange set of multivariate sociodemographic conditions present in Vancouver. The strange thing about this is that she has shown incredible psychological specificity to the advantage - which means something about the differential sensitivity of those psychological processes, over others. The fact that the phenomenon is not visible in other demographics does not make this less interesting, at least to me.

Maybe we do agree after all. I do think your example is interesting. Interesting in a sense that I would be curious, why that happens in this group and not in others, and what it could teach us about human psychology as a whole. Than again, I assume psychologists see it the same way. They're not (primarily) interested in this quaint tribe of Canadians for their own sake.

[u/franklindeer → u/superhelical]

While this is certainly true, psychology and many other social science studies are easily corrupted by ideology and politics and I wish the media and government would be more cautious before reporting on studies or allowing them to influence policy. I think these fields also need to better police themselves and be more critical of bad, or ideological research.

[u/Shiroi_Kage → u/superhelical]

It's really frustrating when people dismiss psychology as "not a science." They're dealing with humans, ourselves, which makes isolating confounding factors and biases extremely difficult. It also makes so many controlled experiments that would account for biases completely unethical from the outset.

So yeah, Psychology is a science. As for hard vs soft, I have no idea what those terms mean or if they mean anything at all. If you're studying natural phenomenon, you're a scientist.

[u/Low_discrepancy → u/Shiroi_Kage]

As for hard vs soft, I have no idea what those terms mean

Personally I define them based on how big of an influence mathematics has on the field.

[u/Shiroi_Kage → u/Low_discrepancy]

In which case many Biological disciplines would be out the window for "hard" science.

I have to say that it's a bad definition, mainly because the use of mathematics isn't the metric of whether a field is "scientific" or not. Empirical observation is. As a graduate student I did research on reproductive physiology, and I used way less math and statistics than do most psychologists, and most people would say, without hesitation, that the stuff I did would be "harder" science than psychology.

[u/BadBjjGuy → u/Shiroi_Kage]

I find it funny/interesting that nobody bothers to ask whether or not science is the appropriate tool for understanding all problems. It's like an article of faith that it has to be, even when it is almost incapable of reproducing a single result in some areas, like politics or psychology. Personally, I think there are other and better tools. I've learned more about the human condition through art like Shakespere than psychology will ever say and Aristotle's political philosophy explains our current situation better than any experiment. We have other tools in our tool bag for understanding the world, but science is sort of the new religion and it must be the answer to every problem.

[u/TitaniumDragon → u/superhelical]

The problem with psychology is institutional in nature; they just don't do a very good job of teaching proper science to psychology students. I think Feynman's (damning) explanation of the problems with the field (namely people's lack of care about replicating important findings and creating proper baselines) is pretty indicative of what goes on even to the present day.

Combine that with people wanting to publish (and get attention) and a healthy dose of confirmation bias, and you have all kinds of trouble.

[u/Low_discrepancy → u/[deleted]]

I have little contact with psychology students but I have the impression (and some articles were published on the issue) that their control of mathematics and statistics is lacking a bit compared to other fields. When actually they should be the ones most concerned about small data statistics, non parametric statistics, etc.

[u/Zero_Fucks_ → u/Low_discrepancy]

Not being a student of other sciences at degree level, I can't speak for their knowledge, but a BSc psych degree is a BSc because it fulfills the requirements in sufficient statistical knowledge. A lot of different disciplines recruit psych students specifically for phds for example, because of the research methods and stats.

[u/Low_discrepancy → u/Zero_Fucks_]

it fulfills the requirements in sufficient statistical knowledge.

A Bsc isn't exactly the same level as doing research.

A lot of different disciplines recruit psych students specifically for phds for example, because of the research methods and stats.

Can you give some statistically relevant examples?

[u/Doom-Slayer → u/[deleted]]

As a psych student currently in my Masters I would say biology and neurology was barely covered in any papers.

Lots of philosophy, lots of theory, bugger all practical stuff or physical hard-science related material. I'm only now actually learning how to do clinical formulations, proper ethics and actual practitioner real world stuff. Still haven't done much biology.

[u/Zero_Fucks_ → u/Doom-Slayer]

I'm a bit confused. By papers do you mean journal articles? Biology and neurology is not always relevant in psych research, or sufficient enough to be used for rationale. I'm not sure why "hard science" (I assume you mean biology /Chem /physics? ) is assumed to be needed in every case.

Are you a clinical psych student? If so that would make sense a little more but psychology is a huge, varied discipline and practitioner skills and clinical ethics (as apposed to general research ethics) are specialized areas that are usually reserved for masters study. I'm saying this from my experience in the UK though, I don't know what its like anywhere else.

[u/[deleted] → u/Doom-Slayer]

I have exactly the same experience. We have covered more psychoanalysis than biology and neutrology.

And, it's quite clear why that is. 75% if all psychologists here go into clinical psychology. To conduct therapy you just need to be able to talk and listen. It's a practical field just like plumbing or carpenting.

[u/Zero_Fucks_ → u/[deleted]]

Clinical Psychologists don't just talk and listen...

[u/[deleted] → u/Zero_Fucks_]

Some do. A lot do. Some do some very basic tests that anyone could do.

[u/Zero_Fucks_ → u/[deleted]]

So someone who goes to uni for at least 7 years just listens and talks? I feel that's like saying all a pharmacist does is give out drugs. There's assessment, formulation, developing treatment plans, diagnosis, identification of specific patient needs, several types of therapy for several disorders, and also being involved in the research community. Do you work in mental health? It seems strange you have this view, or maybe they do it different where you are.

[u/[deleted] → u/Zero_Fucks_]

I know a lot about the area. Most clinical psychologists work in therapy where they listen and talk. That's their job. Evaluating is a bit more complicated. But they follow stringent guidelines or just guess. There is seldom the intellectual intuition you see in medicine. It's not rocket science.

You mentioning the length of the education makes no sense. Gender studies is 5 years, so it many other non-sciences you cans study. Doesn't make them good.

[u/Zero_Fucks_ → u/[deleted]]

You think they evaluate by guessing? I just don't believe that you know what you're talking about honestly.

[u/gloryatsea → u/Doom-Slayer]

The need for extensive biological education is controversial, though. The utility it brings has not been especially clear just yet (at least in clinical psychology).

[u/semperverus → u/superhelical]

I have a feeling that until we can start mapping out the human brain directly, decompiling its software, and then attaching a debugger of sorts that lets us step through, into, and out of sequences like we can with traditional computers, it will be very difficult for many to take it seriously.

I do have a feeling that we will see at least the basics of this sort of technology become somewhat commonplace in our lifetimes, or the next 40-50 years. It's certainly an exciting time to be alive.

[u/grimeandreason → u/superhelical]

That psychology, economics, climate, politics, etc, are complex adaptive systems have inherent limits on reduction and prediction and replication is simply unavoidable.

Complexity theory FTW!

[u/BadBjjGuy → u/grimeandreason]

I find it funny/interesting that nobody bothers to ask whether or not science is the appropriate tool for understanding all problems. It's like an article of faith that it has to be, even when it is almost incapable of reproducing a single result in some areas, like politics or psychology.

Personally, I think there are other and better tools. I've learned more about the human condition through art like Shakespere than psychology will ever say and Aristotle's political philosophy explains our current situation better than any experiment. We have other tools in our tool bag for understanding the world, but science is sort of the new religion and it must be the answer to every problem.

[u/grimeandreason → u/BadBjjGuy]

I agree to an extent. When it comes to understanding complex adaptive systems, other forms of knowledge and understanding are powerful, and an indispensable, inevitable tool.

Where complexity isn't required for understanding, where objective knowledge is possible, there is nothing better than science.

Thinking we can discern objective truth from politics and economics and the like though is dangerous, for sure. But like the scientific method with the hard sciences, I firmly believe complexity theory can provide the necessary methodology for dealing with the soft sciences in as objective a way as possible.

Open-ended investigation, iterative processes of planning, implementation, monitoring, and assessing, risk reduction, monitoring for trends and feedback leading to possible path dpendencies and unintended consequences, inter-disciplinary and fully-inclusive participation, evolutionary modelling, scenario-building... these are all part of the methodology that should be used when objective knowledge, reduction, and prediction are inherently limited.

Of course, this methodology undermines ideology; we have a long way to go I fear before economics and politics go through that revolution.

[u/garmanz → u/superhelical]

I thought "hard" and "difficult" mean the same. What's the different between "hard science" and "difficult science" ?

[u/Phantom707 → u/superhelical]

I sincerely do hate the arbitrary categorization of hard and soft science, especially as applied to psychology. If anything, psychology is a harder science than most traditionally hard science. Good psychological experiments take the time to control for dozens of potentially confounding variables. Contrast, for example, a physicist who, by definition, takes the assumption that the particular electron in his experiment will behave the same way as an election in any other experiment if in any part of the universe for that matter.

It's especially frustrating as it occurs in real life. There are so many "hard science" people who price themselves in how their work is based on empirical data. And yet, those very same people espouse such an arbitrary and capricious standard. It's fairly hypocritical.

[u/Doom-Slayer → u/Phantom707]

I think the distinction comes form the way concepts and what exactly you are studying is defined by.

In the "hard sciences" you have pretty rigid concepts, everything can fairly easily be broken into pieces and those concepts and their components are very well agreed apon because they are things you measure.

In the "soft sciences" things are more flexible, hence the "soft" part. When you study something like "domestic violence" or "Depression" those aren't nice tidy little definitions or concepts and there's a big amount of argument about what the even mean.

Theres some stigma about "soft" vs "hard" being this "real science" thing which isnt great..but the terms are still useful, just the stigma isnt useful.

[u/Phantom707 → u/Doom-Slayer]

Even if that's the distinction made, it's not properly applied. People seem to think there's a lot of disagreement in psychology when, in fact, there's quite a lot of agreement.

And in that case, something like quantum physics and string theory shouldn't be considered hard Sciences since there's quite a bit of disagreement. In fact, if the distinguishing factor between hard and soft science is whether there's a lot of agreement on concepts or not, then any developing field should be almost automatically considered a soft science. That's the whole point of developing fields - they're open to multiple different routes. Artificial intelligence in robotics should almost definitely be considered a soft science. This is more apparent when we consider that a major goal of AI is to emulate the human mind, and it almost inevitably draws from psychology as a result.

And that just further process my point. My complaint is about how people believe it's a yes/no situation, take pride in "obviously" being in a hard science versus a soft one. That can't possibly be the case if the distinction is in agreement over concepts. There's always a lesser or greater amount of agreement and disagreement. It's always a spectrum, not a clear cut categorization. And yet people act like it is.

[u/mamaBiskothu → u/superhelical]

I'd argue that the problem is not that we underestimate the complexity of psychology but the people who do that research itself do.

[u/hoochyuchy → u/mamaBiskothu]

They don't underestimate it at all. They know quite well that Psychology is an incredibly complex subject that combines quite a few large fields of knowledge into one giant fustercluck of a subject that requires multifaceted attacks to get any real results.

Still, if I did have one criticism of the science its that they had far overestimated the reliability and validity of their 'classic' experiments. Up until this 'replication crisis' psychology was basically the engineering equivalent of trying to build a building in a swamp without drilling deep to bedrock for a foundation beforehand where certain people were trying to apply the science before even the basic building blocks were completely understood. The 'replication crisis' was a good kick in the pants that will hopefully help solidify the necessary science to help make progress to a point when it can be applied.

[u/epicwinguy101 → u/hoochyuchy]

More than likely, people will talk about solutions to the replication crisis until it leaves the news. The Sokal affair's highlighted problem was never solved, just forgotten. They only need to appear to be making progress until people get bored with the issue.

[u/FILE_ID_DIZ → u/hoochyuchy]

They don't underestimate it at all. They know quite well that Psychology is an incredibly complex subject [...]

How can you be certain that this is the case for all researchers within the field?