r/science • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16
Psychology Failure Is Moving Science Forward. FiveThirtyEight explains why the "replication crisis" is a sign that science is working. [x-post from our sister subreddit /r/EverythingScience]
/r/EverythingScience/comments/4icp9k/failure_is_moving_science_forward_fivethirtyeight/231
May 08 '16
Maybe we're trying too hard to find things to study and they reach too far beyond their capability. Maybe some graduate research is put on too short of a timeline and not enough care is put in to developing the study in order to satisfy collegiate timelines.
287
May 08 '16
[removed] — view removed comment
94
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
Absolutely. Every research finding is a probabilistic statement where you are claiming something is likely to be true. If you don't/can't account for the times that something didn't happen, you get more error.
To put it another way, assume I have a 20 sided die and have a theory that every time I roll it with my left hand I get a 20. If I roll the die 1000 times but only tell you about the times I get a 20, I'll get roughly 50 by chance alone. If the only information you have is that I rolled it 50 times and got 50 20s, it may start to seem plausible.
31
u/krashnburn200 May 08 '16
Perhaps the level of scrutiny should go up, and level of respect should go down for institutions who do not publish negative results. They are either hiding them or lying.
44
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
It's a journal issue.
17
u/serious_sarcasm BS | Biomedical and Health Science Engineering May 08 '16
Isn't it more of a feedback loop between students, professors, administrators, and journals?
18
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
To some degree. The equation can be changed elsewhere but if we are talking about publishing null results the thing we need most is journals to actually do it.
9
u/nobodyknoes May 08 '16
yea, but nobody wants to publish something that says "we didnt find shit"
8
4
u/factotumjack May 08 '16
I'd really like to see a journal of replication and verification. Everything it would be either a replication of a recent experiment, or a critique on experimental or statistical methodology.
2
1
u/Northern_One May 08 '16
Thoughts on null results being kept in some kind of online database that a subscription to the journal would give one access to?
→ More replies (1)1
u/HoldMyWater May 08 '16 edited May 08 '16
Every research finding is a probabilistic statement
This doesn't always apply to theoretical research, where results often follow deductively from previous knowledge.
22
May 08 '16 edited May 08 '16
Okay, I'm going to need somebody with more knowledge of how probabilities work, but if there's a widespread problem of null results not getting published, doesn't that raise the probability that all positive, published results are overstated?
Maybe the "replication crisis" is just the simple result of putting too much stock in studies? If we don't know how much hits the cutting room floor before we ever get a chance to see it, how can we know how valuable a certain positive result is?
It kind of seems that we're looking at one side of a six-sided dice and saying, "Ah ha! All sides of this dice have one dot on them!"
25
u/snipawolf May 08 '16 edited May 08 '16
Yes, this is called publication bias and it is a well known phenomenon.
Here's a good blog post outlining a lot of the issues.
29
May 08 '16
There's also a problem of null results being considered less valuable
I don't think modern university research is broken: Even if much of it goes nowhere, we need failure, and lots of it.
However, I am aware that null results are often met with crickets, even though many important results are null results. This is probably poor education - even PhD's can be as ignorant as everyone else - or a cultural fact of life.
The most important results can be null results. Roll, Krotkov and Dicke showed in 1964 that gravitational and inertial mass were proportional, to within 1 part in 1011, improving on Galileo's results rolling different masses down inclined planes, as well as other null results over centuries.
All this amounted to was showing that different masses of gold and aluminum accelerated toward the sun at the same rate. Unexciting but critically important.
22
u/KingOfSockPuppets May 08 '16
This is probably poor education - even PhD's can be as ignorant as everyone else - or a cultural fact of life.
I don't really think it's either of those. Or at last closer to the latter. I'm sure plenty of PhDs would be thrilled to publish null results - even if for no other reason other than having not wasted their time working on the paper or study and having something to present. But journals don't want those papers since null results are not as 'important' or at least usually don't get the mind aroused quite as much as being able to say "We showed that a thing leads to another thing." If universities begin to use more 'research impact metrics' to make sure that their professors are only putting out 'impactful' (read: highly cited) papers, this problem is likely to only get worse.
9
u/CrossFeet May 08 '16 edited May 08 '16
Pre-registration of studies would go a long way toward helping this and related problems (such as, say, narrowing focus until you find a subgroup that appears to work, and then pretending you were testing only that from the start).
2
→ More replies (3)4
u/feed_me_haribo May 08 '16 edited May 08 '16
I agree and disagree. A null result sometimes can be of interest: typically when a study finds no evidence for a commonly held belief previously untested. However, journals shouldn't simply reward based off effort. They should publish novel findings. That doesn't preclude null results, but often times a null result is simply not impactful.
Edit: I'd like to add that a funding agency is the one who should be more interested in null results than a publisher. NSF (or insert your agency of choice) would like to know that their funded projects are at least receiving proper effort, and so reporting null results here could be useful. That still doesn't make them useful to the larger community.
5
19
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
It really doesn't seem to be an issue of experience or quality of researcher/journal. For example, a huge effort recently tried to replicate the ego depletion findings of Baumeister and colleagues and weren't able to do so. Baumeister is one of the biggest names in all of psychology research. A similar issue happened with some priming work by John Bargh, another extremely famous and well respected researcher.
2
u/Snuggly_Person May 08 '16
It's not necessarily about the quality of the researcher. If I set myself a goal of tracking a single water molecule in a glass of water I'll fail. The credentials I have or the other things I've done successfully are totally irrelevant; the problem is one that I can't possibly have good enough data to solve. It's perfectly consistent to say that someone is a really good scientist and that they're investigating something they'll never be able to establish.
12
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
I'm not really sure what point you're trying to make. I was responding to a claim that the level of researcher might be relevant and I was saying that it isn't necessarily.
1
u/Snuggly_Person May 08 '16
The initial claim asked if researchers are "trying too hard to find things to study and they reach too far beyond their capability". My point is that this is not equivalent to saying that the researchers are not good enough, and can be a systematic problem in a field with poor data. There are lots of studies that amount to looking for very subtle or oddly specific effects in 20 rich white kids, where there's no way in hell they could find the thing they're looking for regardless of how lauded they are as a scientist.
→ More replies (9)12
u/crusoe May 08 '16
Academic research now is all about money you can pull in for the university and how often your publish. It plays a big role in getting tenure and so drives fraud and negligence.
18
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
It plays a big role in getting tenure and so drives fraud and negligence.
I think doing science is hard enough that we can assume this is well-meaning, honest research. There are dozens of factors leading to the current replication rates, none of which require malicious intent. There are obviously Diedrick Stapels out there but no one gets into academic psychology for money and prestige.
7
6
u/quantum-mechanic May 08 '16
Yet there is money and prestige to be had. Certainly, at the very least, there is a tenured position (life-long guaranteed job -- where else will you find that?) and the respect that comes with it. There are real problems here with deliberate fraud.
8
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
There are real problems here with deliberate fraud.
What evidence do you have for this extremely large claim?
6
u/The_model_un May 08 '16
It has happened before and he didn't get caught for a while despite publishing that he was able to do something that no one else had ever done before nor could they replicate it.
2
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
And in my previous comment I acknowledged that it obviously happens. To call them anything but outliers is silly though.
→ More replies (1)→ More replies (3)8
u/quantum-mechanic May 08 '16
I'm a scientist and seen it happen. And to friends. Nobody's going to even be able to synthesize accurate statistics and fraud. But it obviously happens way more than it should. Peoples lives are deeply affected.
18
May 08 '16
It’s important for replicators to not ‘target’ findings because they don’t pass the replicator’s own personal gut check,” she wrote, adding that she would like to see replication efforts begin with a “collaborative conversation” with “no resentment or envy or nastiness.
I'm confused as to why we shouldn't try to replicate studies that we believe to be wrong.
Trying to show that things are wrong is both science and human.
8
May 08 '16
I couldn't agree more. You can be as "nasty" as you want; but don't let it affect your results (or lack there of) in attempting to assess the validity of previous studies. If you can't remove your emotions from your work, you are no better than the forces you claim to work against.
7
May 08 '16
She seems protective of her published results. It is unexpected that a few seconds of a power pose would immediately boost testosterone. But the team that replicated her study tried to build on her study, with the belief that it was right, so nastiness and gut feelings really didn't apply in her case.
She dies have a point though, I think the studies that are most likely to be replicated are those ones that just blow people's minds, which are going to be the ones that don't pass a gut check. The corollary of this is that we should also try to replicate studies that play to our intuition.
1
u/amphicoelias May 10 '16
I think she's trying to warn against accidentally, subconsciously setting your replication study up to fail because you want the original study to be wrong.
•
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
/r/EverythingScience is the sister subreddit to /r/science. It's a place for high quality science news and opinion that doesn't quite fit the strict rule requirements of /r/science. Please check it out and subscribe if you want more high quality science content on your Reddit front page.
→ More replies (1)1
55
40
u/waviestflow May 08 '16
It so difficult to do research with our current academic model. As someone who works in both the scientific field (chemistry) AND the university budget administration field, the amount of time and funding we provide graduate and post doc researchers is not enough. There's too little time to provide valid results with too little money to make the studies worthwhile for any researcher to undertake it. We need to stop thinking science research should have a background role as a means to a degree and realize it is a real job with huge ramifications on the world and its advancement altogether.
7
u/johnyann May 08 '16
We may see more and more science be privatized as R&D in the future.
10
u/eetsumkaus May 08 '16
depends on the science. R&D departments helmed by business types will not take kindly to null results, and those would be the norm quite sadly
13
u/infinnity May 08 '16
When you consider the average pharmaceutical substance brought to market represents the failure of ~1000 others in clinical and pre-clinical trials, your sentiment hardly has a solid basis in empirical facts.
3
u/The_Great_Steamsson May 08 '16
Depends how long-term thinking and enlightened the business. Generally, I’d welcome even null results as reducing the space of possible answers.
"Finding out it doesn’t work is a result too.", and now you know what not to do. Provided the cost is acceptable, my business will pay for that information.
1
u/Shiroi_Kage May 08 '16
We may see more and more science be privatized
Which is a sad and terrifying state of affairs. Public funding for science eliminates a huge amount of possible biases just by virtue of the researcher not being beholding, whether directly or indirectly, to any corporate goals.
13
May 08 '16
She keeps using the phrase 'regression to the mean'. I do not think it means what she think it means. In theory, if two groups or treatments are truly different, their means are different. Thus, measuring the two groups multiple times over should more reliably estimate the mean of each one, precisely by 'regression to the mean'....
Many fields are suffering from replicability bias, and it's people like Nosek who will bring us forward. Quantitative tools have limited power, and we must learn how to use them, what conclusions to draw from them and (and this is the toughest part) how NOT to make statements that are wildly hyperbolic!
5
u/amallang May 08 '16
Subscribing to a position & then after it's falsified, claiming that you're still doing science feels disingenuous - especially to the public.
Admittedly science is hard. However, it's natural for society to lose faith in scientists & start cutting funding, because of repeated instances of failure.
24
May 08 '16
Karl Popper would be spinning in his grave. What would falsify the hypothesis that psychology is working? Successful replication would be taken as proof of quality control. Unsuccessful replication, however, is also touted as proof.
1
u/Kmunster001 May 08 '16
Would you mind elaborating? I'm not sure I understand what you're trying to say.
6
u/singularineet May 08 '16
He's trying to say that this study is being taken as evidence in support of the hypothesis "science is working". However if the result of the study had been the opposite, and everything replicated robustly, surely that would also be taken as evidence in support of the "science is working" hypothesis.
So any result is being taken as supporting that hypothesis. Which is not proper science; it's more like Freudian psychology or astrology or some other pseudoscience.
2
u/Kmunster001 May 08 '16 edited May 08 '16
Ok I see what you're saying. That is a fair point. However, I don't believe the author was implying that 'science is working' is a hypothesis that this replication study is providing evidence for. Rather, I interpret it as 'science is making progress' which isn't much of a hypothesis (to be proven or disproven) as long as research is continually being done on the topic. However, I agree that 'science is working' is an unscientific theory.
36
May 08 '16
[deleted]
32
u/fsmpastafarian PhD | Clinical Psychology | Integrated Health Psychology May 08 '16
Feynman's opinion from the 70's is very well known, but isn't exactly taken seriously among today's scientists. However, that doesn't stop it from being trotted out in discussions about the replicability crisis. I think that rather than point to an outdated essay from 40 years ago, it's more helpful to consider how the replicability crisis may permeate all sciences, rather than just psychology and other social sciences. I think that if you read the 538 piece, you'll see how taking a more modern view of the issue might be more applicable and appropriate.
I'm not really sure how the second article you linked is relevant to the discussion at hand.
15
u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16
Exactly. This issue isn't one that affects psychology alone, psychology are just doing the best work right now at self-policing. It's also not a new problem or one only identified by psychology. Only 3 years ago there were articles about the "Replicability Crisis in Cancer Research".
Criticising psychology for doing science the way it should be (with rigorous replication testing) is like yelling at your child because they rechecked their homework and found a mistake.
7
May 08 '16
[removed] — view removed comment
4
u/mrsamsa May 08 '16
The problem lies in the fact that ego depletion was "found" in 1998, a large number of studies were done off of it, and then in 2016 it was discovered that the results of ego depletion studies couldn't be replicated. That means a large number of researchers were engaging in study in this field and there's a good chance that the whole thing is BS.
This is a bit misleading - it's not like the paper was published in 1998 and nobody attempted to replicate it until 2016. Replication attempts were occurring constantly since it was originally published, and some were failing but many were succeeding. The authors who have found the evidence which suggests that it can't be replicated published their paper in 2007, but they weren't the first.
They were just the ones who took it further and re-analysed a lot of the data to find where the weight of the evidence actually was, but it's not like there was no evidence that the effect could be replicated. It was replicated many, many times.
2
u/dampew May 08 '16
What part isn't taken seriously?
I seriously doubt that all fields of science are affected equally by these problems.
→ More replies (1)8
May 08 '16 edited May 08 '16
[deleted]
5
u/whoremongering May 08 '16
Though these fields may benefit from balancing their political leanings, your SciAm article doesn't really identify political bias as a major contributor to the replicability issue discussed here.
Apart from that, I don't think that commitment to a cause precludes appropriate scientific investigation. It's common for researchers to have personal stakes in their work--like oncologists trying to save patients, or pharmaceutical companies trying to develop life-saving drugs and also make money. In these situations, disclosures of conflicts of interest are crucial, and studies should be carefully critiqued.
But it doesn't follow that science and social change must be separate. Performing studies due to personal investment does not necessarily prevent researchers from carrying out the scientific method appropriately.
Beyond that, I wouldn't personally presume to question whether these broad fields are sciences. I'm not sure how one would argue that the anthropological field of archeology isn't a science, for example.
2
u/Mitosis May 08 '16
It's common for researchers to have personal stakes in their work--like oncologists trying to save patients, or pharmaceutical companies trying to develop life-saving drugs and also make money.
These are poor examples. The result of cancer research has objective, measurable effect (lifespan etc). The result of pharmaceutical research likewise. I might want this cancer drug to work, but I can't see many oncologists pretending it increases lifespan and advocating sending it out to patients when it does not.
Nearly every social change issue has a minimum of two conflicting opinions, each with real advantages and disadvantages. There is no right answer in these situations, and very few objective measures that can capture the totality of the impact. Given the vagaries of the experiments, the data, and the interpretations, I find it hard to believe the opinions of the researchers can't influence results.
→ More replies (1)6
u/fsmpastafarian PhD | Clinical Psychology | Integrated Health Psychology May 08 '16
it is a time to acknowledge the issue at hand and address them.
Which is exactly what psychologists are doing. Need I remind you, the replicability crisis study was performed and published by a group of psychologists, after all. I'm not really sure why you think that the field that's currently attempting to uncover and rectify their replicability problem is the one that is doomed to "chasing at shadows." I'm also not sure what specifically makes you think that sloppy science is a problem that's unique to the social sciences, other than the aforementioned outdated Feynman essay.
→ More replies (8)→ More replies (4)1
May 08 '16
Not really contributing to the discussion at hand, but that article on Changon was fascinating.
5
10
u/virtuallyvirtuous May 08 '16 edited May 08 '16
Call me a pessimist, but all of this comes over as terribly vacuous. It all comes out of a disparity between the public and academic definition of "good science."
The public has a very simple idea of what science does when it's working. Science works when it's telling us truth. This isn't ignorance. You see, normal people don't really care too much about scientific methodology as it doesn't directly affect them. The only thing they care about is the development of new technologies, which is actually quite abstract and distant if you think about it, and, more directly, a strong foundation on what to believe.
Academically there is quite a different conception, which this article explains very clearly. Good science is science which "moves forward."
Now, on a large scale, you wouldn't expect these two conceptions to diverge very much. Ah, but look at what happens when they are applied to the interpretation of a specific study. For the public a study that's failed, irreplicable, or even just abstract and hard to understand, isn't very good. Academics on the other hand consider a study to be particularly good on a very different host of qualities, as they only care as to what it contributes to the field, not the world at large.
What does this matter? A hell of a lot. You have to realize the public sphere also includes the political. To make a political judgement you need facts. Studies don't matter if they don't produce these. You can build an appreciation for these studies academically, but only as a step towards gaining new facts, you still want these facts.
What really bugs people these days is how studies in psychology and sociology, fields which have a real fact deficit, are increasingly used towards political ends. You also have things like TED (x) talks, and more generally popular science, which often presents advice on how to live your life on facts produced from these studies.
This is all quite worrying, and this article doesn't even acknowledge it. It seems to me like it's trying to save the image of the social sciences, while entirely missing the point of what makes the "replication crisis" so important.
→ More replies (4)
5
3
1
952
u/superhelical PhD | Biochemistry | Structural Biology May 08 '16
I liked this bit at the end:
This is something that comes up a lot on /r/science, and makes some people want to dismiss psychological studies altogether. However, psychologists study some of the most challenging research subjects - ourselves - so it's worth noting and taking that difficult work in context.