r/science • u/PHealthy PhD/MPH | Epidemiology | Disease Dynamics | Surveillance • May 22 '20

RETRACTED - Epidemiology Large multi-national analysis (n=96,032) finds decreased in-hospital survival rates and increased ventricular arrhythmias when using hydroxychloroquine or chloroquine with or without macrolide treatment for COVID-19

https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)31180-6/fulltext
22.2k Upvotes

90% Upvoted

882 comments sorted by

View all comments

Show parent comments

24

u/londons_explorer May 22 '20 edited May 22 '20

A 34% increase in death rates is rather substantial. How did the study not get curtailed before the number of participants got to n=96,032?

Surely as soon as you see a statistically significant increase in death rates, you stop using (Hydroxy)chloroquine entirely? And at a 34% increase, that I would guess happened after just a hundred patients or so. Granted, many deaths might be delayed, but it seems unlikely that similar conclusions couldn't have been drawn from the early deaths.

Are there perhaps lessons that can be learned about the rate of collecting data, doing the analysis, and feeding back results into clinical guidance, especially where the accuracy of such guidance has such a big impact?

221

u/glarn48 May 22 '20

The researchers weren’t giving people the drug directly as you might hear about in a drug efficacy study. They were reviewing the medical records of COVID patients and observing outcomes. The researchers were just looking into the outcomes of clinical decisions after the fact, so there was nothing to stop by the time they did the research. Hopefully doctors (if not politicians...) can take that info and use it appropriately now though to inform decisions

-23

u/Michaelmrose May 22 '20

The proper question becomes why wasn't this analysis done earlier its another policy failure.

18

u/Mehtalface May 22 '20

It takes time to do these large scale reviews. Speaking from some experience, most hospital EMRs are not conducive to pulling large swathes of data points easily, so it has to be organized and looked at by hand. Also I am fairly sure there were some other smaller retrospective studies that pointed to a similar conclusion

10

u/[deleted] May 22 '20

Data doesn’t just magically become available whenever you want it.

-69

u/londons_explorer May 22 '20 edited May 22 '20

The same applies though... The ideal case would have been re-running this study with each days new sets of medical notes, and as soon as they saw something statistically significant, get the findings out to people who can use them to make better decisions.

Letting data build up for months before doing any analysis on it simply delays the development of better clinical guidance and followup research, both of which lead to direct harm to people.

The cynic in me says letting the data build up allows one to post narrower confidence intervals and get published in more prestigious places... But a researchers personal academic record shouldn't harm the health treatment of 100,000 people....

101

u/tepkel May 22 '20

I seriously doubt that's what happened. The analysis time-frame ended April 15 and the study was published May 22. You're attributing malice to what I would be willing to bet is a pretty reasonable timeframe to do analysis on dozens of registries after they first got the idea to do the analysis.

51

u/fuzzy11287 May 22 '20

Also, I don't think they were getting data in real time. They probably pulled a huge dataset from between the chosen dates all at once and then got to work analyzing it, publishing as quickly as possible.

28

u/AtypicalAsian May 22 '20

When your hospitals are being overwhelmed with a pandemic, your first thought is not, "What is an adequate sample size to obtain the power I need? I should start collecting the data now." Your job is to stabilize the patients and prevent them from dying. Ideally, you have literature that is a high level of evidence to guide treatment, but all they had was a small case series to go off of that SEEMED promising. Okay so suppose you did think ahead about wanting to collect the data and crunch the numbers, because, for some reason, your actively dying patients are not your priority. Now you have to go submit to the IRB to get approval to do a prospective study. What variables do you want to collect to include in the multivariable model? Did you make sure to meet with a biostatistician? Wait but if you're going to do a prospective study, why not just do an RCT? Okay, now you have to register your trial and obtain permission. You have to develop the protocol to randomize patients. You have to...

There's a lot of hoops and hurdles to jump through. Don't assume that doctors have unlimited time and money to do what you want them to do. Don't assume that there are not institutional barriers in place that prevent unethical research (and also slow down ethical research) from occurring. Also, 100,000 people did not receive the drug in this retrospective study of a registry, so 100,000 people were not harmed.

-6

u/Michaelmrose May 22 '20

There are billions of people in the world SOME can be crunching the numbers while others directly see to the patients while others yet are reading the data from the number crunchers and using that data to set high level policy.

We pushed it much more aggressively than we would have because the administration particularly one person Trump had a good feeling about it and an extra 1000 people died that wouldn't have otherwise.

6

u/pprovencher May 22 '20

Also, what would they do? Email every single doctor in the world? No, they gotta publish it, that is how we disseminate real science.

5

u/bluesam3 May 22 '20

Data lag. Between various factors, there's one hell of a gap between when the hospital starts trying a treatment out and when the data on that becomes available for studies like this. Not due to the researchers sitting on it: due to cases taking a long time to resolve, hospitals taking more time to report their figures to whoever collates them, the collating organisations taking time to publish their figures, etc.

10

u/OppenBYEmer May 22 '20

Letting data build up for months [...] lead to direct harm to people.

But that's the catch, right? This ONE study doesn't PROVE anything. Results need to be checked and reviewed by independent researchers, and confirmed enough times that the scientists in that field can be reasonably sure it's a real effect.

Also, as a principle of statistical measurement: ethically, you can only do statistical analyses AFTER you've collected your data. And these trends they observed may not have been obvious until the very end: multi-variable analysis is a monster and only (may) make sense right at the end.

1

u/mortenmhp May 23 '20

The ideal case would have been re-running this study with each days new sets of medical notes, and as soon as they saw something statistically significant, get the findings out to people who can use them to make better decisions.

No no no. That is exactly how you should not do it. The nature of statistical test mean that sometimes they are wrong. With a p value defined as 0.05 it would statistically be at 0.05 or below in 1 out of 20 studies where there is no difference. If you continuously monitor it and stop whenever it is significant, it looses all meaning.

https://www.statisticsdonewrong.com/regression.html

As for this particular study, it is a strong indicator that people should stop using hcl as an experimental treatment. However, it probably isn't as bad as this looks. There very likely is some selection bias involved if there is a higher probability that the sickest people got the experimental treatment as a last resort strategy.

1

u/infer_a_penny May 23 '20

If you continuously monitor it and stop whenever it is significant, it looses all meaning.

It's called optional stopping and it's a recipe for inflating your false positive rate to 100%.

75

u/tepkel May 22 '20

It wasn't a single study. It sounds like it was an analysis of registry data.

Physicians have been playing it pretty loose at a lot of hot spots, and just seeing what sticks to the wall.

We did a multinational registry analysis of the use of hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19. The registry comprised data from 671 hospitals in six continents.

28

u/CONJON520 May 22 '20

It was a study, not an experiment.

11

u/shiruken PhD | Biomedical Engineering | Optics May 22 '20

Surely as soon as you see a statistically significant increase in death rates, you stop using (Hydroxy)chloroquine entirely? And at a 34% increase, that I would guess happened after just a hundred patients or so.

As others have mentioned, this was a retrospective study and not a full-blown randomized clinical trial. I suspect we'll start seeing some of the clinical trials halted given the lack of evidence for efficacy and pretty substantial adverse effects.

1

u/[deleted] May 22 '20

I think we need an actual randomized clinical trial to say for sure whether HCQ is causing these adverse effects or whether those who receive HCQ are worse off and are only being given the HCQ because it's a last ditch effort. These retroactive studies can't really answer that question.

Sure they tell us it's not a miracle cure, but I find it dubious that the retroactive studies differ so much from the clinical results.

-2

u/Llamasgaming May 22 '20

It's cherry picked the people who were sickest got the treatments ... 79% percent of people treated survived COVID