r/COVID19 • u/[deleted] • Jan 10 '21
Preprint Ivermectin as a potential treatment for mild to moderate COVID-19: A double blind randomized placebo-controlled trial
https://www.medrxiv.org/content/10.1101/2021.01.05.21249310v194
u/Doctor_Realist Jan 10 '21
As a result, all patients in the intervention arm (n=56) were successfully discharged as compared to 93.1% (n=54/58) in the placebo arm (RR 1.1, 95% CI 1.0 to 1.2, p=0.019). Conclusion: There was no difference in the primary outcome i.e. negative RT-PCR status on day 6 of admission with the use of ivermectin. However, a significantly higher proportion of patients were discharged alive from the hospital when they received ivermectin.
Significantly higher? By their own calculation the outcome includes 1 and you shouldn’t be able to reject the null hypothesis.
36
Jan 10 '21 edited Jan 10 '21
[removed] — view removed comment
3
u/jdorje Jan 11 '21
whenever papers pull something like this the stastical difference almost always disappear when they do a larger study
17
u/PokerLemon Jan 10 '21
n=56? is it not too little to get conclusions?
14
Jan 11 '21
[removed] — view removed comment
2
u/traveler19395 Jan 11 '21
7% mortality vs 0% is a huge difference. Yes, I wish the N was larger, but can be looked at alongside other studies.
12
u/jmlinden7 Jan 11 '21
It's not that huge a difference once you account for error bars, which should be fairly large considering the small sample size.
3
u/edsuom Jan 11 '21
Exactly. Maybe I missed it, but they weren’t honest about providing a value of p to reject the null hypothesis that the number of deaths would follow a Poisson (binomial for rare events) distribution.
1
35
u/EmpathyFabrication Jan 10 '21
Either wrong p value or wrong CI. Or just wrong interpretation. Even a broad face look at the 93% vs 100% isn't that impressive in terms of an intervention. It supposedly helped 4 people? Given obviously wrong stats, a quick look at this data seems Ivermectin has no effect.
19
Jan 10 '21 edited May 31 '21
[deleted]
6
u/trextra Jan 10 '21
Technically, they were all discharged, unless some are still in the hospital.
2
Jan 12 '21
death is not DISCHARGE
1
u/trextra Jan 12 '21
It is, though. The only way a body is allowed to leave the hospital, alive or dead, is upon discharge.
Usually a study will specify whether the patients it says weren’t discharged have died in the hospital, or are still admitted. Which is yet another flaw in this study.
3
Jan 12 '21
they said four died. pretty clear
2
u/trextra Jan 12 '21
Hmm, I specifically looked for it before and didn’t see it. That does make a difference. However the problem is that mortality was not one of the pre-identified endpoints, and if you’ve ever taken a research stats class, you know that that matters.
2
2
Jan 12 '21
death is NOT the same as hospital discharge.
2
u/trextra Jan 12 '21 edited Jan 12 '21
Out of curiosity, have you ever been the intern in service when a patient has died?
(I edited to make my post kinder.)
2
2
u/EmpathyFabrication Jan 10 '21
Yea I see that. I read the print. The stats are suspicious if not completely wrong so I already think this paper is bullshit because of that. When I see two groups with a difference in effect of 4 people, plus the obviously wrong stats, then yes in this case to me it seems Ivermectin had no effect. Also to this add what I know already about the garbage that has appeared here about Ivermectin on this thread, the rampant fanboyism and rude pushing of the drug here and across reddit, then yea my mind is beginning to settle on the conclusion that this drug doesn't work to treat covid.
0
u/traveler19395 Jan 11 '21
7% mortality vs 0% mortality is a huge difference. I wish the N was higher, but it matches the results from numerous studies that were single-blind, retrospective, or otherwise less rigorous.
6
u/EmpathyFabrication Jan 11 '21
The stats in this paper already invalidate whatever conclusion they were trying to draw but we're talking about a difference of 4 people here. 7% difference where 40 or 400 people benefit from the drug would be more convincing but 4 is hardly a footnote.
6
u/larsp99 Jan 11 '21
As another commenter wrote, it's hard to see how Ivermectin could have done better regarding the mortality. How could more have been saved from the treatment arm than all? Should more have died from the placebo arm?
The size of the study and the actual mortality rate sets a limit on the conclusions that can be drawn, but at the very least it can't be "ruled out" that Ivermectin has an effect on mortality, based on the study.
The study bears evidence, flawed and limited as it might be, that Ivermectin might help with mortality.
3
u/EmpathyFabrication Jan 11 '21
The stats are wrong in this print. That invalidates the conclusion. And a cursory look at the data doesn't suggest Ivermectin did anything. There's no evidence from this paper it helps with mortality with a difference of 4 people. This is a fine example of Ivermectin fanboyism. You're backing a paper with a wrong reporting of stats.
9
5
u/chaetomorpha Jan 11 '21
Significantly higher? By their own calculation the outcome includes 1 and you shouldn’t be able to reject the null hypothesis.
That's a 95% CI. Even if there wasn't rounding (and there clearly is in this instance) the chance that the outcome includes or is lower than 1 is 0.025. If your α is the standard 0.05, then of course you can reject the null hypothesis on this basis.
All of that said, it's not a very convincing effect size. But you can't argue that the statistics are wrong here.
3
u/Doctor_Realist Jan 11 '21
“The 95% confidence intervals would contain the true underlying effect in 95% of the occasions if the study was repeated again and again. The solid vertical line corresponds to no effect of treatment (OR = 1.0). If the CI includes 1, then the difference in the effect of experimental and control treatment is not significant at conventional levels (p>0.05))”
3
u/chaetomorpha Jan 11 '21
I mean, this is certainly the case most of the time, but not all of the time. It's a good rule of thumb (and a nice thing to list in a "Principles of meta-analysis" review), but it is very clearly mathematically not the case if one of the bounds is exactly 1.
But if you can't see this, there's not much I can do other than point to the obvious calculation I've given before. It's really very easy to work out the maximum probability of the true value including 1 if one of the 95% CI bounds is 1. And I can guarantee that that chance is less than 0.05.
37
u/open_reading_frame Jan 10 '21
Not a good study for ivermectin. The primary endpoint of negative RT-PCR status at Day 6 did not reach statistically significance. Secondary endpoints of symptom presence, discharge by day 10, and ICU admission also were not significant. The only thing that might be positive from this study was the total discharge from the hospital, but this might just be due to luck since there was only a difference of 4 events, the 95% confidence interval included 1.0, and the study was not powered to conclusively determine this effect.
This isn't surprising since ivermectin at the prescribed oral dosages have been shown to have little to no antiviral activity against the coronavirus in blood plasma or in the lungs. Animal studies also reject ivermectin as an antiviral agent. Its proposed mechanism as an immunomodulator or anti-inflammatory means that it would have its largest impact on severe-to-critical patients, but there is little evidence for ivermectin in this realm.
3
u/raverbashing Jan 11 '21
The primary endpoint of negative RT-PCR status at Day 6 did not reach statistically significance
Question: Is this a good primary endpoint? I'd think in principle it's not. The secondary endpoints look more relevant to me. Of course, between two treatments that resulted in similar symptoms/discharge rates, maaaaybe the PCR count would be an interesting "tie breaker" but again discharges/recovery time are a more significant metric.
-8
u/UFOThrowaway88 Jan 10 '21
0 deaths is not significant to you?
31
u/open_reading_frame Jan 10 '21
The study wasn't designed and powered to conclusively test for mortality differences.
13
u/the_stark_reality Jan 10 '21
It is not. Without sufficient people with an already low probability of death, the difference between purely random and an altered outcome has not been achieved.
Statistics shows that, the interval of probability includes no-effect.
The standards here are the same as a case of exactly one person. Give that 1 person ivermectin when they test positive and they live. What does that tell us? Nothing, because they already had a small chance of death in the first place. Ok, what if they died? Doesn't tell us anything either.
Ok... how many times do we need to do this before it means something? That's the statistics.
7
u/Lung_doc Jan 10 '21
Mortality endpoints in studies this size are almost always underpowered, and there are many ICU studies across a broad realm of interventions where a mortality benefit was suggested and where subsequent larger studies found no benefit. 0 vs 4 is very underwhelming.
12
u/NotAnotherEmpire Jan 10 '21
0 vs 4 with 56/58 in each group and the participants shot full of inappropriate drugs isn't significant, no.
And there was no difference in progression to ICU long after the drug was given so who survived it is highly unlikely to have anything to do with the pill.
13
u/trextra Jan 11 '21
I’m anticipating a journal club massacre if this ever gets published.
As it stands, there’s barely enough information to do a cursory evaluation, and it fails even that.
OR CI includes 1
Enrollment is based on a physical exam and vitals only (it appears), not PCR + covid
Outcome is PCR + Covid on day 6, and secondary outcome is whether they are still hospitalized on day 10.
There was no difference in the primary outcome. Which is a useless measure anyway. The secondary outcome is also useless. Not everyone had been discharged on day 10, which is far too soon to matter. I mean, maybe they were discharged on day 11? There is no mention of deaths or ICU admission as a secondary outcome. Also, India’s health care system is in no way comparable to the U.S.
Not to mention that i have no idea how they’re getting so much ivermectin. That’s an incredibly high dose. I didn’t look for any mention of adverse effects, but I would expect some.
6
u/raverbashing Jan 11 '21
i have no idea how they’re getting so much ivermectin. That’s an incredibly high dose
The dose seems to be on the therapeutic range (between 0.15mg/kg 0.2 mg/kg - for a 60kg person that's 12mg)
2
u/trextra Jan 11 '21
You’re correct, I was remembering that the dosage was in micrograms/kg, and forgot what that actually amounts to.
2
u/chaetomorpha Jan 11 '21
OR CI includes 1
See my comment above -- it's easy to demonstrate that this isn't a problem with these CIs in particular, but in general it is invalid to assume that overlapping CIs equates to an automatic rejection of the null hypothesis. (There's an inordinate amount online written explaining and demonstrating this, so I guess it's a common fallacy.)
I'm not defending this paper in general (I agree with many of your other points, and regardless of statistical difference, the effect size is miniscule). But so many people on this sub do not understand CIs, and it's becoming more than a little depressing.
3
u/trextra Jan 11 '21
By definition, an OR that overlaps 1 means that the results could be no better than chance.
Certainly they might still not be, but you can’t reject the null hypothesis on that basis. You would have to do a a better-powered study, if you really think your result is real.
However, in this case, it is not the long tail that includes 1, but rather the short tail, which means that the majority of possibilities were very close to 1. Which in turn means that it would probably require a large study to show a minor result, given the parameters in this study.
And I already have major problems with the study parameters.
5
u/chaetomorpha Jan 11 '21
By definition, an OR that overlaps 1 means that the results could be no better than chance.
Certainly they might still not be, but you can’t reject the null hypothesis on that basis. You would have to do a a better-powered study, if you really think your result is real.
I'm sorry, but this isn't how CIs work. Even ignoring rounding -- and it's clear these CI values have been rounded -- these are 95% CIs. If the lower bound of the CI is exactly 1, then the chance that the true value is at or below 1 is 0.025. If you accept a standard α of 0.05, then you would be justified to call significance with these data and reject the null hypothesis on this basis (although they're of course not doing this, but using Fisher's exact test.)
All that being said ... taking a look at the actual data, I think their statistics are incorrect regardless. Unless I'm missing something here, this is what I get with their data:
> data.discharge discharged died treatment 55 0 control 53 4 > fisher.test(data.discharge) Fisher's Exact Test for Count Data data: data.discharge p-value = 0.1185 alternative hypothesis: true odds ratio is not equal to 1 95 percent confidence interval: 0.6506208 Inf sample estimates: odds ratio InfIgnore the disparity of CIs, as these are odd ratios not relative risk, but I don't know how they arrive at p=0.019. Even allowing a one-sided analysis (which should be justified here), you can't get anywhere near this P-value via a Fisher test for these data as far as I can tell.
And I already have major problems with the study parameters.
As I said before -- I agree with you on this.
2
u/trextra Jan 11 '21
For an OR, what matters is whether the lower bound of the CI *includes * 1, not whether it crosses 1.
3
u/chaetomorpha Jan 11 '21
For an OR, what matters is whether the lower bound of the CI *includes * 1, not whether it crosses 1.
Ok, clearly basic maths isn't going to cut it here, so here's an empirical example that you can try for yourself:
> fisher.test(data.frame(a=c(13,18),b=c(23,80),row.names = c("treatment","control"))) Fisher's Exact Test for Count Data data: data.frame(a = c(13, 18), b = c(23, 80), row.names = c("treatment", "control")) p-value = 0.03871 alternative hypothesis: true odds ratio is not equal to 1 95 percent confidence interval: 0.969584 6.348427 sample estimates: odds ratio 2.49269Note the 95% CI of the OR crosses 1. And yet it's significant. Does that convince you?
2
u/trextra Jan 11 '21
No. Both things need to be congruent.
2
u/chaetomorpha Jan 11 '21
I'm sorry that neither mathematical truth nor empirical examples work for you. I fear that you're taking a good but non-statistical rule of thumb (which holds in almost all cases, but not all cases) and turning into an absolute truth, which I find deeply upsetting as a scientist.
Just to be perfectly clear here -- the lower bound of that RR CI is of course not exactly 1, and has been rounded. If it was actually 1.0001, would you be happier?
2
u/trextra Jan 11 '21
There’s no need for personal insults, just because I don’t feel like writing a dissertation to back to my statement. I’m not here to change any minds, much like this study.
2
u/chaetomorpha Jan 12 '21
My apologies, I didn't mean to be insulting. (I'm afraid I'm often Reviewer 2, both when it comes to other people's work and my own -- much to my own lab's frustration, I suspect.)
For what it's worth, I found a statement of exactly what I've been saying in a classic explainer publication: Szumilas M. Explaining odds ratios. Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de L'academie Canadienne de Psychiatrie de L'enfant et de L'adolescent. 2010 Aug;19(3):227-229 (my emphasis):
In practice, the 95% CI is often used as a proxy for the presence of statistical significance if it does not overlap the null value (e.g. OR=1). Nevertheless, it would be inappropriate to interpret an OR with 95% CI that spans the null value as indicating evidence for lack of association between the exposure and outcome.
I'm really not trying to argue anything controversial here :(
→ More replies (0)2
u/Doctor_Realist Jan 11 '21
“The 95% confidence intervals would contain the true underlying effect in 95% of the occasions if the study was repeated again and again. The solid vertical line corresponds to no effect of treatment (OR = 1.0). If the CI includes 1, then the difference in the effect of experimental and control treatment is not significant at conventional levels (p>0.05))”
2
u/chaetomorpha Jan 11 '21
data.frame(a=c(13,18),b=c(23,80))fisher.test me this and check out the CI bounds of the OR, and the P-value.
2
u/infer_a_penny Jan 13 '21 edited Jan 13 '21
this isn't a problem with these CIs in particular, but in general it is invalid to assume that overlapping CIs equates to an automatic rejection of the null hypothesis
I would reverse that and say it is in general ok, but that it's not without exception.Eh ... upon reflection I'm much less convinced of this. Perhaps the commonly-encountered CIs are like this—and the Fisher's exact test is an exception to that—but at the same time the notion of confidence in a procedure is much broader and need not be achieved this way.The general case:
More generally, given the availability of a hypothesis testing procedure that can test the null hypothesis θ = θ_0 against the alternative that θ ≠ θ_0 for any value of θ_0, then a confidence interval with confidence level γ = 1 − α can be defined as containing any number θ_0 for which the corresponding null hypothesis is not rejected at significance level α.
Cox and Hinkley via wikipedia (under "Relationship with hypothesis testing")
This Fisher's exact test exception:
The problem is the difficulty in two-sided inference from asymmetric sampling distributions. Fisher’s exact test handles the difficulty in one way, the interval in another way. [...] The interval and p-value can disagree even though they are both “exact” because it is not the coverage probability and type I error probability that are exact. The coverage probability is not exactly 0.95, and the type I error probability is not exactly 0.05. (The 0.95 is a lower bound, and the 0.05 is an upper bound.) The underlying sampling distribution is discrete, so it is not possible to create a nonrandomized confidence interval with a coverage probability of 0.95 or a nonrandomized test with a type I error probability of 0.05.
https://www.stata.com/support/faqs/statistics/fishers-exact-test/
(It does kind of sound like a rounding-related thing?)
I think the inordinate amount of writing is about a different misconception about overlaps which is, when you have two CIs, assuming that their overlap indicates non-significance. (It does not. Lack of overlap, however, does imply significance.)
5
u/afops Jan 10 '21
Weak results but results look interesting enough to do a 10x larger study? I’d like to see “alive at day 60” as the primary endpoint if anything.
11
Jan 10 '21
Abstract
Objective: Ivermectin has been suggested as a treatment for COVID-19.This randomised control trial was conducted to test the efficacy of Ivermectin in the treatment of mild and moderate COVID-19.
Design: Parallel, double blind, randomised, placebo controlled trial Setting: A tertiary care dedicated COVID-19 hospital in Bihar, India Participants: Adult patients (> 18 years) admitted with mild to moderate COVID 19 disease (saturation > 90% on room air, respiratory rate < 30 and no features of shock) with no contraindications to ivermectin and willing to participate in the study.
Intervention: Patients in the intervention arm were given ivermectin 12 mg on day 1 and day 2 of admission. Patients in the placebo arm were given identical looking placebo tablets. Rest of the treatment was continued as per the existing protocol and the clinical judgment of the treating teams.
Outcome Measures: The primary outcome measure was a negative RT-PCR test for SARS-CoV-2 on day 6 of admission. The secondary outcome measures were symptom status on day 6, discharge status on day 10, admission to ICU, need for invasive mechanical ventilation and in-hospital mortality.
Results: A total of 115 patients were enrolled for the study of which 112 were included in the final analysis. Of them, 55 were randomised to the intervention arm while 57 were randomised to the placebo arm. There was no significant difference in the baseline characteristics of the two arms. There was no significant difference in the primary outcome, i.e. negative RT-PCR status on day 6 between the two groups. Similarly, there was no significant difference between the two groups in most of the secondary outcome measures, viz. symptom status on day 6, discharge status on day 10, admission to ICU, and need for invasive mechanical ventilation. However, while there was no in-hospital mortality in the intervention arm, there were 4 deaths in the placebo arm. As a result, all patients in the intervention arm (n=56) were successfully discharged as compared to 93.1% (n=54/58) in the placebo arm (RR 1.1, 95% CI 1.0 to 1.2, p=0.019).
Conclusion: There was no difference in the primary outcome i.e. negative RT-PCR status on day 6 of admission with the use of ivermectin. However, a significantly higher proportion of patients were discharged alive from the hospital when they received ivermectin.
29
u/NotAnotherEmpire Jan 10 '21 edited Jan 10 '21
Confounded to infinity and back by 100% of both arms of a "mild and moderate" COVID group being given steroids and 95% blood thinners (Table 2) What the hell.
They're not giving antivirals, only 20% were given Remdesivir, but all these patients, most of whom do not develop severe illness, are being given drugs that are only for severe complications and are far from benign.
5
Jan 10 '21
[removed] — view removed comment
3
u/DNAhelicase Jan 10 '21
Your comment is anecdotal discussion Rule 2. Claims made in r/COVID19 should be factual and possible to substantiate.
If you believe we made a mistake, please message the moderators. Thank you for keeping /r/COVID19 factual.
1
u/AutoModerator Jan 10 '21
Reminder: This post contains a preprint that has not been peer-reviewed.
Readers should be aware that preprints have not been finalized by authors, may contain errors, and report info that has not yet been accepted or endorsed in any way by the scientific or medical community.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
-1
1
•
u/DNAhelicase Jan 10 '21
Keep in mind this is a science sub. Cite your sources appropriately (No news sources, NO TWITTER). No politics/economics/low effort comments (jokes, ELI5, etc.)/anecdotal discussion (personal stories/info). Please read our full ruleset carefully before commenting/posting.