The Potemkin Argument, Part 12: Bayes Wept

This is a public peer review of Scott Alexander’s essay on ivermectin, of which this is the twelfth part. You can find an index containing all the articles in this series here.

If you were to choose one scientist to answer the ivermectin question, it would be likely to be this one:

He’s published on ivermectin since 2011—focusing on its antiparasitic action—so this is no bandwagon-jumper who doesn’t know what he’s talking about. At the same time, he understands viral infection, having a publication history on HIV stretching all the way back to 2000.

Early Pandemic Research

As one would hope, when the pandemic became a worldwide concern—and ivermectin was noted as a potential treatment—Krolewiecki did not sit on his hands. In July 2020, in collaboration with a team in Barcelona, Spain, he published a small study called Lack of efficacy of standard doses of ivermectin in severe COVID-19 patients. This is an intriguing little retrospective study that found 13 patients that were treated for COVID-19 in the earliest days of the pandemic. They had been given a standard dose of ivermectin (1x 200μg/kg) as part of their treatment regimen—presumably because the doctors suspected there may also be a parasitic infection—and compared them with 13 similar patients who had not been given ivermectin.

Now, if you know me, you’re probably expecting some comment about the overconfident title, given the fact that there were only 26 patients in the study. There will be no such complaint, because the authors actually published a correction, changing the title to clarify exactly this (bold and italics mine):

It is noted that a non-significant result does not confirm the absence of an effect. Therefore, the authors provide the following corrections to errors in the title, and in the Discussion section, in the first sentence of the first paragraph.

The correct title of the manuscript is: Lack of evidence for efficacy of standard doses of ivermectin in severe COVID-19 patients.

Yup. This is how it’s done. Let’s also note that this does not look like the kind of researcher that will balk at publishing a negative result.

Having moved past the hypothesis that standard single doses of ivermectin could have a substantial effect—especially when given late—he and his team ran a small pilot RCT for much higher doses (5x 600 μg/kg).

Scott’s Review

This is the study that Scott Alexander commented on, so let’s see what he has to say:

Krolewiecki et al: Another Argentine study. This one is a real RCT. 30 patients received ivermectin, 15 were the control group (no placebo, again). Primary outcome was difference in viral load on day 5. The trend favored ivermectin but it was not statistically significant, although they were able to make it statistically significant if they looked at a subset of higher-IVM-plasma-concentration patients.

In the case of this study, focusing on statistical significance is particularly odd, because it was designed as a pilot trial—intended to help the design of a future trial which would be powered sufficiently to detect statistical significance.

The paper itself says:

…sample size for a full-scale trial for two study groups with a significance level of 5% and 80% power, a 2:1 randomization and inflated for 10% lost-to-follow-up was calculated in 342 participants and a pilot trial would be at least 31 [19].

So they computed that they’d need 342 patients for a full-size trial that had 80% power to find significance. The reference they used suggests that a pilot trial can be useful in guiding the design of the larger trial with 9% of that sample, which is 31. In order to get some more data on viral load, they doubled the intervention group, ending up with the study size of 45. Statistical significance was never in the cards for this pilot trial. This is the actual result:

Viral load values under the limit of quantification of 10 copies/reaction at day-5 were achieved by 6 of 20 (30%) subjects in the IVM group and in 1 of 12 (8·3%) in the control group without statistical significance between groups.

So, 30% of the treatment group was negative on day 5, while only 8.3% of the control group was.

Let’s pause here for a second. Scott’s motto for Astral Codex Ten (his Substack) is:

“P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary“

If you’ve not heard it before, that expression is also known as Bayes’ theorem, named after Presbyterian minister Thomas Bayes. It represents an approach to probability that is far less rigid, more intuitive, and less prone to dichotomous yes/no answers. This is a topic of deep contention, so I’ll leave it at this.

I’ve previously hinted at this issue of over-reliance on statistical significance to shred useful data, but let’s take the time to break this down. For Bayesians, this kind of brutal application of statistical significance is an affront to all that is holy.

You see, if we analyze the same result in Bayesian terms—using the same parameters as those used by the TOGETHER trial—we would get back a 90.7% probability of superiority. In graphical terms, the likelihood of being PCR-negative on day 5 looks something like this:

gray: untreated control / red: high-dose ivermectin

Focusing on whether significance was achieved or not obscures the reality: if this were the only data we had access to, no sane person would think twice if given a choice which treatment path to follow (if they saw the above graph). To reach statistical significance, almost half the treatment group would have to be negative by day 5, which—given the fact that PCR tests can also pick up fragments of dead virus—is extremely unlikely. I’m pretty certain that no early treatment Scott endorses, be it Paxlovid, Molnupiravir, or Fluvoxamine, would pass such a test.

Scott continues:

They did not find any difference in clinical outcomes.

That’s an interesting turn of phrase, given that if we look for the phrase “clinical outcomes,” the paper itself says:

Limitations of this study include its sample size, which is based on demonstrating the antiviral activity of IVM against SARS-CoV-2 but lacks power to detect differences in clinical outcomes.

So, while what Scott says is technically correct, they also did not NOT find any difference in clinical outcomes, making his statement highly misleading.

Moving on:

A pro-ivermectin person could point out that in the subgroup with the highest ivermectin concentrations, the drug seemed to work.

Given the analysis so far—seeing actual statistical significance in the group that had higher absorption of ivermectin, even though it includes fewer patients—is kind of unexpected. To be clear about what they did, they took blood samples from the patients, and saw which patients had high concentration of ivermectin in their blood. The patients that had blood plasma concentration of over 160ng/mL did a LOT better (since five of nine were PCR negative by day 5), whereas of the remaining eleven patients with low concentration of ivermectin in their blood, only one was negative by day 5. Control patients, similarly, had only one in twelve negative on day 5.

If you’ve ever heard the debate about how “ivermectin should be taken with a fatty meal while study X told patients to take it with an empty stomach,” this blood concentration differential is the underlying concern. The FLCCC has recommended ivermectin be taken with a fatty meal since very early 2021. According to Merck’s own label:

Administration of 30 mg ivermectin following a high-fat meal resulted in an approximate 2.5-fold increase in bioavailability relative to administration of 30 mg ivermectin in the fasted state.

The paper itself also highlights another issue that could affect blood plasma concentration, namely the interaction of ivermectin with P-glycoprotein:

Diet is a key variable affecting oral bioavailability of IVM, with increased plasma concentrations achieved with fed state [27,10]. The interaction of IVM with ABC transporters as P-glycoprotein and the modulation of P-glycoprotein activity after oral administration is well known [28,29]. Thus, variable constitutive and/or induced level of expression and activity of intestinal P-glycoprotein in treated patients, may have contributed to the observed large variability in the pattern of IVM absorption and systemic exposure.

These factors could explain why some patients saw very different results than others, and can help tune dosing parameters in future trials. To imply that these results are somehow illegitimate is… illegitimate.

Let’s get back to Scott’s review of this trial:

A skeptic could point out that this is exactly the kind of subgroup slicing that you are not supposed to do without pre-registering it, which I don’t think this team did. I agree with the skeptic.

He “agrees with the skeptic”—presumably that the result is worthless—since it was not declared prospectively. Not knowing the context of the research program, Scott misses the point of the study. Here, I must recognize that I would probably have made the same mistake, had it not been highlighted to me by a friendly researcher during the course of my literature review.

In context, this isn't a study that aimed to prove ivermectin works with statistical significance. The low patient number combined with the endpoint is nowhere near sufficiently powered for that. As an in-vivo analysis of high-dose ivermectin safety and effect on viral loads—in preparation for a bigger study—it’s invaluable.

I don’t believe I’ve seen another study go this deep into ivermectin absorption in blood plasma in conjunction with viral load reduction. The patient-by-patient charts posted in a corrigendum to the original paper are actually mesmerizing:

Alexandros Marinos @alexandrosM

@AOlavarria To understand the study, it's important to look at the extra information provided in a corrigendum: thelancet.com/journals/eclin… The three groups have very VERY distinct viral load reduction curves, and we are also shown it on a patient-by-patient level.

While the 160ng/ml threshold was identified post-hoc—as the authors freely admit (this was a proof-of-concept study, after all)—the difference in viral clearance is hard to ignore and sets a standard for what kind of dosing we should be looking at, moving forward.

After the RCT

Dr. Krolewiecki followed up the RCT with an in-vitro study that corroborated the 160ng/mL threshold, as well as prior in-vitro results:

In the present study, we demonstrated that a concentration as low as 0.2 μM for 24h produced a similar effect on the inhibition of importin α nuclear to cytoplasmic distribution than a concentration of 2.5 μM for 1 h. This observation suggests that a sustained exposure to lower concentrations of IVM could indeed interfere with the host cell machinery that the virus requires for replication. Interestingly, results from our clinical trial using a high dose of IVM of 0.6 mg/kg/day for five consecutive days in hospitalized COVID-19 patients demonstrated a concentration-dependent effect on the viral load of respiratory secretions. The viral decay rate was significantly greater in patients with IVM plasma levels of 160 ng/mL or higher [7], a concentration close to the one we observed in vitro effects on importin α nuclear localization (0.2 μM that corresponds to a concentration of 175 ng/mL)

So what happened to the trial this trial was a pilot for? You may remember the article in Nature magazine—that we’ve quoted from before—about the wide use of ivermectin in Latin America. What you probably don’t remember is the person with the following quote is the author of this study:

The drug’s popularity “practically cancels” the possibility of carrying out phase III clinical trials, which require thousands of participants—some of whom would be part of a control group and therefore couldn’t receive the drug—to firmly establish safety and efficacy, says Krolewiecki.

In other words, he understood that widespread usage prevented him from proceeding with a large-scale RCT. What did Dr. Krolewiecki do instead? He collaborated in the Mayer observational study, which used the same dosing as this trial. And even though the Mayer study was run just as the deadly Gamma variant was on the ascent in Argentina, it showed statistically significant positive results, especially in high-risk patients. We’ve already covered that study, so I won’t say much more about it here.

While the Mayer study was ongoing, Krolewiecki also published a different study that added safety data for the high-dose ivermectin regimen. He didn’t stop there, either. He published a paper on a nasal spray that could deliver ivermectin straight to the lungs.

All in all, this specific RCT is part of a sequence of six studies on the question of ivermectin for SARS-CoV-2. Following this set of studies, I saw a methodical mind going through the steps of understanding—in detail—the interaction of ivermectin and the virus. Throughout the pandemic, Dr. Alejandro Krolewiecki has published six studies on this topic, each attacking a different angle:

• Retrospective

• Pilot RCT

• In-vitro

• Large observational

• Safety study

• Nasal spray

These studies, read together, make a very strong case that high-dose ivermectin is safe and effective in the treatment of SARS-CoV-2. The case may not be definitive, but drugs have certainly been authorized with much less evidence.

His latest publication happens to be the highest-resolution map of Strongyloides stercoralis prevalence to date.

Something tells me this will become relevant down the line.

---

This is a public peer review of Scott Alexander’s essay on ivermectin, of which this is the twelfth part. You can find an index containing all the articles in this series here.