The Cochrane Review on Ivermectin Violated Its Own Inclusion Criteria for 76% of the Patients It Included
5 of the 11 included studies, covering 2582 of the 3409 patients included in the systematic review did not qualify for inclusion, based on a novel criterion used to exclude many other studies.
I’m having trouble believing this finding is real, but I’m also having trouble understanding how it might be substantially wrong. Let’s rewind all the way to the start of this story and take it step by step:
Cochrane on Ivermectin
Cochrane systematic reviews are considered by many to be the gold standard meta-analyses. So naturally, when the original Popp et al. systematic review was released by Cochrane on July 28, 2021 finding “very low- to low-certainty evidence” about the efficacy and safety of ivermectin for COVID-19, many people took that to be a decisive conclusion.
Of course, the authors only included 14 studies, meaning they excluded the majority of the evidence available. However, one can put that down to Cochrane being excessively picky about the studies they include. As such, many waited to see if the meta-analysis would be updated to include newer studies.
More than a year later, they released an updated version. They somehow managed to add new criteria so that 7 of the 14 studies they had included the previous year were no longer eligible. Even though new trials were added, they ended up with fewer studies in 2022 than in 2021.
It is generally not good practice for a systematic review to modify its protocol between revisions—since it gives the impression of p-hacking taking place—but in an abundance of good faith, we will assume the authors had their reasons for this alteration.
The new criterion—that did much of the heavy lifting in terms of exclusions—was the “trial registration” criterion, specifically the requirement that a trial be prospectively registered:
In their own words:
Is the study prospectively registered?
Check in the trials register the date of protocol submission and first posted. Prospective registration is defined as registration of a trial before enrollment of the first participant as defined by the WHO. It must be determined whether the registers registered (date first posted) without delay at this point in the pandemic. In case of doubt, check for the date first submitted or the authors must be asked for the submission date.
If study is not prospectively registered, exclude the study
They are quite clear that if the date of registration is not before the date of the enrollment of the first participant, the study should be excluded. In fact, on the basis of this criterion, they excluded several studies they had included in their 2021 edition of their systematic review. The excluded trials were: Abd-Elsalam, Biber, Chachar, Okumuş, and Shah Bukhari.
Up to this point, we might quibble about the choice of criteria, but within the criteria they chose, the decisions they made seem justified. The authors inform us that they ended up including 11 trials with a total of 3409 participants:
Main Results
We excluded seven of the 14 trials included in the previous review version; six were not prospectively registered and one was nonrandomized. This updated review includes 11 trials with 3409 participants investigating ivermectin plus standard of care compared to standard of care plus/minus placebo.
Was TOGETHER’s Inclusion Valid?
Looking at the study, I got a bit curious about the registration date of the TOGETHER trial, since I happen to be familiar with its details. Interestingly, the authors had already made an exception to include the TOGETHER trial, since its results were not published within the pre-determined period that the systematic review had committed to consider for its trial pool:
Here’s the problem, though: the TOGETHER trial was not actually prospectively registered, given that the first submission to clinicaltrials.gov was on January 25, 2021, and the study started on January 19th, or January 20th according to the published paper on ivermectin in the New England Journal of Medicine.
As such, the TOGETHER trial should have been excluded, if not for its publication being too late to be considered, then for its registration being retrospective. This puts into question no fewer than 40% of the patients considered in the systematic review:
And somehow, it gets worse.
Examining All 11 Studies
Seeing the inconsistency in the case of TOGETHER, I decided to look into the rest of the studies. The three largest studies (TOGETHER, Vallejos, I-TECH) all seem to have violated the prospective registration rule. In the case of I-TECH, this may be arguable since the registration was submitted the same day the first patient was enrolled. By their definition, however, that registration must be before the first patient is enrolled. I-TECH is also not a valid inclusion. The same is true of two more trials: Kirti, and Gonzalez.
This means that fully 2582 of the 3409 patients included in the systematic review did not qualify for inclusion in the systematic review.
See the details of all the included studies here:
Please let me know if you see any errors in the table above. In order to apply an abundance of caution, I have not relied on the Popp et al. date, which you can see in the “Cochrane Date” column, but I have instead reviewed the clinical trial registrations myself, and often found that the submission date was significantly earlier than the systematic review claims it to be. Still, the date of the first patient enrollment is earlier for almost half of the included studies.
We might want to debate the value of Popp et al. choosing to include this exceedingly harsh criterion as a filter for their meta-analysis, and I suspect most would agree it is erring on the side of excessive caution.
What we cannot debate, however is that the studies that enrolled the vast majority of the included patients did not qualify for inclusion based on this criterion.
If Popp et al. truly wanted to stick by their prospective registration criterion, they should have submitted a systematic review with only six studies, which would include only 827 patients in total. In fact, given that Lopez-Medina managed to disqualify 74 patients by mixing up placebo and treatment for two weeks, they would actually have to rely on the data for fewer than 753 patients, making their systematic review a pointless exercise.
Naturally, this would act as a reductio ad absurdum for the prospective registration criterion, but given that they made the choice to use that criterion, the authors have to live with the consequences of that decision. After all, it was a criterion they, themselves, prospectively declared—at least somewhat.
What Now?
Moving forward—and at the very least—the authors need to revoke this criterion and reinstate the studies they excluded if they want their systematic review to be seen as anything other than an exercise in evidence-shredding.
Unless I’ve made some glaring error, I simply can’t fathom how this systematic review was published.
I’ve seen a lot of facepalm-worthy moments in this pandemic, but this one has my jaw on the floor.
I'm more appalled than shocked to see such a flawed review coming out of the Cochrane Collaboration. I used to be a huge fan of CC and relied heavily, and rather unquestioningly, on their reviews. That all stopped in 2020, after I discovered just how poor their 2018 review of the evidence for the HPV vaccine was. I still remember reading and rereading the 2018 BMJ article by Peter Gotzsche, Lars Jorgensen, and Tom Jefferson laying out the many serious problems with the review, including that the HPV vax review had missed *nearly half of the eligible trials* and was influenced by reporting bias and biased trial designs. And, Cochrane had failed to deal adequately (e.g. they didn't follow their own rules) with the conflicts of interest on the part of the lead author, among others. If folks want to read this critical piece they can check here. https://ebm.bmj.com/content/23/5/165.abstract. You could also read pp 146-150 (Section titled: "Cochrane review of the HPV vaccines was flawed and incomplete") in Gotzsche's 2020 book "Vaccines".