TOGETHER Trial: Interim Analyses (Featuring: The Dubious Termination of the HCQ Arm)

I’ve been trying to figure out when the interim analyses for the TOGETHER trial happened and what they decided. I’ll tell you one thing: it is shockingly difficult to sort out. It is muddy enough that—even after this article is out, which documents everything as well as it can be documented—people will still argue about this topic.

From the trial materials and other sources, I am pretty confident that the interim analyses and other key events were as follows:

• First Epoch starts: June 2020

• Interim Analysis: October/December 2020

  • What went wrong: HCQ was apparently interrupted at 45% of recruitment, while demonstrating 70%+ probability of superiority. We don’t have access to the threshold, but interim analyses don’t tend to stop arms that are performing this well, as 70% is far too high to qualify common-sense and literature definitions of “futility”

• Second epoch starts: Jan 15-20

  • It appears that the single-dose ivermectin arm was terminated in early March 2021, without a recommendation by the DSMC

• First interim analysis: April 3rd

  • What went wrong: metformin was stopped, though all signs point to the fact that its probability of superiority was above the 20% p(superiority) threshold

• Second interim analysis: early May

• Third interim analysis: June 21st

• Fourth interim analysis: August 5th

  • What went wrong: It’s unclear why ivermectin was declared to be terminated for “futility” and later the published paper described it as terminated because it “reached the planned size.” Similarly, it’s unclear why fluvoxamine was allowed to extend to 110% of its planned size but was declared “terminated for superiority”

• Special interim analysis: September 12th

For more details on the above, let’s dig into some primary materials:

First Epoch: June-December 2020

The TOGETHER trial originally started June 2, 2020 trialling HCQ and Lopinavir/Ritonavir. The paper that presents the results says:

The planned sample size was 1476 patients, with interim analyses planned after 500 patients were enrolled. The trial was stopped after the interim analysis for futility with a sample size of 685 patients. Statistical analysis was performed in December 2020.

Interestingly, if we look at the protocol trying to figure out what the stopping rule was, we learn that:

Interim futility assessment
will be done using Cytel’s Bayesian predictive power method where a posterior
predictive probability of rejecting the null hypothesis at the final analysis will be
assessed given the interim data observed at analysis. Data demonstrating futility will lead to the discontinuation of treatment arms with the replacement of other
investigational arms.

OK, this is very similar to the second epoch of the trial in 2021, but what is the actual threshold? The protocol doesn’t say, but it points us to a Statistical Analysis Plan or perhaps an Interim Monitoring Plan that “will be developed.” Well, the year is 2022 and I can’t find those anywhere, so I’m going to assume they weren’t published.

Given that the trial was terminated for futility, I would expect the probability of superiority must have been quite low, so I tried to calculate it.

Turns out, the Bayesian probability of superiority for HCQ at the time the trial was stopped “for futility” was 70.43% (Placebo: 11 events in 227 patients. Treatment: 8 events in 214 patients). While we don’t know what the cutoff was, the next iteration of the TOGETHER trial used 20% for the first analysis (covering 25% of patients), 40% for the second (covering 50% of patients), and 60% for the third (covering 75% of patients). The closest analog is the second analysis, but regardless which one you go with, 70% is greater than any three of the thresholds. I find it hard to believe that after recruiting only 45% of its planned patients, a trial would be terminated with a probability of superiority over 70%. And yet, this seems to be exactly what happened here.

Second Epoch: 2021 and Beyond

According to the ivermectin paper in NEJM, this is what was in the cards regarding interim analyses:

Interim analyses were planned to occur after 25%, 50% and 75% of the maximum number of patient outcomes had been observed, as well as at the trial completion. The posterior efficacy threshold was set at 97.6% and the futility thresholds at 20%, 40% and 60%.

Late December 2020 to January 2021

The second epoch of the TOGETHER trial starts with the Brazil protocol v1.0 dated December 17th, which introduces the new drugs to be tested: fluvoxamine, metformin, low-dose ivermectin.

Fail #1: The Brazilian national ethics committee approval for the trial was granted on January 18th. And yet, according to the metformin paper in The Lancet—as well as the fluvoxamine preprint and previous presentations—the first patients were randomized on January 15th.

According to the Jan 16th update to the trial website, the first interim analysis was planned to take place when each arm had 100 patients randomized.

It’s unclear whether this interim analysis ever happened and was ignored, or if it was never done at all. There is certainly no mention of an interim analysis that happened in March, which would have been when the arms would have each reached 100 patients.

February-March 2021

In February, the TOGETHER team decided to change the ivermectin dosing, even though fewer than 20 patients had been randomized into that arm. This is unusual, and possibly a first time occurrence in a big-time platform trial. Such a serious protocol change within a few days of the start of recruitment of patients is at the very least surprising.

According to the Brazil protocol v2.0, dated February 15th:

[…] to obtain the best clinical results with the use of the drug within the safety observed in several clinical trials conducted using ivermectin for Malaria, Dengue and COVID-19, we are proposing to extend the treatment in this clinical trial to use for 03 days, instead of a single dose […]

This protocol was submitted to the Brazilian authorities on February 19th. Even though they had already declared their decision—subject to approval—to halt the low-dose ivermectin arm and replace it with a higher dose, they did not, in fact, stop recruiting patients into a dosing that, by their own words, would not have “the best clinical results.”

Indeed, as published in Pierre Kory’s Substack, Ed Mills, TOGETHER trial’s principal investigator, said on Mar 6, 2021 with regards to IVM dosing:

(email addresses redacted & emphasis mine)

From: Edward Mills
Sent: Saturday, March 6, 2021 4:48 PM
To: Steve Kirsch
Cc: filipe rafaeli Pierre Kory Patrick Collison
Subject: Re: IVM under dosing in Brazil trial

We are doing three days of dosing. The original protocol had one day of dosing, which was used back in January, when we submitted this for approvals at the ethics and national bodies. It then changed to three days of dosing after an amendment based on emerging trials from Andrew Hill’s synthesis. I have already explained that on several occasions.  Thats what is being administered. Clinicaltrials.gov  needs to be updated. Clinicaltrials.gov is not a protocol, it’s a registry. Gates funded the first part of this trial, when we evaluated HCQ and lopinavir vs placebo.

Thats why they are on the website. Again, this has been clearly explained in the past.

We will do an interim analysis after 800 patients. We aren’t making any changes until then.

Ed

In this email, Mills unambiguously says that at that time (March 6, 2021), TOGETHER was administering ivermectin with three days of dosing, which is the high-dose regimen. A few days later, the TOGETHER trial Twitter account Tweeted out:

Together Trial @TogetherTrial

@socialHATchat @Covid19Critical We have amended our protocol paused allocation to the ivermectin arm while the IRB reviews our amendment to include 3 doses.

This is most likely the true version of events. Low-dose ivermectin recruitment was paused in the first days of March, but high-dose ivermectin had not yet begun.

Digging into the list of files the TOGETHER team submitted to the Brazilian local ethics board (CONEP), I also discovered that they submitted a file with the name “15_TOGETHER_DSMC_Charter_26Mar2021.pdf.”

This is interesting, because the TOGETHER trial’s second epoch has a completely different composition for its Data and Safety Monitoring Committee than the first epoch, so it very well might be that—as implied by the filename—the committee’s charter was only put together by the 26th of March 2021. This would explain why there seems to be no DSMC involvement in stopping of the single-dose ivermectin arm.

April 2021

So, when did the first interim analysis happen? According to the Brazil protocol v3.0, dated April 20th:

As planned the Independent Data and Security Review Committee met recently and after blindly reviewing the data recommended:

(1) the suspension of the Metformin arm of the study due to futility and;

(2) the anticipation of the next interim analysis until after the inclusion of 1,200 patients and at least 14 days of follow-up of the last randomized patient for the evaluation of events and safety regarding the remaining 3 arms in the trial;;

(3) the addition of another arm of the study, at the discretion of the study's steering committee

That “recently” was most likely the 3rd of April, as chronicled in the trial’s publication on metformin:

This is the first indication we have of an interim analysis happening in the trial.

Also in approval letters of the Brazilian local ethics board (CONEP), we see a file named “10_TOGETHER_2_DSMC_Recommendation_Letter_Interim_Evaluation_1.pdf” with a submission timestamp of “25/04/2021 13:53:47,” which fits the picture we have that the first interim review happened some time before the 25th of April 2021.

However, since this was the first interim analysis, metformin most likely was not below the 20% threshold that would justify termination.

The protocol continues with a discussion of resolutions for the next, and second, interim analysis:

By recommendation of the Data Safety and Analysis Committee, the second interim analysis has been anticipated to be performed when there are 1,200 patients randomized and adequately followed up for at least 14 days (first primary clinical endpoint date), considering the event data obtained in the first interim analysis, when only the data from (1) fluvoxamine, (2) ivermectin, and (3) placebo arms will be evaluated.

May 2021

It’s not exactly clear when the second interim review happened, though in approval letters from the Brazilian ethics board (CONEP), we see another file which is named “16_TOGETHER_DSMC_recommendation_letter_interim_evaluation_2.pdf,” with a submission timestamp of “25/05/2021 18:16:50.” This fits the picture we have that the second interim review happened before the 25th of March 2021.

According to Brazil protocol v4.0, dated May 21st, the DSMC decided the following:

The research already has an independent data and safety review committee, which has already conducted two interim reviews

As of May 20, 2020, the research protocol has recorded the randomization of 1602 patients.

As planned, the Independent Data and Security Review Committee met recently and after blindly reviewing the data recommended:

(1) maintaining the progress of the study as planned, without changes; the conduct of an interim safety analysis when 120 patients are included in the investigational product subcutaneous administration arms. This analysis will be restricted to evaluating adverse reactions potentially associated with the investigational products used and adverse events.

(2) Evaluate possible outcomes in current arms of the study according to the statistical analysis plan;

(3) Initially hold the next interim analysis as planned (50% of patients randomized and with complete D14)

As we can see, these decision points are clearly different. It is also written—for the first time—that the DSMC has now conducted two interim reviews. In this one, they decide to make no change to the ongoing arms (as opposed to the previous one that nixed metformin), and conduct an interim safety analysis at 120 patients recruited of any future subcutaneous administration arms.

Intriguingly, it declares that the next interim analysis will happen when 14 days of followup have been completed for 50% of the planned number of patients. Given that the protocol says “…the new sample of patients for the entire study, taking into account since the beginning of the clinical trial, will be increased to 3,645 patients,” we can infer that the next interim review was planned for two weeks after 1823 patients had been recruited. Given that the same protocol records that 1602 patients had already been recruited as of May 20th, we can can expect for the additional recruitment to take a couple of weeks, and so the next interim would be about a month after the 20th of May (2 weeks to recruit plus an additional 2 weeks of followup for the last patient to be recruited).

June 2021

Just as expected, on June 21st, the 3rd interim review of the trial was executed. (I have seen evidence of this—that I find conclusive—but sadly am not at liberty to share).

What we do know from the public domain, is that that does seem to be the starting point for a cluster of events relating to the trial:

• On June 22, the new version of the Statistical Analysis Plan is released and signed (SAP 2.0)

• Also on June 22nd, we see a new version of the Master protocol, v3.0

• On June 24th, the peginterferon lambda arm begins

• That was followed by the initiation of the doxazosin arm two days later

• On July 5th, protocol version 4.0 is submitted to clinicaltrials.gov, almost two weeks after it came into effect

August-September 2021

On August 5th, the 4th interim analysis was conducted by the DSMC. As described in the ivermectin paper in NEJM:

As described in the fluvoxamine paper in The Lancet:

Note that both publications explicitly mention that this was the 4th analysis of those particular arms—and not the 4th analysis of the TOGETHER trial, in general. I note this because some people cite the original interim analysis—that ended up terminating the HCQ and lopinavir/ritonavir arms—as the first interim analysis. While this is technically correct, it is clear here that this is not one of the four analyses being counted, since it did not include the ivermectin or fluvoxamine arm.

Also, as we learned on the Brazil protocol v5.0, dated November 20th, 2021:

The research already has an independent data and safety review committee, which has already conducted four (4) interim reviews.

On August 5, 2021, the Independent Safety and Data Review Committee recommended:

(1) discontinuation of the Fluvoxamine arm due to superiority over the placebo group and;

(2) discontinuation of the ivermectin arm due to futility over the placebo arm.

[…]

On the occasion of the 4ª interim analysis that took place on August 5, 2021 and after blinded data analysis recommended:

(1) The Ivermectin arm was discontinued due to futility compared to the placebo group;

(2) The suspension of the fluvoxamine arm for superiority over the placebo group;

(3) The maintenance of the Lambda pegylated interferon arm without modification;

(4) The definitive suspension of the Lambda Pegylated Interferon arm considering that the sponsor has withdrawn the proposal to sponsor the drug, thus making it unfeasible to perform this arm;

(5) The conduct of a specific interim analysis to evaluate adverse events associated with the doxasozin arm, since there will be a projection of at least 125 patients included in the doxasozin arm (considering 1:1 randomization), to be scheduled by September 15.

In the same document, we also learn that the fate of the doxazosin arm was decided in the fifth interim analysis, conducted on September 12th:

On September 12, 2012, the 5th interim analysis was conducted specifically to assess adverse events regarding the doxasozin arm, and the trial's independent data and safety review committee recommended discontinuation of the 14-day treatment arms.

Somewhere here, our trail runs cold, as the next protocols are much less forthcoming about any interim reviews that may have happened. However, since we’ve covered the period in which the metformin, ivermectin, fluvoxamine, and doxazosin arms were running, it will have to do for now.

Summing It All Up

This was intended as more of a reference post, to help as a place to link to as support for the timing or outcome of a particular interim analysis. However, while writing it, it became clear that the various termination decisions have been anything but obvious.

In at least two cases, there is concern as to the decision to terminate (HCQ and metformin), and many of the other decisions seem to be ambiguous even to the TOGETHER team. For instance, it should be clear whether three-dose ivermectin was terminated for futility or because it reached its planned size; it does seem like the story changed over time. Similarly, it’s unclear why fluvoxamine would have been allowed to extend to more than its planned number of patients when ivermectin was not.