TOGETHER Trial: Doin' Metformin Dirty

The TOGETHER trial as it restarted in 2021 had multiple simultaneous treatment arms: ivermectin, fluvoxamine, metformin. While we’ve discussed about the first two, we’ve not yet discussed the metformin. I have been keeping notes though, and yesterday it all came together.

The Story of the Metformin Trial

Metformin is less talked about because it was cut short. It started in mid-Janurary 2021, and as per the paper published in The Lancet:

On April 3, 2021, the Data and Safety Monitoring Committee recommended stopping enrollment into the metformin arm due to futility.

“Stopping for futility” is a term used in adaptive platform trials such as TOGETHER that means that in an interim analysis, one of the arms performed so badly, that it was determined that the chance of it ending up succesful was miniscule, and therefore the best course of action is to eliminate the particular arm and free up resources for others.

TOGETHER in early 2021 had 3 scheduled interim analyses. This is described well in the ivermectin paper:

Interim analyses were planned to occur after 25%, 50% and 75% of the maximum number of patient outcomes had been observed, as well as at the trial completion. The posterior efficacy threshold was set at 97.6% and the futility thresholds at 20%, 40% and 60%. If the intervention group showed a posterior probability of efficacy by crossing a boundary, it was to be stopped.

The idea is that as the interim analyses cover more and more patients, the minimum amount of efficacy, as expressed in the probability of superiority, that an arm has to demonstrate so that it can be allowed to continue increases, as there’s both more data to make safer conclusions, and fewer patients left to catch up with.

So far so good, what’s all this to do with the metformin trial? When metformin was stopped, its probability of superiority was 28.4%. In other words, it was performing worse than placebo.

When the trial was stopped, it had enrolled 215 patients in treatment and 203 in placebo, roughly 30.7% of its intended size.

This is where we’ve reached a contradiction: The trial was terminated, having 30.7% of the planned number of patients, and a probability of superiority of 28.4%.

If this was the first interim analysis, then the number of patients is about right, but the probability of superiority is too high to trigger stopping for futility. If this was the second interim analysis, then the probability of superiority was sufficiently low to terminate for futility, but the number of patients is far below 50% which the protocol specified.

However, if the second interim analysis was completed on April 3rd, barely 12 weeks into the trial and around 30% of planned patients randomized, when was the first? That makes no sense.

Thankfully in the 3.0 update of the Brazil Protocol, dated 20th April 2021, we get significantly more clarity:

By recommendation of the Data Safety and Analysis Committee, the second interim analysis has been anticipated to be performed when there are 1,200 patients randomized and adequately followed up for at least 14 days (first primary clinical endpoint date), considering the event data obtained in the first interim analysis, when only the data from (1) fluvoxamine, (2) ivermectin, and (3) placebo arms will be evaluated

So, the first interim analysis indeed happened, and yielded data that will be useful for the second one, in which only ivermectin, fluvoxamine, as well as placebo arms will be present.

The obvious explanation is that the metformin arm will be absent in the second interim analysis, since it was eliminated during the first.

We also learn something else: the Data and Safety Monitoring Committee (DSMC), the body tasked with reviewing the interim evaluations, requires at least 14 days of data for each patient. This means that if there was indeed a DSMC meeting on April 3rd, it would have looked at data accumulated at most by March 20th, 2021.

Indeed, this is starting to make a lot more sense. According to our handy cheat-sheet, The week starting March 15, 2021, is the week that the 25% recruitment threshold was crossed, making April 3rd the natural point for the pre-declared interim review by the DSMC.

I think we have our answer now. Metformin was terminated as a result of the first interim analysis, which included patients recruited until March 20th or so, and the DSMC met as soon as the last patient had 14 days of followup data gathered. This termination was most likely done in violation of the trial protocol’s own “termination for futility” rules, which dictated that if the probability of superiority was above 20% in the first interim analysis, the arm in question would continue being active.

The only situation in which the termination could have been done legitimately is one where the patients until March 20 showed substantially lower probability of superiority than the patients on April 3rd. This would need the last 27 patients to have only had one event, an extremely unlikely event, especially since the last two weeks of the trial are ones where epidemiological data from the area is showing a surge in covid infections and severity.

Wordsmiths of the world unite

Knowing this, notice how carefully the metformin paper avoids saying that metformin was dropped in the second interim analysis, while carefully guiding the user to assume that that’s the case:

Planned interim analyses were conducted. Stopping thresholds for futility were established if the posterior probability of superiority was less than 40% for ITT at the second interim analysis.

“Oh, by the way, did we tell you that the futility threshold for the second interim analysis is 40%? Why do we mention this? Oh, no reason, just saying…”.

And again:

A Data and Safety Monitoring Committee provided independent oversight for this trial. We planned a second interim analysis of metformin vs. placebo after 50% of data collected. Herein, we present follow-up of all patients allocated to these arms up to April 03, 2021.

I have no words. Masterclass in deceptive communication. Had they indeed terminated it in the first interim analysis for simply falling below the futility threshold, they would have been able to write that very very simply, rather than insinuating that the second interim analysis was the one in which the termination happened.

Did anything else go wrong in the Metformin trial?

Boy, did it. We’ve discussed before how the first patient was recruited on January 15th, even though the ethics approval from the Brazilian authorities was granted on January 18th.

Alexandros Marinos @alexandrosM

TOGETHER Trial : The Ethics Violations. 🧵 The first violation is perhaps somewhat small. They were recruiting and treating before approval for the trial by the local ethics board. Approval was granted on Jan 18, but the Metformin arm was recruiting since Jan 15.

OK, but that’s just a classic case of enrolling patients for medical experimentation without having cleared it with the locals, you know how it goes. Anything else?

How about this one: The trial reported results for the endpoint “emergency room observation for more than 6 hours or hospitalization”. However, the 6 hour interval is something that was introduced with a protocol alteration enacted on March 22, less than 2 weeks before the last patient was enrolled. For the vast majority of the duration of the trial, the observation time declared in the initial protocol was the one in force, which was twice as long: 12 hours. As such, the endpoint reported was effectively a retrospectively declared one.

Add this to the pile: The paper declares a randomization algorithm that matches version 3.0 of the Statistical Analysis Plan. Except the SAP 3.0 was dated June 22, months after the Metformin trial was terminated.

Got more? As luck would have it, yes. It appears they tuned the dosing to be way too aggressive:

Dr. Eric Oh So Good MD 🗨 @EdoajoEric

While I've defended TOGETHER IVM from what mostly amount to toothless pedantry and stretched presumptions, presented as glaring study flaws, I will say I think they did metformin dirty. No one in practice starts right off that dose - you'd expect adherence to be affected.

(I will sidestep the comment about how the other criticisms of the TOGETHER trial are “toothless pedantry” since Dr. Osgood has not bothered to provide specific criticisms of my arguments)

Let’s focus on his specific prediction: adherence will be affected. Was it? You bet. Placebo had 88% adherence, the treatment arm had 78% adherence. To put it differently 47 patients dropped off in the treatment group, and only 24 in the placebo group. And for you lovers of frequentist statistics, that’s a statistically significant difference (p<0.01).

What if we focus only on patients who completed the protocol appropriately, both in treatment and placebo? That’s called a per-protocol analysis, and the results are frankly shocking:

Metformin’s probability of superiority shoots up from 28.4% to 64.3%, a mere 35.9% difference between ITT and per-protocol. Do numbers even mean anything at this point?

I think we can call Osgood’s point proven beyond reasonable doubt.

His prediction based on clinical practice translated into real-world adherence dropoff, and a massive jump in efficacy if we focus only on patients who completed the treatment.

[Update: I thought about it a little more, read around, and I’ve now got some doubt]

There is another explanation for all this. The paper goes to extreme lengths to explain away the adherence issue by noting that the “per-protocol” placebo group actually contains a blend of 3-day and 10-day placebo patients, and therefore adherence may be artificially inflated by the fact that 3-day placebo patients are more likely to adhere to the treatment than 10-day placebo patients. Here’s the quote:

We observed a large difference between event rates in the ITT and PP populations. To explain this, we looked further into adherence by event category for both placebo and metformin. Looking at adherence rate by event category, there was roughly a 35% reduction in adherence for those who required hospitalization or retention in a COVID-19 emergency setting among the placebo recipients, while adherence dropped by more than 50% in the metformin recipients with the same outcome. As the drop in adherence among those who had events occurred in both the placebo recipients and metformin recipients, this suggests that it is unlikely that reduced adherence leads to event, but rather those with severe disease were less adherent.

In addition, metformin was compared to a common control for all arms, which also might explain the difference in adherence. The control used in the TOGETHER Trial was placebo dosed proportionate to the concurrent active treatment arms; for example, if the trial was randomizing to a 10-day intervention and a 3-day intervention, then half placebo patients would receive a 3-day placebo while the other half received a 10-day placebo. This means that adherence is more likely in a 3-day placebo vs. a 10-day placebo.

This may explain some of the difference, but the difference is large enough that both causes could have contributed to the outcome.

Naturally, someone has to ask why the per-protocol analysis was done this way here, while in the case of ivermectin, only 3-day placebo patients were used, but I digress. Have I already mentioned that the way this trial ran its placebo arm is both completely novel and completely indefensible? Well, the fact that it makes the metformin results unreadable is one more reason.

Did metformin dirty, indeed.

Why?

What I’m mostly interested in is figuring out why the trial was ended prematurely, in contravention of the trial’s protocol. It’s always hard to ascribe intent, and I try to avoid it as much as I can. So instead let’s talk about what the result of this irregular termination was, regardless of whether it was intentional or the result of some cosmic accident.

The ivermectin arm had been officially restarted with a much higher dose on March 23rd, which we now know was the first week of the second quarter of the trial. Due to irregularities in how the placebo arm was allocated, this led to massive catch-up enrollment for ivermectin during the following 3-4 weeks. Those weeks also just happened to be the most lethal weeks of the pandemic. By terminating the Metformin arm when they did, more patients were available to be assigned to the ivermectin arm, making both the disparity more severe, while ensuring that catch-up enrollment would be complete just in time for the second interim analysis, which would have to include patients recruited until some point in late April, 2021.

Was the effect of stopping metformin before its time big? Was it perhaps negligible? The only way to find out is for the investigators to release the data from the trial so that we don’t have to speculate. And if I’m wrong in this speculation, releasing the data will be an excellent way to prove it.