TOGETHER Trial: Doin' Metformin Dirty, Part 3

This article is part of a series on the TOGETHER trial. More articles from this series here.

---

Just when you think you could not find any more primary material, you find more primary material. In particular—for a reason I can’t quite recall—I started revisiting old versions of the TOGETHER trial website on the Wayback Machine. It turns out, there’s gold in them thar hills, all right. I found the preprint of the TOGETHER metformin publication, as well as a summary slide about metformin that had additional information from what was presented on August 6, 2021.

Instead of writing something long and complicated, I want to put this out there with my quick notes, and see if others notice anything else.

The Slide

So, here’s the slide: first in the detailed form, then in the form it was presented:

From Togethertrial.com Website on Aug 10th, 2021:

From the presentation:

Observations from the two slides:

• Population

  • 182 Men, 241 Women (this slide has raw numbers, not just percentages).

  • “…expected hospital stays of <=5 days” (Where is this coming from? What does it mean?)

• Settings/Locations

  • 10 Clinical sites (wrong—only 3-5 sites involved).

• Intervention

  • 423 Patients randomized, 372 analyzed—probably the difference between the total patients as of April 3rd and the patients with sufficient follow-up on April 3rd. This information was removed in the August presentation.

  • 217 Metformin patients—wait—I thought it was 215? Or 216? Or 219? Make up your mind?!

  • 206 placebo patients (we ended up publishing 203 patients, but it’s just placebo, right?)

• Findings (same in both versions)

  • 32/217 here became 34/215 in the published paper…

  • 27/206 here became 28/203 in the published paper…

  • It should be noted that with the numbers above, I find 31.7% probability of superiority, not 27.6% as indicated in the graph. To get closer to 27.6%, I have to use 34/217 and 28/206 respectively, which is the final number of events with the earlier number of patients. A very weird set of coincidences…

The Preprint

What about the Metformin paper preprint? According to the file metadata, it was last modified on July 22, 2021, a full two weeks before the other two arms—which used the same placebo patients—were terminated. It also used the same data as found in the slides, so all three can be considered part of the same work, as they are mutually consistent and produced during the same timeframe.

Things that stand out to me from the preprint:

• It contains adherence numbers not found anywhere else (and very different from the published results). “At the time the recruitment of the metformin arm was stopped, 31 (14.0%) participants in the metformin group did not complete all phases of the study. Twenty (9.7%) participants did not complete the study in the placebo group (figure 1).”

• We learn that the antigen test used was Panbio by Abbott. (evaluation)

• Very specific statement about the time the data snapshot was taken: “We planned an interim analysis of the metformin arm with the data cut from April 03, 2020.”

• “The probability that the event rates were lower in the metformin group compared to placebo was 27.4% for the ITT population and 23.9% for the PP population (figure 2). These posterior efficacy numbers were lower than the pre-specified 40% threshold set for the second interim analysis.”

  • Weirdly, the PP analysis seems to have generated a 23.9% probability of superiority in the preprint, though in the paper it actually ends up being 64.3%.

  • They specify that the comparison point was the 40% for the second interim analysis.

  • The equivalent sentence in the published paper is: “The probability that the event rate was lower in the metformin group compared to placebo was 28.4% for the ITT population and 64.4% for the PP population.”

• Clear statement of DSMC rationale: “The Data Monitoring and Safety Committee advised to stop enrolling patients to metformin because the posterior probability of efficacy was below the 40% futility threshold determined based on 200,000 simulation runs for three different relative risk reductions: 0%, 20%, and 37.5%.” This sentence is absent in the published paper.

• Website indicates that paper had started being written by June 18, 2021 or earlier.

• Doxazosin and IFN-Lambda are reported as “Protocol amendment approved, randomization shortly forthcoming.” This means the paper was written before June 24, 2021—when randomization started.

• File metadata indicates that the version uploaded was last modified on July 22, 2021.

• Wayback Machine indexed the file on July 25, 2021 for the first time.

The Conclusions

This preprint confirms a lot of what we thought we knew about the metformin trial. Even though only 423 patients were randomized (30.6% of the total 1362 patients planned), the termination for futility came as a result of failing to cross the 40% threshold—which had been intended as the threshold for the second interim analysis—once 50% of recruitment was completed. This was implied in the published paper, but it’s spelled out in the preprint.

While it remains theoretically possible that the interim analysis indicated that metformin was performing below the 20% threshold required to fail the first interim analysis, it is both highly unlikely for that to have happened, and very different from what is described in the preprint. The text of the preprint adds substantial evidence to my previously published hypothesis that the trial was stopped early and improperly.

We also see a troubling trend of adding and removing patients from consideration under somewhat unclear criteria. This preprint claims 217 patients were randomized to metformin, whereas the published paper claims 215 (or in the appendix, 216). Which is it? This number is not supposed to be flexible. If a patient took metformin, they should be accounted for.

We can also tell that important data from the trial was seen by staff while the trial was ongoing—even being published on the internet. For instance, this sentence is relevant to the other trials that were concurrent and still recruiting:

The rate of our primary endpoint occurring appears to have increased importantly from the beginning of the trial to the end of the trial. This is likely explained by the emergence of the predominant Gamma (P.1) variant during the conduct of this trial that may exhibit greater transmission and worse clinical outcomes than earlier variants.

However, this being a new piece of primary material, I may have missed more interesting tidbits. Feel free to dive in and let me know what you find!

---

This article is part of a series on the TOGETHER trial. More articles from this series here.