The Potemkin Argument, Part IV: Scott's Synthetic Scorn

This article is part of a public peer review of Scott Alexander’s essay on ivermectin, of which this is Part IV. Follow the links for parts 0, I, I+, II, and III.

In many cases, Scott downgrades papers because he objects to the methodology used—not to the implementation of the methodology, but to the entire class of papers. I find such blanket rules for throwing away evidence quite disconcerting for a blog with a Bayes-endorsing tagline like “P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary,” but here we are.

Today we’ll focus on his approach to synthetic control groups.

Borody et al.

Borody et al: Our last paper!

…is it a paper? I can’t find it published anywhere. It mostly seems to be on news sites. Doesn’t look peer-reviewed.

Given that Scott had no issue including, and praising, the TOGETHER trial when—at the time of writing his article—it was just a single slide and a few minutes of video, it seems flatly inconsistent to leverage a critique here that would apply there, also.

For the record, I am in favor of including data no matter the source. But I am even more in favor of equal standards.

Scott continues:

And it starts with “Note that views expressed in this opinion article are the writer’s personal views.”

Let’s have a look at the page Scott is talking about:

Is it not obvious that the highlighted text is a preamble, obviously appended not by the authors, but by the website in which their preprint is published? Had Scott attached the complete sentence, it would have been clear that the words he clipped could not have been written by the authors of the paper itself:

Note that views expressed in this opinion article are the writer’s personal views and not necessarily those of TrialSite

It’s frustrating to see Scott go so far as to quote a sentence fragment out of context to score a dunk, but here we are. “Look at these complete amateurs, who don’t even know how to publish a paper, lolz” is the general vibe, I suppose.

Let’s keep going:

Whatever. 600 Australians were treated with ivermectin, doxycycline, and zinc. The article compares this to an “equivalent control group” made of “contemporary infected subjects in Australia obtained from published Covid Tracking Data”; this is not how you control group, @#!% you.

“@#!% you.”

Please don’t take this as exaggeration: I don’t think I’ll find eight characters in Scott’s whole essay that sit with me worse than these. Once you understand the context, I think you’ll see what I mean.

Throughout this series, I’ve noted the many cases where researchers are derided, smeared, mocked, caricatured. Here, we see two authors being straight up abused for a cheap laugh.

Here’s the problem: both authors of this paper are highly respected doctors who have each contributed substantially to the world’s health in the course of their careers.

Starting with the second author, Robert Clancy is a Professor Emeritus of Immunology who developed a vaccine for acute bronchitis, among other breakthroughs. He was previously Foundation Chair of Pathology at the University of Newcastle. Before that, he was the first clinical immunologist at the Royal Prince Alfred Hospital, Sydney. He is also a Member of the Order of Australia, among other honors. You know, your standard anti-vaxxer yokel who can’t write a paper.

The first author—Thomas Borody—is responsible for the first truly successful treatment for H. pylori—with an eradication rate greater than 90%—that became the standard treatment for peptic ulcers, back in the 1980s. By third party estimates, that one discovery has saved over a million lives.

For context, Borody perfected the work of John Lykoudis, a Greek physician who treated 10,000 patients in Athens, Greece from the 1960s onward. This very short paper on the life of Lykoudis is a strongly recommended read for all who want to understand just how slow and painful it can be to change a paradigm. Does this quote remind you of anything?

A crucial obstacle, however, seemed to be that Lykoudis was using already approved medications and, at least according to Bayer (Mar 10, 1962; archives of John Lykoudis), his preparation would require sublicensing from the other companies before marketing. We also suspect that, for similar reasons, they thought it easy for competitors to circumvent Lykoudis’ intellectual property protection.

Lykoudis died in disrepute in 1980, a few years before his hypothesis was completed and validated by Barry Marshall and Robin Warren, who were eventually awarded a Nobel Prize in Medicine for their work.

The way he tells the story—in developing his peptic ulcer treatment—Borody realized a pattern of the medical establishment blaming patients’ lifestyle for digestion-related diseases that were actually caused by chronic infections. So he started looking for other instances. That led him to become a pioneer in the Fecal Microbiota Transplantation (FMT) field, which is pretty much what you imagine: using other people’s poop to help your microbiome fight off infections. Borody was decades ahead of his time, with FMT only recently being accepted as treatment for C. diff infections—after successful RCTs were performed around 2013—25 years after Borody started treating patients. His clinic has now treated more than 12,000. Borody is currently pioneering the use of FMT for several other diseases including Irritable Bowel Syndrome, chronic ulcerative colitis, and Crohn’s disease.

I’d also be remiss not to mention that he’s one of the authors of the best titled medical book of all time:

Here’s a short video piece on him by the Australian Broadcasting Corporation:

I hope by now you are starting to get a sense of the immense disrespect involved in Scott Alexander responding to work by Thomas Borody and Robert Clancy with “@#!% you.” Unfortunately, he’s not alone in ignoring decades of contributions when it comes to dissent from the COVID-19 narrative.

Both Borody and Clancy—like hundreds of other scientists—had their Wikipedia pages peppered with delegitimizing content, since they chose to buck the orthodoxy when it came to COVID-19 and early treatment. Borody, in particular, beyond the usual “COVID misinformation,” is accused of not declaring that he had patent applications pending in relation to the ivermectin-based treatment he promoted in Australia. This may sound bad, until you learn that the precise accusation is that he didn’t disclose the patent applications he would file in December 2020, in papers he wrote in August and November 2020—even though they recognize that in papers he wrote after the applications were filed, he did, in fact, disclose them. For more detail, I’ve done a Twitter thread diving into the absurd Guardian Australia piece that serves as the reference for Wikipedia’s allegations.

Synthetic Control 101

It gets worse, though. Let’s remind ourselves of Scott’s actual contention with the paper:

The article compares this to an “equivalent control group” made of “contemporary infected subjects in Australia obtained from published Covid Tracking Data”; this is not how you control group […]

If we take Scott at his word, he thinks that the authors pretty much made a control group out of thin air, using some improvised methodology. His claim is quite definitive, after all: “this is not how you control group.” The issue, unfortunately, is that he’s wrong.

Not only are synthetic control arms an established approach to clinical trials, they avoid several major problems with the RCT paradigm:

• Patients are less reticent to join a trial if they are certain to receive the treatment, making it easier and faster to set up trials.

• Trials are cheaper to run if a large subset of the patients don’t need to be allocated to placebo.

• If the treatment being tested is effective—which the researchers must believe to some extent before starting an RCT—accepting that a large portion of your patients must be left at risk of injury or death is quite a large ethical quandary.

Thomas Borody has actually explicitly cited the fact that he doesn’t believe running RCTs on COVID-19 patients to be ethical.

Perhaps synthetic control groups can’t replace RCTs in all cases. Maybe Prof. Borody’s objections to running RCTs on COVID-19 patients are misplaced. And perhaps we shouldn’t take Borody’s track record as a reason to give more weight to his hypothesis.

What I am pretty sure of, however, is that the appropriate response to reading a study by two researchers with a history of bucking convention, being right, and saving a multitude of lives, cannot possibly be “@#!% you.”

Let’s remember that Scott’s sarcasm was also directed at Flavio Cadegiani for using a synthetic control group:

There was no control group, but the author helpfully designated some random patients in his area as a sort-of-control, and then synthetically generated a second control group based on “a precise estimative based on a thorough and structured review of articles indexed in PubMed and MEDLINE and statements by official government agencies and specific medical societies”.

I’m pretty sure that “helpfully designated some random patients” and “sort-of-control” are not actual quotes from Cadegiani’s paper. I can imagine that the Scott Alexander contemporary of John Lykoudis’ time would probably have written the same kind of missive.

Scott finishes his review of the Borody et al. paper thus:

Then it gets excited about the fact that most patients had better symptoms at the end of the ten-day study period than the beginning (untreated COVID resolves in about ten days). Why are these people wasting my time with this? Let’s move on.

I don’t think I have the words to describe how disturbed I feel about the way Scott approaches this paper. All I can say is that he should—at the very least—have been more curious about the techniques, if not the people, that he is criticizing.

So, What Happened to the Study?

Reader Anon in the comments informed me of a critical piece of context: the study was interrupted by a ban on prescription of ivermectin for COVID-19 by the local regulator. It’s actually described in the paper itself:

The number of patients in this report was capped at 600, as the Australian regulatory body (the TGA) intervened in the middle of Australia’s third Covid wave, to prohibit further prescription of ivermectin for Covid-19 by General Practitioners.

The fact that the preprint was released less than a month after the treatment period for the last patients ended, may in fact speak to some of the shortcomings of the paper, assuming there was a rush to get the results “out there” in the hopes of getting regulators to reconsider.

For the record, the results posted were indeed spectacular:

What would have happened had the study been allowed to extend to thousands of patients, dwarfing any other dataset available? We’ll never know, and the Australian government is directly responsible for that.

Conclusion

At this point, I’m starting to wonder if there’s a legitimate cultural divide here. Maybe it’s something about having grown up in Greece, but it is literally unthinkable for me to use this kind of posture when talking about elders with long careers in a profession such as medicine. Similarly to how I felt when I read Kyle Sheldrick’s baseless accusations against Paul Marik, I find the kind of disrespect Scott shows for people like Borody, Schwartz, Babalola, Cadegiani, Elalfy, and many others simply unthinkable. And as far as I can tell, Scott is not just derisive: he’s derisive and wrong.

What’s more, Scott seems to be aware of the kind of social shunning he’s contributing to:

Here this question is especially tough, because, uh, if you say anything in favor of ivermectin you will be cast out of civilization and thrown into the circle of social hell reserved for Klan members and 1/6 insurrectionists. All the health officials in the world will shout “horse dewormer!” at you and compare you to Josef Mengele. But good doctors aren’t supposed to care about such things. Your only goal is to save your patient. Nothing else matters.

People don’t have to agree with these dissident doctors, but surely a baseline of respect is to be expected. A modicum of pretense, if necessary. I’m having a very hard time imagining myself throwing around the kind of phraseology on display here, even if I was 100% certain I was correct in whatever dispute. After all, this is being pursued in public. If scientific debates are being settled with reputational assassination, career-ending accusations, and just garden variety disrespect, can we truly trust that there haven’t been chilling effects that would bias the seeming consensus?

If you ever wondered what keeps me going in writing this series, you now probably have an idea. Stay tuned for the next installments.