Deconstructing Evidence-Based Medicine... From the Inside Out
Deconstructing Evidence-Based Medicine... From the Inside Out
This piece is a little bit different than my usual fare. It started life as a Twitter thread. I am cross-posting it here because I hope it can serve as a reading list of material for people who have run into the same mess I did a year and a half ago (so they don't also have to spend 18 months reconstructing it).
At first, I was shocked by the kinds of things I have been finding. The kinds of things I have been writing about on this substack. However high an opinion of myself I might have, I realize most of what I have been looking into is not fundamentally intractable. How could anyone else not have seen it?
Well, I was wrong. People have definitely seen it. And have spoken out. In peer-reviewed journals. Nobody (in power) seems to care.
Let's start with this excellent paper by Leonid Hanin:
It’s an incredible piece that is extremely readable, and uncovers the whole mess. Here’s a small part of the conclusion:
"Oh come on, Alex, this is some prof from Idaho with a 23 h-index. Is this your refutation?"
I've not even gotten started yet.
How about Dr. Peter Gøtzsche, over 180 THOUSAND citations, h-index 103 (this means over 100 papers with over 100 citations EACH). Is he credentialed enough? Oh, yeah, he was also a founder and board member of the Cochrane Collaboration.
And here he is explaining how prescription drugs are the third leading cause of death:
Naturally, he was kicked out of Cochrane when he started speaking out.
"Ok fine. You found the village idiot. We keep him around because he has a funny accent." Interesting that you should say that, because I have another dude with a funny accent for you: 456 thousand citations, by some accounts, the most cited living scientist.
Here’s a quote from, perhaps, his most famous paper:
Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.
Now, keep in mind, Ioannidis and Gøtzsche are two of the authors of the PRISMA statement, which is the consensus standard for how meta-analyses and systematic reviews should be done. They are EBM royalty. Naturally, all this had been foretold. All the way back in 1998, Charlton and Miles laid it all out for us. A "science" that is named after the lie that all prior medicine was "not-evidence-based" is a discipline that lives by lies. Here’s a small part:
The author wrote a retrospective 11 years later (13 years ago), basically explaining to us what would happen during the pandemic, when even the basic guardrails were thrown out because "EMERGENCY!!!". The paper is unpaywalled on his blog. Here’s the abstract:
More about this has been written recently, even in the British Medical Journal, such as the much-discussed “The illusion of evidence based medicine.”
The important part to take away is that the findings of evidence-based medicine are a lot weaker than they are made out to be, and that can be shown in mathematically robust ways:
And let’s not even get started about how models are somehow even worse than EBM. Read the following Tweet with the knowledge that people like David Fisman are trying to insert modeling studies as valid medical evidence by using the imaginative name of… EBM+.
It Was Never About Medicine—or Evidence—For That Matter
I am personally partial to this excellent piece that focuses on nutrition “science.” That’s because it puts its finger on the pulse of the issue. It is not a medical—or for that matter, scientific—problem at all. W. Edwards Deming, the father of lean manufacturing, explained the problem of metrics-driven managerial top-down authoritarianism years ago. Watch this video to the end, and don't let his deadpan humor fool you:
Much more material can be found in the book The Tyranny Of Metrics that I will never stop recommending.
This is the way we got to this:
It's not about the evidence. It never was. Incentives rule everything around us.
I actually consider this an optimistic analysis. Whenever I've seen evidence of rot that is this clear, this public, the narrative eventually shifts.
And the evidence against evidence-based medicine is as conclusive as it is damning.
Maybe it will take a decade. Or maybe it will take longer. But it will shift. The question is "to what?" I’ve been tracking narrative shifts for a while now. After the first few, you start to get a feel:
I know this article can be used as an all-purpose rejection of medicine, and this isn’t really my intention. Perhaps it should be read as a warning. If the powers-that-be don’t find a way to turn the ship around, people will start opting out of medicine altogether. Every bad decision, every unfair act of coercion, every time a doubt is shouted down, contributes to this outcome. And when it does occur, they will all, collectively, own it. Never let them blame it on the people who rang the alarm early.
And never forget John Lykoudis, and what his story teaches us about the world:
I’m mid career but worked out a couple of decades ago how crazy the evidence based medical system is.
As a doctor, on the one hand you’re genuinely trying to do the right thing by the patients, on the other hand you’re trying not to get sued. Unfortunately they are not the same thing although of course they should be.
There is a crazy industry now of accreditation, evidenced based medicine and conferences. It feeds off itself and generates billions of dollars annually around the world.
One way of working out best practice is simply a straw poll of experts. I went to conference once where the speaker put up a series of clinical scenarios with options a,b,c,d how to treat. The audience chose via their phone which option they’d choose. Most scenarios had responses of about 70-80% for one option and an even distribution of the other 3.
Although a couple had 90+% one answer and other scenarios had a 4 way even spread.
Everyone practiced in the field and had read the evidence, thus simply asking a group of clinicians what they actually do was heading towards the best answer.
It’s nowhere near perfect but is the best way to summarise opinion. The rest of the time you’re doing what you think’s right based around your exposure to evidence (including your own practice) and available resources.
And if you practice long enough then you’ll likely see a total reversal of practice for a certain condition throughout your career.
The bounds of your practice are limited by “trusted third parties” who define what’s accepted practice, based around their own interpretation of evidence. All good and we’ll until you get a bit older and those people on those regulatory bodies are your peers...then you start to think what does that dufus know?
As an example, I personally know the head of ATAGI here in Australia. He’s not a dufus but he obviously can’t see much wrong with the evidence for the vaccines, contrary to what many commentators claim. I’ve resisted calling him but he’s just like me, nothing particularly special but now he’s making massive decisions
Hanin (2017) is profoundly naïve when writing, "1. Clinical trials should be publicly funded and conducted by biomedical researchers, medical doctors and statisticians with no relation to industry and no conflicts of interest."
Publicly funded research may also have significant relation to industry and conflicts of interest (most often, political, but others as well). Arguably, it will tend to be much worse than industry (the extreme case being Lysenkoism).
Kealey has written plenty about how public funding of science is uncorrelated with economic growth, implying that there are deeper issues than the funding sources.
My unscientific guess is that failures of science are just a symptom of the failures of the overall system. So even harder to fix. But Lykoudis is a great example of how truth can win out _despite_ the corruption of the system.