Alex it looks like this study was indeed poorly conducted - however can you please address the criticism of the Itajai study where it was revealed 94% of participants didn't continue to take the treatment after the third dose. I have yet to see a explanation of this and if it's true it may render the entire Itajai study as worthless as the TOGETHER trial.
I haven't dug into the itajai study and so I don't really base any conclusions on it.
I'll tweet this out in case anyone has any comments but in general pursuing TOGETHER and the Scott Alexander thread have taken a lot more time from me than I have available. Doing things at an appropriate level of detail, including correcting pro-ivm folks who are saying silly things about the TOGETHER trial, is incredibly time consuming.
Hey Alex, thanks for following up on that, I saw the reply from Flavio Cadegiani so I will take that on board. I think his answer boils down to, the effect on the treatment arm was so strong that if there was even better adherence, then you would have seen even better results.
(1) Was the exclusion criterion for vaccination changed mid-trial, as it appears to have been? If so, can the authors speak to why this change was made, the numbers of vaccinated people throughout the trial (it appears this information was collected?), and the effects this might have had on the results?
(2) What were the results for the 1-dose IVM arm? This would be a great mini data set to include in the supporting information, because presumably this data is otherwise going to remain unpublished.
Questions are good, avoid inferences about their motivations and absolutely avoid accusations.
Consider adding: “How many patients received less than 400mcg/kg of Ivermectin?
If these patients are excluded from the analysis does it change the results of the study? If so, how?”
Are you planning to send those questions to the authors or the NEJM?
The authors almost certainly won’t respond to you, even if you get it co-signed by some heavy hitters such as the editor(s) of the BMJ.
The authors will respond to the NEJM however, although if the NEJM take ownership of the questions then they’re not obliged to pass on the results. They can simply say “We’ve received a reply from the authors and are happy with the responses”.
So you’re efforts may be entirely futile, be prepared for that.
However, you are doing great work and may be on the verge of something extraordinary.
But even if you achieve nothing you have been honest, and that is the ultimate victory
> Are you planning to send those questions to the authors or the NEJM?
The authors are well aware of my work, but yes, I want to work towards a letter to the journal. Putting things here is an early review of the material and you maybe are able to see how it is evolving.
Thanks for your suggestions, I'll think on them a bit.
I am learning so much that it's all worthwhile even if nothing happens.
Couple of tips before you send anything to the NEJM.
Despite the fact that obviously you’re highly intelligent and well researched, the NEJM will take it more seriously if it is co-signed by others with academic appointments they can’t ignore. It’s a crappy system, your opinion should count as much as anyone’s but they respond to professors of universities much more than internet researchers.
Secondly, keep the points concise and ensure the wording is utterly unambiguous. They’ll latch on to any ambiguity and return that ambiguity in spades. You’ll probably not get an opportunity to reply to their responses nor enter into a dialogue so be prepared for that.
As it stands, there are a lot of points and questions that you have listed and maybe that can be truncated a little more concisely.
Perhaps focus on the absolute fundamental points and questions that you feel need addressing and that which perhaps can invalidate the paper.
Personally I think clarifying the dose that each and every patient received is a firm, unambiguous statistical parameter that can modify the results. That’s a good starting point.
I’ve dealt with difficult people in one of my previous roles and the process can be incredibly frustrating. At times I felt like I could hold up a piece of white paper, ask them what colour it was and they would say “black”.
An alternative strategy is to simply focus on what you are truly trying to achieve.
Ivermectin being endorsed by every major medical body in the world?
Ivermectin being freely allowed to be prescribed without necessarily being endorsed?
Andrew’s Hill/Owen and Ed Mills being held to account and/or disciplinary action taken against them?
Or something else?
Good luck, give it your best shot and hopefully you achieve everything you want to achieve.
My objective is to get the data released or at least audited by an independent third party that can confirm or deny my suspicions. From there, all sorts of roads open up, from me being proven wrong, to some of the things you mentioned in your message.
Feel free to message me on Twitter if you want to discuss this further, you've clearly got insights and I'd love to continue the conversation. My DMs are open.
My understanding is that administration of ivermectin was on and empty stomach, which is would cause lesser bioavailability for a fat soluble active ingredient. If one were treating stomach or intestinal parasites, this would be the correct method of administration, however for antiviral, this approach will further compound the underdosing. I have not had a chance to reconfirm this as I’m travelling.
Many of these points are addressed in the article. For example point 4 with the placebos the paper clearly states the different groups "corresponded with that of a comparable active-treatment group in the trial". So there is not "no theoretical justification" as you say, but a very pragmatic and clearly stated justification. I believe you did not fully understand the paper.
If we change the endpoint to focus just on mortality, Treatment arm was 21 deaths (3.1% of N=679) and Placebo arm was 24 deaths (3.5% of N=679). The national death rate in Brazil between March-August 2021 was between 2.69-2.88% (source: worldometer, case-positive death rate)
If we follow the theory that both the treatment and control groups had use of ivermectin, then it explains why the results between the two groups were similar, but it doesn't explain why the death rate of these groups was worse than the Brazil national average during that period.
Surely we would expect that if the treatment works, you would still get some kind of favourable outcome compared to the national average, right?
The caveats that I think might change this statistic would be
- removing the BMI>30 deaths from both groups to get a more fair comparison. I don't have the data to do this.
- consider that inclusion to the trial was dependant on the patient having at least one high-risk criterion. Which would make the expected fatality rate higher than the national avg.
Let me know if you have any thoughts on the above. Thank you.
The logical problem is that on the one hand we have the Itajai study,
if we accept the conclusion from Dr Flavio which is;
"Even though the adherance to the protocol was poor (most stopped taking after 4th dose), the treatment was SO STRONG that we observed an obvious effect on mortality anyway."
Whereas in the TOGETHER trial, we are saying that the effect of the drug was watered down and lost because of low dosing and recruitment anomalies and other issues.
It feels to me like if the effect was that strong, as Dr Flavio suggests, we would have seen BOTH treatment + control groups in the TOGETHER trial having really good outcomes compared to the national average.
This is not to dismiss any of the issues you identified with the TOGETHER trial, but just trying to make sense of over-arching issues that could explain the results on mortality (very strong from Itajai, very weak from TOGETHER) from both trials.
I have not looked at itajai at all, but I agree with you. We need an explanation for the whole evidence set.
Now, it may well be that since the itajai study was so much larger in number of patients, a smaller percentage wise effect was more "statistically significant", but as I said, I haven't looked at it at all, so this is just speculation.
If you want to look at Itajai, Prof Greg Tucker Kellogg and Dr Susan Oliver both do negatively biased critical breakdowns of the Itajai study on Youtube (They are colleagues of GMK so take their criticism with a lot of salt). Dr Flavio himself and Dr Lucy Kerr hosted presentations on FLCCC weekly update videos.
Great investigative work overall. And good summary. I think you can probably scratch off the question of “why did the authors adopt this novel endpoint?“ Unless I’m misunderstanding the nature of your question, they do touch on the reason for the composite endpoint in the paper
and further elaborate on the change to a 6 hour threshold in the Supplemental Appendix, specifically the Section titled “Description of Prolonged ER Visits and Modification of Primary Endpoint” on page 8. I think your other critiques about that topic in your linked article are still valid though.
Were they ever declared as separate endpoints in their protocol documentation submitted to authorities? Or are you basing that solely on the distinct “Primary Outcome Measures” on the clinicaltrials.gov registration? I’ve seen many studies registered on CT with several “Primary Outcome Measures”, despite a single “primary endpoint,” e.g. https://clinicaltrials.gov/ct2/show/NCT04368728
It’s unclear to me why CT doesn’t seem to have a way of establishing which is/are the primary endpoint. Or maybe it does and I’m unaware.
If they had a single primary outcome they would have had one bullet point. In clinicaltrials.gov as well as their published protocols, they always have two distinct objectives.
I did. If your position is that they are unconvincing, then I suggest refining your question to be more specific. Just a suggestion, free to be taken or left
Did this abomination really just win an annual best study award?
You betcha.
It just depends on your point of view. They did an excellent job of obscuring the methods and getting the "right" results. A+
Thank you! very helpful and very shareable with others!
Alex it looks like this study was indeed poorly conducted - however can you please address the criticism of the Itajai study where it was revealed 94% of participants didn't continue to take the treatment after the third dose. I have yet to see a explanation of this and if it's true it may render the entire Itajai study as worthless as the TOGETHER trial.
I haven't dug into the itajai study and so I don't really base any conclusions on it.
I'll tweet this out in case anyone has any comments but in general pursuing TOGETHER and the Scott Alexander thread have taken a lot more time from me than I have available. Doing things at an appropriate level of detail, including correcting pro-ivm folks who are saying silly things about the TOGETHER trial, is incredibly time consuming.
Hey Alex, thanks for following up on that, I saw the reply from Flavio Cadegiani so I will take that on board. I think his answer boils down to, the effect on the treatment arm was so strong that if there was even better adherence, then you would have seen even better results.
A couple more questions off the top of my head:
(1) Was the exclusion criterion for vaccination changed mid-trial, as it appears to have been? If so, can the authors speak to why this change was made, the numbers of vaccinated people throughout the trial (it appears this information was collected?), and the effects this might have had on the results?
(2) What were the results for the 1-dose IVM arm? This would be a great mini data set to include in the supporting information, because presumably this data is otherwise going to remain unpublished.
on (2), my educated guess is that the low-dose ivm arm did about as bad as placebo.
TOGETHER obviously did not get their act - or fraud - together.
Questions are good, avoid inferences about their motivations and absolutely avoid accusations.
Consider adding: “How many patients received less than 400mcg/kg of Ivermectin?
If these patients are excluded from the analysis does it change the results of the study? If so, how?”
Are you planning to send those questions to the authors or the NEJM?
The authors almost certainly won’t respond to you, even if you get it co-signed by some heavy hitters such as the editor(s) of the BMJ.
The authors will respond to the NEJM however, although if the NEJM take ownership of the questions then they’re not obliged to pass on the results. They can simply say “We’ve received a reply from the authors and are happy with the responses”.
So you’re efforts may be entirely futile, be prepared for that.
However, you are doing great work and may be on the verge of something extraordinary.
But even if you achieve nothing you have been honest, and that is the ultimate victory
> Are you planning to send those questions to the authors or the NEJM?
The authors are well aware of my work, but yes, I want to work towards a letter to the journal. Putting things here is an early review of the material and you maybe are able to see how it is evolving.
Thanks for your suggestions, I'll think on them a bit.
I am learning so much that it's all worthwhile even if nothing happens.
Couple of tips before you send anything to the NEJM.
Despite the fact that obviously you’re highly intelligent and well researched, the NEJM will take it more seriously if it is co-signed by others with academic appointments they can’t ignore. It’s a crappy system, your opinion should count as much as anyone’s but they respond to professors of universities much more than internet researchers.
Secondly, keep the points concise and ensure the wording is utterly unambiguous. They’ll latch on to any ambiguity and return that ambiguity in spades. You’ll probably not get an opportunity to reply to their responses nor enter into a dialogue so be prepared for that.
As it stands, there are a lot of points and questions that you have listed and maybe that can be truncated a little more concisely.
Perhaps focus on the absolute fundamental points and questions that you feel need addressing and that which perhaps can invalidate the paper.
Personally I think clarifying the dose that each and every patient received is a firm, unambiguous statistical parameter that can modify the results. That’s a good starting point.
I’ve dealt with difficult people in one of my previous roles and the process can be incredibly frustrating. At times I felt like I could hold up a piece of white paper, ask them what colour it was and they would say “black”.
An alternative strategy is to simply focus on what you are truly trying to achieve.
Ivermectin being endorsed by every major medical body in the world?
Ivermectin being freely allowed to be prescribed without necessarily being endorsed?
Andrew’s Hill/Owen and Ed Mills being held to account and/or disciplinary action taken against them?
Or something else?
Good luck, give it your best shot and hopefully you achieve everything you want to achieve.
My objective is to get the data released or at least audited by an independent third party that can confirm or deny my suspicions. From there, all sorts of roads open up, from me being proven wrong, to some of the things you mentioned in your message.
Feel free to message me on Twitter if you want to discuss this further, you've clearly got insights and I'd love to continue the conversation. My DMs are open.
My understanding is that administration of ivermectin was on and empty stomach, which is would cause lesser bioavailability for a fat soluble active ingredient. If one were treating stomach or intestinal parasites, this would be the correct method of administration, however for antiviral, this approach will further compound the underdosing. I have not had a chance to reconfirm this as I’m travelling.
Great work. Keep it up.
You are correct and this is a good point.
Why no question about confounding due to allowing vaccinated participants?
Many of these points are addressed in the article. For example point 4 with the placebos the paper clearly states the different groups "corresponded with that of a comparable active-treatment group in the trial". So there is not "no theoretical justification" as you say, but a very pragmatic and clearly stated justification. I believe you did not fully understand the paper.
I explain the problem with the placebo in more detail here. It's not that simple.
https://doyourownresearch.substack.com/p/together-trial-variable-placebo-an
If you have objections about other points please do let me know.
If we change the endpoint to focus just on mortality, Treatment arm was 21 deaths (3.1% of N=679) and Placebo arm was 24 deaths (3.5% of N=679). The national death rate in Brazil between March-August 2021 was between 2.69-2.88% (source: worldometer, case-positive death rate)
If we follow the theory that both the treatment and control groups had use of ivermectin, then it explains why the results between the two groups were similar, but it doesn't explain why the death rate of these groups was worse than the Brazil national average during that period.
Surely we would expect that if the treatment works, you would still get some kind of favourable outcome compared to the national average, right?
The caveats that I think might change this statistic would be
- removing the BMI>30 deaths from both groups to get a more fair comparison. I don't have the data to do this.
- consider that inclusion to the trial was dependant on the patient having at least one high-risk criterion. Which would make the expected fatality rate higher than the national avg.
Let me know if you have any thoughts on the above. Thank you.
These were high risk patients
The logical problem is that on the one hand we have the Itajai study,
if we accept the conclusion from Dr Flavio which is;
"Even though the adherance to the protocol was poor (most stopped taking after 4th dose), the treatment was SO STRONG that we observed an obvious effect on mortality anyway."
Whereas in the TOGETHER trial, we are saying that the effect of the drug was watered down and lost because of low dosing and recruitment anomalies and other issues.
It feels to me like if the effect was that strong, as Dr Flavio suggests, we would have seen BOTH treatment + control groups in the TOGETHER trial having really good outcomes compared to the national average.
This is not to dismiss any of the issues you identified with the TOGETHER trial, but just trying to make sense of over-arching issues that could explain the results on mortality (very strong from Itajai, very weak from TOGETHER) from both trials.
I have not looked at itajai at all, but I agree with you. We need an explanation for the whole evidence set.
Now, it may well be that since the itajai study was so much larger in number of patients, a smaller percentage wise effect was more "statistically significant", but as I said, I haven't looked at it at all, so this is just speculation.
If you want to look at Itajai, Prof Greg Tucker Kellogg and Dr Susan Oliver both do negatively biased critical breakdowns of the Itajai study on Youtube (They are colleagues of GMK so take their criticism with a lot of salt). Dr Flavio himself and Dr Lucy Kerr hosted presentations on FLCCC weekly update videos.
skip over the first Tucker-Kellogg video as it's basically an ad-hominem attack on Dr Flavio
Great investigative work overall. And good summary. I think you can probably scratch off the question of “why did the authors adopt this novel endpoint?“ Unless I’m misunderstanding the nature of your question, they do touch on the reason for the composite endpoint in the paper
and further elaborate on the change to a 6 hour threshold in the Supplemental Appendix, specifically the Section titled “Description of Prolonged ER Visits and Modification of Primary Endpoint” on page 8. I think your other critiques about that topic in your linked article are still valid though.
They say some words, but the words are unconvincing. See the linked post.
Also I don't believe they address combining the endpoints.
Were they ever declared as separate endpoints in their protocol documentation submitted to authorities? Or are you basing that solely on the distinct “Primary Outcome Measures” on the clinicaltrials.gov registration? I’ve seen many studies registered on CT with several “Primary Outcome Measures”, despite a single “primary endpoint,” e.g. https://clinicaltrials.gov/ct2/show/NCT04368728
It’s unclear to me why CT doesn’t seem to have a way of establishing which is/are the primary endpoint. Or maybe it does and I’m unaware.
If they had a single primary outcome they would have had one bullet point. In clinicaltrials.gov as well as their published protocols, they always have two distinct objectives.
I did. If your position is that they are unconvincing, then I suggest refining your question to be more specific. Just a suggestion, free to be taken or left