My first thought on hearing about the theory it was just publication bias was that when it came to Covid all rules were off. I mean many would have us ignore the huge spike in VAERS because a vaccine campaign like the Covid one was unprecedented. Likewise, what probably is a nice guide for normal times doesn't necessarily apply in the 'war time' that is the pandemic. I still would give the concern merit over in the early going but over time, once it became clear there was stigma to suggesting miracle cures unless you were Fauci talking about the game changer Remdesivir, that to tout a generic Drug would conflate you with Trump, all assumptions about publication bias should probably be tabled.
Great work. You should also be looking at dosis. My recollection is that Vallejos (which has the highest weight in the meta analysis) is badly underdosed. This is exactly the kind of thing that would create funnel plot assymetry: one underdosed large study pulling the central axis of the funnel to the right and lots of smaller correctly dosed studies whose estimated effect sizes float around the true effect size of the correct dose, which is to the left of the central axis.
So apparently I was wrong about Vallejos being particularly underdosed. I haven't checked everything you did, but on a first look you seem to have made a mistake with Okumus. The dose was 200mcg/kg/day for 5 days, so 1000mcg/kg, it should be on the "high" group, not "low".
By the way, Avi Bitterman made a mistake in his forest plot and funnel plot. Both plots were done using the fixed effects model, when he should have used the random effects model. I don't know how to post pictures here, so I'll send you by e-mail. It doesn't make that big of a difference.
Thanks; this was a topic I hoped someone would dive into a little deeper. My instinct when someone throws around 10 scattered points and starts drawing all kinds of serious conclusions has always been 'really'? But ive never taken the time for a deep dive on funnel plots, so it is insightful to see some of the thinking around such matters laid out in some detail.
My first thought on hearing about the theory it was just publication bias was that when it came to Covid all rules were off. I mean many would have us ignore the huge spike in VAERS because a vaccine campaign like the Covid one was unprecedented. Likewise, what probably is a nice guide for normal times doesn't necessarily apply in the 'war time' that is the pandemic. I still would give the concern merit over in the early going but over time, once it became clear there was stigma to suggesting miracle cures unless you were Fauci talking about the game changer Remdesivir, that to tout a generic Drug would conflate you with Trump, all assumptions about publication bias should probably be tabled.
Good point. After the FDA horse tweet, we should probably assume inverse publication bias until proven otherwise.
Great work. You should also be looking at dosis. My recollection is that Vallejos (which has the highest weight in the meta analysis) is badly underdosed. This is exactly the kind of thing that would create funnel plot assymetry: one underdosed large study pulling the central axis of the funnel to the right and lots of smaller correctly dosed studies whose estimated effect sizes float around the true effect size of the correct dose, which is to the left of the central axis.
ok fine, you nerdsniped me. Here you go: https://twitter.com/alexandrosM/status/1573323696680632321?s=20
Summary, not much.
So apparently I was wrong about Vallejos being particularly underdosed. I haven't checked everything you did, but on a first look you seem to have made a mistake with Okumus. The dose was 200mcg/kg/day for 5 days, so 1000mcg/kg, it should be on the "high" group, not "low".
Oh, excellent catch.
By the way, Avi Bitterman made a mistake in his forest plot and funnel plot. Both plots were done using the fixed effects model, when he should have used the random effects model. I don't know how to post pictures here, so I'll send you by e-mail. It doesn't make that big of a difference.
Thanks; this was a topic I hoped someone would dive into a little deeper. My instinct when someone throws around 10 scattered points and starts drawing all kinds of serious conclusions has always been 'really'? But ive never taken the time for a deep dive on funnel plots, so it is insightful to see some of the thinking around such matters laid out in some detail.