I read the article in the NYT with interest and shared it with my sister (MD) and daughter (grad student in public health), and we were very puzzled. The conclusions and headlines didn't make sense or feel right. I shared your write-up, and it answered a lot of our questions.
I really appreciate the work you have done here and wish the NYT would have been more careful before giving the author such a public platform.
Bravo and thank you!
And this is one more example of the truly excellent work you do.
Thomas, thank you for this thorough analysis. The meta-analysis by Jefferson et al. was indeed bad science. They themselves stated that "The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions during the studies hampers drawing firm conclusions." If there's any lesson to be taken from this mess of a paper, it is that poorly enforced mandates with low compliance in which people use masks of variable quality, many worn improperly (below the nose!), and many used too long to remain effective, had a less beneficial effect in reducing transmission than hoped. It seems that Jefferson has an anti-masking agenda and used this poorly designed meta analysis to support his agenda. Letting one's personal agenda bias one's science is sure to result in bad science and weak conclusions. As you point out, the Bangladesh study and other, smaller well-designed studies show a clear benefit to mask use, both for reducing transmission and protecting the wearer. This is why surgical teams have worn masks in operating rooms for the past 100 years.
“If there's any lesson to be taken from this mess of a paper, it is that poorly enforced mandates with low compliance in which people use masks of variable quality, many worn improperly (below the nose!), and many used too long to remain effective, had a less beneficial effect in reducing transmission than hoped.“
I'd like to comment on the extremely high quality of your work. Two of the giants of American medicine, Bob Wachter and Eric Topol, both referred to your article in their response to the mask debate. That is high praise indeed my friend.
A completely neutral approach to information interpretation does not exist. That's really the point of Kuhn's (and Kahneman's and Tversky's among others) work. Every human being has multiple filters, often ones of which they are not aware.
In that context, a decision to investigate the source of a set of information is not bias. It is research, and it is done for a very specific reason:
The issue is not whether crazy folks can have brilliant ideas. It's about whether there is a pre-existing agenda which is driving a particular author's information filter. That absolutely needs to be taken into account and is valid.
Finally, "ad hominem" implies an attack on an individual. Understanding the biases which an author brings to the table is not an attack.
Whenever I am sent a citation that seems to wildly contradict what I know, my first reaction (after calming down the demons of confirmation bias) is to ask - What's the source?
If I find out it's from Epoch Times, I don't even bother to read it.
The second question I ask is: Who are the authors? I.e., are they bringing their own biases to the table and if so, how well known are those biases?
I haven't heard anyone discuss the authors....
How well known are they in epidemiology/statistics?
What are their biases?
What is their agenda?
I suspect there's gold in them thar hills.........
Sorry, one more thing. Compliance and seal really is everything with masks. A surgeon is indeed usually protected. But look up what a surgeon does with his or her masking. And look up the research on what lowers the efficacy of a surgeon’s seal. Then compare that surgical procedure to what we did as a population. Most people masking weren’t actually masking during COVID. ZZ Top with a mask over his beard when he walked to his dinner reservation was signaling, not protecting. Sorry. And that had nothing to do with whether the pilgrims really cared about masking or not. We are lucky COVID wasn’t lethal for the vast majority. But if it were, I promise we’d have all been more fastidious and serious. Isaac Newton lived in the countryside during the Black Death, for example.
Woow! amazing work. You took a scientific paper and did a in-depth analysis to find strong biases on the paper.
Its exactly what a Cochrane review should avoid. From their website: "[Cochrane Database of Systematic Reviews] aimed at minimizing bias, to produce more reliable findings to inform decision-making"
The fact that is a Cochrane Review means they followed protocols, data has been scrutinized and several other scientist review it (PhDs with years of experience in the field). Nobody spotted these facts?
What does it say about the way we are doing science?
The work you did requires deep understanding and hell lot of time. I am used to judging every piece of information I read, but I always considered scientific papers the closest to "ground truth". Not any more. Which is a shame as I (and I guess most people) don't have the knowledge and time to do analyze scientific work the way you did.
I don't think this means that science is bad either, or that Cochrane is poor, or that the integrity of the authors is bad. I think I'm saying this is a very hard topic, very nuanced, where it's hard to make bombastic claims, and the devil is in the details. This is true for science in general.
Does it mean it's not valid? No. It just means we still know so little. We need to be open to learning all the time.
Thank you for all the hard work of analyzing these various studies! Great job! As a Sociologist however, I take these kinds of statistical studies with a big grain of salt. This kind of statistical analysis of human "action"--and the act of putting on a mask is VERY contextual---ignore all sorts of "micro actions".....the "who, what, where, when, and why" of the putting on, keeping on, adjusting, etc. of mask wearing. As you allude to, these studies are not really about mask WEARING but about the validity of various statistical studies. I would offer that a statistic study, which assumes linear causality, ie. "wearing a mask will reduce the likely hood of a respiratory illness" is simply a poor hypothesis. Frankly, the best studies about the efficacy of masks----and not the "wearing of masks" are the studies that show how aerosols are prevented from moving by being blocked by something---and the "something" that blocks the transfer of "human juices" is a K-95 mask. I for one, wore a mask to simply add a layer between my respiratory system and the stuff that people exude when they talk, sneeze, cough, laugh, sing etc. What is really needed is a global, rigorous framework to study what goes into the "act" of wearing some kind of aerosol preventive mechanism.....
In the future, what do you think public health officials should do in cases like this where it’s seems that an intervention will only be effective if it’s done perfectly (which is impossible). Do you advise, and then look like you advise things that don’t work when it’s inevitable shown to not work (appears to be happening now with masking and lockdowns), or do you double and down and harangue the public and say, “If you guys actually did this (masking/lockdown/dieting) right, it would work. It’s not working cause you’re not doing it right”?
- In highly controlled environments, mandates can work. Eg, hospitals.
- In extreme situations, there's immense value in learning. So mandates can be valuable
For other situations, it's a bit harder. There's costs and benefits to wearing masks, costs are low, and benefits depend.
Interestingly, the point of the mandate is not to protect you, it's to protect others. You infect others from projecting aerosols, and for that a surgical mask in front of your face might arguably catch most viruses from the nose and mouth that you eject? So as long as you do wear it on top of them, most of the value is accrued?
Then this value needs to be measured against the loss in freedom and of buying the mask itself. How do you calculate that?
You might end up with special mandates, like having masks in the healthcare system and in multitudinary indoors concerts, and that's it.
This is speculation, but if I were a CDC leader, this is how I'd be thinking about it. I guess this is what they're doing now.
I think that the communication around "masking" needs to be far better developed...as i said above...it's not "the mask" it's the ACT of masking. I think current communication focusses on the "mask" and NOT on the responsibility and accountability of PEOPLE to engage in a form of action that can save lives. We've been using the wrong logic to try and make this into a form of public health disease prevention. As of now the "responsibility" to prevent illness has been placed on the "object" and NOT on people. It's a shift that needs to happen.
There is another way to approach the issue of whether masks (as opposed to mask mandates) work. In countries like Japan mask-wearing is culturally supported, having been employed against particulate pollution for decades. I believe the data on covid-related deaths post lockdowns in Japan supports the view that fewer deaths occur when there is general mask-wearing by the population. Perhaps you could look into this?
I seldom really learn from a blog or podcast, but in the case of Tomas Pueyo every encounter with his thinking has been memorable. Thanks Thomas.
It's an honor to hear this. Thanks Robert!
I read the article in the NYT with interest and shared it with my sister (MD) and daughter (grad student in public health), and we were very puzzled. The conclusions and headlines didn't make sense or feel right. I shared your write-up, and it answered a lot of our questions.
I really appreciate the work you have done here and wish the NYT would have been more careful before giving the author such a public platform.
Bravo and thank you!
And this is one more example of the truly excellent work you do.
Thank you Peter, I appreciate!
Thomas, thank you for this thorough analysis. The meta-analysis by Jefferson et al. was indeed bad science. They themselves stated that "The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions during the studies hampers drawing firm conclusions." If there's any lesson to be taken from this mess of a paper, it is that poorly enforced mandates with low compliance in which people use masks of variable quality, many worn improperly (below the nose!), and many used too long to remain effective, had a less beneficial effect in reducing transmission than hoped. It seems that Jefferson has an anti-masking agenda and used this poorly designed meta analysis to support his agenda. Letting one's personal agenda bias one's science is sure to result in bad science and weak conclusions. As you point out, the Bangladesh study and other, smaller well-designed studies show a clear benefit to mask use, both for reducing transmission and protecting the wearer. This is why surgical teams have worn masks in operating rooms for the past 100 years.
“If there's any lesson to be taken from this mess of a paper, it is that poorly enforced mandates with low compliance in which people use masks of variable quality, many worn improperly (below the nose!), and many used too long to remain effective, had a less beneficial effect in reducing transmission than hoped.“
Spot on
I'd like to comment on the extremely high quality of your work. Two of the giants of American medicine, Bob Wachter and Eric Topol, both referred to your article in their response to the mask debate. That is high praise indeed my friend.
Thank you Doug! I'm glad to hear!
A completely neutral approach to information interpretation does not exist. That's really the point of Kuhn's (and Kahneman's and Tversky's among others) work. Every human being has multiple filters, often ones of which they are not aware.
In that context, a decision to investigate the source of a set of information is not bias. It is research, and it is done for a very specific reason:
The issue is not whether crazy folks can have brilliant ideas. It's about whether there is a pre-existing agenda which is driving a particular author's information filter. That absolutely needs to be taken into account and is valid.
Finally, "ad hominem" implies an attack on an individual. Understanding the biases which an author brings to the table is not an attack.
That’s very fair, Edward. Thank you. I will update my thinking based on this.
Whenever I am sent a citation that seems to wildly contradict what I know, my first reaction (after calming down the demons of confirmation bias) is to ask - What's the source?
If I find out it's from Epoch Times, I don't even bother to read it.
The second question I ask is: Who are the authors? I.e., are they bringing their own biases to the table and if so, how well known are those biases?
I haven't heard anyone discuss the authors....
How well known are they in epidemiology/statistics?
What are their biases?
What is their agenda?
I suspect there's gold in them thar hills.........
There is, and your bias is valid.
But this is an ad hominem attack, and I believe it muddies an objective analysis.
In other words, crazy ppl can come up with great ideas, and vice-versa. Once I have decided to break an idea down, I try to forget who is the author.
Thanks, Tomas and team, for your investigation.
to your second point in opening remarks, 'what can we learn about knowledge from this debate?', this truly is a cautionary tale!
And I'll write more about it in the premium article!
Notice that, unfortunately, right now it's not Tomas and team for the investigation, but just "Tomas". I want to change that.
You performed great surgery to peel away the lies and deceit.
Great article. Thanks for digging into it and making it easier to understand.
Sorry, one more thing. Compliance and seal really is everything with masks. A surgeon is indeed usually protected. But look up what a surgeon does with his or her masking. And look up the research on what lowers the efficacy of a surgeon’s seal. Then compare that surgical procedure to what we did as a population. Most people masking weren’t actually masking during COVID. ZZ Top with a mask over his beard when he walked to his dinner reservation was signaling, not protecting. Sorry. And that had nothing to do with whether the pilgrims really cared about masking or not. We are lucky COVID wasn’t lethal for the vast majority. But if it were, I promise we’d have all been more fastidious and serious. Isaac Newton lived in the countryside during the Black Death, for example.
Woow! amazing work. You took a scientific paper and did a in-depth analysis to find strong biases on the paper.
Its exactly what a Cochrane review should avoid. From their website: "[Cochrane Database of Systematic Reviews] aimed at minimizing bias, to produce more reliable findings to inform decision-making"
The fact that is a Cochrane Review means they followed protocols, data has been scrutinized and several other scientist review it (PhDs with years of experience in the field). Nobody spotted these facts?
What does it say about the way we are doing science?
The work you did requires deep understanding and hell lot of time. I am used to judging every piece of information I read, but I always considered scientific papers the closest to "ground truth". Not any more. Which is a shame as I (and I guess most people) don't have the knowledge and time to do analyze scientific work the way you did.
Many thanks!
I don't think this means that science is bad either, or that Cochrane is poor, or that the integrity of the authors is bad. I think I'm saying this is a very hard topic, very nuanced, where it's hard to make bombastic claims, and the devil is in the details. This is true for science in general.
Does it mean it's not valid? No. It just means we still know so little. We need to be open to learning all the time.
Thank you for all the hard work of analyzing these various studies! Great job! As a Sociologist however, I take these kinds of statistical studies with a big grain of salt. This kind of statistical analysis of human "action"--and the act of putting on a mask is VERY contextual---ignore all sorts of "micro actions".....the "who, what, where, when, and why" of the putting on, keeping on, adjusting, etc. of mask wearing. As you allude to, these studies are not really about mask WEARING but about the validity of various statistical studies. I would offer that a statistic study, which assumes linear causality, ie. "wearing a mask will reduce the likely hood of a respiratory illness" is simply a poor hypothesis. Frankly, the best studies about the efficacy of masks----and not the "wearing of masks" are the studies that show how aerosols are prevented from moving by being blocked by something---and the "something" that blocks the transfer of "human juices" is a K-95 mask. I for one, wore a mask to simply add a layer between my respiratory system and the stuff that people exude when they talk, sneeze, cough, laugh, sing etc. What is really needed is a global, rigorous framework to study what goes into the "act" of wearing some kind of aerosol preventive mechanism.....
Spot on! Thank you Jerri, I agree. Well said.
In the future, what do you think public health officials should do in cases like this where it’s seems that an intervention will only be effective if it’s done perfectly (which is impossible). Do you advise, and then look like you advise things that don’t work when it’s inevitable shown to not work (appears to be happening now with masking and lockdowns), or do you double and down and harangue the public and say, “If you guys actually did this (masking/lockdown/dieting) right, it would work. It’s not working cause you’re not doing it right”?
Very good question. Let me think out loud.
First, we need to segment the situations:
- In highly controlled environments, mandates can work. Eg, hospitals.
- In extreme situations, there's immense value in learning. So mandates can be valuable
For other situations, it's a bit harder. There's costs and benefits to wearing masks, costs are low, and benefits depend.
Interestingly, the point of the mandate is not to protect you, it's to protect others. You infect others from projecting aerosols, and for that a surgical mask in front of your face might arguably catch most viruses from the nose and mouth that you eject? So as long as you do wear it on top of them, most of the value is accrued?
Then this value needs to be measured against the loss in freedom and of buying the mask itself. How do you calculate that?
You might end up with special mandates, like having masks in the healthcare system and in multitudinary indoors concerts, and that's it.
This is speculation, but if I were a CDC leader, this is how I'd be thinking about it. I guess this is what they're doing now.
I think that the communication around "masking" needs to be far better developed...as i said above...it's not "the mask" it's the ACT of masking. I think current communication focusses on the "mask" and NOT on the responsibility and accountability of PEOPLE to engage in a form of action that can save lives. We've been using the wrong logic to try and make this into a form of public health disease prevention. As of now the "responsibility" to prevent illness has been placed on the "object" and NOT on people. It's a shift that needs to happen.
Thank you, Tomas. It’s articles like this that keep me sane. Great work!
loved it!
There is another way to approach the issue of whether masks (as opposed to mask mandates) work. In countries like Japan mask-wearing is culturally supported, having been employed against particulate pollution for decades. I believe the data on covid-related deaths post lockdowns in Japan supports the view that fewer deaths occur when there is general mask-wearing by the population. Perhaps you could look into this?
My prior was that masks work, and I have plenty of evidence to support it. What you say is a good example.
The reason why I looked into this study is because it could potentially add information to change my beliefs.
It didn’t.
So I’m not looking forward to spending many more hours confirming what I already believe!
Great analysis as always! This read is a thorough reminder to read past the headlines and check the sources.
I have two personal questions for you:
1. Until this paper came out, what was your personal mask wearing strategy?
2. After writing this post, has your mask wearing strategy changed and if so, how?
Hasn’t changed
I wear masks in indoor crowds.
Mass transit
Movie cinemas
Conventions
Basically wherever there’s a lot of people indoors and I don’t need to talk much