Vaccine Effectiveness
"Sure, I'm aware of the stats. 94, 95 percent effective...I hear that all the time. Kind of reassuring."
My friend's comment illustrates two things about vaccine effectiveness stats: They're plentiful, and they seem encouraging. In theory, we should know a lot about how much vaccines protect us. What could possibly go wrong?
In this newsletter I'll discuss five reasons statistics on vaccine effectiveness can be uninformative or misunderstood: Ignorance, lies, mistakes, partial translations, and life. In the section on mistakes, I'll discuss what the stats mean in everyday terms.
1. Ignorance.
Some people don't understand effectiveness stats – even if they want to – because they've been struggling to grasp percentages since 4th grade. This is not a complaint or a criticism, but simply an acknowledgment that math is difficult for some people.
At the same time, some people ignore the stats because they don't trust the source. This is a diverse group, ranging from intelligent conservatives who mistrust liberal media, to batshit crazy conspiracy theorists who claim that vaccines contain microchips used by governments to track their citizens (details here). Folks who buy into the microchip theory think that vaccine effectiveness stats are either inflated (to persuade you to get your tracking chip) or false (because COVID-19 doesn't exist).
2. Lies.
Unfortunately, some people with large audiences publicly lie about the stats, and not all of their audience members are vaccinated against deception.
(It’s difficult to tactfully accuse someone of lying, so I'm not even going to try..).
Fox News host Laura Ingraham is a living, breathing textbook of how to be deceptive. You could probably spend a week classifying the many techniques she uses. Here are two that I find especially aggravating, owing to how effectively they can sabotage efforts to connect the general public with science:
(a) Note that expert views on some topic have just changed. Conclude that the experts are confused, or lying. Say this even when experts have changed their views in response to new data.
(b) Simplify findings. Speak in extremes, using terms such "always" or "never", "does" or "doesn't", "all" or "nothing". Exaggerate the implications.
You can find examples of (a) on virtually any "Ingraham Angle" piece related to the pandemic. As for (b), here's an example from last week:
"Masking vaccinated people makes literally zero scientific sense, unless you're conceding that the vaccine doesn't work."
Recommending that vaccinated people wear masks doesn't mean you believe vaccines don't work. It just means you believe vaccines don't work perfectly, and that they may work less well over time or against variants. Unlike Ms. Ingraham, you're thinking probabilistically rather than in all-or-nothing terms. And, you're concerned about a lot of people. Although less than 1 percent of vaccinated Americans have contracted COVID-19 so far, that's still about 125,000 people so far (see here and here). Thus, it makes more than "literally zero" scientific sense to address the spread of COVID-19 through a combination of methods, including vaccines, masks, distancing, etc.
(Evidently, you can say anything you like if your audience allows it. For this reason, I think it's time to reveal that Laura Ingraham is an axe murderer, an international arms dealer, and a shameless abuser of gerbils. You read it here first. Feel free to share these revelations on social media.)
3. Mistakes.
Unfortunately, sources with large audiences sometimes unintentionally misrepresent the stats.
By far the most common mistake during the pandemic has been treatment of effectiveness stats as simple percentages. You can see this when someone assumes (incorrectly) that deeming a vaccine 94% effective means that only 6 out of 100 people will get infected following vaccination, or that any one person only has a 6% chance.
Here's a prominent example of this mistake: In May, a CNN article noted that because currently available COVID-19 vaccines are only roughly 90% effective, "for every million fully vaccinated people who fly, some 100,000 could still become infected." (CNN has since revised the article.)
As commentators pointed out, 90% effectiveness doesn't mean that 10% of vaccinated people are likely to get COVID (or that 90% won't). This figure simply indicates that by getting vaccinated, you've reduced your risk of being infected by 90%.
So, what does that mean exactly? How is it different from what CNN was saying?
Imagine that 1,000 vaccinated people and 1,000 unvaccinated people fly out of a particular airport one day, and all of them are exposed to COVID-19 in the airport. (And, assume this is their only exposure.) Two weeks later, you give all of these people COVID tests.
Scenario A: 100 people in the unvaccinated group test positive for COVID. 10 people in the vaccinated group test positive.
In this scenario, the infection rate for the unvaccinated group is 100 out of 1,000, or 0.1. For the vaccinated group, the infection rate is 10 out of 1,000 or .01. If you subtract those numbers (0.1 minus .01), you get .09. But we're not finished yet. That .09 is an absolute difference in rates of infection. What we want is a relative value. In other words, we want to know how much protection the vaccine gives us relative to not being vaccinated. And so, we divide that .09 by the rate of infection for unvaccinated people (.1). This gives us a value of .9, or 90%. In other words, the vaccine is 90% effective.
I realize the math may still seem a little opaque. (I've added an Appendix that might help.) What's most important to notice is that the effectiveness stat is a ratio that stays constant across situations, even as your risk changes. Here's a simple demonstration:
Suppose that in my airport example the virus is really infectious. Like, zombie-movie infectious. We'll call this Scenario Z.
Scenario Z: All of the 1,000 unvaccinated people test positive for COVID. 100 people in the vaccinated group test positive.
If we do the math, we find that the vaccine is still 90% effective. (1.0 minus .1, divided by 1.0, equals .9). However, a key difference between Scenarios A and Z is that, all else equal, individual risk is greater in Scenario Z.
Specifically, in Scenario A, a vaccinated person has a 1 in 100 chance of infection following exposure at the airport. In Scenario Z, a vaccinated person has a 1 in 10 chance of infection following exposure. That's a big difference. But the vaccine is equally effective at preventing infection in each case. Exactly 90% more effective than doing nothing. The vaccine just has a more formidable adversary in Scenario Z, and so everyone's risk is greater.
Now look back at what CNN wrote. You should be able to see now why it's wrong. For every million people who fly, the number of vaccinated people we expect to get COVID depends on how many unvaccinated people get it. The higher the infection rate among unvaccinated people, the more powerful the virus must be, and the higher the infection rates will be among vaccinated people. But no matter what those rates are, the vaccine still reduces risk by 90%.
At this point you can probably also see why it's so easy to make CNN's mistake. Cognitively speaking, it's easier to grasp absolute values than ratios...
4. Partial translations.
We depend on experts to translate stats into lay terms. However, some of the clearest "translations" of effectiveness stats are incomplete, because the expert shows how effectiveness stats are ratios and then stops (as in this lucid correction of CNN).
In my opinion, once an expert corrects the CNN mistake, they should go on to stress one more key point: Effectiveness stats are poor predictors of individual risk. This is mainly because the stats represent group data – they tell us how much risk is reduced among a group, relative to being unvaccinated. How much risk an individual actually incurs depends on many factors.
Think of it this way: Effectiveness stats tell us we should get vaccinated, because doing so lowers risk of infection. Science tells us that behaviors like getting vaccinated, practicing social distancing, and wearing masks are what reduce the risk. None of these behaviors guarantee that you won't get infected, but each makes a contribution.
How should you think about risk then? Probabilistically, qualitatively, and situationally-specific. Shopping at that store tomorrow slightly increases your risk of infection. So does attending that party, or taking that bus. By how much you can't know exactly, because you don't know how many people you meet will be infected, how infectious they are, how close they'll get, how well the spaces are ventilated, etc. You can at least know that if you're vaccinated, your risk is a lot lower.
In short, since there's no way to generate precise estimate of risk, you have to choose your behaviors in the somewhat murky light of probabilities grounded in unknowns. Fortunately, if you're careful, your chances of infection should be quite small, and you can easily make them smaller.
5. Life.
Our understanding of effectiveness stats is evolving. The Pfizer vaccine now appears to diminish to about 84% effectiveness after four to six months (see here). All of the vaccines seem to offer less protection against the delta variant, and new variants may emerge in the future.
Because the virus as well as the science are changing, effectiveness stats will continue to change too. Our behaviors need to change in conjunction with them. When authoritative sources like the CDC or Anthony Fauci revise their advice, we should listen – and think critically about the data that prompted the revisions, rather than assuming that a change in advice reflects confusion. In other words, we should not be Laura Ingraham. (Did I mention that she's an axe murderer?)
Thanks for reading!
Appendix: More on effectiveness calculations
When people see how effectiveness is calculated, they sometimes ask: Why not just subtract the infection rate for vaccinated people from the infection rate for unvaccinated people, and stop there? Doesn't that difference tell you about effectiveness? Why do you divide the difference by the rate for unvaccinated people?
For example, in Scenario A, 10% of the unvaccinated group and 1% of the vaccinated group get COVID. Why not subtract 1% from 10% and conclude that the vaccine protects 9% more people who would've otherwise gotten infected?
This 9% figure is useful in some respects, but it's purely descriptive, and it gives us a misleading view of the vaccine's effectiveness. To illustrate, compare the following:
Scenario B: In the year 2025, a new virus emerges. In response, a new vaccine is developed. Worldwide, 2% of unvaccinated people become infected with the virus, as compared to 1% of vaccinated people.
Scenario C: In the year 2045, another new virus emerges, and another new vaccine is developed. Worldwide, 30% of unvaccinated people become infected with the virus, as compared to 15% of vaccinated people.
If we just look at differences in infection rates, the Scenario C vaccine seems way better than the one in Scenario B, because it prevents 15% of people from getting sick, as opposed to 1% in B. However, the true effectiveness of the vaccine is identical in these scenarios - 50% in each case. In both scenarios the infection rates among unvaccinated people are twice as high as they are among the vaccinated. The key difference is that the virus in Scenario C is much more powerful.
In sum, we get a clearer, more stable view of a vaccine’s effectiveness by dividing the difference in infection rates between vaccinated vs. unvaccinated people by the rates for the latter, in order to get a relative value telling us the percentage drop in infection rates following vaccination.
Part of what's at stake here is optics. If you announce that a vaccine only prevents 1% of people from getting sick, some people won't get vaccinated because they consider that number too small to merit a change in behavior. (Actually, it's not a small number when you consider that 1% of the world's current population is almost 80 million people. In Scenario B, roughly 80 million people got an infection that could've been prevented through vaccination.)
As for the optics of our actual situation, 9% does seem to be a conservative estimate of the difference between the percentage of the U.S. population who’s had COVID-19 as of this writing (a little less than 10%), and the percentage of vaccinated Americans who later contracted the disease (a little less than 1% - see here). I'm glossing over some complexities to bring these numbers together. My point is simply that with respect to optics – and accuracy – it seems better to speak of 90%+ effectiveness than a roughly 9% reduction in infection rates.