The Best Statistical Errors of 2022
One of the quickest paths to misunderstanding is the treatment of descriptive statistics as facts. The IRS hired 87,000 new employees this year. 57.5% of Twitter users want Elon Musk to step down. Tom Brady is the oldest player in the NFL. All facts, right?
If those were merely facts, this would be a short newsletter. Instead, they made my list of the best statistical errors of the year.
A statistic can be erroneous if it's inaccurate, or it's accurate but misinterpreted. That seems pretty clear. But how could one statistical error be judged "better" than another one? Aren't they all bad?
In my view, a statistical error can be considered "good" when three conditions are met:
(i) Evidence of error is unambiguous. Experts aren't questioning the numbers or how to interpret them.
(ii) The public is unaware of the error, or insufficiently alarmed by it. There's a need for greater accuracy.
(iii) The error illustrates a broad social trend. It's a statistical reflection of a more pervasive phenomenon.
In short, a "good" statistical error is not desirable, but rather informative. Calling it out reveals something important about a current social trend. And so, with that definition in mind, here's my list of the best statistical errors of 2022:
87,000 new IRS agents
Claims that the IRS hired 87,000 new agents this year have been made by a variety of Republican political leaders and candidates.
In fact, the IRS hired just over 4,000 new people in 2022, with the goal of hiring a total of 5,000 for the year.
That 87,000 figure comes from a 2021 estimate by the Treasury Department as to how many employees the IRS hopes to hire over the next 10 years, based on funding from the Inflation Reduction Act.
Republicans who got it wrong were especially vocal during the months preceding the mid-term elections. Their motives are most succinctly captured in a November 1 tweet from the Senate Republicans' official Twitter account:
“Senate Democrats could have secured the border to protect you and your family. They didn’t. Instead, they hired 87,000 IRS agents to audit you.”
This tweet is like one of those "Can you spot 10 things wrong with this picture?" activities. My concern here is the statement about hiring. Apart from citing a grossly inaccurate number, some of the 4,000-plus people actually hired were replacements for others who left, and not all of these people handle audits (only about one-eighth of IRS employees perform that function).
That 87,000 misrepresentation (as well as a social media version in which the new agents were armed) did get some pushback, though arguably not much, perhaps because there's a ton of other disinformation to contend with. And yet, it merits a close look as an exemplar a new species of political lie.
What's special about this lie? For one thing, it's especially brazen. The statistic isn't even close to accurate, disproving it is easy, and there couldn't have been any doubt that the IRS would immediately take steps to do so. I don't know which Republicans were lying and which were merely misinformed, but at least some of them propagated a lie knowing in advance that it would be spotted. I'm not aware of any apologies or retractions.
This kind of lie is also distinctive because it was unnecessary. If Republicans had wanted to raise fears about government oversight and IRS snooping, they could've done so without fabricating the hiring statistics. For example: “Senate Democrats could have secured the border to protect you and your family. They didn’t. Instead, they hired thousands of IRS agents to audit you.” Phrased that way, the statistical portion of the statement, at least, would be accurate.
Finally, I imagine you'd find bipartisan agreement that the number of IRS auditors isn't one of the most vital issues of the day. Unlike the claim that the 2020 election was stolen, excessive hiring at the IRS would not constitute a frontal assault on the Constitution.
In sum, that 87,000 statistic is a poster child for a new type of political lie. The issue is of moderate importance, the statistic is grossly wrong, inessential, and sure to be called out, but no retraction occurred when it was. This is lying carelessly, without fear of consequences.
What should we do when we encounter lies or mistakes like this? For one thing, we should continue to call out disinformation whenever we see it, even if some people use firehose methods of dissemination. At the same time, high school and college teachers should be introducing their students to nonpartisan fact-checkers (PolitiFact, FactCheck, Snopes, etc.) which identify distortions and falsehoods from all sources regardless of political ideology. If we don't speak out against disinformation, we may lose the privilege of doing so.
57.5% of Twitter users say that Elon Musk should step down
"Should I step down as head of Twitter? I will abide by the results of this poll."
Thus tweeted Elon Musk on December 18. The options provided were "Yes" and "No". During the 12 hours that the poll remained open, just over 17.5 million Twitter users voted, with 57.5% choosing Yes, and 42.5% choosing No. (As of today, Musk hasn't abided by the results.)
In my view, that 57.5% figure is the most misunderstood statistic of the year. I'll describe first how it has been erroneously interpreted, then argue that one of the errors has broader, if not immediately obvious, importance.
Everyone describes Musk's poll as "unscientific". I've lost count of how many writers have used this term, including some of the best data journalists in the business (Nate Silver, Philip Bump, etc.). In fact, if you consider the intended purpose of the poll, it's hard to imagine a more scientific approach.
What makes the poll seem unscientific is that it consisted of one question that one highly erratic CEO distributed via tweet. That doesn't seem like a systematic way of understanding how anyone feels about Elon Musk's leadership of the company. And yet, consider this:
––Participation in the poll was easy. In fact, it's almost impossible to make a poll easier to respond to. There were no phone calls, consent forms, or instructions. Just a single question, and a single yes-no decision to be made.
––The sample was large. 17.5 million respondents is roughly 5% of the total number of active Twitter users per month. 5% might not sound like much, but compared to most peer-reviewed scientific studies, it's a enormous percentage. If you were studying social media use among American high school students, for example, in order to sample 5% of them, you'd need to connect with nearly 1 million students.
––The sample was representative. We can't learn from the poll how Americans feel about Musk's leadership or how Twitter users in general feel. However, the sampling does allow us to learn something about a particular subset of Twitter users – i.e., those willing to share their opinions on company leadership. I can't imagine a better, more "scientific" way of finding those folks.
––The measure was appropriate. Although people may have nuanced opinions about Musk, if the options he's actually considering are simply to remain vs. step down as CEO, then his question and the yes-no response options are worded exactly the way they should be.
The only flaw with this poll is the possibility that subscribers created fake accounts in order to respond more than once. This isn't a serious flaw, unless you can make the case that one type of respondent (pro- or anti-Musk) would be more likely to cheat than another. I see no reason to believe that.
My conclusion is that the 57.5% statistic is fairly accurate and meaningful: Among the small percentage of users who are most concerned about Twitter's leadership (and most vocal about their views, and/or most confident that leadership is listening), opinions are divided, with a slight but clear majority preferring that Musk step down.
In sum, one error is to treat that 57.5% as meaningless. Another error is the inference many people have made (including some who call the poll unscientific) that Twitter users want Musk to step down. Again, we only heard from 5% of currently active users. This is a large sample, but the population is enormous. We have no idea how the majority of the Twitter community feels. We just know about the vocal ones.
Why care about this little anecdote? Because it reminds us of something important about interpreting voter behavior.
Musk's poll is analogous to the results of most elections in the U.S. Not all Americans are eligible to vote, not everyone who's eligible actually registers, and not everyone who's registered actually votes. Election results typically represent a minority of possible voters. Turnout rates for state and local elections are famously dismal, and even national elections tend to show a 60-40 pattern (roughly 60% of eligible voters turn out for presidential elections; roughly 40% vote during mid-terms. The figures for 2020 and 2022, 66% and 49%, respectively, were unusually high).
In short, election results aren't likely to accurately portray how Americans feel about particular candidates and issues, any more than Musk's poll tells us about how Americans, or even Twitter users in general, feel about Musk’s leadership. Rather, the results of these elections tell us something about the most vocal people among those who register to vote or tweet.
The moral of the story is a familiar one: Whether you're a Twitter user concerned about the future of the app, or an American concerned about the future of the country, one of the most powerful influences you can have is to persuade others to vote. The difference in votes between the winners and losers of every U.S. presidential election, most elections for Congress, and those who voted for vs. against Musk's leadership, is vastly less than the number of people who didn't vote in each case.
Tom Brady's age
At 45, Tom Brady is, chronologically speaking, the oldest active player in the NFL. I have no quarrel with that statistic, nor would I quarrel with a December 24 NY Times article describing the "Tom Bradys" of other professions (i.e., nine people who are, like Brady, among the oldest 1% of bakers, artists, tour guides, etc.) All the same, there's an error lurking here, one of potentially great importance.
A useful distinction can be made between chronological age and biological age. Chronological age corresponds to what we usually think of as "age" – the amount of time that's passed since birth (or, in some contexts, since conception). However, biological markers of aging, referred to as "epigenetic clocks", often work better than chronological age at predicting later-life outcomes such as illness and mortality rates. Epigenetic clocks are based on a cellular-level change called DNA methylation, which is detectable via a blood test.
In short, the difference between the Tom Bradys of the world and the rest of us may not be luck, talent, perseverance, or unusually strong bodies and minds. It may be DNA methylation. And there's more: Biological aging occurs in different parts of each person's body at different rates. Breast cancer cells, for example, exhibit accelerated aging compared to healthy cells in their vicinity. So, perhaps Tom Brady has a relatively young arm, biologically speaking?
Although scientists don't agree yet on how to measure biological age, this variable may soon supplement – or displace – chronological age as a predictor of health and longevity. This could be a good thing if, for example, it helps prevent discrimination based on chronological age. If I'm 65 years old (chronologically speaking) but live in the body of the average 47 year old (biologically speaking), perhaps I shouldn't have to pay higher life insurance premiums or be subject to mandatory retirement rules. On the other hand, if biological age achieves at least equal status with chronological age, the legal complications could be disruptive.
In 2018, a 69-year-old Dutch citizen requested that his legal age be reduced by 20 years in order to correspond to his biological age. The court refused, but one might ask whether more cases like this are coming. If scientific progress allows epigenetic clocks to outperform chronological age at predicting all meaningful age-related outcomes, then we may see, for example, lawsuits attempting to bar high school athletes from competition if they're too old (biologically speaking), lawsuits requesting exceptions to the minimum legal ages (chronologically) for voting, driving, drinking, etc., and a variety of other challenges to existing laws that pertain to age. Biological age is slightly more accurate than chronological age at predicting certain outcomes, but could those small differences justify so much legal and social upheaval?
Some inspiration
Out with the old, in with the new, but I'm not ready to leave the "old" behind just yet. Some of the most inspiring statistics of 2022 pertain to older adults. These aren't statistical errors though. You could call this final section of the newsletter an abrupt change in topic, or an extended footnote, because I want to close out the year with some happy statistics.
I struggled with terminology here, by the way. Descriptors like "senior citizen" and "elderly" are falling out of fashion. The AMA Manual of Style recommends a phrase like "older persons", which I'll use, but this phrase raises the question: Older than what? Since all of us are older than we were in the past, some people might be offended at being called an "older person", because it seems to emphasize that they're aging (in the chronological and/or biological sense).
Personally, I'm fine with being called older person (I'm 61), but what should my 83-year-old neighbor call me? And what should I call her? An "even older person"?
The problem with euphemisms like "older person" is that they inherit the same negative connotations as the words they replace. I agree that "senior citizen" ought to be retired, because not all of us are citizens, but I see nothing wrong with "elder" or "senior" or "old person". Whatever ageism resides in those terms will reappear in "older person". (Ideally, the mere act of asking people to change their language will sensitize them to their prejudices, but I'm not sure how often that actually happens.)
Apologies for ranting. I must sound like a cranky older person.
Here are three of the statistics from 2022 that I found most inspiring:
–At 95, Angela Alvarez became the oldest person to win a Latin Grammy. (For best new artist, no less!) Ms. Alvarez doesn't sing well for her age; she simply sings well.
–One day after he turned 100, Lester Wright broke the world record for centenarians in the 100-meter dash. (I'll be happy if I can walk to the kitchen when I'm that age.) Although Mr. Wright's time of 26.34 seconds won't earn him a spot at the next Olympics, he's a humbling lesson to runners like me who whine that I'm getting old whenever I feel the slightest twinge in my biologically middle-aged knees.
–Finally there's Brandt Duddy, a 93-year-old technician who works on the largest pipe organ in the world, the Midmer-Losh, located in Atlantic City's Boardwalk Hall. As described in a New Yorker profile, the Midmer-Losh has more than 33,000 pipes – so many that owing to accidents, natural disasters, and general wear-and-tear, it has only been fully playable for two years since it was first built in 1932. Mr. Duddy has been working on this organ since age 17 and has voiced about 10,000 of the pipes, so that now just over half of are playable. Another 10 years will be needed until the organ is entirely restored. "I won't be around to hear it", says Mr. Duddy, "but, as long as I can walk up the steps and do my shtick, I'm happy."
I know the feeling, Mr. Duddy. As long as I can do my schtick, I'm happy too. I'm inspired by your example.
A quick look ahead
I'm grateful to each of you for shticking with me this year. In spring 2023, I'll be rolling out the Statisfied website. My former employer, Southern Methodist University, has kindly awarded me a grant for developing the site, and I'm currently working with a designer on the details. Along with resources for students, journalists, health professionals, and others who engage with statistics in their work, there will be some advanced material, some information for the general public, and a brief section containing "statistics of the week" that seem especially informative – or deceptive.
Thanks for reading. I wish us all a happy 2023!