The Hot Dog Study
"Eating 1 hot dog claims 35 minutes off life, study suggests"
So proclaims a recent Fox News report.
I don't like hot dogs, and I don't like Fox News, but what I like even less is the misuse of statistics – by researchers, journalists, politicians, etc. – to scare people. (Fair warning: you're about to read a pretty cranky newsletter.)
This newsletter isn't actually about hot dogs, or about Fox. It's about a recent study that examines the health effects of 5,853 different foods and ingredients. Ultimately it's about how statistical models can be so complex that (a) deep flaws in the data are hidden, and (b) journalists may end up summarizing the research uncritically – and incorrectly – because the statistical material is too complex and buried for them to evaluate.
By "journalists" I'm not just referring to those at Fox News. This particular story was picked up by Yahoo, CNN, Washington Post, NPR, and many other national and local media. None of the dozens of stories I've seen notes possible limitations to the study, and some of them misrepresent the outcome variable.
In reputable news sources, descriptions of a new study usually include comments (perhaps from experts) on potential limitations. The results should be interpreted cautiously, you might read, because the sample was small, or the researchers over-interpreted their data, or whatever.
In the case of the hot dog study, it would’ve been tough to fold methodological critique into a news report, but it was easy for journalists to present the findings, because they were described so simply in the study. For example, the researchers reported that eating one hot dog reduces how many years of healthy life you'll live by 35.4 minutes.
Notice that my wording above differs slightly from that of the Fox News headline. This is because the researchers didn't actually estimate changes in how many minutes you'll live. Rather, they estimated changes in how many minutes of healthy life you'll live after eating each food. The outcome variable isn't longevity, in other words. It's length of life before developing a disease linked to consumption of each food. Some of the journalists got that wrong.
Now that you know what the outcome variable is, the findings are still weirdly specific. Here are some others: Eating one peanut butter and jelly sandwich increases your healthy life span by 33.1 minutes. Eating one serving of pizza reduces your healthy life span by 3.6 minutes. Eating one serving of fruits increases you're healthy life span by 11.9 minutes. The study reports 5,853 such findings!
At this point, you might be wondering: Is this a non-peer reviewed article published in a crappy journal by researchers with questionable credentials? Well, no. It's a peer-reviewed article published in a reputable journal (Nature Food) by public health researchers at a prestigious institution (University of Michigan).
When I first heard about this study, I thought back to the peanut butter sandwich I'd eaten earlier in the week and wondered: Have I really just gained 33.1 minutes more healthy life? Doesn't that depend on how much jelly I used, and what kind? Shouldn't I care about the type of bread? As for the peanut butter, doesn't it matter whether it's "natural" (just peanuts and salt) or the kind that has corn syrup and other additives?
I'm asking these questions because they lead straight to the core problem with this study. It's known as the false precision problem, and it’s always a possibility when study conclusions emphasize specific numerical values.
To illustrate the false precision problem, let me ask you first: Did you know that 27.6% of statistics are made up on the spot?
That's an old joke that has to be used at least once in a statistics newsletter, and it illustrates the nature of false precision. We know that some statistics are fabricated, but if we try to estimate how many, any number we come up with (e.g., 27.6%) will imply more precision than the evidence could possibly justify. In other words, it's fine to say that "some" statistics are fabricated, because we have evidence for that. It's not fine to say that "27.6%" are fabricated, because we can't know the exact percentage.
The precision of that 35 minute figure immediately raises doubts about the hot dog study, Even if the methodology were strong, we know that however the researchers gathered data, a different sample would've yielded a different estimate. For instance, one hot dog might be found to reduce 29.7 minutes of healthy life, or 38.3 minutes, or whatever, depending on the particular sample. Moreover, calculations such as this don't map well to individuals (and individual hot dogs), for reasons I'll get to in a moment. So, what do the researchers want us to do with such precise figures? In their Conclusion section, they refer vaguely to how their results "could lead to personalized diet solutions" and "could inform" dietary guidelines and programs, but they provide no details. This is ironic. They give us precise estimates for 5,853 foods and ingredients, but only the vaguest hint of what to do with the data.
How did the researchers come up with their estimates? Readers who happen to be public health experts, and are familiar with how Global Disease Burden stats are calculated, will understand their approach pretty easily. Otherwise, you're lost in a swamp. The researchers don't fully explain the statistical approach in their Methods section, or even in the Supplement they refer readers to. You have to examine the minutiae of prior publications in order to piece together all that was done. The complexity, and the need to dig through prior work is, I suspect, part of the reason journalists said nothing about the methodology.
In simple terms, here's how the estimates were generated:
First, standard serving sizes were assumed. For example, for chicken wings the assumption was 85 grams (about 3 ounces).
Second, for any foods consisting of more than one ingredient, assumptions were made about what ingredients each food contained, as well as the amount of each ingredient. For example, the assumptions for one serving of chicken wings were 1.85 grams of polyunsaturated fats (PUFA), 0.0281 grams of calcium, 0.492 grams of sodium, and 0.139 grams of trans fatty acids (TFA). No other ingredients were considered.
Third, existing databases, meta-analyses, and/or studies were consulted to determine the health effects associated with the consumption of one gram of each food or ingredient. For foods containing multiple ingredients, health effects were aggregated. With respect to chicken wings, the researchers calculated that the benefits of PUFA and calcium, combined with the adverse effects of sodium and TFA, would result in 3.3 minutes of healthy life lost per serving. (The researchers assumed that the chicken meat itself would not affect health.)
I'm sure you can already spot a number of problems with this approach. Serving sizes vary. Ingredients vary. Chicken meat itself probably does affect health. Individuals vary in their metabolisms.
The deeper you dig, the worse it gets. Where did those databases etc. get their information? Mainly from studies in which people describe their typical eating habits, either overall or with respect to specific foods, and provided information on their health. Unfortunately, people aren't very accurate when reporting their eating habits over time. Much evidence attests to this (and some studies also show that folks under-report "embarrassing" patterns of consumption). Even without such evidence, you already know that people don't track exact quantities of what they eat. (How much sodium was in the last serving of chicken wings you ate? How many grams of wings were there?)
There's also a lot missing from the hot dog study. The researchers considered individual differences in metabolism for sugar-sweetened beverages and sodium, but not for other ingredients, nor did they (or the studies they reviewed) consistently take into account variables known to affect health such as exercise, stress, alcohol and drug consumption, and so on. The researchers themselves acknowledged a failure to consider cooking methods (which can radically alter the nutritional value of some foods and the harmfulness of others) or adjustments made at table (e.g., whatever you might dip your chicken wings into). They also noted that their statistical methods treated the the effects of each food or ingredient on health as independent, whereas those effects are likely to be interactive.
I could go on, but this should give you a "flavor" of why it's impossible to trust the stats reported in this study. False precision stems from many sources; I've touched on some concrete examples. As a rule, estimates rely on a trade-off between precision and accuracy. If I guess that you're less than 6 feet 10 inches tall, I'm almost surely correct, but that's not a very precise description of your height. Guessing that you're between 5' 5" and 5' 6" is more precise but less likely to be accurate. In the case of the hot dog study, the researchers sacrificed accuracy for precision, again and again.
Could I have done a better job at calculating estimates? No way. I wouldn't want to. In my opinion, epidemiological research liking diet and health doesn't benefit from this kind of excessive precision. It's false precision, and it scares people needlessly.
What people need to know is that if you eat "a lot" of hot dogs, for instance, and you're spending "a lot" of time sitting on the couch, smoking and drinking "a lot", then you're increasing your risk of "a lot" of health problems. People need some rough guidelines for how to define "a lot" in each case, but nobody – not the general public, not policymakers, not health care professionals – needs to be told that eating one hot dog reduces your life span by 35 minutes, because it's just plain dumb and needlessly scary to imply greater precision than the research could possibly sustain.
(It's also important, when you advise people that eating "a lot" of something might hurt or help them, to note whether scientists know anything about the underlying mechanisms. The researchers don't explain why eating one peanut butter and jelly sandwich would improve your health, much less why that improvement would translate into 33.1 extra minutes of healthy life. That estimate is just not believable if you can’t explain why. In contrast, although the fried ice cream at the Texas State Fair is awesome, you wouldn't want to eat “a lot” of it every day for three decades, because scientists have some understanding of the mechanisms by which excessive cholesterol and refined sugar, for instance, damage the cardiovascular system.
So, whatever your next meal happens to be, please enjoy it! It's probably not going to affect how many minutes of healthy life you have ahead of you (unless your next meal happens to be two Big Macs, a quart of vodka, a plate of fried ice cream, in which case I will say now: I'm very sorry to lose you as a reader).
Thanks for reading!