Conflicting Health Advice: A Consumer Guide
"There's a new milk every two weeks....They say don't drink cow's milk, you should drink soy milk. Then they discovered soy milk is all estrogen, and they're like, you should drink rice milk. Then they discovered that it's like drinking carbs, and they're like you should drink almond milk ('cause almonds make milk), but if you have a nut allergy you should drink hemp milk, which is like a nut-free almond milk that's made from rope. If you don't like that you can try this new milk – it's called cow's milk." –Jim Gaffigan
This newsletter is the second of a two-part series. Here I'll be describing four sources of conflicting health advice:
1. Inherently complex issues.
2. Methodologically weak studies.
3. Deceptive or incompetent researchers.
4. Inaccurately-described studies.
To illustrate these sources, I'll be discussing recent reports on aspartame, ketone supplements, ivermectin, pooping, sleep, and the "health of the nation". At the end I'll offer suggestions on dealing with conflicting health data.
1. Inherently complex issues.
Some of the health outcomes we care about most – living long, staying fit, avoiding cancer and heart disease, etc. – are influenced by many variables that each play a small, complicated role.
Well-designed studies may yield conflicting findings simply because it's so hard to tease out the influence of any one variable when so many others are at play.
Complicating matters is that health-related behaviors often need to be measured over long periods of time, but researchers can't follow people around and record, say, how much saturated fat they eat every day.
All of these challenges are illustrated by the latest controversy around the artificial sweetener aspartame.
The aspartame debate
Aspartame is in the news again because the World Health Organization (WHO) announced last week that it may be carcinogenic. This announcement was immediately contradicted by individual scientists, organizations such as the FDA, and even a branch of the WHO itself.
(You read that right. Two different WHO agencies are offering conflicting opinions about the safety of aspartame. The WHO's International Agency for Research on Cancer (IARC) has classified aspartame as a potential carcinogen, while the WHO's Joint Expert Committee on Food Additives (JECFA) has responded that the evidence linking aspartame to cancer is "not convincing".)
After 50+ years of aspartame research– the FDA describes it as one of the most studied food additives – we might wonder: Why do experts still disagree about its effects, and who should we trust? Should we be avoiding Diet Coke, Crystal Light, Trident, or any of the other roughly 6,000 products that contain this additive?
Aspartame-cancer connections?
Between 2014 and 2022, three large longitudinal studies linked aspartame consumption to liver cancer. The effects were small but significant. However, in these studies aspartame consumption estimates were based on how much artificially-sweetened beverage (ASB) each person drank. This guarantees muddy data, because (a) some ASBs contain other artificial sweeteners, and (b) some foods contain aspartame.
In 2022, a team of French scientists reported data on just over 100,000 participants showing that aspartame consumption predicted breast and obesity-related cancers (analyses for liver cancer couldn't be conducted). The effects were again small, but this time, the researchers made a heroic effort to measure aspartame intake accurately. They asked people to record the type, quantity, and, if applicable, brand of each food eaten over the past 24 hours. Then they conducted thousands of assays to determine the aspartame composition of any foods containing the additive. Given such a meticulous approach, the results should be definitive, right?
There's an old joke about a guy poking around on the sidewalk under a streetlight. A cop comes along and asks what the guy is doing.
Guy: "Looking for my car key."
Cop: "Where do you think you dropped it?"
Guy: "Across the street."
Cop: "Then why aren't you looking for it there?"
Guy: "The light's better over here."
Metaphorically speaking, this sort of thing happens a lot in health research – scientists explore what they can rather than what's most needed. Asking people what they just ate and then running chemical assays is a very, very bright streetlight. But it might not reveal the key. Why not? Because in this study, the researchers only asked about dietary intake for two weekdays and one weekend day every six months. That's a tiny sample of what people were eating. Specifically, three days out of every six months is about 1.67% of the time. The key – i.e., the true amount of aspartame intake – is more likely found in that other 98.33% of days rather than in the 1.67% on which the researchers shone their light.
We can't blame the researchers. Tracking the daily food intake of 100,000+ people over a period of years would be logistically infeasible. We have to hope, as the researchers did, that they obtained a representative sample of participants' aspartame intake.
Maybe they did, maybe they didn't. This isn't like a national poll, where a representative sample can be deliberately selected (1.67% of American adults would be 4 million people – way more than needed.) The sampling of dietary intake in this study creates a huge question mark.
Still, I don't find the methods weaker on the whole than those of the many studies suggesting that aspartame is safe. None of these studies can fully manage the complexity of the topic.
So, who should we believe?
I find the IARC's view most reasonable. If none of the observational studies are ideal, but a small number of them link aspartame to cancer, then we should consider the possibility that aspartame is carcinogenic. There may be a genuine, albeit weak signal in the midst of the noise.
However, I wouldn't necessarily recommend that you avoid aspartame products. If this additive increases the risk of cancer, the extent of influence is surely small.
How small? In the 2022 study, for instance, no effects were observed for low levels of aspartame consumption. People whose daily intake averaged 47.42 milligrams per kilogram of body weight were about 15% more likely to develop some sort of cancer.
To put those stats in context, if you weigh around 150 pounds, you'd have to drink well over 10 cans of any brand of diet soda per day in order to reach that 47.42 mg level.
(If you do drink that much diet soda, you might want to scale back, but don't assume that any specific number, such as 9 cans per day, is "safe". At best, group data only allows for estimates of personal risk. And, extrapolations of lifetime risk are imprecise, as most studies aren't cradle-to-grave.)
2. Methodologically weak studies.
Research findings may conflict when one or more of the studies relies on weak methods.
Researchers routinely accuse each other of bad methodology when their results don't agree. Even so, in some cases it's relatively easy to spot the methodological flaws, in which case you can comfortably take a side in the debate. This is illustrated by a new study on ketone supplements and athletic performance.
Ketone supplements and performance
Ketone supplements (drinks, powders, gummies) have become popular among competitive athletes, particular those in endurance sports like distance running and cycling. The assumption is that they improve performance, for the same reason that the "keto diet" promotes weight loss.
The basic idea is that when your blood sugar is low, your fat cells release fatty acids that are converted into ketones by the liver. The ketones then serve as an alternative fuel source. In short, your body relies on fat more than glucose for fuel.
The "keto diet" calls for lots of fat but minimal carbs. This keeps your blood sugar low and tricks your body into metabolizing fat. (Studies show that these diets are effective, though in practice they don't seem ideal. If you resume normal carb intake, you'll probably gain the weight back. If you continue keto-ing, you increase the risk of health problems such as cardiovascular disease.)
Distance runners, triathletes, WorldTour cyclists etc. use ketone drinks and supplements to boost fat metabolism. Our bodies store more fats than carbs, and so with higher ketone levels, athletes should be able to push themselves longer before that sudden, dramatic fatigue fondly (or not so fondly) known as the "bonk" or "hitting the wall."
The new ketone study
Evidence on the actual performance benefits of ketone supplements has been mixed. This April, a new study led by McMaster University researchers suggested that ketones actually impair athletic performance.
The design of this study was rigorous and straightforward: 23 competitive cyclists each came to the lab and cycled as hard as they could for 20 minutes. A week later each one did the same thing again.
During each of the two lab visits, the participant was asked to drink a beverage before cycling. During one visit, the beverage was a ketone drink. During the other visit, it was a placebo. We call this a within-subjects design – each person cycled once after the ketone drink and once after the placebo.
The researchers measured how much power each cyclist generated during each 20-minute trial. (Power is a function of pedaling speed and the gears each cyclist chooses to engage.) The main finding was a roughly 2.4% loss of power on average after ingesting the ketone drink as opposed to the placebo.
Although the sample was small, the results are persuasive, given the rigor of the study methods. Each trial was conducted at the same time of day, following the same warm-up procedure. The amount of beverage consumed by each participant was exactly proportional to body weight (0.35 g/kg). The flavors of each beverage were the same. Et cetera.
So, why do the results of this study conflict with those of some prior studies (and the beliefs of some competitive athletes)? Which findings should we trust?
To put it bluntly, I trust the new study more, on the grounds that the methodology is better.
Here's a key difference: Earlier studies required participants to fast before the lab visit, or to otherwise alter their diets in ways they would not ordinarily do before a competition. In this study, participants were told to eat and drink as they normally would prior to coming to the lab.
In a word, the ecological validity of this study is better. That's a fancy way of saying that the experiment more closely approximated real-world conditions.
I conclude that ketone supplements probably don't boost athletic performance (although, as always, more research would make that conclusion stronger.)
3. Deceptive or incompetent researchers.
Research fraud is the really bad stuff, such as fabricating data. Researchers also pursue activities that are considered wrong but don't rise to the level of fraud. I use the term "malpractice" to describe these activities, since the term encompasses both intentional as well as unintentional wrongdoing. A common example is p-hacking (running lots of analyses until one or more come up significant, then only reporting the significant ones), which some researchers do without realizing that it increases the chances of false positives and other problems.
(Researchers are biased too, but I'm not including that as a separate source of conflicting data, because we're all biased by our particular world views. Bias in this sense informs all studies, whether or not the findings agree.)
Fraud and malpractice have both been responsible for conflicting health data. For instance, a preprint of a study, available now, has identified data fabrication in the work of Sherief Abd-Elsalam, an Egyptian researcher who has already had six peer-reviewed papers retracted for ethical reasons. Among other things, Abd-Elsalam has co-authored several discredited studies promoting the most controversial COVID-19 treatments, including hydroxychloroquine and ivermectin.
Ivermectin works fine for problems like head lice and scabies. As a treatment for COVID-19, the data was initially mixed, but by 2022 researchers had reached a consensus that it has zero effectiveness.
We now know that that the ivermectin data were initially mixed because studies that favored it either reflected extremely poor methodology or fraud. Abd-Elsalam is the most recent of several ivermectin researchers found to have engaged in data fabrication and other forms of misconduct.
There's nothing you can do about health advice grounded in fraud and malpractice, other than to change your thinking once a problem is discovered. Not everyone does that. A sad and frustrating example is the case of Andrew Wakefield, who continues to be supported by fringe groups for promoting anti-vaccination conspiracy theories, in spite of the retraction of his papers on the topic (owing to conflict of interest and data fraud), his permanent ban from practicing medicine, and his reputation as one of the worst medical fraudsters in recent history.
4. Inaccurately-described studies.
Health data may seem contradictory when one or more studies are described inaccurately by health care professionals, journalists, policymakers, bloggers, etc. (Here I'm rephrasing what I wrote about this topic last week, with new examples.)
Inaccuracies bubble up from a variety of sources. Perhaps a study is misunderstood, or the person describing it doesn't realize it has been discredited. Perhaps the person is biased, or intentionally deceptive. Perhaps it's just a semantic issue. (In some alcohol studies, researchers describe the the lowest level of consumption as "moderate" – e.g., one glass of wine per week – but some journalists take that to mean a level in between "high" and "low.")
Here are two examples of inaccurate coverage in the news recently:
Pooping and cognitive decline
Yesterday, CNN.com ran an article on a new study linking constipation to cognitive decline later in life. As the CNN writer put it: "Being chronically constipated, defined by the authors as having a bowel movement only every three or more days, has been linked with a 73% higher risk of subjective cognitive decline."
That's not an inaccurate description of the findings. The problem is that the journalist misrepresented the possible mechanisms. Why would constipation be linked to cognitive decline? (When we think someone isn't reasoning clearly, we might say they're full of shit, but we don't mean it literally.)
The journalist offered a confusing discussion of the gut microbiome, noting that chronic constipation depletes chemicals that prevent bacteria from entering the bloodstream. What's missing here is a clear statement of the researcher's speculation that these bacteria then travel to the brain and impair cognitive functioning. Also missing is any mention of the obvious counter-explanations for the findings.
Specifically, poor nutrition and inactivity are two of the many reasons that elderly people might be chronically constipated and have impaired congition.
In other words, it may wrong to say that chronic constipation causes cognitive decline. Rather, poor nutrition and/or a sedentary lifestyle might be responsible for both. These possibilities should've been addressed by the researchers and noted in the CNN story.
The health of the nation
In a July 10 Atlantic essay entitled "America is doing just fine", Joe Scarborough (MSNBC's "Morning Joe") argued that in spite of countless doom and gloom assessments of the state of our nation (including one published in the Atlantic the same week), we're still leading the world in military power and higher education, and our rates of childhood poverty, teenage pregnancy, and unemployment are at historic lows. In a word, Morning Joe says we're in good health.
I'll pause for a moment while counterexamples come swirling through your brain.
Since this newsletter is already quite long, I won't lay out my concerns here but simply note that Dr. Joe's article is a sort of display case for the different ways that one can cherry-pick statistics.
Cherry-picking is an extremely prevalent source of conflicting health data. Yes, you can find evidence that America is in good health. But, it's also easy to find evidence that we're not. (What does "good health" mean anyway at a national level? Does it even make sense to try aggregate so many different kinds of data?)
When conflict is only apparent
Finally, in some cases health studies don't actually conflict; rather, they're just misrepresented in the news or by some other source. For example, consider these two recent headlines:
"Exercising can wipe out the damage of poor sleep..."
"Short sleep negates benefits of exercise..."
The first one seems to say: Don't worry if you sleep badly – exercise can compensate.
The second one seems to disagree: Whatever you gain from exercise will be lost if you sleep badly.
Actually, the first headline refers to a study about mortality, while the second one references a study on cognitive decline. The results aren't necessarily inconsistent.
If you can trust both studies, here's what they imply: Exercise moderately and you'll live longer, even if you sleep poorly, But if you continue sleeping poorly, you may experience more cognitive decline, in spite of the exercise.
Conclusion: Six practical guidelines
Conflicting health data is a sort of meme, but I'm not sure there's more of it than there is in any other area of research on long-term outcomes that are driven by complex interactions among variables.
Experts do agree about some things (e.g., how to handle the extreme heat this summer). But, discrepancies in health advice grab our attention, because we care about our health, and because the advice often concerns things we have to do but could be doing differently. (Eat more carbs? Fewer carbs? Different carbs? Well, you have to eat some.)
Here are six guidelines for making sense of conflicting health data:
1. Understand what each side of the conflict is claiming.
For instance, among reputable organizations and scientists, the aspartame debate isn't "safe" vs. "carcinogenic". It's "safe-if-not-consumed-in-excess" vs. "may-be-carcinogenic-if-consumed-in-sufficient-quantitities."
2. Consult a reliable source of information about the conflict.
(See the end of last week's newsletter for discussion of what makes a source reliable.)
3. Be cautious about simplistic, single-variable, and/or extreme positions.
Health outcomes are complex. Even if the McMaster study is a fluke, ketone supplements will not win you a gold medal at the next Olympics.
4. Look for financial and/or ideological motives behind one or all sides of a conflict.
Cherry-picking and other forms of misrepresentation are incentivized by ideology and financial rewards. In the extreme case, such as Andrew Wakefield's vaccine-autism study, or some of the ivermectin research, the data itself may be fraudulent.
5. Beware of cultural lag.
Science is progressing more rapidly than it used to, thanks to more powerful computers, better technology, more data sharing, etc. Both knowledge and practice may thus become obsolete more quickly than ever. If you're using ketone drinks to increase your marathon PR, keep in mind that at least one new, well-designed study now suggests that you might be slowing yourself down.
6. Avoid the ecological fallacy.
The ecological fallacy refers to inferences about individuals based on knowledge of groups. This is a fallacy insofar as what's true for a group may not be true for any one its members.
You have your own genetics, metabolism, lifestyle, and living conditions. Health research provides estimates about the benefits or risks of particular activities, but those estimates are statements about the populations studied. None of the conflicting advice about some health-related behavior will necessarily fit you well.
(All the same, I want to assure my friend J, a dedicated runner who uses ketone drinks even though they give him gas, that he can stop with the ketones now. Just stop, J. You'll run faster – and the rest of us will be happier.)
Thanks for reading!