Does Mindfulness Reduce Anxiety?
Two guys are sitting in a bar, commiserating about their low wages. Suddenly Elon Musk walks in. "Cheer up", says one of the guys. "On average everyone in the bar is a billionaire now."
There must be at least a dozen bad jokes about the limitations of arithmetic means. Scholars have complained about these limitations ever since the mean first became a popular statistic among 19th century social scientists. Nonetheless, the mean caught on, and it continues to be a routinely used tool in many fields.
In this week's newsletter I want to share with you a new study on the use of mindfulness techniques for helping people cope with anxiety. In some respects it's a strong study. But it's also a poster child for why means are appealing – and what goes wrong when researchers rely too heavily on them.
Mindfulness
Mindfulness is the practice of attending to the present moment without judgment. You stay aware of whatever thoughts, emotions, and sensations come to mind without "doing" anything with them, such analyzing them or blocking them out.
First discussed in Buddhist teachings as a spiritual practice, mindfulness has evolved into a purely secular approach to self-improvement. For instance, in the U.S. it has been used for nearly half a century to help people deal with chronic pain, stress, and psychological disorders.
Rationale for study
The study I'll be discussing was published in JAMA Psychiatry this November 9 by Georgetown University professor Elizabeth Hoge and colleagues.
In my view, the rationale for conducting this study was exceptionally strong.
1. Anxiety is the most common type of mental health problem. It's estimated that nearly 3 in 10 Americans will experience an anxiety disorder at some point in their lives.
2. Treatment for anxiety is not always desired, accessible, or effective. Researchers estimate that over 40% of Americans with an anxiety disorder don't seek treatment (due to the cost of therapy, mistrust of medication, etc). Statistics on treatment effectiveness vary, but even when people are fully compliant with an established treatment (therapy and/or medication), they don't necessarily improve.
3. Mindfulness practices such as meditation appear to reduce various forms of psychological distress, including anxiety, and these practices are cheaper and often considered more appealing than conventional treatments (e.g., drugs).
4. No experimental studies have directly compared mindfulness to first-line drugs with respect to anxiety relief. ("First-line" means the first treatment prescribed, owing to its known effectiveness.)
Thus, the researchers examined whether an established mindfulness program would be as effective as a widely-prescribed drug at reducing anxiety. (Technically, this is called a "non-inferiority trial", because the purpose is to determine whether a new, possibly more desirable treatment works as well as an existing one. Even though the researchers would be delighted if the mindfulness intervention outperformed the drug, a successful outcome would simply be no differences between the two in effectiveness.)
Study methods
Participants were 276 adults diagnosed by an expert as having an anxiety disorder (generalized anxiety disorder, social anxiety disorder, panic disorder, or agoraphobia).
One strength of this study is the care taken with sampling. The researchers ensured that participants had no other psychological disorders, no prior experience with mindfulness training or meditation, and no current medications that might impact the results of the study.
Roughly half of the participants were randomly chosen to participate in a mindfulness intervention called Mindfulness-based stress reduction (MSBR). MSBR consists of one 2.5 hour class per week for eight weeks, one full-day weekend retreat, and exercises to be carried out at home for 45 minutes per day. The focus is on meditation techniques that attune practitioners to body, breathing, and gentle movement during stretching.
The other half of participants received escitalopram, which is most commonly sold under the brand name Lexapro. Dosage was set at 10 to 20 mg per day.
The choice of conditions (MSBR vs. escitalopram) is another strength of this study. MSBR is an evidence-based approach that has been shown to help with a variety of problems, while escitalopram is a widely-prescribed drug for treating anxiety. In short, a strong intervention was compared to a popular drug.
Finally, the researchers measured anxiety by means of the Clinical Global Impression of Severity Scale (CGI-S). To administer the CGI-S, a clinician interviews a person once, then rates their problem (in this case, anxiety) on a 7 point scale labeled as follows: 1 = normal. 2 = borderline mentally ill. 3 = mildly ill. 4 = moderately ill. 5 = markedly ill. 6 = severely ill. 7 = among the most extremely ill patients.
The CGI-S is not an ideal measure. Instructions for use are minimal – the clinician is simply asked to rely on their own experience in order to choose among the seven options. This is quite subjective and raises the question of how much clinicians might differ in their ratings of the same individual. I did a quick literature search and found that although the CGI-S is widely used, its reliability and validity aren't well-established, and researchers are aware of limitations stemming from its subjectivity.
Study results
Hoge and colleagues summarized their main findings in the figure below:
Three details in this figure are important:
1. At baseline, the two groups were virtually indistinguishable in anxiety (mean CGI-S scores of 4.44 for the MSBR group and 4.51 for the drug group). Methodologically speaking, that's good news. In an experiment like this, you want the groups to be as similar as possible at the outset.
2. Both groups showed a significant decline in anxiety by week 4 that remained lower than it had been at baseline. This is good news too, because it shows that people benefited from participating in the study, regardless of which group they were in. The benefits lasted for at least 24 weeks after baseline.
3. Declines in anxiety among the MSBR and drug groups were not significantly different. In fact, the group means become so similar that by week 8, you can't visually distinguish the two lines on the graph. (At week 24, the mean CGI-S score for each group was exactly 2.92.) Clinically speaking, this seems like great news, because it suggests that MSBR works just as well as the drug. That, indeed, was the researchers' sole conclusion.
An interpretive issue
I really wanted to love this study, because mindfulness meditation would offer people a relatively simple, inexpensive way of managing anxiety. The study findings do seem impressive. However...
The researchers judged the success of MSBR on the basis of mean scores. The fact that the MSBR group improved as much as the escitalopram group did, on average, was the only evidence for concluding that MSBR is just as effective as the drug at reducing anxiety.
Means are appealing because they're convenient, but the problem with exclusive reliance on them is that they're consistent with a variety of individual trends, some of which contradict what the means seem to show. For example, it's possible that most of the MSBR group actually got slightly more anxious over time, but dramatic improvement among some of them made the group mean look favorable. (To be fair, it's also possible that most of the MSBR group improved, but a few people who got dramatically worse lowered the group mean to more modest levels.)
Here's a concrete example of what I have in mind: The means reported by the researchers are consistent with the possibility that 55% of the MSBR group actually increased in anxiety from 4.44 to 4.45, on average, while the other 45% dropped from 4.44 to 1.05. Would you conclude in this case that MSBR is effective? I wouldn't. Rather, I'd say that MSBR was strikingly effective among 45% of participants, but not effective at all among the other 55%. This is quite different from the researchers' conclusion that MSBR is just as effective as the drug.
There are many possible scenarios besides the ones I've described here. The point is that the results are ambiguous owing to exclusive reliance on mean comparisons. Ambiguous to the point of being uninterpretable.
Better approaches
The researchers could've used one or both of the following simple approaches to describe their results more clearly.
1. Frequency analysis.
This could simply mean reporting data on how many people in each group improved, got worse, or stayed the same from one time point to the next. Or, data on how many people improved a lot (i.e., by 2 or more points) declined by a lot (i.e., by 2 or more points), or stayed approximately the same (i.e., by changing 1 point or less in either direction).
2. Subgroup analysis.
This means running analyses that look at effects within subgroups. To take a hypothetical example, it might turn out that age matters for MSBR but not for the drug, in the sense that younger people benefit more from MSBR than older people do, but the effectiveness of the drug is unrelated to age.
Neither approach fully addresses the fundamental question of how to judge the success of an intervention. In the case of frequency analysis, you still have to decide what ratios of people who improve vs. stay the same vs. get worse would constitute success. In the case of subgroup analysis, you still have to consider outcomes for the individuals in each subgroup. Even so, these approaches provide clearer views of group data than means do.
Why is this important?
I'm not the first person to comment on the perils of mean comparisons. There's more than a century of scattered commentary on this topic, and in a future newsletter I'll explain why means became popular anyway.
What's at stake here is much more than just the reputation of mindfulness techniques. Across a variety of disciplines, you'll find that researchers don't always look beyond means to evaluate the results of their experiments. If you're an educator, a health care professional, or a therapist, for example, you often hear about intervention studies, but you may not realize which ones rely on appropriate statistical comparisons and which ones simply compare group means (in which case you may be misled about the effectiveness of the intervention).
This is a widespread problem – I'm not sure yet how to characterize the extent of it – and it's not routinely addressed in media coverage of experimental research. For example, the new mindfulness study was described in articles on the websites of NPR, CNN, NBC, and the CBC, but all of the articles simply noted that MSBR was as effective as the pills.
Is mindfulness advisable anyway?
At this point, you might be thinking: Even if MSBR doesn't help everyone with anxiety, at least some people benefit, and those who don't won't be harmed.
Mindfulness techniques such as meditation do seem innocuous. I'd recommend them to almost anyone. However, I feel obliged to point out two concerns beyond what I've already described:
1. Clinicians have noted that people who don't ordinarily sit quietly and meditate might experience disturbing thoughts and feelings that they're not equipped to deal with. Crudely speaking, it might be risky to take away peoples' defense mechanisms (e.g., staying busy) without offering additional support.
2. A study published in the influential Journal of Personality and Social Psychology this June found that mindful meditation caused people to feel less guilty about harming others and less likely to make amends. The lead researcher, Dr. Andrew Hafenbrack, is an advocate (and practitioner) of mindful meditation, but he notes that meditation instructions often increase self-focus at the expense of attention to other people. As a fix, he recommends that people practice meditation techniques that are grounded in sending happy wishes to others.
None of this is meant to say that mindfulness meditation is a bad thing, either for people in general or for specific individuals. I am a fan. My point is simply that there may be risks – and, studies showing small mean benefits may overestimate the benefits without acknowledging the possibility of those risks.
Conclusion
The strongest way of expressing my theme this week is that in intervention research, sole reliance on mean comparisons is not only misleading but irresponsible. We need to know how individuals respond to treatments. When individual-level data are missing, we can't tell from an apparently successful intervention whether many people improved a little, or a few people improved a lot, or something else – nor do we know how many people failed to improve, or how far they fell short of improvement. It's worth being "mindful" of this next time you hear about a new experimental study…
Sadly, most of the jokes I know about the arithmetic mean make fun of statisticians. You know…a statistician is a person who puts his head in an oven and his feet in a freezer and says "On average I feel fine." Jokes like this, apart from being pretty dumb, imply that statisticians are to blame for the limitations of the mean such as the one I described in this newsletter. In fact, historically, statisticians relied on the mean as part of the foundation for developing more advanced statistical concepts and tools. It's those of us who make use of their tools – i.e., researchers in social science, education, health, medicine, etc. – who sometimes rely too heavily on means as a source of information about groups.
Thanks for reading!
(Next week: Statistics for your Thanksgiving feast.)