The Alzheimer's Drug Debacles
If you need to take medicine – and you can afford it – America is, in theory, a great place to live. Since 1962, all drugs must be extensively tested for both safety and efficacy before receiving FDA approval for sale, and summaries of testing data are freely available on the internet. We now enjoy unprecedented rigor and transparency in drug testing.
At the same time, we take a dim view of the companies that make our drugs. In fact, a 2019 poll found that the pharmaceutical industry is perceived more negatively than any other sector in the U.S. (right below the federal government). Americans are unhappy about the costs of their medicines, as well as Big Pharma's role in the opioid crisis, its lobbying efforts, and its most repugnant players (the Sackler family, Martin Shkreli, etc.). As Bernie Sanders put it, we may be a divided nation, "but we all hate the drug companies."
Unfortunately, Big Pharma's most visible shortcomings sometimes overshadow less obvious problems with the data on drug efficacy and safety. In this newsletter, I'll be talking about two of the newest drugs for treating Alzheimer's disease, one approved by the FDA last year, the other expected to be approved tomorrow. The data in both cases is so weak that approval of the drugs seems scandalous. Because my focus will be on calling out those weaknesses, the substance of this newsletter may feel gloomy, but at the end I'll share some reasons for hope.
What's at stake
Alzheimer's disease is progressive, fatal, and currently affects over 6 million Americans. This number is expected to increase to 13 million by 2050. Currently there's no known cure; existing treatments focus on reducing symptoms and are minimally effective.
In June 2021, the FDA approved the use of Aduhelm (aducanumab) for treatment of Alzheimer's disease, and approval is expected tomorrow for lecanemab. These drugs are the first that allegedly treat the underlying disease rather than simply alleviating symptoms.
New report, old controversy
One week ago today, the U.S. House of Representatives released the report of a joint committee investigation into the 2021 approval and marketing of Aduhelm. In this report the FDA was faulted for its irregular review process, and Biogen, the drug manufacturer, was faulted for an overly aggressive approach to marketing and pricing. Much of the content of the report was covered in the news last year. However, one of the topics that received very little attention, either in the report itself or in media coverage of it this week, is the quality of the actual data on Aduhelm's efficacy and safety.
Why care now about the data? Given that rigorous testing was required for FDA approval, shouldn't we feel reassured that Aduhelm is effective and safe? Sadly, no, the data do not clearly demonstrate either. In the Appendix I describe how the FDA came to approve Aduhelm anyway, and why their decision created so much controversy. Here I'll focus on the data.
Aduhelm trial methods
The FDA initially reviewed two experimental trials on Aduhelm.
Methodologically, the two trials were almost identical. Each looked at a separate sample of over 1,630 patients with early Alzheimer's disease who received either a high dose of Aduhelm, a low dose of Aduhelm, or a placebo. (Participants and experimenters were blind to what each participant received.)
The experiment lasted for 18 months. The primary outcome was changes in the severity of dementia, as recorded by a measure called the Clinical Dementia Rating Scale–Sum of Boxes (CDR-SB). The CDR-SB is based on clinical examination of the patient and family/caregiver interview; scores tells you how well the patient is functioning cognitively (in memory, orientation, and problem-solving) as well as in daily activities (community affairs, home/hobby activities, personal care). Total CDR-SB score ranges from 0 to 18, with higher scores indicating greater severity of dementia.
The results of these trials provided no clear evidence of effectiveness, but they did raise concerns about safety. (Again, this is a drug that has already received FDA approval.)
What was wrong with the data?
1. Only one of the two trials demonstrated effectiveness.
Everyone, including the FDA, agreed that one of the trials was clearly unsuccessful. Patients in that trial who received either dose of Aduhelm were not statistically distinguishable from the placebo group in CDR-SB score changes across the 18 months. That is, all three patient groups showed worsening symptoms, and the extent of decline was not affected by Aduhelm.
In the other trial, the low dose of Aduhelm had positive but nonsignificant effects, while patients who received the high dose showed an absolute difference from the placebo group of 0.39 points on the 18-point scale, a tiny but significant difference. In other words, symptoms of dementia increased in both groups, but the extent of increase was slightly less for the high-dose group.
In sum, one trial showed that a high dose of Aduhelm is ineffective, while the other trial demonstrated marginal effectiveness. The FDA initially concluded that Aduhelm is effective at reducing dementia. However, as FDA advisory committee members and many others have pointed out, it doesn't make sense to selectively favor one finding over the other. This amounts to selectively ignoring half of the data. As Dr. Joel Perlmutter, a former member of the FDA advisory committee put it, in a January 3 e-mail to me:
"It is like seeing bullet holes on road signs on the highway and all of the bullet holes are in the center of [targets on the signs]. That does not indicate good shooting if the targets were drawn after the bullets were fired."
Biogen offered post hoc explanations for the inconsistent data, but none of them were remotely plausible (see here for details).
2. The "successful" trial wasn't very successful.
In the "successful" trial, the difference between the high-dose Aduhelm group and the placebo group was 0.39 points on the 18-point dementia scale. Experts differ somewhat in how to view that difference, but for the most part their opinions range from "no clinical benefit" to "no clear clinical benefit". In everyday terms, this means, in effect, that 18 months from now, someone with mild Alzheimer's disease will seem essentially the same regardless of whether they take a high dose of Aduhelm or nothing at all.
This, by the way, is a sad example of how statistical significance is sometimes confused with practical significance. Statistical significance, in this case, merely provides evidence that the groups differ, not that they differ to any important extent.
3. Long-term effectiveness was not examined.
No data were available beyond 18 months. There's no particular reason to expect that at some later point, Aduhelm would suddenly kick in and begin to substantially delay the progress of dementia.
The surrogate evidence
This is the point where, from a data perspective, the story gets especially ugly.
Earlier I noted that the FDA granted Aduhelm accelerated approval, a special category that allows a drug to be sold while additional testing is carried out. This type of approval is granted when effectiveness is not demonstrated for a clinical outcome (in this case, reduction of dementia) but rather for a different variable that predicts the clinical outcome (in this case, a physical change in the brain). Here's how the FDA explains it:
"The FDA instituted its Accelerated Approval Program to allow for earlier approval of drugs that treat serious conditions, and fill an unmet medical need based on a surrogate endpoint. A surrogate endpoint is a...measure that is thought to predict clinical benefit but is not itself a measure of clinical benefit."
Biogen originally sought regular approval for Aduhelm, which required, among other things, evidence that it reduces dementia. But since the data I described earlier was so weak and universally panned by the FDA's advisory committee (see Appendix below), the FDA was eventually forced to consider a surrogate variable: reduction of amyloid plaques in the brain. (As Alzheimer's disease progresses, these plaques are among the proteins that accumulate in and around neurons, disrupting brain function.) Based on a single study demonstrating amyloid reduction, the FDA granted accelerated approval.
Well, you might be wondering, what's wrong with that? If Alzheimer's disease is caused by a build-up of amyloid plaques and other amino acids, and Aduhelm can reverse or at least slow that accumulation, then the drug must be effective, right?
Sadly, no. Amyloid build-up is associated with the progression of Alzheimer's disease, but it's not clear to what extent this is a symptom as opposed to a cause of the disease. Moreover, amyloid reduction doesn't automatically translate into reduced dementia. There have been more than 25 experimental trials looking at this, and, as succinctly put in a JAMA Internal Medicine article last year,
"Plaque reduction has not been a reliable marker for cognitive function in past trials. Not everyone with plaques has or will get Alzheimer disease, and other plaque-reducing medications have not shown meaningful patient benefit."
Surprisingly, the FDA's own guidance for drug manufacturers, drafted in 2018 and still current, includes the following statement:
"There is no sufficiently reliable evidence that any observed treatment effect on such biomarker measures [beta-amyloid] would be reasonably likely to predict clinical benefit.”
This statement suggests that by granting accelerated approval to Aduhelm, the FDA contradicted its own guidance on surrogate evidence. Yes, Aduhelm may reduce amyloids, but there's no clear evidence that amyloid reduction leads to clinical benefits (i.e., reduced dementia).
And there's more...
The labeling issue
When the FDA granted accelerated approval, they approved the labeling of Aduhelm for all patients with Alzheimer's disease. However, the clinical trial data reviewed by the FDA was obtained from a subset of Alzheimer's disease patients – i.e., those in the early stages of the disease, with mild symptoms. Without prompting from Biogen, the FDA approved much broader use of the drug than the data warranted. Here's a telling excerpt from the House committees report from last week:
"Internal Biogen documents indicate that Biogen’s Alzheimer’s team leaders expressed concern the company could lose credibility by advocating for a broad label indication that exceeded the clinical trial population, and the company even developed a communications strategy to deal with the anticipated fallout. Despite these concerns, Biogen appears to have been unwilling to disagree with FDA and accepted the broad label indication statement initially proposed by FDA, and only sought a label update to the usage and indications section to clarify the appropriate patient population after public backlash."
In other words, Biogen itself didn't want such broad labeling, and they only asked for a correction after experts and others expressed outrage about the FDA's decision.
Aduhelm: Summary of evidence
Aduhelm does not reduce dementia. When evidence of its effectiveness was unpersuasive, the FDA shifted to a surrogate for effectiveness that's not suitable, then approved Aduhelm for use with a broader population than it was tested on. Approval was granted in spite of the unanimous objections of an expert advisory panel, which was not consulted about the shift in criterion for approval (see Appendix below). I haven't even touched on the safety data – a whole newsletter could be written on that topic alone. Suffice to say that Aduhelm routinely causes swelling and bleeding in the brain that is ordinarily easily managed (e.g., by dose reduction), but experts have raised concerns about more serious adverse effects, including fatalities.
In sum, as Dr. Perlmutter put it in his January 3 email to me, Aduhelm has "clear side effects and no clear clinical benefit."
The FDA's defense
In a statement released last year, the FDA, to its credit, directly addressed the concerns raised in this newsletter. However, this statement was, in many places, a masterpiece of deceptive language. For example:
"The data included in [Biogen's] submission were highly complex and left residual uncertainties regarding clinical benefit...As is often the case when it comes to interpreting scientific data, the expert community has offered differing perspectives."
I am unwilling to be polite or nuanced about this statement. It is profoundly deceptive.
First, yes, of course the data itself was complex, but the limitations of the data were not. As I've shown here, you don't need a PhD in biochemistry to recognize those limitations. We all know you shouldn't ignore half of your data just because you like the other half better. Or characterize tiny, clinically meaningless differences as evidence of success. Or ignore 25+ studies that fail to validate the surrogate you use for effectiveness.
Second, yes, some experts viewed the data as leaving residual uncertainties about the clinical benefits of Aduhelm. But others were more certain that the data showed no clinical benefits. In other words, expert opinions ranged from "we're unsure whether this works" to "we see no evidence that this works". These are indeed "differing perspectives", but they don't include anyone saying "this works". As far as I can tell, the only people who consider Aduhelm effective either have a financial interest in saying so, or they work for the FDA.
The FDA statement continues:
"At the end of the day, we followed our usual course of action when making regulatory decisions in situations where the data are not straightforward."
This is false, according to any reasonable definition of the word "usual". As noted in the House committees report from last week, the FDA itself acknowledged flaws in its decision-making process, including deviations from its own protocols and collaborating too closely with Biogen. As for the shift to accelerated approval, the FDA statement acknowledged doing so without consultation with its advisory committee, but no explanation is provided. Instead, the statement insists that Aduhelm reduces amyloid build-up, and that "[t]his reduction in plaques is reasonably likely to result in clinical benefit". According to the experts, this is not what the data shows. If anything, it's reasonably unlikely to produce clinical benefits. Reduction in amyloid plaques may be a good thing, but there's no clear evidence that it would delay the inevitable worsening of dementia.
Finally, because accelerated approval requires additional testing, the FDA's statement contains multiple assurances that Aduhelm can be removed from the market if it turns out to be ineffective. This is not very comforting, given that Biogen has until 2030 to complete the results of additional testing.
The lecanemab trials
The FDA has indicated that it will decide tomorrow (January 6) whether or not to grant accelerated approval to lecanemab.
On November 29 of last year, Biogen and Eisai, the co-manufacturers of lecanemab, released the results of their Phase 3 experimental trial to the general public in the form of a detailed summary as well as a conference presentation. A peer-reviewed report is also available today (behind a paywall) in the New England Journal of Medicine.
In many respects, the methodology of this trial was similar to that used in the Aduhelm trials. 1,795 people with early Alzheimer's disease either received lecanemab or a placebo for 18 months. (The dosage of lecanemab was roughly comparable to what the high dose group received in the Aduhelm trials.) Once again, the primary outcome was changes in scores on the CDR-SB dementia scale mentioned earlier. In addition, for subsets of 257 and 281 participants, surrogate variables (biomarkers of Alzheimer's disease) were measured.
The main evidence of lecanemab's effectiveness also resembles what was found for Aduhelm: By the end of the trial, patients who received lecanemab differed from the placebo group by 0.45 points on the 18-point dementia scale, a tiny but significant difference (and remarkably similar to the 0.39 difference observed for Aduhelm). Once again, the increase in dementia over the 18 month trial was slightly less overall for the lecanemab group than for the placebo group. The surrogate data also suggested that lecanemab slowed or ameliorated the neurophysiological damage associated with Alzheimer's disease.
What's wrong with the lecanemab data?
The lecanemab data suffers from most of the same limitations as the Aduhelm data, including a very small, probably clinically meaningless effect for dementia, as well as reliance on surrogate variables that prior research has not clearly linked to reductions in dementia. As with the Aduhelm data, there was no credible analysis looking at the obvious question of associations between effects on dementia and amyloid changes. The wording of the Biogen/Eisai reports raises further issues.
Statistical semantics
As in the Adulheim data reports, Biogen and Eisai mostly present their effectiveness data as percentages. For instance, they refer to a "27% slowing of decline" for the lecanemab group compared to the placebo group, which is accurate, mathematically, but may create a more favorable impression than simply describing the absolute difference between the groups (0.45 points on an 18-point scale). Given that media reports have typically cited the percentages, I think that in order to prevent unwarranted optimism, the companies – and the FDA – should primarily refer to the absolute differences rather than the percentages, instead of the other way around.
Also, Biogen and Eisai refer to their strongest findings as "highly statistically significant", a phrase that some people (including me) would object to in a context like this. Statistical significance is a kind of evidence that the effect you're looking at (e.g., a difference between groups) is genuine and would be observed in the broader population (e.g., among all Alzheimer's disease patients). A "highly" statistically significant finding might give you much assurance that the effect is genuine, but it doesn't mean that the effect is large. For the lecanemab data, if the effect is indeed genuine, it's a very, very small one.
Finally, as with Aduhelm, safety concerns have emerged in the lecanemab data. For instance, yesterday a third death was reported among lecanemab trial patients due to cerebral hemmorhage. Investigations into these incidents are ongoing.
Conclusion: Statistics, lack of consideration, and hope
In the end, what disturbs me the most, apart from the weakness of the data, is not procedural flaws at the FDA or Big Pharma greed. Rather, it's what feels like a lack of consideration for what Alzheimer's patients and their loved ones must experience. Across all of the data that were gathered, if you ignore the safety risks, the study that didn't show benefits, and the untenable surrogate evidence, the only good news with respect to the lived experience of the disease is that in one study, Aduhelm led to a 0.39 point less of an increase in dementia on an 18 point scale, while in another study, lecanemab led to a 0.45 less of an increase on that scale. How would those differences seem to patients and those around them?
For guidance on this question, I reached out to Dr. David Knopman, a Mayo Clinic neurologist and former member of the FDA advisory committee that had provided expert guidance on the review of Aduhelm data. (The main topic of our January 3 e-mail discussion was patient experience, broadly speaking, and did not pertain to Dr. Knopman's views on Aduhelm, Biogen, or the FDA review process.)
Dr. Knopman pointed out that since what's at issue is not improvement or even stabilization, what statistics like the 0.39 or the 0.45 could only show is a delay in the inevitable worsening of dementia. He went on to note:
"Neither patients nor family members nor physicians can possibly appreciate a delay in decline in a disease like AD, because the point of comparison is hypothetical (ie where the patient would have been at the followup point, which of course is unknowable)."
Dr. Knopman is not saying that a delay in decline is undesirable, but simply that it's not appreciable. What you feel or observe over time is a worsening of symptoms; you don't know how much taking (or not taking) medication would change the rate at which they worsen.
This provides some context for the cautious but enthusiastic statements about Aduhelm and lecanemab that the FDA and Biogen keeping putting forth. Drugs like these, if effective, would be desirable, because patients and their loved ones would benefit. But it seems irresponsible to assure the public of those benefits when, (a) they're tiny, at best, and may not even exist, and (b) they won't be directly perceived.
This newsletter has rather gloomy, but I believe there are reasons for hope. The fact that Aduhelm and lecanemab can slow or reverse amyloid accumulation represents a breakthrough in treating the neurophysiological damage associated with Alzheimer’s disease, even if these drugs don't impact the lived experience of the disease. Drugs that cure or at least delay the onset of Alzheimer's disease may likely build on what has been learned from the current trials. The fact that the FDA and Biogen/Eisai seem over-eager to get new drugs on the market tells us that a mix of genuine compassion and corporate greed will continue to drive the search for new treatments that better support patients. Meanwhile, federal funding for Alzheimer’s disease research increased by $226 million in 2022, bringing the annual total to over $3.7 billion. For these reasons I have hope that the years to come will bring dramatic progress.
Thanks for reading!
Appendix: The Aduhelm approval controversy: An abridged look
In March 2019, after more than three years of research, Biogen actually stopped all testing of Aduhelm after an independent review panel conducted a planned, preliminary analysis and judged it unlikely that the data would meet their primary goal of demonstrating efficacy.
In other words, the data were so weak, it seemed unlikely that further data collection would reveal benefits. Nonetheless, by June 2019, the FDA and Biogen agreed that re-analysis of the existing data was needed on technical grounds, and a series of collaborations between the two organizations ensued. The House committee report published last week criticized the FDA for violating its own protocols on these collaborations, which, long story short, reflected an inappropriately chummy relationship between the FDA and the drug company. (One of the revelations in the House report is that the FDA itself had conducted an internal investigation and found similar problems.)
On November 6, 2020, the FDA convened its Peripheral and Central Nervous System Advisory Committee (the advisory committee I refer to in this newsletter) to discuss the evidence supporting Aduhelm, including the re-analyzed data mentioned above. None of the 11 experts on this committee recommended approval. (10 voted "against"; 1 voted "uncertain".) Nonetheless, on June 7, 2021, the FDA granted accelerated approval to Aduhelm, based on the surrogate data, without further consultation with the advisory committee. Committee members were outraged; three resigned in protest. Former committee members and other experts aired their concerns in a variety of academic and non-academic venues. The weaknesses of the data, and the apparent lack of objectivity on the FDA's part were so striking that organizations ranging from Mount Sinai Health System to the Department of Veterans Affairs chose not to offer Aduhelm, and Medicare refused to cover the drug unless someone was taking it as part of an experimental trial. Biogen meanwhile cut the price of the drug in half (to $28,000), in a so-far failed attempt to garner support. At present, after laying off Aduhelm commercial staff and engaging in other cost-cutting measures, Biogen is primarily focused on continuing the next round of clinical trials for the drug.