Guardian Caps

Even if you know nothing about pro football, you've heard the stories.

Tua Tagovailoa collapsing on the field two weeks ago with his third diagnosed concussion. Hall of Famer Brett Favre revealing on Tuesday that he has Parkinson's disease. Links between football and neurological damage never quite stop making the news nowadays.

In this newsletter I'll be reviewing data on whether a new helmet technology protects football players' brains. But you don't need to know anything about the sport. As it turns out, the data reflects four of the most common ways that health statistics are misrepresented.

In short, the ultimate goal of this newsletter is to help you spot misleading health data, whether the topic is diet, exercise, medicine or anything else.

If you're a football fan – or a parent worrying about your kid taking up the sport – I will also share some conclusions about Guardian Caps. My take is that in spite of the statistical misrepresentations I'll be describing, the Caps do seem to provide a narrow, if not clearly measurable, range of benefits.

The problem

Over time, the NFL, like the tobacco industry, has been forced to acknowledge the dangers of its product after a long period of denial and suppression of evidence.

(Equating the NFL with Big Tobacco isn't meant as a condemnation. Successful industries just don't voluntarily undermine their own financial interests. The NFL is fabulously successful – if it were a country, it would rank 82nd in the world in net worth: $142.87 billion dollars as of 2023, slightly more than Iceland.)

The NFL's early denial of risk involved more than just press releases. In a series of academic papers published in the 2000s, members of the league's now-defunct Mild Traumatic Brain Injury Committee argued that no player has ever experienced chronic brain damage from repeated concussions. In a 2005 paper in the prominent journal Neurology, they wrote that “Professional football players do not sustain frequent repetitive blows to the brain on a regular basis."

(Of course they don't. They sustain frequent repetitive blows to the helmet. Also to the uniform. I hear their shoelaces take a beating too.)

Two decades later, denial is no longer viable. In the court of public opinion, the tragic deaths of Junior Seau, Aaron Hernandez, and other veterans have been attributed to the CTE revealed by their autopsies. (CTE, or chronic traumatic encephalopathy, is a neurodegenerative disease that undermines cognition, mood, and behavior.) In actual court, the NFL agreed to a $765 million dollar legal settlement in 2013 that supports retired players with neurological problems (the support seems far from sufficient though, as the Washington Post reported last month). Meanwhile, worrisome findings continue to accumulate, the most dramatic being a 2017 study that identified CTE in the brains of 99% of former NFL players. This wasn't a representative sample – only brains donated to science were included – but the study did show that the longer someone had played since high school, the greater the extent of damage to their brain.

By 2016, Jeff Miller, the NFL's top health and safety official, admitted that playing football is linked to degenerative brain disorders. This was the first time, after years of denial, the league acknowledged a connection. 

The solutions

Even before Jeff Miller's admission, the NFL had begun taking steps to mitigate head and brain injuries. The rules of play have been revised many times, including a controversial kickoff rule adjustment in 2024. The NFL's Concussion and Return-to-Play Protocols, which I'll discuss later, have been revised several times over the past decade. And there have been equipment changes, including new helmets, and, in recent years, the introduction of Guardian Caps.

Guardian Caps

"Guardian Cap" is the brand name of a soft-shell protective cover that can attach to a football helmet by means of snaps and Velcro. Each Cap weighs just under 7 ounces and is said to both absorb energy (by compressing slightly upon impact) and redirect energy (by reducing friction, and by moving slightly across the helmet when impacted). Guardian, the manufacturer, makes a variety of claims about the Caps' protective benefits.

Thousands of high school and college teams now use Guardian Caps on at least a limited basis. Since 2022, the NFL has mandated use among an increasing number of positions during practice, and in 2024, for the first time, the league allowed Guardian Caps to be worn during games. Keith Jenkins, writing for ESPN, provides further details on the Caps here.

Are Guardian Caps effective?

Kylen Granson, a tight end for the Indianapolis Colts, told the New York Times recently that he plans to wear a Guardian Cap this season. “I would be remiss in not taking every precaution that’s available to me,” he was quoted as saying on Instagram.

That strikes me as wise. In the midst of a dangerous activity, you want to take every precaution. And yet, not all precautions are effective. Fastening a mattress to the front of your car may reduce the impact of a collision, but not in every sort of accident, and not necessarily enough that you would notice. Fastening a mattress to the front of a train would almost never be helpful.

I'm going to discuss the Guardian Cap data under the heading of four principles we should always keep in mind when encountering health data:

1. Statistics are meaningless without context.

2. Statistical significance isn't the same as practical significance.

3. Good statistics can't save bad measures.

4. Statistical tests aren't meant to be one-offs.

Each of these principles is violated by at least some of the Guardian Cap data. You'll see some very common statistical misrepresentations in play here.

At the same time, there's evidence that Guardian Caps are beneficial in a narrowly-defined sense. I'll be as clear as I can about the benefits, particularly for the sake of parents and others worried about the well-being of the roughly 750,000 6 to 12 year olds who currently play tackle football.

1. Statistics are meaningless without context.

The NFL reported that the use of Guardian Caps during training camps in 2022 reduced concussions by 52% compared to training camp rates for the previous three years – 2018, 2019, and 2021, excluding the anomalous 2020 season. (During a league briefing, Jeff Miller alluded to a similar finding of 46% in 2023, but his remark was so vague that it's unclear what was compared, so I will focus on the 52%.)

Guardian has also made a number of claims about lab testing they've conducted, noting on their website, for instance, that the Caps reduce impact by up to 33% percent.

One kind of context that's missing here is the actual studies, including the full details of sampling and measurement, as neither the NFL nor Guardian has published their data.

Obviously both organizations hope the Caps perform well; we need access to the studies so that we can check for biases and any other potential problems.

For instance, I'd want to know why the NFL compared concussion rates for one year of Cap use to combined data for the preceding three years when Caps weren't used. Why not the preceding two years? Why not the preceding four or five? The NFL regularly updates its list of allowable helmets, and the league's Concussion Protocol was modified in 2022, so comparing 2022 data to 2018-2021 data might only show that better helmets and a more inclusive protocol reduced concussions. (I don't know if that's true; I'm just speculating. The point of seeing the published data is you don't have to speculate.)

Here's another example: Dr. Daniel Daneshvar, a concussion expert at Massachusetts General Hospital and Harvard Medical School professor, noted in an interview last week that the NFL study relied on players self-reporting concussion symptoms, but players may ignore or fail to recognize the symptoms. And, if they believe the Guardian Caps are protecting them, they may be disinclined to report a problem. This is another reason that the 52% figure may be misleading.

As for Guardian's data, saying that the Caps reduce impact by up to 33% percent doesn't mean much without knowing how they were tested. "Up to" is a hedge that allows for the possibility that in at least some situations, the Caps don't help at all.

I spoke with Dr. Daneshvar by phone this morning, and he noted that most of the published research in this area uses rigid-necked, helmeted dummies that are struck head on. The necks of actual football players aren't rigid to the same extent, and much of the damage to players' brains is known to come from impacts that cause rotational, left-right movements (think of a collision to one side of the helmet, causing the head to twist). Since Guardian hasn't published their data, we have no way of knowing whether they tested for these angular forces.

Let's assume anyway we can trust the statistics. Even if we do, what seems like good news from both NFL and Guardian doesn't mean much without broader context.

First, think back to that 52% reduction in concussions. Halving the rate of concussions among NFL players still leaves us with a dangerous sport, in part because every year, roughly 10% of NFL players are diagnosed with one or more concussions. If Guardian Caps reduce that figure to 5%, the yearly rate would still be much greater than we see in any other line of work, apart from sports like rugby, MMA, boxing, ice hockey, and, maybe, slap fighting. (Construction workers, for instance, have one of the most dangerous types of job in the US, but each year, only about 1% of them are injured badly enough that they take time off work. This is very much an apples-to-oranges comparison; my point is simply that even if that Guardian Caps halve the concussion rate, pro football will remain a dangerous profession by any standard.)

As for Guardian's data, since we don't know how they calculated a 33% reduction in impact, there's no context for evaluating the change. Is it neurologically meaningful? Is it only focused on one point of impact? One speed of collision? We have no way of knowing. (I reached out to the company to ask but haven't heard back.)

2. Statistical significance isn't the same as practical significance.

The biggest elephant in the room (perhaps it's a linebacker) is that even if you trusted the data, statistically significant differences don't necessarily have practical importance.

What is statistical significance? Statisticians don't agree on the details, but here's the gist: In a particular study, a significant effect is one that's unlikely to have emerged if it weren't genuine.

For instance, when the NFL claims their study showed that Guardian Caps significantly reduced concussion rates, they mean that the chances that the Caps don't actually reduce concussion rates are small. Ditto for Guardians' claims about significantly reduced impact forces.

Earlier I explained why there's insufficient context to evaluate the unpublished NFL and Guardian data. The Caps have also been the focus of a small number of published, peer-reviewed studies. I'll focus on the best and most recent one to illustrate the distinction between statistical and practical significance.

In this 2023 study, published in Annals of Biomedical Engineering, Nicholas Cecchi, Ashlyn Callan, senior author David Camarillo and colleagues at Stanford and USC subjected Guardian Caps to the most comprehensive lab testing to date, looking at the effects of variables such as point of impact, number of impacts, and ambient temperature. (I'm happy to report that mechanical "headforms" rather than people were used for these tests, as shown below.)

Here are four of the many findings:

• Guardian Caps did not consistently reduce head-on forces.

• Guardian Caps significantly reduced angular forces.

• Guardian Caps performed fairly well across repeated collisions and variations in temperature.

• The benefits of Guardian Caps diminished as speed of impact increased.

I'll say a bit more about the last finding, as it illustrates the problem of translating statistically significant effects into practical ones.

The researchers measured a variable called Head Acceleration Response Metric (HARM), which takes into account point of contact, extent of rotational force, and other kinematic details. The researchers found that HARM values were reduced on average by 25% at the slowest speed of impact (3.5 m/s) and 10% at the highest speed (7.4 m/s).

This is good news, but there's no way to translate these numbers into specific, real-world benefits. The study doesn't capture the full range of variability in speed of impact, location of contact (e.g., slightly vs. primarily on a facemask), or the rigidity of the player's neck at the moment of impact. More importantly, we don't know how numbers like these map onto actual neurological effects, either momentarily or in the long run. As Dr. Daneshvar emphasized in our conversation, problems like CTE represent the cumulative effects of a variety of head impacts incurred over a period of years.

All the same, Dr. Daneshvar predicted that any benefits observed in the lab would translate into actual reductions in harm, even if the extent of reduction is tiny. Other experts concur. Maybe then Kylen Granson is right to take every possible precaution.

It may seem like I'm contradicting myself here. On the one hand, I'm suggesting that we can't tell from the lab studies how much Guardian Caps benefit players. On the other hand, I'm suggesting that the Caps do have limited, specific benefits.

There's no real contradiction. The data suggest that at certain speeds and points of impact, a player's brain will experience less trauma if they're wearing a Guardian Cap. That's a statement about physical impact. What's not known is how much this reduction in impact will protect brain health, particularly after years of collisions on the field. Is the Guardian Cap like a mattress fastened to a car that's traveling slowly? Or is it like a mattress fastened to a bullet train moving at top speed? Hopefully time will tell.

3. Good statistics can't save bad measures.

If we can trust the NFL's unpublished data, Guardian Caps will halve the concussion rate. I've already noted that doing so wouldn't make football a very safe profession. Here I want to point out why the NFL can't even be sure the concussion rate was reduced in the first place.

According to NFL rules, a player goes into the Concussion Protocol if they report or show signs of a concussion. "No go" symptoms include loss of consciousness, amnesia, abnormal balance, dysfunctional speech, and so on. Further examination follows, and if a concussion is diagnosed at any point, a complex, five-step Return-to-Play Protocol must be followed before players can fully participate in play again.

These protocols, first implemented in 2013 and revised several times since then, are part of the reason Dr. Daneshvar feels that in recent years, pro football has become safer than it has been at any time in the history of the sport. From the perspective of player well-being, the protocols are useful. However, when treated as a source of data on concussions, the Concussion Protocol is inherently limited. Players may not report symptoms of a concussion because they're not aware of them, or because they don't want to be removed from play. The symptoms might not be detected on the field or during a sideline interview. Experts agree that most concussions end up going undiagnosed. To the extent that this occurs, we can't say for sure whether Guardian Caps are reducing concussion rates. Maybe they are, maybe not.

In our conversation, I asked Dr. Daneshvar how we will know someday whether Guardian Caps are effective, given their introduction at a time when there are also changes to NFL rules, new equipment (including helmets) and evolving Concussion and Return-to-Play protocols. Daneshvar pointed out that the NFL is in a "heads I win, tails you lose" position: If the number of concussions rises from one season to the next, the NFL can boast that the Concussion Protocol is increasingly successful at detecting the problem. If the number of concussions declines, the NFL can attribute this to better rules and equipment, including Guardian Caps, that are improving the safety of the sport. Neither outcome would lead the NFL to say that the Caps have failed.

Two more measurement issues:

(a) In the 2023 study described above, and in a study published this June with NCAA Division I players, specially designed, instrumented mouthguards worn by the players failed to record differences in head impacts when Guardian Caps were vs. were not worn. In both cases, the researchers described their data as "preliminary", which is a euphemism for "not as methodologically strong as it could be", but we can say at least that this approach to measurement could be better, in theory, than testing the Caps under lab conditions, or relying on somewhat subjective, difficult to implement, under-sensitive concussion protocols.

(b) In an editorial published in the British Journal of Sports Medicine this July, Daneshvar and co-authors discussed data from other studies on the lack of consistent relationships between the force of impacts and either the incidence or severity of concussions. Sometimes a lighter hit causes a concussion, while a more powerful hit doesn't. What Daneshvar emphasized in his conversation with me is that neurodegenerative diseases like CTE are the result of cumulative effects rather than concussions per se, and that those powerful but non-concussive impacts play an important role. Some damaging hits aren't even perceptible, even though tens if not hundreds of thousands of neurons may be affected. In short, it's great if Guardian Caps halve the rate of concussions, but it doesn't follow that doing so will alleviate the serious and apparently near-universal problem of CTEs among retired players.

4. Statistical tests aren't meant to be one-offs.

The inferential statistics used to test the benefits of Guardian Caps aren't meant to be one-and-done. We can't conclude much on the basis of a single study, or even a small number of studies. The more data we gather, the more accurately we can judge the benefits of the technology.

In the meantime, it's easy to be mislead by the specificity of individual study statistics. If we trust the NFL's 52% reduction figure, we can only conclude that Guardian Caps are effective. Had the NFL gathered data for a different time period, or included data from regular season play, the extent of reduction would've been different. Maybe higher, maybe lower, but almost surely not exactly 52%. We need more data.

Conclusion

In spite of the granularity of the published lab studies and the promise of the unpublished NFL and Guardian data, we don't really know how much Guardian Caps will benefit players. Physically speaking, the Caps seem to reduce impact to the brain under at least some conditions, but we don't know to what extent this translates into neurological benefits.

Meanwhile, the NFL has increased the number of positions required to wear Guardian Caps during practice, and as of this year the Caps are permitted for use during games. In an email to me, Keith Jenkins, who wrote about the Caps for ESPN, portrayed their use as a growing trend: "More and more players are wearing Guardian Caps and other relatively newer safety items on game day. I think that number will continue to increase as more information becomes available."

Unfortunately, that trend doesn't extend to all ages.

As Dr. Daneshvar pointed out, the NFL's mandated use of Guardian Caps for an increasing number of positions hasn't trickled down yet to the collegiate and high school level, where the vast majority of athletes play, even though there's a push for mandated use in some quarters. Younger brains are not only more vulnerable but also less able to appreciate risk and imagine what life might be like several decades down the road. The financial incentives can be especially distracting. As former quarterback Jordan Palmer told The Athletic, “It’s just hard at 24 to think about your life at 54 [w]hen the system is set up (for a player) to make a lot of money in a short amount of time..."

Part of the solution, in Daneshvar's view, is to delay contact as long as possible. He remarked that no professional player he'd ever spoken with mentioned that contact football at an early age had drawn them to the sport. As currently played though, contact becomes inevitable sooner or later. It's hard to imagine high school football without it.

In the end, I think that players, parents, and anyone else with skin in the game have to address the complex and uncomfortable question of whether the pursuit of a few years of glory and, in the rare case, fabulous wealth, is justified by the risk of neurological impairment for decades after the glory days have faded. This is ultimately a question about values – about how to live one's life. It's easier to articulate this question than to answer it, so I will simply conclude by suggesting that the best we can do for young football players is to encourage them to confront the question openly. Technologies like Guardian Caps can be beneficial, but not to the point that questions like this can be safely avoided.

Thanks for reading!