Stats in Ads: Is the SNOO for you?
This week's newsletter focuses on a specific product for infants. For readers who have little interest in these (sometimes) adorable creatures, my broader purpose is to share some remarks on the use of statistics in advertising. (I call them "remarks" because the topic is too big to treat comprehensively here.)
People often rely on statistics for credibility. Scientists, politicians, journalists, sportscasters....almost anyone labeled an "expert" will trot out statistical factoids from time to time in support of some claim. Same goes for writers of ad copy.
If you're a parent you're probably overwhelmed by the amount of gear that can be purchased for little ones. Ads for these products try to persuade you of their necessity – your kid will benefit if you buy them but lose out if you don't – and to reassure you that they're safe.
In what ways do you need to be cautious about the statistics you find in these (or any) ads?
Consider the SNOO. That is, the SNOO Smart Sleeper Bassinet, which is generating a lot of buzz right now. Swaddle your baby in a SNOO Sack, clip your baby to the SNOO bassinet, turn the thing on, and without further ado, the SNOO soothes the baby with rocking and white noise. What makes the SNOO "smart" is that its rocking and white noise automatically change in response to the baby's crying. (For more details, see here). Happiest Baby, the manufacturer, makes a lot of claims about its effectiveness. Here are three from SNOO's website:
Claim 1: The SNOO "teaches babies to self-soothe with little need to cry-it-out".
Claim 2: The SNOO "often calms crying in under a minute".
Claim 3: The SNOO "Adds 1–2 hours of sleep per night".
Exciting claims! No wonder new parents, exhausted and perhaps desperate, might be willing to spend so much money on a bassinet ($1,495, not including tax and shipping).
Let's have a look at each claim.
Claim 1 is questionable, though not for statistical reasons. The SNOO may soothe, but it's unlikely to teach anything. The baby might learn to associate the SNOO with pleasant sensations (which is a very, very good thing at that moment you're clipping the baby in and praying for sleep), but I think there's zero chance that it teaches emotional self-regulation. This illustrates a problem with ads that we're all familiar with: Some ads contain unsubstantiated claims, and some of these claims turn out to be false. Statistics often support the falsehoods. Famous example: Reebok's widely advertised claims that two of their shoes confer 11% more strength and 28% more ass-sculpting than regular shoes, statistics that were not supported by credible research. In 2011, the FTC forced Reebok to pay $25 million in refunds (see here and here).
Claim 2 ("often calms crying in under a minute") is probably true, because the word "often" is vague, as is the phrase "calms crying" – does that mean less crying, or none at all? Either way, picking up a crying baby "often" calms them in under a minute, as does singing or speaking to them. They "often" calm down by themselves in under a minute. In other words, you could accept Claim 2 without giving any special credit to the SNOO. At minimum, one would want to see data showing that the SNOO calms crying more quickly and/or more fully than other approaches (or just doing nothing).
My concern about Claim 2 applies to statistical factoids in ads and pretty much anywhere else/ Broadly, the concern is that the statistics are meaningless without context. There are many kinds of context, including other statistics. For example, the infant mortality rate in the U.S. is 5.7 deaths per 1,000 live births (according to the most recent CDC data), but that figure doesn't mean much out of context, other than predicting that most American infants will live past their first birthdays. That's a happy thought, but additional statistics provide important context with less happy implications. Two examples:
More than 50 countries have lower infant mortality rates than the U.S. (details).
In the U.S., the infant mortality rate for African-American babies is over twice as high as it is for Caucasian and Asian babies (details).
In short, infant care in the U.S., particularly for some groups, is not as "good" as one would expect in such a wealthy nation. This is important context for the 5.7-in-1,000 figure. In light of other statistics, that 5.7 doesn’t look quite so happy.
Back to the SNOO. I've argued here that Claims 1 and 2 are problematic. How about Claim 3 ("Adds 1–2 hours of sleep per night")? When statistics are quoted, you (should) expect them to be grounded in research. Where is that the research?
Well... according to a news release from Happiest Baby, a study showed that "SNOO increases sleep 1–2 hours per night. On average, these babies woke up one time less per night." That's two findings (increased duration of sleep; reduced number of wakings) and it sounds like good news, but no other details are given. What you can find, on the SNOO website, are two graphs from the study, which I've pasted in below.
The graphs seem to show that the SNOO outperforms traditional beds/bassinets for longest sleep period, and for total time spent asleep (nothing is presented about wakings), but I don't see enough information to judge one way or the other.
First, "sleep" is defined as something that happens between 7 p.m. and 7 a.m. We also need to know what's happening at other times. For example, if the SNOO improves sleep at night but the babies are more wakeful and fussy during the day, it might not be a good investment. (I'm not predicting this; I'm just saying more data are needed.)
Second, without further details about methodology, one can think of many reasons why the findings might not be credible. A gazillion, to be specific. One that Dr. Emily Oster points out here is that it's unclear who the SNOO babies were compared to. All we know is that numbers were drawn from other studies to create comparison data ("Average expected sleep based on literature"). Unknown - but important - is the extent of similarity between SNOO babies and the comparison group, as well as the quality and sample sizes of the studies drawn from the literature, etc.
Again, I'm not saying we should doubt the findings. I'm just saying there's no reason to trust them.
(Even if we did trust mean differences between groups, here's a practical concern: No information is given about variability. As a parent, I’d want to know about that. If the distributions don't overlap – e.g., if all babies whose used SNOOs slept longer than all babies who didn't – the SNOO would be a much safer buy than it would be if there were overlap, in which case you might turn out to be one of the unlucky parents whose baby sleeps even less with the SNOO than without it (even though averaged across all babies, SNOO-use is tied to longer sleep).)
Now for something more troubling: An abstract. Not an entire study; just an abstract published in the journal Sleep. Here we learn that 7,137 SNOO users were compared to "a reference group of a compilation of 13 peer reviewed studies of normative sleep data on sleep in traditional cribs/bassinets." (Again, no info is given about the reference group or study methods.) The abstract notes that SNOO users significantly outperformed the reference group on maximum uninterrupted sleep at night, sleep duration, and night wakings, but the only specific result given is that "Babies in SNOO averaged one less waking per night compared to the reference population (1.09 ±.89 vs 1.89 ±1.1)." To see a bit more data, you have to look at those graphs only available on the Happiest Babies website (and which themselves are missing much detail).
So...should we care that we have to get part of the data from the manufacturer's website, and part of the data from an abstract - and there's still so much missing information? Should we care about the concerns I've raised in this newsletter, statistical and otherwise? Should we care that one of the study's authors turns out to be none other than Dr. Harvey Karp, bestselling author and inventor of the SNOO?
Well, yes. But let's get to the bottom line: Should parents spend $1500+ on a SNOO, or buy something more conventional (and cheaper)? I don't know. For one thing, I'm not sure the SNOO doesn't live up to its promise. I'm only sure I can't find any credible evidence that it does. As the saying goes, absence of evidence doesn't equal evidence of absence. Plus, I feel very sympathetic toward desperate, sleep-deprived new parents. If there's a chance that the SNOO works – surely there must be, at least for some babies – then perhaps it's worth the $1500+ gamble, if you have the money.
Bottom-bottom line: Would I buy a SNOO? Heck no. Here's why not:
1. If a product's effectiveness can be clearly defined (e.g., X more hours of sleep per night), and ads claim that research has demonstrated its effectiveness, the research should be available (and hold up to scrutiny). As I mentioned, very little info about the SNOO study is available. (I contacted SNOO's customer support team to inquire about studies on the effectiveness of the SNOO. Their response: "Unfortunately, we are not able to provide scientific studies or research conducted for SNOO. However, SNOO meets or exceeds all governmental regulations for infant safety.")
2. As Emily Oster points out here, separate pre-SNOO studies have shown that much of what it provides (i.e., swaddling and white noise) does improve infant sleep. So, even if the SNOO is beneficial, it's hard to say whether the benefits exceed what can be provided for a lot less money. Again, this isn’t to say that the SNOO is ineffective or a bad buy. It's just not clear that it outperforms cheaper options.
3. The product reviews aren't what I would want to see. The reviews are generally positive (see Emily Oster's discussion), but, looking at the 2,368 reviewers on the Happiest Baby website, 31% ranked the SNOO less than 5 stars, and if there's any theme that links the comments of parents who gave ratings from 1 to 4 stars, it's the statement or implication that their baby is unique. I think they're right about that. If the SNOO is effective, it's probably more effective with some babies than others. Which is fine, but for $1500+, I’d want reassurance that it's effective with almost all babies – more effective than other, less expensive approaches one could take. The stats don't support that conclusion, and nor do the product reviews.
In sum, no SNOO for me....but it might(?) work for you.