"People are, in fact, fundamentally good." (Airbnb founder and CEO Brian Chesky in a message to employees.)
Ok, people are good. What about Airbnb?
I've used Airbnb as a host as well as a guest, though I'm aware that since the company's launch in 2008 it has been called out for driving up rents, promoting discrimination, and creating nuisances for locals (see here, for example).
Airbnb has been linked to crime too, usually through media reports on horrific incidents. The company remains secretive about internal crime statistics that might give the public a broader, more balanced perspective. Airbnb does state that fewer than 0.1% of stays lead to reported safety issues, but that's not the most reassuring stat you'll ever read, given that by 2019 the company was handling over 200 million bookings a year. (0.1% of 200 million is 200,000. And sure, not all "safety issues" pertain to crime, but still...that's a lot of issues.)
Owing in part to Airbnb's reluctance to share data (see here), the first study showing a causal link between Airbnb and local crime rates appeared only two weeks ago. As you can imagine, the company's public response was not enthusiastic.
I want to talk about this study as well as Airbnb's rebuttal, because the rebuttal illustrates a classic mistake in how to critique statistics.
In this study, published in the renowned journal PLOS ONE, Northeastern university researchers Laiyang Ke, Daniel O'Brien, and Babak Heydari examined the relationship between Airbnb listings and crime in Boston neighborhoods between 2011 and 2018. The researchers considered two possible scenarios:
Scenario 1: Greater Airbnb presence leads to more neighborhood crime, because tourists who use Airbnb are targeted by criminals, and/or because some tourists themselves engage in criminal activity.
Scenario 2: Greater Airbnb presence leads to more neighborhood crime, because greater transience among residents erodes social norms and relationships that help prevent crime. In other words, as Airbnb listings increase, fewer residents will be consistently present to cooperate with each other, keep each other informed, etc.
Although these scenarios aren't exclusive, here's how the researchers assumed they can be distinguished:
When Scenario 1 occurs, the connection between Airbnb presence and crime will be more or less simultaneous. For example, if Airbnb listings increase in 2016, crime rates will increase during the same year.
When Scenario 2 occurs, the impact of greater Airbnb presence will not be immediate. For example, if Airbnb listings increase in 2016, crime rates will begin to increase in, say, 2017 or 2018, because the erosion of social ties that prevent crime will be gradual.
Briefly, the researchers' data mostly supported Scenario 2, with clearest effects for violent crime (as opposed to private conflict or public disorder) that began to emerge a year later for each year studied. In other words, from 2011 through 2018, an annual increase in Airbnb listings within a Boston neighborhood was linked to an increase in violent crime in that neighborhood that only emerged a year later and then persisted over time (at least for the duration of the study). Much more could be said about the findings, but this should be sufficient context for Airbnb's response.
On their News blog, Airbnb published a rebuttal which begins with the admirable but dubious claim that "Airbnb supports rigorous research and analysis to help create shared learnings about the impact of short-term rentals on neighborhoods and communities."
I call this claim "dubious" because law enforcement officials as well as scholars have struggled to gather data from Airbnb that would clarify its impact on local crime (see here for examples). I'm not necessarily faulting Airbnb for this – they withhold data in part to protect the privacy of individuals who use their service. It just seems wrong, or too simple, to say that the company "supports" research.
Airbnb's first substantive criticism of the Northeastern study is that "it uses an unrepresentative sample within one city to make broad nationwide conclusions." To be honest, before I could even think about the merits of this criticism, I was disturbed by the ethics of it. The study is grounded in virtually all available data from Boston, a major U.S. city with a population of nearly 700,000. Even if Boston weren't representative of U.S. cities, shouldn't Airbnb management be concerned that their business is linked to crime in even one city of that size? (At least for the sake of PR, shouldn't they at least sound concerned?)
As for the substance of the criticism, I see no special reason to question the representativeness of this sample. The two scenarios mentioned earlier are drawn from scholarship that applies to any urban setting. If Airbnb promotes crime in Boston, but you want to claim that it won't promote crime in LA or Detroit, you need some sort of rationale for that claim. Airbnb offers no rationale. (Analogy: Scientists in Boston who study brain function sample local residents. If you want to claim that Boston residents aren't representative – i.e., that you can't generalize what you learn about their brains to other Americans – then the burden of proof is on you, because there's no particular reason to expect regional differences in brain function.)
Don't get me wrong: Replicating the Northeastern study in other large cities would make the results more credible (as the researchers themselves acknowledge). My point is simply that there's no obvious reason to doubt their generality. What Airbnb does here – as they do throughout their rebuttal – is to speculate about a problem that's possible in theory, but for which there's no particular evidence.
Airbnb's next criticism is that the study doesn't control for all potentially relevant variables, like new housing construction and overall economic conditions. The gist of the criticism is that if Airbnb growth is correlated with an increase in crime, the correlation may be explained by a third variable. For example, as the local economy declines, more people list their properties on Airbnb, and more people become motivated to engage in crime. In this case we wouldn't say that Airbnb growth causes crime. Rather, a declining economy spurs Airbnb growth, and, through a separate process, the declining economy promotes criminal behavior. This is a classic example of a "third-variable problem".
Airbnb's third-variable criticism makes sense, in theory, it's expressed in a speculative, vague way. Airbnb offers no evidence or rationale for how any third variable could explain the Airbnb-crime association. (After all, it's also possible, in theory, that the New York Yankees' performance is stimulating Airbnb growth and, separately, promoting crime in Boston, but since there's zero evidence for either possibility, it doesn't seem worth considering.)
The researchers did control for variables such as median household income, but Airbnb complained that this variable was based on self-reported five-year estimates, while the main variables in the study were measured annually. I think you could argue that a five-year estimate of income is actually better than a yearly one, because the whole purpose of controlling for income was to filter out any influence it might've had on the main variables (e.g., crime rates). If low household income promotes crime, then struggling financially over a five-year period may have a stronger impact than struggling financially during a one-year period. Maybe, maybe not. In any case, Airbnb provides no rationale for their claim that income was badly defined, other than playing the child's game of saying that five and one don't match.
As I've mentioned, the Northeastern researchers concluded that "the prevalence of Airbnb listings erodes the natural ability of a neighborhood to prevent crime." That's a strong statement. The researchers aren't just saying that Airbnb is "associated with" or "linked to" crime rates. They're saying that Airbnb causes changes in neighborhoods, and one consequence of these changes is greater crime. In response, Airbnb claimed that the researchers used the wrong statistic to establish causality. (No worries if you don't have a background in stats – this won't be a technical discussion.)
According to Airbnb, the statistic that the researchers used, the Granger causality test, "is meant to establish whether one event or action sequentially precedes another – not whether one causes the other." There's a lot wrong with this statement.
1. You often don't need fancy statistics to establish whether one event precedes another one. You just look. Changes in Airbnb listings from 2015 to 2016 clearly precede changes in crime rates from 2017 to 2018. (Duh.)
2. The statistic in question is indeed meant to establish that one variable causally influences another one, regardless of how you define "meant". That is, Nobel Laureate Clive Granger invented his eponymous method in order to establish causality, and many researchers, including the Northeastern team, use the Granger method (or the logic of it) to help establish causal relationships.
3. Granger's method was actually just part of the researchers' approach to demonstrating causality. Their approach (a difference-in-difference (DID) design) is an industry standard.
In real-world settings, if it's unethical or logistically impossible to do a controlled experiment, one way to demonstrate that Variable A causally influences Variable B is to show that change in A predicts later changes in B, but not vice versa. Indeed, the Northeastern researchers showed that growth in Airbnb listings predicted later increases in violent crime, but that growth in violent crime did not predict later increases in Airbnb listings. (They did more too. I just want to stress that they used standard, widely-accepted methods for establishing causality.)
The final concern raised by Airbnb is that the researchers used the "Joined in" date to identify new Airbnb listings, but the date that some people join Airbnb may be years earlier than they actually start hosting. This does makes the definition of "new listing" less than ideal (Airbnb doesn't provide the data that would allow a better definition). Still, it's only possible in theory that inaccuracies introduced by this definition could've led the researchers to falsely link Airbnb growth to crime. Airbnb offers no rationale or even a hint as to how that might happen.
In sum, Airbnb's rebuttal is a pot of speculations, seasoned with one misstatement about causal statistics. The company noted some details of study methodology that might, in theory, be problematic, but no evidence was presented of actual problems.
I'm not suggesting we shouldn't be vigilant. Whether we're conducting research, reading it, or reading about it, we should always consider the kinds of issues Airbnb mentioned. But when we feel that a study's methodology is strong (as I feel in this case), then we need to be brave, draw conclusions, and make use of what we've learned.
Although I've trashed Airbnb's defense of its safety record, I probably won’t stop using the company when I travel. In my opinion, it’s not an unsafe option if you take precautions – read host reviews, research neighborhoods, trust your gut when you interact with a host, etc. The Northeastern study wasn't set up to measure the usefulness of these precautions, but rather to illustrate city-wide trends. (Analogously, studies that tell you the murder rate for a particular city overestimate your actual risk of being murdered if you spend most of your time in the safest part of the city.)
So, I will continue to use Airbnb, unless Brian Chesky reads this newsletter and deactivates my account…
Thanks for reading!