Data Falsification

On Monday, a former metallurgist pled guilty to charges that she falsified data on the strength of steel that the U.S. Navy uses to make submarines. The impact of her misconduct isn't clear yet, but the incident reminded me that in rare instances, stats mislead us not because they're misinterpreted, or grounded in weak methodology, but because someone made them up.

How often do scientists falsify their data? (I'll be using the term "falsify" to mean both making up data as well as changing numbers to enhance outcomes.) According to one meta-analysis, roughly 2% of scientists admit to having done so at least once in their careers. The actual percentage may be higher, given that not everyone cops to bad behavior, even on anonymous surveys, but 2% is already a troubling figure, given other studies suggesting that less than .02% of peer-reviewed papers are retracted for any reason. The discrepancy between that 2% and that .02% hints that some extent of data falsification goes unnoticed.

This possibility is especially concerning in the midst of a pandemic, where a combination of humanitarian, financial, and ideological interests continue to drive research on COVID-19 vaccines and treatments, and studies are being churned out with unusual speed. Assuming that study methods are strong, can we trust that the scientists behind these studies are honest?

It's hard answer a question like that. For the most part we trust scientists because we have to. And yet....we know that every profession has its cheaters. In the meta-analysis I just described, 14.2% of scientists claimed personal knowledge of a colleague who had fabricated or in some way inappropriately modified research data. So, even though you pretty much have to trust scientists (and, if you're like me, you want to trust them), there are reasons to worry.

In this newsletter, I want to provide some reassurance through a look at the editorial practices of one prestigious medical journal. My focus will be on two articles (one on vaccines, one on a COVID-19 treatment) that were published in this journal and then retracted owing to falsified data. In the past decade, the journal and its editor have evolved from dismissiveness and near-inaction to the implementation of strict policies that reduce the risk of falsified data being published.

The journal in question is The Lancet, one of the most influential medical journals in the world (according to common consensus as well as citation rates). Since a clear distinction can't be made between a journal and its editors, I will also be referring to Richard Horton, editor-in-chief of The Lancet since 1995, and a somewhat polarizing figure who’s widely admired yet often criticized for mistakes (like the two I'll discuss in this newsletter), and for an alleged willingness to publish methodologically problematic research if it seems likely to attract publicity for the journal.

The MMR-autism scandal

I'll start with an infamous article that appeared in The Lancet in 1998, three years after Horton became editor. In this article, Andrew Wakefield and colleagues presented data linking the Measles, Mumps, and Rubella (MMR) vaccine to autism.

This case is a pretty scary example of data falsification, because (a) years passed before falsification was proven, (b) one persistent journalist (and luck) were needed to provide proof, (c) public opinion was affected in the interim, and (d) the perpetrator still maintains credibility among anti-vaxxers.

Wakefield's Lancet article reported data on 12 children who all developed autism as well as colitis; 8 of these children began to show symptoms two weeks or less after receiving the MMR vaccine. In the article itself, Wakefield and co-authors noted that their findings were merely correlational, but in press conferences and other public statements, Wakefield began to increasingly push the idea of a causal link between the vaccine and autism.

I agree with many experts that Wakefield's study never merited publication in The Lancet in the first place, and that once it was published, it received more credibility than it deserved. A single study with 12 participants is just too small to be credible. In contrast, less than two years after Wakefield's article appeared, a study that had followed 1.8 million children for 14 years showed no links between MMR vaccinations and autism, and only rare associations with adverse events of any sort. Moreover, between the publication of Wakefield's article and this newer one, a panel of 37 medical experts convened by the Medical Research Council reviewed all available evidence and concluded that the MMR vaccine poses no significant risks to children.

In short, even after Wakefield's article was published, nobody should have taken it seriously. Not unless it could be replicated, which scientists attempted to do, unsuccessfully, for several years. We now have data on tens of millions of children, along with the pronouncements of thousands of experts, indicating no link between vaccines and autism. (Unfortunately, this hasn’t deterred the growing anti-vaccination movement.)

For five years, most scientists considered Wakefield's MMR data merely anomalous, if not mildly suspicious. Then, in 2003, while interviewing parents of the original 12 participants, a journalist named Brian Deer began to notice discrepancies between what the parents said and what Wakefield's article reported. Although children were not named in the article, identification of each child from medical records held by the parents was easy enough given the small sample size. When comparing the medical records to Wakefield's data, Deer found, for example, that some children's autistic symptoms had actually begun to appear before they received the MMR vaccine. For other children, the amount of time between vaccination and their first symptoms was substantially longer than Wakefield claimed. Other children hadn't even received the appropriate diagnosis. Ultimately, Deer found that for every one of the 12 participants, Wakefield's data had been fabricated or otherwise falsified.

Deer uncovered other kinds of misconduct as well. In particular, Wakefield had failed to report a serious conflict of interest, as required by editoral policy (and scientific convention). Prior to conducting the study, Wakefield had been paid by a UK lawyer who was preparing a class action suit against vaccine manufacturers for "damage" caused by their vaccines. This lawyer identified 12 sets of parents who blamed MMR for their children's autism and recruited them for participation in Wakefield's study. However, the article deceptively describes the children as "consecutively referred" and "self-referred" to the hospital.

There's more to the story (see here or here). The short version is that once Deer published his findings in 2004, The Lancet formally acknowledged Wakefield's conflict of interest but not the falsification of data. It wasn’t until 2010, when the UK's General Medical Council concluded that Wakefield had dishonestly falsified data (and banished him from practicing medicine in the UK) that The Lancet fully retracted Wakefield's article.

At this point you might not fault The Lancet for anything but acting slowly. However, the journal's 2010 retraction consists of a single paragraph containing only one substantive statement: "The claims in the original paper that children were “consecutively referred” and that investigations were “approved” by the local ethics committee have been proven to be false." In short, the retraction made no mention of falsified data, though falsification had been clearly documented by Deer, by the General Medical Counsel, and by others.

Along with a failure to acknowledge falsification, Horton also suggested that academic journals can't do anything specific to prevent misconduct. In a 2010 interview with the Guardian, he indicated that by simply being a peer-reviewed journal, The Lancet had done all it could do to establish the validity of Wakefield's research. (As we'll see in a moment, Horton was forced to revise this view last year.)

There's also a particularly ugly side to this story. Deer accused Horton of failing to take his claims seriously, which he’d published in 2004 and shared with Horton in person the same year. Deer claimed that Horton lied about an investigation into Wakefield's conduct by hospital authorities, and even collaborated with Wakefield and other study authors on damage control prior to the General Medical Council investigation in 2010. Horton denies these and other accusations that Deer has made against him.

So there you have it. A totally fabricated study that took 12 years to retract (and wouldn't have been retracted unless a journalist had been talking to participants' families and then decided to act on what he learned). And a prominent journal that at best acted slowly, allowing Wakefield's work to fuel the nascent anti-vaccination movement and ultimately lower vaccination rates among children in the US, UK, and elsewhere.

Now let's fast-forward to 2020 and evidence of progress at The Lancet:

The Surgisphere scandal

On May 22, 2020, Mandeep Mehra and colleagues published an article in The Lancet showing that hydroxychloroquine and chloroquine are ineffective treatments for COVID-19 that increase the risk of arrhythmias and in-hospital deaths among COVID-19 patients.

Unlike the Wakefield study, this one was enormous: 96,032 patients drawn from 671 hospitals representing six continents. However, within days, experts were airing concerns about the statistics and other details of the study. Posts to a prominent blog, and comprehensive summaries of expert response by Catherine Offord, called attention to lack of methodological detail, insufficient statistical control, and an unwillingness to share data (which would’ve been standard practice). Suspicions were strongly aroused by the uniformity of the descriptive stats. As James Watson noted, the prevalence of smoking across continents ranged from 9.4 to 10% and wasn't consistent with known variability in prevalences. The rate of antiviral use ranged from 38.4% to 40.8%, and rates of diabetes and other problems didn't vary much either. Watson's initial response was cautious": "I...am not accusing the authors/data company of anything dodgy, but as they give almost no details about the study and “cannot share the data”, one has to look at things from a skeptical perspective."

On May 30, eight days after the study was published, Watson and more than 180 other experts wrote an open letter to the authors and Lancet editor Richard Horton, summarizing various problems with the data. Along with those already mentioned, the signatories noted that the authors did not identify the hospitals that provided data (as would be standard practice), and that data from Australia were incompatible with government figures (i.e., the number of hospital deaths among COVID-19 patients during the study period exceeded the total number of deaths from any cause that had occurred in the entire country during that period).

Attention quickly turned to Surgisphere Corporation, the Illinois-based company that supposedly provided the researchers with all study data. Efforts to obtain information from Surgisphere proved futile, and no hospital confirmed participation in this study. Nobody in the field had heard about the study either, although 671 hospitals allegedly participated, and in any case, most experts were deeply skeptical that data on 96,032 patients could've been acquired and analyzed during the brief time span of the study. (Data were reported for December 20, 2019 through April 14, 2020; The Lancet published the article on May 22, 2020.)

On June 4, 13 days after the study was published, three of the four study authors requested that it be retracted, because Surgisphere had repeatedly refused to transfer the dataset and other documents to them for an independent review. The only author who did not request a retraction was Sapan Desai, the owner of Surgisphere. On the same day, The Lancet formally retracted the article.

Subsequent investigation revealed that Surgisphere was located in a tiny office, had virtually no online presence, retained six employees (including a science fiction writer and an adult model), and apparently held a contract with just one hospital.

In short, Sapan Desai and his "company" engaged in massive fabrication of data. Exactly how much fabrication has yet to be determined. (The lead author, Harvard professor Mandeep Mehra, never saw the data, and claims to have been misled by Desai.) In response to a deluge of negative publicity following the retraction, the WHO and other agencies temporarily halted clinical trials of hydroxychloroquine as a COVID-19 treatment or prophylaxis. Desai's misconduct thus delayed legitimate research (which eventually confirmed that hydroxychloroquine is ineffective) and gave those who support the drug for ideological reasons more ammunition for claiming that we can't trust mainstream science.

The Lancet's response

Although 13 days is vastly quicker than 12 years, some observers still complained that The Lancet was too slow in retracting this study. The essence of the complaint was that you only need about 13 minutes, not 13 days, to spot deeply suspicious statistical flaws, logistical implausibilities, and missing details.

All the same, I see improvements in the way Richard Horton and his editorial colleagues handled this case, as compared to the Wakefield scandal. These improvements are captured by a statement subsequently published by the editors in The Lancet, entitled Learning from a Retraction. What did the editors appear to have learned from retracting the hydroxychloroquine article? Humility, perhaps, and a sense of professional responsibility for detecting misconduct such as data falsification. In particular, the team announced new additions to their editorial policies. Here are some key examples:

—More than one author will now need to have directly accessed and verified all data. In the case of an academic and commercial partnership, one of the authors who verifies the data must be from the academic team.

—All research papers must now include a data-sharing statement that describes what data and other documents will be shared, when data will become available, and so on.

—Lancet editors must now ensure that at least one peer reviewer is knowledgable about any real-world datasets being reported.

These new policies represent substantial progress since Richard Horton's statement, 10 years earlier, that peer review is the best a journal can do to prevent data falsification and other misconduct.

Conclusion

Scientific misconduct is probably as old as science itself, and it may always be with us. There are powerful incentives to falsify data, including profit, professional advancement, and public acclaim. Although we should remain vigilant, here are three reasons not to lose sleep over the problem:   

1. Editorial practice is evolving. Learning from a Retraction is an example of stricter editorial policy that reduces opportunities to falsify data. Requirements for pre-registration of study protocols prior to data collection, and for sharing data once collected, continue to increase the trustworthiness of published research.

2. Professional organizations are increasingly vigilant. For example, the American Association for Public Opinion Research (AAPOR) and the American Statistical Association (ASA) have collaboratively developed best practice recommendations for preventing and detecting survey data falsification. The AAPOR-ASA recommendations are wide-ranging and span everything from organizational culture to specific algorithms for detecting suspicious patterns in data.

3. Science is a social process. In some instances, falsified data don't get published, because they're spotted by reviewers, editors, or even co-authors. When those gatekeepers fail, expert readers may step up. For example, less than a week after the hydroxychloroquine study appeared, statisticians were airing serious concerns, and science journalist Catherine Offord was providing regular, comprehensive updates on the scandal as it evolved. (As for the metallurgist that inspired this newsletter, she was on the verge of retirement when the person chosen to replace her noticed suspicious test results.)

All the same, assuming that falsification still occurs, is there anything you can do to avoid being duped? Perhaps not, but it never hurts to examine the actual studies. Even if you didn't know that Wakefield falsified his data, you'd be cautious at least about trusting the results of a study with a sample size of 12. As for the hydroxychloroquine study, even if you skimmed it, you might be struck by the lack of detail on where the data came from (e.g., literally zero descriptive information on the hospitals), and if you looked closely enough, you might be suspicious that data on 96,032 patients could’ve been gathered, analyzed, written up, reported, and published in a matter of weeks.

Finally, don't forget that 37% of all statistics are made up on the spot. You read it here first.