Serious credibility questions are now surrounding two coronavirus studies recently published in scientific journals, including one that found little evidence for the drug hydroxychloroquine working as a potential treatment for covid-19. Critics have argued that the data used by the same team of researchers in both studies could have been falsified by the corporation that provided it. Both journals have now expressed concern about the findings while asking for the authors to prove that their data is reliable.
The two studies were published in the New England Journal of Medicine (NEJM) on May 1 and in the Lancet on May 22. They were conducted by members of the same team of researchers, led by Mandeep Mehra, medical director of the Brigham Heart and Vascular Center in Boston, Massachusetts. Both papers were covered by Gizmodo soon after publication.
The NEJM study found evidence that people with preexisting heart conditions were more likely to die from covid-19 than those without them. It also found no link between an increased chance of death from covid-19 and patients taking certain heart medications known as angiotensin-converting–enzyme (ACE) inhibitors and angiotensin-receptor blockers (ARBs).
The more widely circulated Lancet study found evidence that hospitalized patients taking hydroxychloroquine or chloroquine (either alone or in combination with an antibiotic) did not survive with any greater odds than those not taking the drugs; rather, they seemed even more likely to die. The study, which was purported to involve over 90,000 patients, was quickly influential, with the World Health Organization and several countries citing its results in deciding to stop ongoing studies of hydroxychloroquine over fears of endangering patients. This afternoon, the WHO announced it would restart the hydroxychloroquine arm of its covid-19 treatment trials.
The two studies relied on anonymous patient data that was said to be collected from hundreds of hospitals by the company Surgisphere. One of the company’s founders, Sapan Desai, is also listed as a co-author in both papers. But soon after the Lancet study made waves, outside scientists and critics, including James Watson of Oxford University and Australian statistician Peter Ellis, said they spotted irregularities with Surgisphere’s data.
Simply put, the data seems too good to be true. Surgisphere claimed to have collected meticulous electronic records from hospitals in countries where there is relatively little digital health care infrastructure, particularly in Africa. Elsewhere, the data didn’t match up with other documentation. Initially, for instance, the Lancet study reported more deaths from Australia than the official count at the time. The study’s authors issued a correction for that error, claiming they had mixed in data from Asia, but said that the findings weren’t affected. Also suspicious is the high number of cases collected from their claimed sampling of hospitals relative to the total number of cases in a region. The study’s reported North American numbers were slightly higher than the total number of reported covid-19 hospitalizations in the U.S. at the time, for instance.
As others have noticed, Surgisphere also seems to have a remarkably small footprint for having been founded in 2007. Prior to the covid-19 pandemic, no published studies had used Surgisphere’s data. And despite claiming to have secured data-sharing agreements with hundreds of hospitals worldwide, the company only has a handful of listed employees. The company has so far not disclosed the names of any of these hospitals.
On Tuesday, hundreds of researchers signed an open letter to the Lancet and the NEJM highlighting these inconsistencies and demanding that Surgisphere make its data available to the public and that the journals conduct their own investigations. That same Tuesday, the NEJM issued an expression of concern agreeing with the critics and calling for the authors to provide their data; the Lancet followed suit on Wednesday, adding that the results of an independent audit would be out soon.
A third study conducted by Surgisphere cofounder Desai using Surgisphere data that is not yet peer-reviewed is also under question. It concluded that the antiparasitic drug ivermectin can substantially lower the risk of dying from covid-19. Despite not being published yet, the study may have influenced governments in South America, where ivermectin is commonly used, to sanction its experimental use.
A representative for Desai appeared to confirm to Science Magazine on Wednesday that Surgisphere’s data wasn’t shared with the other co-authors. The spokesperson told Science that the company would soon arrange “a nondisclosure agreement that will provide the authors of the NEJM paper with the data access requested by NEJM.” Mandeep Mehra, the lead author of the Lancet and NEJM study, has not responded to a request for comment from Gizmodo.
Other studies related to covid-19 have been criticized as deeply flawed and even occasionally retracted, but this is undoubtedly the largest potential scientific scandal to emerge so far. To be clear, though, even if these studies do turn out to be fraudulent, that doesn’t mean their basic conclusions are necessarily wrong, as other evidence has suggested that ACE inhibitors and ARBs aren’t dangerous for covid-19 patients, while the overall evidence for hydroxychloroquine’s effectiveness remains mixed (some studies supporting its use have faced their own concerns over reliability).
Though any scientific fraud is dismaying, this debacle highlights the willingness for self-criticism within the scientific community. It took less than two weeks for these studies to be torn to shreds once the first whispers of skepticism were heard. It also illustrates the importance for everyone (science reporters included) to not take any single study’s findings as definitive.