Coronavirus: Has flawed science and rushed research failed us on Covid-19?

It is now six months since Covid-19 first emerged in China. More than seven million people have been infected by the Sars-CoV-2 virus, 400,000 have died – and the toll continues to mount. Yet through it all, there has been a belief that scientific research is the golden thread that will lead humanity out of its nightmare. But such confidence is looking increasingly misplaced. Researchers have been forced to withdraw claims about a once-promising drug published in two leading medical journals, and the World Health Organisation has reversed key advice on combating the spread of the virus. Does this simply reflect the evolving state of scientific knowledge, or is it a symptom of a deeper malaise? In the scramble to find treatments for Covid-19, none has been more controversial than the drug hydroxychloroquine. Hints that this anti-malarial might be effective first emerged in lab tests by Chinese scientists in February. Then French researchers claimed it had proved effective in a study of Covid-19 patients. By the end of March, US President Trump was hailing the drug as a “game-changer”, and the US Food and Drug Administration authorised its use for some Covid-19 patients. By then, the drug had become part of the WHO’s global programme for testing promising therapies. But from the outset there were strong reasons to doubt the evidence. Test-tube studies are notorious for failing to predict what happens in real patients, while the French study was riddled with fundamental flaws. The gold standard for finding out whether a drug works is the so-called double-blinded randomised controlled trial (RCT). This involves randomly selecting patients either to receive the new drug or a “control”, in the form of a harmless placebo or existing treatment. In addition, neither the patients nor their doctors are told who is getting what. This “blinding” and randomisation helps prevent the results being undermined by biases introduced unwittingly or otherwise by the researchers. The French study was neither blinded nor properly randomised, and included just 42 patients. Its findings were also reported in a bizarre way that excluded patients who died – thus exaggerating the apparent effectiveness of the drug. Despite all this, the research was accepted for publication by an academic journal, and duly received global coverage. Meanwhile, the WHO pressed on with its programme of large RCTs in the hope of getting reliable insights into whether hydroxychloroquine worked. In the race to uncover the truth, some researchers began exploring existing data from hospitals on the effect of hydroxychloroquine on Covid-19 patients. Late last month the prestigious journal The Lancet published a huge study comparing the fate of nearly 15,000 patients given hydroxychloroquine with 81,000 patient "controls" who did not. The results suggested the drug was literally worse than useless, increasing the risk of death among those patients receiving it. The research sparked global concern and led the WHO to suspend all its studies into the drug pending an urgent investigation. Yet scientists quickly expressed doubts about the study – especially the source of its evidence and how it had been analysed. It emerged that the raw data had come from Surgisphere, a company run by one of the authors of the paper, and requests by other scientists to access it had been refused. Statisticians also raised concerns about the claims made on the basis of the data. Late last week The Lancet retracted the paper, after three of the authors conceded they no longer had confidence in it. Shortly afterwards, another leading medical journal retracted a Covid-19 study based on data from the same company. Around 20 other studies based on Surgisphere data are now said to be under review. So might hydroxychloroquine be safe and effective after all? Ironically, the day after the retraction, The Lancet study's basic conclusion was supported by a gold-standard RCT involving more than 4,600 patients. According to researchers at the University of Oxford, the results “convincingly rule out any meaningful mortality benefit” from giving hydroxychloroquine to Covid-19 patients. These results have yet to be published in a respected scientific journal. But as the French study that sparked the furore and the debacle over the Surgisphere data shows, publication is no guarantee of reliability. For those outside of the scientific community, the idea that such research can so easily be accepted by leading journals will come as a shock. Yet researchers quickly learn to be sceptical of headline-grabbing studies rushed into print by prestigious journals. For them, it is just another example of the gulf between the perception and reality of how science works. After months of refusing to recommend the use of face masks by the public, the WHO last week performed a U-turn. Citing “new evidence”, the organisation advised governments to encourage the public to wear fabric masks where social distancing was difficult, while the over-60s or those with underlying health risks should wear medical-grade masks of the type normally reserved for healthcare workers. The decision has divided the scientific community. Many believe face masks give a modest but valuable level of protection. Others insist any benefit could be cancelled out by people becoming complacent, and ignoring proven measures like social distancing and hand-washing. The constant argument over the pros and cons of so “obvious” a countermeasure will baffle many. But for those familiar with the research, it comes as no surprise. The truth is that – despite long being used in Asian countries to ward off infection – there are no large scale, high quality studies of the effectiveness of face masks. What studies have been done are typically small and leave many questions unanswered – not least about complacency. Attempts to pull together the evidence have also been marred by well-known statistical blunders . The WHO's decision is based largely on a mix of lab studies and pragmatism, and could still prove misguided. Again, none of this comes as a surprise the researchers. Small, poor quality studies incorrectly analysed have long plagued scientific enterprise. Until now, it has gone largely unnoticed by the public – and politicians seeking to be “led by the science” – because it hasn't affected their lives. But now they are getting daily insights into the realities of scientific research – and there is already evidence that they don't like what they are seeing. Robert Matthews is visiting professor of science at Aston University, Birmingham, UK

Coronavirus: Has flawed science and rushed research failed us on Covid-19?

Research into tackling Covid-19 is often far less robust than it seems