Image: © Tomas Skopal/Stock.adobe.com
As researchers around the world race to find ways to stop the coronavirus in its tracks, contradicting information may be appearing online at a rapid pace.
When a tech company is faced with a particularly nasty computer virus that could leave its customers facing a major privacy headache, it will typically rush out a patch to fix the issue as soon as possible.
But in the world of science, where people’s lives are at stake, researchers cannot simply patch out a flaw. It takes many months – and possibly years – to test every possible scenario to make sure that a solution not only solves a problem, but doesn’t have unintended consequences.
The world currently finds itself in the grips of the Covid-19 pandemic, which has touched nearly every corner of the globe. At the time of writing, there are almost 2m confirmed cases worldwide, with no known vaccine or definitive cure.
Such situations are like a call to action for researchers. In the space of a few months, there are already more than 140 experimental drug treatments and vaccines in development across the world, in addition to around 250 clinical trials testing drugs developed for other medical issues to see whether they could be used against Covid-19.
Knowing what is ‘good’ science
Under the current circumstances, timelines that typically last months have been shrunk down into a matter of days. Researchers developing a device to detect the virus in saliva using lasers recently admitted they managed to put together a team and write a funding proposal in as little as eight days.
As Johnson and Johnson’s chief scientific officer recently put it to The Wall Street Journal: “We have never gone so fast with so many resources in such a short time frame”.
But is there a trade-off happening and how many of the claims being made by recent studies can we take as being ‘good’ science?
US president Donald Trump continues to make headlines for his claims regarding hydroxychloroquine, a long-established antimalarial drug, that he said could be effective against Covid-19.
This stemmed from a recent French study that said it had found evidence that a combination of hydroxychloroquine and the antibiotic azithromycin could be effective against Covid-19. However, the science around hydroxychloroquine and Covid-19 is so far inconclusive.
In a recent press conference, Trump said “what do you have to lose” when asked about people taking the drug combination. “In France, they had a very good test,” he added. “But we don’t have time to go and say, ‘Gee, let’s take a couple of years and test it out, and let’s go and test with the test tubes and the laboratories.’”
Spotting red flags
When trying to identify potential red flags in a study – whether related to Covid-19 or otherwise – some aspects of the French study stand out.
For one, it involved just 80 subjects, most of whom had mild Covid-19 symptoms. With only 15pc of those in the study showing signs of a fever, it is likely they would have naturally recovered from the virus.
Another major red flag was that the study appeared on a pre-print server for anyone to access on 16 March. It was approved a day later and published on 20 March. This remarkable turnaround raised eyebrows, especially given that one of the paper’s co-authors is also the editor-in-chief of the journal it featured in, the International Journal of Antimicrobial Agents.
Not long after it began making headlines, the International Society of Antimicrobial Chemotherapy (ISAC), which publishes the journal, distanced the organisation from the study, saying it did not meet the society’s standards.
“Although ISAC recognises it is important to help the scientific community by publishing new data fast, this cannot be at the cost of reducing scientific scrutiny and best practices,” said Andreas Voss, ISAC president.
Gaetan Burgio, an Australian National University expert on drug resistance, went as far as to describe the study’s findings as “insane”. In another study released online prior to peer review, Chinese research into hydroxychloroquine raised a number of questions over its focus and how it was translated.
Looking at supplementary Table 1, most of the controls had viral load qualitatively detected or the PCR was not done !!!! . Only 4 out of 16 controls had a proper measure of the viral load !!!! This is insane ! pic.twitter.com/hDhKewVcTu
— Dr Gaetan Burgio, MD, PhD. (@GaetanBurgio) March 21, 2020
‘You obviously can’t cut corners’
“While we applaud research publishers and journals and how to work differently and quickly, you obviously can’t cut corners,” said Emily Jesper-Mir, director of partnerships with the sound science campaigning charity Sense About Science.
“Even when something is peer reviewed and published, there still needs to be a critique of the research by anyone who is looking at that study. If there’s people who post-publication critiqued it and flagged significant flaws in the study design, then that can’t be used by Donald Trump to give it oxygen that might be dangerous.”
So why are scientists so quick to publish research that is not peer reviewed and may be lacking in many key areas? According to Jesper-Mir, such data is not invalid but helps build a bigger picture.
“Uncertainty is something researchers are used to,” she said. “And [in these instances] researchers aren’t saying there’s nothing, they’re just being transparent.”
“But it’s really important that when models are released, the researchers are clear about what the uncertainties are and they aren’t saying we know nothing. Showing their work is an essential part of how we get more knowledge and have these models, and uncertainty is embedded in that.”
A need for research, fast
Research groups seem to share the same belief. Just as the crisis was starting to make waves in Europe, the Wellcome Trust and dozens of research groups and publications backed a call for all Covid-19 pre-print data to be immediately made public for scrutiny, and that peer-reviewed research be immediately shared with the World Health Organization and made open access.
This means that everyone – not just those working in academia – needs to be aware that in the middle of a pandemic, contradicting information may be appearing online at a rapid pace. So if you see a study that seems to be making some unusual conclusions, Jesper-Mir said, engage in some critical thinking before you take it as fact.
“If something’s being spread over social media or you’re getting it on WhatsApp but you can’t spot what the source is, then it’s probably quite likely to be rubbish,” she warned.
“People need governments to not just be putting out clear soundbites on what the public take-home messages are, but to also be transparent about their policy, statistics and evidence, and how it’s changing over time.”
