In these uncertain times manuscripts on SARS-CoV-2 or COVID-19 get published at high speed, sometimes as pre-publications on sites at MedRxiv and BioRxiv. This is an important way to share knowledge rapidly in the hope of increasing insight into the disease, finding a treatment, and developing vaccines. For those who dont know: the regular publication process of a manuscript can take a long time, time we do not have currently.
The disadvantage of these rapid publications is that pre-publications are not peer reviewed at all and that many of the fast publications have gone through a rapid peer review process that was too fast to catch problems. As a result, along with the fast publication of important advances also a lot of bad science is published.
So how can you find out whether an article that was rapidly published indeed contains sound research, mainly bad science, or something in between? Where can you find whether a (COVID-19) publication was flagged for irregularities? Below are a few sources that can help to spot bad science but that are of course also very useful at other times.
Preprint server
When a manuscript is pre-published on a preprint server such as MedRxiv or BioRxiv it will have links below it to any tweets that mention the manuscript, blogs written about it, and comments published on PubPeer (see below). In addition, it has a section below the article where peers can leave their comments. This way, dubious results or plain bad science are often quickly identified, which is especially important when manuscripts get a lot of -undeserved- attention in the media.
Public peer review
There is also a website called PubPeer, that serves as an online journal club. Here any article that raises questions is discussed. Often these are minor points that can be dealt with before an article is published in its final form. There are however also many instances where serious issues are found with an article. Obvious examples are various of the publications on hydroxychloroquine for treatment of covid-19, such as the one by Gautret et al. that received over 65 comments on PubPeer pointing out many problems with patient inclusion, endpoints, analyses, and results. Or another paper by Ren et al. where one patient (n=1!) was successfully treated for COVID-19 with traditional Chinese medicine but never had a positive test…
Another place where a lot of public peer review is published is twitter. Look for instance for the hashtags #ImageForensics and #RetractionWatch.
PREreview
PREreview is another website where reviews are posted, this website however deals with pre-published articles only, such as the ones published on MedRxiv, BioRxiv, and Preprints.org. The reviews on this website are more like journal club reviews in the sense that the whole article is discussed, not just the flaws. Quite a lot of effort appears to have gone into the reviews.
In addition, one can request the review of a specific manuscript, which is helpful if you lack the knowledge to evaluate a manuscript yourself.
A recently set up database specifically collects reviews of prepublished manuscripts about COVID-19 or SARS-CoV-2. The reviews are written by scientists at the Human Immune Monitoring Center at Mount Sinai (New York). Not sure whether this site is user friendly, the reviews are however also posted by the Sinai Immunol Review Project under the articles at the preprint servers.
Science Integrity Digest
The scientist Elisabeth Bik has made it her job to identify duplicated images in science papers, she reports these duplications on twitter @MicrobiomDigest and on her website Science Integrity Digest. In some journals duplication of images is more common than in other, Bik even found one journal in which 15% of 675 articles contained duplicated images. This is probably a result of sloppy or absent peer review.
Several of her recent blogs are about bad science in COVID-19 research, such as this study in seniors where the participants in the control and treatment groups were not randomized and were not equally sick. You can find a list of other dubious COVID-19 articles that Bik discussed here.
She also brings to light other science integrity problems, ranging from animal ethics misconduct to publications based on bad science in general.
For Better Science
The website For Better Science is a website maintained by Leonid Schneider that focuses on integrity in Life Sciences and Biomedicine. It contains many well-researched stories about questionable scientists, such as Mauro Ferrari. Ferrari was appointed as president of the European Research Council (ERC) in January 2020 and was fired three months later. There are also several stories about various claims made by COVID-19 researchers, the one about Didier Raoult sticks out most.
Retraction Watch
The Center for Scientific Integrity (CSI) maintains the Retraction Watch website. This is a website that keeps track of any papers ever published that were retracted, or should have been. Several COVID-19 manuscripts have already made it onto Retraction Watch, and there is a separate page dedicated to retracted COVID-19 papers.
If you would like to keep updated on this topic you may want to follow @RetractionWatch on twitter.
Bad Science – Other sources
Please let me know in case you know of any other good websites that can help identify bad science.
Recent Comments