09/03/2022 10:58 Dozens of recent clinical trials may contain wrong or falsified data, claims study | Peer review

er review and scientific publishing | The …

Peer review and scientific publishing

This article is more than 4 years old

Dozens of recent clinical trials may contain wrong or

falsified data, claims study
Fresh concerns over reliability of papers published in journals as
suspicious statistical patterns prompt investigations into some of
the identified trials

Stephen Buranyi and Hannah Devlin

Mon 5 Jun 2017 16.39 BST

Dozens of recent clinical trials contain suspicious statistical patterns that could
indicate incorrect or falsified data, according to a review of thousands of papers
published in leading medical journals.

The study, which used statistical tools to identify anomalies hidden in the data, has
prompted investigations into some of the trials identified as suspect and raises new
concerns about the reliability of some papers published in medical journals.

The analysis was carried out by John Carlisle, a consultant

anaesthetist at Torbay Hospital, who previously used similar statistical tools to
expose one of the most egregious cases of scientific fraud on record, involving a

https://www.theguardian.com/science/2017/jun/05/dozens-of-recent-clinical-trials-contain-wrong-or-falsified-data-claims-study 1/3
09/03/2022 10:58 Dozens of recent clinical trials may contain wrong or falsified data, claims study | Peer review and scientific publishing | The …

Japanese anaesthesiologist who was found to have fabricated data in many of his
183 retracted scientific papers.

In the latest study, Carlisle reviewed data from 5,087 clinical trials published during
the past 15 years in two prestigious medical journals, Jama and the New England
Journal of Medicine, and six anaesthesia journals. In total, 90 published trials had
underlying statistical patterns that were unlikely to appear by chance in a credible
dataset, the review concluded.

“This raises serious questions about data in some studies,” said Carlisle. “Innocent
or not, the rate of error is worrying as we determine how to treat patients based
upon this evidence,” he added.

Dr Andrew Klein, the editor-in-chief of Anaesthesia, which has published the new
analysis, called for the studies identified as potentially flawed to be reviewed
urgently.

“It’s very scary that we may be treating patients based on false evidence.” he said.
“It may be the case that certain treatments may need to be withdrawn from use.”

The tool works by comparing the baseline data, such as the height, sex, weight and
blood pressure of trial participants, to known distributions of these variables in a
random sample of the populations.

If the baseline data differs significantly from expectation, this could be a sign of
errors or data tampering on the part of the researcher, since if datasets have been
fabricated they are unlikely to have the right pattern of random variation. In the
case of Japanese scientist, Yoshitaka Fuji, the detection of such anomalies triggered
an investigation that concluded more than 100 of his papers had been entirely
fabricated.

Klein said that in some cases the anomalies could be put down to
“misinterpretation, statistical error or plain simple mistakes”, such as transcribing
data wrongly or mislabelling a column.

The latest study identified 90 trials that had skewed baseline statistics, 43 of which
with measurements that had about a one in a quadrillion probability of occurring by
chance.

The review includes a full list of the trials in question, allowing

Carlisle’s methods to be checked but also potentially exposing the authors to
criticism. Previous large scale studies of erroneous results have avoided singling out
authors.

https://www.theguardian.com/science/2017/jun/05/dozens-of-recent-clinical-trials-contain-wrong-or-falsified-data-claims-study 2/3
09/03/2022 10:58 Dozens of recent clinical trials may contain wrong or falsified data, claims study | Peer review and scientific publishing | The …

Relevant journal editors were informed last month, and the editors of the six
anaesthesiology journals named in the study said they plan to approach the authors
of the trials in question, and raised the prospect of triggering in-depth
investigations in cases that could not be explained.

Klein called for journals to adopt similar screening tools, in the interests of scientific
integrity. “There is no excuse for any medical journal reporting randomised
controlled trials not to [do so],” he said.

Sir David Spiegelhalter, professor for the public understanding of risk at the
University of Cambridge, said that the outlier papers ought to be looked at closely,
but cautioned against attributing a cause for the discrepancies at this stage. “This is
a real danger of a classic ‘prosecutor’s fallacy’,” he said. “Just because fabricators
had a good chance of producing non-random data, does not mean that non-
randomness means a good chance of fabrication.”

A spokeswoman for the New England Journal of Medicine said they had not had
access to the list of papers cited until publication, but that the editors were taking
the issue seriously and would carefully review all information.

Howard Bauchner, the editor-in-chief of Jama, also responded, saying: “We receive
numerous allegations about various issues related to the articles we publish. After
we assess the validity of the allegation, we will determine next steps. We certainly
believe authors have a right to respond to allegations that are important.”

https://www.theguardian.com/science/2017/jun/05/dozens-of-recent-clinical-trials-contain-wrong-or-falsified-data-claims-study 3/3