Data fabrication and other reasons for

non-random sampling in 5087 RCTs in both

anaesthetic and medical journals

An Article Review
 Author:
Dr. John Carlisle

About
Journal: Anaesthesia
Impact factor: 5.431

Funding: Nil
Competing interests: An editor of
“Anaesthesia”
Background

● Fraudulent research not a new phenomenon

● Statistical methods described in 1941 by JBS Haldane discredited the the
genetic experiments of Franz Moewus
● Similar methods were used by Carlisle in 2012 to highlight discrepancies in
167 RCTs carried out by Yoshitaka Fujii
● On the leaderboard of biomedical authors who have been forced to retract
publications, three of the top fifteen are anaesthetists
 Culprit anaesthesiologists...

#1 Yoshitaka Fujii #2 Joachim Boldt #15 Scott Reuben

● 183 papers retracted ● 96 papers retracted ● 24 papers retracted

● University of Tsukuba ● University of Giessen ● Tufts University
● RCTs of anti-emetics in ● Colloids ● Pain medicine
PONV
Rationale for this study

● Are trials published by anaesthetists more likely to be retracted?

● Are anaesthetists more likely to fabricate data?
● Would the statistical methods used to discover issues with Fujii also
retrospectively find aberrations in Boldt and Reuben?
The questions

1. Does the distribution of baseline means correspond to the expected

distribution, and are discrepancies shared by both anaesthetic and
non-anaesthetic medical journals

2. Is there a difference in retraction rates between leading anaesthetic and

non-anaesthetic journals

3. Is data corruption discoverable by these new statistical methods in those

articles that had been retracted?
Study Design
Summary Criteria
● Review of 5087 RCTs in 8 leading ● Pulished between 2000 and 2015
medical journals, 6 anaesthesia and 2 ● RCT
non-anaesthesia ● Participant allocation described as
○ Anaesthesia random
○ Anaesthesia and Analgesia ● Baseline summary data had to be
○ Anesthesiology continuous variable
○ BJA ● Excluded trials with stratified variables
○ Canadian Journal of Anaesthesia
○ European Journal of
Anaesthesiology
○ NEJM
○ JAMA
 Primary end-point

Distribution of p-values, calculated

for:
Statistical ● differences between means;

approach ● Individual variables; and

● When these were combined

within trials
Simply put...

● In a properly conducted RCT, differences in baseline characteristics are by

definition random
● P-value: probability of random sampling resulting in a difference as large or
larger than the observed difference
● Carlisle’s method performs analysis of baseline data in RCTs to identify
instances where sampling in RCTs may not have been random
Results
Question 1

● P values deviate from what is

expected at values close to
zero and one
● 15.6% (794/5087)
Question 1

● Demonstrated in all eight

journals, both anaesthesia
(3634 trials) and
non-anaesthesia (1453 trials)
Question 2

● 6/1435 trials in non-anaesthesia journals retracted

● 66/3634 trials in anaesthesia journals retracted
Question 3

● To test this, the author set an initial threshold of p<0.0001

○ only flagged 8 of 72 (11%) of retracted trials
○ 7 of which had Fujji as first author!
○ Flagged 82/5105 (1.6%) of unretracted trials
● Not particularly sensitive or specific
● Recommendation: to group all RCTs by one author together and analyse
(described by author as “laborious”)
Summary

1. Distribution of means in RCTs inconsistent with random sampling

○ Fraud
○ Unintentional error
○ Poor methodology
2. Distortion of expected distribution indistinguishable between anaesthesia
and non-anaesthesia journals analysed
3. Admittedly neither very sensitive or specific in identifying definite error, but
rather guiding editors toward “suspect” trials.
Anaesthesia editorial

● “Widening the search for suspect data - is the flood of retractions about to
become a tsunami”
● Author: JA Loadman: An Editor of Anaesthesia
● “The Carlisle Method” - a “statistical exposé”
● Adoption of the method by Anaesthesia going forward
Anesthesiology
editorial

“When the scientific content of

Anesthesiology and the integrity of our
authors are pubically questioned, it is
the journal’s responsibility to evaluate
and inform our readers of such claims…
and the results of our evaluation of the
claims”

Authors: Kharasch (Editor in Chief) and

Houle (Statistical Editor)
Anesthesiology editorial

“Were there only some valid algorithm that could detect fabrication and
falsification…”

● 12 RCTs identified by Carlisle published in Anesthesiology

● “Factually incorrect”: Notes that 6 of these are animal trials and not human
trials
● “Methodologically flawed”: Carlisle is assuming randomised variables, when
in fact it is the subject that is randomised
Anesthesiology editorial

“Were there only some valid algorithm that could detect fabrication and
falsification…”

● 12 RCTs identified by Carlisle published in Anesthesiology

● “Factually incorrect”: Notes that 6 of these are animal trials and not human
trials
● “Methodologically flawed”: Carlisle isassuming randomised variables, when
in fact it is the subject that is randomised
● “Ethically questionable”: Defied COPE guidelines on how to handle research
misconduct by “calling out” authors
● Decries the method: “Not the Holy Grail for detecting fabrication”
Thank you