Journal of Clinical Epidemiology 61 (2008) 455e463

Adjusted indirect comparison may be less biased than direct comparison

for evaluating new pharmaceutical interventions
F. Songa,b,*, I. Harveya, R. Lilfordc
a
School of Medicine, Health Policy and Practice, University of East Anglia, Norwich, Norfolk NR4 7TJ, UK
b
School of Allied Health Professions, University of East Anglia, Norwich, NR4 7TJ, UK
c
Department of Public Health and Epidemiology, University of Birmingham, Birmingham, B15 2TT, UK
Accepted 5 June 2007

Abstract
Objective: To investigate discrepancies between direct comparison and adjusted indirect comparison in meta-analyses of new versus
conventional pharmaceutical interventions.
Study Design and Setting: Results of direct comparison were compared with results of adjusted indirect comparison in three meta-
analyses of new versus conventional drugs. The three case studies are (1) bupropion versus nicotine replacement therapy for smoking ces-
sation, (2) risperidone versus haloperidol for schizophrenia, and (3) fluoxetine versus imipramine for depressive disorders.
Results: In all the three cases, effects of new drugs estimated by head-to-head trials tend to be greater than that by adjusted indirect
comparisons. The observed discrepancies could not be satisfactorily explained by the play of chance or by bias and heterogeneity in ad-
justed indirect comparison. This observation, along with analysis of possible systematic bias in the direct comparisons, suggested that the
indirect method might have produced less biased results. Simulations found that adjusted indirect comparison may counterbalance bias un-
der certain circumstances.
Conclusion: Adjusted indirect comparison could be used to cross-examine the validity and applicability of results from head-to-head
randomized trials. The hypothesis that adjusted indirect comparison may provide less biased results than head-to-head randomized trials
needs to be investigated by further research. Ó 2008 Elsevier Inc. All rights reserved.
Keywords: Adjusted indirect comparison; Head-to-head comparison; Bias; Meta-analysis; Clinical trials; Pharmaceutical intervention

1. Introduction evidence generated by adjusted indirect comparison can also

Adjusted indirect comparison has been increasingly used
in systematic reviews to evaluate relative effects of compet-
ing pharmaceutical interventions due to a lack of head-to-
head randomized trials [1e3]. Suppose that interventions
A and C were compared in a trial, and that interventions
B and C were compared in another trial. Then, the indirect
comparison of interventions A and B can be carried out by
using the results of their independent comparisons with the
common intervention C (e.g., placebo). This allows the
strength of randomized trials to be partially preserved in an
indirect comparison [2]. To improve statistical power,
Fujian Song conceived the initial idea and conducted analyses. Ian
Harvey and Richard Lilford contributed to the interpretation of results
and the preparation of the manuscript. Fujian Song had full access to all
of the data in the study and takes responsibility for the integrity of the data
and the accuracy of the data analysis.
* Corresponding author. Tel.: þ44-1603-591253; fax: þ44-1603-593166.
E-mail address: fujian.song@uea.ac.uk (F. Song).
0895-4356/08/$ e see front matter Ó 2008 Elsevier Inc. All rights reserved.
doi: 10.1016/j.jclinepi.2007.06.006
be combined with evidence from head-to-head trials [4,5].
However, the potential usefulness of adjusted indirect
comparison is often overshadowed by concern about possi-
ble bias resulting from this approach [1,3]. For the adjusted
indirect comparison of intervention A versus B to be valid,
trials that compare intervention A versus placebo should be
on average similar to trials that compare intervention B ver-
sus placebo in terms of moderators of relative treatment
effect (e.g., characteristics of trial participants) [1e3]. Ad-
justed indirect comparison will provide misleading results
if this key assumption is not fulfilled. In addition, any possi-
ble biases in trials used in adjusted indirect comparison
would also affect the validity of indirect comparison [6].
We need to remember that head-to-head comparison tri-
als are also susceptible to bias because of inadequate meth-
odology and operational difficulties [7e10]. In existing
studies of bias in clinical trials, it was usually assumed that
bias would tend to beget exaggerated estimates of treatment
effect, and the observed association between certain trial
456 F. Song et al. / Journal of Clinical Epidemiology 61 (2008) 455e463
methodological features and treatment effects was often
considered as empirical evidence of bias [8e10]. For exam-
ple, inadequate concealment of treatment allocation is con-
sidered to be associated with bias because empirical
evidence revealed that trials with inadequate or unclear
concealment of allocation reported treatment effects that
were on average 30% greater than trials with adequate con-
cealment of allocation [10].
If the logic above was true and adjusted indirect compar-
ison was indeed more likely to be biased, adjusted indirect
comparison would on average exaggerate treatment effects
as compared with direct comparison. However, empirical
evidence from 44 meta-analyses in a previous study indi-
cates that the results of adjusted indirect comparisons usu-
ally agree with the results of head-to-head trials over a wide
range of competing interventions [6]. Significant discrep-
ancies were found in only four comparisons after eliminat-
ing trials with obvious heterogeneity, and in two cases the
direct comparisons produced greater effect sizes than the
adjusted indirect comparisons [3,6].
Subsequent to this study, we observed a further case in
which the adjusted indirect comparison provided a signifi-
cantly smaller effect size than the direct comparison when
a new drug, bupropion, was compared with a conventional
nicotine replacement therapy (NRT) for smoking cessation
[11]. We noted that masking of treatment allocation was in-
adequate in this case [12]. Such failure of masking may
lead to ‘‘optimism bias’’ in head-to-head trials of new drugs
[13]. Because optimism bias and other documented meth-
odological problems in clinical trials may exaggerate the
effects of new treatments [7,8,10], we hypothesized that
there may be circumstances where, perhaps counterintui-
tively, indirect comparisons could provide a less biased es-
timate than the direct comparison. To investigate this
possibility, we sought further instances in which both the
direct and adjusted indirect methods could be used to com-
pare new and conventional drugs. We discuss circumstances
where adjusted indirect comparisons could indeed correct
for bias in direct comparisons.
2. Methods
2.1. Identification of further case studies
Our source material consisted of systematic reviews of
direct comparisons of new and conventional drugs in which
the first author has been involved. We identified two further
cases in which both direct and adjusted indirect compari-
sons could be conducted to compare new and old drugs.
These were risperidone versus haloperidol for schizophre-
nia [14] and fluoxetine versus imipramine for depressive
disorders [15]. In this paper, we report results of all three
case studies; the original case that had suggested the hy-
pothesis to us (bupropion versus NRT for smoking cessa-
tion) and the above newly ascertained case studies.
Since the three original systematic reviews have been
conducted many years ago, we searched the Cochrane Li-
brary to identify recently updated systematic reviews rele-
vant to the three case studies [16e23]. We also searched
MEDLINE and the references of retrieved reviews to iden-
tify additional placebo-controlled trials for adjusted indirect
comparisons. Trials were excluded if intention-to-treat
analysis was impossible, and we used only the first phase
from crossover trials.
2.2. Statistical methods
Treatment effect is measured using odds ratio (OR). For
direct comparisons, the relative effect of active interven-
tions, and the effect of active drugs versus placebo are es-
timated by within-trial comparisons. When there are two or
more trials for the same comparison, the random-effects
model is used to quantitatively combine results of individ-
ual trials, weighted by the inverse of corresponding vari-
ances. Heterogeneity across studies and asymmetry of
funnel plot in meta-analyses are statistically tested [24,25].
The adjusted indirect comparison uses the method sug-
gested by Bucher et al. in which the indirect comparison
of the new and conventional interventions is calculated
from the results of their direct comparisons with a common
intervention control (e.g., placebo) [1]. According to find-
ings from a study of methods for indirect comparison, stan-
dard deviations of treatment effects estimated in adjusted
indirect comparison were on average underestimated by
the fixed effect model but overestimated by the random-
effects model [3]. We use the random-effects model for
meta-analyses involved in adjusted indirect comparison to
prevent the underestimation of standard errors (SEs) in ad-
justed indirect comparison.
The discrepancy between the direct estimate (logOR,
TAB) and the adjusted indirect estimate (T0AB) is measured
by the difference (D) between the two estimates:
0
D 5 TAB  TAB :
Its SE is
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
 0 2
2
SEðDÞ 5 SEðTAB Þ þSE TAB ;
in which SE(TAB) and SE(T0AB) are the estimated SEs for
the direct estimate and the adjusted indirect estimate, re-
spectively. The estimated discrepancy could be standard-
ized by its SE to obtain a value of z5D/SE(D). The
discrepancy (D) is in fact the log ratio of odds ratios
(ROR). ROR has been used in methodological studies to
measure bias in controlled trials [8,10]. In this paper,
ROR ! 1.0 (or D ! 0) indicates that the treatment effect
estimated by the direct comparison is greater than that by
the adjusted indirect comparison.
Some trials may have three or more arms, including a new
drug, an old drug, and a placebo arm. Data from these trials
could be used to directly compare the new drug and the old
 F. Song et al. / Journal of Clinical Epidemiology 61 (2008) 455e463
drug, and could also be used to indirectly compare the new
and the old drug adjusted by the placebo control. In this case,
the point estimate of discrepancy between the direct and ad-
justed indirect estimate is zero. That is, when the same data
are used in both the direct and adjusted indirect comparison,
the discrepancy will be underestimated. To ensure that direct
estimates and adjusted indirect estimates are independent, as
assumed when calculating the discrepancy, any placebo-
controlled trials that directly compared a new drug and an
old drug are used only in direct comparison.
3. Results
The discrepancies in estimated effects between direct
and adjusted indirect comparisons for all the three case
studies, including the results of post hoc sensitivity analy-
ses are shown in Table 1. Details about individual case
studies are provided below.
3.1. Case-1: bupropion versus NRT for smoking
cessation
We conducted a review of bupropion and NRT for smok-
ing cessation in 2001 [11]. In only one trial, bupropion has
been directly compared with conventional NRT therapy
[12]. Two recently updated Cochrane systematic reviews
[16,17] were used to identify relevant placebo-controlled
trials for the adjusted indirect comparison. After excluding
one trial that directly compared bupropion and NRT patch,
there are nine placebo-controlled trials of bupropion and 19
placebo-controlled trials of NRT patch.
Fig. 1 shows results by different comparisons. The single
head-to-head comparison trial found that the proportion of
patients who failed to give up smoking at 12 months is statis-
tically significantly lower in the bupropion group than in the
NRT patches group (OR 0.48, 95% confidence interval
[CI] 5 0.28e0.82) [12]. When the two active drugs are sep-
arately compared to placebo, the effect of bupropion (OR
0.51, 95% CI 5 0.36e0.73) is similar to that of NRT patch
(OR 0.57, 95% CI 5 0.48e0.67) (Fig. 1). Consequently,
the adjusted indirect comparison found no difference be-
tween bupropion and NRT patch (OR 0.90, 95%
CI 5 0.61e1.34). The OR of bupropion versus NRT esti-
mated by the direct comparison is 47% greater than that by
the adjusted indirect comparison (ROR 0.53, 95%
CI 5 0.28e1.03; P 5 0.06) (Table 1).
There are no significant differences in effect for different
NRT patch regimens [16], and the results of the 19 NRT patch
trials are not significantly heterogeneous (I2 5 12%;
P 5 0.30). However, the results of the nine bupropion trials
are statistically significantly heterogeneous (I2 5 54%;
P 5 0.03). This statistical heterogeneity could be reduced
by using trials that recruited only community volunteers
(I2 5 25%; P 5 0.25). Egger’s test indicates that smaller pla-
cebo-controlled trials of NRT patch are associated with
457
greater treatment effect (Fig. 1). We conducted sensitivity
analyses using trials that included only community volunteers,
or with intensive support, or only large trials (N O 200). The
discrepancies between the direct and indirect estimates are
almost unaffected in the sensitivity analyses (Table 1).
3.2. Case-2: risperidone versus haloperidol
for schizophrenia
Risperidone was directly compared with conventional
antipsychotics (mostly haloperidol) in a Cochrane system-
atic review [18], and haloperidol was compared with pla-
cebo in another Cochrane systematic review [19] for the
treatment of schizophrenia. We searched MEDLINE and
identified three trials that compared risperidone with pla-
cebo for schizophrenia.
A pooling of direct comparison trials shows that risper-
idone is associated with fewer patients leaving the study
early (OR 0.74, 95% CI 5 0.60e0.91) and fewer patients
not showing clinical improvement (OR 0.77, 95%
CI 5 0.58e1.02) (Fig. 2). However, the results of adjusted
indirect comparison shows that risperidone is not superior
to haloperidol in terms of total dropouts (OR 1.43; 95%
CI 5 0.75e2.75) and clinical improvement (OR 2.21,
95% CI 5 1.04e4.92). The treatment effect of risperidone
versus haloperidol estimated by the direct comparison is
48% greater for total dropouts (ROR 0.52, 95%
CI 5 0.26e1.03) and 65% greater for clinical improvement
(ROR 0.35, 95% CI 5 0.16e0.78) than that by the adjusted
indirect comparison (Table 1).
The type of patient included, dose of drug, and treatment
duration were similar in the included trials. There is no sta-
tistically significant heterogeneity across the direct compar-
ison trials, and across the placebo-controlled trials of
haloperidol (Fig. 2). However, there is heterogeneity and
funnel plot asymmetry among the three placebo-controlled
risperidone trials, which could be explained by one trial
that used long-acting injectable risperidone [26]. After ex-
cluding this trial, results of two placebo-controlled trials of
risperidone are no longer heterogeneous (I2 5 0%), but the
discrepancy between the direct and adjusted indirect com-
parisons is increased (Table 1).
It should also be noted that the criteria for a marked clin-
ical improvement may be different across trials. In most tri-
als of risperidone, the clinical improvement was defined as
a 20% or more reduction in the Positive and Negative Syn-
drome Scale or Brief Psychiatric Rating Scale total scores.
In all placebo-controlled trials of haloperidol, it was rated
by clinicians using Clinical Global Impression or other
scales. The number of trials available is not sufficient to
conduct a sensitivity analysis about different criteria for
clinical improvement.
The doses of haloperidol used in many direct compari-
son trials are high [27], and it has been shown that beyond
a certain threshold high doses of antipsychotics increase the
risk of adverse reactions without additional therapeutic
458 F. Song et al. / Journal of Clinical Epidemiology 61 (2008) 455e463

Table 1
Discrepancy between the direct and adjusted indirect comparisons in three case studies: main results and sensitivity analyses
Number of trials (patients)a
Direct Indirect ROR (95% CI)b
Bupropion versus NRT for smoking cessation
Failed to give up smoking at 12 months
All trials 1 (488) 9 (3,544)/19 (9,598) 0.53 (0.28e1.03)c
Community volunteers only 1 (488) 6 (2,642)/9 (5,149) 0.47 (0.24e0.94)d
Intensive support trials 1 (488) 9 (3,544)/11 (4,665) 0.54 (0.28e1.06)c
NRT trials N O 300 1 (488) 9 (3,544)/5 (7,135) 0.58 (0.30e1.12)
NRT trials since 1992 1 (488) 9 (3,544)/14 (8,805) 0.55 (0.28e1.06)c

Risperidone versus haloperidol for schizophrenia

Dropouts
All trials 13 (2,539) 3 (848)/11 (628) 0.52 (0.26e1.03)c
Excluding injectable risperidone trial 13 (2,539) 2 (448)/11 (628) 0.40 (0.22e0.73)d
Haloperidol maximal dose O10 mg/d 11 (1,142) 3 (848)/9 (516) 0.43 (0.21e0.89)d
Haloperidol maximal dose <10 mg/d 2 (1,397) 3 (848)/2 (112) 3.96 (0.17e91.55)
Haloperidol trials since 1997 13 (2,539) 3 (848)/2 (139) 0.69 (0.28e1.69)
Lack of clinical improvement
All trials 10 (2,206) 3 (848)/9 (374) 0.35 (0.16e0.78)d
Excluding injectable risperidone trial 10 (2,206) 2 (448)/9 (374) 0.28 (0.13e0.62)d
Haloperidol maximal doseO10 mg/d 8 (809) 3 (848)/6 (238) 0.31 (0.13e0.74)d
Haloperidol maximal dose <10 mg/d 2 (1,397) 3 (848)/3 (136) 0.23 (0.03e2.06)
Haloperidol trials since 1980 10 (2,206) 3 (848)/3 (109) 0.16 (0.05e0.50)d

Fluoxetine versus Imipramine for depression

Dropouts
All trials 13 (1,289) 14 (2,144)/39 (4,830) 0.78 (0.46e1.33)
Major depression only 11 (1,198) 10 (1,754)/18 (2,670) 0.77 (0.42e1.40)
Fluoxetine 60e80 mg/d 12 (1,257) 4 (412)/39 (4,830) 0.63 (0.33e1.22)
Outpatient trials only 8 (912) 9 (1,482)/31 (4,229) 0.64 (0.35e1.17)
Imipramine trials N O 200 6 (942) 14 (2,144) 7 (2,416) 0.79 (0.45e1.40)
Imipramine trials since 1987 13 (1,289) 14 (2,144)/19 (2,693) 0.86 (0.50e1.47)
Lack of clinical improvement
All trials 9 (1,049) 12 (1,916)/39 (4,862) 0.66 (0.41e1.07)c
Major depression only 9 (1,049) 8 (1,526)/16 (2,559) 0.63 (0.38e1.04)c
Fluoxetine 60e80 mg/d 9 (1,049) 3 (328)/39 (4,862) 0.58 (0.31e1.08)c
Outpatients trials only 6 (673) 8 (1,398)/30 (4,249) 0.60 (0.39e0.94)d
Impramine trials N O 200 5 (823) 12 (1,916)/7 (2,416) 0.71 (0.42e1.20)
Imipramine trials since 1987 9 (1,049) 12 (1,916)/17 (2,605) 0.70 (0.43e1.15)
a
For indirect comparison, number of new drug trials/old drug trials.
b
ROR ! 1 indicates that the treatment effects estimated by the direct comparison are greater than that by the adjusted indirect comparison.
c
0.05 < P ! 0.10.
d
P ! 0.05.

benefit [28,29]. In two direct comparison trials and three
placebo-controlled haloperidol trials, the maximal daily
dose of haloperidol was lower than 10 mg. Only a few trials
used lower doses of haloperidol (<10 mg/d) so that the re-
sults of sensitivity analysis of low dose haloperidol are lack
of precision, with wider CIs for both dropouts and clinical
improvement (Table 1). The result remains unchanged by
using trials in which the maximal dose of haloperidol was
greater than 10 mg per day (Table 1).
3.3. Case-3: fluoxetine versus imipramine for depressive
disorders
In several Cochrane systematic reviews, selective seroto-
nin reuptake inhibitors (SSRIs) have been directly
compared with either conventional antidepressants [20,21]
or placebo [22,23] for depressive disorders. In this case
study, the effects of fluoxetine (SSRI) and imipramine (a
conventional antidepressant) are compared. We searched
MEDLINE and references of review articles to identify ad-
ditional placebo-controlled trials of fluoxetine or imipra-
mine for depression.
Pooling results from 13 direct comparison trials indicate
that the proportion of patients who left trials early is statis-
tically nonsignificantly lower in the fluoxetine group than in
the imipramine group (OR 0.73; 95% CI 5 0.48e1.12)
(Fig. 3). In placebo-controlled trials, there is no significant
difference in the proportion of patients who left trials early
between fluoxetine and placebo (OR 0.92, 95%
CI 5 0.70e1.22) and between imipramine and placebo
 F. Song et al. / Journal of Clinical Epidemiology 61 (2008) 455e463 459

Odds ratio (95% CI)

0.1 1 10

Heterogeneity Funnel plot

Smoking at 12 months Estimate I2% (p value) Egger’s test

Bupropion v Placebo 0.51 (0.36 to 0.73) 54% (0.03) p=0.47

NRT patch v Placebo 0.57 (0.48 to 0.67) 12% (0.30) p=0.04

Adjusted indirect 0.90 (0.61 to 1.34)

Direct comparison 0.48 (0.28 to 0.82) Only one trial

Fig. 1. Results of dire Himmel ct and adjusted indirect comparisons of bupropion versus nicotine replacement therapy (NRT patch) for smoking cessation.

(OR 0.98, 95% CI 5 0.84e1.14). The difference in total
dropout between the two groups is 22% greater by the in-
direct comparison than by the adjusted indirect comparison
(ROR 0.78, 95% CI 5 0.46e1.33), although the discrep-
ancy is not statistically significant (P 5 0.36) (Table 1).
Direct comparison trials found no significant difference
between the fluoxetine and the imipramine group in the
number of patients not showing marked clinical improve-
ment (OR 0.89, 95% CI 5 0.61e1.31) (Fig. 3). However,
the adjusted indirect comparison indicates that fluoxetine
is statistically significantly less effective than imipramine
in terms of clinical improvement (OR 1.35, 95%
CI 5 1.01e1.82) (Fig. 3). The relative effect of fluoxetine
for clinical improvement estimated by the direct compari-
son is 34% greater than that by the adjusted indirect com-
parison (ROR 0.66, 95% CI 5 0.41e1.07) (Table 1).
There are statistically significant heterogeneity in results
across trials, and the funnel plot of dropout results of pla-
cebo-controlled imipramine trials are asymmetric (Egger’s
test P 5 0.05) (Fig. 3). We conducted sensitivity analyses
in terms of diagnostic criteria for depressive disorders (ma-
jor depression only), dose of fluoxetine (60e80 mg daily),

Odds ratio (95% CI)

0.1 1 10

Heterogeneity Funnel plot

Total dropouts Estimate I 2% (p value) Egger’s test

Risperidone v Placebo 0.75 (0.45 to 1.24) 62% (0.07) p=0.02

Haloperidol v Placebo 0.52 (0.35 to 0.79) 0% (0.57) p=0.97

Adjusted indirect 1.43 (0.75 to 2.75)

Direct comparison 0.74 (0.60 to 0.91) 0% (0.45) p=0.04

Not clinically improved

Risperidone v Placebo
0.40 (0.26 to 0.62) 37% (0.20) p=0.74

Haloperidol v Placebo 0.18 (0.10 to 0.34) 11% (0.34) p=0.30

Adjusted indirect 2.22 (1.04 to 4.72)

Direct comparison 0.77 (0.58 to 1.02) 14% (0.31) p=0.19

Fig. 2. Results of direct and adjusted indirect comparisons of risperidone versus haloperidol for schizophrenia.
460 F. Song et al. / Journal of Clinical Epidemiology 61 (2008) 455e463
Odds ratio (95% CI)
0.1 1 10
Total dropouts
Fluoxetine v Placebo
Imipramine v Placebo
Adjusted indirect
Direct comparison
Not clinically improved
Fluoxetine v Placebo
Imipraminel v Placebo
Adjusted indirect
Direct comparison
Fig. 3. Results of direct and adjusted indirect comparisons of fluoxetine versus imipramine for depressive disorders.
setting of care (outpatients only), and size of imipramine
trials. However, the observed heterogeneity across trials
could not be satisfactorily explained by these study-level
variables. Sensitivity analyses provide results that are simi-
lar to the overall analyses in terms of discrepancies between
the direct and adjusted indirect comparisons (Table 1).
4. Discussion
In the three case studies, the treatment effects of new
drugs estimated by head-to-head comparison trials tend to
be greater than those from adjusted indirect comparisons, al-
though observed discrepancies may not be statistically sig-
nificant in individual cases (Table 1). There are several
possible explanations for this observed phenomenon. These
include the play of chance, bias in head-to-head comparison
trials, bias in adjusted indirect comparisons, and clinically
meaningful heterogeneity across different sets of trials.
4.1. Play of chance
According to findings from a previous study [3,6], the
direction of discrepancies between adjusted indirect esti-
mates and direct estimates was unpredictable. That is, ad-
justed indirect comparisons might by chance produce
greater or smaller effect sizes than direct comparisons. It
is possible that we acquired cases in which the relative ef-
fects of new drugs estimated by the direct comparison were
Heterogeneity Funnel plot
Estimate I 2% (p value) Egger’s test
0.92 (0.70 to 1.22) 36% (0.09) p=0.50
0.98 (0.84 to 1.14) 22% (0.12) p=0.05
0.94 (0.69 to 1.30)
0.73 (0.48 to 1.12) 61% (0.00) p=0.17
0.55 (0.44 to 0.68) 12% (0.33) p=0.26
0.41 (0.33 to 0.49) 52% (0.00) p=0.47
1.35 (1.01 to 1.82)
0.89 (0.61 to 1.31) 50% (0.04) p=0.24
greater than that by indirect comparison by chance. How-
ever, the direction of possible bias in head-to-head compar-
ison trials used in the current study is relatively clear due to
‘‘optimism bias’’ in favor of new drugs [13], whereas it was
mostly difficult to specify in the previous study [3,6]. In ad-
dition, the hypothesis that the adjusted indirect comparison
may be less biased than the direct comparison in some cir-
cumstances was actually conceived after examining the first
case of bupropion and NRT for smoking cessation. The dis-
crepancy between the direct and adjusted indirect compari-
sons was in the same direction in further two additional
cases. The play of chance thus becomes a less satisfactory ex-
planation, and other possible causes should be considered.
4.2. Bias in head-to-head comparison trials
Bias may be introduced into randomized controlled tri-
als through many different (overt or hidden) ways due to,
for example, lack of allocation concealment, inadequate
blinding, and imbalanced withdrawals [7,8,30]. Bias in
head-to-head trials might not always be the result of poor
methodological quality research. It may be very hard to
avoid; for example, if people prefer the newer of two treat-
ments in circumstances where outcomes are influenced by
this preference and in a context where full masking cannot
be achieved. Even when trials are double blind, new drugs
may have different adverse effects from conventional drugs,
and it has been shown that this scenario may provide clini-
cians with a clue to the patient’s treatment and hence might
 F. Song et al. / Journal of Clinical Epidemiology 61 (2008) 455e463
bias results [31]. This might lead to bias as a result of prior
expectation that the new treatment will be more efficacious,
so called optimism bias [13]. In our second and third case
studies, the conventional drugs (haloperidol and imipra-
mine) have specific adverse effects that are qualitatively
different from those of the new drugs such that blinding
may have been undermined.
4.3. Bias in adjusted indirect comparison
The key assumption in adjusted indirect comparison is
that results of trials involved in the adjusted indirect com-
parison are exchangeable across different sets of trials in-
volved in the comparison [1,3]. In the case studies, we
have attempted to gather trials that are as homogeneous
as possible in terms of patients included, interventions in-
vestigated, and outcomes measured. Although heterogene-
ity was found in some of the meta-analyses in our case
studies, the results of sensitivity analyses were not signifi-
cantly different from the main analyses (Table 1).
Adjusted indirect comparison is statistically much less
precise than direct comparison trials. The result of adjusted
indirect comparison using four trials is statistically (mea-
sured by SEs) as precise as the result of a single direct com-
parison trial, given the same sample size in each of the
trials involved [3]. In addition, there is further uncertainty
due to confounding factors and heterogeneity in trials that
will affect adjusted indirect comparison. Therefore, results
of adjusted indirect comparisons will theoretically tend to
be more unpredictable and extreme than that of direct com-
parisons. However, lack of precision by adjusted indirect
comparison could not be used to satisfactorily explain the
consistent discrepancies between the adjusted indirect com-
parison and direct comparison in the three case studies.
Placebo-controlled trials used in adjusted indirect com-
parisons may also be biased. To examine the impact of bias
in placebo-controlled trials on the results of adjusted indi-
rect comparison, we conducted a simple numerical simula-
tion. Assume that RORAC is the level of bias in placebo-
controlled trials of new drugs, and RORBC is the level of
bias in placebo-controlled trials of conventional drugs
(for instance, RORAC 5 0.8 indicates that the effect of
a new drug versus placebo is overestimated by 20%). Then
bias of adjusted indirect comparison of A versus B will
mathematically be RORAC/RORBC. Fig. 4 shows results
of simulations to investigate the impact of bias in pla-
cebo-controlled trials on adjusted indirect comparisons. If
placebo-controlled trials of new drugs are biased to the
same extent as placebo trials of conventional drugs, ad-
justed indirect comparison will counterbalance such bias
so that the estimated effect of the new drug versus the con-
ventional drug by adjusted indirect comparison will no lon-
ger be biased (Fig. 4).
When placebo-controlled trials of the new drug are less
(or more) biased than placebo-controlled trials of the con-
ventional drug, the adjusted indirect comparison may
461
underestimate (or exaggerate) the effect of the new drug.
For example, when the results are biased by 10% (i.e.,
RORAC 5 0.9) in trials of the new drug and by 15%
(RORBC 5 0.85) in trials of the conventional drug, the ad-
justed indirect comparison will underestimate the effect of
the new drug by 6% (i.e., 1.06 5 0.90/0.85). Even so, ad-
justed indirect comparison may still be able to yield an esti-
mate that is less biased than the direct comparison, depending
on the scale of bias in head-to-head comparison trials.
Placebo-controlled trials of conventional drugs were of-
ten conducted many years before trials of newer drugs. The
quality of trials may improve over time. If trials of new
treatments conducted more recently are less biased than
earlier trials of conventional treatments, adjusted indirect
comparison would underestimate the effect of new treat-
ments. For the three case studies, we excluded any early tri-
als in which intention-to-treat analysis was impossible
because of lack of data on withdrawals. The quality of pla-
cebo-controlled trials of the new drugs is not different from
placebo-controlled trials of the conventional drugs in terms
of adequate allocation concealment.
Noting that bias over time might still be possible, we
conducted additional post hoc sensitivity analyses using on-
ly placebo-controlled trials of conventional drugs that were
published during the same period as trials of new drugs.
The results of these sensitivity analyses are similar to re-
sults from the main analyses (Table 1). This suggests that
the indirect comparisons were not simply manifesting time
bias, and along with the genuine risk of bias in the head-to-
head comparisons, supports our hypothesis that indirect
comparisons may actually correct for bias.
4.4. Clinically meaningful heterogeneity
This paper focuses on the validity of direct and adjusted
indirect comparisons. However, it may also be plausible
that discrepancies between direct and adjusted indirect
1.8
Bias in placebo-controlled
1.6
comparison (RORAB-ind)
Bias in adjusted indirect
1.1 trials of new drugs (RORAC)
1.0
1.4
0.8
1.2 0.6
1.0
0.8
0.6
0.4
0.4 0.6 0.8 1.0 1.2
Bias in placebo-controlled trials
of conventional drugs (RORBC)
Fig. 4. The impact of bias in placebo-controlled trials on the extent of bias
in adjusted indirect comparisons (ROR ! 1.0 indicates that the treatment
effect is overestimated, and ROR O 1.0 indicates that the effect is under-
estimated. Results of four levels of bias [RORAC 5 1.1, 1.0, 0.8, and 0.6,
respectively] in placebo-controlled trials of new drugs are presented).
462 F. Song et al. / Journal of Clinical Epidemiology 61 (2008) 455e463
comparisons could be explained by some clinically mean-
ingful heterogeneity. For example, patients entered in trials
of new drugs are likely to be those who had responded
poorly to conventional interventions. Assuming that the
poor response to the conventional drug would not affect
the patient’s response to the new drug, the results of direct
comparison trials may be only applicable to patients who
have failed to respond to the conventional drug, whereas
the results of adjusted indirect comparison may be more ap-
plicable to general patients. This kind of information can be
used by clinicians to decide when and for whom the new
drug should be used.
4.5. Implications
We focus on the comparison of new and conventional
drugs, because the direction of possible bias in trials is rel-
atively clear as a consequence of optimism bias [13]. Find-
ings presented in this paper may also be relevant to the
evaluation of nonpharmaceutical interventions, and the
common control intervention used in adjusted indirect com-
parison may not necessarily be placebo.
According to generally agreed recommendations [1,3],
we should accept the results from randomized direct com-
parison trials and dismiss conflicting evidence from indirect
approaches. The findings presented in this paper suggest
that adjusted indirect comparison could be used to chal-
lenge and cross-examine the results of head-to-head ran-
domized trials. Therefore, the use of adjusted indirect
comparison should be extended to cases where direct com-
parison trials are plentiful, particularly for the evaluation of
new interventions. Discrepancies between direct and ad-
justed indirect comparisons should be carefully investigated
for possible explanations, including methodological inade-
quacies and possible clinically meaningful heterogeneity
across trials.
It should be stressed that we are not arguing against the
importance of randomized trials. After all, adjusted indirect
comparison relies on randomized trials that compare active
interventions with a common comparator. Adjusted indirect
comparison is a logical extension of utilizing data from ran-
domized trials.
Acknowledgments
There is no specific funding for this work. Fujian Song
and Ian Harvey are employees of the University of East An-
glia, Norwich, UK. Richard Lilford is partially supported
by the UK NHS R&D Methodology Programme.
References
[1] Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct
and indirect treatment comparisons in meta-analysis of randomized
controlled trials. J Clin Epidemiol 1997;50:683e91.
[2] Song F, Glenny AM, Altman DG. Indirect comparison in evaluating
relative efficacy illustrated by antimicrobial prophylaxis in colorectal
surgery. Control Clin Trials 2000;21:488e97.
[3] Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ,
D’amico R, et al. Indirect comparisons of competing interventions.
Health Technol Assess 2005;9:1e144.
[4] Higgins JP, Whitehead A. Borrowing strength from external trials in
a meta-analysis. Stat Med 1996;15:2733e49.
[5] Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of
multiple treatments: combining direct and indirect evidence. BMJ
2005;331:897e900.
[6] Song F, Altman DG, Glenny AM, Deeks JJ. Validity of indirect com-
parison for estimating efficacy of competing interventions: empirical
evidence from published meta-analyses. BMJ 2003;326:472e5.
[7] Gotzsche PC. Methodology and overt and hidden bias in reports of
196 double-blind trials of nonsteroidal anti-inflammatory drugs in
rheumatoid arthritis. Control Clin Trials 1989;10:31e56.
[8] Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence
of bias: dimensions of methodological quality associated with esti-
mates of treatment effects in controlled trials. JAMA 1995;273:
408e12.
[9] Moher D, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P.
Does quality of reports of randomised trials affect estimates of in-
tervention efficacy reported in meta-analyses? Lancet 1998;352:
609e13.
[10] Juni P, Altman D, Egger M. Systematic reviews in health cared
assessing the quality of controlled clinical trials. BMJ 2001;323:
42e6.
[11] Song F, Raftery J, Aveyard P, Hyde C, Barton P, Woolacott N. Cost-
effectiveness of pharmacological interventions for smoking cessa-
tion: a literature review and a decision analytic analysis. Med Decis
Making 2002;22:S26e37.
[12] Jorenby DE, Leischow SJ, Nides MA, Rennard SI, Johnston JA,
Hughes AR, et al. A controlled trial of sustained-release bupropion,
a nicotine patch, or both for smoking cessation. N Engl J Med
1999;340:685e91.
[13] Chalmers I, Matthews R. What are the implications of optimism bias
in clinical research? Lancet 2006;367:449e50.
[14] Song F. Risperidone in the treatment of schizophrenia: a meta-analy-
sis of randomized controlled trials. J Psychopharmacol 1997;11:
65e71.
[15] Song F, Freemantle N, Sheldon TA, House A, Watson P, Long A,
et al. Selective serotonin reuptake inhibitors: meta-analysis of effi-
cacy and acceptability. BMJ 1993;306:683e7.
[16] Silagy C, Lancaster T, Stead L, Mant D, Fowler G. Nicotine replace-
ment therapy for smoking cessation. Cochrane Database Syst Rev
2004;CD000146.
[17] Hughes J, Stead L, Lancaster T. Antidepressants for smoking cessa-
tion. Cochrane Database Syst Rev 2004;CD000031.
[18] Hunter RH, Joy CB, Kennedy E, Gilbody SM, Song F. Risperidone
versus typical antipsychotic medication for schizophrenia. Cochrane
Database Syst Rev 2003;CD000440.
[19] Joy CB, Adams CE, Lawrie SM. Haloperidol versus placebo for
schizophrenia. Cochrane Database Syst Rev 2001;CD003082.
[20] Barbui C, Hotopf M, Freemantle N, Boynton J, Churchill R,
Eccles MP, et al. Selective serotonin reuptake inhibitors versus tricy-
clic and heterocyclic antidepressants: comparison of drug adherence.
Cochrane Database Syst Rev 2000;CD002791.
[21] Geddes JR, Freemantle N, Mason J, Eccles MP, Boynton J. SSRIs
versus other antidepressants for depressive disorder. Cochrane Data-
base Syst Rev 2000;CD001851.
[22] Lima MS, Moncrieff J. A comparison of drugs versus placebo for the
treatment of dysthymia. Cochrane Database Syst Rev 2000;
CD001130.
[23] Wilson K, Mottram P, Sivanranthan A, Nightingale A. Antidepressant
versus placebo for depressed elderly. Cochrane Database Syst Rev
2001;CD000561.
 F. Song et al. / Journal of Clinical Epidemiology 61 (2008) 455e463
[24] Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis
detected by a simple, graphical test. BMJ 1997;315:629e34.
[25] Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-anal-
ysis. Stat Med 2002;21:1539e58.
[26] Kane JM, Eerdekens M, Lindenmayer JP, Keith SJ, Lesem M, Karcher K.
Long-acting injectable risperidone: efficacy and safety of the first long--
acting atypical antipsychotic. Am J Psychiatry 2003;160:1125e32.
[27] Safer DJ. Design and reporting modifications in industry-sponsored
comparative psychopharmacology trials. J Nerv Ment Dis
2002;190:583e92.
463
[28] Bollini P, Pampallona S, Orza MJ, Adams ME, Chalmers TC. Anti-
psychotic drugs: is more worse? A meta-analysis of the published
randomized control trials. Psychol Med 1994;24:307e16.
[29] Waraich PS, Adams CE, Roque M, Hamill KM, Marti J. Haloperidol
dose for the acute phase of schizophrenia. Cochrane Database Syst
Rev 2002;CD001951.
[30] Kaptchuk T. The double-blind, randomized, placebo-controlled trial:
gold standard or golden calf? J Clin Epidemiol 2001;54:541e9.
[31] Thomson R. Side effects and placebo amplification. Br J Psychiatry
1982;140:64e8.