Frontiers_Rinchuse.

qxd 5/7/08 2:18 PM Page 167

Frontiers IN CLINICAL RESEARCH

Donald J. Rinchuse, DMD,

MS, MDS, PhD1 DECONSTRUCTING EVIDENCE IN
Daniel J. Rinchuse, DMD, ORTHODONTICS: MAKING SENSE OF
MS, MDS, PhD1

Sanjivan Kandasamy,
SYSTEMATIC REVIEWS, RANDOMIZED
BDSc, BScDent, DocClin-
Dent, MOrthRCS2
CLINICAL TRIALS, AND META-ANALYSES
Marc B. Ackerman, DMD3 Much has been written in support of systematic reviews and the ran-
domized clinical trials and meta-analyses upon which they are based.
Clearly, the medical profession (as opposed to the dental profession)
has been the leader in publishing the benefits of systematic reviews
over the traditional, qualitative narrative reviews. At the same time,
the medical profession also appears to be ahead of the dental profes-
sion in recognizing the limitations of such reviews. That said, there are
a number of inherent problems with systematic reviews, as well as the
randomized clinical trials and meta-analyses that back them up. To
better facilitate evidence-based decision making, this article discusses
the shortcomings of systematic reviews so that practitioners are fully
aware of their drawbacks, as well as their benefits. World J Orthod
2008;9:167–176.

1Clinical Professor, Department of

Orthodontics and Dentofacial Ortho-
pedics, University of Pittsburgh
School of Dental Medicine, Pitts-
burgh, Pennsylvania, USA.
2Senior Lecturer, Department of
Orthodontics, School of Dentistry,
University of Western Australia,
Perth, Australia; private orthodontic
practice, Perth, Australia.
3Associate Professor and Director,
Fellowship in Orthodontic Clinical
Research, Jacksonville University
School of Orthodontics, Jacksonville,
Florida, USA.
Correspondence
Dr Sanjivan Kandasamy
University of Western Australia
17 Monash Avenue
Nedlands, Western Australia 6009
Australia
E-mail: sanj@kandasamy.com.au
t has been said that the 20th century
I was the age of science, while the 21st
century ushered in the age of evidence.1
Evidence-based dentistry is defined as
“an approach to oral health care that
requires the judicious integration of sys-
temic assessments of clinically relevant
scientific evidence relating to the
patient’s oral and medical condition and
history, with the dentist’s clinical exper-
tise and patient’s treatment needs and
preference.”2 The ultimate goal of evi-
dence-based dentistry is for clinicians to
provide the best possible care to
patients. It is difficult for clinicians to
read and assimilate into their practice
the information from the many articles
published each year. Therefore, it takes
a systematic approach to summarize
the large volume of literature. Further,
the evidence-based approach argues
that lower-level evidence, such as expert
opinions and even case studies, is “not
sufficiently strong or credible evidence
in the decision-making process.”2 The
best evidence is considered that which
is the “least biased in terms of design,
analyses, or interpretation.”2 The least
biased evidence is said to come from
prospective, longitudinal, double-blind
studies utilizing randomized clinical
trials; this is considered the gold stan-
dard by which new treatments are best
evaluated.

COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO 167
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
Frontiers_Rinchuse.qxd 5/7/08 2:18 PM Page 168
FRONTIERS IN CLINICAL RESEARCH
HIERARCHY OF EVIDENCE
(Based on the quality of research)
Systematic reviews based on RCTs (highest level)
Meta-analyses
RCTs
Non-RCTs
Cohort studies
Case-control studies
Cross-over studies
Cross-sectional studies
Case studies
Consensus opinion of experts
Anecdotal reports and testimonies
According to the evidence-based para-
digm, there exists a hierarchy of evi-
dence with systematic reviews utilizing
randomized clinical trials at the top (ie,
the best); the experience-based view that
includes testimonies and anecdotal
reports is at the lowest level 2 (Fig 1).
There are 3 hierarchical models for evi-
dence-based decision-making as sug-
gested by Ismail and Bader.2 At the low-
est level is model level No. 1, which
consists of evidence derived solely from
the practitioner’s experience. Although
model level No. 1 is considered within
the framework of the evidence-based
dentistry paradigm, it is really not an evi-
dentiary approach. Model level No. 2 is
the next highest level and is evidence
gleaned from both the clinical experience
of the practitioner, but more importantly,
knowledge and information acquired
from a review of selected published arti-
cles and research. The literature search
in the model No. 2 approach is not
exhaustive and may be as limited as a
cursory reading of several publications
readily available to the clinician. A major
limitation of this approach is that it
requires a constant search for evidence.
Of note, there can be quite a range and
disparity in the quality of the evidence in
model level No. 2 clinical decision-mak-
ing. For instance, a clinician can review
only several articles while a more diligent
practitioner may do an online search
168 COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
WORLD JOURNAL OF ORTHODONTICS
Fig 1 Hierarchy of evidence with the high-
est and strongest evidence at the “top.”
RCTs (randomized clinical trials).
Evidence-based
Experience-based
using various scientific databases such
as PubMed. Nonetheless, in the litera-
ture review in model level No. 2, evi-
dence is in a narrative format and the
interpretations and evaluation of the arti-
cles are qualitative. Model level No. 3 is
the highest level of evidence and
involves a systematic review of the litera-
ture that incorporates an exhaustive
search for the best scientific evidence
(eg, PubMed, Medline, Cochrane Library),
followed by the application of a quantita-
tive, statistical analysis: the meta-analy-
sis. Again, the best scientific evidence is
considered to be derived from random-
ized clinical trials.2
According to Shekelle et al,3 there is a
6-level hierarchy of evidence:
Ia: Evidence from systematic reviews of
randomized clinical trials
Ib: Evidence from at least 1 randomized
clinical trial
IIa: Evidence from at least 1 controlled
study with randomization
IIb: Evidence from at least 1 other type of
quasiexperimental study
III: Evidence from nonexperimental
descriptive studies, such as compara-
tive studies, correlation studies, and
case-control studies
IV: Evidence from expert committee
reports or opinions or clinical experi-
ence of respected authorities.
Frontiers_Rinchuse.qxd 5/7/08
VOLUME 9, NUMBER 2, 2008
COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
2:18 PM Page 169
In an attempt to facilitate evidence-
based decision making, guidelines such
as CONSORT (Consolidated Standards of
Reporting Trials), QUOROM (Quality of
Reporting of Meta-analyses), and MOOSE
(Meta-analysis of observational studies in
epidemiology) were established by a
team of dedicated journal editors, epi-
demiologists, and statisticians. 4,5 For
sure, evidence-based dentistry does not
suggest that orthodontists consult only
systematic reviews and papers that
report randomized clinical trials. There
still exists in the evidence-based para-
digm a useful place for traditional, narra-
tive, qualitative literature reviews.
The value of systematic reviews and
randomized clinical trials have been well-
documented.2,3,5 And the present authors
clearly support the modern-day evidence-
based paradigm including systematic
reviews. Nonetheless, there are some
inherent problems and limitations of
some systematic reviews and randomized
clinical trials that are not as well-publi-
cized and recognized in dentistry as they
are in medicine. The purpose of this arti-
cle is to point out and discuss some of the
shortcomings of systematic reviews, ran-
domized clinical trials, and meta-analyses
(fully aware and appreciative of the bene-
fits of such reviews). A recurrent theme of
this paper is that orthodontic practitioners
and journal editors alike should not erro-
neously accept as holy writ the conclu-
sions of all systematic reviews.
SYSTEMATIC REVIEWS AND
RANDOMIZED CLINICAL TRIALS
A systematic review is a critical review of
primary studies that involves the selec-
tion of only those studies that have used
explicit and reproducible methods and
then applies an appropriate statistical
technique that quantitatively appraises
the chosen studies.6 As previously men-
tioned, we ardently support science and
the current evidence-based decision-
making paradigm, including randomized
clinical trials and systematic reviews.
With this said, we are also very much
aware of the limitations of randomized
RINCHUSE ET AL
clinical trials, meta-analyses, and system-
atic reviews. For example, Dietrich 7
argued that “the conduct of high-quality
RCTs may not be a realistic goal for many
dental interventions and procedures.
Well-designed observational studies may
be a viable, cost-efficient alternative.”
Likewise, Keim,8 editor of the Journal of
Clinical Orthodontics, as well as Williams
and Garner,9 have challenged the obsti-
nate adherence to evidence-based sys-
tematic reviews because of the very
restrictive inclusion criteria used while
ignoring other valid sources of knowl-
edge and clinical experience. As noted by
Johnston,10 “ . . . I doubt that we need to
wait 20 years for an ostensibly perfect
answer from a randomized clinical trial
[RCT] that in the end will be diminished
by sample attrition, ‘Hawthorne effect,’
lack of blinding, ‘data peeking,’ and other
shortcomings that are intrinsic to ortho-
dontic versions of the medical trials we
seek to imitate.”
There are other limitations of systematic
reviews. First, Flores-Mir et al11 found that
the search and selection methods of cur-
rent systematic reviews in orthodontics (ie,
from 2000 to 2004) are limited in that key
methodological components are frequently
absent or not appropriately described. For
the 16 orthodontic systematic reviews for
this time period, many failed to search
more than Medline (56%), failed to docu-
ment the database names and search
dates (37%), failed to document the search
strategy (62%), did not use several experts
to select studies (75%), and did not include
all languages (81%).11
This same research group also
assessed dental systematic reviews pub-
lished for time periods between January
1, 2000, and July 14, 2005, and found
that many failed to search more than
Medline (20%), document the search
strategy (20%), use multiple reviewers for
selecting studies (25%), and include all
languages (39%).12 On the plus side, in
2005, most systematic reviews docu-
mented database names and search
dates (90%), used electronic search
terms (95%) and appropriate inclusion-
exclusion criteria (95%), and employed
secondary searches (100%). Parentheti-
169
Frontiers_Rinchuse.qxd 5/7/08 2:18 PM
FRONTIERS IN CLINICAL RESEARCH
cally, search and selection methods were
best for the most current (at the time of
this writing) year of 2005.12
Next, by asking general and broad
questions, systematic reviews often pro-
duce results and conclusions with ques-
tionable validity: “Poorly focused questions
in systematic reviews lead to unclear deci-
sions about what research to include and
how to summarize it.”13 Part of the prob-
lem is that some systematic reviews, or
the research studies they are based upon,
have not accounted for confounders that
may preclude appropriate interpretations.
For example, a recent systematic review of
the effect of rapid maxillary expansion by
Lagravere et al14 did not identify the types
of malocclusions (phenotypes) the rapid
maxillary expansion was used to treat; all
malocclusions were “lumped” together.
Was rapid maxillary expansion used for
crowding or Class II correction in the
absence of a posterior crossbite as advo-
cated by McNamara?15–17 Or, was rapid
maxillary expansion used only when a pos-
terior crossbite was present as suggested
by Gianelly?18 Parenthetically, Gianelly is
baffled by transverse maxillary expansion
in the absence of a posterior crossbite.19
Thus, how much clinical validity can this
review have? Asking the right questions is
of utmost importance in systematic
reviews. Even the authors14 of this exhaus-
tive systematic review were aware of the
limitations of their review and gave this
caveat and disclaimer as to the value of
what they concluded: “All of the studies
had methodological deficiencies. Lack of
intra- and interexaminer agreement report-
ing was common; only 1 study clearly
reported these values. The report of con-
founders and dropouts in the sample
analysis, as well as descriptions of the
inclusion criteria, was absent in the major-
ity of studies. Therefore, the scientific evi-
dence we found in this meta-analysis
should be interpreted carefully. Clinicians
will have to consider their experience, the
opinion of experts, and the limited evi-
dence on RME [rapid maxillary expansion]
to decide whether to use this treatment on
patients.”
Another example of the ambiguity and
controversy regarding systematic reviews
was pointed out by Johnston10 in regard
170 COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
Page 170
WORLD JOURNAL OF ORTHODONTICS
to a systematic review20 dealing with the
mandibular effects produced by func-
tional appliances in the treatment of
Class II malocclusion. Johnston, a critic
of the proposition that functional appli-
ances can “grow mandibles,” 1 0 , 2 0 , 2 1
offered this critique: inappropriate inclu-
sion and exclusion criteria; bias in the
weighting of articles (a tendency of the
authors to select their own studies, even
when the finding may not be statistically
significant); duplication of a previous sys-
tematic review that already had
addressed the question; and so forth.
Poor systematic reviews will invariably
lead to inaccurate conclusions that will
then negatively impact upon clinical prac-
tice. Further, when systematic reviews
are based on randomized clinical trials
that are also poorly defined and directed,
the error and impact on clinical practice
multiplies. There are perhaps more nega-
tive ramifications from poor systematic
reviews in medicine than in dentistry,
given that many of their clinical ques-
tions are life-threatening. Moreover, the
results and conclusions of medical sys-
tematic reviews vis-a-vis dental reviews
are more likely to be thoroughly scruti-
nized and the research they are based
upon often replicated using larger trials.
With regard to the problem in medicine,
Rees and Ebrahim22 offered this caveat
for basing medical decisions (ie, cardiol-
ogy) on initial and preliminary systematic
reviews utilizing rather small trials: “In
general, smaller trials are completed
before larger trials, and if meta-analyzed,
may show a beneficial effect, making
subsequent large clinical trials unneces-
sary and even unethical. The results of
meta-analyses of small trials may also be
more applicable at the population level
than a single large trial, in terms of allow-
ing stratified analyses by patient type,
severity of disease, age, and so on. Even
in cardiology, where very large random-
ized controlled trials have been per-
formed, it is unlikely that single large tri-
als of all potentially useful treatments
will be undertaken.”
Furthermore, Richards13 believed that
many systematic reviews in dentistry are
compromised because of poorly defined
outcomes measures with surrogate,
Frontiers_Rinchuse.qxd 5/7/08
VOLUME 9, NUMBER 2, 2008
COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
2:18 PM Page 171
rather than true, endpoints. True end-
points are “tangible to the patients and
directly measure how a patient feels,
functions or survives (eg, tooth loss or
pain or oral health-related quality-of-life
measurements).”13 Surrogate endpoints
are “intangible to the patient, such as
changes in probing attachment level or
gingival crevicular fluid level.”13 Richards
wrote that dentistry tends to focus on
surrogate endpoints or outcomes rather
than on true endpoints dealing with oral
health-related quality of life measures.13
MORE ON RANDOMIZED
CLINICAL TRIALS
Randomized clinical trials24–26 are some-
times also referred to as randomized
controlled trials.7 There is certainly a dif-
ference between a randomized clinical
trial and a randomized controlled trial.
For instance, the Class II early treatment
study at the University of Pennsylvania by
Gharfari et al26 was said to be a random-
ized clinical trial (and appropriately so)
and could only be described as a clinical
trial, not a controlled trial. Although the
trial was prospective and the subjects
were randomly assigned to 1 of 2 treat-
ment groups, there was no control group.
That is, subjects were not assigned to a
no-treatment (control) group.
Randomized controlled trials are con-
sidered the best type of studies because
they are, theoretically, not susceptible to
various biases; the distribution of charac-
teristics of all known and unknown vari-
ables are identical among the study
groups.9 Conversely, retrospective stud-
ies, the traditional method of assessing
the efficacy of orthodontic treatments,
have been criticized because of selection
bias, inadequate sample size, lack
of contemporaneous controls, poor
research design, and so on.24
Limitations of Randomized
Clinical Trials
As in other types of prospective studies,
randomized clinical trials are susceptible
to biases of compliance and long-term
RINCHUSE ET AL
attrition. Not all subjects comply with the
regimen to which they are assigned, and
for studies that require long follow-up peri-
ods, there is a natural tendency for a high
dropout rate. Moreover, because it takes
such a long time to complete prospective
trials, by the time the studies are com-
plete, the appliance and/or procedure
that was investigated may not even be
considered and/or utilized in practice (ie,
the question may be obsolete).27
Although the goal of conducting high-
quality randomized clinical trials is noble,
the reality is that many orthodontic clini-
cal research questions are amenable to
well-designed and cost-effective, obser-
vational (cross-sectional) studies such as
cohort or case-control studies. 28 Fein-
stein stated: “Although I have great rever-
ence, admiration, and devotion for ran-
domized trials, they have a limited
spectrum of application, and they cannot
be used to answer all the clinical ques-
tions that need to be answered. For rea-
sons of ethics, feasibility, or costs, we
simply will not be able to use randomized
trials to answer all the cause-effect ques-
tions that arise about both therapy and
etiology in clinical medicine.”28
As mentioned, there are ethical con-
cerns involving randomized clinical trials
studies using human subjects. In general,
there is the moral foundation that health
care providers should not disadvantage
subjects on account of their research par-
ticipation. 7 There must be a genuine
uncertainty on behalf of the expert ortho-
dontic community concerning the merits
of each trial arm (clinical equipoise); oth-
erwise, obtaining proper informed consent
becomes an issue. 7 More importantly,
when there is no clinical equipoise, there
may be an additional ethical concern with
randomized clinical trials due to random-
ization into experimental and control
groups whereby subjects in the control
group may be disadvantaged significantly
by not receiving the more appropriate
treatment in the long term (eg, extraction
versus nonextraction, orthodontics versus
surgery, and long versus short treat-
ments). In addition, a researcher cannot
ethically create a disease or disorder in 1
group of subjects, study the effects of the
disease (and several treatment modali-
171
Frontiers_Rinchuse.qxd 5/7/08 2:18 PM
FRONTIERS IN CLINICAL RESEARCH
ties), and compare it with a sample that
does not have the disease.
Importantly, there are many research
projects (and questions) that are not
favorable to a true experiment design, ie,
longitudinal, prospective, double-blind (or
at least single blind), randomized clinical
trial research protocol. For instance, it is
not possible or logical to consider a
prospective, randomized clinical trial
design for descriptive (epidemiological)
studies that do not involve issues of
cause/effect. Descriptive studies in medi-
cine investigate the incidence or preva-
lence of a certain disease state, patho-
logic condition, diagnostic factor, health
variable in a population, and so forth. In
orthodontics, an example would be a sur-
vey of the frequency of individuals in a
certain location who possess the various
Angle’s static occlusion types (ie, normal,
Class I, II, III)29 to address perhaps the ulti-
mate question as to which of the various
Angle types does this population possess.
Class II Early Treatment
Randomized Clinical Trials
Arguably, the best studies conducted in
orthodontics are the Class II early treat-
ment, multicenter randomized clinical tri-
als. The orthodontic profession has
clearly recognized and embraced the
value and strengths of these trials. How-
ever, these trials have some inherent
weaknesses. Certainly, we believe the
strengths of these studies far outweigh
their shortcomings. The strengths of the
early Class II randomized clinical trials
will be discussed first. The Class II early
treatment randomized clinical trials were
aimed essentially at answering 2 contro-
versies once and for all; the ability of
functional appliances to modify dentofa-
cial growth and the effectiveness of
1- versus 2-phase treatments of Class II
malocclusions (overjets greater than 7
mm). They generally found that 2-phase
treatments are less beneficial than 1
comprehensive phase. 23,25,26,30–35 The
early treatment randomized clinical trials,
as well as the compelling evidence from
the literature, seem to be converging on
the conclusion that mandibles cannot be
172 COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
Page 172
WORLD JOURNAL OF ORTHODONTICS
grown any more than the given morpho-
genetic pattern of the individual subjects
and that 2-phase growth-modification
treatments have little effect above and
beyond what would be expected of more
conventional 1-phase treatments.21,26,30–35
A strength of early treatment random-
ized clinical trials is their assignment of
treatments at random in order to mini-
mize bias. Proponents of the early treat-
ment randomized clinical trials also argue
the view that whether 1 particular person
or someone else could have assigned the
treatments more appropriately in isolated
instances or whether a Twin Block,
Herbst, or a Bionator was used to posture
the mandible in a forward position for it
to grow more is beside the point. The ran-
domized clinical trials were designed to
test for an effect. Had 1 treatment or
another been able, on average, to do
something that the others could not, the
randomized clinical trials certainly would
have had the power to detect it. Had an
effect been detected, everyone could
then apply the proper treatment in the
best interest of our patients.
The individuals, or groups, within the
orthodontic profession who still dismiss
the results of these current Class II early
treatment randomized clinical trials base
their criticisms on the following: ambigu-
ous definition of Class II; objectives of
phase I treatments not clearly defined as
would be in private practice (university
setting); did not use headgear long
enough; evaluated Bionator and Twin
Block (as opposed Frankel or others) as
the choice of functional appliances;
could have used a more contemporary
appliance or method including fixed
appliances; no retention between
phases; lack of blinding or control for
Hawthorne effect; too many uncontrolled
variables; large costs; and so forth.36,37
Meikle27 pointed out some additional
limitations of randomized clinical trials
that have evaluated the efficacy of func-
tional appliances: “Given the variability in
the timing, magnitude, and duration of
pubertal dentofacial growth, differing lev-
els of motivation and patient compliance,
the inherent inaccuracy of cephalometry
and the questionable validity of the mea-
surements themselves used to quantita-
Frontiers_Rinchuse.qxd 5/7/08
VOLUME 9, NUMBER 2, 2008
COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
2:18 PM Page 173
tive change, it is not surprising that the
conclusions have not been as clear-cut
as anticipated.”
Furthermore, regarding the issue of
what exactly constitutes a Class II, division
1 malocclusion as used in the early treat-
ment Class II randomized clinical trials
and whether all Class II malocclusions are
alike, Darendeliler37 made several com-
ments. The early treatment RCTs report
the average effect of treatment on a Class
II (1) malocclusion that result from a com-
bination of different deformities or mor-
phologies (phenotypes).37 The problem is:
“Not all patients with an increased overjet
or increased ANB angle have the same
malocclusion.” For example, a Class II (1)
malocclusion with an overjet of greater
than 7 mm may be due to maxillary dental
protrusion, mandibular retrognathism,
maxillary prognathism, retroclined
mandibular anterior teeth, a combination
thereof, and so on. It is therefore argued
that a stratified, refined sample selection
of Class II (1) malocclusions would have
provided more accurate data.37
All considered, we believe the early
treatment Class II randomized clinical tri-
als did answer the questions they initially
proposed. That is, we cannot grow
mandibles and that in general, 2-phase
treatments are less efficient that 1 com-
prehensive phase (at least under the
conditions of the random clinical trial
research design).23,25,26,30–35
The issue does not really appear to be
whether randomized clinical trials are
capable of addressing various controver-
sies in dentistry and orthodontics: They
are. The issue is whether we can justify
the large costs and time associated with
such trials when simple, cost-effective ret-
rospective or observational cohort studies
may arguably reveal the same results.38–43
In a specialty for which only limited
research funding is available, we must
seek not only evidence but also frugality.
Retrospective studies are quick, cost-
effective, and ethically unambiguous.40 A
major problem with retrospective studies
is achieving an approximation of a bias-
free sample, which is a prominent feature
in prospective trials. Failure to control for
these biases when comparing samples
retrospectively will invariably lead to
RINCHUSE ET AL
skewed and uninterpretable results. Care-
fully designed, planned, and statistically
analyzed retrospective studies aimed at
controlling or minimizing biases (selection,
detection/exclusion, proficiency, and
transfer) have been shown to yield highly
accurate results.40,42 Methods of control-
ling for these biases have been clearly
outlined by Johnston.42
META-ANALYSES
A meta-analysis is a mathematical and
quantitative (statistical) synthesis of the
results of 2 or more primary studies that
address the same hypothesis or topic in
the same way.6 Part of the responsibility
of the meta-analysis is to tabulate rele-
vant information on “the inclusion crite-
ria, sample size, baseline patient charac-
teristics, withdrawal rate, and results of
primary and secondary end points of all
the studies included.” 6 It is important
that the methods used for the review are
reliable, valid, and well-characterized.
Meta-analyses are by all accounts supe-
rior to qualitatively based evaluations of
numerous studies. The preliminary
aspects of the meta-analysis (prior to
applying the actual statistical test), how-
ever, are subjective (even though there
are certain rules and guidelines); there is
the subjective judgment in deciding
which studies to include.43
Eysenck44 lists a number of problems
that are inherent to meta-analyses:
“regressions are often nonlinear; effects
are often multivariate rather than univari-
ate; coverage can be restricted; bad stud-
ies may be included; the data summa-
rized may not be homogeneous; grouping
different causal factors may lead to mean-
ingless estimates of effects; and the the-
ory-directed approach may obscure dis-
crepancies.” Ultimately, Eysenck
concluded, “Meta-analysis may not be the
best method for studying the diversity of
fields for which it has been used.”44
Let us now discuss several additional
limitations of meta-analysis as described
by Eysenck.44 First, there is the possibility
that the data used in meta-analyses are
not homogeneous. Homogeneity is an
important requisite of meta-analyses.
173
Frontiers_Rinchuse.qxd 5/7/08 2:18 PM
FRONTIERS IN CLINICAL RESEARCH
Homogeneity means: “The results of each
individual trial are mathematically com-
patible with the results of any of the oth-
ers.”6 It implies that treatments, patients,
and end points must be similar or at least
comparable. Clinical heterogeneity is a
problem for meta-analyses. Greenhalgh6
cited a poignant example of a fallacious
meta-analysis based partly on heterogene-
ity: “ . . . Reservations about meta-analysis
are borne out in the infamously discred-
ited meta-analysis which showed (wrongly)
that giving intravenous magnesium to
people who had had heart attacks was
beneficial. A subsequent megatrial involv-
ing 58,000 patients (ISIS-4) failed to find
any benefit, and the meta-analysts’ mis-
leading conclusions were subsequently
explained in terms of publication bias,
methodological weaknesses in the
smaller trials, and clinical heterogeneity.”
Altman45 believed that the meta-analy-
ses (and systematic reviews) of prognos-
tic studies are difficult. Prognostic stud-
ies include clinical studies of variables
predictive of future events as well as epi-
demiological studies of etiological risk
factors. Authors often have concluded
that a meaningful meta-analysis for prog-
nostic studies is not possible due to a set
of studies being too diverse and/or too
weak. That is, a high proportion of prog-
nostic studies are methodologically poor.
Altman45 stated, “The poor quality of the
published literature is a strong argument
in favor of systematic reviews but also an
argument against formal meta-analysis . .
. Meta-analysis of prognostic studies
using individual data from patients may
overcome many of these difficulties.”
When there are serious methodologi-
cal difficulties in the primary studies, it is
generally impossible to conduct a sensi-
ble meta-analysis without first having
access to the individual patients’ raw
data. 45 The problem here may be that
many of the prognostic studies are obser-
vational rather than experimental (ran-
domized clinical trials) and the individual
raw data are not readily available. Fur-
ther, meta-analyses are sometimes used
incorrectly to recover something from
poorly designed studies; studies with
insufficient statistical power and studies
resulting in apparent contradictions.45 No
174 COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
Page 174
WORLD JOURNAL OF ORTHODONTICS
statistical test can overcome and rectify
the methodological shortcomings of
poorly designed primary studies. In sum-
mary, there is no doubt that the meta-
analysis has its place in the evidence-
based dentistry paradigm and is an
integral part of a systematic review; how-
ever, the validity of its findings is greatly
dependent on the quality of the individual
studies incorporated into the analysis.
CONCLUSIONS
Ackerman 46 appropriately stated, “The
challenge facing orthodontists in the 21st
century is the need to integrate the
accrued scientific evidence into clinical
orthodontic practice.” However, exactly
what constitutes the evidence is some-
what debatable. In orthodontic practice,
clinical judgment entails an integration of
clinical experience and systematic
assessment of relevant scientific evi-
dence in the context of the patient’s
orthodontic condition, treatment need
and preference.47 Currently, systematic
reviews of the literature favoring studies
utilizing randomized clinical trials are con-
sidered the gold standard and the top of
the hierarchical pyramid of evidence.
Although systematic reviews are the best
available source of evidence, a critical
appraisal and a cautious interpretation of
the results are essential. The notion that
systematic reviews and randomized clini-
cal trials are without limitations is a
rather naïve view and detrimental to
appropriate clinical decision-making.
Extrapolation of findings from systematic
reviews into clinical practice must con-
sider the methodological quality and
external validity of the studies used in
such reviews. Importantly, well-designed,
planned, and statistically analyzed retro-
spective studies (observational) aimed at
controlling or minimizing biases (selec-
tion, detection/exclusion, proficiency, and
transfer) may also yield valuable informa-
tion for the clinician. Finally, it is inappro-
priate in the age of evidence for health
care practitioners to totally abrogate their
responsibility to critically evaluate the evi-
dence, including the quality of systematic
reviews.
Frontiers_Rinchuse.qxd 5/7/08 2:18 PM Page 175

VOLUME 9, NUMBER 2, 2008 RINCHUSE ET AL

REFERENCES 18. Gianelly A. Rapid palatal expansion in the

absence of crossbites: Added value? Am J
Orthod Dentofacial Orthop 2003;124:362–365.
1. Turpin DL. Consensus builds for evidence-
19. Gianelly A. Dr. Lawrence White, JCO editor,
based methods. Am J Orthod Dentofacial
interviews Dr. Anthony Gianelly on current
Orthop 2004;125:1–2.
issues in orthodontics. J Clin Orthod 1996;30:
2. Ismail AI, Bader JD. Evidence-based dentistry in
439–445.
clinical practice. J Am Dent Assoc 2004;135:
20. Cozza P, Baccetti T, Franchi L, De Toffol L,
78–83.
McNamara Jr JA. Mandibular changes pro-
3. Shekelle PG, Woolf SH, Eccles M, Grimshaw J.
duced by functional appliances in Class II mal-
Clinical guidelines: Developing guidelines. Br
occlusion: A systematic review. Am J Orthod
Med J 1999;318:593–596.
Dentofacial Orthop 2006:129:599.e1–599.e12.
4. Turpin DL. CONSORT and QUOROM guidelines
21. Johnston Jr LE. If wishes were horses: Func-
for reporting randomized clinical trials and sys-
tional appliance and growth modification. Prog
tematic reviews. Am J Orthod Dentofacial
Orthod 2005;6:36–47.
Orthop 2005;128:681–686.
22. Rees K, Ebrahim S. Promises and problems of
5. Stroup D, Berlin J, Morton S, et al. Meta-analy-
systematic reviews. Heart Drug
sis of observational studies in epidemiology:
2001;1:247–248.
A proposal for reporting. JAMA 2000;283:
23. Keeling SD, Wheeler TT, King GJ, et al. Antero-
2008–2012.
posterior skeletal and dental changes after
6. Greenhalgh T. How to read a paper: Papers
early Class II treatment with bionator and head-
that summarize other papers (systematic
gear. Am J Orthod Dentofacial Orthop 1998;
reviews and meta-analyses) Brit Med J
113:40–50.
1997;315: 672–675.
24. Sadowsky PL. Craniofacial growth and the tim-
7. Dietrich T. Commentary: Evidence-based den-
ing of treatment. Am J Orthod Dentofacial
tistry and the absence of evidence from ran-
Orthop 1998;113:19–23.
domized controlled trials for common dental
25. Tulloch JFC, Phillips C, Proffit WR. Benefit of
procedures. In: Pena RA (ed). Dental Abstracts
early Class II treatment: Progress report of a
2006;51:192–193.
two-phase randomized clinical trial. Am J
8. Keim RG. The weight of evidence. J Clin Orthod
Orthod Dentofacial Orthop 1998;113:62–72.
2004;38:121–122.
26. Ghafari J, Shaffer S, Jacobsson-Hunt U,
9. Williams DDR, Garner J. The case against “the
Markowitz DL, Laster LL. Headgear versus
evidence:” A different perspective on evidence-
function regulator in the early treatment of
based medicine. Br J Psychol 2002;180:8–12.
Class II, division 1 malocclusion: Randomized
10. Johnston Jr LE. Commentary. Am J Orthod
clinical trial. Am J Orthod Dentofacial Orthop
Dentofacial Orthop (online only) 2006:
1998; 113:51–61.
129(5):e1–e4. [Cozza P, Baccetti T, Franchi L,
27. Meikle MC. What do prospective randomized
De Toffol L, McNamara Jr JA. Mandibular
clinical trials tell us about the treatment of
changes produced by functional appliances in
class II malocclusions? A personal viewpoint.
Class II malocclusion: A systematic review. Am
Eur J Orthod 2005;27:105–114.
J Orthod Dentofacial Orthop (online only)
28. Feinstein AR. The clinician as scientist. In: PS
2006:129:599.e1–599.e12.]
Vig, KA Ribbens (eds). Science and Clinical
11. Flores-Mir C, Major MP, Major PW. Search and
Judgment in Orthodontics. Monograph 19-
selection methodology of systematic reviews in
Craniofacial Growth Series: Center for Human
orthodontics (2000-2004). Am J Orthod Dento-
Growth and Development. Ann Arbor: The Uni-
facial Orthop 2006;130:214–217.
versity of Michigan, 1986.
12. Major MP, Major PW, Flores-Mir C. An evalua-
29. Angle EH. Classification of malocclusion. Den-
tion of search and selection methods used in
tal Cosmos 1899;41:248–264.
dental systematic reviews published in English.
30. O’Brien K, Wright J, Conboy F, et al. Effective-
J Am Dent Assoc 2006;137:1252–1257.
ness of early orthodontic treatment with the
13. Richards D. Outcomes, what outcomes? Evid
twin-block appliance: A multicenter, random-
Based Dent 2005;6:1.
ized, controlled trial. Part 1: Dental and skele-
14. Lagravere MO, Heo G, Major PW, Flores-Mir C.
tal effects. Am J Orthod Dentofacial Orthop
Meta-analysis of immediate changes with rapid
2003;124:234–243.
maxillary expansion treatment. J Am Dent
31. O’Brien K, Wright J, Conboy F, et al. Effective-
Assoc 2006;137:44–53.
ness of early orthodontic treatment with the
15. McNamara JA. Maxillary transverse deficiency.
twin-block appliance: A multicenter, random-
Am J Orthod Dentofacial Orthop 2000;117:
ized, controlled trial. Part 2: Psychosocial
567–570.
effects. Am J Orthod Dentofacial Orthop 2003;
16. McNamara JA Jr. Early intervention in the trans-
124:488–494.
verse dimension: Is it worth the effort? Am J
32. O’Brien K, Wright J, Conboy F, et al. Effective-
Orthod Dentofacial Orthop 2002;121:572–574.
ness of treatment for class II malocclusion with
17. McNamara JA Jr, Brudon WL. Orthodontics and
the herbst or twin-block appliances: A random-
dentofacial orthopedics. Ann Arbor: Needham
ized controlled trial. Am J Orthod Dentofacial
Press, 2001.
Orthop 2003;124:128–137.

COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO 175
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER
Frontiers_Rinchuse.qxd 5/7/08 2:18 PM Page 176

FRONTIERS IN CLINICAL RESEARCH

33. Tulloch JFC, Phillips C, Koch G, Proffit WR. The

effect of early treatment on skeletal pattern in
class II malocclusion: A randomized clinical
trial. Am J Orthod Dentofacial Orthop 1997;
111:391–400.
34. Tulloch JFC, Proffit WR, Phillips C. Outcomes in
a 2-phase randomized clinical trial of early
Class II treatment. Am J Orthod Dentofacial
Orthop 2004;125:657–667.
35. Koroluk LD, Tulloch JFC, Phillips C. Incisor Jarabak Endowed Professorship
trauma and early treatment for Class II division
1 malocclusion. Am J Orthod Dentofacial The Joseph R. and Louise Ada Jarabak Professorship is a full-time faculty
Orthop 2003;123:117–125. position endowed at the full Professor level. The specified purpose of the
36. Dugoni S, Lee JS, Varela J, Dugoni A. Early Professorship is to enhance and further orthodontic education; stimulate
mixed dentition treatment: Postretention evalu- research in anatomy, physiology, and the function of the stomatognathic
ation of stability and relapse. Angle Orthod system; continue research in anatomy, physiology, and the function of the
1995;65:311–320. stomatognathic system; continue research in TMJ disorders; and stimulate
37. Darendeliler MA. Validity of randomized clinical a continued development of orthodontic teachers. All successful candi-
trials in evaluating the outcomes of class II dates must either have a DDS/DMD degree, clinical specialty training in
treatment. Semin Orthod 2006;12:67–79. orthodontics and eligibility for Indiana dental licensure. In addition the can-
38. Johnston LE Jr. A comparative analysis of Class didates must either have: (1) completed an NIH Fellowship in biochemistry,
II treatments. In: Vig PS, Ribbens KA (eds). Sci- molecular biology, or immunology with a PhD in anatomy, physiology, or
ence and Clinical Judgment in Orthodontics. similar qualifications; (2) a MS or PhD in biophysics or a basic science
Ann Arbor: Center for Human Growth and equivalently related to orthodontics. Development of an independent
Development, 1986. research program is required. Opportunity for intramural or extramural pri-
39. Luppannapornlarp S, Johnston LE Jr. The vate practice is available. The position will remain open until filled.
effects of premolar-extraction treatment:
A long-term comparison of outcomes in “clear-
cut” extraction and nonextraction Class II Interested individuals should email an electronic application, an electronic
patients. Angle Orthod 1993;63:257–272. curriculum vitae and three electronic reference letters to:
40. Livieratos FA, Johnston LE Jr. A comparison of
Dr Jeffrey A. Dean
one- and two-stage non-extraction alternatives
Executive Associate Dean
in matched Class II samples. Am J Orthod
Dentofacial Orthop 1995;108:118–131. Indiana University School of Dentistry
41. Paquette DE, Beattie JR, Johnston LE Jr. A long- 1121 W. Michigan Street
term comparison of nonextraction and premo- Indianapolis, IN 46202-5186
lar-extraction edgewise therapy in “borderline” jadean1@iupui.edu
Class II patients. Am J Orthod Dentofacial
Indiana University is an Equal Opportunity/Affirmative Action Employer.
Orthop 1992;102:1–14.
42. Johnston LE Jr. Moving forward by looking
back: “Retrospective” clinical studies. J Orthod
2002;29:221–226.
43. Concato J, Shah N, Horwitz RI. Randomized
controlled trials, observational studies, and the
hierarchy of research designs. N Engl J Med
2000;342:1887–1892.
44. Eysenck HJ. Systematic reviews: Meta-analysis
and its problems. Br Med J 1994;309:789–792.
45. Altman DG. Systematic reviews in health care:
Systematic reviews of evaluations of prognos-
tic variables. Br Med J 2001;323:224–228.
46. Ackerman M. Evidence-based orthodontics for
the 21st century. J Am Dent Assoc 2004;135:
162-167.
47. Ackerman JL, Kean MR, Ackerman MB. Evi-
dence-bolstered orthodontics. Aust Orthod J
2006;22:69–70.

176 COPYRIGHT © 2008 BY QUINTESSENCE PUBLISHING CO, INC. PRINTING OF THIS DOCUMENT IS RESTRICTED TO PERSONAL USE ONLY. NO
PART OF THIS ARTICLE MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM WITHOUT WRITTEN PERMISSION FROM THE PUBLISHER