Schizophrenia Bulletin vol. 32 no. 4 pp.
655665, 2006
doi:10.1093/schbul/sbl009
Advance Access publication on July 18, 2006
Enhancing Validity in Co-occurring Disorders Treatment Research
Gregory J. McHugo1,2, Robert E. Drake2, Mary F.
Brunette2, Haiyi Xie2, Susan M. Essock3, and
Alan I. Green4
2
Dartmouth Psychiatric Research Center, 2 Whipple Place, Suite
202, Lebanon, NH 03766; 3Division of Health Services Research,
Mt Sinai School of Medicine; 4Department of Psychiatry,
Dartmouth Medical School
Despite the high prevalence of co-occurring mental health
and substance-use disorders, there has been a relative lack
of treatment research with this population, and the existing
research often has limited validity. This article explores
some of the barriers to the conduct of research on promising
interventions for substance-abuse treatment for people with
co-occurring disorders, using the concepts of external and
ecological validity to make recommendations for future in-
vestigation. The central recommendation is to move rapidly
from efficacy studies to more credible and valid effective-
ness studies, in order to facilitate the adoption of evidence-
based interventions in routine practice settings.
Key words: inferential validity/external validity/
ecological validity/co-occurring disorders/
dual diagnosis/effectiveness research
Introduction
Co-occurring severe mental illness and substance abuse,
often called dual diagnosis or dual disorders, is a major
public health problem.1 This fact has been well estab-
lished since the 1980s, yet there has been relatively little
research on treatment for people with co-occurring dis-
orders. These patients are unstable,2 diagnostically com-
plex,3 difficult to recruit to studies,4 difficult to engage in
treatment,5 and especially difficult to retain in treat-
ment.6,7 These factors lead to the exclusion of patients
with co-occurring disorders from controlled research
studies and to difficulty completing studies aimed at
this population. In addition, the presence of separate cen-
ters within National Institutes of Health for the study of
alcohol (National Institute on Alcohol Abuse and Alco-
1
To whom correspondence should be addressed; tel: 603-448-
0263, fax: 603-448-3976, e-mail: gregory.mchugo@dartmouth.edu.
The Author 2006. Published by Oxford University Press on behalf of the Maryland Psychiatric Research Center. All rights reserved.
For permissions, please email: journals.permissions@oxfordjournals.org.
655
holism), drugs (National Institute on Drug Abuse), and
mental illness (National Institute of Mental Health) has
led to an unfortunate lack of responsibility and cooper-
ation concerning the study of co-occurring disorders.
Treatment research of co-occurring disorders has been
summarized recently in terms of specific psychosocial in-
terventions,8 residential interventions,9 treatment princi-
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
ples,10 and pharmacotherapies.11 Although more than 40
controlled studies show advantages for specific interven-
tions, there have been few replications. In many cases, the
experimental intervention represents a closer integration
of mental health and substance-abuse treatments than the
control intervention, but there is little consistency across
studies in terms of designs, patients, interventions, and
outcome measures.8 Many of the studies are quasi-exper-
imental rather than experimental, different types of
patients are included in studies, many of the interventions
are complex amalgams, and outcomes and measures vary
considerably.10 Thus, after 20 years of research, there
remains a lack of strong and clear evidence regarding
effective engagement, treatment, and rehabilitation
interventions for people with co-occurring disorders.
Furthermore, even where the evidence base is consistent,
there has been reluctance among service providers to im-
plement new interventions for co-occurring disorders,
leading to the well-documented science to service gap.1
We believe that it is time to examine the reasons why
there has been a lack of research and adoption of inter-
ventions for patients with severe mental illness and a
co-occurring substance-use disorder. We propose that
progress in the past has been slow because conventional
research methods often lead to studies with low validity
when applied to this population.
Validity
Based on the work of Campbell and colleagues, validity
refers to the truthfulness of inferences drawn from re-
search findings. Inferential validity was originally dis-
cussed as internal vs external validity,12 although more
recently, internal validity has been expanded to include
statistical conclusion validity and external validity has
been expanded to include construct validity.13,14 Internal
validity pertains to the elimination of bias from the cause-
effect relationship. Statistical conclusion validity pertains
to the appropriate use of statistics. Construct validity
G. J. McHugo et al.
pertains to the generalizability of the findings across the
operations and measures of a study, and external validity
pertains to the generalizability of the findings across peo-
ple, settings, and time. Co-occurring disorders research
has been weak in all 4 domains of inferential validity. As
the evidence-based medicine movement grows within
behavioral health, the lack of valid research will hinder
the development of evidence-based practices, treatment
guidelines and algorithms, and decision supports for
co-occurring disorders.
The accrual of evidence in support of an intervention
comes from many directions and with various claims to
scientific rigor and inferential validity. The challenge be-
fore the co-occurring disorders field now is to design in-
tervention studies that address key clinical questions in
a manner that is scientifically sound and that generalize
to practitioners and patients in routine practice settings.
This brings to the fore the tension between efficacy and
effectiveness research. The basic issue in balancing effi-
cacy vs effectiveness involves the extent to which the re-
search reflects ideal scientific conditions (high internal
validity) vs real-world practice conditions (high external
validity).15
The randomized controlled trial (RCT) is the gold
standard of efficacy research because it provides the
strongest basis for drawing valid causal inferences.
That is, by eliminating potential sources of bias, the
well-conducted RCT can isolate the intervention as the
cause of the outcome. The common approach in most ef-
ficacy research is to maximize internal validity by using
a narrowly defined population, interventions designed
specifically for this narrow population, highly trained re-
search clinicians, tightly controlled randomized trials,
short research follow-ups, and proximal outcome meas-
ures. Efficacy research maximizes scientific control and
inferential validity concerning the treatment effect, but
studies of this type often produce such narrow and non-
generalizable findings that they do not apply to typical
patients, do not lead to the proposed next stages of inter-
vention development, and fail to influence routine clinical
practice.1618 Moreover, this approach assumes the stage-
wise development, validation, and dissemination of inter-
ventions (eg, Rounsaville et al19), but this process can be
so slow that new interventions displace the tested ones
before the tested ones are widely disseminated.
While researchers often test theoretically sound inter-
ventions in carefully controlled situations, common real-
world interventionsthose that are regularly used by
clinicians, believed to be effective, and therefore warrant
testingare often difficult to study in RCTs. Examples
from the co-occurring disorders field include injectable
antipsychotic medications, clozapine, dual-recovery self-
help interventions, and long-term residential treatment.
Each of these interventions, except clozapine, is widely
used in existing dual-diagnosis programs but has not
been studied carefully, for different reasons. For exam-
656
ple, residential treatment programs must follow counter-
acting and restrictive regulations of the Department of
housing and urban development (HUD), and clinicians
are often reluctant to assign unstable patients to cloza-
pine by randomization. Despite several quasi-experimen-
tal studies that support residential dual-diagnosis
treatment9 and clozapine,11 there have been no success-
fully completed rigorous RCTs of either intervention.
Despite the power and prestige of the RCT, there are
many situations in which true experiments are impossi-
ble, unnecessary, premature, or uninformative.20 Even
if randomization is possible in a given situation, experi-
mental control may not be, so each threat to inferential
validity has to be evaluated carefully. For example, ran-
domization to a community treatment team vs standard
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
case management may go smoothly only to have clients
fail to receive the assigned intervention for a wide variety
of client-level reasons, such as being jailed, moving out of
state, or deciding to discontinue treatment, or system-
level reasons, such as a reconfiguration of available
services during the course of the study. The longer the
treatment and follow-up periods of a study, the more
likely such challenges to intervention fidelity and sample
retention become. A study with substantial attrition
would indicate how many individuals, if offered the treat-
ment under these circumstances, would participate and
would highlight the challenges to mounting the interven-
tion in question but would not provide the answer to the
basic clinical question regarding effectiveness.
In many cases where an RCT has not been possible,
dual-disorders studies have employed a range of quasi-
experimental designs to address important research ques-
tions often in naturalistic settings.10 These designs are
useful for the study of system-level policy changes, the
examination of a longer term perspective on outcome,
those situations when feasibility and pilot data are
needed, and occasions when there are barriers to random-
ization. In addition, qualitative research methods can
yield deeper understanding of specific issues due to their
focus on subjective experience and narrative data. They
have been used to study the validity of measures, patient
attitudes and motivations, reasons for noncompliance,
and system barriers to dissemination and implementa-
tion.21 For the study of co-occurring disorders, qualita-
tive methods have been used to understand recovery from
substance abuse from the patients perspective, to assess
natural social support networks, and to explain treatment
refusal or dropout.22,23
This discussion highlights the value of the RCT for de-
termining intervention efficacy, but it also reveals the
shortcomings of efficacy research for developing the
evidence base in support of a given intervention. Other
designs and methods may be more practical and infor-
mative, and they may yield findings with higher external
validity. The various approaches can be viewed as com-
plementary rather than oppositional because evidence of

the efficacy of an intervention is necessary but seldom
sufficient to ensure its widespread adoption in routine
practice settings. We believe that the widespread adop-
tion of interventions to treat substance abuse among peo-
ple with co-occurring disorders will occur only when
studies lead to inferences with high external validity
and include procedures with high ecological validity.
External Validity
A study has high external validity if its findings apply to
the intended population of patients, providers, and set-
tings. Despite the importance of external validity, the de-
liberate focus on internal validity among investigators
has led to a great deal of irrelevant and nongeneralizable
research. Rothwell17 documented the lack of attention to
external validity in research across a wide range of med-
ical conditions and proposed higher standards for both
single studies and systematic reviews. Most telling is
the oft-repeated claim by providers that the failure to
adopt new practices is due to the lack of credibility
and applicability of the evidence base.
To illustrate this problem, consider the following
hypothetical question, which is based on approximate
population estimates. If only 50% of people with schi-
zophrenia and co-occurring substance abuse are in
treatment,24 if only 20% of those in treatment take anti-
psychotic medications as prescribed,25 and if only 10% of
those who take medications are interested in adding an-
other medication for substance abuse,4 do we want to
study the new medication for the 1% of eligible patients
or do we want to study interventions for engagement in
long-term medication management for the 99% of eligible
patients? This example is not farfetched. By the time in-
clusion and exclusion criteria have been imposed, the
conditions of study participation have been revealed,
and other recruitment procedures have been imple-
mented, the enrolled sample may differ in important
ways from the majority of patients who are seen in rou-
tine practice. It may be more sensible for clinicians to ig-
nore a studys results entirely rather than to try to adjust
the risks and benefits of a treatment that was evaluated in
a patient group that is only remotely similar to theirs.
Ecological Validity
In addition to stressing the importance of external
validity, we also stress the need for studies with greater
ecological validity. Although defined variously (eg,
Bronfenbrenner26 and Brunswick27), we use ecological
validity here to mean the extent to which the methods,
settings, and interventions of an experiment approximate
the real-life situation that is under study. Therefore, to
enhance ecological validity, the challenge is to conduct
studies that mimic the clinical realities of the engage-
ment, treatment, and rehabilitation of patients with co-
occurring disorders.
Enhancing Validity
One such reality is the decision-making process in
which clinicians and patients engage regularly. If, on
the one hand, a study contains a rigid protocol for mak-
ing decisions about treatment dose and duration without
regard to co-occurring conditions, engagement in serv-
ices, adherence to treatment, and patient preferences, it
will have low ecological validity because its procedures
may not simulate the complex decision process that is
used in routine practice. If, on the other hand, a study
protocol reflects the usual decision making that deter-
mines which intervention to try first, how long a given
trial lasts, what outcomes are tracked, how to sequence
interventions, and how to handle nonadherence and
dropout, it will more closely mimic the decision process
in routine care and have higher ecological validity.
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
The remainder of this article focuses on ways to
enhance both external and ecological validity in dual-
disorders research, without compromising internal valid-
ity. The primary advantage is that the dissemination and
implementation of effective interventions will be facili-
tated due to the credibility and clinical meaningfulness
of the research rather than assumed due to the statistical
significance of the findings.
Design
Ideally, design strategies reflect current clinical thinking
and practice and also include the largest proportion of
participants, test the most relevant interventions, and
account for patient preferences. Unfortunately, dual-
disorders research has been hampered by the unquestion-
ing adoption of RCT methods for effectiveness trials.
Alternatives to the conventional RCT have emerged re-
cently,28 and they may lead to greater external and eco-
logical validity, while maintaining internal validity, in
studies of co-occurring disorders.
Sequential Adaptive Treatment Designs
In treating chronic relapsing disorders, clinicians make
decisions over time that depend on multiple factors,
such as the availability of treatments and the allocation,
adherence, and response to them. Historically, treatment
research for chronic disorders has not evaluated what
clinicians actually do. Instead, new treatments are
evaluated in isolation relative to standard care, thereby
bearing little on actual practice. Fortunately, recent
developments have led to the elaboration of experimental
designs that permit evaluation of adaptive treatment
strategies (eg, Collins et al29, Lavorie and Dawson30,
and Murphy31). In cases where treatment guidelines
are well specified, fixed adaptive designs are appropriate.
These designs contrast algorithmic treatment strategies
with each other or with usual care in order to evaluate
the effectiveness of adhering to a predefined sequence
of decisions concerning treatment. For example, fixed
657
G. J. McHugo et al.
adaptive designs have been used to evaluate the treatment
of depression in primary care settings.32,33 In these stud-
ies, rule-based but flexible strategies for care manage-
ment, which included education, medication, specialty
care, and adherence, side effect, and outcome monitor-
ing, were contrasted with usual care and found to be
more effective.
Because treatment for co-occurring disorders does not
have well-established guidelines currently, randomized
adaptive designs are an appealing option. By adding ran-
dom assignment at each decision point, studies can con-
trast multiple treatment strategies, each of which involves
a sequence of options that depend on factors such as re-
sponse to prior treatment, adherence, and intervention
modality. These studies are efficient in developing and
refining sequential adaptive treatment strategies, al-
though they require large sample sizes. Ideally, each
effective sequence is then codified as a treatment
algorithm and contrasted with standard treatment in
a conventional 2-group RCT. As examples, randomized
adaptive designs have been used in recent trials to eval-
uate treatment for depression,34 Alzheimer disease,35 al-
cohol addiction,36 and cocaine addiction.37 For the study
of treatment of alcohol abuse among people with co-
occurring disorders, such a design might include an initial
period of dual-diagnosis case management, after which
nonresponders are assigned randomly to next-level inter-
ventions (eg, a medication strategy or a dual-diagnosis
group intervention). Continued nonresponse at the
next assessment point could lead to a switching or aug-
menting of interventions, with or without randomization.
Responders at any level of the design could be randomly
assigned to one of several relapse prevention interven-
tions. With sufficient sample size, such a design could
rapidly test multiple interventions and could determine
the potency of various sequences of interventions in re-
lation to the long-term outcome of sustained abstinence.
By evaluating long-term outcomes and allowing an in-
formed sequence of treatment decisions, these designs
have greater ecological validity and could move the field
more quickly toward useful guidelines for the timing,
matching, and sequencing of treatments.
Equipoise-Stratified Randomization
Randomized trials of multiple interventions often are
limited by the size and representativeness of the enrolled
sample because each participant must be willing to accept
all possible treatment options in order to be assigned ran-
domly to any one of them. If a prospective participant
rejects (eg, refuses medication), or is not appropriate
for (eg, has no family), a specific intervention, that par-
ticipant cannot be included in the study. One proposed
solution to this problem is equipoise-stratified randomi-
zation.38 Equipoise means that prospective study partic-
ipants (patients, clinicians, or both) regard a set of
658
treatment options as essentially equivalent in terms of
the likelihood of success. Equipoise-stratified randomiza-
tion is a method for retaining the maximum number of
participants possible, while still allowing contrasts be-
tween conditions (interventions) to which participants
have been assigned randomly. At the point of consent,
participants are informed of the treatment options of-
fered in the experiment, and they decide which options
are acceptable to them. As participants enroll in the ex-
periment, a number of strata will emerge depending on
the subsets of the treatment options that participants
are willing (and eligible) to accept. The only people
who are excluded are those who are willing to accept
only one or none of the available options. Otherwise,
the study can accommodate all volunteers willing to be
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
assigned to at least 2 of the treatments under study,
thereby enhancing external validity. More work needs
to be done concerning the analysis and interpretation
of studies using equipoise-stratified randomization, but
the emergence of this approach underscores the need
for solutions to problems of recruitment, which often
result in irrelevant and ignorable findings.
These innovations in experimental design can greatly
enhance the external and ecological validity of research
concerning co-occurring disorders. Studies that examine
the timing and sequencing of psychosocial, medication,
and residential interventions are sorely needed. Factors
such as intervention cost and burden could be taken
into account in the sequencing, and the treatment deci-
sion process could more closely approximate that used in
routine settings. The addition of equipoise-stratified ran-
domization could enable the findings to generalize to a
much broader segment of the dual-diagnosis population.
Length of Follow-up
Long-term outcomes should be the goal for research that
seeks to understand chronic, fluctuating illnesses. Brief
interventions and short-term follow-ups may test
short-term compliance rather than sustained remission
and personal recovery. For example, studies of injectable
antipsychotic medications tend to show short-term gains
but no long-term effects.39 Long-term treatment outcome
studies are expensive and difficult to conduct, and yet, the
answers to many of the most important questions facing
the field can only be answered by such studies. If we are to
bridge the gap between efficacy and effectiveness studies,
one solution will be to pay more attention to identifying
short-term outcomes that predict long-term recovery.
When studies of co-occurring disorders are designed to
evaluate adaptive treatment sequences, the natural focus
will be on long-term outcomes. An effectiveness study to
evaluate medication options to treat alcohol abuse may
take a year to conduct, with another year of follow-up,
and therefore, an end-point outcome might be whether
or not participants attain remission or sustained abstinence.

A typical efficacy trial of one of the medications might
last for 36 months, and the outcome might be the change
in the average daily quantity of alcohol consumed.
Surely, the long-term perspective of the former study
leads to outcomes that are more meaningful and clinically
significant for patients and clinicians, although short-
term studies to establish treatment efficacy and tolerabil-
ity are needed first. One implication of this is that efficacy
studies should include short-term (proximal) outcomes
that predict long-term (distal) outcomes rather than
solely using short-term outcomes that are supposed sur-
rogates of the long-term outcomes but may not actually
predict them.
Recovery from both mental illness and substance
abuse is a longitudinal process, not a short-term out-
come. On the one hand, long-term illnesses need to be
studied in relation to long-term outcomes because the
short-term course involves fluctuations and brief changes
that may not predict eventual recovery. Everything we
know about dual disorders implies a long-term course
of recovery.40 On the other hand, system-level outcomes
involve improving the processes of care. Measuring such
relatively short-term events as changes in the likelihood
of clients receiving particular evidence-based inter-
ventions (processes of care) is a means of characterizing
system-level outcomes. To the extent that long-term
recovery depends on short-term outcomes (eg, treatment
access and adherence), research must take both into
account.
Outcomes
Current ethical, clinical, and research perspectives man-
date that we study outcomes that are meaningful to both
clinicians and patients.1 Patients value goals that are em-
bodied in the concept of recovery, which is defined by the
Presidents Commission on Mental Health1(p1) as living,
learning, working, and participating fully in ones com-
munity. In focus groups, ethnographies, and research
interviews with dual-diagnosis patients, we have found
that they also identify several other goals: avoiding neg-
ative health outcomes, like HIV and hepatitis C infection;
avoiding aversive living states, like homelessness and hos-
pitalization; avoiding criminalization and incarceration;
managing their own illnesses, including mental illness and
substance abuse; avoiding negative side effects related to
medications; and avoiding victimization, stigma, and in-
terpersonal abuse. Sometimes, they identify goals that are
qualitatively different from the outcomes that are typi-
cally assessed by researchers. For example, dual-diagno-
sis patients often identify having friends and intimate
partners who are not substance abusers as an important
component of their recovery.23,40 Likewise, for people
with co-occurring disorders, treatment discontinuation
at any stage of recovery is associated with negative out-
comes and should be considered an important outcome in
Enhancing Validity
its own right. Too often, treatment nonadherence and
dropout are considered nuisances rather than key out-
comes, although they are among the preconditions
that determine the magnitude and scope of the treatment
effect.
Deciding what outcomes to measure is a central issue
when studying co-occurring disorders. Mental health and
substance-abuse outcomes are relatively independent
dimensions, and they are largely independent of out-
comes in other domains, such as employment, quality
of life, general health, and residential status.41,42 Having
used a multidimensional assessment, the challenge then is
to decide whether to treat the outcome within each do-
main separately and hierarchically (ie, primary vs second-
ary outcomes) or to combine outcomes into a single
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
indicator of a complex construct like recovery (eg, Drake
et al40 and Xie et al43). One proposed way to circumvent
this decision is to use an analytic approach that selects the
outcome of greatest change for each participant prior to
testing the contrast between groups.44 This method
acknowledges that multiple outcomes may be affected
by an intervention and that change may happen on dif-
ferent outcomes for different people.
A further consideration concerning outcomes is inter-
preting their clinical significance, not just their statistical
significance. Continuous measures have advantages for
statistical analysis, but they do not readily translate
into clinically meaningful conclusions. Proposed solu-
tions to this problem include comparison with established
norms45 or use of the reliable change index.46,47 As an
alternative to statistical solutions, investigators often
define outcomes a priori in clinically meaningful ways.
By doing so, a statistically significant difference can be
evaluated more readily for its clinical significance.
Often the immediate questions following demonstra-
tion of a treatment effect are for whom and how does
it work? Moderator and mediator effects, respectively,
take direct aim at these questions. A priori specification
of a model of the treatment and recovery process is
important when deciding what to measure as possible
moderators or mediators. Recent discussions by method-
ologists have improved the understanding and analysis of
moderator and mediator effects in longitudinal outcome
studies48 and clinical trials.49
Moderators are often attributes of the study partici-
pants that influence the response to treatment. Modera-
tor effects are most often detected statistically as 3-way
interactions in longitudinal data analysis (Treatment
Group by Moderator Level by Time). Because they are
observed characteristics of study participants, moderator
effects are necessarily correlational and must be inter-
preted accordingly. For example, among patients with
co-occurring disorders, antisocial personality disorder
or the presence of severe physiological dependence
may moderate the impact of treatment for alcohol abuse,
but causal mechanisms cannot be specified.
659
G. J. McHugo et al.
Mediators are links in a causal chain of outcomes,
whereby levels of an intervening (proximal) outcome
causally influence more distal outcomes. Mediators are
often process variables, such as services received or
knowledge gained, which occur after enrollment in an in-
tervention but before outcomes are assessed. For exam-
ple, the effect of naltrexone on alcohol abuse may depend
on adherence with the medication regimen. In this case,
treatment adherence mediates the effect of naltrexone on
alcohol use. A range of statistical methods has been pro-
posed to detect mediator effects, and they differ in the
trade-off between Type I errors and statistical power.50
Measuring Substance Use
Whether the primary outcome or not, substance use will
be part of any study of an intervention for co-occurring
disorders. Unfortunately, there are limitations on the re-
liability and validity of measures of substance use from
any source, and the target behavior is not easily measured
over extended periods of time. Moreover, there are nu-
merous possible sources of bias in verbal reports in gen-
eral, many of which are likely when reporting illegal
and socially undesirable behavior like drug use. There
are also limitations with collateral reports and biological
measures.
Currently, the most reliable and valid way to assess
substance use is to gather assessments from multiple
sources, using multiple methods, and to bring those
measures into a rule-based process for developing con-
sensus ratings. Specifically, we have used participant
self-report, clinician ratings, and biological indicators
to triangulate on a reliable and valid assessment of sub-
stance-use disorder (eg, Drake et al51 and Essock et al52).
Given the uncertainty surrounding any single measure of
substance use, a standardized process for combining in-
formation from multiple sources provides the most valid
measure possible, especially when the outcome of interest
is clinically meaningful, such as presence or absence of
stable abstinence, the status of alcohol-use disorder, or
the days of drug use in the past 6 months.
Interventions
Because we are interested in interventions that improve
outcomes of usual care, clarifying and documenting usual
care is critical. Based on the current state of the evidence,
what is ethical and evidence-based to include in usual care
for patients with co-occurring disordersclinical case
management, cognitive behavioral therapy, referral to
self-help, or illness self-management? Unfortunately, the
evidence is not yet strong enough for numerous specific
dual-disorders interventions to make this decision. Yet, it
is crucial to standardize usual care in order to improve the
inferences drawn from studies of co-occurring disorders.
Many clinics and other settings are already providing
660
some form of dual-diagnosis treatment; the challenge
is to bring it under algorithmic control via explicit pro-
tocols so that interventions are tested against meaningful
standards.
Even if usual care can be standardized prior to the start
of a research project, it may be difficult to keep it stan-
dardized over time. For example, randomization within
clinics to either usual care or a new intervention can lead
to numerous threats to inferential validity. It is difficult to
control drift, diffusion, and compensatory efforts when
control group (ie, usual care) clinicians become aware
of experimental interventions. In addition, various efforts
to rescue clients, such as transferring them to supervised
housing or changing medications, may also, of necessity,
undermine experimental control. Quasi-experimental re-
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
search comparing clinics (eg, one provides usual care and
one provides usual care plus the intervention) may be
more feasible, although some of these same threats to in-
ferential validity apply to them as well, and the lack of
randomization leaves inferences vulnerable to selection
bias. Even in the most rigorously designed studies, nu-
merous trade-offs must be evaluated because randomiza-
tion can cause, as well as overcome, threats to validity.
Given high uncertainty, due to the lack of direct evi-
dence, how should interventions be selected? It may
make sense to start with interventions that have shown
efficacy in small trials in the dual-disorders population
or in related populations. At this time, motivational
interviewing linked with cognitive behavioral therapy,
contingency management, residential treatment, and spe-
cial medications for substance abuse (eg, naltrexone and
acamprosate) are good candidates.8,11 They have shown
promise in the primary substance-abuse treatment
population and are ripe for testing in the dual-disorders
population.
When considering interventions, adherence to treat-
ment arises as a critical issue. Individuals with dual dis-
orders are notoriously difficult to engage in treatment,
for a host of reasons, and therefore, study protocols
must anticipate adherence problems. Studies may in-
crease adherence through design features, monitoring
protocols, and specific interventions such as contingency
management to increase participation.53 The goal is to
preserve the integrity of the experiment following ran-
domization, but artificially enhanced adherence comes
with the downside of compromising external validity if
adherence would be different without the intervention
provided by the research protocol. Hence, interventions
should not include provisions to maximize adherence that
would be infeasible in routine practice settings.
Medications
Patients involvement with medication treatment for sub-
stance abuse can be conceptualized within a stagewise
model. Attitudes about medication are affected by lack

of acknowledgment of a mental illness or denial and min-
imization regarding substance-use disorder, either of
which can lead to disinterest in treatment, in general,
and medications, in particular, in the early stages of treat-
ment. As patients learn more about their disorders and
become interested in managing them, they are often
more willing to try medications as a management tool.
Consequently, studies of medications for people with
dual disorders are more feasible if they recruit people
who are already in either the active treatment or the re-
lapse prevention stages of recovery. Restricting the trea-
ted population in this manner limits the generalizability
of the study, but paradoxically, it may enhance its exter-
nal validity, by allocating treatment in the research as it
would be allocated in routine practice.
Medications effective for alcohol-use disorders in the
general population, such as acamprosate and naltrexone,
have modest effects.54,55 Consequently, studies of these
medications need large sample sizes in order to detect sta-
tistically significant differences between groups, and
researchers need to be clear about what magnitude of dif-
ference is clinically significant. Studies of medications are
usually short term and involve measures of substance use
that are sensitive to immediate and modest change. These
studies do not provide sorely needed evidence concerning
the persistence of the changes and the effects of long-term
use of the medications.
Studies of medications often incorporate psychosocial
interventions to enhance adherence. The question of
which psychosocial interventions to study along with
medication interventions brings up issues regarding com-
bination treatments vs solo treatments. If the effect of the
medication is modest and concomitant psychosocial
interventions are usually offered in real-world settings,
a more externally valid design is to study the combina-
tion of medication plus psychosocial treatment. Placebo
plus psychosocial treatment controls could help to assess
what portion of improvement is due to the medication
component.
Some medications, such as clozapine, may treat psychi-
atric and substance-use disorders together,56 whereas
other medications are designed to treat the substance-
use disorder only and must be added to a regimen that
treats the mental illness. The advantage of a single med-
ication approach includes the use of fewer pills, which
could be more cost effective, less burdensome, and pos-
sibly more appealing to participants, resulting in better
adherence. The disadvantages include the requirement
to change the primary psychiatric medication and the in-
ability to adjust medication regimens based on symptoms
of each disorder.
Because patients with dual disorders tend to experience
high levels of psychosocial instability and are often
poorly engaged into any sort of services, it is difficult
to recruit and retain them in medication studies. Techni-
ques for tracking this population (eg, maintaining multi-
Enhancing Validity
ple contacts from the patients social world such as
landlords, friends, coworkers, and family) work well for
longitudinal studies,57 but they do not ensure compliance
with regular and frequent research appointments to mon-
itor medication treatment. Instead, frequent reminders,
transportation, child-care support, and community out-
reach may be needed in order to enhance patients ability
to participate regularly in this type of study. A further
complication is that patients in later stages of treatment
who are more stable are often reluctant to participate in
medication studies. Clinicians may view their stability as
tenuous, and there may be reluctance on their part to
stress the patient by involvement in research or by chang-
ing medication. Studies of add-on medications, such as
naltrexone for alcohol-use disorders, may be easier to im-
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
plement because a switch of the medication treating the
mental illness is not required for the study.
This discussion underscores the value of, and need for,
both efficacy and effectiveness studies of medications for
people with co-occurring disorders. Efficacy studies need
blinding, placebo control groups, and other procedures
of RCTs in order to isolate the treatment effect from
other influences. Once a medication effect has been found
under ideal conditions, the focus should change rapidly
to real-world effectiveness trials where the concern is with
such issues as acceptance and adherence, timing and se-
quencing, clinically meaningful outcomes, persistence of
changes over time, long-term side effects, and medication
interactions. Findings from both types of studies are
needed before evidence-based treatment guidelines and
algorithms can be developed.
Practitioners
A difficult trade-off exists when deciding who should pro-
vide specialized psychosocial interventions. On the one
hand, it may not be best to use highly trained research
clinicians because their training, experience, and skills
do not generalize to clinicians in routine settings. On
the other hand, it is desirable to study a competent imple-
mentation of each intervention in order to have a valid
test of effectiveness. A reasonable compromise may be
to train clinicians in routine settings to a standard of com-
petence and to monitor their practice throughout the
study, both clinically (supervision) and empirically (treat-
ment adherence and model fidelity). This has been done
in effectiveness studies of integrated treatment for indi-
viduals with co-occurring disorders with good success.51,52
Settings
Patients with co-occurring disorders involving severe
mental illness often receive services in community mental
health centers, hospitals, jails, and homeless shelters, if
at all. Studies in routine settings such as these have a
good chance of recruiting people who are in different
661
G. J. McHugo et al.
stages of recovery and are more representative of the
population.
One problem with conducting studies in routine set-
tings is that usual care varies widely from setting to set-
ting. As discussed above, there is a pressing need to
standardize usual care in order to clarify the effects of
new medication and psychosocial interventions. In addi-
tion to clarifying usual care, critical features of routine
settings can be expected, based on empirical studies, to
influence outcomes. For example, the local criminal jus-
tice system,58 the types and amounts of housing programs
that exist,11 and the amounts of family contact and sup-
port that patients have59 may moderate treatment effects.
In addition, medications for mental and physical health
may interact with treatment for substance abuse and
must be monitored closely. Settings will also differ in
the extent to which they assess intervention fidelity
and assure adherence with the treatments, whether
psychosocial or pharmacological.
Patients
A fundamental problem in studies of interventions for co-
occurring disorders is the heterogeneity of the patient
population. People with co-occurring disorders are di-
verse, not only in socioeconomic features, personal
assets, and community supports but also in clinical
and comorbid features. Mental health and substance-
use disorders can range widely in type and severity.
Patients are also affected by a range of common co-
occurring conditions, such as antisocial personality dis-
order, posttraumatic stress disorder, traumatic brain
injury, neurocognitive problems, hepatitis C, and chronic
medical problems (eg, diabetes, obesity, and cardiovascu-
lar disease). Restricting the population to those with se-
vere mental illness may have modest impact on reducing
heterogeneity, although it does increase external validity,
because these are the patients who are treated in the pub-
lic mental health system.
We have already discussed how the typical solution to
this problema narrow focus on one specific clinical
groupoften results in studies that severely limit external
validity and clinical applicability because they pertain to
so few people. But how can we consider heterogeneity in
a way that enhances clinically relevant research? We sug-
gest looking for more natural points of cleavage in the
populationthose that correspond to research findings
and clinical realities.
Naturalistic follow-ups, as well as intervention studies,
can identify meaningful subgroups of patients with co-
occurring disorders. For example, because psychiatric
diagnosis among the severe mental illnesses has little
or no predictive validity concerning long-term recovery
from substance abuse, it may be a poor selection criterion
for studies, unless there is a diagnosis-specific interven-
tion. Rather, longitudinal research on substance-abuse
662
recovery suggests that there are distinct subgroups within
the larger population of patients with co-occurring dis-
orders, which are characterized in part by their rela-
tionships and responses to treatment over time.60 One
group rejects community-based treatments and has uni-
formly poor outcomes over at least 10 years. A second
group engages in treatment rapidly, enters substance-
abuse recovery rapidly, and has a stable course over 10
years. The majority of patients occupy 2 groups that
have intermediate courses, one with a fluctuating course
over 10 years, and another with a course of slow but
steady improvements over 10 years. If these subgroups
can be validated across samples and characterized by
common individual differences and clinical features, it
would make sense to classify patients early in treatment
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
for the sake of efficiently providing services that they
need, avoiding services to which they may not respond,
and designing separate studies for each group.
In addition, studies can be designed to reflect clinical
decision making and shared decision making. As de-
scribed above, adaptive treatment strategies enable the
evaluation of care for chronic relapsing disorders in a
more ecologically valid fashion. The goal is to make treat-
ment decisions that take the patients history, current sta-
tus, and preferences into account and then to monitor
adherence and outcomes in order to adapt appropriately.
For example, research on behavior change indicates that
treatment is more effective when clinicians and patients
think about recovery from substance-use disorders in
steps or stages. Based on an approach described by Osher
and Kofoed,61 the patient must first be engaged in
a therapeutic relationship, defined at minimum by regu-
lar participation in treatment. As an example of a study at
this stage, Corrigan and colleagues62 compared motiva-
tional interviewing, financial incentives, and barrier re-
duction in terms of their effect on regular treatment
participation at 6 months for clients with substance abuse
and traumatic brain injury. They showed that both finan-
cial incentives and barrier reduction were superior to
motivational interviewing and an attention control.
Following engagement in treatment, many patients
with co-occurring disorders remain unmotivated to
manage their illnesses (persuasion phase). That is, they
attend treatment sessions but deny problems with sub-
stance abuse, mental illness, or both, and they do not
take responsibility for managing these illnesses by using
active strategies. The common intervention at this stage
is motivational counseling of some type,8 but other
strategies, such as contingency management, family psy-
choeducation, supported employment, long-acting med-
ications, and mandated treatment, could be tested.
Regardless of approach, the goal is to increase the behav-
ioral manifestations of illness management.
Once patients are actively working to acquire the skills
and supports for managing their illnesses, they are in
the active treatment stage. At this stage, a variety of

pharmacological, cognitive behavioral, social network-
based, self-help, and other strategies could be tested.
When patients are securely in remission (eg, at least
6 months without a relapse of either disorder), they are
in the relapse prevention stage. A variety of relapse pre-
vention interventions have been developed for mental ill-
nesses and for substance-use disorders, but none has been
tested specifically with dual-diagnosis patients.63 Because
patients at this stage of recovery are participating in treat-
ment reliably, are relatively stable, and yet relapse is com-
mon, this would be an ideal group for RCTs to examine
interventions for successful relapse prevention.
Summary
There are fundamental differences between efficacy and
effectiveness research, and we have reviewed a number of
ways to enhance the validity of the latter, with special em-
phasis on external and ecological validity. Efficacy stud-
ies are necessary in identifying promising interventions
that yield a discernible treatment effect under ideal con-
ditions, but their success does not guarantee adoption of
the tested interventions. That is, in an efficacy study, un-
der conditions where everything else is equal, a treatment
effect can be detected and attributed conclusively to an
intervention, but until the intervention has been tested
under routine conditions, its actual value remains un-
known. Therefore, efficacious interventions need to be in-
vestigated more rapidly and extensively under conditions
that enhance external and ecological validity. To this end,
effectiveness studies can include heterogeneous patient
samples, preference-based decisions, adherence interven-
tions and monitoring, routine clinicians and clinical
decision making, stage-appropriate interventions, stan-
dardized usual care in routine settings, and long-term
outcomes that are valued by patients themselves. Ef-
fectiveness studies can use innovative designs and proce-
dures to simulate routine clinical processes and to recruit
more representative patients, clinicians, and settings.
Abelson64 included generality and credibility among
the 5 criteria that he proposed to evaluate the merits
of an experiment. Even if an experiment shines in all other
ways, if it is not credible in its methods and generalizable
in its findings, it will neither have an impact scientifically
nor influence practice. By paying more attention to ex-
ternal and ecological validity, effectiveness studies of
treatment for co-occurring disorders will gain easier ac-
ceptance among patients, providers, and policy makers,
and will be more likely to bridge the gap between science
and practice.
Acknowledgments
This work was supported by grants from the National
Institute on Alcohol Abuse and Alcoholism, National
Institute on Drug Abuse, National Institute of Mental
Health, and the West Family Foundation.
Enhancing Validity
An earlier version of this article was presented at the
National Institute on Alcohol Abuse and Alcoholism/
National Institute on Drug Abuse/National Institute
of Mental Health workshop on Methodology of
Conducting Pharmacologic Clinical Trials in Patients
with Alcohol/Drug Dependence and Psychiatric Comor-
bidity, Bethesda, MD, in February 2006.
References
1. New Freedom Commission on Mental Health. Achieving the
Promise: Transforming Mental Health Care in America. Rock-
ville, Md. DHHS Publication No. SMA-033831; 2003.
2. Drake RE, Brunette MF. Complications of severe mental ill-
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
ness related to alcohol and other drug use disorders. In:
Galanter M, ed. Recent Developments in Alcoholism. Volume
XIV, Consequences of Alcoholism. New York: Plenum;
1998:285299.
3. Caton CLM, Drake RE, Hasin D, et al. Differences between
early phase primary psychotic disorders with concurrent
substance use and substance-induced psychoses. Arch Gen
Psychiatry. 2005;62:137145.
4. Petrakis IL, Nich CI, Ralevski E. Schizophrenia and alcohol
abuse: a review of pharmacotherapeutic strategies and a
report on the effectiveness of naltrexone and disulfiram.
Schizophr Bull. In press.
5. Lehman AF, Herron JD, Schwartz RP, Myers CP. Rehabili-
tation for young adults with severe mental illness and
substance use disorders: a clinical trial. J Nerv Ment Dis.
1993;181:8690.
6. Chandler DW, Spicer G. Integrated treatment for jail recidi-
vists with co-occurring psychiatric and substance use disor-
ders. Community Ment Health J. 2006;42:In press.
7. Hellerstein DJ, Rosenthal RN, Miner CR. A prospective
study of integrated outpatient treatment for substance-abus-
ing schizophrenic patients. Am J Addict. 1995;4:3342.
8. Mueser KT, Drake RE, Sigmon S, Brunette MF. Psychoso-
cial interventions for adults with severe mental illnesses and
co-occurring substance use disorders: a review of specific
interventions. J Dual Disord Diagn. 2005;1:5782.
9. Brunette MF, Mueser KT, Drake RE. A review of research
on residential programs for people with severe mental illness
and co-occurring substance use disorders. Drug Alcohol Rev.
2004;23:471481.
10. Drake RE, Mueser KT, Brunette MF, McHugo GJ. Review
of treatments for persons with severe mental illness and co-
occurring substance use disorder. Psychiatr Rehabil J. 2004;
27:360374.
11. Brunette MF, Noordsy DL, Buckley P, Green AI. Pharmaco-
logic treatments for co-occurring substance use disorders in
patients with schizophrenia: a research review. J Dual Diagn.
2005;1:4155.
12. Campbell DT, Stanley JC. Experimental and Quasi-
Experimental Designs for Research. Chicago, Ill: Rand
McNally; 1963.
13. Cook TD, Campbell DT. Quasi-Experimentation: Design and
Analysis Issues for Field Settings. Chicago, Ill: Rand
McNally; 1979.
14. Shadish WR, Cook TD, Campbell DT. Experimental and
Quasi-Experimental Designs for Generalized Causal Inference.
Boston, Mass: Houghton Mifflin; 2002.
663
G. J. McHugo et al.
15. Wells KB. Treatment research at the crossroads: the scientific
interface of clinical trials and effectiveness research. Am J
Psychiatry. 1999;156:510.
16. Carroll KM, Rounsaville BJ. Bridging the gap: a hybrid
model to link efficacy and effectiveness research in substance
abuse treatment. Psychiatr Serv. 2003;54:333339.
17. Rothwell PM. External validity of randomized controlled tri-
als: to whom do the results of this trial apply? Lancet.
2005;365:8293.
18. Zimmerman M, Chelminski I, Posternak MA. Generalizabil-
ity of antidepressant efficacy trials: differences between de-
pressed psychiatric outpatients who would and would not
qualify for an efficacy trial. Am J Psychiatry. 2005;162:
13701372.
19. Rounsaville BJ, Carroll KM, Onken LS. A stage model of be-
havioral therapies research: getting started and moving on
from stage I. Clin Psychol Sci Pract. 2001;8:133142.
20. Essock SM, Drake RE, Frank RG, McGuire TG. Random-
ized clinical trials in evidence-based mental health care: get-
ting the right answer to the right question. Schizophr Bull.
2003;29:115123.
21. Whitley R, Crawford M. Qualitative research in psychiatry.
Can J Psychiatry. 2005;50:108114.
22. Alverson H, Alverson M, Drake RE. An ethnographic study
of the longitudinal course of substance abuse among people
with severe mental illness. Community Ment Health J.
2000;36:557569.
23. Alverson H, Alverson M, Drake RE. Social patterns of sub-
stance use among people with dual diagnoses. Ment Health
Serv Res. 2001;3:314.
24. Wang PS, Demier O, Kessler RC. Adequacy of treatment for
serious mental illness in the United States. Am J Public
Health. 2002;92:9298.
25. Owen RR, Fischer EP, Booth BM, Cuffel BJ. Medication
noncompliance and substance abuse among patients with
schizophrenia. Psychiatr Serv. 1996;47:853858.
26. Bronfenbrenner U. The Ecology of Human Development:
Experiments by Nature and Design. Cambridge, Mass: Har-
vard University Press; 1979.
27. Brunswick E. Perception and the Representative Design of
Psychological Experiments. 2nd ed. Berkeley, Calif: Univer-
sity of California Press; 1956.
28. TenHave TR, Coyne J, Slazer M, Katz I. Research to im-
prove the quality of care for depression: alternatives to the
simple randomized clinical trial. Gen Hosp Psychiatry.
2003;25:115123.
29. Collins LM, Murphy SA, Nair V, Strecher V. A strategy for
optimizing and evaluating behavioral interventions. Ann
Behav Med. 2005;30:6573.
30. Lavorie PW, Dawson R. A design for testing clinical strate-
gies: biased individually tailored within-subject randomiza-
tion. J R Stat Soc Ser A Stat Soc. 2000;163:2938.
31. Murphy SA. An experimental design for the development of
adaptive treatment strategies. Stat Med. 2005;24:14551481.
32. Katon W, Von Korff M, Lin E, et al. Stepped collaborative
care for primary care patients with persistent symptoms of
depression: a randomized trial. Arch Gen Psychiatry. 1999;
56:11091115.
33. Unutzer J, Katon W, Williams JW, et al. Improving primary
care for depression in late life: the design of a multicenter ran-
domized trial. Med Care. 2001;39:785799.
34. Fava M, Rush AJ, Trivedi MH, et al. Background and ratio-
nale for the sequenced treatment alternatives to relieve
664
depression (STAR*D) study. Psychiatr Clin North Am.
2003;26:457494.
35. Schneider LS, Tariot PN, Lydetosos CG, et al. National In-
stitute of Mental Health Clinical Antipsychotic Trails of In-
tervention Effectiveness (CATIE): Alzheimer Disease trial
methodology. Am J Geriatric Psychiatry. 2001;9:346360.
36. Breslin FC, Sobell MB, Sobell LC, Cunningham JA, Sdao-
Jarvie K, Borsoi D. Problem drinkers: evaluation of
a stepped-care approach. J Subst Abuse. 1999;10:217232.
37. Brooner RK, Kidorf M. Using behavioral reinforcement
to improve methadone treatment participation. Sci Pract
Perspect. 2002;1:3848.
38. Lavorie PW, Rush AJ, Wisniewski SR, et al. Strengthening
clinical effectiveness trials: equipoise-stratified randomiza-
tion. Biol Psychiatry. 2001;50:792801.
39. Fenton WS, Blyler CR, Heinssen RK. Determinants of med-
ication compliance in schizophrenia: empirical and clinical
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
findings. Schizophr Bull. 1997;24:637651.
40. Drake RE, McHugo GJ, Xie H, Fox M, Packard J,
Helmstetter B. Ten-year recovery outcomes for clients
with co-occurring schizophrenia and substance use disor-
ders. Schizophr Bull. 2006;32:464473.
41. McDermott PA, Alterman AI, Brown L, Zaballero A, Snider
EC, McKay JR. Construct refinement and confirmation for
the Addiction Severity Index. Psychol Assess. 1996;8:182189.
42. McLellan AT, Luborsky L, Woody GE, OBrien CP. An im-
proved diagnostic evaluation instrument for substance abuse
patients. J Nerv Ment Dis. 1980;168:2633.
43. Xie H, McHugo GJ, Helmstetter BS, Drake RE. Three-year
recovery outcomes for long-term patients with co-occurring
schizophrenic and substance use disorders. Schizophr Res.
2005;75:337348.
44. Boothroyd RA, Banks SM, Evans ME, Greenbaum PE,
Brown E. Untangling the web: an approach to analyzing
the impacts of individually tailored, multicomponent treat-
ment interventions. Ment Health Serv Res. 2004;6:143153.
45. Kendall PC, Marrs-Garcia A, Nath SR, Sheldrick RC. Nor-
mative comparisons for the evaluation of clinical significance.
J Consult Clin Psychol. 1999;67:285299.
46. Jacobson NS, Truax P. Clinical significance: a statistical ap-
proach to defining meaningful change in psychotherapy re-
search. J Consult Clin Psychol. 1991;59:1219.
47. Speer DC. Clinically significant change: Jacobson and Truax
(1991) revisited. J Consult Clin Psychol. 1992;60:402408.
48. Kenny DA, Calsyn RJ, Morse GA, Klinkenberg WD, Winter
JP, Trusty ML. Evaluation of treatment programs for per-
sons with severe mental illness: moderator and mediator
effects. Eval Rev. 2004;28:294324.
49. Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Media-
tors and moderators of treatment effects in randomized clin-
ical trials. Arch Gen Psychiatry. 2002;59:877883.
50. MacKinnon DP, Lockwood CM, Hoffman JM, West SG,
Sheets V. A comparison of methods to test mediation and
other intervening variable effects. Psychol Methods.
2002;7:83104.
51. Drake RE, McHugo GJ, Clark RE, et al. A clinical trial of
assertive community treatment for patients with co-occurring
severe mental illness and substance use disorder. Am J Ortho-
psychiatry. 1998;68:201215.
52. Essock SM, Mueser KT, Drake RE. Comparison of ACT and
standard case management for delivering integrated treat-
ment for co-occurring disorders. Psychiatr Serv. 2006;57:
185196.

53. Sigmon SC, Stitzer ML. Use of a low-cost incentive interven-
tion to improve counseling attendance among methadone
maintained patients. J Subst Abuse Treat. 2005;29:253258.
54. Kranzler HR, Van Kirk J. Efficacy of naltrexone and acam-
prosate for alcoholism treatment: a meta-analysis. Alcohol
Clin Exp Res. 2001;25:13351341.
55. Mann K, Lehert P, Morgan MY. The efficacy of acamprosate
in the maintenance of abstinence in alcohol-dependent indi-
viduals: results of a meta-analysis. Alcohol Clin Exp Res.
2004;28:5163.
56. Drake RE, Xie H, McHugo GJ, Green AI. The effects of clo-
zapine on alcohol and drug use disorders among patients with
schizophrenia. Schizophr Bull. 2000;26:441449.
57. Scott CK. A replicable model for achieving over 90% follow-
up rates in longitudinal studies of substance abusers. Drug
Alcohol Depend. 2004;74:2136.
58. Swartz MS, Swanson JW, Wagner HR, Burns BJ, Hiday VA.
Effects of involuntary outpatient commitment and depot
antipsychotics on treatment adherence in persons with severe
mental illness. J Nerv Ment Dis. 2001;189:583592.
Enhancing Validity
59. Clark RE. Family support and substance use outcomes for
persons with mental illness and substance use disorders.
Schizophr Bull. 2001;27:93101.
60. Xie H, Drake RE, McHugo GJ. Are there distinctive trajec-
tory groups in substance abuse remission over 10 years? An
application of the group-based modeling approach. Adm
Policy Ment Health Ment Health Serv Res. 2006;33:423432.
61. Osher FC, Kofoed LL. Treatment of patients with psychiatric
and psychoactive substance use disorders. Hosp Community
Psychiatry. 1989;40:10251030.
62. Corrigan JD, Bogner J, Lamb-Hart G, Heinemann AW,
Moore D. Increasing substance abuse treatment compliance
for persons with traumatic brain injury. Psychol Addict
Behav. 2005;19:131139.
63. Drake RE, Wallach MA, McGovern MP. Preventing relapses
to substance use disorder among clients with severe mental ill-
nesses: what do we know and what do we do next? Psychiatr
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on November 5, 2015
Serv. 2005;56:12971302.
64. Abelson RP. Statistics as Principled Argument. Hillsdale, NJ:
Lawrence Erlbaum; 1995.
665