Psychological Assessment Copyright 2002 by the American Psychological Association, Inc.

2002, Vol. 14, No. 2, 209 –220 1040-3590/02/$5.00 DOI: 10.1037//1040-3590.14.2.209

Convergent Validity of the Agnew Relationship Measure

and the Working Alliance Inventory

William B. Stiles Roxane Agnew-Davies

Miami University Refuge

Michael Barkham Alison Culverwell

University of Leeds East Kent Community Trust

Marvin R. Goldfried Jeremy Halstead

State University of New York at Stony Brook Dewsbury Health Care Trust

Gillian E. Hardy Patrick J. Raue

University of Leeds and University of Sheffield Weill Medical College, Cornell University

Anne Rees David A. Shapiro

University of Leeds University of Leeds and University of Sheffield

The convergent validity of the Agnew Relationship Measure (ARM) and the Working Alliance Inventory
(WAI) was assessed in samples drawn from 2 comparative clinical trials of time-limited psychotherapies
for depression. In 1 sample, clients (n ⫽ 18) and therapists (n ⫽ 4) completed self-report versions of both
measures after every session (n ⫽ 198). In the other sample, clients (n ⫽ 39) and therapists (n ⫽ 6)
completed the ARM, and observers subsequently rated selected audiotaped sessions (n ⫽ 78) using the
WAI. In both samples, the ARM’s core alliance scales (Bond, Partnership, and Confidence) were
correlated with the WAI’s scales (Bond, Tasks, and Goals) strongly when assessed within client and
therapist perspectives and, with some qualifications, moderately when assessed between client, therapist,
and observer perspectives, supporting the assumption that the ARM and the WAI measure some of the
same core constructs.

The alliance between client and therapist is perhaps the most
written-about and measured construct in the psychotherapy pro-
cess research literature (e.g., Constantino, Castonguay, & Shut,
William B. Stiles, Department of Psychology, Miami University; Rox-
ane Agnew-Davies, Refuge, London, England; Michael Barkham and
Anne Rees, Psychological Therapies Research Centre, University of Leeds,
Leeds, United Kingdom; Alison Culverwell, East Kent Community Trust,
Canterbury, United Kingdom; Marvin R. Goldfried, Department of Psy-
chology, State University of New York at Stony Brook; Jeremy Halstead,
Dewsbury Health Care Trust, Dewsbury, United Kingdom; Gillian E.
Hardy and David A. Shapiro, Psychological Therapies Research Centre,
University of Leeds and Department of Psychology, University of Shef-
field, Sheffield, United Kingdom; Patrick J. Raue, Department of Psychi-
atry, Weill Medical College, Cornell University.
Portions of this research were supported by the Medical Research
Council of the United Kingdom and by Grant MH 40196 from the National
Institute of Mental Health. Some staff members involved in this project
were supported by funding from the Leeds Community Mental Health and
Teaching Trust.
Correspondence concerning this article should be addressed to William
B. Stiles, Department of Psychology, Miami University, Oxford, Ohio
45056. E-mail: stileswb@muohio.edu
2002; Horvath & Bedi, in press; Horvath & Greenberg, 1994;
Horvath & Luborsky, 1993; Orlinsky, Grawe, & Parks, 1994;
Safran & Muran, 1998). Following Bordin’s (1979, 1994) seminal
conceptualization, which characterized the alliance as encompass-
ing (a) the affective bond between client and therapist, (b) agree-
ment on the goals of treatment, and (c) agreement on treatment
tasks, or means of achieving those goals, most alliances research-
ers have understood the alliance as multidimensional. However,
researchers have not agreed on the boundaries of the alliance
construct or on the number or names of the dimensions (see
Horvath & Greenberg, 1994). Many additional and overlapping
dimensions have been posited and assessed, as noted later. The
intense interest reflects the alliance’s replicated correlations with
measures of psychotherapy outcome across a wide range of ther-
apeutic approaches (Horvath & Bedi, in press; Horvath & Sy-
monds, 1991; Krupnick et al., 1996; Orlinsky et al., 1994; Raue &
Goldfried, 1994; Stiles, Agnew-Davies, Hardy, Barkham, & Sha-
piro, 1998; Watson & Greenberg, 1994).
Different alliance scales have only occasionally been compared
in the same study (Bachelor, 1991; Cecero, Fenton, Nich, Frank-
forter, & Carroll, 2001; Safran & Wallner, 1991; Tichenor & Hill,
1989). Thus, the assumption that they are measuring the same
construct has seldom been tested directly. In this article, we report

209
210 STILES ET AL.
direct comparisons between two alliance measures, the Working
Alliance Inventory (WAI; Horvath & Greenberg, 1986, 1989) and
the Agnew Relationship Measure (ARM; Agnew-Davies, Stiles,
Hardy, Barkham, & Shapiro, 1998), in samples drawn from two
comparative clinical trials of time-limited psychotherapies for de-
pression. Although our comparison bears on the convergent valid-
ity of both measures, our motivation was to assess the convergence
of the ARM, which is a newer measure, with the more widely used
WAI. We examined convergent validity at both the dyad level
(comparisons across client–therapist dyads) and the session level
(comparisons across each dyad’s sessions).
Structure of the WAI and the ARM
The WAI includes three scales, which were derived from Bor-
din’s (1979) transtheoretical conception of the alliance (Horvath,
1994). The Bond scale measures the therapeutic bond, which
encompasses mutual liking, attachment, and trust. The Tasks scale
measures agreement on joint tasks, including the strategies and
techniques of treatment. The Goals scale measures agreement
about treatment goals, including the areas targeted for change.
Although the WAI was developed as a self-report instrument
(Horvath & Greenberg, 1986, 1989), Tichenor and Hill (1989)
“adapted [an observer-rated version] from the client and therapist
forms by altering the pronouns to fit an observer perspective” (p.
197). This observer form was formally specified in a rating manual
developed by Raue, Goldfried, and Barkham (1997) and used in
one of our samples.
The ARM was developed using a mixed conceptual– empirical
strategy, with items constructed to encompass scale content from
many previous measures, including the WAI, and then refined
through three iterations of item selection and rewriting, adminis-
tration to therapy dyads, and factor analysis (Agnew-Davies et al.,
1998). Items were selected for five scales using three criteria:
statistical coherence (based on the factor analyses), conceptual
coherence (judged from item content), and comparability of items
across client and therapist forms. The five scales are Bond, which
concerns the friendliness, acceptance, understanding, and support
in the relationship; Partnership, which concerns working jointly on
therapeutic tasks and toward therapeutic goals; Confidence, which
concerns optimism and respect for the therapist’s professional
competence; Openness, which concerns the client’s felt freedom to
disclose personal concerns without fear or embarrassment; and
Client Initiative, which concerns the client’s taking responsibility
for the direction of the therapy. Items and scales are parallel across
client and therapist forms (Agnew-Davies et al., 1998). Advan-
tages of the ARM include incorporation of content areas drawn
broadly from previous alliance work, a simple format cast in
language appropriate for most therapeutic approaches, and parallel
forms developed for therapists and clients. As noted by Horvath
and Luborsky (1993), in previous instruments, “therapists’ scales
are direct rewordings of client instruments; thus far no effort has
been made to investigate the specific impressions and experiences
that therapists associate with the clients’ experience of positive
alliance” (p. 565). The ARM assesses broader aspects of the
therapeutic relationship than do most previous instruments, en-
compassing client initiative, openness, and confidence in addition
to the core components of bond and partnership described by
Bordin (1979).
Many studies have demonstrated a positive association of WAI
scales with gains in treatment (Horvath, 1994; Horvath & Bedi, in
press). The ARM has not yet been studied so extensively, but
comparisons based on one of the samples from which the present
study was drawn broadly confirmed the positive association of
alliance with treatment outcomes. Some ARM scales shared up to
40 –50% of the variance in some comparisons (Stiles et al., 1998).
For example, therapist mean ARM Confidence scores were corre-
lated .44 with residual gains on the Beck Depression Inventory
(Beck, Ward, Mendelson, Mock & Erbaugh, 1961) and .51 with
residual gains on the Rosenberg Self-Esteem Scale (O’Malley &
Bachman, 1979) assessed at end of treatment in a sample of 79
clients treated in time-limited therapies for depression (boths ps ⬍
.001). (Note that, unless the alliance is considered as a subscale of
treatment outcome, the more appropriate coefficient of determina-
tion is r, not r2; Ozer, 1985.) However, as in studies involving
other alliance measures (see review by Horvath & Bedi, in press),
the strength of the alliance– outcome association varied in complex
ways across alliance subscales, outcome measures, occasions of
outcome assessment (posttreatment, follow-up), and the point in
treatment at which the alliance was measured (Stiles et al., 1998).
As a contrasting example, client mean ARM Bond scores were
correlated only .16 with residual gains on the Symptom Check-
list—90 —Revised (Derogatis, Lipman, & Covi, 1973) and .16
with residual gains on the Inventory of Interpersonal Problems
(Horowitz, Rosenberg, Baer, Ureno, & Villasenor, 1988) assessed
at 6-month follow-up in the same sample ( p ⬎ .05). Current
conceptualizations of the alliance do not offer clear accounts of
these complex variations (Stiles et al., 1998). Furthermore,
whereas some previous authors concluded that alliance– outcome
associations are stronger when the alliance is assessed from the
client rather than from the therapist perspective and assessed early
rather than late in treatment (see reviews by Constantino et al.,
2002; Horvath & Symonds, 1991), Stiles et al. (1998) found that
similar correlations with residual gains were obtained from ther-
apist and from client ratings (reflecting, perhaps, the closer atten-
tion given to the therapist form in the ARM’s construction) and
that alliance– outcome correlations were generally higher when
alliance was measured later rather than earlier in treatment.1 As-
sessing the extent to which different alliance scales measure the
same thing must be an early step in any attempt to understand such
complexities empirically.
The Bond scales on the ARM and the WAI were meant to
measure the same construct, and the ARM’s Partnership scale was
meant to measure the constructs measured by the WAI’s Tasks and
Goals scales considered together (Agnew-Davies et al., 1998). The
ARM’s Confidence, Openness, and Initiative scales were meant to
measure constructs described elsewhere in the alliance literature;
1
Across ARM alliance scales and across measures and occasions of
assessment, the prediction of residual gains from alliance scores tended to
improve from earlier to later sessions. The mean improvement-in-
prediction correlation was .33 for client-rated alliance and .43 for therapist-
rated alliance (N ⫽ 140 improvement-in-prediction correlations for each
mean). Each improvement-in-prediction correlation was the correlation
between session number and the correlation of residual gain (five mea-
sures ⫻ three occasions of assessment, except one measure was not
administered at one of the assessments) with alliance (five ARM scales),
calculated for each of two treatment groups (Stiles et al., 1998).
 CONVERGENT VALIDITY OF THE ARM AND THE WAI
for example, the content of the ARM Openness scale overlaps with
the Patient Working Capacity scale of the California Psychother-
apy Alliance Scales (CALPAS; Marmar, Horowitz, Weiss, &
Marziali, 1986; Marmar, Weiss, & Gaston, 1989). The content of
the ARM Confidence scale overlaps with therapist and client
Confident Collaboration factors identified in analyses of pools of
items drawn from several alliance measures (Hatcher, 1999;
Hatcher & Barends, 1996). Empirically, in comparisons made
within each instrument and within client, therapist, or observer
perspectives, the ARM Bond, Partnership, and Confidence scales
and, similarly, the three WAI scales (Bond, Tasks, and Goals) have
been highly intercorrelated (r ⬎ .80 in most comparisons; e.g.,
Agnew-Davies et al., 1998; Horvath, 1994; Horvath & Greenberg,
1986, 1989; Raue et al., 1997). The ARM Openness and Initiative
scales were less strongly correlated with other ARM scales (.18 ⱕ
r ⱕ .66; Agnew-Davies et al., 1998).
The high intercorrelations among some alliance scales would
justify collapsing them into a core alliance index, for example,
aggregating the ARM Bond, Partnership, and Confidence scales or
the WAI Bond, Tasks, and Goals scales. For example, Kivlighan
and Shaughnessy (2000) measured the alliance as the average of
the three WAI scales in their study of patterns of relationship
development in counseling dyads. Other alliance investigators,
however, continue to use the scales separately to retain the con-
ceptual distinctions, and there have been some findings suggesting
differential relations with other variables (Horvath, 1994; Horvath
& Greenberg, 1989; Horvath & Luborsky, 1993). For example, in
one of the clinical trials from which the present samples were
drawn, clients’ ARM Confidence mean (i.e., Confidence scores
averaged across sessions) was more highly correlated with residual
gain at end of treatment on the Symptom Checklist—90 —Revised
(r ⫽ .30, p ⬍ .01) than was their ARM Bond mean (r ⫽ .14, ns;
Stiles et al., 1998). We examined convergence for each scale
separately, and we have addressed the issue of overlap in our
discussion.
Previous Research on the Convergent Validity
of Alliance Measures
The few previous direct comparisons of alliance measures with
each other have tended to show strong convergence within client,
therapist, and observer perspectives but moderate to poor conver-
gence between these different perspectives. Tichenor and Hill
(1989) compared client, therapist, and observer versions of the
WAI with observer-rated versions of three other measures of
working alliance, the CALPAS, the Penn Helping Alliance Scales
(PENN; Alexander & Luborsky, 1986), and the Vanderbilt Ther-
apeutic Alliance Scale (VTAS; Hartley & Strupp, 1983), each
aggregated across its constituent scales to form a (core) alliance
index. These measures were averaged across four sessions from
each of eight therapist– client dyads. All of the clients were
women 32 to 60 years old; therapists included four men and four
women 34 to 78 years old, with 5 to 42 years postdoctoral
experience (described more fully by Hill, 1989). The CALPAS,
PENN, and VTAS were all highly correlated with the observer-
rated WAI (.71 ⱕ r ⱕ .84; N ⫽ 8) and had more mixed intercor-
relations among themselves (.34 ⱕ r ⱕ .80). None of the four
observer-rated measures, however, was significantly correlated
with either the therapist or client self-report WAI ratings.
211
In a somewhat similarly designed study with a larger sample of
clients (N ⫽ 60) drawn from a clinical trial of three treatments for
substance abuse, Cecero et al. (2001) compared client-and
therapist-rated WAI scales with observer-rated WAI, CALPAS,
PENN, and VTAS scales. The clients included 15 (25%) women
and 30 (50%) minority participants; 35 (58%) were unemployed.
All met criteria for a diagnosis of cocaine dependence and alcohol
abuse or dependence and had been randomly assigned to
cognitive– behavioral treatment (n ⫽ 21), 12-step facilitation (n ⫽
14), or clinical management (n ⫽ 25). Cecero et al. observed,
“There was a pattern of strong positive correlations among the
observer-rated measures and more modest yet significant correla-
tions between the observer-rated measures and the therapist ver-
sion of the WAI” (p. 7). The client-rated WAI “was not signifi-
cantly related to any of the observer-rated measures” (p. 6). Within
the pattern of strong correlations among observer-rated measures,
however, there were anomalies for some scales. For example, the
CALPAS Mutual Goals scale was correlated only .19 with the
WAI Goals scale, although it was correlated .45 with the VTAS
Mutuality scale.
Safran and Wallner (1991) compared self-report versions of the
CALPAS and the WAI in a sample of 22 clients (11 men and 11
women, 24 to 52 years old) who presented with depression-related
symptoms (54%), anxiety-related symptoms (32%), or a combina-
tion of both depression- and anxiety-related symptoms (14%) and
received time-limited (20-session) cognitive therapy from 1 of 9
therapists (5 master’s level and 4 doctoral level, with 1 to 5 years
of experience with the approach). Clients completed both measures
after their third session. Safran and Wallner found high correla-
tions of the CALPAS Patient Commitment and Therapist Positive
Contribution scales with the three WAI scales (.64 ⱕ r ⱕ .82).
CALPAS Goal Disagreement was correlated strongly with WAI
Goals (r ⫽ .73) and Tasks (r ⫽ .68) scales but more moderately
with WAI Bond (r ⫽ .39). CALPAS Patient Working Capacity
and Therapist Negative Contributions scales had lower correla-
tions with the WAI scales (.07 ⱕ r ⱕ .48).
Bachelor (1991) compared self-report versions of the PENN, the
Vanderbilt Psychotherapy Process Scale (VPPS; Suh, Strupp, &
O’Malley, 1986), and the Therapeutic Alliance Rating System
(TARS; Marziali, 1984), each completed two or three times
(roughly, at the 3rd, 10th and final sessions, averaged for analyses)
by 37 female clients (M age ⫽ 31.2 years) and 10 male clients (M
age ⫽ 28.9 years), including students and community residents,
seen for therapy in a university consultation service, and by their
therapists, who were master’s-level students in clinical psychol-
ogy. Diagnoses, as noted on the service’s form, included 33%
interpersonal problems, 28% personality disorders, 35% psycho-
neuroses, and 4% marital or sexual problems, or both. Although
both client- and therapist-rated alliance measures were gathered,
only within-perspective correlations were reported. These showed
moderate to good agreement between conceptually corresponding
scales; the strongest correlations involved scales that described
positively toned characteristics of the therapist, for example, the
PENN Type 1 scale, which reflects the client’s experience of
receiving help or a helpful attitude from the therapist, the VPPS
Therapist Warmth and Friendliness scale, and the TARS Therapist
Positive scale (.62 ⱕ r ⱕ .82 within each perspective).
In a series of studies, Hatcher and colleagues (Hatcher, 1999;
Hatcher & Barends, 1996; Hatcher, Barends, Hansell, & Gut-
212 STILES ET AL.
freund, 1995) studied client and therapist self-report versions of
the WAI, the CALPAS, and (in the first two studies) the PENN,
collected at one assessment occasion per dyad (after varying
numbers of sessions) in a university psychology clinic. They did
not directly compare the scales usually scored on these instruments
but instead investigated the factor structure of the global alliance
(total) scores (in the first study) or of the aggregate pools of items
(in the second and third studies). First, Hatcher et al. used confir-
matory factor analysis on global alliance scores by 38 therapists
and their 144 clients and confirmed three general alliance fac-
tors—a shared-view factor incorporating client and therapist
views, along with separate client and therapist factors. The load-
ings on the shared-view factor suggested that clients and therapists
tended to agree on helpfulness and on therapist clarity about goals
and tasks. Next, Hatcher and Barends used exploratory factor
analysis on ratings by 231 clients and identified six factors in the
full pool of items, notably a Confident Collaboration factor that
drew items from all three measures and that showed the highest
correlation of any factor with clients’ estimates of improvement,
gathered at the same assessment occasion (r ⫽ .37). Finally, using
Perfect Congruence Analysis (Ten Berge, 1986), Hatcher first
identified components in the WAI and the CALPAS in a survey
sample (251 therapists who each rated one client selected from
their current practice) and then confirmed the components in
ratings gathered in the university psychology clinic (ratings by 63
therapists of 259 clients). In separate analyses, he confirmed four
components in the WAI and five components in the CALPAS,
none of which corresponded closely to the scales usually scored.
Then he confirmed a Therapists’ Confident Collaboration factor,
which drew items from both measures and had the highest corre-
lation with therapist estimates of improvement (r ⫽ .64 and .62 in
the two samples). In a subsample of the clinic group (n ⫽ 190),
Therapists’ Confident Collaboration also had a small but signifi-
cant correlation with patients’ estimates of improvement (r ⫽ .17,
p ⬍ .02). Correlations of the therapist factors with the client
alliance factors previously identified by Hatcher and Barends
based on this subsample ranged from negligible to moderate
(⫺.05 ⱕ r ⱕ .38), consistent with previous interperspective
comparisons.
Dyad and Session Levels of Analysis
We assessed convergent validity of the ARM and the WAI at
two levels: the dyad level (correlations of means across therapist–
client pairs) and the session level (correlations of deviation scores
across sessions within dyads), capitalizing on previously collected
samples in which both the ARM and the WAI were applied across
multiple sessions of each dyad. As we use the term, dyad-level
mean is the mean of a scale across a dyad’s sessions. Dyad-level
means thus reflect characteristics of a particular client–therapist
pairing averaged across sessions.2 A session-level deviation score
is the deviation of a raw scale score from the mean score for that
case (i.e., the difference between the raw score and the correspond-
ing dyad-level mean). Session-level deviation scores thus reflect
session-to-session variation within a case. To illustrate, alliance is
considered as a dyad-level variable when it is used to predict
treatment outcome but as a session-level variable when it is used
to follow alliance rupture and repair cycles, which are thought to
be a therapeutically important arena for in-session work on clients’
relationship problems (Safran, Crocker, McMain, & Murray, 1990;
Safran & Muran, 1996). Correlations among alliance scores at the
dyad level are independent of correlations among alliance scores at
the session level, and the interpretations of dyad-level correlations
are different from those of session-level correlations (Dill-
Standiford, Stiles, & Rorer, 1988; Norman, 1967). In the interscale
comparisons reviewed earlier (Bachelor, 1991; Cecero et al., 2001;
Hatcher, 1999; Hatcher & Barends, 1996; Hatcher et al., 1995;
Safran & Wallner, 1991; Tichenor & Hill, 1989), investigators
correlated alliance scores only across dyads, even when data were
gathered from several of each dyad’s sessions.
Aims and Design
We aimed to assess convergent validity for multiple dimensions
of the alliance as measured by the ARM and the WAI within and
between the perspectives of client, therapist, and observer at dyad
and session levels. We were particularly interested in assessing
convergence for the ARM scales. Whereas the WAI scales had
shown good convergence with core alliance scales on other instru-
ments (albeit only within perspectives at the dyad level; Bachelor,
1991; Cecero et al., 2001; Safran & Wallner, 1991; Tichenor &
Hill, 1989), the ARM was relatively new, and its convergent
validity was previously untested.
According to the measures’ conceptualization, the strongest
convergence should be (a) between the Bond scales on the two
measures and (b) between ARM Partnership and the WAI Goals
and Tasks scales. Other correlations should be weaker. However,
more pragmatically, the previously noted findings of high inter-
correlations among ARM Bond, Partnership, and Confidence
scales and among WAI Bond, Tasks, and Goals scales led us to
expect substantial correlations between these sets of scales, with
ARM Openness and Initiative appearing more distinct.
Our data were drawn from two previously reported comparative
clinical trials of brief therapy for depression, a collaborative psy-
chotherapy project (CPP) carried out in three outpatient facilities
of the National Health Service (NHS) of the United Kingdom
(Barkham et al., 1996) and the Second Sheffield Psychotherapy
Project (SPP2), conducted in a university-based research clinic in
the United Kingdom (Shapiro et al., 1994). The CPP was designed
as a replication and extension of SPP2. In both projects, the ARM
was the primary measure of the alliance, completed by clients and
therapists after each session. The WAI was used only in subsets of
the sessions, and the present study considered only the sessions for
which both ARM and WAI data were available. In one half of the
CPP cases, both therapists and clients completed the WAI after
each session (yielding data on 198 sessions of 18 clients and 4
therapists); these data have not been previously reported. In SPP2,
observers applied the WAI to selected audiotaped sessions (78
sessions of 39 clients and 5 therapists). There have been previous
2
We did not assess convergent validity separately at the therapist level
(mean scores aggregated across each therapist’s clients) and client level
(deviation of client-level means from therapist-level means). Most appli-
cations of alliance measures consider the dyad to be the relevant unit (e.g.,
for comparisons with outcome), folding effects of therapist differences into
a dyad-level mean, whose convergent validity was thus of focal interest
(see reviews by Horvath & Bedi, in press; Horvath & Symonds, 1991;
Orlinsky et al., 1994; but see Hatcher et al., 1995, for an exception).
 CONVERGENT VALIDITY OF THE ARM AND THE WAI 213

reports of the ARM data in the full SPP2 sample (Agnew-Davies
et al., 1998; Stiles et al., 1998) and of the observer WAI data (Raue
et al., 1997), but these ARM and WAI data have not previously
been compared.
CPP and SPP2 had very similar overall designs. Briefly, clients
who met criteria that included primarily a diagnosis of major
depressive episode were randomly assigned to receive either 8
or 16 sessions of either cognitive– behavioral (CB) or
psychodynamic–interpersonal (PI) therapy. Clients’ degree of
change was assessed at the end of treatment and at follow-up
assessments 3 months and 1 year after treatment. The treatments
studied in CPP and SPP2 were generally effective. Clients in all
cells of the design averaged substantial improvement in both
studies. These results, along with results of comparisons among
experimental conditions, have been reported previously (Barkham
et al., 1996; Shapiro et al., 1994, 1995).
Study 1: Comparisons Within and Between Client and
Therapist Perspectives
To study how the ARM converged with the WAI within and
between the client and therapist perspectives, we drew data from
the CPP (Barkham et al., 1996).
Method
The WAI. The WAI (Horvath & Greenberg, 1986, 1989) is composed
of 36 items on 7-point scales, with parallel items in the client and therapist
self-report versions. It includes three scales, each composed of 12 items:
therapeutic Bond (e.g., “My therapist and I understand each other”),
agreement on Tasks (e.g., “I am clear about what my responsibilities are in
therapy”), and agreement about Goals (e.g., “The goals of these sessions
are important for me”). Internal consistency reliabilities of the three WAI
scales are reported in Table 1.
The ARM. The ARM (Agnew-Davies et al., 1998) is composed of 28
sentences describing the client, the therapist, and the client–therapist rela-
tionship, rated on parallel forms by clients and therapists using 7-point
scales anchored from strongly disagree to strongly agree. Instructions on
the form read, “Thinking about today’s meeting, please indicate how
strongly you agree or disagree with each statement.” Parallel items concern
the same person’s experience as viewed from two perspectives. For exam-
ple, the item “I feel friendly towards my therapist” in the client version is
considered as parallel to the item “My client is friendly towards me” in the
therapist version (note that the latter item asks whether the client “is
friendly,” a judgment from observation, rather than whether the client
“feels friendly,” an inference about the client’s private feeling).
Table 1
Means, Standard Deviations, and Internal Consistency of the
ARM and the WAI
CPP SPP2
(n ⫽ 198 sessions)a (n ⫽ 78 sessions)b

Participants. Clients (n ⫽ 18) were 11 women and 7 men with a mean Alliance scale M SD ␣ M SD ␣
age of 39 years (range ⫽ 19 –55) who had received a diagnosis of major
depressive episode and had met other inclusion and exclusion criteria, Clients
including (a) continuous history of the presenting disorder less than 2 ARM
years, (b) no more than three sessions of formal psychotherapy within Bond (6) 6.08 0.98 .87 6.07 0.77 .81
previous 5 years, and (c) no significant change in psychotropic medication Partnership (4) 6.06 0.97 .81 5.96 0.90 .78
within the previous 6 weeks. A further criterion that clients be employed in Confidence (7) 5.91 0.98 .84 5.96 0.85 .86
a professional, managerial, or other white-collar occupation was aban- Openness (5) 5.75 1.05 .73 5.72 1.06 .78
doned part way through the study because of difficulty finding NHS clients Initiative (4) 4.23 1.00 .59 4.48 0.84 .54
who met that criterion. The clients were seen for psychotherapy in three WAI
NHS hospitals, in Leicester, Huddersfield, and Sheffield, England. By Bond (12) 5.79 0.97 .91
Tasks (12) 5.73 1.01 .92
design, they represented a randomly selected half of the 36 CPP clients
Goals (12) 5.48 1.07 .90
(Barkham et al., 1996) who completed the WAI as well as the ARM after
each of their sessions. The other half of the CPP clients instead completed Therapists
measures dealing with another topic. All of the participating clients gave
written informed consent for their data to be used for research. ARM
The therapists in CPP were 4 clinical psychologists (1 man and 3 Bond (6) 5.91 0.69 .83 5.38 0.91 .87
Partnership (4) 5.52 0.90 .83 5.15 1.03 .77
women, with 0 to 6 years of experience since completion of professional
Confidence (7) 5.31 0.85 .89 4.96 1.08 .87
training) employed in the clinics where the study was conducted. They Openness (5) 5.70 0.95 .89 5.07 1.08 .85
were investigators in the project, but, as clients were told, they did not have Initiative (4) 4.99 0.73 .47 4.62 0.59 .41
access to research data until after treatment was completed. The therapists WAI
were selected for, and encouraged to maintain, a balanced belief in the Bond (12) 5.82 0.68 .90
effectiveness of both CB and PI therapies. All of the therapists were trained Tasks (12) 5.46 0.85 .93
in both CB and PI treatment protocols, and each therapist’s clients were Goals (12) 5.28 0.95 .92
distributed approximately evenly across all cells in the design.
Observers
Treatments. The CB and PI therapies have been described in the
previous reports and in manuals developed for the Sheffield projects (Firth WAI
& Shapiro, 1985; Shapiro & Firth, 1985). Briefly, the CB treatment was a Bond (12) 6.13 0.46 .95
multimodal method emphasizing the provision of a wide range of cognitive Tasks (12) 6.01 0.50 .94
and behavioral strategies, including anxiety-control training, self- Goals (12) 6.07 0.48 .94
management, and cognitive restructuring (Beck, Rush, Shaw, & Emery,
Note. Numbers in parentheses indicate the number of items on the sub-
1979; Goldfried & Merbaum, 1973; Snaith, 1974). The PI treatment was
scales. ARM ⫽ Agnew Relationship Measure; WAI ⫽ Working Alliance
based on Hobson’s (1985) Conversational Model of therapy and used a Inventory; CPP ⫽ Collaborative Psychotherapy Project; SPP2 ⫽ Second
combination of psychodynamic, interpersonal, and experiential concepts. It Sheffield Psychotherapy Project.
focused on the client–therapist relationship as a vehicle for revealing and a
Means based on 186 to 196 sessions because of missing data on some
resolving interpersonal difficulties, which were viewed as primary in the items. b Means based on 75 to 77 sessions because of missing data on
origins of depression. some items.
214 STILES ET AL.

The ARM includes five scales. The following examples are from the
client version (see Agnew-Davies et al., 1998, for a list of all of the items).
The scales were Bond (6 items, e.g., “My therapist accepts me no matter
what I say or do”), Partnership (4 items, e.g., “My therapist follows his/her
own plans, ignoring my views of how to proceed”; reversed), Confidence
(7 items, e.g., “I have confidence in my therapist and his/her techniques”),
Openness (5 items, e.g., “I feel I can openly express my thoughts and
feelings to my therapist”), and Client Initiative (4 items, e.g., “I take the
lead when I’m with my therapist”). Two of the ARM’s 28 items were not
used in any scale because their factor loadings were low or inconsistent
across client and therapist perspectives. Internal consistency reliabilities of
the five ARM scales are reported in Table 1.
CPP procedure. Parallel procedures for client selection were used at
the three NHS sites. Referral letters were scanned for reference to depres-
sion as a presenting problem, and background information was checked for
consistency with criteria for admission to the study. Possible referrals were
brought to a weekly meeting attended by all of the participating therapists.
In the absence of excluding evidence, clients were mailed an invitation, an
information sheet on the project, and screening measures for completion.
Clients returning materials who appeared likely to meet criteria were then
offered a clinical interview.
Clients were interviewed by independent assessors. The interviews were
structured to gather sufficient information to determine Diagnostic and
Statistical Manual of Mental Disorders (3rd ed.; American Psychiatric
Association, 1980) diagnoses of major depressive episode, generalized
anxiety disorder, and panic disorder. Clients also completed a battery of
self-report assessment measures. Clients were excluded for psychotic,
manic, or obsessional symptoms, or if depression was attributable to
organic illness. Clients meeting criteria were randomly assigned to one of
the four treatment conditions. All of the participating clients signed a
consent form describing the treatment they were to receive and outlining
the schedule for assessments. The 18 clients who provided ARM and WAI
data included 6 assigned to 8 sessions of CB, 5 assigned to 8 sessions of
PI, 3 assigned to 16 sessions of CB, and 4 assigned to 16 sessions of PI. For
further details regarding client selection and assessment procedures, see
Barkham et al. (1996).
Sessions took place weekly, and missed sessions were rescheduled.
Clients and therapists completed the ARM and the WAI immediately after
each session. Completed forms were returned to clinic secretaries with the
understanding that they would not be examined until the therapy was
completed. We used WAI and ARM data from all 198 of the 18 clients’
sessions (2 scheduled sessions of 1 client’s treatment were not conducted).
On some forms, however, a few items were not completed (less than 1%),
so scores could not be calculated on a few scales for a few sessions.
Data reduction for analysis. We combined data from CB and PI
treatments and for 8- and 16-session treatments in our analyses. Hetero-
geneity in types of treatments is appropriate for assessing convergent
validity of alliance measures, insofar as one potential use of these measures
is to compare the alliance across treatments.
Raw scores of each WAI and ARM scale were calculated as the means
of constituent items (each scored 1–7, reversed for negatively worded
items) for each session. Scale scores were treated as missing if any
constituent item on the scale was missing. Means, standard deviations, and
internal consistencies for each scale are shown in Table 1. Internal con-
sistencies for all of the scales were good, except for the ARM Initiative
scale; they were slightly higher for WAI scales than for ARM scales,
possibly reflecting the WAI scales’ greater number of items.
We calculated dyad-level means as the means of each WAI or ARM raw
score on each scale across each client’s sessions. We then calculated
session-level deviation scores as deviations of the raw scores from each
therapist’s or client’s corresponding dyad-level mean.
Results
The first six columns of Table 2 show the dyad-level correla-
tions between the ARM and the WAI in the CPP. Dyad-level
interscale correlations indicate the degree to which respondents
who reported generally strong or weak alliances on the ARM
reported similarly strong or weak alliances on the WAI.
The first six columns of Table 3 show the session-level corre-
lations between the ARM and the WAI in the CPP. Session-level
interscale correlations indicate the extent to which alliance scores
covaried across sessions for an average dyad.
The number of correlations we calculated made it inappropriate
(because of family-wise Type 1 error), as well as impractical, to
consider each nominally significant result separately. A full Bon-
ferroni correction for the 90 correlations in each table (including

Table 2
Correlations of Dyad-Level Means of the ARM and the WAI

WAI scale

CPP clients (n ⫽ 18) CPP therapists (n ⫽ 18) SPP2 observers (n ⫽ 39)

ARM scale Bond Tasks Goals Bond Tasks Goals Bond Tasks Goals

Clients
Bond .91** .89** .86** .74** .65* .69* .31 .11 .14
Partnership .91** .92** .91** .58* .56* .59* .44* .42* .42*
Confidence .90** .96** .91** .66* .66* .70* .30 .19 .24
Openness .71* .53* .63* .41 .32 .36 .24 .11 .12
Initiative .21 .50* .45 .13 .14 .12 .05 ⫺.08 ⫺.12
Therapists
Bond .65* .62* .61* .97** .85** .83** .20 .07 .11
Partnership .60* .67* .64* .91** .96** .91** .35* .47* .44*
Confidence .69* .68* .66* .82** .96** .92** .21 .30 .35
Openness .70* .58* .55* .82** .79** .75** ⫺.09 .06 .08
Initiative .32 .28 .40 .55* .46 .49* .07 .15 .12

Note. ARM ⫽ Agnew Relationship Measure; WAI ⫽ Working Alliance Inventory; CPP ⫽ Collaborative
Psychotherapy Project; SPP2 ⫽ Second Sheffield Psychotherapy Project.
* nominal p ⬍ .05. ** nominal p ⬍ .0005 (Bonferroni-corrected p ⬍ .05).
 CONVERGENT VALIDITY OF THE ARM AND THE WAI 215

Table 3
Correlations of Session-Level Deviation Scores of the ARM and the WAI, Adjusted for Mean
Differences Among Therapist–Client Pairs

WAI scale

CPP clients CPP therapists SPP2 observers

(n ⫽ 198 sessions)a (n ⫽ 198 sessions)a (n ⫽ 78 sessions)b

ARM scale Bond Tasks Goals Bond Tasks Goals Bond Tasks Goals

Clients
Bond .70** .60** .63** .29** .23** .32** .48** .43** .46**
Partnership .54** .54** .61** .20* .15* .19* .36* .45** .51**
Confidence .56** .63** .61** .21* .22* .27** .25* .29* .32*
Openness .43** .34** .37** .27** .13 .19* .22 .40** .35*
Initiative .24* .21* .23* .15 .00 .10 .09 .14 ⫺.09
Therapists
Bond .33** .23* .28** .81** .67** .57** .60** .61** .61**
Partnership .38** .42** .36** .79** .80** .80** .55** .54** .59**
Confidence .31** .43** .38** .74** .85** .83** .55** .59** .59**
Openness .34** .30** .27* .65** .52** .66** .28* .39** .31*
Initiative .24* .34** .26* .40** .43** .50** ⫺.03 .05 .04

Note. ARM ⫽ Agnew Relationship Measure; WAI ⫽ Working Alliance Inventory; CPP ⫽ Collaborative
Psychotherapy Project; SPP2 ⫽ Second Sheffield Psychotherapy Project.
a
Correlations based on 168 to 188 sessions because of missing data on some items. b Correlations based on 75
to 77 sessions because of missing data on some items.
* nominal p ⬍ .05. ** nominal p ⬍ .0005 (Bonferroni-corrected p ⬍ .05).

SPP2 as well as CPP analyses) demands a nominal significance
level of .000555 to achieve a conventional .05 significance level
for any single correlation. Correlations that met this criterion are
indicated in the tables. Ignoring all of the correlations that failed to
achieve this corrected significance level, however, would yield an
unacceptable rate of Type 2 errors—falsely accepting the null
hypothesis and thus overlooking relations that may be real. We
offer some observations based on the broad patterns of correlations
and the magnitude of the effects. Our observations focus on this
study’s goal of assessing convergent validity of ARM and WAI
dimensions within and between client and therapist perspectives at
both dyad and session levels.
1. Within client and therapist perspectives, the ARM Bond,
Partnership, and Confidence scales were strongly correlated with
all three WAI scales (Bond, Tasks, Goals) at both dyad and session
levels. For example, the Bond scales on the two measures were
correlated .91 for clients and .97 for therapists at the dyad level
(Table 2) and .70 for clients and .81 for therapists at the session
level (Table 3).
2. The conceptual expectation that the correlations between
ARM and WAI Bond scales and between ARM Partnership and
WAI Tasks and Goals scales should be distinctively higher than
other intercorrelations was not fulfilled. Instead, the within-
perspective correlations of all of these scales with each other (and
with ARM Confidence) appeared generally similar (Tables 2 and 3).
3. The ARM Openness and Initiative scales had relatively lower
correlations with WAI scales, although most of them were positive
and some were substantial.
4. WAI–ARM correlations within client and therapist perspec-
tives were generally higher than correlations between these two
perspectives at both dyad and session levels (Tables 2 and 3).
Study 2: Comparisons of the Observer Perspective With
Client and Therapist Perspectives
To study how the client- and therapist-rated ARM converged
with the observer-rated WAI, we drew data from SPP2 (Shapiro et
al., 1994).
Method
Participants. Clients (n ⫽ 39) were 25 women and 14 men with a
mean age of 41 years (range ⫽ 23– 60) who met the same criteria as clients
in CPP and, in addition, worked in professional, managerial, and other
white-collar occupations. They were self-referred or referred by general
practitioners or occupational health workers for treatment of depression.
They were seen for treatment in a research clinic in Sheffield, United
Kingdom. All of the participating clients gave written informed consent for
their data to be used for research.
The 39 SPP2 clients whose sessions were measured with both the ARM
and the WAI were a subset of 117 clients who participated in SPP2. They
represented the intersection of two subgroups: (a) those who completed the
final version of the ARM (n ⫽ 79) and (b) those whose sessions were
subsequently rated using the observer version of the WAI (n ⫽ 57).
Because the ARM was being developed while SPP2 was in progress, the
final version was used only by the last 79 SPP2 clients (Agnew et al.,
1998). After the completion of SPP2, 2 sessions from each of the 57 clients
who had been assigned to 16-session treatments were rated on the observer
version of the WAI by Raue et al. (1997), as described later.
The therapists in SPP2 were 5 research clinical psychologists working in
the research clinic (3 men and 2 women, with 1 to 17 years of experience
since completion of professional training). None of them were therapists in
CPP. As in CPP, however, the therapists were investigators in the project
and trained in both CB and PI therapies, and their clients were distributed
across cells in the design. Treatments followed the same protocols as those
in CPP.
216 STILES ET AL.

The observer-rated WAI. The observer-rated version of the WAI (Raue three observer WAI scales than did the ARM Openness and
et al., 1997; Tichenor & Hill, 1989), like the self-report version, is com- Initiative scales, as was the case at both levels in Study 1.
posed of 36 items on a 7-point scale. Items in the three 12-item scales are 2. As in Study 1, the conceptual expectation of distinctively
parallel to those in the self-report versions but reworded to represent an higher correlations between ARM and WAI Bond scales and
external perspective (Bond, e.g., “There is a good understanding between
between ARM Partnership and WAI Tasks and Goals scales was
the client and therapist”; Tasks, e.g., “There is agreement about what the
not fulfilled.
client’s responsibilities are in therapy”; and Goals, e.g., “There is a per-
ception that the goals of the sessions are important for the client”). Internal 3. At the dyad level (Table 2), the ARM Partnership scale was
consistency reliabilities of the three observer-rated WAI scales are reported more highly correlated with the observers’ WAI scales than were
in Table 1. the other ARM scales. This pattern was apparent for both clients’
SPP2 procedure. As in CPP, clients who met screening criteria were and therapists’ ARM ratings.
invited for an assessment interview at which the battery of assessment 4. At the dyad level (Table 2), most of the correlations of
measures was administered. Clients who were diagnosed with major de- observers’ WAI scales with clients’ and therapists’ ARM scales
pressive episode and met the other criteria were invited to join the study (Partnership excepted) were surprisingly low. As noted earlier, the
and, if they accepted, were randomly assigned to one of the treatment two sessions that contributed to each dyad-level mean in SPP2
conditions. Informed consent was obtained before randomization. Clients
were selected as extreme with respect to therapist-judged helpful-
were seen weekly, and missed sessions were rescheduled. Of the 39 SPP2
ness. In an exploratory analysis we recalculated these correlations
clients in this study, 20 were assigned to 16 sessions of CB therapy and 19
were assigned to 16 sessions of PI therapy. Batteries of assessment mea- separately for the least helpful and the most helpful sessions (Table
sures were readministered at the end of treatment, at 3-month follow-up, 4). For clients, the resulting pattern of correlations in the least and
and at 1-year follow-up (see Shapiro et al., 1994, 1995, for further details most helpful sessions were similar to each other and to that shown
regarding SPP2 procedures). in Table 2. However, the correlations of therapist ARM scores
As in CPP, clients and therapists completed the ARM immediately after with observer WAI scores were much higher in the least helpful
each session. Completed forms were returned to clinic secretaries with the sessions than in the most helpful sessions, as shown in Table 4.
understanding that they would not be examined until the therapy was 5. Looking across studies, at the session level (Table 3), corre-
completed. lations of clients’ and therapists’ ARM ratings with observers’
Session selection and observer WAI rating. Two sessions from each of
WAI ratings (SPP2) were generally higher than were correlations
the 39 clients were rated on the observer version of the WAI (Raue et al.,
between client ARM and therapist WAI ratings or between ther-
1997). These sessions were selected mainly on the basis of therapists’
global ratings of session helpfulness on a 7-point scale, completed imme- apist ARM and client WAI ratings (CPP). For example, client
diately after each session. One was the session rated as the most helpful, Partnership was correlated .51 with observer Tasks but only .19
and the other as the least helpful, among Sessions 4 –13 (i.e., excluding the with therapist Tasks. Put another way, at the session level, thera-
first 3 and last 3 sessions). In cases of helpfulness ratings that were tied or pists and clients seemed to converge more with the observers than
within one point of each other (40% of the cases), the selection from among they did with each other. This comparison should be considered
these sessions was based on therapists’ ratings on the Session Evaluation cautiously, however, insofar as it was based on two different
Questionnaire (Stiles, Reynolds, Hardy, Rees, Barkham, & Shapiro, 1994), samples.
which was also completed after every session. (See Raue et al., 1997, for
further details regarding session selection.)
The raters for the observer form of the WAI were 6 students in a U.S.
clinical psychology doctoral program who were trained for approximately Table 4
one month to adequate reliability among themselves (intraclass correlation Correlations of Therapists’ Dyad-Level ARM Means With
coefficient ⬎ .60). They also met regularly throughout the rating period to Observers’ Dyad-Level WAI Means in SPP2 Sessions Judged by
maintain calibration (see Raue et al., 1997, for further details of rater the Therapist as Least or Most Helpful
characteristics, selection, and training). Rotating pairs of raters indepen-
dently rated the sessions by listening to the audiotape and reading the Observer WAI scale
transcript. Thus, each rater rated one third of the sessions, which were
presented in randomized order at the rate of approximately two per week. Therapist ARM scale Bond Tasks Goals
Data reduction for analysis. ARM and WAI scores, including dyad-
level means and session-level deviation scores, were calculated in the same Least helpful sessions (n ⫽ 39)a
Bond .43* .32* .32*
way as in Study 1. Means, standard deviations, and internal consistencies
Partnership .62** .58** .56**
are shown in Table 1. Confidence .47* .45* .44*
Openness .04 .19 .11
Results Initiative .16 .27 .23
Most helpful sessions (n ⫽ 39)a
The last three columns of Table 2 show the dyad-level correla- Bond ⫺.01 ⫺.21 ⫺.21
tions of the client- and therapist-rated ARM with the observer- Partnership ⫺.08 .08 .06
Confidence ⫺.06 .06 .12
rated WAI in SPP2. The last three columns of Table 3 show the
Openness ⫺.21 ⫺.11 ⫺.08
corresponding session-level correlations. Continuing our strategy Initiative ⫺.11 ⫺.01 ⫺.05
in reporting the Study 1 results, and mindful that there were many
correlations addressing similar questions, we focused on a few Note. Observer WAI ratings drawn from SPP2. ARM ⫽ Agnew Rela-
broad patterns that bear on this study’s goal of assessing the tionship Measure; WAI ⫽ Working Alliance Inventory; SPP2 ⫽ Second
Sheffield Psychotherapy Project.
convergent validity of ARM and WAI dimensions. a
Some correlations based on only 37 or 38 sessions because of missing
1. At the session level (Table 3), the ARM Bond, Partnership, data on some ARM items.
and Confidence scales had generally higher correlations with the * p ⬍ .05. ** p ⬍ .0005.
 CONVERGENT VALIDITY OF THE ARM AND THE WAI
General Discussion
For the core alliance scales on the ARM and the WAI, the
within-perspective dyad-level comparisons showed excellent con-
vergent validity, extending previous findings (Bachelor, 1991;
Safran & Wallner, 1991; Tichenor & Hill, 1989). Specifically, the
correlations of the ARM Bond, Partnership, and Confidence scales
with the WAI Bond, Goals, and Tasks within client and therapist
perspectives in CPP (Table 2) were all in the .80s and .90s. For
measuring these core aspects of the alliance at the dyad level, then,
the ARM and the WAI seemed strong and nearly equivalent.3 It
should be noted that averaging WAI and ARM scores across all of
each client’s 8 or 16 sessions in the CPP sample probably made
these dyad-level means particularly strong and stable estimates of
the alliance in these dyads, in contrast to the weaker dyad-level
estimates in the SPP2 sample, discussed later.
Within-perspective convergent validity for the core alliance
scales also seemed strong at the session level (Table 3), although
the correlations were somewhat lower (.54 ⱕ r ⱕ .70 for clients;
.57 ⱕ r ⱕ .85 for therapists). Convergence at this level, which had
not been shown previously, suggests that these ARM and the WAI
scales should yield results that are generally similar to each other
in studies of session-to-session changes in the alliance, such as
rupture and repair or sudden gains within a case (cf. Agnew,
Harper, Shapiro, & Barkham, 1994; Tang & DeRubeis, 1999), as
they should do when they are used for addressing dyad-level
issues, such as predicting outcome. The correlations may have
been lower at this level simply because the session-level deviation
scores were based on fewer ratings and were therefore less reliable
than the dyad-level means (internal consistency of the scales was
high, but there was no assessment of test–retest reliability, which
would have involved asking participants to rate the relationship
twice on the same instrument after each session). Alternatively, it
may be that the ARM and the WAI are differentially sensitive to
some transitory aspects of the relationship. For example, clients’
reactions to session-to-session variation in the partnership may
have diverged in some way from their sense of agreement on
treatment tasks, so that their session-level deviation scores varied
around their dyad-level means differently for ARM Partnership
than for WAI Tasks, leading to lower session-level convergence
(r ⫽ .54) than dyad-level convergence (r ⫽ .92).
Finding moderate convergence between therapists and clients
for the core alliance scales at both levels in CPP represents a
further confirmation of the alliance construct, insofar as it implies
some mutual or shared experience of the relationship. This inter-
perspective convergence contrasts with the null results reported by
Tichenor and Hill (1989), but it is consistent with correlations
reported by Cecero et al. (2001) and with the shared-view global
alliance factor identified by Hatcher et al. (1995), reviewed earlier.
The correlations were of the same magnitude as those between
client and therapist dyad-level means on the ARM’s core alliance
scales in SPP2, reported in a previous study (Agnew et al., 1998).
It should be expected that the alliance will be experienced some-
what differently from different vantage points. For example, ther-
apists may tend to be more prospective, focused on making inter-
ventions likely to have a helpful impact on clients, whereas clients
may tend to be more immediate or retrospective, focused on the
comfort, safety, understanding, and life changes achieved up to
that point. So it is not surprising that the between-perspective
217
correlations were lower than the within-perspective correlations in
CPP at both levels. The notably lower client–therapist convergence
at the session level (Table 3) than at the dyad level (Table 2) could
reflect therapists being less cognizant of session-to-session fluctu-
ations in clients’ evaluations than of the enduring qualities of this
particular alliance.
For the core alliance scales at the session level in SPP2, both
clients’ and therapists’ convergence with observers was substantial
and generally higher than clients’ and therapists’ convergence with
each other in CPP (rightmost three columns of Table 3). This
comparison could reflect sample differences, but, alternatively,
seems plausibly understood as reflecting the observers’ taking both
client and therapist perspectives into account, yielding intermedi-
ate estimates of alliance qualities. Note that by selecting extreme
sessions to represent each client in SPP2, we ensured a large
within-client variance in alliance ratings (insofar as alliance ratings
tended to be higher in the most helpful sessions and lower in the
least helpful sessions; Raue et al., 1997) and thus probably opti-
mized convergence at the session level.
At the dyad level, the convergence between participants and
observers in SPP2, shown in the rightmost three columns of
Table 2, may have been anomalously low. Because the two ses-
sions that contributed to each dyad-level mean were selected as
extreme with respect to therapist-judged helpfulness, they may
have been divergent and unrepresentative. The contrast between
the negligible therapist– observer convergence in the therapists-
judged most helpful sessions and the substantial convergence in
the least helpful sessions (Table 4) raises several possibilities. (a)
The low correlations for the most helpful sessions may have
reflected a restriction of range—a ceiling effect due to uniformly
high ratings. Variances for most of the alliance scales (all except
ARM Initiative) were numerically smaller for the most than for the
least helpful sessions; however, these differences were small for
many of the scales, and they were statistically significant (by
Levene’s test for equality of variances) only for the ARM Confi-
dence scale and the WAI Goals scale. (b) As a second possibility,
perhaps therapists’ estimates of the alliance tended to be exagger-
ated or distorted in sessions they judged to be extremely helpful,
leading to low correlations with other indexes. Interperspective
convergence may be stronger when therapists take a more sober
view of the session’s accomplishments. (c) As a third possibility,
perhaps observers can more easily or accurately see when the
alliance is problematic than when it is good; that is, it may be
relatively easy to identify moments of rupture, getting stuck,
conflict, miscommunication, nonengagement, and so forth, but
relatively difficult to assess the extent of liking, trust, deepening,
and the like. Several of these factors may have contributed to the
contrasting patterns shown in Table 4.
Our results failed to show the expected differentiation among
the core alliance scales on the two instruments (ARM Bond,
3
To underline this within-perspective equivalence, we constructed core
alliance indexes as the mean of the 17 items on the ARM Bond, Partner-
ship, and Confidence scales and the mean of all 36 WAI items in the CPP
sample. The within-perspective dyad level correlations of these two in-
dexes were .98 for clients and .97 for therapists. The within-perspective
session-level correlations of the ARM and WAI core alliance indexes were
.79 for clients and .91 for therapists.
218 STILES ET AL.
Partnership, and Confidence and WAI Bond, Tasks, and Goals).
Theoretically, on the basis of Bordin’s (Bordin’s 1979, 1994)
conceptualization and the design and construction of the ARM
(Agnew et al., 1998), one would expect relatively higher correla-
tions (a) between the Bond scales on the two measures and (b)
between ARM Partnership and WAI Tasks and Goals than be-
tween other combinations of scales. The same pattern would be
expected within and between perspectives at both dyad and session
levels. Instead, we found no consistent pattern, and most of the
correlations between the core scales were of roughly comparable
magnitude within each set of comparisons. This lack of differen-
tiation helps justify the common practice of combining the sepa-
rate alliance scales into measures of global alliance (e.g., Hatcher
et al., 1995; Kivlighan & Shaughnessy, 2000; Tichenor & Hill,
1989).
The main exception to the lack of differentiation among the core
alliance scales was the distinctive convergence of participants’
ARM Partnership ratings with all three observer-rated WAI scales
at the dyad level (Table 2), more specifically, in the least helpful
sessions (Table 4). Such unexpected observations should be rep-
licated before they are strongly credited. But, perhaps, in distin-
guishing among therapeutic dyads, observers applying the WAI
scales in an undifferentiated way were distinctively sensitive to
relationship qualities that participants experienced as a partner-
ship—working as a team, sharing a view of the therapeutic tasks
and goals—as distinct from alliance aspects measured by ARM
Bond and Openness. Such occasional evidence that the core scales
are measuring something different from each other (see also Hor-
vath, 1994; Stiles et al., 1998) supports the argument for consid-
ering the core scales separately.
Most of the correlations of the WAI scales with ARM Openness
and Initiative were relatively lower than were those with the ARM
core alliance scales (Bond, Partnership, Confidence) for corre-
sponding comparisons at both levels—particularly for the within-
perspective comparisons in CPP. For example, at the dyad level in
CPP (Table 2), all of the correlations of client WAI scales with
client ARM Openness and Initiative scales were lower (.21 ⱕ r ⱕ
.71) than all of the correlations of the client WAI scales with the
client ARM core alliance scales (.86 ⱕ r ⱕ .96). These results
offered some discriminant validity. They suggest that the ARM
measures distinct aspects of the alliance not measured by the
WAI—the feeling of freedom or constriction in disclosing per-
sonal concerns and the degree to which the client took responsi-
bility for session content and process. The distinctiveness of the
Openness scale was consistent with Safran and Wallner’s (1991)
finding of relatively lower within-perspective correlations of WAI
scales with the CALPAS Patient Working Capacity scale, with
which the Openness scale overlaps conceptually (Agnew et al.,
1998). The much lower correlations involving Initiative could
partly reflect that scale’s weaker internal consistency (Table 1) as
well as client characteristics, such as motivation for change, that
are relatively independent of the core alliance dimensions.
Limitations of this study include its restrictions to clients diag-
nosed with depression and to a relatively narrow and small sample
of therapists who were also investigators. We know of no reason
why convergent validity of alliance instruments should be dis-
torted by diagnostic category. The therapist–investigators’ invest-
ment in the project might have led them to be particularly careful
and diligent in completing the numerous questionnaires and, per-
haps unwittingly, to convey their investment to their clients. Such
care and diligence should improve validity; however, personal
investment expressed as a bias toward positive ratings could im-
pair convergent validity correlations by restricting the range of
scores. The ARM and the WAI use similar rating formats (7-point
Likert scales), and although none of the items are worded identi-
cally, there is a good deal of overlap in content, raising the
possibility that method variables contributed to the convergence,
particularly between the self-report versions in CPP. So far, there
is no observer version of the ARM, so we were unable to assess
convergence within the observer perspective.
In summary, the ARM and the WAI appear to measure at least
some of the same things. Our results suggest that investigators who
favor a global alliance measure could justifiably either aggregate
the three ARM core alliance scales or use the aggregated WAI
scales. Investigators who take a multidimensional view of the
alliance would do better with the ARM. Finding convergence at
the session level as well as the dyad level, at least within client and
therapist perspectives, suggests that session-to-session changes in
the alliance can be studied with either measure, just as can case-
to-case variation. The substantially lower convergence between
perspectives than within perspectives underlines the importance of
assessing the alliance from multiple viewpoints. Research exam-
ining the different perspectives on the alliance could contribute to
fine tuning of training in the skills needed for improving the
alliance.
References
Agnew, R. M., Harper, H., Shapiro, D. A., & Barkham, M. (1994).
Resolving a challenge to the therapeutic relationship: A single case
study. British Journal of Medical Psychology, 67, 155–170.
Agnew-Davies, R., Stiles, W. B., Hardy, G. E., Barkham, M., & Shapiro,
D. A. (1998). Alliance structure assessed by the Agnew Relationship
Measure (ARM). British Journal of Clinical Psychology, 37, 155–172.
Alexander, L. B., & Luborsky, L. (1986). The Penn Helping Alliance
scales. In L. S. Greenberg & W. M. Pinsof (Eds.), The Psychotherapeu-
tic Process: A Research Handbook (pp. 325–366). New York: Guilford
Press.
American Psychiatric Association. (1980). Diagnostic and statistical man-
ual of mental disorders (3rd ed.). Washington, DC: Author.
Bachelor, A. (1991). Comparison and relationship to outcome of diverse
dimensions of the helping alliance as seen by client and therapist.
Psychotherapy, 28, 534 –539.
Barkham, M., Rees, A., Shapiro, D. A., Stiles, W. B., Agnew, R. M.,
Halstead, J., Culverwell, A., & Harrington, V. M. G. (1996). Outcomes
of time-limited psychotherapy in applied settings: Replicating the Sec-
ond Sheffield Psychotherapy Project. Journal of Consulting and Clinical
Psychology, 64, 1079 –1085.
Beck, A. T., Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive
therapy of depression. New York: Guilford Press.
Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961).
An inventory for measuring depression. Archives of General Psychia-
try, 4, 561–571.
Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of
working alliance. Psychotherapy: Theory, Research, and Practice, 16,
252–260.
Bordin, E. S. (1994). Theory and research on the therapeutic working
alliance: New directions. In A. O. Horvath & L. S. Greenberg (Eds.), The
working alliance: Theory, research and practice (pp. 13–37). New
York: Wiley.
Cecero, J. J., Fenton, L. R., Nich, C., Frankforter, T. L., & Carroll, K. M.
 CONVERGENT VALIDITY OF THE ARM AND THE WAI
(2001). Focus on the therapeutic alliance: The psychometric properties
of six measures across three treatments. Psychotherapy, 38, 1–11.
Constantino, M. J., Castonguay, L. G., & Shut, A. J. (2002). The working
alliance: A flagship for the scientist–practitioner model in psychother-
apy. In G. S. Tryon (Ed.), Counseling based on process research (pp.
81–131). New York: Allyn & Bacon.
Derogatis, L. R., Lipman, R. S., & Covi, L. (1973). SCL–90: An outpatient
rating scale. Preliminary report. Psychopharmacology Bulletin, 9, 13–
20.
Dill-Standiford, T. J., Stiles, W. B., & Rorer, L. G. (1988). Counselor–
client agreement on session impact. Journal of Counseling Psychol-
ogy, 35, 47–55.
Firth, J. A., & Shapiro, D. A. (1985). Prescriptive therapy manual for the
Sheffield Psychotherapy Project (PTRC Memo No. 734). (Available
from the Psychological Therapies Research Centre, University of
Leeds, 17 Blenheim Terrace, Leeds LS2 9JT, United Kingdom)
Goldfried, M. R., & Merbaum, M. (1973). Behavior change through
self-control. New York: Holt, Rinehart & Winston.
Hartley, D. E., & Strupp, H. H. (1983). The therapeutic alliance: Its
relationship to outcome in brief psychotherapy. In J. Masling (Ed.),
Empirical studies of psychoanalytic theories (Vol. 1, pp. 1–37). Hills-
dale, NJ: Analytic Press.
Hatcher, R. L. (1999). Therapists’ views on treatment alliance and collab-
oration in therapy. Psychotherapy Research, 9, 405– 425.
Hatcher, R. L., & Barends, A. W. (1996). Patients’ view of the alliance in
psychotherapy: Exploratory factor analysis of three alliance measures.
Journal of Consulting and Clinical Psychology, 64, 1326 –1336.
Hatcher, R. L., Barends, A., Hansell, J. & Gutfreund, M. J. (1995).
Patients’ and therapists’ shared and unique views of the therapeutic
alliance: An investigation using confirmatory factor analysis in a nested
design. Journal of Consulting and Clinical Psychology, 63, 636 – 643.
Hill, C. E. (1989). Therapist techniques and client outcomes: Eight cases
of brief psychotherapy. Newbury Park, CA: Sage.
Hobson, R. F. (1985). Forms of feeling: The heart of psychotherapy.
London: Tavistock.
Horowitz, L. M., Rosenberg, S. E., Baer, B. A., Ureno, G., & Villasenor,
V. S. (1988). Inventory of Interpersonal Problems: Psychometric prop-
erties and clinical applications. Journal of Consulting and Clinical
Psychology, 56, 885– 892.
Horvath, A. O. (1994). Empirical validation of Bordin’s pantheoretical
model of the alliance: The Working Alliance Inventory perspective. In
A. O. Horvath & L. S. Greenberg (Eds.), The working alliance: Theory,
research and practice (pp. 259 –286). New York: Wiley.
Horvath, A. O., & Bedi, R. P. (in press). The alliance. In J. C. Norcross
(Ed.), Psychotherapy relationships that work: Therapist contributions
and responsiveness to patient needs. New York: Oxford University
Press.
Horvath, A. O., & Greenberg, L. S. (1986). The development of the
Working Alliance Inventory. In L. S. Greenberg & W. M. Pinsof (Eds.),
The psychotherapeutic process: A research handbook (pp. 529 –556).
New York: Guilford Press.
Horvath, A. O., & Greenberg, L. S. (1989). Development and validation of
the Working Alliance Inventory. Journal of Counseling Psychology, 36,
223–233.
Horvath, A. O., & Greenberg, L. S. (Eds.). (1994). The working alliance:
Theory, research and practice. New York: Wiley.
Horvath, A. O., & Luborsky, L. (1993). The role of the therapeutic alliance
in psychotherapy. Journal of Consulting and Clinical Psychology, 61,
561–573.
Horvath, A. O., & Symonds, B. D. (1991). Relation between working
alliance and outcome in psychotherapy: A meta-analysis. Journal of
Counseling Psychology, 38, 139 –149.
Kivlighan, D. M., Jr., & Shaughnessy, P. (2000). Patterns of working
219
alliance development: A typology of working alliance ratings. Journal of
Counseling Psychology, 47, 362–371.
Krupnick, J. L., Sotsky, S. M., Simmens, S., Moyer, J., Elkin, I., Watkins,
J., & Pilkonis, P. A. (1996). The role of the therapeutic alliance in
psychotherapy and pharmacotherapy outcome: Findings in the National
Institute of Mental Health Treatment of Depression Collaborative Re-
search Program. Journal of Consulting and Clinical Psychology, 64,
532–539.
Marmar, C. R., Horowitz, M. J., Weiss, D. S., & Marziali, E. (1986). The
development of the therapeutic alliance rating system. In L. S. Green-
berg & W. M. Pinsof (Eds.), The psychotherapeutic process: A resource
handbook (pp. 367–390). New York: Guilford Press.
Marmar, C. R., Weiss, D. S., & Gaston, L. (1989). Toward the validation
of the California Therapeutic Alliance Rating System. Psychological
Assessment, 1, 46 –52.
Marziali, E. (1984). Three viewpoints on the therapeutic alliance: Similar-
ities, differences and associations with psychotherapy outcome. Journal
of Nervous and Mental Disease, 172, 417– 423.
Norman, W. T. (1967). On estimating psychological relationships: Social
desirability and self-report. Psychological Bulletin, 67, 273–293.
O’Malley, P. M., & Bachman, J. G. (1979). Self-esteem and education: Sex
and cohort comparisons among high school seniors. Journal of Person-
ality and Social Psychology, 37, 1153–1159.
Orlinsky, D. E., Grawe, K., & Parks, B. K. (1994). Process and outcome in
psychotherapy—Noch einmal. In A. E. Bergin & S. L. Garfield (Eds.),
Handbook of psychotherapy and behavior change (4th ed., pp. 270 –
376). New York: Wiley.
Ozer, D. J. (1985). Correlation and the coefficient of determination. Psy-
chological Bulletin, 97, 307–315.
Raue, P. J., & Goldfried, M. R. (1994). The therapeutic alliance in
cognitive– behavior therapy. In A. O. Horvath & L. S. Greenberg (Eds.),
The working alliance: Theory, research and practice (pp. 131–152).
New York: Wiley.
Raue, P. J., Goldfried, M. R., & Barkham, M. (1997). The therapeutic
alliance in psychodynamic–interpersonal and cognitive– behavioral ther-
apy. Journal of Consulting and Clinical Psychology, 65, 582–587.
Safran, J. D., Crocker, P., McMain, S., & Murray, P. (1990). Therapeutic
alliance rupture as a therapy event for empirical investigation. Psycho-
therapy, 27, 154 –165.
Safran, J. D., & Muran, J. C. (1996). The resolution of ruptures in the
therapeutic alliance. Journal of Consulting and Clinical Psychology, 64,
447– 458.
Safran, J. D., & Muran, J. C. (Eds.). (1998). The therapeutic alliance
in brief psychotherapy. Washington, DC: American Psychological
Association.
Safran, J. D., & Wallner, L. K. (1991). The relative predictive validity of
two therapeutic alliance measures in cognitive therapy. Psychological
Assessment, 3, 188 –195.
Shapiro, D. A., Barkham, M., Rees, A., Hardy, G. E., Reynolds, S., &
Startup, M. J. (1994). Effects of treatment duration and severity of
depression on the effectiveness of cognitive/behavioral and psychody-
namic/interpersonal psychotherapy. Journal of Consulting and Clinical
Psychology, 62, 522–534.
Shapiro, D. A., & Firth, J. A. (1985). Exploratory therapy manual for the
Sheffield Psychotherapy Project (Memo No. 733). (Available from the
Psychological Therapies Research Centre, University of Leeds, 17 Blen-
heim Terrace, Leeds LS2 9JT, United Kingdom)
Shapiro, D. A., Rees, A., Barkham, M., Hardy, G., Reynolds, S., & Startup,
M. (1995). Effects of treatment duration and severity of depression on
the maintenance of gains following cognitive– behavioral and
psychodynamic–interpersonal psychotherapy. Journal of Consulting and
Clinical Psychology, 63, 378 –387.
Snaith, R. P. (1974). Psychotherapy based on relaxation techniques. British
Journal of Psychiatry, 124, 473– 481.
220 STILES ET AL.
Stiles, W. B., Agnew-Davies, R., Hardy, G. E., Barkham, M., & Shapiro,
D. A. (1998). Relations of the alliance with psychotherapy outcome:
Findings in the Second Sheffield Psychotherapy Project. Journal of
Consulting and Clinical Psychology, 66, 791– 802.
Stiles, W. B., Reynolds, S., Hardy, G. E., Rees, A., Barkham, M., &
Shapiro, D. A. (1994). Evaluation and description of psychotherapy
sessions by clients using the Session Evaluation Questionnaire and
the Session Impacts Scale. Journal of Counseling Psychology, 41, 175–
185.
Suh, C. S., Strupp, H. H, & O’Malley, S. S. (1986). The Vanderbilt process
measures: The Vanderbilt Psychotherapy Process Scale (VPPS) and the
Vanderbilt Negative Indicators Scale (VNIS). In L. S. Greenberg &
W. M. Pinsof (Eds.), The psychotherapeutic process: A research hand-
book (pp. 285–324). New York: Guilford Press.
Tang, T. Z., & DeRubeis, R. J. (1999). Sudden gains and critical sessions
in cognitive– behavioral therapy for depression. Journal of Consulting
and Clinical Psychology, 67, 262–266, 894 –904.
Ten Berge, J. M. F. (1986). Rotation to perfect congruence and the
cross-validation of component weights across populations. Multivariate
Behavioral Research, 21, 41– 64.
Tichenor, V., & Hill, C. E. (1989). A comparison of six measures of
working alliance. Psychotherapy, 26, 195–199.
Watson, J. C., & Greenberg, L. S. (1994). The alliance in experiential
therapy: Enacting the relationship conditions. In A. O. Horvath & L. S.
Greenberg (Eds.), The working alliance: Theory, research and practice
(pp. 153–172). New York: Wiley.
Received August 17, 2000
Revision received February 2, 2002
Accepted February 12, 2002 䡲