Clinical Psychology and Psychotherapy

Clin. Psychol. Psychother. 24, 451–461 (2017)

Published online 6 April 2016 in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/cpp.2014

Therapeutic Alliance in Internet-Delivered

Cognitive Behaviour Therapy for Depression or
Generalized Anxiety
Heather D. Hadjistavropoulos,1* Nicole E. Pugh,1† Hugo Hesser2 and
Gerhard Andersson3
1
Department of Psychology, University of Regina, Regina, SK, Canada
2
Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden
3
Department of Behavioural Sciences and Learning, Linköping University and Department of Clinical Neuroscience,
Karolinska Institute, Linköping, Sweden

There has been limited research on therapeutic alliance in the context of therapist-assisted Internet-
delivered cognitive behaviour therapy (ICBT) when delivered in clinical practice. The present study
investigated therapeutic alliance in ICBT delivered to patients seeking treatment for symptoms of de-
pression (n = 83) or generalized anxiety (n = 112) as part of an open dissemination trial. ICBT was pro-
vided by 27 registered therapists or 28 graduate students working in six geographically dispersed
clinics; therapist-assistance was delivered primarily through secure messages and occasionally tele-
phone calls. The Generalized Anxiety Disorder-7 and Patient Health Questionnaire-9 were collected
pre-, mid- and post-treatment, and the Therapeutic Alliance Questionnaire was assessed mid- and
post-treatment. Therapeutic alliance ratings were high both at mid-treatment and post-treatment (above
80%). There was no relationship between therapeutic alliance ratings and improvement on primary
outcomes. Among patients treated for depression, lower ratings of mid-treatment alliance were
associated with concurrent treatment by a psychiatrist and fewer phone calls and emails from their
therapist. Among patients treated for generalized anxiety, ratings of mid-treatment alliance were higher
among registered providers as compared to graduate students. Multiple directions for future research
on therapeutic alliance in ICBT are offered, including suggestions for developing a new measure of
therapeutic alliance specific to ICBT and measuring therapeutic alliance throughout the treatment
process. Copyright © 2016 John Wiley & Sons, Ltd.

Key Practitioner Message:

• This research demonstrated that therapeutic alliance ratings were very strong at both mid- and post-
treatment among patients who received Internet-delivered cognitive behaviour therapy (ICBT) for
depression or anxiety in clinical practice.
• Among patients receiving ICBT for depression, lower ratings of therapeutic alliance were associated
with patients reporting concurrent treatment by a psychiatrist and with the receipt of fewer phone calls
and emails from the therapist.
• Among patients receiving ICBT for generalized anxiety, ratings of alliance were higher when patients
were treated by registered providers as compared to graduate students.
• Therapeutic alliance ratings did not predict outcome in ICBT for depression or anxiety.
• Practitioners have reason to be confident that a therapeutic relationship can be formed in ICBT when
delivered in clinical practice.

Keywords: Depression, Generalized Anxiety, Internet delivered, Cognitive Behaviour Therapy, Therapeu-
tic Alliance

Depression and anxiety are prevalent, disabling and costly

*Correspondence to: Heather Hadjistavropoulos, Department of Psy- conditions (Kessler, Berglund, Demler, Jin, & Walters, 2005)
chology, University of Regina, 3737 Wascana Parkway, Regina, SK,
that often involve patients either failing to seek out or
Canada, S4S 0A2.
E-mail: hadjista@uregina.ca significantly delaying treatment (Wang et al., 2005).
† Cognitive-behavioural therapy (CBT) is efficacious for
Present address: Nicole E. Pugh, 715 East 12th Avenue, 2nd Floor,
Room 232, Vancouver, BC V5Z1M9, Canada treating depression and anxiety (Hollon, Stewart, & Strunk,

Copyright © 2016 John Wiley & Sons, Ltd.

452
2006) with growing research extending the efficacy to an
Internet-delivered CBT format (ICBT; G. Andersson, 2015).
In ICBT, patients review psychoeducational information that
is commonly provided in face-to-face cognitive behavioural
therapy over the Internet. This information is presented in
the form of weekly modules presented in an engaging man-
ner. Treatment can either be self- or therapist-guided. The ap-
proach overcomes barriers to accessing care such as living in
a rural or remote location, having limits on mobility or time,
or being embarrassed about seeking care (G. Andersson,
2015). A recent systematic review examined the clinical effi-
cacy of ICBT for a range of disorders (Hedman, Ljotsson, &
Lindefors, 2012). The review included 108 randomized con-
trolled trials (RCTs) of ICBT with or without therapist assis-
tance. The results were promising and consistently found
large within group effect sizes for depression and anxiety dis-
orders (mean d ranged from 0.94 to 1.94). ICBT appeared
equally beneficial when compared to face-to-face CBT. Given
the strength of these findings, it is not surprising that there is
increasing interest in offering ICBT in clinical practice and
considerable evidence that the efficacy of ICBT generalizes
to clinical practice (G. Andersson & Hedman, 2013).
When examining the research literature, ICBT that involves
some form of guidance appears superior when compared
with self-guided ICBT (Johansson & Andersson, 2012). For
instance, a review of 14 RCTs investigated the impact of guid-
ance on ICBT treatment outcomes (Baumeister, Reichler,
Munzinger, & Lin, 2014). Results indicated that guided inter-
ventions were significantly superior to unguided interven-
tions with respect to efficacy, completed modules and
program completion rates. Only one study in this review fo-
cused on the dose–response relationship and found no signif-
icant effect when comparing a lower dose (i.e., one email
contact a week) to a higher dose of guidance (i.e., at least
three emails per week). Four trials in the review examined
different levels of Internet therapist qualifications (e.g., grad-
uate students versus clinical psychologists) and found no re-
lationship to treatment outcome. Finally, one trial in the
review compared synchronous and asynchronous communi-
cation and found no significant differences between groups
in symptom improvement, number of modules completed
or program completion rates.
Various aspects of therapeutic alliance have been examined
in the context of therapist-assisted ICBT in diverse clinical
samples, such as post-traumatic stress disorder (e.g., Klein
et al., 2010), panic disorder (Kiropoulos et al., 2008), health
anxiety (Hedman, Andersson, Lekander, & Ljotsson, 2015),
obsessive compulsive disorder (E. Andersson et al., 2015), tin-
nitus (Jasper et al., 2014), depression, generalized anxiety and
social anxiety (G. Andersson et al., 2012). Across trials, a
strong therapeutic relationship has consistently been found
using standardized measures, such as the Working Alliance
Inventory-12 item (Tracey & Kokotovic, 1989) and the Thera-
peutic Alliance Questionnaire (Kiropoulos et al., 2008). Some
researchers have examined therapeutic alliance at different
Copyright © 2016 John Wiley & Sons, Ltd.
H. D. Hadjistavropoulos et al.
time points and identified that therapeutic alliance increases
over the course of treatment (e.g., from pre-to mid-treatment
or mid- to post-treatment; Bergman Nordgren, Carlbring,
Lina, & Andersson, 2013; Knaevelsrud & Maercker, 2007).
Others have compared therapeutic alliance in ICBT to other
therapeutic modalities, and found no significant differences
in ratings of therapeutic alliance at post-treatment in ICBT
as compared to face-to-face CBT (Kiropoulos et al., 2008;
Preschl, Maercker, & Wagner, 2011), telephone CBT (Lindner
et al., 2014) or group CBT (Jasper et al., 2014).
In contrast to the clarity of the above findings, research
regarding how therapeutic alliance in ICBT impacts treat-
ment outcome has been mixed, with some studies
reporting no relationship between alliance ratings and
symptom improvement (G. Andersson et al., 2012; Preschl
et al., 2011), and other studies reporting ICBT outcomes are
related to therapeutic alliance measured at various time
points, such at the beginning of treatment (Hedman
et al., 2015), at mid-treatment (E. Andersson et al., 2015;
Bergman Nordgren et al., 2013), at the end (Knaevelsrud
& Maercker, 2007) or very close to the end of treatment
(Knaevelsrud & Maercker, 2006). In terms of comparison
to the face-to-face therapy literature, a meta-analysis of
studies examining therapeutic alliance in face-to-face ther-
apy has revealed a small, but nevertheless positive robust
relationship between the therapeutic alliance and treat-
ment outcome (r = 0.275; Del Re, Fluckiger, Horvath,
Symonds, & Wampold, 2012; Horvath, Del re, Flückiger,
& Symonds, 2011).
The purpose of the present study was to extend the pre-
vious research by investigating therapeutic alliance re-
ported by patients who completed a therapist-assisted
ICBT program for depression or generalized anxiety as
part of an open trial designed to facilitate the use of ICBT
in multiple settings (Hadjistavropoulos et al., 2014). In
contrast to most past research that either involved small
sample pilot studies or RCTs with stricter criteria and a
small number of therapists, this study was unique. Inclu-
sion criteria to participate were not as strict (e.g., patients
did not need to meet diagnostic criteria to participate).
Additionally, patients were screened centrally, but were
then treated by a large number of registered providers
(n = 27) or supervised graduate students (n = 28) working
in one of six clinical settings.
Using this data set we examined: (1) the strength of alli-
ance ratings measured at mid- and post-treatment, (2)
whether therapeutic alliance measured at mid-treatment
predicted symptom change from pre- to post-treatment,
and (3) predictors of therapeutic alliance, including back-
ground variables (i.e., sex, age, married or not, children
or not, working or not), clinical characteristics assessed
at pre-treatment (i.e., diagnosis of major depressive epi-
sode (MDE), diagnosis of generalized anxiety disorder
(GAD), use of psychotropic medication, concurrent psy-
chiatric treatment, presence of comorbid medical
Clin. Psychol. Psychother. 24, 451–461 (2017)
Therapeutic Alliance in Internet-Delivered Therapy
condition) and treatment variables (i.e., comfort with text,
days in ICBT, number of modules initiated, number of
messages received from therapist, phone calls from thera-
pist, student versus registered provider, psychology ver-
sus other professional background). The latter analysis
represents a novel direction in the ICBT literature and
was designed to improve understanding of variables asso-
ciated with a strong therapeutic alliance; this could pro-
vide valuable information for clinicians offering ICBT in
clinical practice.
Based on past research (e.g., Klein et al., 2010), we pre-
dicted that therapeutic alliance would be rated as strong
(i.e., score above 80% on the measure of therapeutic alli-
ance used in this study). Despite mixed findings in the
literature, we predicted a positive relationship between
mid-treatment therapeutic alliance and symptom im-
provement; this prediction was made because we antici-
pated that with more therapists and a more diverse
patient sample there would be greater variability in the
sample and thus a greater opportunity to find a relation-
ship between therapeutic alliance and outcome. We ex-
plored the relationship between mid-treatment alliance
ratings and symptom improvement so as not to con-
found post-treatment alliance ratings with symptom im-
provement at post-treatment (Knaevelsrud & Maercker,
2007). Examination of predictors of therapeutic alliance
was considered exploratory and no hypotheses were for-
mulated. Overall, the study is consistent with recommen-
dations in past studies to examine therapeutic alliance in
ICBT in clinically representative samples (G. Andersson
et al., 2012).
METHODS
Participants
This study involved analysis of data from a previously
published study of 195 patients who participated in an
open dissemination trial investigating the effectiveness
of therapist-assisted ICBT for depression (n = 83) or an-
xiety (n = 112) when delivered in clinical practice
(ISRCTN48160673). Patients learned of ICBT through
providers, family, friends, advertisements and media
stories (see Hadjistavropoulos et al., 2014 for additional
details). Patients could be self- or provider-referred. In-
terested patients underwent a standardized telephone
screening interview, which included the administration
of the MINI International Neuropsychiatric Interview
(MINI; Sheehan et al., 1998) conducted by trained
screeners who either held a graduate degree in or were
graduate students in Social Work or Clinical Psychology.
Patients were offered treatment if they: (1) were at least
18 years of age, (2) resided in Saskatchewan, Canada,
(3) had access to and were comfortable using computers
Copyright © 2016 John Wiley & Sons, Ltd.
453
and the Internet, (4) provided a physician as an emer-
gency contact, (5) reported either no or stable medication
for a month, (6) reported no current or recent problems
with psychosis, mania, substance-related disorders, or a
suicide plan or intent, and (7) a score above 5 on the Pa-
tient Health Questionnaire-9 (PHQ-9; Kroenke, Spitzer,
& Williams, 2001) or Generalized Anxiety Disorder-7
(GAD-7; Spitzer, Kroenke, Williams, & Lowe, 2006)
reflecting the presence of at least self-reported mild
symptoms of depression (Kroenke & Spitzer, 2002)
and/or anxiety (Spitzer et al., 2006). It was not necessary
to meet diagnostic criteria for a disorder to participate. ICBT
was optional. Patients who were not interested in ICBTcould
seek services through publically funded mental health clinics
or medication/support from publically funded family physi-
cians. Waiting times for mental health clinics can be long and
typically require patients to attend in person during day time
hours. Limited time, distance to clinic, mobility issues and
concerns about stigma can be a barrier to attending the men-
tal health clinics and thus likely influenced patient interest in
ICBT.
In total, 380 individuals underwent screening for this
open trial, but were not all included for the following rea-
sons: no response to subsequent attempts to begin treatment
(n = 62); self-reported problems other than depression or
anxiety (n = 32); reported receiving other psychological ser-
vices (n = 26); described minimal symptoms of depression
or anxiety (n = 19); did not meet basic requirements (e.g.,
18 or older, resident of province, computer access; n = 19)
or received ICBT for panic disorder (n = 26).
Mean age of the patients in the trial was 40.22 years
(SD = 12.57 years); 69.5% were women, 65.8% were mar-
ried or in a common-law relationship, 56.4% reported hav-
ing children and 73.5% reported being employed on a
part- or full-time basis. A large number of patients re-
ported having co-morbid medical conditions (42.3%). Psy-
chotropic medication was used by 61.2% of the sample
and 12.2% described being in regular contact with a psy-
chiatrist. The majority (73.3%) reported previously receiv-
ing mental health treatment for depression or anxiety;
78.3% of patients who received ICBT for depression met
diagnostic criteria for a MDE and 75.9% of patients who
received ICBT for generalized anxiety met diagnostic
criteria for GAD.
Procedure
Treatment
Following the telephone screening, eligible patients
were provided with access to a secure web application.
Once logged in, patients provided electronic informed
consent and completed pre-treatment background and
symptom questionnaires. Treatment consisted of 12 mod-
ules focused on either depression or generalized anxiety
offered at no cost. The programs were available for
Clin. Psychol. Psychother. 24, 451–461 (2017)
454
patients reporting symptoms consistent with MDE or
GAD, as well as sub-clinical symptoms as ICBT was de-
signed to assist with treatment of disorders as well as pre-
vent symptom deterioration. The program content was
licenced from Swinburne University of Technology
National eTherapy Centre in Australia (see Klein, Meyer,
Austin, & Kyrios, 2011 for a review). Modules covered ba-
sic psychoeducation regarding the targeted disorder, cog-
nitive (e.g., identifying and challenging thoughts) and
behavioural strategies (e.g., relaxation, sleep manage-
ment, problem-solving, exposure, behavioural activation,
communication), as well as relapse prevention (see
Table 1 for a list of themes by module for each program).
Each module began with check-in questions that were
submitted to a designated therapist and concluded with
assigned homework activities. Patients worked on each
module for at least one week, but could take longer if ad-
ditional time was needed. The average time from first
login to last login was 19 weeks (SD = 11.53 weeks).
Therapist Assistance
Therapists worked in one of six settings. In terms of
community therapists, clinic managers were contacted,
approved of the study and identified therapists to partici-
pate in the trial. Approximately half of the therapists in
the tiral (n = 27) were registered psychologists, social
workers or nurses with CBT experience who regularly
treated patients with depression and anxiety in commu-
nity mental health clinics. The remaining therapists were
supervised graduate students in clinical psychology or so-
cial work who sought this experience to complement their
graduate training (n = 28). Patients were assigned to work
with the first available community therapist; if a commu-
nity therapist was unavailable, then, with patient consent,
patients were assigned to work with a graduate student.
Table 1. Topics covered by module in the Internet-delivered cognitive behaviour therapy programs for depression and generalized
anxiety
Module Depression program
1 Psychoeducation on depression and the cognitive
behavioural model
2 Behavioural activation and goal setting
3 Behavioural activation continued
4 Relaxation
5 Sleep management
6 Introduction to cognitive restructuring
7 Cognitive restructuring continued
8 Problem solving
9 Communication
10 Mindfulness
11 Time management
12 Relapse prevention
Copyright © 2016 John Wiley & Sons, Ltd.
H. D. Hadjistavropoulos et al.
Therapists on average worked with 3 patients (SD = 2.34;
Median = 3; Mode = 1; Range 1 to 12).
All therapists participated in a one-day workshop that
was both didactic and experiential in nature cover-
ing research as well as professional practice issues
(Hadjistavropoulos et al., 2012). Supervision was provided
either by a registered psychologist or registered social
worker; supervisors were able to review patient progress
on the website as well as all messages sent between therapists
and patients. Supervision was most commonly provided
over secure email. Therapists would send questions or their
planned messages to the supervisor for review. Supervisors
would respond to requests for supervision on a daily basis.
Therapists could also call or request a face-to-face supervi-
sion. The model of supervision was developmental in nature,
with greater supervision provided to students and registered
providers with less experience and as students and therapists
gained experience supervision was reduced. More specifi-
cally, initially all emails were reviewed prior to being sent
to patients. Once the supervisor had confidence in the thera-
pists’ skills, then emails would only be reviewed as requested
by therapists. Therapists were instructed to seek supervision
when patients were: (1) not improving or deteriorating, (2)
not responding to attempts to contact, (3) describing dissatis-
faction with ICBT or the therapeutic relationship, (4) present-
ing as at risk of suicide or in crisis, or (5) presenting with any
issue that led the therapist to feel uncertain about how to re-
spond. In addition to this supervision, supervisors audited
therapist’s messages to patients. Supervisors contacted thera-
pists to provide feedback on these audits as needed.
Patients were informed of the name and qualifications
of the designated therapist. The therapist’s photo and
brief work biography was available on the ICBT website.
Each week therapists reviewed patient progress. This
would include reviewing homework sheets completed
by patients and any additional messages sent from pa-
tients to therapists. After reviewing homework sheets
Generalized anxiety
Psychoeducation on generalized anxiety and the cognitive
behavioural model
Relaxation
Relaxation continued
Cognitive restructuring
Cognitive restructuring continued
Beliefs about worry and uncertainty
Problem solving and assertiveness
Overcoming avoidance
Changing worry behaviours
Sleep
Mindfulness
Relapse prevention
Clin. Psychol. Psychother. 24, 451–461 (2017)
Therapeutic Alliance in Internet-Delivered Therapy
and messages, therapists then messaged assigned patients
on a set day. Messages were tailored to the individual pa-
tient. Therapists were instructed to: (1) offer support and
encouragement, (2) answer any questions about the mate-
rial, (3) provide feedback on homework, (4) direct patients
to past or upcoming modules that applied to the patient
concerns, and (5) assist patients with any difficulties they
faced with motivation or engagement with the program.
Therapists had the option to phone or email patients more
than once a week if they were concerned that patients
were not participating or required additional support. In
total, there were 12 homework review sheets (one per
module). Patients, on average, completed 8.51 modules
(SD = 4.15) and therefore on average sent eight homework
sheets to therapists. Patients, on average, sent therapists
10.19 (SD = 11.14; Mdn = 7; Range 0–76) secure email mes-
sages in addition to the homework sheets. On average, pa-
tients participated in the program for 19 weeks
(SD = 11.23; Mdn = 18.14; Range 0–62). Therapists, on aver-
age, sent 19.64 (SD = 11.53; Mdn = 19; Range 0–62) mes-
sages to patients and phoned patients an average 1.96
(SD = 2.56 Mdn = 1.0; Range 0–16) times. Research ethics
board approval was obtained from the Universities of
Regina and Saskatchewan, as well as the Regina
Qu’Appelle, Saskatoon, Cypress, Five Hills and Sun
Country health regions.
Measures
Background information collected prior to treatment in-
cluded age, sex, married/common law or not, had chil-
dren or not, completed university degree or not, part- or
full-time employment or not, diagnosis of GAD or not, di-
agnosis of MDE or not, use of psychotropic medication or
not and regular psychiatric treatment or not (i.e., patient
reported regular contact with a psychiatrist for symptom
monitoring, medication management or support).
The GAD-7 (Spitzer et al., 2006) and the PHQ-9
(Kroenke et al., 2001) served as the primary outcome mea-
sures and were administered pre-treatment prior to mod-
ule 1, mid-treatment after completion of module 6 and
post-treatment prior to completion of module 12. The
GAD-7 (Spitzer et al., 2006) asked patients how often they
had experienced a list of seven problems in the past
2 weeks and one question inquired about how difficult
the problems had made their daily lives. Items were rated
on a scale ranging from 0 (not at all) to 3 (nearly every day)
with a maximum score of 21. The GAD-7 has demon-
strated strong psychometric properties (Bandelow &
Brasser, 2009) and scores of 10 or greater have been used
to identify individuals who are likely to meet diagnostic
criteria for GAD (Spitzer et al., 2006).
The PHQ-9 (Kroenke et al., 2001) assessed symptoms
of depression. Nine items were rated on a scale from 0
(not at all) to 3 (nearly every day) with a maximum score
Copyright © 2016 John Wiley & Sons, Ltd.
455
of 27 and a cut-off score of 10 or greater used to identify
individuals with a likely diagnosis of major depression
(Manea, Gilbody, & McMillan, 2012). The PHQ-9 has
strong psychometric properties (Cameron, Crawford,
Lawton, & Reid, 2008; Kroenke et al., 2001).
The Therapeutic Alliance Questionnaire (TAQ;
Kiropoulos et al., 2008; Klein et al., 2010) assessed the over-
all perceived helpfulness of the relationship with the
Internet-therapist and was administered after module 6
and prior to completing module 12. The TAQ consists of
17 items adapted from the 19-item Helping Alliance
Questionnaire-II (HAQ-II), which is a psychometrically
sound measure of therapeutic alliance (e.g., internal con-
sistency, test–retest, convergent validity) used in face-to-
face therapy (Luborsky et al., 1996). The TAQ consists of
16 of the original items with minor wording modifications
to reflect ICBT as compared to face-to-face therapy (e.g., ‘I
feel that I am working together with the therapist in a joint
effort’ was changed to ‘My therapist and I worked well to-
gether throughout the treatment program’). Three of the
original HAQ-II were deleted (i.e., ‘at times I distrust the
therapists’ judgement’, ‘I want very much to work out
my problems’, and ‘from time to time, we both talk about
the same important events in my past’) and one item was
added (i.e., ‘I felt comfortable with my therapist’s ability
to guide me through the program’) to form the TAQ. Sim-
ilar to the HAQ-II, patients rated a series of statements on
a scale ranging from 1 (strongly disagree) to 6 (strongly
agree). The measure has previously been used to assess
the therapeutic alliance in ICBT for PTSD (Klein et al.,
2009; Klein et al., 2010) and panic disorder (Kiropoulos
et al., 2008). This particular measure was chosen as we
were interested in overall rating of therapeutic alliance.
Consistent with past studies, items were summed to pro-
duce a total score ranging from 17 to 102; the summed
score was then divided by the maximum total score
(102), and therapeutic alliance ratings above 80% were
considered to reflect high therapeutic alliance (Klein
et al., 2009; Klein et al., 2010). Patients were informed that
the TAQ ratings would not be shared with their therapist,
but instead would be de-identified and available to the re-
search team. The internal consistency of the measure in
this study was very high (Cronbach’s alpha = 0.96).
The collected treatment variables included the number
of days patients participated in ICBT from first login to
last login to the website, number of modules started, num-
ber of messages received from the therapist, number of
phone calls with therapist, comfort with writing text
(rated on a 1 ‘not very comfortable’ to 5 ‘very comfortable’
scale), training of therapist (student versus registered pro-
vider) and educational background of therapist (psychol-
ogy versus social work/nursing background). To assess
treatment satisfaction, at treatment completion, patients
responded to two questions asking, ‘How much did you
like the treatment program?’ and ‘How much did you
Clin. Psychol. Psychother. 24, 451–461 (2017)
456
enjoy communicating with your therapist?’ These two
questions were rated on a 0 (not at all) to 7 (very much so)
scale.
Statistical Analyses
Descriptive statistics were used to examine TAQ alli-
ance ratings with TAQ ratings above 80% considered
to reflect high therapeutic alliance similar to past
studies (Klein et al., 2009; Klein et al., 2010). A depen-
dent t-test examined if patient TAQ ratings mid-
treatment differed from patient TAQ ratings provided
post-treatment.
Regression models were estimated using Mplus
software version 5.2 (Muthén & Muthén, 2008) within
the structural equations modelling (SEM) framework
and fitted with full maximum likelihood estimation
(with non-normality robust standard errors). Maximum
likelihood estimation allows observed variables in-
cluded in the data model to be related with missing data
(Enders, 2010). Regression models estimated within
SEM retained patients with at least one valid observa-
tion and sample size varied as a function of how many
patients provided usable data in the model. Statistical
significance was determined by the Wald ratio (esti-
mate/standard error) with an alpha level of 0.05 (i.e.,
z-value above 1.96). Confidence intervals are provided
with a 95% margin. We ran separate analyses for each
patient-group (i.e., generalized anxiety and depression)
using the disorder-specific measure as the outcome var-
iable (i.e., GAD-7, PHQ-9).
To examine whether the mid-treatment TAQ pre-
dicted post-treatment symptoms, we employed linear
regression analysis (within SEM framework); this model
examined the association between TAQ ratings at mid-
treatment and the primary symptom measure at post-
treatment while controlling for pre-treatment symptom
measure. Mid-treatment alliance was examined rather
than post-treatment alliance as past researchers have
commented that post-treatment alliance scores are
likely to be highly associated with outcome as they are
measured at the same point in time (Knaevelsrud &
Maercker, 2007).
To explore predictors of TAQ ratings at mid-
treatment, a series of linear multiple regression analyses
were computed. In three separate regression models,
we examined whether demographic variables (i.e., age,
sex, married or not, children or not, university education,
part- or full-time employment), clinical characteristics at
baseline (i.e., diagnosis of GAD, diagnosis of MDE, med-
ication use, concurrent psychiatric treatment, other med-
ical condition) and treatment variables (i.e., days in
treatment, number of modules started, number of mes-
sages received from therapist, number of phone conver-
sations, comfort with text, student versus registered
Copyright © 2016 John Wiley & Sons, Ltd.
H. D. Hadjistavropoulos et al.
provider, psychology versus social work/nursing back-
ground) predicted TAQ ratings at mid-treatment. In ad-
dition, mid-treatment TAQ ratings were examined to
allow for a greater sample size for the analysis as fewer
patients provided post-treatment TAQ ratings.
RESULTS
As noted in the previously published study examining
patient outcomes (Hadjistavropoulos et al., 2014), of the
83 patients assigned to the ICBT program for depression,
80 completed pre-treatment measures, 57 mid-treatment
measures and 41 post-treatment measures. Of the 112
patients assigned to ICBT for generalized anxiety, 107
completed pre-treatment measures, 78 completed mid-
treatment measures and 58 completed post-treatment
measures. Analysis of outcomes using latent growth
curve modelling identified significant reductions in
PHQ-9 and GAD-7 scores that were large (d = 0.91–
1.25); sensitivity analyses suggested that missing data
did not influence the findings and overall conclusions
(Hadjistavropoulos et al., 2014).
Alliance Ratings Mid- to Post-treatment
For patients who received ICBT for generalized anxiety,
mid-treatment (M = 85.82, SD = 10.85; 84.14%) and post-
treatment TAQ ratings (M = 86.93, SD = 12.42; 85.23%)
were considered high (> than 80%) and a dependent t-test
did not reveal a statically significant mean difference be-
tween measurement points, t(57) = 1.26, p = 0.21. Similarly,
for patients who received ICBT for depression, mid-
treatment (M = 83.47, SD = 13.89; 81.83%) and post-
treatment TAQ ratings (M = 83.20, SD = 15.35; 81.56%)
were regarded as high (> than 80%), and a dependent t-
test did not reveal a statically significant mean difference
between measurement points, t(40) = 0.51, p = 0.61.
A regression analysis (within SEM) with TAQ assessed
at mid-treatment as the criterion and type of ICBT pro-
gram as dummy coded predictor revealed no significant
differences in TAQ ratings between patients who received
ICBT for generalized anxiety and those who received
ICBT for depression, (n = 196), b = 2.36, 95% [ 6.72,
2.04], z = 1.06, p = 0.29.
Relationship Between Therapeutic Alliance and
Outcome
In the regression model examining TAQ ratings made by
patients in the ICBT for generalized anxiety program at
mid-treatment as a predictor of the GAD-7 scores at
post-treatment while controlling for pre-treatment GAD-
7 scores, mid-treatment TAQ scores were not significantly
Clin. Psychol. Psychother. 24, 451–461 (2017)
Therapeutic Alliance in Internet-Delivered Therapy
correlated with GAD-7 post-treatment scores (n = 107),
b = 0.06, 95% [ 0.13, 0.14], z = 1.56, p = 0.12, r = 0.16.
Similarly, no significant relationship was observed for pa-
tients who received ICBT for depression, when examining
if the mid-treatment TAQ scores predicted post-treatment
PHQ-9 scores while controlling for pre-treatment PHQ-9
scores (n = 80), b = 0.06, 95% [ 0.18, 0.06], z = 1.04,
p = 0.30, r = 0.10.
Predictors of Therapeutic Alliance
For patients in the ICBT for generalized anxiety program,
regression analyses revealed no statistically significant as-
sociations between TAQ ratings at mid-treatment and de-
mographic variables, clinical characteristics, or treatment
variables (all p’s > 0.07); the only exception was that the
level of training was associated with mid-treatment TAQ
ratings, b = 6.61, 95% [1.30, 11.93], z = 2.44, p = 0.015. This
relationship indicated that registered providers
(M = 88.20; SD = 11.16) had, on average, higher scores on
the TAQ than student providers (M = 84.95; SD = 10.39),
corresponding to a standardized mean difference effect
size of d = 0.30.
For patients provided with ICBT for depression, these
same analyses revealed that TAQ ratings at mid-treatment
were negatively associated with concurrent psychiatric
treatment at intake, b = 12.66, 95% [ 24.03, 1.30],
z = 2.19, p = 0.029. This result indicates that patients were
more likely to report lower mid-treatment TAQ ratings
when they were receiving care from a psychiatrist at the
beginning of ICBT for depression. In the ICBT for depres-
sion sample, TAQ ratings were also positively associated
with number of messages received, b = 0.52, 95% [0.15,
0.90], z = 2.79, p = 0.005, and number of phone conversa-
tions with therapists, b = 1.64, 95% [0.42, 2.85], z = 2.64,
p = 0.008, indicating that a greater number of messages
and phone conversations were associated with higher alli-
ance ratings at mid-treatment. Other examined variables
were not significantly associated with alliance ratings at
mid-treatment in the depression sample (all p’s > 0.09).
Therapeutic Alliance and Treatment Satisfaction
As a secondary analysis, we examined the relationship
between ratings of therapeutic alliance and treatment
satisfaction using Pearson correlations. For patients re-
ceiving ICBT for generalized anxiety, mid-treatment
and post-treatment TAQ scores, respectively, were
moderately and positively correlated with overall satis-
faction with the ICBT program (r = 0.30, p = 0.03 and
r = 0.36, p = 0.001) and satisfaction with therapist commu-
nication (r = 0.48, p = 0.001 and r = 0.76, p < 0.001). For pa-
tients receiving ICBT for depression, mid-treatment
approached and post-treatment alliance scores were
Copyright © 2016 John Wiley & Sons, Ltd.
457
moderately and positively correlated with overall satis-
faction with the ICBT program (r = 0.27, p = 0.10 and
r = 0.40, p = 0.01). Mid-treatment and post-treatment alli-
ance scores also positively correlated with satisfaction
with therapist communication (r = 0.34, p = 0.04 and
r = 0.35, p = 0.03).
DISCUSSION
There is a growing body of research examining therapeu-
tic alliance in the delivery of ICBT and how it relates to
outcome. Much of this research, however, has involved
small samples or has been conducted as part of RCTs
and there has been a call for additional research with clin-
ically representative samples (G. Andersson et al., 2012).
The present research contributed to the literature by ex-
amining the therapeutic alliance in therapist-assisted ICBT
when offered as part of an open dissemination trial
whereby ICBT was implemented in multiple settings by
a large number of registered providers and students
working with a diverse clinical sample. We were inter-
ested in exploring therapeutic alliance measured both at
mid-treatment (after module 6) and post-treatment (mea-
sured before module 12) in a clinical setting where there
was potential for greater variance in ratings of therapeutic
alliance.
Of particular note was that despite the large number
of therapists involved in the study trial, including
both registered providers and graduate students who
were new to ICBT, therapeutic alliance ratings were
high among patients treated with ICBT for depression
or generalized anxiety (i.e., >80%). In this study, a
global measure of therapeutic alliance was used and
ratings of the alliance were similar to ratings (e.g., >
80%) that have been observed in past studies of ICBT
using this measure (Kiropoulos et al., 2008; Klein et al.,
2009; Klein et al., 2010). In general, the findings appear
consistent with past research on face-to-face CBT and
ICBT showing that a strong therapeutic alliance is
formed when patients are treated with ICBT for
depression and anxiety (e.g., Kiropoulos et al., 2008;
Preschl et al., 2011)
In this study, therapeutic alliance ratings were similar at
mid- and post-treatment. This is in contrast to some past
researchers who have observed a slight improvement in
therapeutic alliance ratings over the course of ICBT (e.g.,
from pre-to mid-treatment or mid- to post-treatment;
Bergman Nordgren et al., 2013; Knaevelsrud & Maercker,
2007). In the present study, it should be noted that post-
treatment alliance ratings were only available for patients
who completed all 12 modules. Findings could differ from
past studies for this reason. It is possible that the variabil-
ity in findings also reflects differences in settings, as well
as differences in the measures selected as the former
Clin. Psychol. Psychother. 24, 451–461 (2017)
458
studies utilized the Working Alliance Inventory Short
form (WAI-S; Tracey & Kokotovic, 1989) as compared to
the TAQ. The WAI-S assesses empathetic bond, but also
assesses agreement between the patient and therapist
concerning therapeutic tasks and goals. It is possible that
the WAI-S changes over the course of treatment because
it is measuring additional components of therapeutic
alliance.
Consistent with a number of other studies examining al-
liance and outcome in ICBT (G. Andersson et al., 2012;
Preschl et al., 2011), in the present study, therapeutic alli-
ance was not related to symptom improvement. It should
be noted, however, that this research contrasts with other
ICBT research, where researchers have reported small re-
lationships between symptom changes and therapeutic al-
liance measured at various time points, such at the
beginning of treatment (Hedman et al., 2015), at mid-
treatment (E. Andersson et al., 2015; Bergman Nordgren
et al., 2013), and at the end (Knaevelsrud & Maercker,
2007) or very close to the end of treatment (Knaevelsrud
& Maercker, 2006). It also contrasts with the face-to-face
therapy literature where a research synthesis of over 200
papers found a small, but, nevertheless, positive relation-
ship between therapeutic alliance and treatment outcome
(r = 0.275;Horvath et al., 2011).
Unfortunately, it is not readily apparent why findings
on therapeutic alliance and treatment outcome diverge
across studies. Differences in findings are not likely a func-
tion of our use of the TAQ as a measure of therapeutic al-
liance as findings are also mixed among studies that used
the WAI-S. For example, using the WAI-S, some re-
searchers have not found therapeutic alliance to be related
to outcome (G. Andersson et al., 2012; Preschl et al., 2011)
while others have (Bergman Nordgren et al., 2013). Differ-
ences also do not seem to reflect measurement periods.
Similar to our study, others have studied mid-treatment
alliance; sometimes mid-treatment alliance relates to out-
comes (Bergman Nordgren et al., 2013) while other times
it does not (G. Andersson et al., 2012).
One possibility is that the ICBT studies may have dif-
fered in the amount of variability in the therapeutic alli-
ance. All ICBT studies are described as involving
therapist-assistance, but generally little detail on the na-
ture of therapeutic assistance is provided. There could be
potential differences in studies regarding the amount
and nature of contact between therapists and patients.
This could lead to more or less variance in the therapeutic
relationship. With more variance, it is possible that a
greater relationship could emerge between therapeutic
alliance and outcomes in ICBT. In the current study,
although there were a large number of therapists with di-
verse educational backgrounds, there was significant
training and supervision that was provided. Perhaps this
resulted in less variation than other studies and thus less
opportunity for a relationship to outcome. In the current
Copyright © 2016 John Wiley & Sons, Ltd.
H. D. Hadjistavropoulos et al.
study, the mid-treatment therapist alliance rating was
M = 85.82, SD = 10.78 for patients receiving ICBT for gen-
eralized anxiety and M = 83.47, SD = 13.75 for patients re-
ceiving ICBT for depression. Review of the standard
deviations suggests that scores were very elevated and
thus potentially restricted. This potential explanation
seems consistent with some past research showing limited
variation in therapist outcomes among therapists using
ICBT to treat anxiety disorders (Almlov et al., 2011) or
depression (Almlov, Carlbring, Berger, Cuijpers, &
Andersson, 2009).
Even though therapeutic alliance scores were not related
to outcome, we were still interested in examining whether
certain variables predicted higher ratings of therapeutic
alliance in order to gain a better understanding of the ther-
apeutic alliance in ICBT. For patients who were treated for
depression, we identified that mid-treatment therapeutic
alliance ratings were higher if patients received more mes-
sages and more phone calls from their therapist. This find-
ing may suggest that it may be helpful for therapists to
message and call patients with depression in an effort to
bolster the therapeutic alliance. It should be acknowl-
edged, however, that the findings are correlational in na-
ture. Another plausible explanation for the relationship
between therapist contact and therapeutic alliance is that
because the therapeutic alliance was high, the therapist
may have been more inclined to put in extra effort and call
and message the client more often. It is also possible that
because therapeutic alliance was high the patient
requested additional phone calls and messages, and this
relationship is more patient rather than therapist driven.
Ultimately, the findings suggest that therapists should be
mindful of the relationship between alliance and phone
calls and messages and that this may be an area worthy
of additional research (e.g., by systematically examining
the impact of varying number of phone calls and emails
on the therapeutic alliance). It would be particularly inter-
esting to examine if increased therapist contact is impor-
tant for therapeutic alliance for all patients, or only for
the patients who are struggling with or failing to benefit
from ICBT. It also needs to be acknowledged that even
though increased phone calls and messages may be asso-
ciated with greater ratings of therapeutic alliance, this
does not appear to be related to improved outcomes.
For patients treated for depression, therapeutic alliance
ratings were found to be lower if patients had concurrent
psychiatric treatment. This relationship is interesting and
perhaps reflects that greater severity of illness is associ-
ated with more difficulties forming a therapeutic alliance,
which has also been found in the face-to-face therapy liter-
ature (e.g., Constantino, Arnow, Blasey, & Agras, 2005).
Such factors provide insight for therapists delivering
ICBT. It is notable, however, that these variables were
not predictive of the therapeutic alliance among patients
treated with ICBT for generalized anxiety. It is important,
Clin. Psychol. Psychother. 24, 451–461 (2017)
Therapeutic Alliance in Internet-Delivered Therapy
therefore, for therapists to also be mindful that different
factors could be relevant for understanding the therapeu-
tic alliance among patients with different mental health
conditions.
For patients treated with ICBT for generalized anxiety,
patients rated the therapeutic relationship more positively
when treated by registered providers as compared to stu-
dents. The difference was statistically significant, but
small in magnitude. Specifically, there was only a very
small difference in therapeutic alliance ratings between
registered providers (M = 88.20; SD = 11.16) and student
providers (M = 84.95; SD = 10.39), corresponding to a stan-
dardized mean difference effect size of d = 0.30. This small
difference ultimately may not be meaningful, but may be
worthy of future research attention to understand what
factors account for lower ratings of alliance in students
(e.g., lower expectations among patients, lack of skill
among students). It is interesting to note that in a system-
atic review of the ICBT literature, it was concluded that
the level of qualification (e.g., student versus registered
provider) was unrelated to treatment efficacy (Baumeister
et al., 2014). The present study adds to this review and
suggests that while level of training is not related to treat-
ment efficacy, level of qualification appears to be related at
least in a small way to ratings of the therapeutic alliance.
It is generally quite striking how few variables were re-
lated to therapeutic alliance among patients with GAD,
but also patients with depression. It is quite possible that
it is difficult to find predictors of therapeutic alliance be-
cause the ratings were very high and perhaps there were
very few cases of low alliance making prediction difficult
from a statistical perspective. It is also possible, however,
that we failed to measure important variables that predict
therapeutic alliance. In the face-to-face literature, a num-
ber of studies have found lower therapeutic alliance rat-
ings to be related to low pre-treatment expectations
(Constantino et al., 2005; Meyer et al., 2002) as well as
interpersonal problems (Muran, Segal, Samstag, &
Crawford, 1994; Saunders, 2001); these variables were
not measured in the present study, but may represent im-
portant variables that predict therapeutic alliance in ICBT
as well.
Of interest, as a secondary analysis, we examined the
relationship between therapeutic alliance and treatment
satisfaction. We identified that both mid-treatment and
post-treatment ratings of therapeutic alliance were related
to overall treatment satisfaction and satisfaction commu-
nicating with the therapists. As this research is correla-
tional in nature, it is not possible to know the direction
of the relationship, but, nevertheless, suggests that thera-
pists should be aware that treatment satisfaction and ther-
apeutic alliance are inter-related.
In terms of future research directions, it is helpful to
consider the study limitations. In this study we used
the TAQ, which has been used previously in several
Copyright © 2016 John Wiley & Sons, Ltd.
459
ICBT studies (e.g., Kiropoulos et al., 2008; Klein et al.,
2010). Adding confidence to the measure, we identified
that the TAQ had very high internal consistency and
demonstrated variability in terms of relationship with
therapist training, as well as association with therapeutic
phone calls and emails. Nevertheless, it is possible that
the TAQ was limited in terms of its ability to measure
therapeutic alliance in ICBT and a more specific measure
of therapeutic alliance tailored for ICBT is needed.
Andersson et al. (2012) previously highlighted that the
therapeutic alliance is different in ICBT in that there is
typically considerably less contact and alliance is not
only related to interactions with the therapist, but also
the treatment content. Similarly, Bergman Nordgren
et al. (2013) highlighted that a two-factor model of thera-
peutic alliance may be more appropriate for ICBT, with
one factor focused on the relationship and one factor fo-
cused on agreement/confidence in the program. Devel-
opment of an ICBT specific measure of therapeutic
alliance would likely be a valuable direction for future
research. This would allow researchers to explore if alli-
ance with the program as compared to alliance with the
therapist predicts treatment outcome.
In terms of other measurement limitations, in this study,
we did not capture therapeutic ruptures, such as when a
patient expressed dissatisfaction with the therapist. We
also did not measure potentially negative therapist behav-
iours, such as failing to email on a planned date, making
unsupportive comments or failing to show responsiveness
to patient concerns (e.g., not responding to patient ques-
tions or expressing appropriate empathy). It is possible
that therapeutic ruptures or negative behaviours are asso-
ciated with poor outcome. Furthermore, we did not cap-
ture what has been referred to as ‘rupture–repair’
episodes, in which there is some form of rupture in the
therapeutic relationship that is subsequently repaired.
There is some evidence in face-to-face therapy that
rupture–repair in the therapeutic relationship is associated
with improved outcomes (e.g., Stiles et al., 2004). These all
represent interesting directions for research on the thera-
peutic relationship in ICBT. An additional measurement
limitation is that while patients were informed that the
TAQ would not be shared with their therapist and that it
was only collected for research purposes, it is possible that
patients may have been reluctant to honestly express their
views and thus the therapeutic ratings could have been
impacted by this bias.
With respect to measurement timing, we measured ther-
apeutic alliance at two time points (mid- and post- treat-
ment). Measurement after each module could be helpful
in understanding the temporal relationship between ther-
apeutic alliance and outcome. In future studies, it would
be valuable to explore ratings of symptom and therapeutic
alliance on a weekly basis over the course of treatment.
This relationship would allow for examination of how
Clin. Psychol. Psychother. 24, 451–461 (2017)
460
early symptom changes and program satisfaction impact
the therapeutic alliance. In this study, we examined pre-
dictors of mid-treatment alliance rather than post-
treatment alliance in order to allow for a larger sample
size. Ideally, in the future, with a larger sample size, pre-
dictors of post-treatment alliance could also be examined
to determine if there are differences in predictors of alli-
ance at mid- and post-treatment.
In terms of additional future directions, in order to gain
insight into the therapeutic relationship in ICBT, one pos-
sibility is to explore the extent to which the therapeutic al-
liance is related to the content in the therapeutic emails. In
a previous study, the content in therapists’ emails was
classified into the following themes: task reinforcement,
self-efficacy shaping, task prompting, alliance bolstering,
psychoeducation, empathetic utterances and self disclo-
sure (Paxling et al., 2013). In the future, it would be valu-
able to examine how these themes in content relate to
therapeutic alliance. Furthermore, in this study, we cap-
tured total emails and total phone calls sent to patients. It
would be valuable to consider length of emails and phone
calls and when contact goes beyond one contact per week.
In conclusion, this study provides additional evidence
that the therapeutic relationship is strong at mid- and
post-treatment in therapist-assisted ICBT for depression
and generalized anxiety when delivered in clinical set-
tings with a large number of therapists new to the deliv-
ery of ICBT. In the case of individuals treated with ICBT
for depression, mid-treatment alliance appears to be
rated lower by those who reported concurrent psychiat-
ric care and those who received fewer phone calls and
messages. For individuals treated with ICBT for gener-
alized anxiety, mid-treatment therapeutic alliance was
rated as stronger when treated by a registered provider
as compared to a student. Ratings of therapeutic alliance
by patients were not related to symptom improvement
in ICBT. Future research using an ICBT specific measure
of therapeutic alliance assessed over the course of ICBT
would provide additional insight into the relationship
between the therapeutic alliance and outcomes in ICBT.
ACKNOWLEDGEMENT
This work was supported by funding provided by the
Canadian Institutes of Health Research (reference number
101526) and Saskatchewan Health Research Foundation.
Funders had no involvement in the study design, collec-
tion, analysis or interpretation of the data. The GAD
Online content was licenced from Swinburne University
of Technology through Drs. Britt Klein and David Austin.
The authors would like to acknowledge Marcie N. Nugent
and Max Ivanov as well as the many therapists and
patients of the Online Therapy Unit for Service, Education
and Research, University of Regina.
Copyright © 2016 John Wiley & Sons, Ltd.
H. D. Hadjistavropoulos et al.
REFERENCES
Almlov, J., Carlbring, P., Berger, T., Cuijpers, P., & Andersson, G.
(2009). Therapist factors in Internet-delivered cognitive behav-
ioural therapy for major depressive disorder. Cognitive Behav-
iour Therapy, 38(4), 247–254. DOI:10.1080/16506070903116935.
Almlov, J., Carlbring, P., Kallqvist, K., Paxling, B., Cuijpers, P., &
Andersson, G. (2011). Therapist effects in guided internet-
delivered CBT for anxiety disorders. Behavioural Cognitive
Psychotherarpy, 39(3), 311–322. DOI:10.1017/S135246581000069X.
Andersson, E., Ljotsson, B., Hedman, E., Enander, J., Kaldo, V.,
Andersson, G., … Rück, C. (2015). Predictors and moderators
of Internet-based cognitive behavior therapy for obsessive–
compulsive disorder: Results from a randomized trial. Journal
of Obsessive-Compulsive and Related Disorders, 4, 1–7.
DOI:10.1016/j.jocrd.2014.10.003.
Andersson, G. (2015). The internet and CBT: A clinical guide. Boca
Raton: CRC Press, Taylor & Francis Group.
Andersson, G., & Hedman, E. (2013). Effectiveness of guided
internet-based cognitive behavior therapy in regular clinical set-
tings. Verhaltenstherapie, 23(3), 140–148. DOI:10.1159/000354779.
Andersson, G., Paxling, B., Wiwe, M., Vernmark, K., Felix, C. B.,
Lundborg, L., … Carlbring, P. (2012). Therapeutic alliance in guided
internet-delivered cognitive behavioural treatment of depression,
generalized anxiety disorder and social anxiety disorder. Behaviour
Research and Therapy, 50(9), 544–550. DOI:10.1016/j.brat.2012.05.003.
Bandelow, B., & Brasser, M. (2009). Clinical suitability of GAD-7
scale compared to hospital anxiety and depression scale-A for
monitoring treatment effects in generalized anxiety disorder.
European Neuropsychopharmacology, 19, S604–S605.
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014). The im-
pact of guidance on Internet-based mental health interventions
—A systematic review. Internet Interventions, 1(4), 205–215.
DOI:10.1016/j.invent.2014.08.003.
Bergman Nordgren, L., Carlbring, P., Lina, E., & Andersson, G.
(2013). Role of the working alliance on treatment outcome in
tailored internet-based cognitive behavioural therapy for anxi-
ety disorders: Randomized controlled pilot trial. JMIR Research
Protocols, 2, e4. DOI:10.2196/resprot.2292.
Cameron, I. M., Crawford, J. R., Lawton, K., & Reid, I. C. (2008).
Psychometric comparison of PHQ-9 and HADS for measuring
depression severity in primary care. British Journal of General
Practice, 58(546), 32–36. DOI:10.3399/bjgp08X263794.
Constantino, M. J., Arnow, B. A., Blasey, C., & Agras, W. S. (2005).
The association between patient characteristics and the thera-
peutic alliance in cognitive-behavioral and interpersonal ther-
apy for bulimia nervosa. Journal of Consulting and Clinical
Psychology, 73(2), 203–211. DOI:10.1037/0022-006X.73.2.203.
Del Re, A. C., Fluckiger, C., Horvath, A. O., Symonds, D., &
Wampold, B. E. (2012). Therapist effects in the therapeutic
alliance-outcome relationship: A restricted-maximum likeli-
hood meta-analysis. Clinical Psychology Review, 32(7), 642–649.
DOI:10.1016/j.cpr.2012.07.002.
Enders, C. K. (2010). Applied missing data analysis. New York:
Guilford Press.
Hadjistavropoulos, H. D., Thompson, M., Klein, B., & Austin, D.
(2012). Dissemination of therapist-assisted Internet Cognitive Be-
haviour Therapy: Development and open pilot study of a work-
shop. Cognitive Behaviour Therapy, 41, 230–240. DOI:10.1080/
16506073.2011.645550.
Hadjistavropoulos, H. D., Pugh, N. E., Nugent, M. M., Hesser, H.,
Andersson, G., Ivanov, M., … Austin, D. W. (2014). Therapist-
assisted Internet-delivered cognitive behavior therapy for depression
Clin. Psychol. Psychother. 24, 451–461 (2017)
Therapeutic Alliance in Internet-Delivered Therapy
and anxiety: Translating evidence into clinical practice. Journal of
Anxiety Disorders, 28, 884–893. DOI:10.1016/j.janxdis.2014.09.018.
Hedman, E., Andersson, E., Lekander, M., & Ljotsson, B. (2015). Pre-
dictors in Internet-delivered cognitive behavior therapy and be-
havioral stress management for severe health anxiety. Behaviour
Research and Therapy, 64, 49–55. DOI:10.1016/j.brat.2014.11.009.
Hedman, E., Ljotsson, B., & Lindefors, N. (2012). Cognitive behavior
therapy via the Internet: A systematic review of applications, clinical
efficacy and cost-effectiveness. Expert Review of Pharmacoeconomics
and Outcomes Research, 12(6), 745–764. DOI:10.1586/erp.12.67.
Hollon, S. D., Stewart, M. O., & Strunk, D. (2006). Enduring effects
for cognitive behavior therapy in the treatment of depression
and anxiety. Annual Review of Psychology, 57, 285–315.
DOI:10.1146/annurev.psych.57.102904.190044.
Horvath, A. O., Del Re, A. C., Flückiger, C., & Symonds, D.
(2011). Alliance in individual psychotherapy. Psychotherapy,
48, 9–16. DOI:10.1037/a0022186.
Jasper, K., Weise, C., Conrad, I., Andersson, G., Hiller, W., &
Kleinstauber, M. (2014). Internet-based guided self-help versus
group cognitive behavioral therapy for chronic tinnitus: A ran-
domized controlled trial. Psychotherapy and Psychosomatics,
83(4), 234–246. DOI:10.1159/000360705.
Johansson, R., & Andersson, G. (2012). Internet-based psychological
treatments for depression. Expert Review if Neurotherapeutics, 12(7),
861–869 quiz 870. DOI:10.1586/ern.12.63.
Kessler, R. C., Berglund, P., Demler, O., Jin, R., & Walters, E. E. (2005).
Lifetime prevalence and age-of-onset distributions’ of DSM-IV dis-
orders in the national comorbidity survey replication. Archives of
General Psychiatry, 62(6), 593–602. DOI:10.1001/archpsyc.62.6.593.
Kiropoulos, L. A., Klein, B., Austin, D. W., Gilson, K., Pier, C., Mitchell,
J., & Ciechomski, L. (2008). Is internet-based CBT for panic disorder
and agoraphobia as effective as face-to-face CBT? Journal of Anxiety
Disorders, 22(8), 1273–1284. DOI:10.1016/j.janxdis.2008.01.008.
Klein, B., Meyer, D., Austin, D. W., & Kyrios, M. (2011). Anxiety
online: a virtual clinic: Preliminary outcomes following com-
pletion of five fully automated treatment programs for anxiety
disorders and symptoms. Journal of Medical Internet Research, 13
(4), e89. DOI:10.2196/jmir.1918.
Klein, B., Mitchell, J., Abbott, J., Shandley, K., Austin, D., Gilson, K.,
Redman, T. (2010). A therapist-assisted cognitive behavior therapy
internet intervention for posttraumatic stress disorder: Pre-, post-
and 3-month follow-up results from an open trial. Journal of Anxi-
ety Disorders, 24(6), 635–644. DOI:10.1016/j.janxdis.2010.04.005.
Klein, B., Mitchell, J., Gilson, K., Shandley, K., Austin, D.,
Kiropoulos, L., … Cannard, G. (2009). A therapist-assisted
Internet-based CBT intervention for posttraumatic stress disor-
der: Preliminary results. Cognitive Behaviour Therapy, 38(2),
121–131. DOI:10.1080/16506070902803483.
Knaevelsrud, C., & Maercker, A. (2006). Does the quality of the
working alliance predict treatment outcome in online psycho-
therapy for traumatized patients? Journal of Medical Internet Re-
search, 8(4), e31. DOI:10.2196/jmir.8.4.e31.
Knaevelsrud, C., & Maercker, A. (2007). Internet-based treatment
for PTSD reduces distress and facilitates the development of a
strong therapeutic alliance: A randomized controlled clinical
trial. BMC Psychiatry, 7, 13. DOI:10.1186/1471-244X-7-13.
Kroenke, K., & Spitzer, R. L. (2002). The PHQ-9: A new depression di-
agnostic and severity measure. Psychiatric Annals, 32(9), 509–515.
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001). The PHQ-9
—Validity of a brief depression severity measure. Journal of
General Internal Medicine, 16(9), 606–613. DOI:10.1046/j.1525-
1497.2001.016009606.x.
Copyright © 2016 John Wiley & Sons, Ltd.
461
Lindner, P., Linderot Olsson, E., Johnsson, A., Dahlin, M.,
Andersson, G., & Carlbring, P. (2014). The impact of telephone
versus e-mail therapist guidance on treatment outcomes, ther-
apeutic alliance and treatment engagement in internet-
delivered CBT for depression: A randomised pilot trial. Internet
Interventions, 1, 182–187. DOI:10.1016/j.invent.2014.09.001.
Luborsky, L., Barber, J. P., Siqueland, L., Johnson, S., Najavits, L.
M., Frank, A., & Daley, D. (1996). The Revised Helping Alli-
ance Questionnaire (HAq-II). Journal of Psychotherapy Practice
and Research, 5, 260–271.
Manea, L., Gilbody, S., & McMillan, D. (2012). Optimal cut-off
score for diagnosing depression with the Patient Health Ques-
tionnaire (PHQ-9): A meta-analysis. Canadian Medical Associa-
tion Journal, 184(3), E191–E196. DOI:10.1503/Cmaj.110829.
Meyer, B., Pilkonis, P. A., Krupnick, J. L., Egan, M. K.,
Simmens, S. J., & Sotsky, S. M. (2002). Treatment expectan-
cies, patient alliance, and outcome: Further analyses from
the National Institute of Mental Health Treatment of De-
pression Collaborative Research Program. Journal of Con-
sulting and Clinical Psychology, 70(4), 1051–1055.
Muran, J. C., Segal, Z. V., Samstag, L. W., & Crawford, C. E.
(1994). Patient pretreatment interpersonal problems and thera-
peutic alliance in short-term cognitive therapy. Journal of
Consulting and Clinical Psychology, 62(1), 185–190.
Muthén, L., & Muthén, B. (2008). Mplus user’s guide. Los Angeles,
CA: Muthen & Muthen.
Paxling, B., Lundgren, S., Norman, A., Almlov, J., Carlbring, P.,
Cuijpers, P., & Andersson, G. (2013). Therapist behaviours in
internet-delivered cognitive behaviour therapy: Analyses of
e-mail correspondence in the treatment of generalized anxiety
disorder. Behavioural and Cognitive Psychotherapy, 41(3), 280–289.
DOI:10.1017/s1352465812000240.
Preschl, B., Maercker, A., & Wagner, B. (2011). The working alli-
ance in a randomized controlled trial comparing online with
face-to-face cognitive-behavioral therapy for depression.
BMC Psychiatry, 11. DOI:10.1186/1471-244x-11-189.
Saunders, S. M. (2001). Pretreatment correlates of the therapeutic
bond. Journal of Clinical Psychology, 57(12), 1339–1352.
DOI:10.1002/jclp.1101[pii].
Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs,
J., Weiller, E., et al. (1998). The Mini-International Neuropsychi-
atric Interview (MINI): The development and validation of a
structured diagnostic psychiatric interview for DSM-IV and
ICD-10. Journal of Clinical Psychiatry, 59, 22–33.
Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Lowe, B.
(2006). A brief measure for assessing generalized anxiety
disorder—The GAD-7. Archives of Internal Medicine, 166(10),
1092–1097. DOI:10.1001/archinte.166.10.1092.
Stiles, W. B., Glick, M. J., Osatuke, K., Hardy, G. E., Shapiro,
D. A., Agnew-Davies, R., … Barkham, M. (2004). Patterns
of alliance development and the rupture-repair hypothesis:
Are productive relationships U-shaped or V-shaped? Journal
of Counseling Psychology, 51(1), 81–92. DOI:10.1037/0022-
0167.51.1.81.
Tracey, T. J., & Kokotovic, A. M. (1989). Factor structure of the Work-
ing Alliance Inventory. Psychological Asessment, 1(3), 207–210.
Wang, P. S., Berglund, P., Olfson, M., Pincus, H. A., Wells, K. B., &
Kessler, R. C. (2005). Failure and delay in initial treatment contact
after first onset of mental disorders in the national comorbidity
survey replication. Archives of General Psychiatry, 62(6), 603–613.
DOI:10.1001/archpsyc.62.6.603.
Clin. Psychol. Psychother. 24, 451–461 (2017)