Available online at www.sciencedirect.

com

ScienceDirect
Behavior Therapy 53 (2022) 334–347

www.elsevier.com/locate/bt

Chatbot-Delivered Psychotherapy for Adults With Depressive and

Anxiety Symptoms: A Systematic Review and Meta-Regression ,,
Shi Min Lim
National University Hospital, National University Health System
Chyi Wey Claudine Shiau
Tan Tock Seng Hospital, National Healthcare Group
Ling Jie Cheng
Saw Swee Hock School of Public Health, National University of Singapore

Ying Lau
Alice Lee Centre for Nursing Studies, Yong Loo Lin School of Medicine,
National University of Singapore

ment of heterogeneity was done using v2 and I2 tests. A

Although psychotherapy is a well-established treatment for
depression and anxiety, chatbot-delivered psychotherapy is
an emerging field that has yet to be explored in depth. This
review aims to (a) examine the effectiveness of chatbot-
delivered psychotherapy in improving depressive symp-
toms among adults with depression or anxiety, and (b)
evaluate the preferred features for the design of chatbot-
delivered psychotherapy. Eight electronic databases were
searched for relevant randomized controlled trials. Meta-
analysis and random effects meta-regression was con-
ducted using Comprehensive Meta-Analysis 3.0 software.
Overall effect was measured using Hedges’s g and deter-
mined using z statistics at significance level p < .05. Assess-
We would like to acknowledge authors for sharing their study in
our meta-analysis.
We received no funding from an external source.
SML and CWCS conducted the systematic literature search with
help of the senior librarian. SML and CWCS performed the title
and abstract screening, data extraction, thematic data analysis,
and data synthesis. SML, CWCS, LJC, and YL participated in the
conception and the design of the study, the development of the
methodology, data management, and reference editing. SML, YL,
and LJC wrote and formatted the paper and SML, CWCS, LJC,
and YL read and approved the final version of the paper.
Address correspondence to Ying Lau, Ph.D., Level 2, Clinical
Research Centre, Block MD11, 10 Medical Drive, Singapore
117597. e-mail: nurly@nus.edu.sg.
0005-7894/Ó 2021 Association for Behavioral and Cognitive Therapies.
Published by Elsevier Ltd.
meta-analysis of 11 trials revealed that chatbot-delivered
psychotherapy significantly improved depressive symptoms
(g = 0.54, 95% confidence interval [ 0.66, 0.42], p <
.001). Although no significant subgroup differences were
detected, results revealed larger effect sizes for samples of
clinically diagnosed anxiety or depression, chatbots with
an embodiment, a combination of types of input and out-
put formats, less than 10 sessions, problem-solving ther-
apy, off-line platforms, and in different regions of the
United States than their counterparts. Meta-regression
did not identify significant covariates that had an impact
on depressive symptoms. Chatbot-delivered psychotherapy
can be adopted in health care institutions as an alternative
treatment for depression and anxiety. More high-quality
trials are warranted to confirm the effectiveness of
chatbot-delivered psychotherapy on depressive symptoms.
PROSPERO registration number: CRD42020153332.
Keywords: artificial intelligence; psychotherapy; depression; meta-
analysis; systematic review
DEPRESSIVE DISORDERS ARE DISTINGUISHED by pes-
simism, diminished interest levels, experience of
guilt, and depleted self-worth (World Health
Organization, 2017), whereas anxiety disorders
are distinguished by feelings of excessive worry
and fear (World Health Organization, 2017). It
 chatbot-delivered psychotherapy
was estimated that adults with depressive disor-
ders surged by 18.4% from 2005 to 2015 (Vos
et al., 2016) and adults diagnosed with anxiety dis-
orders increased by 14.9% from 2005 to 2015
(Vos et al., 2016). Comorbidity of depression
and anxiety are frequently observed, where
85.0% of depressive patients experienced anxiety
symptoms and 90.0% of anxious patients also
experienced depressive symptoms (Tiller, 2013).
Both disorders are associated with poor social
functioning of the individual and the patients are
at increased risk of suicide attempts (Heeren
et al., 2018). Substantial health care burden and
expenditure are placed on national health systems
due to the relapsing and recurrent nature of
depression and anxiety (Kuyken et al., 2015),
and contribute to losses in worldwide work partic-
ipation and productivity (Chisholm et al., 2016).
Psychotherapy including cognitive-behavioral
therapy (CBT) and problem-solving therapy (PST)
are well supported for the treatment of depression
and anxiety (Weitz et al., 2018). CBT posits that
dysfunctional cognitions bring about the perpetua-
tion of distressed emotions and maladaptive behav-
iors (Beck, 1970). CBT techniques target change in
dysfunctional cognitions, which subsequently
improve emotional distress and maladaptive
behaviors (Hofmann et al., 2012). According to
D’Zurilla and Goldfried (1971), PST aims to gener-
ate individual appraisal of the problem and
prompts the individual to identify effective solu-
tions to cope with the problem. PST reduces the
perceived severity of the problem by inducing the
individual to develop different strategies to counter
a problem and enhance his or her internal locus of
control (Warmerdam et al., 2010).
Although effectiveness of traditional face-to-
face psychotherapy has been established, only a
minority of depressed and anxious individuals
receive treatment in a timely and adequate manner
(Patel et al., 2010) due to treatment barriers
related to accessibility, convenience, worries about
stigmatization, and costs (Lindhiem et al., 2015).
Chatbots are innovative subsets of technological
interventions that may be utilized to deliver psy-
chotherapy while overcoming barriers of tradi-
tional psychotherapy. Chatbots are automated
software that hold a text-based or speech-based
dialogue with humans through an interactive inter-
face (Abdul-Kader & Woods, 2015). Chatbots are
conducted on platforms independent of time and
place, making them widely accessible to individu-
als (Lin et al., 2017). Anonymity of the user is pre-
served, allaying worries about stigmatization with
regard to the sensitive topics discussed (Schnyder
et al., 2017).
335
The mechanism by which chatbots work can be
explained by the persuasive systems design (PSD)
model (Scholten et al., 2017). According to Kegel
and Wieringa (2014), the PSD model consists of
three elements: the intent, the event, and the strat-
egy. First, chatbots identify the event, which is the
problem experienced by the user through analyz-
ing the context of user response. Next, the strategy
consisting of messages that utilizes persuasive tech-
niques is communicated to the user. Technology-
mediated persuasive techniques are structured to
change the cognitive stance of the individual via
the provision of support in four dimensions: “pri-
mary task support” that assists users in task
accomplishment, “dialogue support” that guides
and offers constructive feedback to the user, “cred-
ibility support” that confirms the trustworthiness
of the feedback generated, and “social support”
that encourages the user throughout the behavior
change process (Oinas-Kukkonen & Harjumaa,
2009). The intent refers to the overall intended
behavior or attitude change that the chatbot aims
to achieve. The process of analyzing user response
and provision of persuasive messages continues
until the intent is achieved. Chatbot-delivered psy-
chotherapy combines the delivery of psychothera-
peutic content with a range of persuasive
techniques, simulating an interaction similar to a
therapeutic conversation (Bendig et al., 2019).
Chatbots can be classified according to the pres-
ence of embodiment (presence of human features),
variation in input and output formats (variation in
ways of responding to and receiving responses
from), and can be based on online or off-line
(without an Internet connection) platforms (Abd-
alrazaq et al., 2019).
Given the emerging nature of chatbots in the
field of psychotherapy, similar reviews (Abd-
alrazaq et al., 2019; Montenegro et al., 2019;
Vaidyam et al., 2019) focused on conceptualizing
the scope of chatbots and their uses in clinical psy-
chology, instead of evaluating their effects on men-
tal health outcomes. Existing meta-analytic
reviews that reported effects on depressive symp-
toms (Bennett et al., 2019; Moman et al., 2019;
Wahle et al., 2017) were restricted to mixed inter-
ventions as they included a broad scope of
technology-delivered mental health interventions,
leading to high statistical heterogeneity in their
results. Only one meta-analysis specifically ana-
lyzed chatbot-delivered psychotherapy (Twomey
et al., 2017), but the included trials involved vary-
ing degrees of therapist involvement that induced
variability in the efficacy of the intervention. Given
the potential that chatbot-delivered psychotherapy
has on improving depressive symptoms, further
336 lim et al.
research must be performed to evaluate their effec-
tiveness. It was hypothesized that chatbot-
delivered psychotherapy would improve depressive
symptoms in adults with depression or anxiety.
The review objectives include (a) synthesize
available evidence that examines the effectiveness
of chatbot-delivered psychotherapy in improving
depressive symptoms among adults with depression
or anxiety, and (b) identify the preferred features in
designing chatbot-delivered psychotherapy.
Method
The Preferred Reporting Items for Systematic
Reviews and Meta-Analyses (PRISMA) was
employed in the conduct of this systematic review
and meta-analysis (Moher et al., 2009). This
review is registered in the PROSPERO database
at the Centre for Reviews and Dissemination in
the United Kingdom (CRD42020153332).
inclusion and exclusion criteria
We included all types of randomized controlled
trials (RCTs) that had adults with subclinical or
clinically diagnosed depression, anxiety, or both.
Trials were included if they incorporated autono-
mous chatbots that involved synchronous two-
way exchanges with the user, delivered psy-
chotherapeutic content with the intent to treat
depression or anxiety, and were conducted via
mobile or computer devices. Due to the emerging
nature and the associated heterogeneous terminol-
ogy used to describe chatbot technology, the crite-
ria by McTear et al. (2016a) is used in this review
to identify an autonomous chatbot. First, an appli-
cation interface must be present for users to input
their responses (spoken language or text). Second,
software present in the chatbot must indepen-
dently interpret the meaning of words by the user.
Third, the chatbot must formulate a response to
the user based on the input given (spoken language
or text). Comparators included wait-list control,
treatment as usual (TAU), or information-only
control, all of which lacked chatbot technology.
Primary outcome includes depressive symptoms
measured at the immediate postintervention per-
iod. No limits were in place for the publication
year, and only trials reported in the English lan-
guage were included. Details of the eligibility crite-
ria are found in the supplementary material
information.
search strategy
A three-step extensive search was performed in
line with the guidelines proposed by the Cochrane
Handbook for Systematic Reviews of Interven-
tions (Higgins & Green, 2011). First, we searched
the electronic databases PubMed, Embase,
Cochrane Library, CINAHL, PsycINFO, Scopus,
and IEEE Xplore from study inception until
December 15, 2019, and updated in March 18,
2020, to identify eligible trials. A detailed search
strategy for eight databases is found in the supple-
mentary material. Second, a search was conducted
for unpublished trials from a variety of clinical
trial registries. Finally, reference lists of similar
systematic reviews and included studies, gray liter-
ature, and target journals were hand searched to
maximize potential trials. Original authors of rele-
vant trials were reached via e-mail correspondence
to request missing information or to seek clarifica-
tions. Studies with title or abstracts that met the
selection criteria were selected for full-text evalua-
tion by two independent reviewers. Reasons of
exclusion were documented. Interrater agreement
using the Cohen kappa test statistic was calculated
(McHugh, 2012). A third author was involved for
the resolution of any disagreements.
data extraction
Data items extracted for characteristics of
included trials consist of author, design, year,
country, mean age, nature of sample, intervention,
control, sample size, attrition rate, missing data
management, intention-to-treat (ITT) analysis,
trial registration, published protocol, and grant
support. Data items extracted relevant to
chatbot-delivered psychotherapy included plat-
form, user input format, chatbot output format,
psychotherapy, presence of embodiment, number
of sessions, length of session, session frequency,
intervention duration, follow-up, and outcomes.
quality assessment
Two authors appraised the quality of the eligible
trials independently by utilizing the Cochrane
risk-of-bias tool (Higgins & Green, 2011). Six
assessment domains were categorized as “high
risk,” “low risk,” or “unclear risk”—namely, (a)
generation of random sequence, (b) concealment
of allocation sequence, (c) blinding of participants
and personnel, (d) blinding of outcome assess-
ment, (e) incomplete outcome data, and (f) selec-
tive reporting. Interrater agreement using the
Cohen kappa test statistic was calculated
(McHugh, 2012). Any disagreements were
resolved by LJC. The Grading of Recommenda-
tions, Assessment, Development and Evaluation
(GRADE) framework was utilized to evaluate the
overall evidence quality across trials (Guyatt
et al., 2011). The quality of evidence was
appraised based on five evidence factors: (a)
methodological limitations, (b) inconsistency, (c)
 chatbot-delivered psychotherapy
indirectness, (d) imprecision, and (e) publication
bias. Each factor was rated “not serious,” “seri-
ous,” or “very serious.”
statistical analysis
Comprehensive Meta-Analysis Software (Version
3; Borenstein et al., 2013) was used to conduct
the meta-analyses by pooling data of the same out-
comes under a random effects model. A random
effects model was adopted as it accounts for the
statistical assumption of variation in the estima-
tion of the mean across all trials (Borenstein
et al., 2010). Z statistics at a significance level of
p < .05 was used to analyze the overall effect.
Pooled mean effect sizes of continuous outcome
scores were estimated using Hedges’s g, as it pro-
vides a more accurate estimation of the overall
effect size when the included trials have small sam-
ple sizes (Rosenthal et al., 1994). Effect size was
interpreted as small (0.2), medium (0.5), large
(0.8), and very large (1.2; Hedges & Olkin,
2014). The inverse–variance statistical method
was utilized to analyze continuous outcomes
(Higgins & Green, 2011). Statistical heterogeneity
was evaluated based on Cochran’s Q test, where p
< .10 indicates the presence of significant statistical
heterogeneity (Higgins & Green, 2011). The I2
statistic was used to quantify the degree of hetero-
geneity, where I2 indicates the variation percent-
age between the sample estimates of trials that is
due to heterogeneity instead of sampling error
(Sedgwick, 2013). The I2 was interpreted as 0–
40% (might not be important), 30–60% (moder-
ate), 50–90% (substantial), and 75–100% (consid-
erable; Higgins & Green, 2011).
Sensitivity analyses were applied to identify
heterogeneous trials that were removed to main-
tain overall homogeneity (Higgins & Green,
2011). Subgroup analyses were conducted to
reduce overall heterogeneity across studies (I2 >
40%; Sedgwick, 2013) and to compare the treat-
ment effects among features of intervention
(Borenstein & Higgins, 2013). Predefined sub-
groups included the nature of sample, type of com-
parator, type of psychotherapy, input format,
output format, chatbot embodiment, intervention
duration, and platform. Publication bias was
examined using visual inspection of asymmetry
of funnel plot and Egger’s test (Egger et al.,
1997) when the review included 10 or more trials
(Sterne et al., 2011). Meta-regression was con-
ducted to explain whether between-trial hetero-
geneity was attributed to covariates (Borenstein
et al., 2011). A random effects multivariate
meta-regression model was conducted to assess
whether year of publication, mean age of sample,
337
duration of intervention, and sample size influ-
enced the effect size of chatbot-delivered psy-
chotherapy. We adopted a p < .05 significance
level for random effects meta-regression analysis.
Results
study selection
The PRISMA flow diagram depicts the process of
trial selection as shown in Figure 1. A total of 11
RCTs were selected for meta-analysis. Interrater
agreement of .79 was calculated using Cohen’s
kappa for trial selection, which indicates consider-
able agreement between two authors regarding
trial inclusion (McHugh, 2012). Included trials
are presented in Table 1 and are marked with an
asterisk (*) in the reference list. Reasons for eligi-
bility of intervention considered as chatbot tech-
nology in this review is included in the
supplementary material information.
study characteristics
Table 1 summarizes the trial characteristics among
1,099 participants from 2009 (Meyer et al., 2009)
to 2017 (Berger et al., 2017; Fitzpatrick et al.,
2017; Zwerenz et al., 2017) in Switzerland
(Berger et al., 2011, 2017), the United Kingdom
(Burton et al., 2016), the United States (Cartreine
et al., 2012; Fitzpatrick et al., 2017; Sandoval
et al., 2017), and Germany (Meyer et al., 2009,
2015; Moritz et al., 2012; Schröder et al., 2014;
Zwerenz et al., 2017). Sample sizes ranged from
14 (Cartreine et al., 2012) to 396 (Meyer et al.,
2009). Seven trials included adults with a clinical
diagnosis of depression or anxiety, while four tri-
als included adults with a subclinical disorder of
depression, anxiety, or both. Three trials had chat-
bots with embodiment. Two trials utilized CBT
only, while two trials utilized PST only. Three tri-
als had chatbots that were delivered via off-line
platforms. Three trials had chatbots with response
options and written as the input format. Three tri-
als had chatbots that used a combination of writ-
ten, spoken, and gestures as the output format.
The intervention duration was between 2 weeks
(Fitzpatrick et al., 2017) and 12 weeks (Meyer
et al., 2015; Zwerenz et al., 2017).
Seven trials had an overall unclear risk of bias
attributed mainly to insufficient details on the
domain of performance bias, while four trials
had an overall high risk of bias attributed mainly
to the presence of attrition bias. Interrater agree-
ment of .81 was calculated using Cohen’s kappa
for risk of bias assessment of individual trials,
which indicates near perfect agreement
(McHugh, 2012).
338 lim et al.

5180 records idenfied from database

Idenficaon
searching (ll 31st December 2019)
PubMed (N=879), EMBASE (N=1072),
CINHAL (N=347), Cochrane (N =1440),
PsycINFO (N =523), Scopus (N =655), Addional records idenfied
ProQuest Dissertaons and Theses (N =86), through reference list searching
IEEExplore (N =178) (N = 2)

2889 records were idenfied as

duplicates and removed using
Screening

ENDNOTE soware

Records screened (N = 2293)

Reasons for 2220 records exclusion:

Irrelevant tle (N= 1825)
Irrelevant abstract (N= 395)
Eligibility

Full-text arcles assessed

for eligibility
(N = 73)

Reasons for n= 62 full-text arcles

exclusion:
No chatbot (N = 27)
Human support involved (N = 13)
Populaon without depressive or
anxiety symptoms (N = 11)
Non-RCT (N = 3)
Study protocol (N = 3)
Included

Abstract only (N = 3)

Trials included in
meta-analysis
(N= 11)

FIGURE 1 Preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram.

depressive symptoms
A meta-analysis on depressive symptoms involv-
ing 1,099 participants was conducted in the
selected 11 trials as shown in Figure 2. Compared
to control conditions, results of the meta-analysis
showed a significant improvement in depressive
symptom scores (z = 8.64, p < .001), with a med-
ium effect size (g = 0.54), favoring chatbot-
delivered psychotherapy using a random effects
model. Sensitivity analysis was conducted to
exclude trials with a high risk of bias (Lundh &
Gøtzsche, 2008). Results after the sensitivity
analysis indicated minimal change in depressive
symptom scores (z = 7.16, p < .001) and effect
size (g = 0.58), favoring chatbot-delivered
psychotherapy.
subgroup analyses
Subgroup analyses discovered larger effect sizes for
chatbots with embodiment (g = 0.88), using off-
Table 1
Characteristics of Included Randomized Controlled Trials
Author Design Nature of sample/mean age ± Sample Intervention (name) Control Outcomes Attrition ITT/missing Protocol/
(year)/country SD size (measures) rate (%) data registration/grant
management support
Berger et al. 3-arm Adults diagnosed with MDD or T: 51 Unguided Internet-based Wait-list Depressive 13.7 Yes/yes No/no/yes
(2011)/ RCT dysthymia/38.80 ± 14.0 I: 25 program (Deprexis) control symptoms
Switzerland C: 26 (BDI-II)
Berger et al. 2-arm Adults diagnosed with SAD, T: 139 Unguided Internet TAU Depressive 13.7 No/no No/yes/yes
(2017)/ RCT PDA, or GAD/41.95 ± NR I: 70 intervention (Velibra) symptoms
Switzerland C: 69 (BDI-II)

chatbot-delivered psychotherapy
Burton et al. 2-arm Adults diagnosed with MDD/ T: 28 Embodied virtual agent- TAU Depressive 25.0 No/no No/yes/yes
(2016)/United RCT 38.65 ± NR I: 14 based system symptoms
Kingdom C: 14 (Help4Mood) (BDI-II)
Cartreine et al. 2-arm Adults diagnosed with minor T: 14 Electronic problem-solving Wait-list Depressive 21.4 Yes/yes No/yes/yes
(2012)/United RCT depression/50.40 ± NR I: 7 treatment (ePST) control symptoms
States C: 7 (HDI)
Fitzpatrick et al. 2-arm Adults with subclinical anxiety T: 70 Fully automated Information Depressive 20.0 Yes/yes No/no/yes
(2017)/United RCT and depressive symptoms22.21 I: 34 conversational agent only symptoms
States ± NR C: 36 (Woebot) (PHQ-9)
Meyer et al. 2-arm Adults with subclinical T: 396 Integrative online treatment Wait-list Depressive 45.5 No/yes No/yes/yes
(2009)/Ger- RCT depression/34.76 ± 11.60 I: 320 (Deprexis) control symptoms
many C: 76 (BDI)
Meyer et al. 2-arm Adults diagnosed with MDD or T: 163 Internet-based treatment TAU Depressive 17.8 Yes/yes No/yes/yes
(2015)/Ger- RCT Dysthymia/42.00 ± 11.39 I: 78 (Deprexis) symptoms
many C: 85 (PHQ-9)
Moritz et al. 2-arm Adults with subclinical T: 210 Online self-help program Wait-list Depressive 19.0 No/no No/yes/no
(2012)/Ger- RCT depression/38.60 ± NR I: 105 for depression (Deprexis) control symptoms
many C: 105 (BDI)
Sandoval et al. 2-arm Adults diagnosed with MDD or T: 45 Interactive media-based TAU Depressive 0 No/no No/no/yes
(2017)/United RCT Dysthymic Disorder/28.60 ± NR I: 25 problem-solving therapy symptoms
States C: 20 (imbPST) (HSCL)
Schröder et al. 2-arm Adults with subclinical T: 78 Psychological online Wait-list Depressive 26.9 No/no No/yes/no
(2014)/Ger- RCT depression and epilepsy/37.53 I: 38 intervention for depression control symptoms
many ± NR C: 40 (Deprexis) (BDI-I)
Zwerenz et al. 2-arm Adults diagnosed with T: 229 Online self-help (Deprexis) Information Depressive 6.11 Yes/yes Yes/yes/yes
(2017)/Ger- RCT depression/47.98 ± 9.79 I: 115 only symptoms
many C: 114 (BDI-II)
Note. SD = standard deviation; ITT = intent to treat; RCT = randomized controlled trials; MDD = major depressive disorder; T = total; I = intervention; C = control; BDI = Beck Depression
Inventory; SAD = social anxiety disorder; PDA = panic disorder with or without agoraphobia; GAD = generalized anxiety disorder; NR = not reported; HDI = Hamilton Depression Inventory; PHQ-
9 = Patient Health Questionnaire–9; TAU = treatment as usual; HSCL = Hopkins Symptom Checklist.

339
340 lim et al.
Study name Statistics for each study Sample size Year of Publication Age Country Hedges's g and 95% CI
Hedges's Chatbot
g Z-Value p-Value Psychotherapy Control
Berger et al (2011) -0.65 -2.31 0.02 25 26 2011 38.80 Europe
Berger et a; (2017) -0.56 -3.05 0.00 57 63 2017 41.95 Europe
Burton et al (2016) -0.47 -1.09 0.28 12 9 2016 38.65 Europe
Cartreine et al (2012) -1.24 -2.25 0.02 7 7 2012 50.40 US
Fitzpatrick et al (2017) -0.62 -2.30 0.02 31 25 2017 22.21 US
Meyer et al (2009) -0.64 -4.06 0.00 159 57 2009 34.76 Europe
Meyer et al (2015) -0.57 -3.22 0.00 61 73 2015 42.00 Europe
Moritz et al (2012) -0.43 -2.78 0.01 80 90 2012 38.60 Europe
Sandoval et al (2016) -0.98 -3.14 0.00 25 20 2016 28.60 US
Schroder et al (2014) -0.22 -0.82 0.41 25 32 2014 37.53 Europe
Zwerenz et al (2017) -0.44 -3.20 0.00 108 107 2017 47.98 Europe
-0.54 -8.64 0.00

-2.00 -1.00 0.00 1.00 2.00

Chatbot Psychotherapy Control

FIGURE 2 Forest plot of effect size (Hedges’s g) in depressive symptoms scores for Chatbot psychotherapy and control group.

line platforms (g = 0.88), delivering PST (g = 1.05), Discussion

having combined forms of input format (g = 0.84), Our review showed significant improvements on
less than 10 sessions (g = 0.75), and combined depressive symptoms of medium effect size.
forms of output format (g = 0.88). Results are pre- Heterogeneity was low and no publication bias
sented in Table 2. In addition, there are larger was detected. Random effect meta-regression did
effect sizes among those samples with either clini- not identify any impact of the covariates on effect
cally diagnosed anxiety (g = 0.56) or depression (g size.
= 0.55) in the United States (g = 0.83). All of the
analyses revealed statistically significant effects quality assessment
for chatbot-delivered psychotherapy. No signifi- The risk of bias domain was downgraded owing to
cant subgroup differences were detected. the presence of performance bias in the majority of
included trials. Performance bias arose from the
meta-regression lack of blinding of participants in all included tri-
We conducted a random effect meta-regression to als. Due to the lack of blinding of participants, the
evaluate the impact of covariates on the effect size placebo effect cannot be ruled out (Charlesworth
of depressive symptoms. We included the follow- et al., 2017). The imprecision domain was down-
ing covariates: year of publication, mean age of graded owing to the presence of small sample sizes
sample, intervention duration, and sample size, in the majority of included trials, which diminishes
as shown in Table 3. Our findings suggested that the probability of discovering true effects (Button
the year of publication (b = .01, p = .67), mean et al., 2013). ITT analyses were utilized in only 5
age of sample (b = .01, p = .38), intervention dura- out of 11 trials. According to the Consolidated
tion (b = .02, p = .47), and sample size (b < .001, p Standards of Reporting Trials (CONSORT) state-
= .46) had no impact on depressive symptoms. ment, all RCTs should adopt ITT analyses and
evaluate results according to participants’ original
overall evidence assessment group assignments. Failure to perform ITT analy-
The overall quality of evidence for depressive ses can lead to the disruption of prognostic balance
symptoms is rated as low for the RCTs as per among treatment allocation groups and the overes-
the GRADE criteria. The risk-of-bias factor was timation of treatment effects (Abraha et al., 2015).
downgraded due to a high risk for performance
bias. The imprecision factor was downgraded effectiveness on depressive symptoms
due to small sample sizes present in more than half Our meta-analysis found that chatbot-delivered
of the included trials. The GRADE Summary of psychotherapy significantly reduced depressive
Findings table is found in the supplementary mate- symptoms among adults with depression or anxi-
rial information. Publication bias was not detected ety. Our review included only trials that contained
for trials that reported depressive symptoms as chatbots as part of the intervention, which can be
observed by the symmetrical distribution of considered a step forward in literature. Chatbot-
included trials on the funnel plot (Egger’s test, P delivered psychotherapy is a form of guided com-
= 0.07) as presented in Supplementary material 9. puterized self-help in the absence of human inter-
Table 2
Subgroup Analyses of Chatbot-Delivered Psychotherapy for Depressive Symptoms
Design Subgroups Number of trialsref Sample size I2 (%) g Overall effect (p value) Subgroup differences (p
value)
Population Clinical diagnosis 7a-d,g,i,k 600 0 0.57 z = 6.86 (p < .001) Q = 0.27 (p = .601)
Subclinical disorder 4e,f,h,j 499 0 0.50 z = 5.28 (p < .001)
Nature of sample Anxiety only 1b 120 0 0.56 z = 3.05 (p < .001) Q = 0.43 (p = .81)
Depression with 2e,j 113 13.78 0.42 z = 2.04 (p = .04)
comorbidities
Depression only 8a,c,d,f,g,h,i,k 866 0 0.55 z = 7.82 (p < .001)

chatbot-delivered psychotherapy
Region Europe 8a,b,c,f,g,h,j,k 984 0 0.51 z = 7.65 (p < .001) Q = 2.62 (p = .11)
United States 3d,e,i 115 0 0.83 z = 4.34 (p < .001)
Embodiment With embodiment 3c,d,i 80 0 0.88 z = 3.83 (p < .001) Q = 2.37 (p = .124)
Without embodiment 8a,b,e–h,j,k 1,019 0 0.51 z = 7.90 (p < .001)
Platform Online 8a,b,e–h,j,k 1,019 0 0.51 z = 7.90 (p < .001) Q = 2.37 (p = .124)
Offline 3c,d,i 80 0 0.88 z = 3.83 (p < .001)
Psychotherapeutic CBT only 2c,e 77 0 0.58 z = 2.53 (p = .01) Q = 3.74 (p = .154)
content
PST only 2d,i 59 0 1.05 z = 3.84 (p < .001)
Mixed psychotherapy 7a,b,f–h,j,k 963 0 0.51 z = 7.57 (p < .001)
Input format Response options only 8a-c,f–h,j,k 984 0 0.51 z = 7.65 (p < .001) Q = 2.62 (p = .105)
Response options + written 3d,e,i 115 0 0.84 z = 4.34 (p < .001)
Output format Written only 8a,b,e,f–h,j,k 1,019 0 0.51 z = 7.90 (p < .001) Q = 2.37 (p = .124)
Written + spoken + gestures 3c,d,i 80 0 0.88 z = 3.83 (p < .001)
Number of sessions <10 3b,d,i 179 13.20 0.75 z = 4.23 (p < .001) Q = 1.73 (p = .42)
10 6a,f,g,h,j,k 843 0 0.50 z = 6.94 (p < .001)
Not reported 2c,e 77 0 0.58 z = 2.53 (p = .012)
Note. I2 = I2 statistic; g = Hedges’ g; Q Cochran’s = Q statistic; z = z statistics; CBT = cognitive-behavioral therapy; PST = problem-solving therapy.
a
Berger et al. (2011).
b
Berger et al. (2017).
c
Burton et al. (2016).
d
Cartreine et al. (2012).
e
Fitzpatrick et al. (2017).
f
Meyer et al. (2009).
g
Meyer et al. (2015).
h
Moritz et al. (2012).
i
Sandoval et al. (2017).
j
Schröder et al. (2014).
k
Zwerenz et al. (2017).

341
342
Table 3
Random Effects Meta-Regression Models of Chatbot-Delivered Psychotherapy by Various Covariates
Covariate b Standard error
Year of publication 0.01
Mean age of sample 0.01
Duration of intervention 0.02
Sample size <0.001
Note. b = regression coefficient; z = z statistics.
vention (McTear et al., 2016b). Previous reviews
found comparable effects between traditional
face-to-face psychotherapy and computerized psy-
chotherapy (Carlbring et al., 2018). In addition,
previous reviews have shown that guided psy-
chotherapy is more effective than unguided psy-
chotherapy attributed to external pressure
fostering individual accountability and positive
reinforcements from the source of guidance (i.e.,
therapist; Alfonsson et al., 2017).
Chatbots offer a solution to the shortage of
mental health workers, as they are able to function
autonomously while providing guided psychother-
apy via automated script-based dialogue (Oh
et al., 2017), and are suggested to form therapeutic
alliances with individuals similar to those formed
with human therapists (Kiluk et al., 2014). Previ-
ous meta-analyses (Twomey et al., 2017; Wahle
et al., 2017) that included chatbots as part of the
intervention reported similar effect sizes as did
our review, which demonstrates the consistency
of effect sizes when chatbots are adopted as part
of the intervention. The meta-analyses of Moman
et al. (2019) and Bennett et al. (2019) that
included various technological interventions (dis-
cussion forums, online workbooks, video games)
but did not involve chatbots reported lower effect
sizes.
One possible reason for the lower effect sizes
may be the superiority of chatbot technology over
other forms of technological interventions, and the
absence of chatbot intervention was reflected in
the lower effect sizes. Another possible reason
for the lower effect sizes may be attributed to the
large heterogeneity among the trial interventions.
Clinically heterogeneous trials contribute to the
variability of effect estimates (Kriston, 2013).
Another reason for the improvement in depressive
symptoms may be attributed to the incorporation
of the well-established theoretical frameworks of
CBT and PST for the treatment of depression
and anxiety (Cuijpers et al., 2018; Newby et al.,
2016) into scripts programmed within the chatbot
software that allow for conversations that mimic
therapeutic discussions (D’Alfonso et al., 2017).
These scripts include techniques of cognitive
restructuring and behavioral activation that
lim et al.
95% lower 95% upper z p value
0.02 0.03 0.05 0.43 .67
0.01 0.01 0.03 0.87 .38
0.03 0.03 0.07 0.72 .47
<0.001 < 0.001 <0.001 0.72 .46
improve negative cognitions (Fernández &
Mairal, 2017), while techniques of problem
appraisal and problem orientation reduce per-
ceived problem severity (Bell & D’Zurilla, 2009).
Both reduction in dysfunctional attitudes and
increased feelings of control played a mediational
role within CBT and PST that in turn improved
depressive symptoms (Warmerdam et al., 2010).
preferred features of chatbot-
delivered psychotherapy
Our subgroup analyses highlighted that embodied
chatbots achieve more favorable effect sizes. Mul-
tiple studies (Araujo, 2018; Go & Sundar, 2019;
Schroeder et al., 2013) also supported the notion
that users preferred to interact with an embodied
agent instead of a text-only interface. Araujo
(2018) suggested that embodiment increases the
perception of human likeness and aids in the sim-
ulation of human-to-human interaction. One pos-
sible reason for the effectiveness of this feature is
that the presence of embodiment increases the
credibility of the chatbot and induces a feeling of
social partnership in the user (Mayer & DaPra,
2012).
Our subgroup analyses also suggested that users
preferred to interact with chatbots that utilize a
combination of input and output formats. In line
with the review by Montenegro et al. (2019), the
combination of having response options and a
written format as a form of input introduces dia-
logue variability into the interaction between user
and chatbot. The combination of different types of
input format avoids perceived repetitiveness and
subsequent question fatigue in the users
(Henrichsen & Allwood, 2013). In addition, the
use of combined types of output formats (written,
spoken, and gestures) aids in the establishment of a
level of realism, especially when new technologies
are involved (Montenegro et al., 2019). Having a
combination of types of output format allows for
greater expressivity of the chatbot to deliver infor-
mation, where the inclusion of nonverbal behav-
ior—like facial expression, gaze, gestures, and
postures—powerfully influences feelings of rap-
port between chatbot and user (Burgoon et al.,
2016).
 chatbot-delivered psychotherapy
Congruent to our subgroup analyses favoring
off-line platforms, the study by Heer et al. (2011)
recognizes the security challenges faced by utilizing
the Internet as a platform for information exchange,
such as the leaking of confidential information or
instances of identity theft. Off-line platforms offer
greater user privacy based on the virtue that it is sep-
arate from the Internet, away from a global network
that anyone can participate in and access the per-
sonal information disclosed (Vitak, 2012). Future
trials should seek to validate this finding given the
increased utilization of computer technologies
within mental health interventions.
Our subgroup analyses observed that PST
achieved a greater intervention effect compared
to CBT. Our findings are consistent with the
review by Zhang et al. (2018) that demonstrated
higher effect sizes of PST compared to CBT for
the treatment of primary care depression and anx-
iety using technology-delivered psychotherapies.
In PST, the individual adopts a more directive role
in the therapeutic process from problem identifica-
tion to solution implementation (Cuijpers et al.,
2018), while the process of traditional CBT usu-
ally entails the individual following the instruc-
tions of the therapist (Lee et al., 2013). Unlike in
face-to-face psychotherapy where human thera-
pists are able to alter therapeutic content to facili-
tate client elaboration, chatbot responses are
limited to the scope of their programmed scripts,
which restricts the adaptiveness of chatbot-to-
user responses (Rahman et al., 2017). Higher user
involvement in exploration and examination of
their experiences during the therapeutic process
might be a possible explanation for the higher
effect size of chatbot-delivered PST. These findings
should be further confirmed with future high-
quality trials given the small number of trials
included in the subgroup analyses due to the
emerging nature surrounding the types of
chatbot-delivered psychotherapy.
Although no statistically significant subgroup
differences were detected, this review intended to
examine the relative effect sizes of subgroup fea-
tures as a guide for future studies on the emerging
topic of chatbot-delivered psychotherapy.
strengths and limitations
We identified several strengths in our review. First,
our review adopted a comprehensive search strat-
egy that included both computing and medical
databases to identify more potential trials. Second,
low statistical heterogeneity was also achieved, as
evidenced by the low I2 values in the meta-
analysis. Third, we performed subgroup analyses
to identify preferred features in the design of chat-
343
bots, a step beyond previous reviews that only
identified design features of general technology-
delivered interventions (Morrison et al., 2012;
Whitton et al., 2015). Fourth, publication bias
was not detected among our included trials. Last,
our review utilized the GRADE system to assess
overall evidence quality across trials.
Limitations identified include that all trials were
carried out in Western countries and are hence
restricted to the English language, thereby restrict-
ing the generalization of effects across all coun-
tries. Next, small sample sizes detected in half of
the included trials might induce small study
effects. And, the low quality of overall evidence
might reduce the internal validity of findings.
Next, the small number of trials included in the
meta-analysis might have caused small study
effects (Button et al., 2013). Last, the majority of
the subgroups identified have an uneven covariate
distribution (Richardson et al., 2019) that could
explain the absence of statistical difference within
subgroups, hence the results should be interpreted
with caution.
future research
First, considering the low quality of the overall evi-
dence assessed by GRADE, well-designed RCTs
that follow the recommendations of the CON-
SORT statement should be conducted in future tri-
als, especially efforts to minimize performance
bias by blinding participants and personnel so as
to facilitate proper research implementation and
reporting. Larger trials are needed to strengthen
the evidence. Next, this review also addressed the
need for large sample sizes when conducting trials,
especially in non-Western countries. Last, given
that only limited forms of chatbot-delivered psy-
chotherapy were evaluated in this review, future
trials should explore the effectiveness of other
forms of psychotherapy, such as mindfulness-
based therapy and acceptance and commitment
therapy delivered via chatbots, both of which have
been delivered using other technological modali-
ties in previous RCTs (Lin et al., 2015; Mak
et al., 2015).
clinical implications
Chatbot-delivered psychotherapy provides an
alternative method for the delivery of psychother-
apy to individuals (Clement et al., 2015), and
expands the availability of psychotherapy to indi-
viduals who are unable to access mental health ser-
vices due to limitations of time or location (Ebert
et al., 2018). Chatbot-delivered psychotherapy
also promotes collaboration between the fields of
health sciences and computing, improves the qual-
344 lim et al.
ity of health care services delivered by encouraging
the integration of technology into traditional
health care practices (Kuo, 2011), and allows
health care institutions to upscale their mental
health services with ease since chatbot-serverless
systems do not require dedicated physical infras-
tructure to function (Yan et al., 2016).
conclusions
Chatbot-delivered psychotherapy was found to
produce significant improvements in depressive
symptoms among adults with depression or anxi-
ety. Subgroup analyses found that features includ-
ing samples clinically diagnosed with either
anxiety or depression, chatbots with an embodi-
ment, combination of types of input and output
formats, less than 10 sessions, PST, off-line plat-
forms, and in the United States are more favorable
in the design of chatbots. Meta-regression analyses
did not find the impact of year of publication,
mean age, country, nature of population, embodi-
ment, duration, platform, psychotherapy, input
formats, and output formats as covariates on
depressive symptoms. Future trials should adhere
to the recommendations of the CONSORT state-
ment (Schulz et al., 2010), as more high-quality
trials are needed to fully derive the effectiveness
of chatbot-delivered psychotherapy on depressive
symptoms.
Conflict of Interest Statement
The authors declare that there are no conflicts of interest.
Supplementary data to this article can be found online at
https://doi.org/10.1016/j.beth.2021.09.007.
References
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B.
M., Gardner, P., & Househ, M. (2019). An overview of the
features of chatbots in mental health: A scoping review.
International Journal of Medical Informatics, 132. https://
doi.org/10.1016/j.ijmedinf.2019.103978 103978.
Abdul-Kader, S. A., & Woods, J. (2015). Survey on chatbot
design techniques in speech conversation systems.
International Journal of Advanced Computer Science and
Applications, 6(7). https://doi.org/10.14569/IJACSA.
2015.060712.
Abraha, I., Cherubini, A., Cozzolino, F., De Florio, R.,
Luchetta, M. L., Rimland, J. M., Folletti, I., Marchesi,
M., Germani, A., & Orso, M. (2015). Deviation from
intention to treat analysis in randomised trials and treat-
ment effect estimates: Meta-epidemiological study. BMJ,
350. https://doi.org/10.1136/bmj.h2445 h2445.
Alfonsson, S., Johansson, K., Uddling, J., & Hursti, T. (2017).
Differences in motivation and adherence to a prescribed
assignment after face-to-face and online psychoeducation:
An experimental study. BMC Psychology, 5(1), 3. https://
doi.org/10.1186/s40359-017-0172-5.
Araujo, T. (2018). Living up to the chatbot hype: The
influence of anthropomorphic design cues and commu-
nicative agency framing on conversational agent and
company perceptions. Computers in Human Behavior,
85, 183–189. https://doi.org/10.1016/j.chb.2018.03.051.
Beck, A. T. (1970). Cognitive therapy: Nature and relation to
behavior therapy. Behavior Therapy, 1(2), 184–200.
https://doi.org/10.1016/S0005-7894(70)80030-2.
Bell, A. C., & D’Zurilla, T. J. (2009). Problem-solving therapy
for depression: A meta-analysis. Clinical Psychology
Review, 29(4), 348–353. https://doi.org/10.1016/j.
cpr.2009.02.003.
Bendig, E., Erb, B., Schulze-Thuesing, L., & Baumeister, H.
(2019). The next generation: Chatbots in clinical psychol-
ogy and psychotherapy to foster mental health—A scoping
review. Verhaltenstherapie, 1–13. https://doi.org/10.1159/
000501812.
Bennett, S. D., Cuijpers, P., Ebert, D. D., McKenzie Smith, M.,
Coughtrey, A. E., Heyman, I., Manzotti, G., & Shafran, R.
(2019). Practitioner review: Unguided and guided self-help
interventions for common mental health disorders in
children and adolescents: A systematic review and meta-
analysis. Journal of Child Psychology and Psychiatry.
https://doi.org/10.1111/jcpp.13010.
*Berger, T., Hammerli, K., Gubser, N., Andersson, G., &
Caspar, F. (2011). Internet-based treatment of depression:
A randomized controlled trial comparing guided with
unguided self-help. Cognitive Behaviour Therapy, 40(4),
251–266. https://doi.org/10.1080/16506073.2011.616531.
*Berger, T., Urech, A., Krieger, T., Stolz, T., Schulz, A.,
Vincent, A., Moser, C. T., Moritz, S., & Meyer, B. (2017).
Effects of a transdiagnostic unguided Internet intervention
(“velibra”) for anxiety disorders in primary care: Results of
a randomized controlled trial. Psychological Medicine, 47
(1), 67–80. https://doi.org/10.1017/s0033291716002270.
Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H.
(2010). A basic introduction to fixed-effect and random-
effects models for meta-analysis. Research Synthesis Meth-
ods, 1(2), 97–111. https://doi.org/10.1002/jrsm.12.
Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H.
(2011). Introduction to meta-analysis. Wiley.
Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H.
(2013). Comprehensive meta-analysis (Version 3). Biostat.
Borenstein, M., & Higgins, J. (2013). Meta-analysis and
subgroups. Prevention Science, 14(2), 134–143. https://doi.
org/10.1007/s11121-013-0377-7.
Burgoon, J. K., Guerrero, L. K., & Floyd, K. (2016).
Nonverbal communication. Routledge, 10.4324/
9781315663425.
*Burton, C., Szentagotai Tatar, A., McKinstry, B., Matheson,
C., Matu, S., Moldovan, R., Macnab, M., Farrow, E.,
David, D., Pagliari, C., Serrano Blanco, A., & Wolters, M.
(2016). Pilot randomised controlled trial of Help4Mood,
an embodied virtual agent-based system to support treat-
ment of depression. Journal of Telemedicine and Telecare,
22(6), 348–355. https://doi.org/10.1177/
1357633x15609793.
Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint,
J., Robinson, E. S., & Munafò, M. R. (2013). Power
failure: Why small sample size undermines the reliability of
neuroscience. Nature Reviews Neuroscience, 14(5), 365.
https://doi.org/10.1038/nrn3475.
Carlbring, P., Andersson, G., Cuijpers, P., Riper, H., &
Hedman-Lagerlöf, E. (2018). Internet-based vs. face-to-
face cognitive behavior therapy for psychiatric and somatic
disorders: An updated systematic review and meta-analysis.
 chatbot-delivered psychotherapy
Cognitive Behaviour Therapy, 47(1), 1–18. https://doi.org/
10.1080/16506073.2017.1401115.
*Cartreine, J. A., Locke, S. E., Buckey, J. C., Sandoval, L., &
Hegel, M. T. (2012). Electronic problem-solving treatment:
Description and pilot study of an interactive media
treatment for depression. JMIR Research Protocols, 1(2).
https://doi.org/10.2196/resprot.1925 e11.
Charlesworth, J. E., Petkovic, G., Kelley, J. M., Hunter, M.,
Onakpoya, I., Roberts, N., Miller, F. G., & Howick, J.
(2017). Effects of placebos without deception compared
with no treatment: A systematic review and meta-analysis.
Journal of Evidence-Based Medicine, 10(2), 97–107.
https://doi.org/10.1111/jebm.12251.
Chisholm, D., Sweeny, K., Sheehan, P., Rasmussen, B., Smit,
F., Cuijpers, P., & Saxena, S. (2016). Scaling-up treatment
of depression and anxiety: A global return on investment
analysis. The Lancet Psychiatry, 3(5), 415–424. https://doi.
org/10.1016/S2215-0366(16)30024-4.
Clement, S., Schauman, O., Graham, T., Maggioni, F., Evans-
Lacko, S., Bezborodovs, N., Morgan, C., Rüsch, N.,
Brown, J., & Thornicroft, G. (2015). What is the impact
of mental health-related stigma on help-seeking? A sys-
tematic review of quantitative and qualitative studies.
Psychological Medicine, 45(1), 11–27. https://doi.org/
10.1017/S0033291714000129.
Cuijpers, P., de Wit, L., Kleiboer, A., Karyotaki, E., &
Ebert, D. D. (2018). Problem-solving therapy for adult
depression: An updated meta-analysis. European
Psychiatry, 48(1), 27–37. https://doi.org/10.1016/j.eurpsy.
2017.11.006.
D’Alfonso, S., Santesteban-Echarri, O., Rice, S., Wadley, G.,
Lederman, R., Miles, C., Gleeson, J., & Alvarez-Jimenez,
M. (2017). Artificial intelligence-assisted online social
therapy for youth mental health. Frontiers in Psychology,
8, 796. https://doi.org/10.3389/fpsyg.2017.00796.
D’Zurilla, T. J., & Goldfried, M. R. (1971). Problem solving
and behavior modification. Journal of Abnormal Psychol-
ogy, 78(1), 107–126. https://doi.org/10.1037/h0031360.
Ebert, D. D., Van Daele, T., Nordgreen, T., Karekla, M.,
Compare, A., Zarbo, C., Brugnera, A., Øverland, S.,
Trebbi, G., & Jensen, K. L. (2018). Internet- and mobile-
based psychological interventions: Applications, efficacy,
and potential for improving mental health. European
Psychologist, 23(2), 167–187. https://doi.org/10.1027/
1016-9040/a000318.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997).
Bias in meta-analysis detected by a simple, graphical test.
BMJ, 315(7109), 629–634. https://doi.org/10.1136/
bmj.315.7109.629.
Fernández, E. N., & Mairal, J. B. (2017). Behavioral activa-
tion versus cognitive restructuring to reduce automatic
negative thoughts in anxiety generating situations. Psi-
cothema, 29(2), 172–177.
*Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017).
Delivering cognitive behavior therapy to young adults with
symptoms of depression and anxiety using a fully auto-
mated conversational agent (Woebot): A randomized
controlled trial. JMIR Mental Health, 4(2). https://doi.
org/10.2196/mental.7785 e19.
Go, E., & Sundar, S. S. (2019). Humanizing chatbots: The
effects of visual, identity and conversational cues on
humanness perceptions. Computers in Human Behavior,
97, 304–316. https://doi.org/10.1016/j.chb.2019.01.020.
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G.,
Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., &
Debeer, H. (2011). GRADE guidelines: 1. Introduction—
GRADE evidence profiles and summary of findings tables.
345
Journal of Clinical Epidemiology, 64(4), 383–394. https://
doi.org/10.1016/j.jclinepi.2010.04.026.
Hedges, L. V., & Olkin, I. (2014). Statistical methods for
meta-analysis. Academic Press.
Heer, T., Garcia-Morchon, O., Hummen, R., Keoh, S. L.,
Kumar, S. S., & Wehrle, K. (2011). Security challenges in
the IP-based Internet of things. Wireless Personal Commu-
nications, 61(3), 527–542. https://doi.org/10.1007/s11277-
011-0385-5.
Heeren, A., Jones, P. J., & McNally, R. J. (2018). Mapping
network connectivity among symptoms of social anxiety
and comorbid depression in people with social anxiety
disorder. Journal of Affective Disorders, 228, 75–82.
https://doi.org/10.1016/j.jad.2017.12.003.
Henrichsen, P. J., & Allwood, J. (2013). Predicting the
attitude flow in dialogue based on multi-modal speech
cues. NEALT proceedings: Northern European Associa-
tion for Language and Technology [Symposium]. 4th
Nordic Symposium on Multimodal Communication,
November 15–16, Gothenburg, Sweden.
Higgins, J. P., & Green, S. (2011). Cochrane handbook for
systematic reviews of interventions (Vol. 4). Wiley.
Hofmann, S. G., Asnaani, A., Vonk, I. J., Sawyer, A. T., &
Fang, A. (2012). The efficacy of cognitive behavioral
therapy: A review of meta-analyses. Cognitive Therapy and
Research, 36(5), 427–440. https://doi.org/10.1007/s10608-
012-9476-1.
Kegel, R. H., & Wieringa, R. J. (2014). Persuasive technolo-
gies: A systematic literature review and application to pisa
[Technical Report No. TR-CTIT-14–07]. Centre for
Telematics and Information Technology, University of
Twente, Enschede.
Kiluk, B. D., Serafini, K., Frankforter, T., Nich, C., & Carroll,
K. M. (2014). Only connect: The working alliance in
computer-based cognitive behavioral therapy. Behaviour
Research and Therapy, 63, 139–146. https://doi.org/
10.1016/j.brat.2014.10.003.
Kriston, L. (2013). Dealing with clinical heterogeneity in
meta-analysis: Assumptions, methods, interpretation.
International Journal of Methods in Psychiatric Research,
22(1), 1–15. https://doi.org/10.1002/mpr.1377.
Kuo, M.-H. (2011). Opportunities and challenges of cloud
computing to improve health care services. Journal of
Medical Internet Research, 13(3). https://doi.org/10.2196/
jmir.1867 e67.
Kuyken, W., Hayes, R., Barrett, B., Byng, R., Dalgleish, T.,
Kessler, D., Lewis, G., Watkins, E., Brejcha, C., & Cardy,
J. (2015). Effectiveness and cost-effectiveness of mindful-
ness-based cognitive therapy compared with maintenance
antidepressant treatment in the prevention of depressive
relapse or recurrence (PREVENT): A randomised con-
trolled trial. The Lancet, 386(9988), 63–73. https://doi.org/
10.1016/S0140-6736(14)62222-4.
Lee, J. A., Neimeyer, G. J., & Rice, K. G. (2013). The
relationship between therapist epistemology, therapy style,
working alliance, and interventions use. American Journal
of Psychotherapy, 67(4), 323–345. https://doi.org/10.1176/
appi.psychotherapy.2013.67.4.323.
Lin, J., Lüking, M., Ebert, D. D., Buhrman, M., Andersson,
G., & Baumeister, H. (2015). Effectiveness and cost-
effectiveness of a guided and unguided Internet-based
acceptance and commitment therapy for chronic pain:
Study protocol for a three-armed randomised controlled
trial. Internet Interventions, 2(1), 7–16. https://doi.org/
10.1016/j.invent.2014.11.005.
Lin, J., Paganini, S., Sander, L., Lüking, M., Ebert, D. D.,
Buhrman, M., Andersson, G., & Baumeister, H. (2017).
346 lim et al.
An Internet-based intervention for chronic pain: A three-
arm randomized controlled study of the effectiveness of
guided and unguided acceptance and commitment therapy.
Deutsches Ärzteblatt International, 114(41), 681.
Lindhiem, O., Bennett, C. B., Rosen, D., & Silk, J. (2015).
Mobile technology boosts the effectiveness of psychother-
apy and behavioral interventions: A meta-analysis. Behav-
ior Modification, 39(6), 785–804. https://doi.org/10.1177/
0145445515595198.
Lundh, A., & Gøtzsche, P. C. (2008). Recommendations by
Cochrane review groups for assessment of the risk of bias
in studies. BMC Medical Research Methodology, 8(1), 22.
https://doi.org/10.1186/1471-2288-8-22.
Mak, W. W., Chan, A. T., Cheung, E. Y., Lin, C. L., & Ngai,
K. C. (2015). Enhancing web-based mindfulness training
for mental health promotion with the health action process
approach: Randomized controlled trial. Journal of Medical
Internet Research, 17(1). https://doi.org/10.2196/jmir.3746
e8.
Mayer, R. E., & DaPra, C. S. (2012). An embodiment effect in
computer-based learning with animated pedagogical
agents. Journal of Experimental Psychology: Applied, 18
(3), 239. https://doi.org/10.1037/a0028616.
McHugh, M. L. (2012). Interrater reliability: The kappa
statistic. Biochemia Medica, 22(3), 276–282. https://doi.
org/10.11613/BM.2012.031.
McTear, M., Callejas, Z., & Griol, D. (2016a). The conver-
sational interface (Vol. 6). Springer.
McTear, M., Callejas, Z., & Griol, D. (2016b). Creating a
conversational interface using chatbot technology. In The
conversational interface: Talking to smart devices. Springer
(pp. 125–159). Springer. https://doi.org/10.1007/978-3-
319-32967-3.
*Meyer, B., Berger, T., Caspar, F., Beevers, C., Andersson, G.,
& Weiss, M. (2009). Effectiveness of a novel integrative
online treatment for depression (Deprexis): Randomized
controlled trial. Journal of Medical Internet Research, 11
(2). https://doi.org/10.2196/jmir.1151 e15.
*Meyer, B., Bierbrodt, J., Schröder, J., Berger, T., Beevers, C.
G., Weiss, M., Jacob, G., Späth, C., Andersson, G., Lutz,
W., Hautzinger, M., Löwe, B., Rose, M., Hohagen, F.,
Caspar, F., Greiner, W., Moritz, S., & Klein, J. P. (2015).
Effects of an Internet intervention (Deprexis) on severe
depression symptoms: Randomized controlled trial. Inter-
net Interventions, 2(1), 48–59. https://doi.org/10.1016/j.
invent.2014.12.003.
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009).
Preferred reporting items for systematic reviews and meta-
analyses: The PRISMA statement. Annals of Internal
Medicine, 151(4), 264–269. https://doi.org/10.7326/0003-
4819-151-4-200908180-00135.
Moman, R. N., Dvorkin, J., Pollard, E. M., Wanderman, R.,
Murad, M. H., Warner, D. O., & Hooten, W. M. (2019).
A systematic review and meta-analysis of unguided elec-
tronic and mobile health technologies for chronic pain—is
it time to start prescribing electronic health applications?
Pain Medicine, 20(11), 2238–2255. https://doi.org/
10.1093/pm/pnz164.
Montenegro, J. L. Z., da Costa, C. A., & da Rosa Righi, R.
(2019). Survey of conversational agents in health. Expert
Systems With Applications, 129, 56–67. https://doi.org/
10.1016/j.eswa.2019.03.054.
*Moritz, S., Schilling, L., Hauschildt, M., Schroder, J., &
Treszl, A. (2012). A randomized controlled trial of Inter-
net-based therapy in depression. Behaviour Research and
Therapy, 50(7–8), 513–521. https://doi.org/10.1016/j.
brat.2012.04.006.
Morrison, L. G., Yardley, L., Powell, J., & Michie, S. (2012).
What design features are used in effective e-health inter-
ventions? A review using techniques from critical interpre-
tive synthesis. Telemedicine and e-Health, 18(2), 137–144.
https://doi.org/10.1089/tmj.2011.0062.
Newby, J. M., Twomey, C., Li, S. S. Y., & Andrews, G.
(2016). Transdiagnostic computerised cognitive beha-
vioural therapy for depression and anxiety: A systematic
review and meta-analysis. Journal of Affective Disorders,
199, 30–41. https://doi.org/10.1016/j.jad.2016.03.018.
Oh, K.-J., Lee, D., Ko, B., & Choi, H.-J. (2017). A chatbot for
psychiatric counseling in mental healthcare service based
on emotional dialogue analysis and sentence generation
[Paper presentation]. 2017 18th IEEE International Con-
ference on Mobile Data Management (MDM).
Oinas-Kukkonen, H., & Harjumaa, M. (2009). Persuasive
systems design: Key issues, process model, and system
features. Communications of the Association for Informa-
tion Systems, 24(1), 28. https://doi.org/10.17705/
1CAIS.02428.
Patel, V., Maj, M., Flisher, A. J., De Silva, M. J., Koschorke,
M., Prince, M., Zonal, W., Representatives, M. S., Tem-
pier, R., Riba, M., & Sanchez, M. (2010). Reducing the
treatment gap for mental disorders: A WPA survey. World
Psychiatry, 9(3), 169–176. https://doi.org/10.1002/j.2051-
5545.2010.tb00305.x.
Rahman, A., Al Mamun, A., & Islam, A. (2017). Program-
ming challenges of chatbot: Current and future prospective
[Paper presentation]. 2017 IEEE Region 10 Humanitarian
Technology Conference (R10-HTC).
Richardson, M., Garner, P., & Donegan, S. (2019). Interpre-
tation of subgroup analyses in systematic reviews: A
tutorial. Clinical Epidemiology and Global Health, 7(2),
192–198. https://doi.org/10.1016/j.cegh.2018.05.005.
Rosenthal, R., Cooper, H., & Hedges, L. (1994). Parametric
measures of effect size. Handbook of Research Synthesis,
621(2), 231–244.
*Sandoval, L. R., Buckey, J. C., Ainslie, R., Tombari, M.,
Stone, W., & Hegel, M. T. (2017). Randomized
controlled trial of a computerized interactive media-based
problem solving treatment for depression. Behavior
Therapy, 48(3), 413–425. https://doi.org/10.1016/j.beth.
2016.04.001.
Schnyder, N., Panczak, R., Groth, N., & Schultze-Lutter, F.
(2017). Association between mental health-related stigma
and active help-seeking: Systematic review and meta-
analysis. British Journal of Psychiatry, 210(4), 261–268.
https://doi.org/10.1192/bjp.bp.116.189464.
Scholten, M. R., Kelders, S. M., & Van Gemert-Pijnen, J. E.
(2017). Self-guided web-based interventions: Scoping
review on user needs and the potential of embodied
conversational agents to address them. Journal of Medical
Internet Research, 19(11). https://doi.org/10.2196/
jmir.7351 e383.
*Schröder, J., Brückner, K., Fischer, A., Lindenau, M., Köther,
U., Vettorazzi, E., & Moritz, S. (2014). Efficacy of a
psychological online intervention for depression in people
with epilepsy: A randomized controlled trial. Epilepsia, 55
(12), 2069–2076. https://doi.org/10.1111/epi.12833.
Schroeder, N. L., Adesope, O. O., & Gilbert, R. B. (2013).
How effective are pedagogical agents for learning? A meta-
analytic review. Journal of Educational Computing
Research, 49(1), 1–39. https://doi.org/10.2190/EC.49.1.a.
Schulz, K. F., Altman, D. G., & Moher, D. (2010).
CONSORT 2010 statement: Updated guidelines for report-
ing parallel group randomised trials. BMC Medicine, 8(1),
18. https://doi.org/10.1186/1741-7015-8-18.
 chatbot-delivered psychotherapy
Sedgwick, P. (2013). Meta-analyses: Heterogeneity and sub-
group analysis. BMJ, 346. https://doi.org/10.1136/bmj.
f4040 f4040.
Sterne, J. A., Sutton, A. J., Ioannidis, J. P., Terrin, N., Jones,
D. R., Lau, J., Carpenter, J., Rücker, G., Harbord, R. M.,
& Schmid, C. H. (2011). Recommendations for examining
and interpreting funnel plot asymmetry in meta-analyses of
randomised controlled trials. BMJ, 343. https://doi.org/
10.1136/bmj.d4002 d4002.
Tiller, J. W. (2013). Depression and anxiety. Medical Journal
of Australia, 199(6), S28–S31.
Twomey, C., O’Reilly, G., & Meyer, B. (2017). Effectiveness
of an individually-tailored computerised CBT programme
(Deprexis) for depression: A meta-analysis. Psychiatry
Research, 256, 371–377. https://doi.org/10.1016/j.
psychres.2017.06.081.
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan,
M. S., & Torous, J. B. (2019). Chatbots and conversational
agents in mental health: A review of the psychiatric
landscape. Canadian Journal of Psychiatry, 64(7),
456–464. https://doi.org/10.1177/0706743719828977.
Vitak, J. (2012). The impact of context collapse and privacy
on social network site disclosures. Journal of Broadcasting
and Electronic Media, 56(4), 451–470. https://doi.org/
10.1080/08838151.2012.732140.
Vos, T., Allen, C., Arora, M., Barber, R. M., Bhutta, Z. A.,
Brown, A., Carter, A., Casey, D. C., Charlson, F. J., &
Chen, A. Z. (2016). Global, regional, and national
incidence, prevalence, and years lived with disability for
310 diseases and injuries, 1990–2015: A systematic anal-
ysis for the Global Burden of Disease Study 2015. The
Lancet, 388(10053), 1545–1602. https://doi.org/10.1016/
S0140-6736(16)31678-6.
Wahle, F., Bollhalder, L., Kowatsch, T., & Fleisch, E. (2017).
Toward the design of evidence-based mental health infor-
mation systems for people with depression: A systematic
literature review and meta-analysis. Journal of Medical
Internet Research, 19(5). https://doi.org/10.2196/jmir.7381
e191.
Warmerdam, L., van Straten, A., Jongsma, J., Twisk, J., &
Cuijpers, P. (2010). Online cognitive behavioral therapy
347
and problem-solving therapy for depressive symptoms:
Exploring mechanisms of change. Journal of Behavior
Therapy and Experimental Psychiatry, 41(1), 64–70.
https://doi.org/10.1016/j.jbtep.2009.10.003.
Weitz, E., Kleiboer, A., Van Straten, A., & Cuijpers, P. (2018).
The effects of psychotherapy for depression on
anxiety symptoms: A meta-analysis. Psychological
Medicine, 48(13), 2140–2152. https://doi.org/10.1017/
S0033291717003622.
Whitton, A. E., Proudfoot, J., Clarke, J., Birch, M.-R., Parker,
G., Manicavasagar, V., & Hadzi-Pavlovic, D. (2015).
Breaking open the black box: Isolating the most potent
features of a web and mobile phone-based intervention for
depression, anxiety, and stress. JMIR Mental Health, 2(1).
https://doi.org/10.2196/mental.3573 e3.
World Health Organization (2017). Depression and other
common mental disorders: Global health estimates. Author.
Yan, M., Castro, P., Cheng, P., & Ishakian, V. (2016).
Building a chatbot with serverless computing [Paper
presentation]. Proceedings of the 1st International Work-
shop on Mashups of Things and APIs, Trento, Italy. https://
dl.acm.org/doi/10.1145/3007203.3007217.
Zhang, A., Franklin, C., Jing, S., Bornheimer, L. A., Hai, A.
H., Himle, J. A., Kong, D., & Ji, Q. (2018). The
effectiveness of four empirically supported psychotherapies
for primary care depression and anxiety: A systematic
review and meta-analysis. Journal of Affective Disorders,
245, 1168–1186. https://doi.org/10.1016/j.jad.2018.
12.008.
*Zwerenz, R., Becker, J., Knickenberg, R. J., Siepmann, M.,
Hagen, K., & Beutel, M. E. (2017). Online self-help as an
add-on to inpatient psychotherapy: Efficacy of a new
blended treatment approach. Psychotherapy and Psycho-
somatics, 86(6), 341–350. https://doi.org/10.1159/
000481177.
RECEIVED: April 10, 2021
ACCEPTED: September 21, 2021
AVAILABLE ONLINE: 12 OCTOBER 2021