Addictive Behaviors 122 (2021) 107027

Contents lists available at ScienceDirect

Addictive Behaviors
journal homepage: www.elsevier.com/locate/addictbeh

Evaluation of a virtual reality E-cigarette prevention game for adolescents

Veronica U. Weser a, b, *, Lindsay R. Duncan b, c, Brandon E. Sands a, b, Andrew Schartmann b, d,
Sandra Jacobo b, e, Bernard François f, Kimberly D. Hieftje a, b
a
Department of General Medicine, Yale School of Medicine, New Haven, CT, United States
b
play4REAL XR Lab at the Yale School of Medicine, New Haven, CT, United States
c
Department of Kinesiology and Physical Education, McGill University, Montreal, Quebec, Canada
d
Department of Music Theory, New England Conservatory, Boston, MA, United States
e
University of California, Berkeley, United States
f
PreviewLabs Inc, New Haven, CT, United States

A R T I C L E I N F O A B S T R A C T

Keywords: Objective: To test preliminary efficacy and acceptability of Invite Only VR: A Vaping Prevention Game (Invite Only
Serious games VR), an electronic cigarette (e-cigarette) prevention game that uses virtual reality (VR) hardware.
Intervention Methods: 285 adolescents (146 boys; ages 11–14; M = 12.45 years; SD = 0.57) were enrolled in this non-
Adolescents
equivalent control groups design through their middle school. Participants who played Invite Only VR for 1.5
E-cigarettes
Virtual reality
h were compared to treatment as usual at 4 timepoints: baseline, post gameplay, 3 and 6 months. Evidence of the
Nicotine addiction efficacy of the game to influence e-cigarette use, e-cigarette knowledge, nicotine addiction knowledge, perceived
addictiveness of e-cigarettes, perceived likelihood of using e-cigarettes, perceptions of harm, self-efficacy to
refuse, social approval of e-cigarettes, and e-cigarette social perceptions was assessed using mixed between-
within subjects ANOVAS. Intervention acceptability was indicated by measures of gameplay experience/satis­
faction, VR experience, and perceived responsibility for game-based decisions.
Results: From baseline to 6 months, Invite Only VR players improved in e-cigarette knowledge, nicotine addiction
knowledge, perceived addictiveness of e-cigarettes, perceptions of harm, and social perceptions about e-cigarette
use compared to the control group. No significant changes were observed for the other dependent variables,
including e-cigarette use. Ratings of gameplay experience and satisfaction, VR experience, and game-based de­
cisions were high.
Conclusions: Invite Only VR may help players develop knowledge about e-cigarettes and nicotine addiction, as well
as encourage the development of healthy harm and social perceptions regarding e-cigarette use that persist 6
months beyond gameplay. Adolescents’ satisfaction with the VR gameplay indicates preliminary intervention
acceptability.

1. Introduction
The popularization of electronic nicotine delivery systems (ENDS),
also known as electronic cigarettes (e-cigarettes), is a rising public
health concern. E-cigarettes are now the most commonly used nicotine-
containing products among U.S. middle- and high school students (Na­
tional Institute on Drug Abuse, 2020). The results of the 2019 Youth Risk
Behavior Survey revealed that 50.1% of teens aged 14–18 years had
used an e-cigarette at least once, and 10.7% of teens in this age group
vape e-cigarettes on 20 or more days per month (Centers for Disease
Control and Prevention, 2019). Alarmingly, 5.7% of U.S. middle school
students aged 11–13 years reported past 30-day e-cigarette use (Wang
et al., 2020). According to the CDC, 99% of the e-cigarettes sold in the U.
S. contain nicotine, and although some labels do not disclose the nico­
tine content of the product, many e-cigarettes marketed as 0% nicotine
nevertheless contain the chemical responsible for addiction (Centers for
Disease Control and Prevention, 2012). The nicotine contained in e-
cigarettes is easily absorbed by the body and, notably, the most addic­
tive form of nicotine: highly oxidizing free-base nicotine (Goel et al.,
2015). The addictive nature of e-cigarettes was highlighted in a recent
nationally representative survey in which 27% of individuals who tried
an e-cigarette went on to use the product regularly (Levy et al., 2017).

* Corresponding author at: 2 Church St. South, New Haven, CT 06510, United States.
E-mail address: veronica.weser@yale.edu (V.U. Weser).

https://doi.org/10.1016/j.addbeh.2021.107027
Received 19 March 2021; Received in revised form 18 June 2021; Accepted 19 June 2021
Available online 23 June 2021
0306-4603/© 2021 Elsevier Ltd. All rights reserved.
V.U. Weser et al.
The health risks to adolescents, who are especially susceptible to
addiction (Jordan & Andersen, 2017), are of particular concern: using
devices that contain nicotine during adolescence can impair parts of the
brain responsible for attention, learning, mood, and impulse control (US
Department of Health & Human Services, 2016). Moreover, a recent
meta-analysis found that adolescents and young adults aged 14–30 years
who had ever vaped an e-cigarette were nearly 4 times more likely to
start smoking traditional combustible cigarettes, even after controlling
for potentially confounding demographic, psychosocial, and behavioral
risk factors (Soneji et al., 2017). Given that 90% of adult smokers used
nicotine-containing products as adolescents (Kong & Krishnan-Sarin,
2017), and the adverse outcomes for adolescent e-cigarette use, the
need to reach adolescents before they engage in risky behaviors, such as
vaping, is of vital importance. Therefore, research focused on the
development and testing of theory-driven and evidence-based in­
terventions to prevent e-cigarette use among young adolescents is
imperative.
Although e-cigarettes use far exceeds traditional combustible ciga­
rette use among adolescents (National Institute on Drug Abuse, 2020),
most existing e-cigarette prevention programs are adaptations of to­
bacco control programs, and not specifically designed for e-cigarette use
prevention (O’Connor et al., 2019). Adolescents try e-cigarettes to
sample fun flavors and attempt “vaping tricks,” reasons far different
from those given for trying combustible cigarettes (Evans-Polce et al.,
2018; Pepper et al., 2017). This calls into question the extent to which
traditional tobacco use prevention initiatives can be generalized to e-
cigarettes.
There exists one school-based program that was developed and
evaluated specifically for e-cigarette prevention: CATCH My Breath
(CATCH Global Foundation, 2020). Sixteen months after completing the
program’s 4 class modules, e-cigarette use initiation at middle schools
that implemented CATCH My Breath was significantly lower than e-
cigarette use initiation at comparable schools that did not implement the
program (4.9% vs 8.9%). Students who used CATCH My Breath also had
significant improvements in e-cigarette knowledge and perceived posi­
tive outcomes for not using e-cigarettes compared to students at control
schools (Kelder et al., 2020). Despite the potential promise of this e-
cigarette prevention intervention, CATCH My Breath requires special­
ized training for the teacher facilitator, which may reduce the feasibility
of implementing this program at scale. E-cigarette interventions that can
be delivered in school settings without specialized personnel or addi­
tional teacher training requirements have the potential to reach far more
adolescents than extant programs.
One promising and scalable method of intervention delivery is
through the development of a research-informed videogame. Given that
90% of adolescents aged 13–17 years report playing videogames regu­
larly (Anderson & Jiang, 2018), gaming provides a means of delivering
health intervention content in a manner that adolescents find fun and
engaging (see Baranowski et al., 2008; Papastergiou, 2009, for reviews).
Recent systematic reviews have demonstrated the effects of serious
games on improving health behaviors in the contexts of diabetes,
asthma, physical activity, nutrition, medication adherence, eating dis­
orders, sexually transmitted diseases, management of chronic disease,
depression, and anxiety (Primack et al., 2012; DeSmet et al., 2014, 2015;
Gao et al., 2015; Lu et al., 2013; Johnson et al., 2016; Holtz et al., 2018;
Lau et al., 2017; Hieftje et al., 2013; Merino-Campos et al., 2016; Theng
et al., 2015; Barnes et al., 2018; Villani et al., 2018).
Virtual reality (VR) is an emerging videogame technology that sim­
ulates reality and real-life situations (Slater & Sanchez-Vives, 2016),
making it optimal for use in contexts as diverse as entertainment, ar­
chitecture, industrial design, business conferencing, simulation, and the
delivery of learning content. VR places an individual into a computer-
generated world, viewed through a head-mounted display. VR is
widely used in education, simulation, and training due to its potential
for simulating interactivity, motivation, and immersive properties
(Seymour, 2008; see Patle et al., 2019; Checa & Bustillo, 2020, for
2
Addictive Behaviors 122 (2021) 107027
reviews of VR in specific industries). A recent systematic review of VR in
education documented consistent evidence of significant advantages for
VR-based instruction compared with less immersive pedagogical
methods, such as desktop computer games and standard lectures with
slideshows (Hamilton et al., 2020). Taken as a whole, VR technology is a
promising vehicle for the delivery of e-cigarette prevention content.
To our knowledge, Invite Only VR: A Vaping Prevention Game (Invite
Only VR) is the only videogame intervention for e-cigarette prevention
that utilizes VR to enhance learning through immersion. It is a theory-
informed videogame intervention that teaches young adolescents
about the health risks of vaping and nicotine use while providing a
virtual environment to practice refusing peer pressure to vape. In the
current study, a non-equivalent control groups design (NEGD) was
conducted to examine 3 outcomes: 1) behavioral changes in e-cigarette
use; 2) intervention efficacy via indicators of prevention, such as in­
creases in e-cigarette knowledge and harm perceptions; and 3) gameplay
satisfaction as an index of intervention acceptability.
2. Methods
2.1. Study design
We conducted a NEGD in 3 middle schools in a Connecticut school
district between the fall of 2019 and spring of 2020. The partnership
with the school district stipulated that participants not be randomized
individually, but rather that control and intervention conditions be
assigned based on a participant’s class section to facilitate the continuity
of the regular school curriculum. Each school had 2 class sections
designated as intervention and 2 sections designated as control per
grade level (grades 7th and 8th). Dividing intervention and control
conditions by classes also had the advantage of reducing the likelihood
of contamination between conditions, as individuals who participated in
the research would primarily interact with classmates assigned to the
same condition. In total, individuals from 12 intervention class sections
and 12 control class sections participated. The Yale School of Medicine
Human Investigation Committee approved the research.
2.2. Participants
Eligible participants were adolescents, aged 11–14 years, who pro­
vided assent and parental/guardian consent, were enrolled in a partner
middle school, were willing to wear a VR headset to play Invite Only VR
in 2–3 sittings for a total of 45–120 min, and were willing to answer
questions before and after playing. All students enrolled in the middle
school district were invited to participate. At the time of the study, the
median annual household income in the district was $91,783 USD and
31% of students qualified for free or reduced-cost lunch.
Recruitment was conducted through fliers, emails, and VR demon­
strations during lunch hour. Participants were not offered compensation
for their participation. Instead, each participating school was gifted 30
Oculus Go VR headsets at the conclusion of the study.
2.3. Intervention. Invite Only VR
VR, story-based videogame intervention that teaches adolescents
aged 11 to 16 years about the health risks of using e-cigarettes while
providing a virtual environment to practice refusing peer pressure to
vape e-cigarettes. A sample script from the game appears in Appendix A.
A recent publication described in detail the development of the proto­
type of Invite Only VR, including the development of a logic model, and
the pilot evaluation of the game’s preliminary impact using a pre- post-
design with 47 adolescents aged 13–15 years (Weser et al., 2020). Invite
Only VR was designed as a seated VR experience for the Oculus Go VR
headset (Oculus, 2020). The game utilizes voice recognition software
that allows the player to use their voice to engage with virtual peers. The
game facilitates the repetitive and meaningful practice of navigating
V.U. Weser et al.
peer pressure situations involving e-cigarettes, including refusal, and
correcting e-cigarette misperceptions. See Fig. 1 for representative im­
ages from the game.
2.4. Control
The control condition was not exposed to Invite Only VR during the
study and received treatment as usual.
2.5. Measures
Demographic characteristics (e.g., age, grade, sex, ethnicity, and
race) were collected in the baseline survey. Assessments given at each
interval included questions adapted from national surveys (Youth Risk
Behavior Survey (YRBS), Kann et al., 2018; National Youth Tobacco
Survey (NYTS), 2019). To assess Outcome 1, e-cigarette use behaviors,
participants reported whether they had ever tried using e-cigarettes like
JUUL. See Table 1 for the measures used to assess Outcomes 2 and 3.
2.6. Procedure
Outcome measures were collected at baseline, immediately after
completion of gameplay for the intervention condition, approximately 1
week after baseline for the control condition, and at 3 and 6 months for
both conditions. Each assessment timepoint had a specified window
within which collection of the assessment data would be considered “on-
time” as indicated: 3-month assessments (10 days); 6-month assessments
(30 days).
All assessments were collected using the online data collection
platform Qualtrics. Research assistants were present to answer partici­
pants’ questions regarding specific items and confidentiality of re­
sponses during baseline, post-gameplay, and 3-month follow up
assessments. Participants were assured that their responses would be
completely confidential, that responses to assessment items would not
be shared with the school or with their parents. Data collection,
including gameplay, took place during participants’ regular physical
education class. Participants in the intervention condition played Invite
Only VR for 2–4 sessions for approximately 40 min per session. In­
dividuals who elected to take fewer or shorter breaks completed the
game in fewer sessions than did players who took more breaks or more
time to respond to in-game prompts. Participants were instructed to take
breaks whenever they liked, with research assistants encouraging a
break after 20 min of play, in line with Oculus Go device recommen­
dations (Oculus, 2020).
The 6-month data collection window opened in March 2020 when
Fig. 1. Screen captures from Invite Only VR. Panel A: When the virtual peer character Anthony makes the claim that vaping is safe, the player activates their power to
correct peer misperceptions and is prompted with the line shown in the figure. The player must read the prompt aloud, using their own voice to correct a
misperception about e-cigarettes. Panel B. An example of the gameplay elements used to convey information about e-cigarette and nicotine addiction. The buttons
shown are dialog options the player can select to learn more about the two available topics. Facts gleaned from this interaction become useable as rebuttals to be
spoken out loud when correcting peer misperceptions in subsequent parts of the game.
3
Addictive Behaviors 122 (2021) 107027
the U.S. declared the COVID-19 pandemic a national emergency,
prompting school closures. Thus, 6-month follow-up data were collected
remotely through a distance learning platform via emails containing
links to the 6-month assessment on Qualtrics.
2.7. Statistical analysis
The data were screened to detect missing values and outliers, and to
test for conformity with the assumptions of parametric statistical tests.
To assess changes in the dependent variables across the 4 timepoints, we
conducted mixed between-within subjects ANOVAs. Partial eta-squared
values were used to determine the effect sizes of the multivariate tests
and were interpreted using common guidelines (0.01 = small; 0.06 =
moderate; 0.14 = large effect) (Cohen, 1988, p. 284–287).
3. Results
A total of 287 participants (146 male; 8th grade: n = 135; 7th grade:
n = 152; mean age 12.45 years; SD = 0.57) provided signed parent or
guardian consent and were then assigned to either the intervention
videogame condition (n = 155) or to the control condition (n = 132)
based on their class section. See Fig. 2 for the CONSORT flow diagram
that describes the number of students assigned to each condition across
all study timepoints (adapted from Boutron et al., 2017). Table 2 depicts
the sample demographics at baseline. Overall, the study had a high
retention rate with 93% of students completing assessments immedi­
ately after gameplay, 92% at 3 months, and 70% at 6 months (Fig. 2
includes the reason participants were lost to follow-up at each
timepoint).
3.1. Outcome measures
Across all measures, no significant differences were found based on
participant sex or grade level. (see table 3)
3.1.1. Outcome 1: Behaviors
At baseline, very few participants reported having ever used e-cig­
arettes or JUUL (n = 9; 3%), traditional combustible cigarettes (n = 3;
1%), alcohol (n = 16; 5.5%), cigars (n = 2; <1%), hookah (n = 1; <1%),
or other tobacco products (n = 2; <1%). No participants reported having
ever tried smoking or vaping marijuana. Most participants (n = 207;
72%) reported that they had no friends who vape. Among participants
who reported having friends who vape, 48 (60%) indicated that “a few
of them” vape, 3 (3.75%) indicated that “half of them” vape, and 2
(2.5%) indicated that “most of them” vape, with the remainder (n = 27;
V.U. Weser et al. Addictive Behaviors 122 (2021) 107027

Table 1 experimental or control groups, for all participants, perceived likelihood

Measures. of using e-cigarettes remained low (M = 1.18 on a scale ranging from 1
Number Response options Sample item [not at all likely] to 4 [very likely]), perceived social approval of e-
of items cigarette remained low (M = 1.5 on a scale ranging from 1 [strongly
Outcome 2: disagree] to 4 [strongly agree]), and self-efficacy to refuse e-cigarettes
Indicators of
remained high (M = 3.80 on a scale ranging from 1 [not at all sure] to 4
Prevention
E-cigarette 15 True, False, Not “The vapor from e- [definitely sure]) across the study period. Descriptive statistics for the
knowledge surea cigarettes is just water, dependent variables for intervention and control participants at all study
which is safe to inhale” timepoints are presented in Table 4.
Nicotine addiction 3 True, False, Not “Addiction to nicotine
knowledge surea as a teen can rewire
3.1.3. Outcome 3: Post-gameplay experience assessments
your brain to become
easily addicted to Of the 149 participants in the intervention condition, 79.6% finished
other drugs” the game (Mean playtime = 1 h and 40 min; SD = 33 min). On average,
Perceptions of e- 2 Likert scale: 1 (very “Once someone has intervention participants completed 93.6% (SD = 17%) of the game.
cigarette likely or very easy) started vaping or
Among participants in the experimental condition, scores on a scale
addictiveness to 5 (not at all likely JUULing, do you think
or very difficult) it would be difficult for ranging from 1 (strongly agree) to 4 (strongly disagree) were high for
them to quit?” gameplay experience (M = 3.08, SD = 0.59) and satisfaction with the
Perceived 1 Likert scale: 1 (not at “Do you think you will virtual reality experience (M = 2.91, SD = 0.51) and perceived re­
likelihood of e- all likely) to 4 (very vape or use an e- sponsibility for game-based decisions (M = 3.19 out of 4.0, SD = 0.68).
cigarette use likely) cigarette, like JUUL, in
Descriptive statistics for the individual gameplay experience variables
the next year?”
Harm perceptions 5 Likert scale: 1 (no “How much do you are presented in Table 4.
harm) to 4 (a lot of think people harm
harm) themselves when they 4. Discussion
breathe in other
people’s e-cigarette or
JUUL vapor?” Invite Only VR is the only VR videogame intervention for adolescent
Refusal self- 6 Likert scale: 1 (not at “How sure are you that e-cigarette use prevention. It is a theory-informed videogame interven­
efficacy all sure) to 4 you can refuse if a tion that provides a virtual environment where players can practice
(definitely sure) friend offers you a hit refusing peer pressure to vape while learning about the health risks of e-
off a JUUL?”
Social approval 4 Likert scale: 1 “My friends approve of
cigarette use (Weser et al., 2020).
(strongly disagree) teens my age vaping or Regarding the first outcome, the results indicate no difference in e-
to 4 (strongly agree) JUULing” cigarette use behaviors between the control group and those who played
Social perceptions 4 Likert scale: 1 “I think people who Invite Only VR at the 6-month follow-up timepoint. The goal of Invite
(strongly disagree) vape or JUUL have
Only VR is to prevent e-cigarette use by adolescents and the lack of e-
to 4 (strongly agree) more friends” (reverse
coded item) cigarette use in either group indicates we were successful in enrolling a
sample from our target audience: adolescents who in general had not yet
*Outcome 3:
Intervention
experienced much exposure to e-cigarettes. It is possible that if we had
Acceptability followed the participants beyond 6 months, group differences might
Gameplay 7 Likert scale: 1 “I enjoyed playing the have emerged. We hypothesize that Invite Only VR would have a pre­
experience and (strongly disagree) game” ventive effect such that over a longer period, rates of e-cigarette use
satisfaction to 4 (strongly agree)
would increase among control participants and remain lower among
VR experience 12 Likert scale: 1 “In Invite Only VR, I
(strongly disagree) had a sense of “being participants from the intervention group. It is also possible that because
to 4 (strongly agree) there” that was similar of the COVID-19 pandemic and subsequent lockdown, participants in
to my normal both groups had reduced access to e-cigarettes since contact with friends
experience of being in and opportunities for e-cigarette purchase may have been limited by
a place”
social distancing restrictions.
Perceived 2 Likert scale: 1 “I felt responsible for
responsibility for (strongly disagree) the choices I made in As for the second outcome, playing Invite Only VR significantly
game-based to 4 (strongly agree) the game” increased key indicators of successful prevention and healthy behavior
decisions maintenance: e-cigarette knowledge, nicotine addiction knowledge,
a
Responses of “not sure” were always scored as incorrect. perceived addictiveness of e-cigarettes, perceptions of e-cigarette harm,
and social perceptions related to e-cigarette use. This finding is prom­
33.75%) specifying that they have “1 or 2 friends who vape.” ising because high rates of e-cigarette misperceptions, including that e-
cigarettes are less harmful and addictive than combustible cigarettes,
3.1.2. Outcome 2: Indicators of prevention are often cited as major contributors to high e-cigarette use rates among
Mixed between-within subjects ANOVAS indicated that from base­ adolescents (Gorukanti et al., 2017; Amrock, Lee, & Weitzman, 2016;
line to 6 months, compared to the control group, participants who Cooper et al., 2016). Therefore, increasing knowledge about e-ciga­
played Invite Only VR had greater improvements in general e-cigarette rettes, such as the fact that they contain nicotine and can lead to
knowledge [Wilk’s λ = 0.658, F(3, 171) = 29.68, p < .001, η2p = 0.342 addiction, may help individuals decide to avoid e-cigarette initiation.
(large)] (Fig. 3), nicotine addiction knowledge [Wilk’s λ = 0.865, F(3, Likewise, acknowledging the potential for harm can also influence
180) = 9.389, p < .001, η2p = 0.135 (moderate)] (Fig. 4 Panel A), health behaviors. Both factors provide the motivation to avoid e-ciga­
perceived addictiveness of e-cigarettes [Wilk’s λ = 0.955, F(3, 178) = rette use and are important steps towards the development of perceived
2.775, p = .043, η2p = 0.045 (small)] (Fig. 4 Panel B), perceptions of e- behavioral control, which contributes to avoiding e-cigarette use (Ajzen,
cigarette harm [Wilk’s λ = 0.914, F(3, 178) = 5.61, p = .001, η2p = 0.086 1991). Finally, lasting changes to social perceptions about e-cigarettes
(moderate)] (Fig. 4 Panel C), and social perceptions about e-cigarette mean that adolescents may be less likely to engage in e-cigarette use
use [Wilk’s λ = 0.973, F(3, 175) = 1.65, p = .028, η2p = 0.027 (small)] behaviors to satisfy social pressures. Given the game is focused on the
(Fig. 4 Panel D). Though no significant changes were observed for social pressures of e-cigarette use and providing opportunities to engage
virtual peers in conversations about the risks of e-cigarettes and

4
V.U. Weser et al.
Fig. 2. CONSORT Flow Diagram depicting participant enrolment and retention across all study timepoints. Adapted from Boutron et al. (2017).
subsequent addiction, these outcomes signal successful accomplishment
of the intervention’s primary goals.
This study’s third and final outcome was to examine gameplay
experience and satisfaction data to index preliminary intervention
acceptability. This approach is justified since an enjoyable intervention
is more likely to be impactful because participants are motivated to play
to completion (Dishman et al., 2005; Ryan et al., 2006). Indeed, nearly
80% of participants finished Invite Only VR, and gameplay experience
ratings were high, indicating engagement and enjoyment. Moreover, the
low dropout rate during the intervention is a testament to the comfort of
the VR experience and suggests that no participants found the
5
Addictive Behaviors 122 (2021) 107027
experience so uncomfortable that they needed to withdraw from the
study.
Altogether, these findings indicate that the intervention was not only
successful in changing e-cigarette knowledge, nicotine addiction
knowledge, perceived addictiveness of e-cigarettes, harm perceptions,
and social perceptions, but that the game was also enjoyable to play.
Participants reported feeling responsible for their decisions in the game,
many of which profoundly impacted the virtual characters. We believe
the immersive game experience may have contributed to the success of
the intervention. As a result, we plan to conduct larger follow-up studies
to investigate how VR experiences and perceived responsibility for
V.U. Weser et al. Addictive Behaviors 122 (2021) 107027

Table 2 Table 3
Demographics collected at baseline. Descriptive statistics for dependent variables for intervention and control groups
Demographics
across 4 study timepoints.
Baseline Post 3 Months 6 Months
Male Female Prefer to Total
identify as M (SD) M (SD) M (SD) M (SD)
something else
n n= n=2 N= E-cigarette knowledge
=146 130 279 Intervention 11.29 13.97 13.60 13.39
(1.91) (1.27) (1.29) (1.96)
Grade Control 10.33 10.45 11.17 11.02
8th 73 60 1 (<1%) 134 (2.57) (2.92) (2.88) (2.78)
(26%) (21%) (48%) Nicotine addiction
7th 73 71 1 (<1%) 145 knowledge
(26%) (26%) (52%) Intervention 2.37 2.80 2.70 2.64
Age (0.58) (0.47) (0.46) (0.60)
11 2 3 (1%) 0 5 (2%) Control 2.17 2.11 2.14 2.25
(>1%) (0.82) (0.88) (0.85) (0.85)
12 68 80 1 (<1%) 149 Perceived addictiveness of
(24%) (29%) (53%) e-cigarettes
13 71 46 1 (<1%) 118 Intervention 4.11 4.17 4.20 4.27
(26%) (17%) (42%) (0.52) (0.63) (0.54) (0.50)
14 5 (2%) 2 (1%) 0 7 (3%) Control 3.89 3.84 3.79 3.76
Race (0.69) (0.72) (0.72) (0.70)
White 91 90 1 (<1%) 181 Perceived likelihood of e-
(33%) (33%) (65%) cigarette use
Black or African 10 7 (3%) 0 17 Intervention 1.12 1.08 1.18 1.11
American (4%) (6%) (0.44) (0.27) (0.41) (0.32)
Asian 19 16 0 35 Control 1.25 1.26 1.28 1.21
(7%) (6%) (13%) (0.55) (0.51) (0.56) (0.44)
American Indian 4 (1%) 2 0 6 (2%) Harm perceptions
or Alaskan (<1%) Intervention 2.56 2.69 2.65 2.64
Native (0.29) (0.24) (0.19) (0.20)
Native Hawaiian 0 1 0 1 Control 2.57 2.56 2.57 2.55
or other Pacific (<1%) (<1%) (0.30) (0.35) (0.30) (0.23)
Islander Refusal self-efficacy
Multi-Racial 8 (3%) 9 (3%) 1 (<1%) 18 Intervention 3.87 3.89 3.90 3.90
(7%) (0.31) (0.37) (0.24) (0.23)
Prefer to identify 14 7 (3%) 0 21 Control 3.73 3.72 3.77 3.83
as a different (5%) (8%) (0.62) (0.62) (0.53) (0.44)
racial group Social approval
Ethnicity Intervention 1.49 1.41 1.45 1.45
Hispanic or 24 19 0 43 (0.41) (0.44) (0.42) (0.42)
Latino (9%) (7%) (16%) Control 1.49 1.50 1.59 1.58
(0.50) (0.46) (0.57) (0.56)
Note. percentages reflect percent of total sample.
Social perceptions
Intervention 3.38 3.37 3.36 3.48
game-based decisions affect long term health behavior outcomes for (0.46) (0.49) (0.50) (0.44)
Control 3.21 3.24 3.28 3.28
players. Indeed, this line of inquiry aligns with previous research sug­
(0.54) (0.54) (0.55) (0.58)
gesting increased levels of immersive content stimulate multisensory
engagement and ultimately lead to more effective learning outcomes in
VR experiences compared to traditional learning contexts (Webster, perceptions of social approval of e-cigarettes. However, all these factors
2016). reflected relatively healthy attitudes towards avoiding e-cigarettes at
baseline for both intervention and control groups, and they remained
healthy across all follow-up timepoints. The failure to detect changes in
4.1. Limitations
these important outcome variables can be addressed by conducting a
larger effectiveness trial with a more racially and ethnically diverse
Because the collaboration with the school district precluded the use
population that would follow participants from middle school to high
of random assignment, our study cannot rule out confounding variables
school, a time when adolescents are more likely to experiment with e-
such as preexisting differences between the class periods that were
cigarette use. A larger effectiveness trial could also investigate the
designated control vs. intervention. Moreover, national data suggests
connections between VR experience, such as feelings of presence in the
that 10.5% of middle schoolers have used an e-cigarette, only 5% of our
virtual environment, and primary intervention outcomes. This is a
participants reported ever having tried e-cigarettes. The study’s findings
largely untapped area of research in health behavior and intervention
may be a result of participants being told they would be playing a game
science that merits future investigation as VR becomes an increasingly
about the dangers of e-cigarettes, which may have discouraged current
popular method of delivering educational and health-related content.
e-cigarette users from participating or admitting to vaping. Although we
took several measures to remind the participants about the confidenti­
5. Conclusion
ality of their responses, self-reported measures about substance use are
often affected by social desirability concerns, which can lead to under
The findings from this preliminary efficacy and effectiveness study
reported prevalence rates, particularly in adolescent samples (Fan et al.,
are promising. From baseline to 6 months, participants who played Invite
2006). Given the small number of self-reported e-cigarette use in our
Only VR had greater improvements in possible mediators associated
sample, we were unable to detect changes in e-cigarette use behavior in
with behavior change compared to the control group. Participants also
the 6 months following the intervention.
enjoyed playing the intervention, with the majority of participants
Additionally, players did not show changes in their perceived like­
playing the game to completion. These findings suggest that participants
lihood of e-cigarette use, self-efficacy to refuse e-cigarettes, or their

6
V.U. Weser et al. Addictive Behaviors 122 (2021) 107027

could use the VR game to gain valuable practice refusing e-cigarettes

and correcting peer e-cigarette misperceptions, skills that may readily
translate to their everyday lives. Our results indicate the need for further
investigation of the potential of this intervention with a larger sample.
Overall, this study suggests that the use of Invite Only VR as an enjoyable
intervention approach has the potential to reach many adolescent
players and help them to develop knowledge about and motivations to
avoid the use of e-cigarettes.

6. Role of Funding sources

Funding for this study was provided by Oculus. Oculus had no role in
the study design, collection, analysis or interpretation of the data,
writing the manuscript, or the decision to submit the paper for
publication.

CRediT authorship contribution statement

Veronica U. Weser: Conceptualization, Methodology, Investigation,

Fig. 3. Change in e-cigarette knowledge for experimental vs control groups
from baseline to 6-month follow-up. Scores range from 0 to 15 based on the sum
of correct responses. Points represent means and error bars depict stan­
dard deviation.
Data curation, Writing - original draft, Visualization, Project adminis­
tration. Lindsay R. Duncan: Formal analysis, Data curation, Writing -
review & editing, Visualization. Brandon E. Sands: Conceptualization,
Methodology, Investigation, Project administration. Andrew Schart­
mann: Software. Sandra Jacobo: Software. Bernard François: Soft­
ware. Kimberly D. Hieftje: Conceptualization, Methodology,
Resources, Writing - review & editing, Supervision, Funding acquisition,
Project administration.

Fig. 4. Changes in secondary outcomes from baseline to 6-month follow up assessment for intervention vs. control condition. Points represent means and error bars
depict standard deviation. Panel A: Knowledge of nicotine addictiveness. Scores ranged from 0 to 3 based on the sum of correct response. Panel B: Perceptions of e-
cigarette addictiveness. Items measured on a Likert scale ranging from 1 (very likely or very easy) to 5 (not at all likely or very difficult). Panel C: Perceptions of E-
cigarette Harm. Items measured on a Likert scale ranging from 1 (no harm) to 4 (a lot of harm). Panel D. Social perceptions of e-cigarette use. Items measured on a
Likert scale ranging from 1 (strongly disagree) to 4 (strongly agree).

7
V.U. Weser et al.
Table 4
Descriptive statistics for individual gameplay experience items.
Item
Gameplay Experience and Satisfaction
I would play this game again
I was frustrated with this game*
I enjoyed playing the game
I liked the way the game looked
I felt connected to the other characters in the game
I felt connected to my character in the game
I would tell my friends to play this game
VR Experience
I liked using my own voice in the game
The sounds I heard in the game helped me to feel like I was visiting
a real place
My experience in the virtual environment seemed like my real-
world experience
In Invite Only VR, I had a sense of “being there” that was similar to
my normal experience of being in a place
I felt that the virtual world was all around me
I felt like I was just seeing pictures or a movie, rather than visiting a
place*
I felt like I was acting in the virtual space, rather than using a
device
I felt present in the virtual space
I was not aware of my real environment
I was completely captivated by the virtual world
In order to use the controller, I had to keep looking at it while I was
playing the game*
Using my own voice made me feel more like part of the game
Perceived Responsibility for Game-Based Decisions
I would make decisions in life like I made in the game
I felt responsible for the choices I made in the game
Note. Items assessed with Likert scale ranging from 1 (strongly disagree) to 4
(strongly agree). Starred items were reverse coded in the analysis.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Acknowledgements
This project would not have been possible without the support of
Oculus, our ongoing partnership with game prototype developer, Pre­
viewLabs, Inc, and the support of the Milford Public School District. We
would also like to thank all the adolescents, teachers, and focus group
participants who contributed to the development of this project.
Appendix A. Supplementary data
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.addbeh.2021.107027.
Addictive Behaviors 122 (2021) 107027
References
Ajzen, I. (1991). The theory of planned behavior. Organizational behavior and human
M (SD)
decision processes, 50(2), 179-211.
Amrock, S. M., Lee, L., & Weitzman, M. (2016). Pediatrics, 138(5).
3.26 Anderson, M., & Jiang, J. (2018). Teens, Social Media & Technology 2018. Retrieved
(0.74) October 07, 2020, from https://www.pewresearch.org/internet/2018/05/31/teens-
social-media-technology-2018/.
1.77
Baranowski, T., Buday, R., Thompson, D. I., & Baranowski, J. (2008). Playing for real:
(0.77)
video games and stories for health-related behavior change. American Journal of
3.40 Preventive Medicine, 34(1), 74–82.
(0.71) Barnes, S., & Prescott, J. (2018). JMIR serious games, 6(1). https://doi.org/10.2196/
3.09 games.9530.
(0.79) Boutron, I., Altman, D. G., Moher, D., Schulz, K. F., & Ravaud, P. (2017). Annals of
2.59 internal medicine, 167(1), 40–47.
(0.81) CATCH Global Foundation. (2020). CATCH My Breath a nicotine vaping prevention
2.78 program. Retrieved from https://www.catch.org/.
(0.82) Checa, D., & Bustillo, A. (2020). Multimedia Tools and Applications, 79(9), 5501–5527.
3.15 Centers for Disease Control and Prevention. (2019). Youth Risk Behavior Survey Data
(0.78) (YRBS). Available at: www.cdc.gov/yrbs. Accessed on October 07, 2020.
Centers for Disease Control and Prevention. (2019). National Youth Tobacco Survey
(NYTS). CDC website. https://www.cdc.gov/tobacco/data_statistics/surveys/nyts/
2.65
index.htm. 2019.
(0.92)
Centers for Disease Control and Prevention. (2012). Preventing tobacco use among youth
3.11
and young adults: a report of the surgeon general.
(0.72) Cohen, J. W. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).
2.94 Hillsdale, NJ: Lawrence Erlbaum Associates.
(0.69) Cooper, M., Harrell, M. B., Pérez, A., Delk, J., & Perry, C. L. (2016). Tobacco regulatory
3.01 science, 2(3), 278–289.
(0.75) DeSmet, A., Shegog, R., Van Ryckeghem, D., Crombez, G., & De Bourdeaudhuij, I. (2015).
3.21 Games for health journal, 4(2), 78–90. https://doi.org/10.1089/g4h.2014.0110.
(0.73) DeSmet, A., Van Ryckeghem, D., Compernolle, S., Baranowski, T., Thompson, D.,
2.24 Crombez, G., Poels, K., Van Lippevelde, W., Bastiaensens, S., Van Cleemput, K.,
(0.84) Vandebosch, H., & De Bourdeaudhuij, I. (2014). A meta-analysis of serious digital
2.94 games for healthy lifestyle promotion. Preventive medicine, 69, 95–107. 10.1016/j.
ypmed.2014.08.026.
(0.73)
Dishman, R. K., Motl, R. W., Saunders, R., Felton, G., Ward, D. S., Dowda, M., &
3.05
Pate, R. R. (2005). Medicine and Science in Sports and Exercise, 37(3), 478–487.
(0.70)
Evans-Polce, R. J., Patrick, M. E., Lanza, S. T., Miech, R. A., O’Malley, P. M., &
2.53 Johnston, L. D. (2018). Reasons for vaping among US 12th graders. Journal of
(0.84) Adolescent Health, 62(4), 457–462.
2.77 Fan, X., Miller, B. C., Park, K. E., Winward, B. W., Christensen, M., Grotevant, H. D., &
(0.80) Tai, R. H. (2006). Field methods, 18(3), 223–244.
1.83 Gao, Z., Chen, S., Pasco, D., & Pope, Z. (2015). A meta-analysis of active video games on
(0.77) health outcomes among children and adolescents. Obesity reviews: an official
2.75 journal of the International Association for the Study of Obesity, 16(9), 783–794.
(0.89) 10.1111/obr.12287.
Goel, R., Durand, E., Trushin, N., Prokopczyk, B., Foulds, J., Elias, R. J., & Richie, J. P., Jr
3.22 (2015). Chemical research in toxicology, 28(9), 1675–1677.
(0.75) Gorukanti, A., Delucchi, K., Ling, P., Fisher-Travis, R., & Halpern-Felsher, B. (2017).
Preventive medicine, 94, 65–71.
3.14
Hamilton, D., McKechnie, J., Edgerton, E., & Wilson, C. (2020). Journal of Computers in
(0.80)
Education, 1–32.
Hieftje, K., Edelman, E. J., Camenga, D. R., & Fiellin, L. E. (2013). JAMA pediatrics, 167
(6), 574–580.
Holtz, B. E., Murray, K., & Park, T. (2018). Games for health journal, 7(5), 291–301.
https://doi.org/10.1089/g4h.2018.0024.
Johnson, D., Deterding, S., Kuhn, K. A., Staneva, A., Stoyanov, S., & Hides, L. (2016).
Internet interventions, 6, 89–106. https://doi.org/10.1016/j.invent.2016.10.002.
Jordan, C. J., & Andersen, S. L. (2017). Developmental cognitive neuroscience, 25, 29–44.
Kann, L., McManus, T., Harris, W. A., Shanklin, S. L., Flint, K. H., Queen, B., & Lim, C.
(2018). MMWR Surveillance Summaries, 67(8), 1.
Kelder, S. H., Mantey, D. S., Van Dusen, D., Case, K., Haas, A., & Springer, A. E. (2020).
Public Health Reports, 135(2), 220–229.
Kong, G., & Krishnan-Sarin, S. (2017). A call to end the epidemic of adolescent E-
cigarette use. Drug and alcohol dependence, 174, 215.
Lau, H. M., Smit, J. H., Fleming, T. M., & Riper, H. (2017). Frontiers in psychiatry, 7, 209.
https://doi.org/10.3389/fpsyt.2016.00209.
Levy, D. T., Yuan, Z., & Li, Y. (2017). The prevalence and characteristics of e-cigarette
users in the US. International journal of environmental research and public health,
14(10), 1200.
Merino-Campos, Carlos; Del Castillo Fernande-, Héctor. (2016). The Benefits of Active
Video Games for Educational and Physical Activity Approaches: A Systematic
Review. Journal of New Approaches in Educational Research, 5(2), 115-122.
Lu, A. S., Kharrazi, H., Gharghabi, F., & Thompson, D. (2013). Games for health journal, 2
(3), 131–141. https://doi.org/10.1089/g4h.2013.0025.
National Institute on Drug Abuse. (2020). What are electronic cigarettes? Retrieved
October 07, 2020, from https://www.drugabuse.gov/publications/research-reports/
tobacco-nicotine-e-cigarettes/what-are-electronic-cigarettes.
O’Connor, S., Pelletier, H., Bayoumy, D., & Schwartz, R. (2019). Interventions to Prevent
Harms from Vaping. Ontario Tobacco Research Unit.
Oculus (2020). Oculus VR | Oculus Go – all-in-on headset made for entertainment.
[online] Oculus VR. Available at: https://www.oculus.com/go.
Papastergiou, M. (2009). Exploring the potential of computer and video games for health
and physical education: A literature review. Computers & Education, 53(3), 603–622.
Patle, D. S., Manca, D., Nazir, S., & Sharma, S. (2019). Virtual Reality, 23(3), 293–311.
8
V.U. Weser et al.
Pepper, J. K., Lee, Y. O., Watson, K. A., Kim, A. E., Nonnemaker, J. M., & Farrelly, M. C.
(2017). Journal of Adolescent Health, 61(5), 599–605.
Primack, B. A., Carroll, M. V., McNamara, M., Klem, M. L., King, B., Rich, M.,
Chan, C. W., & Nayak, S. (2012). American journal of preventive medicine, 42(6),
630–638. https://doi.org/10.1016/j.amepre.2012.02.023.
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). Motivation and Emotion, 30(4),
344–360.
Seymour, N. E. (2008). World journal of surgery, 32(2), 182–188.
Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual
reality. Frontiers in Robotics and AI, 3, 74.
Soneji, S., Barrington-Trimis, J. L., Wills, T. A., et al. (2017). Association Between Initial
Use of e-Cigarettes and Subsequent Cigarette Smoking Among Adolescents and
Addictive Behaviors 122 (2021) 107027
Young Adults: A Systematic Review and Meta-analysis. JAMA Pediatr, 171(8),
788–797. https://doi.org/10.1001/jamapediatrics.2017.1488.
Theng, Y. L., Lee, J. W., Patinadan, P. V., & Foo, S. S. (2015). Games for Health Journal, 4
(5), 352–361.
US Department of Health & Human Services. (2016) E-cigarette use among youth and
young adults: a report of the surgeon general.
Villani, D., Carissoli, C., Triberti, S., Marchetti, A., Gilli, G., & Riva, G. (2018). Games for
health journal, 7(2), 85–99. https://doi.org/10.1089/g4h.2017.0108.
Wang, T. W., Neff, L. J., Park-Lee, E., Ren, C., Cullen, K. A., & King, B. A. (2020).
Morbidity and Mortality Weekly Report, 69(37), 1310.
Webster, R. (2016). Interactive Learning Environments, 24(6), 1319–1333.
Weser, V. U., Duncan, L. R., Pendergrass, T. M., Fernandes, C. S., Fiellin, L. E., &
Hieftje, K. D. (2020). Addictive Behaviors, 106639.