International Journal of Human–Computer Interaction

ISSN: 1044-7318 (Print) 1532-7590 (Online) Journal homepage: https://www.tandfonline.com/loi/hihc20

Understanding Continuance Intention

toward Crowdsourcing Games: A Longitudinal
Investigation

Xiaohui Wang, Dion Hoe-Lian Goh & Ee-Peng Lim

To cite this article: Xiaohui Wang, Dion Hoe-Lian Goh & Ee-Peng Lim (2020): Understanding
Continuance Intention toward Crowdsourcing Games: A Longitudinal Investigation, International
Journal of Human–Computer Interaction, DOI: 10.1080/10447318.2020.1724010

To link to this article: https://doi.org/10.1080/10447318.2020.1724010

Published online: 12 Feb 2020.

Submit your article to this journal

Article views: 21

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=hihc20
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION
https://doi.org/10.1080/10447318.2020.1724010

SURVEY ARTICLE

Understanding Continuance Intention toward Crowdsourcing Games: A

Longitudinal Investigation
a
Xiaohui Wang , Dion Hoe-Lian Gohb, and Ee-Peng Limc
a
Department of Journalism, Hong Kong Baptist University, Hong Kong; bWee Kim Wee School of Communication and Information, Nanyang
Technological University, Singapore; cSchool of Information Systems, Singapore Management University, Singapore

ABSTRACT
Given the increasing popularity of gamified crowdsourcing, the study reported here involved examining
determinants of users' continuance intention toward crowdsourcing games, both with longitudinal data
and reference to a revised unified theory of acceptance and use of technology (UTAUT). At three time
points, data were collected from an online survey about playing crowdsourcing games. Time-lagged
regression, cross-temporal correlation, and structural equation modeling were performed to examine
determinants of the acceptance of crowdsourcing games. Results indicate that the revised UTAUT2 is
applicable to explaining the acceptance of crowdsourcing games. Not only did effort expectancy,
hedonic motivation, and social influence directly affect users’ continuance intention toward crowdsour-
cing games, but time-based variations also emerged in users’ perceptions and acceptance of the games
and in how their perceptions affect their acceptance. The findings answer the call for a context-specific
acceptance model and the identification of factors of adopting gamification.

In recent years, videos games have experienced consistently
accelerated growth (Entertainment Software Association,
2017). Part of the reason why games are popular could be that
video gaming brings together innovative and creative technolo-
gies to create engaging, immersive, and even breathtaking
experiences (Kahn & Williams, 2015). As a consequence,
researchers have placed a significant amount of effort into study-
ing gamification – that is, the use of game-like experiences to
motivate individuals in less engaging environments (Deterding,
Dixon, Khaled, & Nacke, 2011; Huotari & Hamari, 2017; Wang,
Goh, Lim, Vu, & Chua, 2017). A major area in which gamifica-
tion has been used is crowdsourcing, which many organizations
have employed in outsourced tasks (Morschheuser, Hamari,
Koivisto, & Maedche, 2017). In a similar way, gamification
could be an effective channel to mobilize large numbers of
volunteers, since users of gamification are motivated not by
altruism or financial incentives but by their intrinsic desire to
be entertained (Law & Von Ahn, 2011).
In research responding to the increasing popularity of
gamification, Von Ahn and Dabbish (2008) have posited
that the most important aspect of gamified crowdsourcing
systems is that outputs are produced in enjoyable ways.
More recently, Sørensen et al. (2016) have discussed the
possibility of applying crowdsourcing and gamification to
solve optimization problems in quantum physics. In that
study, examining the online video game Quantum Moves,
the authors showed that even in quantum physics, players’
motivation can lead to new scientific insights. Unsurprisingly,
most studies on gamification have reported the positive
results of their implementations (Morschheuser et al., 2017).
Despite the popularity of gamification, few studies have been
conducted to explore the dynamic behaviors of users of gamified
crowdsourcing systems. In response, the study presented here
contributes to knowledge about the continuance of gamification.
The chief contributions of the study are twofold. First, we inves-
tigate the direct and indirect determinants of behavioral inten-
tion to use integrated models, particularly by following the
framework of the extended unified theory of acceptance and
use of technology (UTAUT2; Venkatesh, Thong, & Xu, 2012)
and the extended the acceptance of technology model (TAM;
Davis, Bagozzi, & Warshaw, 1992). Second, we explore the
dynamic nature of those determinants and their effects on con-
tinuance intention – that is, to an individual’s decision to con-
tinue using new technology beyond its first usage (Bhattacherjee,
2001; Sun & Jeyaraj, 2013). In studying users’ behavioral inten-
tions toward using a technology, it is important to examine the
phenomenon over an extended period instead of within a mere
cross-sectional snapshot (Sonderegger, Zbinden, Uebelbacher, &
Sauer, 2012). When confronting new technologies for the first
time, users make decisions about their acceptance of those
technologies that systematically differ from their subsequent
decisions (Venkatesh, Morris, & Ackerman, 2000). In that
light, the goal of the study was to examine how predictors of
acceptance influence users’ continued use of gamified crowd-
sourcing systems over time.
In what follows, the next section of the paper introduces
the concepts of gamification, crowdsourcing, and the accep-
tance of technology, followed by an overview of two models of
such acceptance, as well as their extensions into the field of
gamification. After the Methods section presents the

CONTACT Xiaohui Wang vincentwx@hkbu.edu.hk Department of Journalism,Hong Kong Baptist University, Kowloon Tong, Hong Kong.
© 2020 Taylor & Francis Group, LLC
2 X. WANG ET AL.
instruments developed for the study and the procedures of
data collection used, the Results section presents the findings
of data analyses. Major findings are elaborated upon in the
Discussion section, after which the paper concludes by
addressing the implications of the findings and the limitations
of the study.
1. Research background
1.1. Crowdsourcing games
Gamification refers to the process of enhancing a service with
affordances for gameful experiences (Deterding et al., 2015).
The approach aims at evoking users’ intrinsic motivations by
designing systems reminiscent of games in order to support
overall value creation for users (Hamari & Koivisto, 2015).
Principles and features of games can be used to attract, moti-
vate, and engage users, which not only reduces perceived
barriers to using systems, including low incentive and poor
rates of adoption (Baptista & Oliveira, 2017), but also trans-
forms game-based interactions into desirable outputs
(Zichermann & Linder, 2010). Gamification continues to
experience rapid growth and has spawned numerous applica-
tions for health, business, education, and sustainability as well
as persuasive technology (Deterding et al., 2015).
Meanwhile, crowdsourcing refers to the use of contribu-
tions, monetary or otherwise, from internet users in order to
offer desirable services or ideas (Howe, 2006). The primary
goal of crowdsourcing is either to reduce costs by distributing
the work required by immense tasks to a crowd or to solve
complex problems with crowd intelligence. Since crowdsour-
cing generally relies on the participation of large numbers of
volunteers, gamification is an effective way to attract and
engage potential users. By transforming crowdsourcing tasks
into game-like activities, gamification provides users with
motives other than monetary compensation (Morschheuser
et al., 2017). Developed for crowdsourcing tasks such as
multimedia annotation, geolocation tagging, natural-
language processing, scientific puzzle solving, and knowledge
creation, gamification systems include Google Image Labeler
for labeling images (Google Image Labeler, 2007), the online
video game Foldit for identifying well-folded protein struc-
tures (Cooper et al., 2010), the online game OnToGalaxy for
mining semantic relations (Carranza & Krause, 2012), and the
aforementioned Quantum Moves for solving optimization
problems in quantum physics (Sørensen et al., 2016), to
name a few.
Parallel to the increasing popularity of gamification in
crowdsourcing, academic research on the topic has steadily
increased and can be expected to continue increasing in the
future (Morschheuser et al., 2017). To date, studies examining
gamification mainly focused on three aspects: (1) the effective-
ness of individual gamification affordance, such as points and
leaderboard (Prestopnik & Tang, 2015; Talasila, Curtmola, &
Borcea, 2016), storytelling and avatar (Prandi, Nisi, Salomoni,
& Nunes, 2015; Sakamoto & Nakajima, 2014), and competitive
or cooperative gaming design (Morschheuser et al., 2017; Pe-
Than, Goh, & Lee, 2017); (2) the efficiency of gamified crowd-
sourcing tasks in terms of participation (Eickhoff, Harris, de
Vries, & Srinivasan, 2012; Kawajiri, Shimosaka, & Kahima,
2014), output engagement (Goncalves, Hosio, Ferreira, &
Kostakos, 2014), and psychological outcomes (e.g., enjoyment,
Altmeyer, Lessel, & Krüger, 2016); and (3) the motivations of
using gamified crowdsourcing, such as social influence
(Hamari & Koivisto, 2015), intrinsic motivations (Mekler,
Brühlmann, Tuch, & Opwis, 2017), and esthetic experience
(Wang, Goh, Lim, & Vu, 2016; Wang et al., 2017). Such results
provide empirical evidence for gamification developments,
providing designers with guidelines in ways of encouraging
and engaging users in the crowdsourcing tasks.
Largely missing from the literature, however, are investiga-
tions into the individuals’ long-term interactions with the
gamified systems (Rapp, Hopfgartner, Hamari, Linehan, &
Cena, 2019). Furthermore, very little empirical work has yet
systematically examined the antecedents of gamification
acceptance based on nuanced theories (Morschheuser et al.,
2017). Seaborn and Fels (2015) noticed that a major issue of
gamification studies was the disconnection between theoreti-
cal and practical work. Gamification studies would benefit
from a wider use of theories to account for the complexity
of human behavior (Rapp et al., 2019). In this light, this study
investigates to examine the predictors of individuals’ accep-
tance and continuance of crowdsourcing games based on an
integrated theoretical framework.
1.2. UTAUT2 and extended TAM
Despite the success stories of implementing gamification,
most developed systems struggle with increasing their user
base (Deterding, 2015; Doan, Ramakrishnan, & Halevy, 2011).
Indeed, many innovations do not survive initial deployment
due to having too few installations (Siu & Riedl, 2016), and
only when significant numbers of users participate in the
crowdsourcing process can large-scale problems be solved.
However, such problems could be largely overcome by clar-
ifying the driving factors behind users’ intention to use such
systems.
Proposed by Venkatesh, Morris, Davis, and Davis (2003), the
unified theory of acceptance and use of technology (UTAUT)
was synthesized using eight prominent theories from psychology
and sociology in order to elucidate why people intend to use
information systems. UTAUT proposed four predictors of beha-
vioral intentions toward and the use of technology: performance
expectancy, effort expectancy, social influence, and facilitating
conditions. Whereas performance expectancy is defined as the
degree to which using a technology will provide individuals with
performance-related benefits, effect expectancy is the degree of
ease in using a technology. By contrast, social influence refers to
the extent to which users perceive that their important ones
think that they should use a technology. Last, facilitating condi-
tions are individuals’ perceptions of the necessary resources and
support available to perform certain behaviors (Venkatesh et al.,
2003).
More recently, Venkatesh et al. (2012) also proposed
UTAUT2 by extending UTAUT with three additional con-
structs – hedonic motivation, price value, and habit – and
thereby tailoring the theory to the context of individual
consumers’ acceptance of technology. In particular, hedonic

motivation, also called perceived enjoyment, is defined as the
fun or pleasure experienced by using a technology, and
UTAUT2 has indeed been examined in hedonic contexts
(Baptista & Oliveira, 2017; Chang, Liu, & Chen, 2014; Oh
& Yoon, 2014). For instance, Baptista and Oliveira (2017)
utilized UTAUT2 to explain the acceptance of a gamified
banking service and found that all of the proposed factors
except facilitating conditions directly affected behavioral
intentions.
A limitation of UTAUT2, however, is the underdevelop-
ment of secondary factors of acceptance. Although perfor-
mance expectancy, effect expectancy, social influence, and
hedonic motivation are prominent predictors of behavioral
intentions, UTAUT2 does not address the inter-relationships
among their antecedents. By contrast, the extended TAM
model specifies the hierarchical relationships among the ante-
cedents of behavioral intentions. Originally proposed to iden-
tify determinants of the use of information systems in the
workplace (Davis, 1989), TAM hypothesizes that initial per-
ceived ease of use, which is conceptually similar to effort
expectancy in UTAUT, and perceived usefulness, which is
conceptually similar to performance expectancy (Martins,
Oliveira, & Popovič, 2014), affect behavioral intentions by
way of attitude. To accommodate TAM in contexts of gami-
fication, researchers have sought to extend the model with
hedonic constructs. For instance, Shin and Shin (2011) have
proposed that perceived enjoyment, which is conceptually
similar to hedonic motivation, affect behavioral intentions to
play games in social networks via attitude.
In the light of the interactions among TAM’s constructs,
we propose a revised UTAUT that integrates the hierarchical
relationships in the extended TAM, as shown in Figure 1.
Specifically, we propose that attitude, social influence, and
hedonic motivation form the first layer of antecedents of
behavioral intentions, whereas performance expectancy, effort
expectancy, and hedonic motivation form a second layer of
antecedents mediated by attitude. We excluded facilitating
conditions from our model since it has been characterized
as a nonsignificant predictor of behavioral intention in recent
studies (Hoque & Sorwar, 2017; Martins et al., 2014).
1.3. Longitudinal investigation
An overlooked consideration in the evaluation of the accep-
tance of technology is the temporal nature of the beliefs,
Hedonic Motivation
Effort Expectancy
Performance
Expectancy
Social Influence
Figure 1. The integrated model of gamification acceptance.
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 3
intentions, and, in turn, behaviors related to using technology
(Sonderegger et al., 2012). Most research examining users’
behavioral intentions has focused on one-time snapshots
instead of whether players’ perceptions and intention change
over time. In cross-sectional studies of users, participants are
typically unfamiliar with the system being evaluated, and the
results of such work provide information about the initial
phase of human–computer interaction. By contrast, a long-
itudinal study involving multiple testing sessions is expected
to represent a more valid modeling of actual usage. It is also
suggested that, in the earliest stages of the introduction of new
technology, users make decisions about their acceptance of
the technology, which have been shown to systematically
differ from their continuance-related decisions as their experi-
ence with the technology accumulates (Venkatesh et al.,
2000). In short, Therefore, to gain a thorough understanding
of the underlying mechanisms of the continued acceptance of
technology, we investigated the role of perceptions in users’
initial acceptance-related decisions as well as their continu-
ance-related ones.
Hanus and Fox (2015) have conducted one of the few
studies investigating the temporal nature of gaming experi-
ences. Assessing longitudinal data representing students’ per-
ceptions of gamification in a classroom setting, they found
that users’ satisfaction with the gamified curriculum was sig-
nificantly less in the later stages of using it than during the
initial encounter, although significant differences in other
perceptions over time did not emerge. Added to their find-
ings, longitudinal studies on information technology may also
clarify the continued intention to use gamification systems.
For instance, Venkatesh and Davis (2000) tested the extended
TAM in four organizational systems with longitudinal data
and measured their model’s constructs at three different
points in time over a period of four months. They found
that perceived usefulness was the most stable construct across
the three time points, whereas the stability of the perceived
ease of use was systematically lower. In another longitudinal
study, Luse, Mennecke, and Triplett (2013) examined changes
in users’ intention to play virtual-world technology over time,
with participants recruited to play a game and finish a survey
three times in two weeks. Their results demonstrated that
a user’s intention to utilize virtual-world technologies in his
or her future work decreased significantly over time. Most
recently, Mou, Shin, and Cohen (2017) tested the extended
TAM in the online health-information context with data
collected in two phases during a five-week period. Among
Attitude Behavioral Intention
4 X. WANG ET AL.
their findings, perceived ease of use became less important to
the formation of positive behavioral intentions in the second
phase.
2. Methodology
2.1. Game development
To execute the study, a mobile crowdsourcing game called
KpopRally was developed as a question-and-answer app
designed to collect tags for music videos. Because annotating
music videos with tags – that is, brief, text-based tokens
describing certain aspects of a video – enables the retrieval
of music videos, having appropriate tags associated with each
music video was expected to allow more accurate search
results and recommendations (Dulačka & Bieliková, 2012).
The collected tags could also function as metadata for
machine learning geared toward automatic annotation in the
future (Michelucci & Dickinson, 2016).
The mechanism of the crowdsourcing system is simple;
users contribute tags by answering questions related to the
music videos. During the crowdsourcing session, a music
video clip played along with the question displayed at the
bottom of the screen. Users were required to choose an
answer from a list of options, and as incentives, they earned
points as well as reputation within the system. Points for
questions were awarded based on the percentage of other
users who agreed with the chosen answer. In that way, the
answer with the highest percentage of agreement was deemed
the correct answer for the corresponding question and saved
as metadata.
The main activity of KpopRally is that users earn points in
the task session. Game elements with regards to points, lea-
derboard, achievements, challenge, and social element are
embedded in this system. The usage scenario of KpopRally is
as follows. First, users are required to login with Facebook
account or register with a unique username. The first page of
the system present users with a usage tutorial, user profile,
and the navigation to the task session (Figure 2a). Users can
choose between two play modes, solo play and competing
with others. The tasks in the different modes are the same,
that is, five rounds of watching a music video and answering
(a) Mainpage (b) gameplay
Figure 2. Screenshots of the crowdsourcing game.
questions (Figure 2b). At the end of the task session, the total
score achieved is presented to users as feedback for their
progress. The difference between the two modes is that users
who choose the competition mode will be matched with
another user – a competitor – and gain extra points if they
beat the competitor. Those who choose the solo mode gain
points based on answers to the questions only. After the task
session, scores are accumulated and updated on users’ profile.
Scores are also associated with users’ rankings in the leader-
board (Figure 2c) and achievement (Figure 2d) in the system.
Through game elements such as gaining points and ranking in
leaderboard, collecting achievements, challenging others, and
social interactions, users of KpopRally may gain enjoyment
during the gameplay.
2.2. Data collection
In our online survey study, participants were recruited via
e-mail invitation, poster advertisements, and oral invita-
tions. Criteria for inclusion were having an Android smart-
phone with the screen size larger than 3.5 inches and
agreeing to install the app on that smartphone. Participants
were asked to register themselves on the study’s website
(Figure 3) and required to test the app and complete
a questionnaire three times: at first-time use, a week later,
and two weeks after the first use. The approach was consis-
tent with the procedures used in previous longitudinal stu-
dies (Luse et al., 2013; Vigo & Harper, 2017). During the
study, participants were first briefed about the concept and
purpose of the crowdsourcing game, the procedures of the
study, and the required tasks. Next, they were instructed to
download and install the app on their smartphones and
required to test the app according to a usage scenario.
After completing the tasks with the app, participants were
directed to the online questionnaire.
Upon finishing the first evaluation, participants were
encouraged to continue using the app for at least two more
weeks and reminded that they needed to test the app and
complete the questionnaire at two other time points: one week
and two weeks after the initial use. Via e-mail and in-app
notifications, participants were notified to complete the eva-
luations at the corresponding time points, and ones who did
(c) leaderboard (d) achievements

Figure 3. Website of the online study.
not complete the survey at the required time point received
a follow-up notification. If participants did not complete the
questionnaires within 10 days after the first use and within
17 days after the second use, then their data were considered
to be invalid. At the end of the study, promised incentives
were given to participants as a reward for their participation.
2.3. Measurements
The questionnaire survey (see Appendix I for questionnaire
items) was designed to gather individuals’ perceptions and
acceptance of crowdsourcing. The constructs measured in
the study were behavioral intention, attitude, hedonic motiva-
tion, effort expectancy, performance expectancy, and social
influence. The operationalization of those constructs was
obtained from measures reported in previous research on
UTAUT2 and TAM (Davis, Bagozzi, & Warshaw, 1989;
Venkatesh et al., 2012), and items corresponding to the con-
structs were rated on a scale of 1 (strongly disagree) to 7
(strongly agree). The questionnaire also included two open-
ended items to collect participants’ reflections on the system:
“What do you like about the system and its features?” and
“What could be improved in this system?” A pilot test was
conducted to ensure the feasibility of the study design, as well
as the content validity and reliability of the questionnaire. Six
participants invited to review the study procedures provided
feedback that guided modifications of the wording of items.
To ensure that the items appropriately measured the con-
structs, factor analysis was also conducted to test the reliabil-
ity and validity of the scale.
3. Results
3.1. Sample and demographic features
The survey was administered to 110 participants at the first-time
evaluation of the game (T1), to 82 participants at the second stage
of evaluation (T2), and to 72 participants at the third stage (T3).
Attrition rates at T2 and T3 were 25.5% and 12.2%, respectively,
which were considered to be acceptable for longitudinal studies
(Gustavson, von Soest, Karevold, & Røysamb, 2012). Data with
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 5
missing values exceeding 5% were excluded from analysis, and the
multiple imputation technique was used to manage all other
missing values in the data. Of the valid respondents in T1, 44
were male and 66 were female, with ages ranging from 20 to 40
and an average of 23.21 years. The majority of the participants
watched music videos online (60%) and played mobile games
(61%) frequently (once a week or more).
3.2. Psychometric properties of measures
Confirmatory factor analysis was conducted to test the relia-
bility and validity of the measurement. Because data were
gathered at three time points, three confirmatory factor ana-
lyses corresponding to those time points were conducted.
Specifically, convergent validity, discriminant validity, and
composite reliability (CR) tests were employed to evaluate the
psychometric properties of the measurements. Consistent with
the benchmark recommended by Fornell and Larcker (1981),
all factor loadings of items on their corresponding construct
exceeded the acceptable value of 0.4, all CR values exceeded the
cutoff value of 0.7, and the average variance extracted was
greater than the recommended 0.5 threshold. Such results
suggest the good convergent validity and reliability of the
measurement. The correlation matrix in Table 4 presents the
discriminant validity of the measurement, the results for which
show that inter-construct correlations were less than the var-
iances extracted by the constructs (i.e., the values on the diag-
onal). Overall, it can be concluded that the measurement was
reliable and valid at all three points in time.
3.3. Explaining continuance intention
The study’s design avoided a major limitation associated with self-
reported data: common-method variance, which results from
measuring all concepts with a single questionnaire. Specifically,
the study involved examining the relationship between percep-
tions measured at one time point and self-reported acceptance
measured at the subsequent time point, such that the measure-
ment of intention and its determinants were separated by one
week. To the extent that measurements were taken at different
6 X. WANG ET AL.

Table 1. TAM regression results predicting continuance intention. 3.5. Summary of results pooled across periods
Attitude Continuance Intention
A pooled data analysis was performed to examine the revised
Time of Measurement R2 β R2 β
UTAUT in the context of smartphones by following the
First period (T1~ T2, N = 82) 0.33 0.39
Hedonic Motivation 0.54*** 0.32** method of Venkatesh and Davis (2000). Specifically, data
Effort expectancy 0.02 0.12 were pooled across periods, which yielded a sample of 264
Performance Expectancy 0.10 0.18
Social Influence 0.29** questionnaires used for estimating the proposed research
Second period (T2~ T3, N = 72) 0.68 0.56 model. Structural equation modeling (SEM) was conducted
Hedonic Motivation 0.55*** 0.44** to test the hypotheses; as shown by the actual and recom-
Effort expectancy 0.38*** 0.21*
Performance Expectancy 0.10 0.04 mended values of the model fit indices in Table 3, all the
Social Influence 0.25* recommended indices were better than the recommended
*p < 0.05, **p < 0.01 ***p < 0.001 thresholds (Schreiber, Nora, Stage, Barlow, & King, 2006).
Such results demonstrated a good fit between the model and
times in different places, they were able to reduce systematic the data. As shown by the results of SEM in Figure 4, all the
covariation and thus minimize common method variance proposed relationships were supported except for perfor-
(Podsakoff, MacKenzie, Lee, & Podsakoff, 2003). mance expectancy. Effort expectancy and hedonic motivation
Table 1 shows the effects of hedonic motivation, perfor- had a significant effect on individuals’ attitudes toward the
mance expectancy, effort expectancy, and social influence on crowdsourcing game, although no significant relationship
continuance intention. In particular, hedonic motivation was emerged between performance expectancy and attitude. At
a strong determinant of attitude and intention to play at T2 the same time, attitude, hedonic motivation, and social influ-
and T3, whereas social influence was a strong determinant of ence directly affected behavioral intention.
intention to play at both times. Effort expectancy was not
a significant predictor at T2 but became a primary signifi-
4. Discussion
cant determinant of attitude and intention at T3. However,
performance expectancy remained consistently nonsignifi- The study was conducted to examine constructs of UTAUT in
cant across all time points. The regression models explained gamified crowdsourcing and the varying influences of differ-
between 33% and 68% of the variance in attitude and ent factors over time. The model was empirically validated
between 39% and 56% of the variance in behavioral using data gathered at three points in time from an online
intention. study evaluating a mobile crowdsourcing game. To answer the
research questions, time-lagged regression, cross-temporal
correlation, and SEM were performed, the results of which
3.4. Cross-temporal correlations
are discussed in what follows.
The cross-temporal correlations of acceptance-related con- First, an important contribution to the literature concerns
structs were also examined to better understand the inher- the temporal dynamics of the determinants of users’ accep-
ent changes of each construct. That is, the analysis involved tance of technology. The findings indicate time-based changes
examining the stability of users’ perceptions and acceptance in terms of users’ perceptions and acceptance, as well as the
of the technology at different time points, particularly by effect of their perceptions on acceptance. Results of the cross-
investigating the within-construct correlations measured at
different points in time (Venkatesh & Davis, 2000). Most of Table 3. The recommended and actual values of model fit indices.
the constructs showed strong, stable correlations across
Fit index χ2(df) χ2/df CFI TLI RMSEA SRMR
time, except for perceived output quality, correlations for
Recommended value <3 > .95 > .95 < .08 < .05
which were lower than the others (Table 2). Interestingly, Model fits 336.80 (157) 2.22 .96 .95 .07 .05
correlations in the later period (range: 0.58–0.81) were χ2/df is the ratio between the Chi-square and the degrees of freedom; CFI is the
systematically higher than correlations in the earlier period Comparative Fit Index; TLI is the Tucker–Lewis coefficient; RMSEA is the Root
(range: 0.47–0.67), which indicates that users’ perceptions Mean Square Error of Approximation; SRMR is the Standardized Root Mean
Square Residual.
during later usage were more stable than ones from initial
use. As shown by the results of cross-temporal correlation
analysis, perceptions and acceptance among users varied Table 4. Inter-correlations between constructs (validity and reliability).
across different points in time representing different stages The UTAUT2 M SD CR AVE 1 2 3 4 5 6
of use. 1.Continuance 4.25 1.36 0.93 0.81 0.90
Intention
2.Effort 5.23 1.13 0.84 0.64 0.68 0.80
Table 2. Cross-temporal correlations. expectancy
3.Performance 4.83 1.12 0.89 0.72 0.64 0.78 0.85
Constructs T1~ T2 T2~ T3 expectancy
Hedonic Motivation 0.67*** 0.81*** 4.Social 4.46 1.41 0.95 0.87 0.69 0.61 0.61 0.93
Effort expectancy 0.60*** 0.68*** influence
Performance Expectancy 0.55*** 0.80*** 5.Hedonic 4.44 1.41 0.96 0.89 0.79 0.72 0.65 0.58 0.94
Social Influence 0.47*** 0.58*** motivation
Attitude 0.56*** 0.73*** 6.Attitude 4.46 1.32 0.94 0.83 0.86 0.76 0.68 0.61 0.82 0.91
Continuance Intention 0.66*** 0.75*** M = Mean; SD = Standard Deviation; CR = Composite Reliability; AVE = Average
***p < 0.001 Variance Extracted

Hedonic Motivation
.60***
.31***
Effect Expectancy
Performance
Expectancy
Social Influence
The Integrated Model of Gamification Acceptance
Figure 4. The proposed integtated model. R2(attitude) = .79; R2(intention) = .80.
temporal correlations show that the within-construct correla-
tions among users’ perceptions and acceptance in the first
stage were lower than those in the second stage, which sug-
gests that their perceptions and acceptance in later periods
differed starkly from that at the first use of the technology.
Such results are consistent with the notion found in prior
studies that individuals’ perceptions and decisions about
acceptance vary according to the time of use (Hassenzahl &
Tractinsky, 2006; Sun & Jeyaraj, 2013; Venkatesh et al., 2000).
Possibly, users’ perceptions and decisions about acceptance
depend upon their familiarity with the system and would
change as they gain more experience (Sonderegger et al.,
2012). The study thus confirms a limitation of cross-
sectional studies, in which participants are typically unfamiliar
with the system being evaluated, for results that capture only
the first phase of human–computer interaction but not the
temporal aspects of users’ perceptions and acceptance. In
response, it is important to trace users’ experiences and beha-
viors over time, as measured in the study presented here.
Second, regression analysis revealed that users’ hedonic
motivation and perceptions of social influence significantly
affected their acceptance at all points in time, which under-
scores the important role of enjoyable experiences and social
elements in the continuous use of a technology. That finding
confirms the hypotheses that hedonic and social factors are
strong determinants of the sound acceptance of gamification
(e.g., Gerow, Ayyagari, Thatcher, & Roth, 2013; Hamari &
Koivisto, 2015; Morschheuser et al., 2017). Put differently, the
use of crowdsourcing games is primarily motivated by the
individual’s desire to be entertained. Accordingly, it is valu-
able to account for how engaging the experience is as well as
social elements in designing crowdsourcing games.
Third, determinants of intention to play the crowdsourcing
game in the second stage also included effort expectancy,
which was not a significant factor in the first stage. That result
shows that as users gained more hands-on experience with the
app, they relied more on the ease of use in forming their
attitudes and intentions to play the game. A possible explana-
tion could be that initially using the system is part of a routine
of trial and practice, in which error and failure can be
expected and thus tolerated by novices (Mendoza & Novick,
2005). However, issues with usability become problems as
users develop from novices into experienced users and, in
turn, judge the system based on their experience
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 7
.17*
.60***
Attitude Continuance Intention
.22***
(Sonderegger et al., 2012). That finding emphasizes the evolu-
tionary nature of the belief of usability.
Fourth, performance expectancy of the app did not signifi-
cantly relate to intention to play. It seems that players paid
more attention to their subjective experiences than the sys-
tem’s practicality, which contrasts the major finding in the
literature that usability significantly affects the acceptance of
technology (e.g., Morschheuser et al., 2017; Sun & Jeyaraj,
2013). Such results suggest that crowdsourcing games differ
from utilitarian systems, although they could also reflect that
the game used in the study was framed as a game. After all,
participants were mainly motivated by a desire to be enter-
tained instead of concerns with the efficiency of crowdsour-
cing, which de-emphasizes the usefulness of the system.
Overall, the results offer considerable support for the proposi-
tion that predictors of acceptance differ in different contexts.
Last, the revised UTAUT found support from the results of
the study, which suggest that UTAUT can be generalized to
the context of crowdsourcing games. The revised UTAUT
provides a detailed account of the key factors underlying
players’ acceptance of crowdsourcing games, and the research
model explained up to 80% of the variance in continuous
intention. Therein, hedonic motivation had the dominant
total effect on behavioral intention among all predictors, and
the hierarchical relationships between the antecedents were
examined.
5. Conclusion
5.1. Implications
The results of the study have several important implications
for future research on crowdsourcing games. First, we validate
UTAUT in the context of gamification on smartphone apps.
Although the acceptance of technology has received consider-
able attention in different types of computer systems, they
have seldom been investigated in the context of gamification.
The findings of the study thus indicate a new context for the
application of UTAUT2. Second, the results imply that indi-
viduals are subject to different influences at their respective
times of first using a system and deciding to continue to use
it. Individuals are subject to hedonic and social factors during
early stages of use, and hedonic, social, and usability-related
factors during later stages. Subjective and social feelings are
8 X. WANG ET AL.
always the focus of players’ experiences, whereas usability
plays an important role only in later stages of acceptance. In
future research on crowdsourcing games, acceptance at dif-
ference stages should thus be predicted with different factors.
Third, the study poses methodological implications for
designing future studies on gamification, notably by suggest-
ing the need for longitudinal approaches.
For practitioners, the findings of the study first suggest that
game developers should emphasize an engaging experience,
which exerts a sustained impact on individuals’ acceptance of
crowdsourcing games. Qualitative feedback from participants
may provide recommendations for system development.
When asked about their favorite aspects of the system, parti-
cipants responded with comments such as “Imaginative,”
“Relaxing and makes me feel at ease,” and “Appears to be
interactive, I enjoy using this system.” The results provide
empirical evidence of how casual funs, such as the inspired
curiosity, relaxation, and capacity to kill time with less invol-
vement (Kultima, 2009; Lazzaro, 2008), facilitate gamification
acceptance. Future development of gamification systems
could therefore address these elements. Alternatively, the
study also points out that underemphasizing usability, as
many studies have, is not advisable. Results of regression
analysis show that whereas usability might not affect intention
at the initial use of a technology, it plays an important role in
attitudes and intentions during later use. As users continue to
use the system, usability-related issues become important and
may hinder the system’s acceptance. We attribute that
dynamic to the known role of direct hands-on experience in
forming such beliefs (Venkatesh & Davis, 1996). Designing
crowdsourcing games with short learning curves and intuitive
interfaces is thus important, since it can reduce users’ cogni-
tive efforts in their continued use of a technology.
5.2. Limitations and future work
The findings of the study should be interpreted in light of their
limitations. First, the research was conducted on one game
employed on a single smartphone app, which may limit the
generalizability of the findings. Second, the study involved col-
lecting data over a two-week period, which might have reflected
only the initial stage of the implementation of crowdsourcing
games in reality. By contrast, future research could involve
collecting data during a longer time span in order to gain
greater insights into the later stages of game implementation.
Third, the sample for the third wave of data collection included
only 72 participants, which might jeopardize the power of the
statistical tests. Moreover, some nonsignificant findings from
regression analyses might have been due to the insufficient
power of the data. Future studies with larger sample sizes and
more diverse groups of users could increase the significance of
the results. At the same time, findings from those analyses were
consistent with the results of previous studies, which reduced
the likelihood of spurious findings.
Last, future research should seek to further examine the
dynamic attributes of other theoretical constructs beyond
UTAUT, including those of the theory of planned behavior
(Lee, 2009), motivation theory (Ryan, Rigby, & Przybylski,
2006), and theory about use and gratification (Sherry, Lucas,
Greenberg, & Lachlan, 2006). Future research could be devel-
oped to investigate whether constructs in other theoretical
models show similar variety over time.
ORCID
Xiaohui Wang http://orcid.org/0000-0003-2126-1712
References
Altmeyer, M., Lessel, P., & Krüger, A. (2016). Expense control:
A gamified, semi-automated, crowd-based approach for receipt cap-
turing. In Proceedings of the 21st International Conference on
Intelligent User Interfaces (pp. 31–42). New York, USA: ACM Press.
Baptista, G., & Oliveira, T. (2017). Why so serious? Gamification impact
in the acceptance of mobile banking services. Internet Research, 27(1),
118–139. doi:10.1108/IntR-10-2015-0295
Bhattacherjee, A. (2001). Understanding information systems continu-
ance: An expectation-confirmation model. MIS Quarterly, 25,
351–370. doi:10.2307/3250921
Carranza, J., & Krause, M. (2012). Evaluation of game designs for human
computation. Paper presented at the Eighth Artificial Intelligence and
Interactive Digital Entertainment Conference. doi:10.1094/PDIS-11-
11-0999-PDN
Chang, I.-C., Liu, -C.-C., & Chen, K. (2014). The effects of hedonic/
utilitarian expectations and social influence on continuance intention
to play online games. Internet Research, 24(1), 21–45. doi:10.1108/
IntR-02-2012-0025
Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., …
Players, F. (2010). Predicting protein structures with a multiplayer
online game. Nature, 466(7307), 756–760. doi:10.1038/nature09304
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user
acceptance of information technology. MIS Quarterly, 16, 319–340.
doi:10.2307/249008
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of
computer technology: A comparison of two theoretical models.
Management Science, 35(8), 982–1003. doi:10.1287/mnsc.35.8.982
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1992). Extrinsic and
intrinsic motivation to use computers in the workplace. Journal of
Applied Social Psychology, 22(14), 1111–1132. doi:10.1111/
jasp.1992.22.issue-14
Deterding, S. (2015). The lens of intrinsic skill atoms: A method for
gameful design. Human–Computer Interaction, 30(3–4), 294–335.
doi:10.1080/07370024.2014.993471
Deterding, S., Canossa, A., Harteveld, C., Cooper, S., Nacke, L. E., &
Whitson, J. R. (2015). Gamifying research: Strategies, opportunities,
challenges, ethics. In Proceedings of the 33rd Annual ACM Conference
Extended Abstracts on Human Factors in Computing Systems, Seoul,
Republic of Korea (pp. 2421–2424).
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design
elements to gamefulness: Defining “gamification”. In Proceedings of the
15th International Academic MindTrek Conference: Envisioning Future
Media Environments, Tampere, Finland (pp. 9–15).
Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing
systems on the world-wide web. Communications of the ACM, 54(4),
86–96. doi:10.1145/1924421
Dulačka, P., & Bieliková, M. (2012). Validation of music metadata via
game with a purpose. In Proceedings of the 8th International
Conference on Semantic Systems, Graz, Austria (pp. 177–180).
doi:10.1177/1753193411427832
Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P. (2012).
Quality through flow and immersion. In Proceedings of the 35th
International ACM SIGIR Conference (pp. 871–880). Portland,
Oregon, USA: ACM. doi:10.1177/1753193412441761

Entertainment Software Association. (2017). Essential facts about the
computer and video game industry. Retrieved from http://www.
theesa.com/about-esa/essential-facts-computer-video-game-industry/
Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation
models with unobservable variables and measurement error. Journal
of Marketing Research, 18(1), 39–50. doi:10.1177/002224378101800104
Gerow, J. E., Ayyagari, R., Thatcher, J. B., & Roth, P. L. (2013). Can we
have fun@ work? The role of intrinsic motivation for utilitarian
systems. European Journal of Information Systems, 22, 360–380.
doi:10.1057/ejis.2012.25
Goncalves, J., Hosio, S., Ferreira, D., & Kostakos, V. (2014). Game of
words: Tagging places through crowdsourcing on public displays. In
Proceedings of the 2014 Conference on Designing Interactive Systems
(pp. 705–714). Vancouver, BC, Canada: ACM.
Google Image Labeler. (2007). First report of alternaria dianthicola
causing leaf blight on Withania somnifera from India. Plant Disease.
91. Available at http://images.google.com/imagelabeler/.
Gustavson, K., von Soest, T., Karevold, E., & Røysamb, E. (2012).
Attrition and generalizability in longitudinal studies: Findings from
a 15-year population-based study and a Monte Carlo simulation
study. BMC Public Health, 12(1), 918. doi: 10.1186/1471-2458-12-918
Hamari, J., & Koivisto, J. (2015). Why do people use gamification
services? International Journal of Information Management, 35(4),
419–431. doi:10.1016/j.ijinfomgt.2015.04.006
Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the
classroom: A longitudinal study on intrinsic motivation, social com-
parison, satisfaction, effort, and academic performance. Computers &
Education, 80, 152–161. doi:10.1016/j.compedu.2014.08.019
Hassenzahl, M., & Tractinsky, N. (2006). User experience - a research
agenda. Behaviour & Information Technology, 25, 91–97. doi:10.1080/
01449290500330331
Hoque, R., & Sorwar, G. (2017). Understanding factors influencing the
adoption of mHealth by the elderly: An extension of the UTAUT
model. International Journal of Medical Informatics, 101, 75–84.
doi:10.1016/j.ijmedinf.2017.02.002
Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), 1–4.
Huotari, K., & Hamari, J. (2017). A definition for gamification:
Anchoring gamification in the service marketing literature.
Electronic Markets, 27(1), 21–31. doi:10.1007/s12525-015-0212-z
Kahn, A. S., & Williams, D. (2015). We’re all in this (game) together.
Communication Research, 43(4), 487–517. doi:10.1177/0093650215617504
Kawajiri, R., Shimosaka, M., & Kahima, H. (2014). Steered crowdsensing:
Incentive design towards quality-oriented place-centric crowdsensing.
In Proceedings of the 2014 ACM International Joint Conference on
Pervasive and Ubiquitous Computing (pp. 691–701). Seattle,
Washington, USA: ACM. doi:10.1094/PDIS-08-13-0831-PDN
Kultima, A. (2009). Casual game design values. In Proceedings of the 13th
International MindTrek Conference: Everyday Life in the Ubiquitous
Era, Tampere, Finland (pp. 58–65). doi:10.1177/1753193408096015
Law, E., & Von Ahn, L. (2011). Human computation. Synthesis Lectures
on Artificial Intelligence and Machine Learning, 5(3), 1–121.
doi:10.2200/S00371ED1V01Y201107AIM013
Lazzaro, N. (2008). The four fun keys. In K. Isbister & N. Schaffer (Eds.),
Game usability: Advancing the player experience (pp. 315–344). Boca
Raton, FL: CRC press.
Lee, M.-C. (2009). Understanding the behavioural intention to play online
games: An extension of the theory of planned behaviour. Online
Information Review, 33, 849–872. doi:10.1108/14684520911001873
Luse, A., Mennecke, B., & Triplett, J. (2013). The changing nature of user
attitudes toward virtual world technology: A longitudinal study.
Computers in Human Behavior, 29(3), 1122–1132. doi: 10.1016/j.
chb.2012.10.004
Martins, C., Oliveira, T., & Popovič, A. (2014). Understanding the Internet
banking adoption: A unified theory of acceptance and use of technology
and perceived risk application. International Journal of Information
Management, 34(1), 1–13. doi:10.1016/j.ijinfomgt.2013.06.002
Mekler, E. D., Brühlmann, F., Tuch, A. N., & Opwis, K. (2017). Towards
understanding the effects of individual gamification elements on
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 9
intrinsic motivation and performance. Computers in Human
Behavior, 71, 525–534. doi:10.1016/j.chb.2015.08.048
Mendoza, V., & Novick, D. G. (2005). Usability over time. In Proceedings
of the 23rd Annual International Conference on Design of
Communication: Documenting & Designing for Pervasive Information
(pp. 151–158). Coventry, UK: ACM.
Michelucci, P., & Dickinson, J. L. (2016). The power of crowds. Science,
351(6268), 32–33. doi:10.1126/science.aad6499
Morschheuser, B., Hamari, J., Koivisto, J., & Maedche, A. (2017).
Gamified crowdsourcing: Conceptualization, literature review, and
future agenda. International Journal of Human-Computer Studies,
106, 26–43. doi:10.1016/j.ijhcs.2017.04.005
Mou, J., Shin, D. H., & Cohen, J. (2017). Understanding trust and
perceived usefulness in the consumer acceptance of an e-service: A
longitudinal investigation. Behaviour & Information Technology, 36
(2), 125–139. doi: 10.1080/0144929X.2016.1203024
Oh, J.-C., & Yoon, S.-J. (2014). Predicting the use of online information
services based on a modified UTAUT model. Behaviour & Information
Technology, 33(7), 716–729. doi:10.1080/0144929X.2013.872187
Pe-Than, E. P. P., Goh, D. H. L., & Lee, C. S. (2017). Does it matter how
you play? The effects of collaboration and competition among players
of human computation games. Journal of the Association for
Information Science and Technology, 68, 1823–1835. doi:10.1002/
asi.23863
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003).
Common method biases in behavioral research: a critical review of the
literature and recommended remedies. Journal of Applied Psychology,
88(5), 879–903. doi: 10.1037/0021-9010.88.5.879
Prandi, C., Nisi, V., Salomoni, P., & Nunes, N. J., 2015. From gamifica-
tion to pervasive game in mapping urban accessibility. In Proceedings
of the 11th CHItaly (pp. 126–129). Rome, Italy: ACM.
Prestopnik, N. R., & Tang, J. (2015). Points, stories, worlds, and diegesis:
Comparing player experiences in two citizen science games.
Computers in Human Behavior, 52, 492–506. doi:10.1016/j.
chb.2015.05.051
Rapp, A., Hopfgartner, F., Hamari, J., Linehan, C., & Cena, F. (2019).
Strengthening gamification studies: Current trends and future oppor-
tunities of gamification research. International Journal of Human-
Computer Studies, 127, 1–6. doi:10.1016/j.ijhcs.2018.11.007
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). The motivational pull
of video games: A self-determination theory approach. Motivation and
Emotion, 30, 344–360. doi:10.1007/s11031-006-9051-8
Sakamoto, M., & Nakajima, T. (2014). Gamifying social media to encou-
rage social activities with digital-physical hybrid role-playing. In
Proceedings of the 2014 International Conference on Social
Computing and Social Media (pp. 581–591). Heraklion, Crete,
Greece: Springer. doi:10.1080/09593330.2013.837964
Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006).
Reporting Structural Equation Modeling and Confirmatory Factor
Analysis Results: A Review. The Journal of Educational Research, 99
(6), 323–338. doi:10.3200/JOER.99.6.323-338
Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A
survey. International Journal of Human-computer Studies, 74, 14–31.
doi:10.1016/j.ijhcs.2014.09.006
Sherry, J. L., Lucas, K., Greenberg, B. S., & Lachlan, K. (2006). Video
game uses and gratifications as predictors of use and game pre-
ference. In P. Vorderer & J. Bryant (Eds.), Playing video games:
Motives, responses, and consequences (pp. 213–224). Abingdon, UK:
Routledge.
Shin, D.-H., & Shin, Y.-J. (2011). Why do people play social network
games? Computers in Human Behavior, 27(2), 852–861. doi:10.1016/j.
chb.2010.11.010
Siu, K., & Riedl, M. O. (2016). Reward systems in human computa-
tion games. In Proceedings of the 2016 Annual Symposium on
Computer-Human Interaction in Play, Austin, Texas, USA
(pp. 266–275).
Sonderegger, A., Zbinden, G., Uebelbacher, A., & Sauer, J. (2012). The
influence of product aesthetics and usability over the course of time:
10 X. WANG ET AL.
A longitudinal field experiment. Ergonomics, 55, 713–730.
doi:10.1080/00140139.2012.672658
Sørensen, J. J. W. H., Pedersen, M. K., Munch, M., Haikka, P.,
Jensen, J. H., Planke, T., … Sherson, J. F. (2016). Exploring the
quantum speed limit with computer games. Nature, 532(7598),
210–213. doi:10.1038/nature17620
Sun, Y., & Jeyaraj, A. (2013). Information technology adoption and
continuance: A longitudinal study of individuals’ behavioral inten-
tions. Information & Management, 50, 457–465. doi:10.1016/j.
im.2013.07.005
Talasila, M., Curtmola, R., & Borcea, C. (2016). Crowdsensing in the wild
with aliens and micropayments. IEEE Pervasive Computing, 15, 68–77.
doi:10.1109/MPRV.2016.18
Venkatesh, V., & Davis, F. D. (1996). A model of the antecedents of
perceived ease of use: Development and test. Decision Sciences, 27(3),
451–481. doi: 10.1111/deci.1996.27.issue-3
Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the
technology acceptance model: Four longitudinal field studies.
Management Science, 46, 186–204. doi:10.1287/mnsc.46.2.186.
11926
Venkatesh, V., Morris, M. G., & Ackerman, P. L. (2000). A longitudinal
field investigation of gender differences in individual technology
adoption decision-making processes. Organizational Behavior and
Human Decision Processes, 83, 33–60. doi:10.1006/obhd.2000.2896
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User
acceptance of information technology: Toward a unified view. MIS
Quarterly, 27(3), 425–478. doi:10.2307/30036540
Venkatesh, V., Thong, J. Y., & Xu, X. (2012). Consumer acceptance and
use of information technology: Extending the unified theory of accep-
tance and use of technology. MIS Quarterly, 36(1), 157–178.
doi:10.2307/41410412
Vigo, M., & Harper, S. (2017). Real-time detection of navigation pro-
blems on the world ‘wild’web. International Journal of Human-com-
puter Studies, 101, 1–9. doi: 10.1016/j.ijhcs.2016.12.002
Von Ahn, L., & Dabbish, L. (2008). Designing games with a purpose.
Communications of the ACM, 51(8), 58–67. doi:10.1145/1378704
Wang, X., Goh, D. H.-L., Lim, E.-P., & Vu, A. W. L. (2016).
Understanding the determinants of human computation game
acceptance: The effects of aesthetic experience and output quality.
Online Information Review, 40(4), 481–496. doi:10.1108/OIR-06-
2015-0203
Wang, X., Goh, D. H.-L., Lim, E.-P., Vu, A. W. L., & Chua, A. Y. K.
(2017). Examining the effectiveness of gamification in human
computation. International Journal of Human–Computer Interaction,
33(10), 813–821. doi:10.1080/10447318.2017.1287458
Zichermann, G., & Linder, J. (2010). Game-based marketing: Inspire
customer loyalty through rewards, challenges, and contests. NJ, USA:
John Wiley & Sons.
About the Authors
Xiaohui Wang is a Research Assistant Professor at Department of
Journalism, Hong Kong Baptist University. His research focuses on
user experience, social media, health informatics, and computational
social science.
Dion Hoe-Lian Goh is an Associate Professor at Wee Kim Wee School of
Communication and Information, Nanyang Technological University.
His research areas and specialties are social media practices and percep-
tions, gamification techniques for shaping user perceptions and motivat-
ing behavior, and mobile information sharing and seeking.
Ee-Peng Lim is a Lee Kong Chian Professor of Information Systems at
the School of Information Systems, Singapore Management University.
His main research interests are data mining and analytics, social media
content mining and analysis, social network mining and analysis, and
digital libraries.
Appendix I
Theories, Constructs, items, and measurement sources
Core constructs Items
Continuance It is worthwhile to use this system
Intention I am more likely to use this system in the future
I would like to use this system if it is available
Attitude I feel that using this system is a good way to kill time
I feel I was absorbed in using this system.
I was so involved in this system that I lost track of time.
Effort Expectancy My interaction with the system is clear and
understandable
It is easy for me to become skillful at using the system
I find the system ease to use
Learning to operate the system is easy for me
Performance I find the system is useful for creating new keywords for
Expectancy videos
I find the system encourages me to create new keywords
for videos
I find the system produces keywords that are useful for
describing videos
Social influence People who influence my behavior think that I should use
the system
People who are import to me think that I should use the
system
People whose opinion that I value prefer that I use the
system
Hedonic Motivation Using this system is enjoyable.
While using this system, I experienced pleasure.
Overall, I believe that this system is enjoyable.