Accepted Manuscript

Learning Executive Function Skills by Playing Focused Video Games

Jocelyn Parong, Richard E. Mayer, Logan Fiorella, Andrew MacNamara, Bruce

Homer, Jan Plass

PII: S0361-476X(17)30315-6
DOI: http://dx.doi.org/10.1016/j.cedpsych.2017.07.002
Reference: YCEPS 1634

To appear in: Contemporary Educational Psychology

Received Date: 23 August 2016

Revised Date: 27 June 2017
Accepted Date: 10 July 2017

Please cite this article as: Parong, J., Mayer, R.E., Fiorella, L., MacNamara, A., Homer, B., Plass, J., Learning
Executive Function Skills by Playing Focused Video Games, Contemporary Educational Psychology (2017), doi:
http://dx.doi.org/10.1016/j.cedpsych.2017.07.002

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers
we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and
review of the resulting proof before it is published in its final form. Please note that during the production process
errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
 LEARNING BY PLAYING GAMES 1

Learning Executive Function Skills by Playing Focused Video Games

Jocelyn Parong

University of California, Santa Barbara

Richard E. Mayer

University of California, Santa Barbara

Logan Fiorella

University of Georgia

Andrew MacNamara

New York University

Bruce Homer

City University of New York

Jan Plass

New York University

Date submitted: August 24, 2016

Revision submitted: March 20, 2017

Revision submitted: June 26, 2017

 LEARNING BY PLAYING GAMES 2

Abstract


The objective of the present study was to determine whether it is possible to design a video game

that could help students improve their executive function skill of shifting between competing

tasks, and the conditions under which playing the game would lead to improvements on

cognitive tests of shifting. College students played a custom video game, Alien Game, which

required the executive function skill of shifting between competing tasks. When students played

for 2 hours over 4 sessions they developed significantly better performance on cognitive shifting

tests compared to a control group that played a different game (d = 0.62), but not when they

played for 1 hour over 2 sessions. Students who played Alien Game at a high level of challenge

(i.e., reaching a high level in the game) developed significantly better performance on cognitive

shifting tests compared to controls when they played for 2 hours (Experiment 1, d = 1.44), but

not when they played for 1 hour (Experiment 2). Experiment 3 replicated the results of

Experiment 1 using an inactive control group, showing that playing Alien Game for 2 hours

resulted in significant improvements in shifting skills (d = 0.78). Results show the effectiveness

of playing a custom-made game that focuses on a specific executive function skill for sufficient

time at an appropriate level of challenge. Results support the specific transfer of general skills

theory, in which practice of a cognitive skill in a game context transferred to performance on the

same skill in a non-game context.

Keywords: video games, serious games, computer games, executive function, cognitive training
 LEARNING BY PLAYING GAMES 3

Learning Executive Function Skills by Playing Focused Video Games

1. Introduction

1.1. Objectives

The goal of the present study was to determine the instructional effectiveness of a custom-

designed video game—Alien Game—intended to improve the executive function skill of shifting

between competing tasks. Executive function refers to cognitive skills for managing one’s

cognitive processing, and shifting, or task switching is a type of executive function skill that is

the focus of this study. We sought to determine the conditions under which playing Alien Game

leads to improvements in shifting skill. Shifting skill is the ability to change from one cognitive

task to another with minimum disruption, such as changing from sorting objects based on shape

to sorting based on color, or from sorting letter number pairs on the basis of the letter

characteristics to sorting based on number characteristics. Instead of relying on an off-the-shelf

game originally intended for entertainment, we developed a focused video game that requires

players to repeatedly engage in task switching at increasing levels of challenge within a varied

game environment. Because this game was designed with the specific goal of improving an

executive function (EF) skill, and because our design process employed principles from

Evidence Centered Design (Mislevy, Almond, & Lukas, 2003), we have reason to expect this

game to be more effective than games not designed for this purpose or games that were designed

by gamifying tasks intended to measure EF skills.


1.2. Rationale

Executive function (EF) is the set of cognitive processes used for effortful, controlled,

goal-directed thinking and behavior (Banich, 2009; Best, 2012). In short, EF skills serve to

manage one’s cognitive processing aimed at achieving a goal. EF skills are not only important
 LEARNING BY PLAYING GAMES 4

for everyday activities, but they also have critical academic implications because they predict

academic success in children (Best, 2014; St Clair-Thompson & Gathercole, 2006). EF is

comprised of three closely related, but distinguishable component skills: shifting, (i.e., switching

from one task to another), updating (i.e., keeping track of multiple events), and inhibition (i.e.,

not attending to irrelevant features), all three of which are fundamental in a classroom setting

(Miyake, Friedman, Emerson, Witzki, & Howerter, 2000). For this reason, we consider EF skills

to be educationally relevant skills.

Many strong claims are made for the potential of games to revolutionize education. For

example, visionaries have said that “kids learn more positive, useful things for their future from

their video games than they learn in school” (Prensky, 2006, p. 4); “the key to solving the current

crisis in education will be to use the power of computer and video games to give all children

access to experiences and build interest and understanding” (Shaffer, 2006, p. 67); and “good

games are problem-solving spaces that create deep learning, learning that is better than what we

often see today in our schools” (Gee, 2007, p. 10). In support of these claims, however, they

present weak evidence or none at all.

In contrast, game researchers offer a more cautious assessment of the research base on

computer games for learning. For example, research reviews conclude: “Many strong claims are

made for the educational value of computer games, but there is little strong empirical evidence to

back up those claims” (Mayer, 2011a, p. 281); “...there is relatively little research evidence on

the effectiveness of simulations and games for learning” (Honey & Hilton, 2011, p. 21); “...there

is considerably more enthusiasm for describing the affordances of games and their motivating

properties than for conducting research to demonstrate that those affordances are used to attain

instructional aims...This would be a good time to shelve the rhetoric about games and divert
 LEARNING BY PLAYING GAMES 5

those energies to conducting needed research” (Tobias, Fletcher, Dai, & Wind, 2011, p. 206);

and “While effectiveness of game environments can be documented in terms of intensity and

longevity of engagement ... as well as the commercial success of the games, there is much less

solid empirical information about what outcomes are systematically achieved by the use of

individual and multiplayer games to train participants in acquiring knowledge and skills. Further,

there is almost no guidance for game designers and developers on how to design games that

facilitate learning” (O’Neil & Perez, 2008, p. ix).

Our approach is to close the gap between strong claims and weak evidence for the value

of educational games by focusing on the use of games to foster specific cognitive skills, the area

in which game research findings to date are the most encouraging. There is promising evidence

that playing certain action video games can improve perceptual attention skills (Green &

Bavelier, 2003; Feng, Spence, & Pratt, 2007; Mayer, 2014). For example, playing a first-person

shooter game, Unreal Tournament, resulted in greater improvements in perceptual attention

measures such as useful field of view (Green & Bavelier, 2006a), enumeration (Green &

Bavelier, 2006b), and contrast sensitivity (Li, Polat, Makous, & Bavelier, 2009) as compared to

playing a puzzle game or simulation game. There is also promising evidence that playing certain

games can improve spatial cognition skills. For example, several studies showed that playing the

puzzle game Tetris resulted in improvements in spatial cognition skills such as 2D mental

rotation (DeLisi & Woolford, 2002; Okagaki & Frensch, 1994; Sims & Mayer, 2002;

Subrahmanyan & Greenfield, 1994) and 3D mental rotation (Okagaki & Frensch, 1994; Terlecki

et al., 2008), although in some cases the effects were weak or did not transfer to rotation of new

shapes. However, there is not substantial evidence for learning of other cognitive skills through

playing off-the-shelf games (Mayer, 2014).

 LEARNING BY PLAYING GAMES 6

Though research on learning EF skills with games is minimal and most studies do not

report strong effects (Mayer, 2014), there is some promising evidence. For example, Green,

Sugarman, Medford, Klobusicky, and Bavelier (2012) found that playing the first-person shooter

game Unreal Tournament resulted in improvements in task-switching speed, and Whitlock et al.,

(2012) found that playing the massive multi-player role-playing game World of Warcraft

improved performance on the Stroop task in older adults as compared to playing non-action

games. Similarly, Bialystok (2006) found video game play to be related to faster performance on

a number of executive control tasks. In light of these and related results, there is reason to pursue

a plan of optimizing mini-games for teaching specific subskills associated with EF. The rationale

for designing mini-games that focus on specific executive function skills is that most off-the-

shelf games do not target these skills. Off-the-shelf games were designed mainly for

entertainment rather than cognitive training. Therefore, in this study we examine whether it

might be possible to design a game that is fun to play--like off-the-shelf games--and helps

players practice a specific EF skill at increasing levels of challenge--unlike most off-the-shelf

games. In this way, our study takes a scientific approach to testing claims about the effectiveness

of games for learning, particularly for helping students improve on educationally- relevant

cognitive skills.


1.3. Literature Review

Although video games have been shown to be effective in improving perceptional

attention skills (Anderson & Bavelier, 2011; Green, 2014; Mayer, 2014), research on games for

teaching EF skills is in its infancy. Evidence for how video game play affects EF has been

mixed. While some research has shown negligible effects of video game play on EF skills

(Powers, Brooks, Aldrich, Palladino, & Alfieri, 2013), other research has shown that video game
 LEARNING BY PLAYING GAMES 7

play can enhance shifting abilities (Mayer, 2014). Arcade-style games and games designed for

the Wii platform yield significant training effects on some EF skills (Powers & Brooks, 2014).

For the shifting component of EF, video games that are cognitively engaging, such as

action or strategy games, require players to constantly shift from one set of relevant instructions

to another. For example, in some first-person shooter games, a player must respond to a certain

enemy with one action and to another enemy with a different action, which trains the shifting

component of EF (Best, 2014). The improvement in shifting within the video game may then

transfer to other cognitive tasks that require shifting in non-game contexts.

Some experimental studies have provided support for the notion that video game play can

enhance shifting skill, but the amount of game play was substantial. For example, participants

trained in 50 hours of two action shooter video games, Unreal Tournament 2004 and Call of

Duty 2, showed improvements in shifting compared to training in a control game, The Sims 2

(Green et al., 2012). In another study, Basak, Boot, Voss, and Kramer (2008) found that

participants trained in a strategy game, Rise of Nations, for 23.5 hours improved on a task

switching assessment compared to a no-video game control group. In the present study, we

examine the effectiveness of only 1 or 2 hours of exposure to a game specifically designed to

provide concentrated practice on shifting skills in a variety of game contexts and with increasing

levels of challenge, because requiring 20 to 50 hours of game-based training on a single skill

might be impractical for many learners. Experiments 1 and 2 are identical except that

participants played for 2 hours across 4 sessions in Experiment 1, and they played for 1 hour

across 2 sessions in Experiment 2. In Experiment 3, participants also played for 2 hours, but

were compared to an inactive control group.

 LEARNING BY PLAYING GAMES 8

1.4. Theoretical Framework

Transfer of cognitive skill refers to the effects of past learning on one’s ability to apply a

skill to new tasks (Mayer, 1987; Mayer & Wittrock, 1996; Pellegrino & Hilton, 2012).

Traditionally, theories of transfer have been divided into specific transfer—the idea that prior

learning of a skill on one task can affect performance on new tasks to the extent that the tasks

have identical elements in common (Thorndike, 2006; Thorndike & Woodworth, 1901)—and

general transfer—the idea that learning one skill will result in improvements on tasks requiring

unrelated skills by improving the mind in general (Singley & Anderson, 1989). Research

spanning more than 100 years repeatedly shows strong evidence for specific transfer but not for

general transfer (Mayer & Wittrock, 1996; Pellegrino & Hilton, 2012; Singley & Anderson,

1989). One explanation, based on research in expert skill performance (e.g., Ericsson, Charness,

Feltovich, & Hoffman, 2006; Singley & Anderson, 1989) is that skilled performance depends on

domain specific skills rather than general heuristics.

Given that education often is seen as an attempt to improve one’s ability to apply what

one has learned to new situations, scholars and researchers over the years have sought to

determine ways to promote learning that goes beyond specific transfer (Pellegrino & Hilton,

2012; Sims & Mayer, 2002). In this study, we examine a third view of transfer of cognitive skill,

the cognitive theory of specific transfer of general skills, which posits that a cognitive skill

trained in one situation can be applied to another situation that requires the same general skill

(Mayer, 2014). The idea that learning a skill in a broad way can lead to transfer on other tasks

requiring the same general skill or principle has its roots in early research on skill transfer dating

back to Judd (1908) and on meaningful learning of procedures inspired by the Gestalt

psychologists (Katona, 1940; Wertheimer, 1945), and is the basis for what can be called
 LEARNING BY PLAYING GAMES 9

transferable knowledge (Pellegrino & Hilton, 2012). In applying the cognitive theory of specific

transfer of general skills to game-based training, we seek to test the proposal that appropriately

designed video games can allow players to practice and train a cognitive skill within a game and

then apply it to new contexts outside the game that also specifically require that cognitive skill

(Mayer, 2014). In the present study, we designed a game that required players to repeatedly

exercise the executive function skill of shifting at increasing levels of challenge and in varying

game contexts with clear feedback, as recommended by Anderson and Bavelier (2011). If the

game is successful in promoting specific transfer of general skill, we expect Alien Game players

will show improvements on tasks requiring shifting outside of the game context—such as on

cognitive tests of shifting—as compared to participants who play a game that does not require

shifting.

Although research on training and transfer of cognitive skills has a long and somewhat

disappointing history in educational psychology and cognitive psychology (Pellegrino & Hilton,

2012), more recently the search for transferable cognitive skill training has expanded to include

the potential of video games (Anderson & Bavelier, 2011; Green, 2014; Lieberman, Biely, Thai,

& Peinado, 2014). As noted by Powers and Brooks (2014, p. 302), however, the criterion by

which to judge the efficacy of video games as vehicles for cognitive skill training concerns the

degree to which skills learned in a game context can be applied successfully in non-game

contexts: “a critical question for application of video games to professional training, education,

or rehabilitation is the extent to which skills enhanced through video game play transfer to tasks

outside the game environment.”

An important distinction within the theory of specific transfer of general skill can be

drawn between near transfer—applying the learned skill in similar contexts (such as applying a
 LEARNING BY PLAYING GAMES 10

skill learned in a game to performance on classic cognitive tests requiring the same skill)—and

far transfer—applying the learned skill in dissimilar contexts (such as applying a skill learned in

a game to performance on an academic task requiring the same skill; Lieberman, et al., 2014;

Mayer, 2014). Thus, in applying the theory of specific transfer of general skill to game-based

training, we can search for a chain of transfer effects in which exercising a skill in a game

context can cause (1) an improvement on cognitive tests requiring the same skill (near transfer to

a similar but non-game context), and (2) an improvement on academic tasks requiring the same

skill (far transfer to a dissimilar and non-game context). In the present study, we focus on the

first link in this proposed causal chain of the game-based theory of specific transfer of general

skill—game-playing leads to improvements in targeted skills in a similar but non-game

context—as a first step in investigating the efficacy of games as vehicles for training

educationally-relevant cognitive skills. Given the long-standing lack of support for general

transfer, we do not also examine the link between learning one skill in a game and subsequent

performance on unrelated skills in a non-game context.

1.5 Predictions

In the present study, we compare posttest scores (and gains scores) on shifting skills for

students who are assigned to play a game intended to promote shifting skill (Alien Game group)

versus students who play a game unrelated to shifting skill (Bookworm group). Alien Game

requires players to respond to aliens who appear on the screen by shooting up food or drink

based on rules, such as red aliens need food and blue aliens need drink. The rules continually

change throughout the game, requiring the player to shift criteria for shooting up food or drink.

This requires the player to continually monitor the status of the screen and to alter response

contingencies when change is required. Thus, Alien Game is intended to give players practice in
 LEARNING BY PLAYING GAMES 11

task switching. In contrast, Bookworm is a word search game that does not require task

switching. We chose Bookworm as the control because it is a game that does not involve shifting,

it is somewhat similar in on-screen appearance as Alien Game, and participants found word

search games to be challenging in previous research (Adams, Pilegard, & Mayer, 2016).

We measure transfer of switching skill through versions of two cognitive tasks in non-

game contexts-- the dimensional change card sort task (DCCS) and the letter-number task (LN).

In the DCCS, participants are given objects (e.g., blue truck) to sort based either on color (e.g.,

yellow or blue) or type of object (e.g., truck or ball). In the letter-number task, participants must

judge whether a presented letter-number pair contains a vowel or a consonant on some trials and

whether it contains an even or odd number on other trials. Switching cost is determined by

subtracting the time (or accuracy) to make a judgment that is based on a different rule as the

previous trial from the time (or accuracy) to make a judgment that is based on the same rule as

the previous trial, with greater switching cost indicating poorer switching skill. We recognize

that we are setting a rather low bar for transfer of switching skill by using measures that require

the same skill (being able to rapidly and accurately shift from one rule to another) as in the game,

but in a non-game cognitive task. This research can be considered an initial step in determining

whether a skill exercised in a game can transfer to non-game environments that require the same

skill but do not have the same surface features as the game.

According to the specific transfer of general skill hypothesis, we predict the Alien Game

group will show greater posttest scores (and gain scores) on cognitive tests of switching than the

Bookworm group. In contrast, the specific transfer hypothesis predicts no difference between the

groups.
 LEARNING BY PLAYING GAMES 12

2. Method


2. 1. Experiment 1


2.1.1. Participants and design

In Experiment 1, the participants were 45 undergraduate students (26 women, ages 18-24;

M = 19.67, SD = 1.50) from the University of California, Santa Barbara. Twenty-six participants

were assigned to the experimental group, which played Alien Game, and 19 were assigned to the

control group, which played Bookworm.


2.1.2. Procedure

Participants signed up for a four-part experiment via an online subject recruitment

website and were paid $40 for their participation. Participants were tested individually in

cubicles containing a computer workstation in a lab setting. First, the experimenter explained the

study and obtained informed consent from all participants. Participants were randomly assigned

to play one of two video games: Alien Game (experimental group) or Bookworm (control group).

In the course of four sessions spread over 2 weeks, participants completed a pretest of 2

cognitive assessments, engaged in 2 hours of video game play of either Alien Game or

Bookworm, completed a posttest of the same 2 cognitive assessments, and took a demographic

questionnaire. The first session included the pretest and 30 minutes of game play. The second

and third sessions each included 30 minutes of gameplay, and the fourth session included 30

minutes of game play, followed by the posttest and the participant questionnaire. We followed

standards for ethical treatment of human subjects and obtained IRB approval for the study.


2.1.3. Materials

The materials consisted of two cognitive skill assessments, two computer games, and a

computer-based participant questionnaire, all administered online through a website created for
 LEARNING BY PLAYING GAMES 13

the experiment. Two computer-based cognitive assessments measuring the shifting component of

executive function, the Dimensional Change Card Sort test (DCCS) and Letter-Number task

(LN), were used in both the pretest and posttest. As shown in Figure 1, in the DCCS (Zelazo,

2006), participants were prompted with the word color or shape and then presented with an

object (e.g., a blue or yellow tennis ball or truck). When the word color was presented, the

participant pressed a key to indicate whether the object was blue or yellow. When the word

shape was presented, the participants pressed a key to indicate whether the object was a tennis

ball or truck. Task switching skill was required when the word presented changed from color to

shape or vice versa. The test-retest reliabilities for the DCCS among children aged 3 to 15 years

were high (ICCs = .90–.94; Beck, Schafer, Pang, & Carlson, 2011; Zelazo, Anderson, Richler,

Wallner-Allen, & Beaumont, 2013). Among adults, there was also excellent test-retest reliability

(ICCs = .71-.87) and convergent validity as it correlated with the NIH Toolbox Flanker task

(Zelazo et al., 2014).

As shown in Figure 2, in the LN task (Miyake et al. 2000), participants were presented

with a letter-number pair (e.g. G6) in one of four quadrants. When the letter-number pair was

presented in the top two quadrants, the participant decided if the number was odd (1, 3, 5, or 7)

or even (2, 4, 6, or 8) and pressed a keyboard key accordingly. When the letter-number pair was

presented in the bottom two quadrants, the participants decided if the letter was a vowel (A, E, I,

or U) or a consonant (G, K, M, or R) and pressed a keyboard key accordingly. Participants were

presented with letter-number pairs in a clockwise pattern starting in the top left quadrant. The

bottom right and top left quadrants required a shift in task (i.e., were switch trials), whereas the

bottom left and top right quadrants did not (i.e., were non-switch trials). The reliability estimate

for the letter-number task, .91, was high, which was calculated by adjusting split-half (odd–even)
 LEARNING BY PLAYING GAMES 14

correlations with the Spearman-Brown prophecy formula (Miyake et al., 2000).

As shown in Figure 3, Alien Game presented participants with aliens, who were either

thirsty or hungry, falling from the top of the screen. The narrative of the game stated, “Welcome

Space Ranger! The aliens on this strange planet have been waiting for your ship to arrive. Some

of the aliens are very hungry, and some are very thirsty. Good thing you have plenty of food and

drinks on board of your ship to give them. Right now, red aliens are thirsty, and blue aliens are

hungry. Your job as space ranger is to take care of the aliens and give them exactly what they

want [ ... ] Be careful, This is a strange planet and rules are always changing. [...]” The player’s

job was to satisfy the aliens’ thirst and hunger by shooting up food and drinks from a dispenser at

the bottom of the screen. Players were instructed that aliens were hungry or thirsty depending on

two properties of the aliens, their color and/or number of eyes (e.g., red aliens are hungry and

blue aliens are thirsty), but that they often changed their minds as indicated by a change of rules

to follow (e.g.. “Now blue aliens are hungry and red aliens are thirsty”). Another kind of switch

occurred when the rule changed based on a different property of the aliens (e.g., a rule based on

number of eyes rather than color). The game difficulty increased with increasing levels, which

involved more complex rules, higher accuracy requirements, and higher speed. Points were

awarded for each alien fed correctly, and subtracted for each alien fed incorrectly. The game

recorded in a user log for each player how many aliens were fed correctly, how many were fed

incorrectly, and how many were missed altogether. In addition, the game recorded which level

the player achieved. The game is available at the following URL [ blinded for review ].

As shown in Figure 4, Bookworm (Pop Cap Games, 2001) was a word search game in

which participants created letter chains to form words among letter tiles on the screen. Words

could be created in any direction as long as the letter tiles were adjacent to the previous letter.
 LEARNING BY PLAYING GAMES 15

This same basic rule was applied to all levels of the game. When a word was found, the tiles

disappeared, the remaining tiles shifted down into their spots, and new tiles appeared at the top.

Each word was awarded a number of points, and longer and less commonly used words earned

more points. The objective of the game was to create words until the player earned enough points

required to pass each level. As the levels progressed, the point requirements increased, and

certain letter tiles were required in creating a word. If the indicated tile was not used within a

certain number of moves, the game ended.

The participant questionnaire solicited information in the participant’s age, gender, and

video game playing experience.


2.1.4. Apparatus

All materials were presented to participants on iMac or Dell computers, with 20-inch

color screens.

2.2. Experiment 2

In Experiment 2, the participants were 49 undergraduate students (37 women, ages 18-35;

M = 19.96, SD = 2.48) from the University of California, Santa Barbara. Twenty-five

participants were randomly assigned to the Alien Game group, and 24 were randomly assigned to

the Bookworm group. Participants signed up for a two-part experiment via an online subject

recruitment website and were paid $20 for their participation. The second experiment followed

the same procedure and used the same materials and apparatus as the first experiment, except

participants completed only 1 hour of training across two sessions over a week. The first session

included the cognitive pretest and 30 minutes of game play. The second session included 30

minutes of gameplay, followed by the posttest and the participant questionnaire.

 LEARNING BY PLAYING GAMES 16

3. Results

3.1 Scoring

The DCCS was scored according to the NIH Toolbox guidelines from a range of 0 to 10,

with up to 5 points for accuracy and up to 5 points for reaction time. Higher scores indicated

better performance. In the LN task, a switch cost was calculated as an index of shifting ability.

The switch cost was calculated by subtracting the average correct reaction times of the non-shift

trials from the average correct reaction times of the shift trials. Lower switch costs indicated

better performance. To determine the educational impact of game playing, we calculated Cohen's

d (Cohen. 1988) as our measure of effect size. Cohen (1988) suggested that an effect size of .2 is

small, .5 is medium, and .8 is large. Consistent with Hattie (2009), we consider effect sizes

greater than d = .40 to be educationally important, and consistent with Cohen (1988) we consider

effect sizes lower than d = .20 to be negligible.


3.2. Do the Groups Differ on Basic Characteristics?

The first step was to determine whether the groups were equivalent on basic

characteristics. In Experiments 1 and 2, participants in the Alien Game group and Bookworm

group did not differ significantly (at p < .05 based on a t-test) in their age, average video game

play reported per week, mean DCCS pretest score, mean LN pretest score, and pretest reaction

times on the non-switch trials in the LN task. The groups also did not differ significantly (at p <

.05 based on a chi-square test) in the proportion of men and women. We conclude that the groups

did not differ on basic characteristics.


3.3. Does Playing Alien Game Increase Shifting Skill?

According to the specific transfer of a general skill theory, the Alien Game group should

do better on the DCCS and LN tasks than the Bookworm group. The top two portions of Table 1
 LEARNING BY PLAYING GAMES 17

show the means and standard deviations of the two groups on the two cognitive tests. ANCOVAs

were run on the posttest scores of the DCCS and LN, with respective pretest scores as a

covariate, for each experiment. In the first experiment, the Alien Game group performed

significantly better than the Bookworm group on the LN task, F(1,42) = 7.15, p = .011, d = 0.82,

but the difference did not reach statistical significance on the DCCS, F(1,42) = 0.47, p = .498, d

= .08. We conclude that playing the Alien Game for 2 hours produced significant effects with a

large effect size on one of two cognitive measures of task switching. One reason the DCCS did

not yield significant results may be that it was designed for use with children, so it may not be as

highly sensitive for the college students which is exacerbated by not having a large sample size.

In the second experiment, the Alien Game group and the Bookworm group did not differ

significantly on the DCCS, F(1,46) = 3.79, p = .058, d = 0.41, or the LN task, F(1,46) = 1.14, p =

.291, d = .07. We are not able to conclude that playing the Alien Game for 1 hour produced

strong and significant effects on cognitive measures, although the effect size greater than .4 on

the DCCS encourages further investigation with larger sample sizes.

In order to test the hypothesis combining all the available data, we created a composite

shifting score for each player by adding the standardized score on the DCCS to the standardized

score on the LN and dividing by 2. We created the composite score in order to maximize the

power available in the data set to test the hypothesis, because the Alien Game group

outperformed the Bookworm group on both tests, and because we had no theoretical reason to

investigate the two switching tests separately. The mean composite shifting score and standard

deviation on the posttest for each group in each experiment is shown in the bottom portion of

Table 1. For Experiment 1, an ANCOVA with composite pretest score as the covariate showed

that the Alien Game group achieved a significantly higher composite shifting score on the
 LEARNING BY PLAYING GAMES 18

posttest than the Bookworm group, F(1,42) = 6.55, p = .014 d = 0.54. Not only is the effect of

game playing statistically significant (at the .05 level), it also produced a sufficiently large effect

size (i.e., greater than .4) to be considered educationally relevant. For Experiment 2, an

ANCOVA with composite pretest score as the covariate showed no significant difference

between the groups on composite shifting score on the posttest, F(1,46) = 0.63, p = .432. We are

not able to conclude that playing the Alien Game for one hour causes a substantial improvement

in task switching skill. The same pattern of significant results was obtained for ANCOVAs on

1
gains scores (i.e., posttest score – pretest score) with pretest as a covariate . Overall, these results

provide support for the hypothesis that playing the Alien Game improves shifting ability, at least

at sufficient dosage (i.e., 2 hours of training in Experiment 1).

Finally, we examined whether the effects of game playing might be strong for players

who reached high levels in the Alien Game, suggesting they played at a high level of challenge.

The role of game performance in training of shifting ability was examined by using a median

split on the level reached in Alien Game during the training period, with students above the

median considered to be performing at a high level of challenge. ANCOVAs were run to

compare the high performance Alien Game group and the Bookworm group on their scores on the

DCCS and LN tasks. The two portions of Table 2 show the means and standard deviations of

these groups in each of the two experiments. In the first experiment, the high Alien Game group

significantly outperformed the Bookworm group on the LN task, F(1,30) = 6.31, p = .018, d =

1.08, but the difference did not reach statistical significance on the DCCS, F(1,30) = 3.75, p =

.072, d = 1.04. In the second experiment, the high Alien Game group did not outperform the

Bookworm group significantly on the DCCS task, F(1,36) = 2.61, p = .115, or the LN task, F =

0.15, p = .696. As shown in the bottom row of Table 2, using composite scores, an ANCOVA
 LEARNING BY PLAYING GAMES 19

with pretest scores as a covariate showed that the high Alien Game group did significantly better

than the Bookworm group in Experiment 1, F(1,30) = 10.63, p = .003, d = 1.44, but not in

Experiment 2, F(1,36) = 0.59, p = .446. The same pattern of significant results was obtained for

2
ANCOVAs on gains scores (i.e. posttest score – pretest score) with pretest as a covariate . Thus,

the effectiveness of game playing is strong when we focus on students who play for a sufficient

period of time and at a high level of challenge.

4. Experiment 3

A primary goal of Experiment 3 was to test whether the results of Experiment 1 could be

replicated. In short, we conducted a media comparison study comparing gains in targeted

executive function skills by students who played Alien Game for 2 hours over 4 sessions versus

those in an inactive control group who did not play Alien Game. Replication is recognized as one

of the six principles for conducting educational research (Shavelson & Towne, 2002), and is

particularly important in light of the increased role of meta-analyses of research on instructional

interventions (Hattie, 2009) and in light of the conversation about the so-called "crisis of

replication" in which well-known effects in psychology sometimes cannot be replicated (Pashler

& Wagenmakers, 2012; Stroebe & Strack, 2014).

A secondary goal was to conduct a value-added study comparing the gains in targeted

executive skills by students whom played the base version of Alien Game versus those who

played an enhanced version that presented screens that stated performance goals before each

level of the game.

4.1 Method


4.1.1. Participants and design

The participants were 89 college students recruited as in Experiments 1 and 2, with a

 LEARNING BY PLAYING GAMES 20

mean age of 19.87 years (SD = 1.59) and 62 of the participants being women. Thirty-one

participants were assigned to the no goals group, which was identical to the Alien Game group in

Experiment 1, 33 participants were assigned to the goals group, which was identical to the Alien

Game group in Experiment 1 except a goal setting screen was presented before each level of the

game, and 25 participants served in the control group, which did not play the Alien Game. As in

Experiments 1 and 2, the groups did not differ significantly (p < .05) on average age, average

video game play reported per week, mean DCCS pretest score, mean LN pretest score, or

proportion of men and women.


4.1.2. Procedure, Materials and Apparatus

The procedure, materials, and apparatus for the control group was the same as in

Experiment 1 except participants solely took the pretests and posttest and did not play a game in

the lab (so there was no Bookworm game). We decided to use an inactive control in Experiment

3, because Experiment 1 had already established the game with an active control group. The

procedure, materials, and apparatus for the no goal group were identical to the Alien Game group

in Experiment 1. The procedure, materials, and apparatus for the goal group was identical to the

Alien Game group in Experiment 1 except a goal setting screen was presented after each level.

The goal setting screen displayed the participant’s accuracy (in percent correct) and reaction time

(in milliseconds) from the just completed level and recommended a target accuracy and target

reaction time for the next level. Based on the accuracy and reaction time of the previous level,

the new goals for the upcoming level were a 5% increase in the percentile of those scores for the

new level based on norms from previous players. The tests of executive function were scored as

in Experiment 1.

 LEARNING BY PLAYING GAMES 21

4.2 Results


4.2.1 Does Adding Goal Setting Screens to the Alien Game Increase Shifting Skill?

Table 3 shows the mean posttest scores (and SDs) on the DCCS and LN tasks for each of

the three groups. Based on an ANCOVA with pretest score as a covariate, there was no

significant difference between the goal group and no goal group on mean posttest scores adjusted

for pretest score for the DCCS task, F(1,61) = 3.17, p = .080, the LN task, F(1,61) = 0.31, p =

.581, or for a composite score computed as in Experiment 1, F(1,61) = .46, p = .501. The same

conclusions were supported based on an ANCOVA with all three groups in Table 3, followed up

by pairwise Tukey tests comparing the goal and no goal groups. As can be seen in Table 3, for

each measure, the goal group scored lower than the no goal group, albeit at non-significant

levels. We conclude that there is no evidence that adding goal setting screens improved learning.

In retrospect, we suspect that the goal setting screens did not provide useful information that

participants could use to guide their game playing and that the goal setting screens may have

been largely ignored.


4.2.2 Does Playing the Alien Game Increase Shifting Skill?

The primary goal of Experiment 3 was to determine whether 2 hours of playing Alien

Game could improve shifting skill as was found in Experiment 1. Given the foregoing analysis

showing that the goal and no goal groups did not differ significantly, we combined them into a

game group, which we compared to the control group. Although combining the game groups

listed in Table 3 created unequal sample sizes in the game and control groups, the benefit of

doing so is to increase statistical power without violating any statistical requirements (Hayes,

1988). Consistent with predictions, based on an ANCOVA with pretest score as a covariate, the

game group (i.e., the combined goal and no goal groups) achieved significantly higher posttest
 LEARNING BY PLAYING GAMES 22

scores (adjusted for pretest score) than the control group on the DCCS (M = 8.65, SD = 0.62 for

the game group, M = 8.16, SD = 0.70 for the control group; F(1,86) = 16.38, p < .001, d = 0.75)

and on the composite score (M = 0.16, SD = 0.78 for game group; M = -0.43, SD = 0.76 for

control group; F(1,86) = 11.18 p = .001, d = 0.78). The game group (M = 364.88, SD = 208.45)

did not achieve significantly higher scores than the control group (M = 457.95, SD = 203.87) on

the LN task, F(1,86) = 2.75, p = .101, d = 0.45. As in Experiment 1, the same pattern of

significant results was present when we conducted an ANCOVA on gain scores with pretest as a

covariate. Also, the same conclusions were supported when we conducted an ANCOVA

comparing only the no goal group (which is the same as the Alien Game group in Experiment 1)

and the control group.

Finally, as in Experiment 1, the same pattern of significant results was obtained when we

compared the highly challenged players in the game group (i.e., those reaching game levels

above the median) to the control group, including finding larger effect sizes favoring the high-

challenge game group on composite score (d = 1.11), DCCS (d = 1.14), and LN (d = .55 ).

Overall, these results show that game effects based on two hours of gameplay can be found

across two independent experiments, and strengthen our conclusion that playing Alien Game for

a sufficient period of time improves the executive function skills targeted by the game.


5. General Discussion


5. 1. Empirical Contributions

The present study provides somewhat rare empirical evidence that playing a custom-

designed game that requires repeatedly exercising shifting between competing tasks for a modest

amount of time can transfer to improved performance on classic cognitive tasks of shifting

outside of the game context as compared to a control group. An important boundary condition is
 LEARNING BY PLAYING GAMES 23

that the effects are strong for longer periods of game play (i.e., 2 hours over 4 sessions) but not

for shorter periods of game play (i.e., 1 hour over 2 sessions). These replicated findings help

establish an empirical research base on the effective use of custom-designed video games for

improving specific executive function skills.


5.2. Theoretical Implications

A theoretical implication is that the results support the cognitive theory of specific

transfer of general skills theory in that the general skill of shifting was transferred from a game

context to a cognitive testing context. In this case, we focused on a general executive function

skill—shifting from one task to another—and found that practicing that skill in a game context

can transfer to improvements in performance on tasks that require that skill in a non-game

context. In particular, we focused on the near transfer link between training in shifting in a game

context to performance on a cognitive test of shifting in a non-game context. Thus, this study

contributes to an emerging new path for research on transfer—using games as a vehicle for

promoting transfer of cognitive skills (Anderson & Bavelier, 2011; Mayer, 2014; Singley &

Anderson, 1989).


5.3. Practical Implications

A practical implication is that it might be more fruitful to custom-design computer games

that target specific executive function skills rather than rely on off-the-shelf games that are

designed primarily for entertainment. The advantage of building a custom-designed game is that

it can focus on game tasks that require the player to repeatedly engage in the target cognitive

skill in varied contexts within the game and at increasing levels of challenge. Importantly,

substantial effects can be created with just 2 hours of game play, rather than 20 to 50 hours of

play required to produce effects using off-the-shelf games intended for entertainment.
 LEARNING BY PLAYING GAMES 24

5.4. Limitations and Future Directions

This study was a first step in examining the efficacy of video games as a training tool for

cognitive skills that may have academic implications. A fruitful next step would be to examine

whether the effects of game playing transfer to academic tasks that require shifting, and

ultimately to academic achievement in subject areas (i.e., the far-transfer link). Another

important issue concerns whether other executive function skills—such as inhibition and

updating (Miyake et al, 2000)—can be improved by playing games that are designed to exercise

those skills. Given that some cognitive strategy effects do not appear until delayed tests

(Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013), it would be useful to conduct

experiments that contain both immediate and delayed tests. Additionally, by including a delayed

test (e.g. after days or weeks), one would be able to determine whether the effects of enhancing

executive function through video games are relatively permanent rather than potentially a result

of transient arousal from playing the video game. Finally, it would be useful to pinpoint the

dosage required to establish substantial improvement in executive function skills, such as, how

many hours of playing and how much time per session is optimal.

Acknowledgement: This project was supported by Grant R305A150417 from the Institute of

Education Sciences of the U.S. Department of Education.

 LEARNING BY PLAYING GAMES 25

References

Adams, D. M., Pilegard, C., & Mayer, R. E. (2016). Evaluating the cognitive consequences of

playing Portal for a short duration. Journal of Educational Computing Research, 54, 173-

195.

Anderson, A. F., & Bavelier, D. (2011). Action game play as a tool to enhance perception,

attention, and cognition. In S. Tobias & J. D. Fletcher (Eds.), Computer games and

instruction (pp. 307-329). Charlotte, NC: Information Age Publishing.

Banich, M. T. (2009). Executive function: The search for an integrated account. Current

Directions In Psychological Science, 18(2), 89-94.

Basak, C., Boot, W.R., Voss, M.W., & Kramer, A.F. (2008). Can training in a real-time strategy

video game attenuate cognitive decline in older adults? Psychology And Aging, 23(4),

765-777.

Beck, D. M., Schaefer, C., Pang, K., & Carlson, S. M. (2011). Executive function in preschool

children: Test-retest reliability. Journal of Cognition and Development, 12(2), 169-193.

Best, J. (2012). Exergaming immediately enhances children’s executive function. Developmental

Psychology, 48(5), 1501-1510.

Best, J. R. (2014). Relations between video gaming and children's executive functions. In F. C.

Blumberg (Ed.), Learning by playing: Video gaming in education (pp. 42-53). New York,

NY, US: Oxford University Press.

Bialystok, E. (2006). Effect of bilingualism and computer video game experience on the simon

task. Canadian Journal of Experimental Psychology/Revue Canadienne De Psychologie

Expérimentale, 60(1), 68-79. doi:http://dx.doi.org/10.1037/cjep2006008

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Hillsdale, NJ:
 LEARNING BY PLAYING GAMES 26

Erlbaum.

Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013).

Improving students' learning with effective learning techniques: Promising directions

from cognitive and educational psychology. Psychological Science in the Public Interest,

14(1), 4-58.

DeLisi, R., & Wolford, J. L. (2002). Improving children’s mental rotation accuracy with

computer game playing. The Journal of Genetic Psychology, 163, 272-282.

Ericsson, K. A., Charness, N., Feltovich, P. J., & Hoffman, R. R. (Eds.). (2006). The Cambridge

handbook of expertise and expert problem solving. New York: Cambridge University

Press.

Feng, J., Spence, I., & Pratt, J. (2007). Playing an action video game reduces gender differences

in spatial cognition. Psychological Science, 18(10), 850-855.

doi:http://dx.doi.org/10.1111/j.1467-9280.2007.01990.x

Gee, J. P., (2007). What Video Games Have to Teach Us About Learning and Literacy. New

York, NY: St. Martin’s Press.

Green, C. S. (2014). The perceptual and cognitive effects of action video game esperience. In F.

C. Blumberg (Ed.), Learning by playing: Video gaming in education (pp. 29-41). New

York, NY, US: Oxford University Press.

Green, C.S., & Bavelier, D. (2003). Action video game modifies visual selective attention.

Nature, 423, 534–537. doi:10.1038/nature01647

Green, C. S., & Bavelier, D. (2006a). Effect of action video games on the spatial distribution of

visuospatial attention. Journal of Experimental Psychology: Human Perception and

Performance, 32(6), 1465-1478. doi:http://dx.doi.org/10.1037/0096-1523.32.6.1465

 LEARNING BY PLAYING GAMES 27

Green, C. S., & Bavelier, D. (2006b). Enumeration versus multiple object tracking: The case of

action video game players. Cognition, 101(1), 217-245.

doi:http://dx.doi.org/10.1016/j.cognition.2005.10.004

Green, C. S., Sugarman, M. A., Medford, K., Klobusicky, E., & Bavelier, D. (2012). The effect

of action video game experience on task-switching. Computers In Human Behavior,

28(3), 984-994. doi:10.1016/j.chb.2011.12.020

Hattie, J. (2009). Visible learning. New York: Routledge.


Hayes, W. L. (1988). Statistics (4t ed). New York: Holt, Rinehart, and Winston.

Honey, M. A. & Hilton, M. (2011). Learning Science Through Computer Games and

Simulations. Washington D.C.: National Acadamies Press.


Judd, C. H. (1908). The relation of special training and general intelligence. Educational Review,

36, 28-42.
Katona, G. (1940). Organizing and memorizing. New York: Columbia

University Press.


Li, R., Polat, U., Makous, W., & Bavelier, D. (2009). Enhancing the contrast sensitivity function

through action video game training. Nature Neuroscience, 12(5), 549-551.

doi:http://dx.doi.org/10.1038/nn.2296


Lieberman, D. A., Biely, E., Thai, C. L., & Peinado, S. (2014). Transfer of learning from video

game play to the classroom. In F. C. Blumberg (Ed.), Learning by playing: Video gaming

in education (pp. 189-203). New York, NY, US: Oxford University Press.

Mayer, R. E. (1987). The elusive search for teachable aspects of problem solving. In J. A. Glover

& R. R. Ronning (Eds.), Historical Foundations of Educational Psychology (pp. 327-

347). New York: Plenum.

Mayer, R. E. (2011). Multimedia learning and games. In S. Tobias & J. D. Fletcher (Eds.)
 LEARNING BY PLAYING GAMES 28

Computer Games and Instruction (pp. 281-306). Charlotte, NC: Information Age

Publishing.

Mayer, R. E. (2014). Computer games for learning: An evidence-based approach. MIT Press,

Cambridge, MA.

Mayer, R. E., & Wittrock, M. C. (1996). Problem solving and transfer. In D. Berliner & R.

Calfee (Eds.), Handbook of educational psychology (pp. 45-61). New York: Macmillan.

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., & Howerter, A. (2000). The unity

and diversity of executive functions and their contributions to complex 'frontal lobe'

tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49-100.

Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003), A brief introduction to evidence-based

design. ETS Research Report Series, RR-03-16. Princeton, NJ: Educational Testing

Service. doi:10.1002/j.2333-8504.2003.tb01908.x

Okagaki, L. & Frensch, P. (1994). Effects of video game playing on measures of spatial

performance: Gender effects in late adolescence. Journal of Applied Developmental

Psychology, 15, 33-58.

O’Neil, H. F., & Perez, R. S. (Eds.). (2008). Computer games and team and individual learning.

Oxford, UK: Elsevier.

Pashler, H., & Wagenmakers, E.J. (2012). Editor's introduction to the special issue on

replicability in psychological science: A crisis in confidence? Perspectives in

Psychological Science, 7, 528-530.


Pellegrino, J. G., & Hilton, M. (Eds). (2012). Education for life and work: Developing

st
transferable knowledge and skills in the 21 century. Washington, DC: National

Academies Press.

 LEARNING BY PLAYING GAMES 29

Pop Cap Games. (2002). Bookworm. Electronic Arts, Pop Cap. Retrieved from

http://www.popcap.com/bookworm


Powers, K. L., Brooks, P. J., Aldrich, N. J., Palladino, M. A., & Alfieri, L. (2013). Effects of

video-game play on information processing: A meta-analytic investigation. Psychonomic

Bulletin & Review, 20(6), 1055-1079. doi:http://dx.doi.org/10.3758/s13423-013-0418-z

Powers, K. L., & Brooks, P. J. (2014). Evaluating the specificity of effects of video game

training. In F. C. Blumberg (Ed.), Learning by playing: Video gaming in education. (pp.

302-329) New York, NY: Oxford University Press.


Prensky, M. (2006). Don’t Bother Me Mom, I’m Learning!. St Paul., MN: Paragon House.

Shaffer, D. (2006). How Computer Games Help Children Learn. New York, NY: Palgrave

Macmillan.
Shavelson, R. J., & Towne, L. (2002). Scientific research in education.

Washington, DC: National Academy Press.


Sims, V. K., & Mayer, R. E. (2002). Domain specificity of spatial expertise: The case of video

game players. Applied Cognitive Psychology, 16, 97-115.


Singley, M. K., & Anderson, J. R. (1989). The transfer of cognitive skill. Harvard University

Press, Cambridge, MA.


St Clair-Thompson, H. L., & Gathercole, S. E. (2006). Executive functions and achievements in

school: Shifting, updating, inhibition, and working memory. The Quarterly Journal of

Experimental Psychology, 59(4), 745-759.

doi:http://dx.doi.org/10.1080/17470210500162854


Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication.

Perspectives on Psychological Science, 9(1), 59-71.


 LEARNING BY PLAYING GAMES 30

Subrahamanyam, K. & Greenfield, P. M. (1994). Effect of video game practice on spatial skills

in girls and boys. Journal of Applied Developmental Psychology, 15, 13-32.


Terlecki, M. S., Newcombe, N. S., & Little, M. (2008). Durable and generalized effects of spatial

experience on mental rotation: Gender differences in growth patterns. Applied Cognitive

Psychology, 22(7), 996-1013. doi:http://dx.doi.org/10.1002/acp.1420


Thorndike, E. L. (1906). Principles of teaching. New York: Seiler.


Thorndike, E. L., & Woodworth, R. S. (1901). The influence of improvement in one mental

function upon efficiency of other functions. Psychological Review, 8, 247-261.


Tobias, S., Fletcher, J.D., Dai, D.Y., Wind, A.P. (2011). Review of research on computer games.

In: Tobias, S., Fletcher, J.D. (eds.) Computer Games and Instruction (pp. 127–222).

Charlotte, NC: Information Age Publishing.


Wertheimer, M. (1945). Productive thinking. New York: Harper & Row.


Whitlock, L.A., McLaughlin, A., & Allaire, J.C. (2012). Individual differences in response to

cognitive training: Using a multi-modal, attentionally demanding game-based

intervention for older adults. Computers In Human Behavior, 28(4), 1091-1096.

doi:10.1016/j.chb.2012.01.012

Zelazo, P. D. (2006). The Dimensional Change Card Sort (DCCS): A method of assessing

executive function in children. Nature Protocols, 1(1), 297-301.

Zelazo, P. D., Anderson, J. E., Richler, J., Wallner-Allen, K., Beaumont, J. L., Conway, K. P., ...

Weintraub, S. (2014). NIH Toolbox Cognition Battery (CB): validation of executive

function measures in adults. Journal of the International Neuropsychological Society,

20(6), 620–9. https://doi.org/10.1017/S1355617714000472

Zelazo, P. D., Anderson, J. E., Richler, J., Wallner-Allen, K., Beaumont, J. L., & Weintraub, S.
 LEARNING BY PLAYING GAMES 31

(2013). National institutes of health toolbox cognition battery (NIH toolbox CB):

Validation for children between 3 and 15 years: II. NIH toolbox cognition battery (CB):

Measuring executive function and attention. Monographs of the Society for Research in

Child Development, 78(4), 16-33.

 LEARNING BY PLAYING GAMES 32

Footnote

1. Composite shifting gain scores were calculated by first subtracting the pre-test scores from

post-test scores in the DCCS and post-test scores from pre-test scores in the LN (as lower scores

indicate better performance) for each player, and then adding the standardized gain scores for

both tasks and dividing by 2. In Experiment 1, a t-test showed that the Alien Game group (M =

0.07, SD = 0.50) had a significantly higher composite gain scores than the Bookworm group (M=

-0.26, SD = 0.54, t = 2.09, p = .043, d = 0.62). In Experiment 2, a t-test showed that the Alien

Game group (M = 0.06, SD = 0.59) did not have significantly different composite gain scores

than the Bookworm group (M = -0.01, SD = 0.44, t = 0.45, p = .660).

2. There was no correlation between the composite pre-test shifting scores and subsequent game

performance in Alien Game in both Experiment 1 (r = 0.37, p = .060) and Experiment 2 (r =

0.37, p = .067). An ANOVA revealed that there were no differences in composite pretest scores

between the high Alien Game group (M = 0.38, SD = 0.48), low Alien game group (M = -0.23,

SD = 1.07), and bookworm group (M = 0.01, SD = 0.57, F = 2.41, p = .102). However, in

Experiment 2, there were differences in composite pretest scores between the high Alien Game

group (M = 0.26, SD = 0.60), low Alien game group (M = -0.45, SD = 0.61), and Bookworm

group (M = -0.01, SD = 0.62, F = 3.43, p = .041); post hoc tests revealed there was a significant

difference in pre-test scores between the low and high Alien game groups (p =.012).
 LEARNING BY PLAYING GAMES 33

Table 1

Mean Score and Standard Deviation on Two Cognitive Tests of Shifting for the Alien Game

Group and the Bookworm Group in Experiment 1 and 2

Experiment 1 Experiment 2

(2 hour training) (1 hour training)

Pre-test Post-test Pre-test Post-test

M SD M SD M SD M SD

DCCS

Alien Game 7.84 1.10 8.03 1.08 7.66 0.99 8.08 0.57

Bookworm 7.89 0.46 7.96 0.62 7.75 0.60 7.86 0.51

d = .08 d = .41

LN

Alien Game 514.69 290.54 294.72* 215.75 525.71 327.52 378.49 249.53

Bookworm 584.47 313.66 487.07 249.40 546.69 283.41 326.87 301.13

d = .82 d = .19

Composite

Alien Game 0.09 0.85 0.18* 0.79 -0.02 0.77 0.05 0.84

Bookworm 0.01 0.57 -0.25 0.58 -0.01 0.62 -0.05 0.73

d = .62 d = .13

Note: * indicates the Alien Game group performed significantly better than the Bookworm group

at p <.05. DCCS = Dimensional Change Card Sort task; LN = Letter Number task.; Composite =

average of standardized scores on DCCS and LN. For the LN task, lower score is better.
 LEARNING BY PLAYING GAMES 34

Table 2

Mean Score and Standard Deviation on Two Cognitive Tests of Shifting for Highly Challenged

Students in the Alien Game Group and the Bookworm Group in Experiment 1 and 2

Experiment 1 Experiment 2

(2 hour training) (1 hour training)

Pre-test Post-test Pre-test Post-test

M SD M SD M SD M SD

DCCS

High AG 8.20 0.45 8.45 0.31 8.02 0.53 8.25 0.57

Bookworm 7.89 0.46 7.96 0.62 7.75 0.60 7.87 0.51

d = 1.08 d = .70

LN

High AG 469.56 210.88 268.94* 161.24 474.60 290.10 307.21 231.74

Bookworm 584.47 313.66 487.07 249.40 546.69 283.41 326.87 301.13

d = 1.04 d = .07

Composite

High AG 0.38 0.48 .46* .39 0.26 0.60 .33 .77

Bookworm 0.01 0.57 -.25 .58 -0.01 0.62 -.05 .73

d = 1.44 d = .51

Note: * indicates the Alien Game group performed significantly better than the Bookworm group

at p <.05. DCCS = Dimensional Change Card Sort task; LN = Letter Number task. For LN task,

lower score is better.

 LEARNING BY PLAYING GAMES 35

Table 3

Mean Score and Standard Deviation on Two Cognitive Tests of Shifting for the Goal, No Goal,

and Inactive Control groups in Experiment 3

Pre-test Post-test

M SD M SD

DCCS

Goal 8.36 0.64 8.61 0.63

No Goal 8.01 1.41 8.71 0.61

Control 8.20 0.60 8.16 0.70

LN

Goal 437.73 223.80 339.97 221.82

No Goal 542.37 415.30 391.38 193.28

Control 559.10 279.22 457.95 203.87

Composite

Goal 0.23 0.47 0.18 0.79

No Goal -0.09 0.92 0.14 0.75

Control -0.04 0.57 -0.43 0.76

Note. DCCS = Dimensional Change Card Sort task; LN = Letter Number task.; Composite =

average of standardized scores on DCCS and LN. For LN task, lower score is better.
 LEARNING BY PLAYING GAMES 36

Figure 1. Sample trial of the Dimensional Card Change Sort task (DCCS).
 LEARNING BY PLAYING GAMES 37

Figure 2. Sample trial of Letter-Number task (LN).

 LEARNING BY PLAYING GAMES 38

Figure 3. Example of Alien Game gameplay

 LEARNING BY PLAYING GAMES 39

Figure 4. Example of Bookworm gameplay.

 LEARNING BY PLAYING GAMES 40

Highlights

Developed a custom game aimed at teaching the executive function skill of shifting.

Selected a control game that did not require shifting.

Playing the custom game improved shifting skill more than playing the control game.

Shifting effects were found for 2 hours of game playing but not for 1 hour.

Shifting effects were stronger for players who reached higher levels in the custom game.