Assessing Performance Competence in Training Games

Hiran Ekanayake1,2,3, Per Backlund1, Tom Ziemke1, Robert Ramberg2, and Kamalanath Hewagamage3,
University of Skvde, P.O. Box 408, SE-541 28 Skvde, Sweden 2 Stockholm University, Forum 100, SE-164 40 Kista, Sweden 3 University of Colombo School of Computing, 35, Reid Avenue, Colombo 7, Sri Lanka hiran@dsv.su.se, {per.backlund, tom.ziemke}@his.se, robban@dsv.su.se, kph@ucsc.cmb.ac.lk
1

Abstract. In-process assessment of trainee learners in game-based simulators is a challenging activity. This typically involves human instructor time and cost, and does not scale to the one tutor per learner vision of computer-based learning. Moreover, evaluation from a human instructor is often subjective and comparisons between learners are not accurate. Therefore, in this paper, we propose an automated, formula-driven quantitative evaluation method for assessing performance competence in serious training games. Our proposed method has been empirically validated in a game-based driving simulator using 7 subjects and 13 sessions, and accuracy up to 90.25% has been achieved when compared to an existing qualitative method. We believe that by incorporating quantitative evaluation methods like these future training games could be enriched with more meaningful feedback and adaptive game-play so as to better monitor and support player motivation, engagement and learning performance. Keywords: Serious Games, Performance Evaluation, Motivation, Driver Training

Introduction

There is an increasing demand for the use of computer and video games for purposes other than pure entertainment for example, education, skill training and health care. Proponents of such serious games claim three major benefits: they are intrinsically motivating, they provide immediate feedback to users and they provide ample learning opportunities [1].There is no doubt that advancement in rendering of graphics and realistic simulation of real-world aspects in games from the viewpoint of the player character (referred to as the first-person perspective) have contributed to this tremendous growth in the game play community. However, the use of gamebased technologies for education and training purposes is still hindered by lack of methods and tools to measure and relate its learning effect to other, more conventional types of training/learning. For instance, in classroom-based learning, assessment is conducted in two ways called formative and summative. However, in game-based learning these assessment methods become inappropriate in the same

sense, because negative feedback could cause a cautious learner to lose motivation and to stop playing. Despite the abovementioned benefits, however, misconceptions and lack of awareness have caused a wide public opinion that educational use of computer and video games is not a valid alternative to classroom-based learning. The purpose of this paper is to propose an automated, formula-driven quantitative method to measure the learning effect, especially in the psychomotor skills domain, in serious training games, which is based on performance-oriented achievement motivation of a learner. Currently, assessment of performance competence in training games is often conducted by human instructors using subjective qualitative methods [6][22]. These methods have their own drawbacks in addition to not scaling enough to one instructor per student vision in game-based learning. We believe that our method provided a first step towards future research on adaptive game-play and real-world effects of learning. The paper is organized as follows: In section 2, we discuss the evolution of learning theories and aspects relevant to learning, such as the context and transfer, concerned to game-based learning environments and real-world. Section 3 focuses mainly on assessment methods, the connection between classroom-based learning and game-based learning, and a psychological perspective on the crucial role of motivation in learning. In section 4, we present the proposed formula-driven approach to assessing performance competence of a learner. Section 5 presents the steps involved in the empirical validation of the proposed formula-driven approach. In section 6, we discuss the results, findings, limitations and future work.

The Validity of Game-Based Simulator Training

Shuell (as interpreted by [2]) defined learning as follows: Learning is an enduring change in behavior, or in the capacity to behave in a given fashion, which results from practice or other forms of experience. The above definition accounts for behaviorist view of learning which virtually denies the existence of the mind. Therefore, it fails to give an account of internal cognitive processes involved [3] or acquisition of higher level skills, such as language development and problem solving [2]. On the contrary, contemporary school of learning theories falls in cognitive and constructivist camps which accounts for internal mental processes involved and environmental conditions [3]. Constructivism differs from cognitivism in its subjectivistic assumption, i.e. [knowledge] is a function of how the individual creates meaning from his or her own experiences [2]. According to constructivism the memory is always evolving with new uses and new situations through three processes known as externalism, internalism and incongruousness, and learning must include three crucial factors: activity (practice), concept (knowledge) and culture (context). Although Blooms taxonomy identifies three domains in which learning could occur, namely cognitive, affective and psychomotor [4], from the constructivist point of view any type of learning is explained as a change in a corresponding mental model [3].

Learning is successful if what is learnt can be successfully transferred to new situations and contexts. [5] explains two types of transfer mechanisms of learning in such contexts: low-road transfer and high-road transfer. The low-road transfer involves triggering of well-practiced semi-automatic responses (reflexive) as a result of similarities between prior and transfer learning contexts. The high-road transfer, on the other hand, is more likely to happen when the learning contexts are more remote and it involves mindful and searching of deliberate connections between the two learning contexts. A recent trend in learning is to use computer games for educational purposes. The formal study into this aspect is called serious games and it studies the use of computer games for purposes beyond pure entertainment, such as for education, skills training, health care, and military [1][6][7][8]. Among the other advantages, serious games with advanced simulator technologies offer repeating of critical situations, such as operation on slippery roads and night driving, which are otherwise dangerous with ordinary training sessions [7]. According to [9], the four primary learning principles in serious games are practicing skills, knowledge gain through exploration, cognitive problem solving, and social problem solving. Here our focus is mainly on skills training using serious training games. From a general perspective, all computer games represent alternate realities like in dreams [10]. Therefore, the serious games initiative is challenged by various parties either criticizing its theoretical foundation or practical usage. The primary advantage of serious games is in its entertainment value which serves to engage and motivate the learner to the gaming task. However, the entertainment value of a serious game depends on both the subject's performance and the complexity of the game [11]. Three complexity factors, physical realism of the environment and characters, affective realism of characters, and social complexity and realism of their interactions determine the complexity of the game [12]. For subject's performance the engagement to the gaming task is crucial, for which adaptive challenge is a necessity. [7] identifies this adaptive challenge and appropriate feedback as the two essential aspects in promoting self-efficacy. Moreover, engagement is considered as an important parameter which can improve the learning performance [13]. Although uncertainty remains on the validity of simulator-based training, [14] and [15] using two sufficiently complex game-based simulator training environments, a fire-fighter training simulator and a driving simulator, show that simulator-based training is indeed a valid alternative for classroom-based training. Simulator-based learning has also been characterized as facilitating model-based thinking, because it helps bridge between concrete experiences and more abstract and systematic understanding [16].

Learning Goals and Performance Evaluation in Game-Based Simulator Training

Assessment is a powerful tool that can be used to direct students' attention to what matters in their learning [17]. Conventional assessment methods designed for

classroom-based learning cannot be applied in game-based learning systems in the same sense. According to [18], educational serious games and classroom-based learning have connections between intersections of feedback, aesthetics and mechanics. In classroom-based learning feedback can be given in two forms: summative and formative. Summative feedback comes as a quantitative measure of the learners performance after a summative assessment in the completion of a learning session. In games, summative feedback is typically a kind of a scoring system, such as experience points, level progression, item collection, different endings, or just simple points. Experience point systems are considered very motivational in games; however, [18] find it difficult to convert a course grade system into an experience point system. The difference exists as games usually offer quick early leveling to motivate the players to stay within the game, whereas in courses the students have a failing grade through much of the course. Achievement-based systems can work well in both games and classroom, such as unlock special features in games or incentives in the classroom. Formative feedback is often a qualitative feedback given during the process of learning and it allows the learner to formulate opinions, share problem areas and reflect on how to accomplish his/her goals. In games formative feedback can result in adaptive game play, in which the game will become easier or harder depending on the players game play behavior. One important factor facilitating this adaptive game play is called the flow experience, in which the learner loses track of time and the outside world while completely absorbed by the activity he/she is currently engaged in [19][20]. Interruptions in flow cause the learner to leave the game. The flow is a result of constant cycle of cognitive disequilibrium and resolution, and this is to be created without foiling the expectations and exceeding the capacity of the player. Since the focus of this paper is on assessment of psychomotor performance in serious training games more emphasis is put towards aspects important for such games. Simulation games invite experimentation and exploration while improving strategic thinking and insight, psychomotor skills, analytical and spatial skills, iconic skills and visual selective attention [8][21]. However, when it comes to assessment, there is no appropriate computer-based assessment methodology designed for the needs of simulation-based serious training games. Currently the only valid assessment method is by an expert teacher monitoring the learner constantly or by inspecting application logs of user interactions [6][22], which becomes inappropriate when considering the expected benefits of digital game-based learning as reducing instructional time and costs [20] and facilitating an adaptive challenge. From the constructivist point-of-view learning is a subjectivistic mental process and achievement motivation plays a major role in its success. Although entertainment value of serious games is considered to serving motivation for gameplay, motivational researchers contrast this individual interest as intrinsic and situational. Intrinsic interest is a stable motivational construct that orients the individual to develop competence and task mastery, while situational interest is a transitory motivational construct which orients one's focus to a certain condition or stimuli in the environment which may or may not last for a longer period [19][23]. Since digital games are basically designed for situational interest, suitability of using digital games

for practicing skills is questionable [9]. Several researchers aim to identify the current emotional state, physiological state and many other aspects of the player and use these information in game dynamics to improve the game interaction which is studied under themes of affective feedback games and biofeedback games [24][25[26][27]. In a learning setting, learners adopt to different achievement goals. The specific type of achievement goal the learner has been adopted triggers relevant behavioral patterns including how an individual approach, engage in, and respond to achievement tasks [23][28]. This creates differences between learners. As discussed in the theory of academic risk taking [19], an adventuresome learner is a learner typically with high ability and want to be challenged with difficult tasks, take risks of failure, and even stable on the face of negative emotions. In contrast, a cautious learner prefers fewer and easier tasks and takes fewer risks. Achievement goal theorists contrast this difference as to which goals learners have adopted to: a performance goal focused on the demonstration of competence relevant to others, or a mastery goal focused on the development of competence and task mastery [28]. Further to their discussion, [28] identify three achievement orientations: mastery goal, performance-approach goal and performance-avoidance goal. Both mastery and performance-approach goals are characterized as self-regulation according to potential positive outcomes while performance-avoidance goal is characterized as self-regulation according to potential negative outcomes.

The Proposed Formula for the Assessment of Performance Competence

Our proposed method for the assessment of performance competence in training (assumed as an indirect way to measure the task mastery) is based on an evaluation of behavioral patterns of the player. We hypothesize that the achieved performance (P) of a player depends on favorable outcomes (F), unfavorable outcomes (U) and the physical effort (E), as given by the following formula, (1) Formula 1 distinguishes one learner who achieves a certain performance putting in less effort as compared to another learner who achieves the same training outcome putting in a greater effort. To account for an expert level of effort, which is characterized as neither more than required nor less than required effort, a measurement of optimal effort (O) is introduced to the formula 1, which now becomes, (2) Further, a threshold value is used to avoid division by zero and higher peaks when the divisor reaches zero.

5
5.1

The Empirical Validation of the Proposed Method

Data Collection

The virtual driving environment was designed using the VDrift open source racing simulator game engine [29]. Physically realistic steering wheel and pedals were used as the game controller and the video output has been projected to a large white-board using a multimedia projector. The player was sitting about two meters away from the screen. Two web cameras recorded the front screen with player's position from back and the player's position from front. A total of seven volunteers (four male and three female) of age between 25 and 44 with a mean age of 32 participated in the experiment. All of them have obtained their driving license, but differences exist in their driving experience. They were asked to play in the driving simulator considering it as a real-world like driving situation. Each participant played at least two sessions in the simulator, where each session consisted of three laps and taken approximately ten minutes to complete. Differences were made by changing the type of track between two possibilities. The VDrift game engine has been customized to render data representing the current game state to an external data logging program. The program recorded the players pressure to throttle, brake and steering wheel, as well as, speed of the car, crashes to walls and other cars, and off-the-track driving. 5.2 Data Analysis

The data analysis has been started by annotating recorded videos into segments of two types, straight road ahead (type 1) and turn/bend ahead (type 2) , as how the player would see the situation ahead, using Transana annotation tool [30]. Although, 14 sessions were recorded, one recording was dropped due to some errors in the recording, which resulted in 13 session recordings for the seven subjects. The evaluation of each player's driving behavior has been obtained using two methods: a qualitative method and the proposed formula-driven quantitative method. The qualitative assessment has been carried out by an author of this paper who has about 15 years of driving experience, which otherwise should have been done by an independent expert or a panel of evaluators. Table 1 lists the guidelines followed in the qualitative assessment of performance.
Table 1. Guidelines for the Qualitative Assessment Code Very good Good Average Value 2 1 0 Guidelines No observable mistake; near expert level of driving Car balancing not good; drive very closer to road boundary or slightly drift beyond boundary Car balancing is very bad; Crosses into boundaries, but not much disturbance on velocity/speed of the car; no damage to the car

7 Bad Very bad -1 -2 Go off the track and disturbances to driving; minor damage to the car and objects in the environment Crash into walls or other cars

The Formula-Driven Quantitative Method. The idea was to analyze each player's driving behavior for each session within the identified situation types. For example, player A's driving behavior in straight roads against player B's. However, data showed that the time spent on each situation differs significantly, because of different track geometries and player's condition at that period of time. To overcome this problem, segments of each type (1 and 2) have been further categorized into three categories (named as A, B and C) considering their length of time. All data processing has been done with the assistance of Matlab [31]. When substituting values in the formula 2, it was assumed that a player's favorable outcome is to achieve a good speed while minimizing unfavorable outcomes, such as crashes or car to go off-the-road. Effects of priority incidents (e.g. crash in to a wall) has been multiplied in the formula. A way to study psychologically significant learner behaviors and situations in game simulator-based driver training is presented in [32]. The players effort has been determined using the physical pressure the player puts in the game controller to control the situation in the game world, i.e. pressure to the throttle, the brake and the steering wheel. Since the current study has not been designed to find an accurate measurement for the optimal effort and the threshold, they have been determined by trial and error, for which both has lead to 0.1. 5.3 Results

Fig. 2 represents the relationships between qualitative method and proposed formuladriven method for two categories 1C and 2C.

Fig. 1. Relationships between the qualitative method and the proposed formula-driven method for the situation categories 1C and 2C

The correlation coefficients for the relationships between the qualitative method and the formula-driven method for the four situation categories 1B, 1C, 2B and 2C are 13.52%, 82.69%, 55.84% and 90.25%, respectively. The category A has been omitted from further analysis, because the number of segments was not sufficient.

Discussion and Conclusion

This paper has presented a formula-driven method to obtain a quantitative assessment of performance competence in serious training games. The proposed method has been validated against a qualitative method using 13 sessions and seven subjects in a driving simulator game. In addition to the formula, the paper has discussed its empirical validation, which has included the following steps: identify different types of situations and categories, determine values for variables in the formula, and compare obtained results against an existing qualitative approach. However, for simplicity, only a certain number of game and user action variables have been considered. Moreover, the two parameters, optimal performance and threshold, were determined using trial and error. However, the comparison has reported as high as 82.69% and 90.25% correlation coefficients respectively for 1C and 2C cases and low to moderate 13.52% and 55.84% correlation coefficients respectively for 1B and 2B cases. The low values for correlation coefficients may be because fewer numbers of situations to compare with or the length of segments is shorter than that for the category C. On the other hand, it may be because less accuracy of the qualitative method. Future research is planned to improve the formula and validate it against different types of training games, learning aspects, situations, learners and applicability in realworld. Another improvement in the formula to stand in general-purpose games is to

extend the definition of favorable and unfavorable outcomes and their means of verification (which also depends on the particular game in question). Moreover, we are in the process of collecting psychophysiological signals to determine learners mindful interaction and effort to accurately determine the engagement and effort. Apart from that, our method could be improved to include learners affect or satisfaction, as measured by their facial expressions or other indicators, or by questionnaires. Thereby, we hope that the method can be enhanced to capture more learning aspects into the formula, such as exploration, experiencing, self-efficacy and flow. Acknowledgments. The authors wish to sincerely thank the staff of InGaMe lab at University of Skvde and all participants in the experiment.

References
1. Ritterfeld, U., Cody, M., Vorderer, P.: Serious Games: Explication of an Oxymoron: Introduction. In: Ritterfeld, U., Cody, M., Vorderer, P. (eds.) Serious Games: Mechanics and Effects, pp. 3--9. New York: Routledge (2009) 2. Ertmer, P., Newby, T.: Behaviorism, cognitivism, constructivism: comparing critical features from an instructional design perspective. In: Performanc. Improvement Quarterly, 6(4): pp. 50--72 (1993) 3. Goel, L., Johnson, N., Junglas, I., Ives, B.: Situated Learning: Conceptualization and Measurement. In: Decision Sciences Journal of Innovative Education, vol. 8, issue 1, pp. 215--240 (2010) 4. Clark, D.: Bloom's Taxonomy of Learning Domains: The Three Types of Learning, http://www.skagitwatershed.org/~donclark/hrd/bloom.html 5. Perkins, D. N., Salomon, G.: Transfer of learning. In: International Encyclopedia of Education, 2nd ed., Oxford, UK: Pergamon Press (1992) 6. Brennecke, A.: A General Framework for Digital Game-Based Training Systems. In: Doctoral thesis, University of Rostock (2009) 7. Backlund, P., Engstrm, H., Johannesson, M., Lebram, M.: Games for traffic education: An experimental study in a game-based driving simulator. In: Simulation & Gaming, vol. 41, issue 2, DOI: 10.1177/1046878107311455 (2010) 8. Mitchell, A., Savill-Smith, C.: The use of computer and video games for learning: A review of the literature. Learning and Skills Development Agency, pp. 1--93, London (2004) 9. Ratan, R., Ritterfeld, U.: Serious Games: Explication of an Oxymoron: Classifying Serious Games. In: Ritterfeld, U., Cody, M., Vorderer, P. (eds.) Serious Games: Mechanics and Effects, pp. 10--24. New York: Routledge (2009) 10. Hsu, J.: Video Games and Control Dreams, Study Suggests, http://www.livescience.com/culture/video-games-control-dreams-100525.html 11. Piselli, P., Claypool, M., Doyle, J.: Relating Cognitive Models of Computer Games to User Evaluations of Entertainment. In: 4th International Conference on Foundations of Digital Games, Orlando, Florida (2009) 12. Hudlicka, E.: Affective Game Engines: Motivation and Requirements. In: 4th International Conference on Foundations of Digital Games, Orlando, Florida (2009) 13. Carini, R. M., Kuh, G. D., Klein, S. P.: Student Engagement and Student Learning: Testing the Linkages. In: Research in Higher Education, 47 (1), pp. 1--32 (2006)

10 14. Backlund, P., Engstrom, H., Hammar, C., Johannesson, M., Lebram, M.: Sidh - a Game Based Firefighter Training Simulation. In: 11th International Conference Information Visualization (IV '07), pp. 899--907 (2007) 15. Backlund, P., Engstrm, H., Johannesson, M., Lebram, M., Sjdn, B.: Designing for selfefficacy in a game based simulator: An experimental study and its implications for serious games design. In: Bannatyne, M. et al. (eds.) Vis2008, pp. 106--113, ISBN: 978-0-76953271-4, London, UK (2008) 16. Gee, J. P.: Theories and Mechanisms: Deep Learning Properties of Good Digital Games: How Far Can They Go? In: Ritterfeld, U., Cody, M., Vorderer, P. (eds.) Serious Games: Mechanics and Effects, pp. 67--82. New York: Routledge (2009) 17. Gibbs, G., Habeshaw, T.: Preparing to Teach: An Introduction to Effective Teaching in Higher Education. Technical and Educational Services Ltd., Bristol, UK (1992). 18. Bayliss, J. D., Schwartz, D. I.: Instructional Design as Game Design. In: 4th International Conference on Foundations of Digital Games, Orlando, Florida (2009) 19. Graesser, A., Chipman, P., Leeming, F., Biedenbach, S.: Theories and Mechanisms: Deep Learning and Emotion in Serious Games. In: Ritterfeld, U., Cody, M., Vorderer, P. (eds.) Serious Games: Mechanics and Effects, pp. 83--102. New York: Routledge (2009) 20. Eck, R. V.: Digital Game-Based Learning: It's Not Just the Digital Natives Who Are Restless. In: EDUCAUSE Review, vol. 41, no. 2, pp. 16--30 (2006) 21. Debul, P.: Game on!: Game-based learning. In: T.H.E. Journal, 33(6), 41, pp. 30--33, http://www.thejournal.com/articles/17788 22. Picard, R., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D. et al.: Affective learning: A manifesto. In: BT Technology Journal, 22(4), pp. 253264 (2004) 23. Hidi, S., Harackiewicz, J. M.: Motivating the academically unmotivated: A critical issue for the 21st century. In: Review of Educational Research, vol. 70, no. 2, pp. 151--179 (2000) 24. Gilleade, K. M., Dix, A., Allanson, J.: Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me. In: DiGRA 2005, Vancouver, BC, Canada (2005) 25. Bersak, D., McDarby, G., Augenblick, N., McDarby, P., McDonnell, D., MdDonald, B., Karkun, R.: Intelligent Biofeedback using an Immersive Competitive Environment. In: Designing Ubiquitous Computing Games Workshop at UbiComp 2001, Atlanta, GA, USA (2001) 26. Kim, J., Bee, N., Wagner, J., Andr, E.: Emote to win: affective interactions with a computer game agent. In: GI Jahrestagung: (1), pp. 159--164 (2004) 27. Fairclough, S. H.: Psychophysiological Inference and Physiological Computer Games. In: Brainplay07: Brain-Computer Interfaces and Games. Advances in Computer Entertainment, Salzburg, Austria (2007) 28. Elliot, A. J., Church, M. A.: A hierarchical model of approach and avoidance achievement motivation. In: Journal of Personality and Social Psychology, vol. 72, no. 1, pp. 218232 (1997) 29. VDrift.net, http://vdrift.net/ 30. Transana version 2.42, http://www.transana.org/ 31. MATLAB version 7.6.0. Natick, Massachusetts: The MathWorks Inc. (2003) 32. Ekanayake, H., Backlund, P., Ziemke, T., Ramberg, R., Hewagamage, K.: Game Interaction State Graphs for Evaluation of User Engagement in Explorative and Experience-based Training Games. In: International Conference on Advances in ICT for Emerging Regions (ICTer), Colombo, Sri Lanka (2010)