11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

FDG 

RESEARCH-ARTICLE
    
Moment-to-moment Engagement Prediction through the Eyes of the
Observer: PUBG Streaming on Twitch

Authors: David Melhart, Daniele Gravina, Georgios N. Yannakakis

Authors Info & Claims

FDG '20: Proceedings of the 15th International Conference on the Foundations of Digital Games
• September 2020 • Article No.: 60 • Pages 1–10 • https://doi.org/10.1145/3402942.3402958

This website
Published: uses cookies
17 September 2020 Publication History
We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
 8 information
 271that you’ve provided to them or that they’ve collected from your use of their services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.

Use necessary cookies only Allow selected cookies

  Allow all
cookies
Get Access

Necessary Preferences Statistics Marketing Show details

FDG '20: Proceedings of the 15th International Conference on the Foundations of Digital Games
Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on
Twitch
Pages 1–10

Previous Next



ABSTRACT

Is it possible to predict moment-to-moment gameplay engagement based solely on game telemetry?

Can we reveal engaging moments of gameplay by observing the way the viewers of the game behave?

To address these questions in this paper, we reframe the way gameplay engagement is defined and we

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 1/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

view it, instead, through the eyes of a game’s live audience. We build prediction models for viewers’

engagement based on data collected from the popular battle royale game PlayerUnknown’s

Battlegrounds as obtained from the Twitch streaming service. In particular, we collect viewers’ chat logs
FDG 
and in-game telemetry data from several hundred matches of five popular streamers (containing over

100,000 game events) and machine learn the mapping between gameplay and viewer chat frequency
during play, using small neural network architectures. Our key findings showcase that engagement

models trained solely on 40 gameplay features can reach accuracies of up to 80% on average and 84%

at best. Our models are scalable and generalisable as they perform equally well within- and across-

streamers, as well as across streamer play styles.

References

1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay
Ghemawat, Geoffrey Irving, Michael Isard, 2016. Tensorflow: A system for large-scale machine learning. In 12th
USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265–283.
This website uses cookies
We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
2. Sander CJ Bakkes,
information about yourPieter HMsiteSpronck,
use of our andmedia,
with our social Giel advertising
van Lankveld. 2012.
and analytics Player
partners whobehavioural
may combine itmodelling
with other for video
information that you’ve provided to them or
games. Entertainment Computing 3, 3 (2012), 71–79.that they’ve collected from your use of their services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.

3. Francesco Barbieri, Luis Espinosa Anke, Miguel Ballesteros, Juan Soler, and Horacio Saggion. 2017.
Use necessary cookies only Allow selected cookies Allow all cookies
Towards the understanding of gaming audiences by modeling twitch emotes. In Proceedings of the 3rd
WorkshopNecessary Preferences Text.
on Noisy User-generated Statistics
11–20. Marketing Show details

4. Richard Bartle. 1996. Hearts, clubs, diamonds, spades: Players who suit MUDs. Journal of MUD research 1,
1 (1996), 19.

5. Christian Bauckhage, Anders Drachen, and Rafet Sifa. 2015. Clustering game behavior data. IEEE
Transactions on Computational Intelligence and AI in Games 7, 3(2015), 266–278.

6. Rafael A Calvo, Sidney D’Mello, Jonathan Matthew Gratch, and Arvid Kappas. 2015. The Oxford handbook
of affective computing. Oxford University Press, USA.

7. Alessandro Canossa, Sasha Makarovych, Julian Togelius, and Anders Drachenn. 2018. Like a DNA String:
Sequence-Based Player Profiling in Tom Clancy’s The Division. In Proceedings of the 14th Artificial Intelligence
and Interactive Digital Entertainment Conference.

8. Peter Carruthers and Peter K Smith. 1996. Theories of theories of mind. Cambridge University Press.

9. Peter Chapman, Sanjeebhan Selvarajah, and Jane Webster. 1999. Engagement in multimedia training
systems. In Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999.
HICSS-32. Abstracts and CD-ROM of Full Papers. IEEE, 9–pp.

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 2/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

10. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network
learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289(2015).

11. Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations.
FDG 
In Proceedings of the 10th ACM conference on recommender systems. ACM, 191–198.

12. James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta,
Yu He, Mike Lambert, Blake Livingston, 2010. The YouTube video recommendation system. In Proceedings of
the fourth ACM conference on Recommender systems. ACM, 293–296.

13. Anders Drachen, Alessandro Canossa, and Georgios N Yannakakis. 2009. Player modeling using self-
organization in Tomb Raider: Underworld. In 2009 IEEE symposium on computational intelligence and games.
IEEE, 1–8.

14. Anders Drachen and Shawn Connor. 2018. Games Analytics for Games User Research. In Games User
Research, Anders Drachen, Pejman Mirza-Babaei, and Lennart E Nacke (Eds.). Oxford University Press, 333–
355.

15. Magy Seif El-Nasr, Anders Drachen, and Alessandro Canossa. 2016. Game analytics. Springer. 792 pages.

This website uses cookies

We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
16. Colin Ford,about
information Danyour
Gardner, Leah
use of our Elaine
site with Horgan,
our social media,Calvin Liu,
advertising AM
and Tsaasan,
analytics partnersBonnie
who mayNardi,
combineand Jordan
it with other Rickman.
information that you’ve provided to them or that they’ve collected from your use of their
2017. Chat speed op pogchamp: Practices of coherence in massive twitch chat. In Proceedings of theservices. Use the check boxes below to 2017
choose the types of cookies you consent to have stored on your device.
CHI Conference Extended Abstracts on Human Factors in Computing Systems. 858–871.

Use necessary cookies only Allow selected cookies Allow all cookies
17. William A Hamilton, Oliver Garretson, and Andruid Kerne. 2014. Streaming on twitch: fostering participatory
Necessary
communities Preferences
of play within Statistics
live mixed media. Marketing
In Proceedings of the SIGCHI conferenceShow details factors in
on human
computing systems. 1315–1324.

18. Morgan Jennings. 2000. Theory and models for creating engaging and immersive ecommerce websites. In
Proceedings of the 2000 ACM SIGCPR conference on Computer personnel research. ACM, 77–85.

19. Nicole Lazarro. 2004. Why we play games: Four keys to more emotion without story. In Game developer’s
conference, San Jose.

20. Antonios Liapis, Daniele Gravina, Emil Kastbjerg, and Georgios N Yannakakis. 2019. Modelling the Quality
of Visual Creations in Iconoscope. In International Conference on Games and Learning Alliance. Springer,
129–138.

21. Lassi A Liikkanen. 2013. Three Metrics for Measuring User Engagement with Online Media and a YouTube
Case Study. arXiv preprint arXiv:1312.5547(2013).

22. Tobias Mahlmann, Anders Drachen, Julian Togelius, Alessandro Canossa, and Georgios N Yannakakis.
2010. Predicting player behavior in Tomb Raider: Underworld. In Proceedings of the Symposium on
Computational Intelligence and Games (CIG). IEEE, 178–185.

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 3/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

23. Konstantinos Makantasis, Antonios Liapis, and Georgios N Yannakakis. 2019. From Pixels to Affect: A
Study on Games and Player Experience. In 2019 8th International Conference on Affective Computing and
Intelligent Interaction (ACII). IEEE, 1–7.

FDG 
24. Hector P Martinez, Georgios N Yannakakis, and John Hallam. 2014. Don’t classify ratings of affect; rank
them!IEEE transactions on affective computing 5, 3 (2014), 314–326.

25. Akhil Mathur, Nicholas D Lane, and Fahim Kawsar. 2016. Engagement-aware computing: Modelling user
engagement from mobile contexts. In Proceedings of the 2016 ACM International Joint Conference on
Pervasive and Ubiquitous Computing. ACM, 622–633.

26. David Melhart, Ahmad Azadvar, Alessandro Canossa, Antonios Liapis, and Georgios N Yannakakis. 2019.
Your Gameplay Says it All: Modelling Motivation in Tom Clancy’s The Division. Proceedings of IEEE
Conference on Games (CoG) (2019).

27. David Melhart, Antonios Liapis, and Georgios N Yannakakis. 2019. PAGAN: Platform for Audiovisual
General-purpose ANnotation. In 2019 8th International Conference on Affective Computing and Intelligent
Interaction Workshops and Demos (ACIIW). IEEE, 75–76.

28. David Melhart, Antonios Liapis, and Georgios N Yannakakis. 2019. PAGAN: Video Affect Annotation Made
Easy.This website
In 2019 uses cookies
8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE,
We
130–136.occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
information that you’ve provided to them or that they’ve collected from your use of their services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.
29. Ilya Musabirov, Denis Bulygin, Paul Okopny, and Ksenia Konstantinova. 2018. Between an arena and a
sports bar: Online chats of esports spectators. arXiv preprint arXiv:1801.02862(2018).
Use necessary cookies only Allow selected cookies Allow all cookies

Necessary
30. Heather Preferences
L O’Brien and Statistics
Elaine G Toms. 2008. WhatMarketing Showframework
is user engagement? A conceptual details for
defining user engagement with technology. Journal of the American society for Information Science and
Technology 59, 6 (2008), 938–955.

31. Rui Pan, Lyn Bartram, and Carman Neustaedter. 2016. TwitchViz: a visualization tool for twitch chatrooms.
In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems.
ACM, 1959–1965.

32. África Periáñez, Alain Saas, Anna Guitart, and Colin Magne. 2016. Churn prediction in mobile social
games: towards a complete assessment using survival ensembles. In Proceedings of the International
Conference on Data Science and Advanced Analytics (DSAA). 564–573.

33. Matthias Rauterberg. 1995. About a framework for information and information processing of learning
systems. In Information System Concepts. Springer, 54–69.

34. Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster
analysis. Journal of computational and applied mathematics 20 (1987), 53–65.

35. Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell system technical journal 27,
3 (1948), 379–423.

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 4/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

36. Max Sjöblom and Juho Hamari. 2017. Why do people watch others play video games? An empirical study
on the motivations of Twitch users. Computers in Human Behavior 75 (2017), 985–996.

37. Max Sjöblom, Maria Törhönen, Juho Hamari, and Joseph Macey. 2017. Content structure is king: An
FDG 
empirical study on gratifications, game genres and content type on Twitch. Computers in Human Behavior 73
(2017), 161–171.

38. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.
Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research
15, 1 (2014), 1929–1958.

39. Ruck Thawonmas and Tomohiro Harada. 2017. AI for Game Spectators: Rise of PPG. In Workshops at the
Thirty-First AAAI Conference on Artificial Intelligence.

40. Elaine G Toms. 2002. Information interaction: Providing a framework for information architecture. Journal of
the American Society for Information Science and Technology 53, 10 (2002), 855–862.

41. Markus Viljanen, Antti Airola, Jukka Heikkonen, and Tapio Pahikkala. 2018. Playtime measurement with
survival analysis. IEEE Transactions on Games 10, 2 (2018), 128–138.

This website uses cookies

42. Joe H Ward Jr. 1963. Hierarchical grouping to optimize an objective function. Journal of the American
We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
statistical association
information 58,
about your use301(1963),
of our site with236–244.
our social media, advertising and analytics partners who may combine it with other
information that you’ve provided to them or that they’ve collected from your use of their services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.
43. Sarah Wassermann, Nikolas Wehner, and Pedro Casas. 2019. Machine Learning Models for YouTube QoE
and User Engagement Use
Prediction in Smartphones.
necessary cookies only ACM SIGMETRICS
Allow Performance Evaluation
selected cookies Review 46, 3
Allow all cookies
(2019), 155–158.
Necessary Preferences Statistics Marketing Show details

44. Tim Wulf, Frank M Schneider, and Stefan Beckert. 2018. Watching players: An exploration of media
enjoyment on Twitch. Games and Culture (2018), 1555412018788161.

45. Georgios N Yannakakis, Roddy Cowie, and Carlos Busso. 2017. The Ordinal Nature of Emotions. In 2017
Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 248–255.

46. Georgios N Yannakakis, Roddy Cowie, and Carlos Busso. 2018. The Ordinal Nature of Emotions: An
Emerging Approach. IEEE Transactions on Affective Computing(2018).

47. Georgios N Yannakakis, Pieter Spronck, Daniele Loiacono, and Elisabeth André. 2013. Player modeling. In
Dagstuhl Follow-Ups, Vol. 6. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

48. Georgios N. Yannakakis and Julian Togelius. 2018. Artificial Intelligence and Games. Springer Nature.
http://gameaibook.org.

49. Nick Yee. 2006. Motivations for play in online games. CyberPsychology & behavior 9, 6 (2006), 772–775.

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 5/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

Recommendations

Streaming on twitch: fostering participatory communities of play within live mixed media
FDG 
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Previously, video streaming sites were at the fringes of online social media. In the past two years, live streams of video
games, on sites such as Twitch.tv, have become very popular. Live streams serve as meeting grounds for player…

Read More
Social motivations of live-streaming viewer engagement on Twitch

Abstract
Little is known about the motivations underlying viewer engagement in the rapidly growing live-streaming multimedia…
Highlights
Live-stream viewer engagement was explored from a socio-motivational perspective.

Read More
The Role of Uncertainty in Moment-to-Moment Player Motivation: A Grounded Theory
CHI PLAY '19: Proceedings of the Annual Symposium on Computer-Human Interaction in Play

Uncertainty is widely acknowledged as an engaging characteristic of games. Practice and research have proposed various
types and factors of game uncertainty, yet there is little work explaining when and why different kinds of uncertainty motivat…

Read More
This website uses cookies
We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
information that you’ve provided to them or that they’ve collected from your use of their services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.
Comments
Use necessary cookies only Allow selected cookies Allow all cookies

Login options Preferences

Necessary Statistics Marketing Show details

Check if you have access through your login credentials or your institution to get full access on this
article.

Sign in

Full Access

Get this Publication

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 6/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

FDG 

Information Contributors

Published inuses cookies

This website
We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
FDG '20: Proceedings of the 15th International
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
information that you’ve provided to them or that they’ve collected from your use of their services. Use the check boxes below to
Conference on the Foundations of Digital Games
choose the types of cookies you consent to have stored on your device.

September 2020 804 pages

Use necessary cookies only Allow selected cookies Allow all cookies

Necessary ISBN:
Preferences 9781450388078
Statistics Marketing Show details

DOI: 10.1145/3402942

Editors: Georgios N. Yannakakis, Antonios Liapis,

Penny Kyburz, Vanessa Volz, Foaad Khosmood,

Phil Lopes

Copyright © 2020 ACM

Permission to make digital or hard copies of all or part of this work for personal or classroom use is
granted without fee provided that copies are not made or distributed for profit or commercial
advantage and that copies bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than the author(s) must be honored. Abstracting with credit
is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from [email protected].

Publisher

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 7/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

Association for Computing Machinery

New York, NY, United States

FDG 
Publication History
Published: 17 September 2020

Permissions
Bibliometrics Citations 8

Article Metrics

8 271

Total Total
Citations Downloads

This
View website
Citations uses cookies
We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
information
Downloads that you’ve
(Last provided to them or that they’ve collected from your use of their services. Use the check boxes below to 49
12 months)
choose the types of cookies you consent to have stored on your device.

Downloads (Last 6 weeks)

Use necessary cookies only Allow selected cookies Allow all cookies
3

Necessary Preferences Statistics Marketing Show details

Other Metrics

View Author Metrics

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 8/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

FDG 
PDF Format
View or Download as a PDF file.

 PDF

eReader
View online with eReader.

 eReader

HTML Format
This this
View website
articleuses cookies
in HTML Format .
We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
 View
information HTML
that you’ve Format
provided to them or that they’ve collected from your use of their services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.

Use necessary cookies only Allow selected cookies Allow all cookies

Necessary Preferences Statistics Marketing Show details

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 9/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

FDG 

This website uses cookies

We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
information that you’ve provided to them or that they’ve collected from your use of their services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.

Use necessary cookies only Allow selected cookies Allow all cookies

Necessary Preferences Statistics Marketing Show details

Figures Other

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 10/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

FDG 

This website uses cookies

We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
information that you’ve provided to them or that they’ve collected from your use of their services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.

Use necessary cookies only Allow selected cookies Allow all cookies

Necessary Preferences Statistics Marketing Show details

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 11/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

FDG 

Share this Publication link

This website uses cookies
https://dl.acm.org/doi/abs/10.1145/3402942.3402958
We occasionally run membership recruitment campaigns on social media channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
information that you’ve provided to them or that they’ve collected from your use of their services. Use the check boxes below to
choose the types
 Copy of cookies you consent to have stored on your device.
Link

Use necessary cookies only Allow selected cookies Allow all cookies

Necessary
on SocialPreferences
Statistics Marketing Show details
Share Media

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 12/13
11/17/23, 2:50 AM Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch | Proceedings of …

FDG 

0
      

View Table Of Contents

Categories About

Journals About ACM Digital Library

Magazines ACM Digital Library Board

Books Subscription Information

This website uses cookies
Proceedings Author
We occasionally run membership recruitment campaigns on social media Guidelines
channels and use cookies to track post-clicks. We also share
information about your use of our site with our social media, advertising and analytics partners who may combine it with other
SIGs information that you’ve provided to them or that they’ve collected fromUsing ACM
your use Digital
of their Library
services. Use the check boxes below to
choose the types of cookies you consent to have stored on your device.
Conferences All Holdings within the ACM Digital Library
Collections Use necessary cookies only AllowACM Computing
selected cookies Classification
AllowSystem
all cookies

People Necessary Preferences Statistics

Digital Library Accessibility Show details
Marketing

Join Connect

Join ACM  Contact

Join SIGs  Facebook

Subscribe to Publications  Twitter

Institutions and Libraries  Linkedin

 Feedback

 Bug Report

The ACM Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.

Terms of Usage Privacy Policy Code of Ethics

https://dl.acm.org/doi/abs/10.1145/3402942.3402958 13/13