2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)

An investigation on the automatic generation of

music and its application into video games
Germán Ruiz Marcos
School of Computing and Communications
The Open University
Milton Keynes, UK
german.ruiz-marcos@open.ac.uk

Abstract—This paper presents a description of the author’s
PhD research plan and its progress to date. By way of intro-
duction, some gaps and challenges are pointed out concerning
algorithmic composition and its literature. Motivated by these, a
set of research questions are given, which explore the possibility
of generating music matching tension and its applications in video
games. To give a brief overview of the background, the most
relevant models of tension in music are introduced, as well as the
most recent pieces of related work. The research approach is then
presented as a summary of the scope of the problem, according to
the gaps motivating the project and the challenges that come from
the related work, and the appropriate methodology to explore the
research questions. A brief review of the work to date is included,
emphasising the design of an automatic music generator and the
empirical study carried out to test its capabilities. To conclude,
the steps presented in the methodology are transformed into a
future plan and some research contributions.
Index Terms—algorithmic composition, tension, video games,
tonal music
I. I NTRODUCTION
Algorithmic composition refers to the process of combining
musical elements into a whole composition according to a
sequence of rules [1]. During the past decades, algorithmic
composition has been widely studied within the field of
artificial intelligence [2]–[4]. However, the music generated by
most of these studies lacks specific intention and so can sound
meaningless [2]. To address this issue, Papadopoulos and
Wiggins [2] proposed that, in future, models should evaluate
and refer to a specific feature, such as musical tension 1
or expectancy. This would allow human composition to be
simulated more closely.
A. Motivations
The generation of music in terms of tension can be used in
many applications. For instance, in video games music plays
an essential role, both as a medium of affective immersion and
as support of the interaction between the player and the nar-
rative of the game. Thereby, procedural music, “composition
that evolves in real time according to a specific set of rules or
control logics” [6], which is in constant demand in markets
such as those of films or video games, would definitely benefit
from a tension-based approach.
1 Musical tension is commonly used for evoking emotions in the literature
concerning affective computing [5].
The process of automatically generating music has also been
aided by improvements in memory consumption. It is attractive
as a source of real-time, original, endless material [7], either
as a final product or to provide inspiration [8], and to reduce
effort and costs [9].
B. Research questions
Motivated by this, my research explores the following
research question: how to automatically generate music match-
ing the flow of tension within narratives and how can it be
applied into video games?
This question can be split into four sub-questions, which
determine my research methodology. These questions are:
a) How to model (tonal) tension in order to implement an
automatic algorithmic composition system?
b) How to generate harmonic, melodic and rhythmic con-
tent matching (tonal) tension?
c) Which features influence the generation, evolution and
perception of tension within video games and their
narratives?
d) How to map the musical model of tension to that
generated by the game?
II. BACKGROUND AND RELATED WORK
In the literature, musical tension has been studied as a
matter of expectation within hierarchies, as it is the case of
tonality, but also as depending on the psychological impact
of sensory features, such as loudness, rhythmic patterns or
dissonances, among others [10]–[12]. These latter studies
investigate tension by focusing on a single feature each, which
precludes the generalization of a theory of tonal tension. On
the other hand, hierarchical models attempt to describe the
general role of tension in a given context, what fits best with
the purpose of this investigation.
A. Musical context
Tonality is one of the main idioms in the Western music
tradition 2 . It can be defined as a hierarchical system that
organizes musical events according to their level of importance
with regard to a reference, known as the centre of the tonality.
Those relationships are based on the nature of sound itself,
2 The term “tonal” is used in this paper as being in the context of tonality.

978-1-7281-3891-6/19/$31.00 ©2019 Crown

but also on the implicit knowledge of tonality, as the result
of the long-term exposure to this context during most of the
history of the Western music tradition.
Notes grouped together and played simultaneously are
known as chords. Within tonality, chords have a specific role
or function given by the individual importance of the notes
shaping each chord. Said chords’ ratings could be considered
as implicit in listeners [13]. So that, a note or a chord generates
an expectation on the listener on what will be played next,
based on its level of importance within tonality [14]. Thus,
the expectation of a given note or chord depends on its tonal
function, but also on the distance to the previous note or chord:
the more important the note or chord is within a tonality and
the closer it is to its preceding note or chord, the more it is
expected to be played, and vice-versa.
In this sense, musical tension can be described as a matter
of expectation, following a hierarchical organization [11]. If
a note or chord is played but it was not expected to follow
its previous note or chord, the change from one to the other
will be perceived as tense. If, on the contrary, the output note
or chord was expected, we will probably feel the transition as
non-tense.
B. Modelling tonal tension
There have been different approaches to modelling tension
between chords in the literature. However, some of them have
not been empirically tested [15], do not give a complete
perspective of the problem [16], or fail to consider features
such as melodic contour and rhythm [12]. In 2007, Lerdahl
and Krumhansl [17] presented a quantitative theory to model
tonal tension 3 , based on the structural analysis of music and
the role of notes and chords within tonality. This theory has
shown empirical evidence both statistically [17] and at the
neuroscientific level [20].
The relationship between melody and tension is studied in
the literature as a matter of melodic expectancy. Margulis [14]
succeeded in providing quantitative predictions of melodic
expectancy. Given a note in a melodic line, Margulis’ model
would calculate the expectancy values of the note to follow
based on the role of the chord of the original note [20],
the distance of the original note to its preceding note, and
the continuation or reverse of the direction of the preceding
melody.
Concerning the rhythmic perspective of tension, some con-
siderations had already been taken into account in Lerdahl and
Krumhansl’s model [17] (as it is based on the GTTM). Some
others refer to the musicological behaviour of chords within
tonality, as going from a more stable chord to a less stable
one is perceived as tense, and to the speed of change of the
notes between chords; the faster the change, the tenser it is
perceived [21].
3 Based on the well-known Generative Theory of Tonal Music [18] (GTTM)
and the Tonal Pitch Space [19] (TPS).
C. Tension and game scoring
Dynamic music is that able to react and adapt to game play
[22]. It can support narratives by pointing out and describing
emotions, characterizing settings and physical activities, com-
municating values, directing attention and focusing on detail,
or on the contrary masking [23].
There have been some contributions to this matter in the past
recent years. AudioInSpace [24] presented a creative fusion of
generative audio and narratives, implemented through multiple
Artifitial Neural Networks. The Sonancia [25] project carried
out sonification of game levels according to probabilistic
transitions, which were weighted according to pre-annotated
levels of suspense. ARTMG [26] developed a music generator
capable of composing music in real-time in style of a training
piece, using Hierarchical and Hidden Markov models. In FATE
[27], adaptive music is generated in order to set interactive
experiences, according to probabilistic transitions based on
context-sensitive grammars. Metacompose [28] proved capable
of generating affective music in real time by using genetic
algorithms. Finally, Escape point [29] presented a template-
based chord generator that supports a dynamically changing
emotional-narrative.
III. R ESEARCH APPROACH
A. Scope of the problem
According to the literature, the lack of intention of music
generators within algorithmic composition stands out as an
essential gap to be filled. This gap, which motivates my
research, sets the starting point at finding a suitable definition
of tension in the context of music applicable into video games.
To approach this task, an interpretation and transformation of
the theoretical descriptions of tension in music and games is
needed, to support an implementation of a generative strategy.
New challenges also arise from the limitations of the
models introduced as part of the related work. Taking into
account that game narratives are non-linear, game music has
to dynamically adapt in real-time. But it has to be an enjoyable
adaptation to game play, avoiding boredom. Likewise, most of
the music generators developed to date generate game music
as variations over pre-composed material, which reduces the
chances of exploring new and more creative ideas, as oppose
to generating total compositions of new material.
B. Methodology
Taking into account the scope of the problem, the steps
to be followed to answer the research questions could be
summarised as follows:
a) How to model (tonal) tension in order to implement an
automatic algorithmic composition system?
• to carry out a critical review of the literature con-
cerning theories of tension in music.
• to combine the most suitable theories of tension
in music into a single model, keeping in mind the
empirical essence of the desired output.
 b) How to generate harmonic, melodic and rhythmic con-
tent matching (tonal) tension?
• to transform the theoretical model from analytical
into generative.
• to define and set the value of the parameters of the
generative theory based on theoretical aspects.
• to test those decisions taking into account the re-
lationship between the generative model and what
listeners actually perceive.
• to hypothesize about ideas of improvement on the
generative model based on empirical data.
c) Which features influence the generation, evolution and
perception of tension within video games and their
narratives?
• to explore the role of the different features that relate
to tension within game narratives.
• to design a theoretical framework concerning the
implication of these features in the generation of
tension through game narratives.
• to test the theoretical framework, qualitatively, with
experts in the field of Game Design and Game
Music.
d) How to map the musical model of tension to that
generated by the game?
• to propose a mapping system based on the musical
and gaming tension frameworks.
• to implement the final model into a video game.
• to test the influence of the generated game music in
terms of immersion.
• to test the correlation between the tension extracted
from the game narratives, the tension theorized by
the music generator, and the tension perceived by
the players.
• to come up with relevant conclusions and prospects
for future work.
IV. W ORK TO DATE
A. Design of a music generator
I have designed and implemented a music generative system
whose output matches tonal tension. It is divided in three
layers, each corresponding to three basic elements of music:
harmony (sequence of chords), rhythm (arrangement of dura-
tions of chords and notes) and melody (sequence of notes).
Based on Lerdahl and Krumhansl’s [17] quantitative model
of tonal tension, it is possible to, given a chord, decide which
chords could be played next to match a specific level of
tension. To deal with all the possible transitions between
chords, the harmonic generator is implemented as a Markov
chain. A Markov chain is a stochastic model where the
possible events of a transition are given a probability to
follow the sequence depending only on the previous event.
Those transition probabilities are set according to the tension
values. That is to say, in a high-tension scenario, the tenser
the transition, the higher the probability to occur; and, in a
low-tension scenario, the tenser the transition, the lower the
Fig. 1. Interface used in the empirical study.
probability. Thereby, following the transition probabilities, a
chord sequence is generated matching a given level of tension.
The durations of those chords also affect musical tension.
According to Swain’s theory of rhythmic tension [21], tran-
sitions from longer to shorter durations are perceived as an
increase in tension, and vice-versa. Likewise, the harmonic
and rhythmic influence on tension must be congruent. So that,
chords’ durations are implemented again as a probabilistic
model, taking into account the level of tension of the transition
between chords: if the transition goes to a more important
chord in the tonality, tension decreases, so it is more probable
to find a longer chord; if the transition goes to a less important
chord, tension increases, so it is more probable to find a shorter
chord. When the level of tension remains the same, it is more
probable to repeat the previous duration.
A melody could be played on top of the generated harmonic
sequence, matching again tonal tension. Margulis’ model of
melodic expectancy [14] was used to determine transition
probabilities between notes in order to stochastically generate
a melody.
B. Empirical study
I carried out an experiment to test the power and capability
of the described system. Ten musicians and ten non-musicians
took part in the study, which was done face-to-face and
lasted between thirty and fifty minutes per participant. An
audiovisual interface (see Fig. 1) was specifically designed for
the experiment where three areas were shown. Each area was
associated with a different tension level: minimum, medium
or maximum. Participants were asked to listen to the music
generated in each area and select, from the three possible
areas, which sequence they perceived as least tense (minimum)
and as most tense (maximum). Each participant listened and
labelled five chord sequences, five melodies, and five melodic
and harmonic sequences, all of them automatically generated
by the described model.
The system’s and participants’ tension labels were compared
for each tension level (minimum, medium, maximum), in
each experimental phase (harmonic, melodic, accompanied
melodies). The agreement between a participant’s and the
system’s labelling was defined as equal to 1 if the system’s
tension label was the same as the participant’s, 0 otherwise 4 .
A summary of the average degree of agreement is shown in
Tables I and II.
4 Cohen’s kappa coefficient was not use because its overall random agree-
ment, pe , would be affected by the conditional probabilities.
 TABLE I TABLE III
AVERAGE DEGREE OF AGREEMENT, %, BETWEEN THE PARTICIPANTS ’ F UTURE PLAN
AND THE SYSTEM ’ S TENSION LABELLINGS IN THE CASE OF MUSICIANS .
mid 2019 late 2019 early 2020 mid 2020 late 2020
Improvements on the generative system
minimum medium maximum AVERAGE Melodic expectancy VS melodic attraction
harmonic 84% 70% 92% 82% Game narrative and tension review
melodic 42% 40% 92% 58% Game features extraction
acc. mel. 68% 64% 94% 75% Design game tension theoretical framework
Interview study with game experts
AVERAGE 65% 68% 93% Design game-music tension mapping system
Implement mapping strategy into video game
Final experiment
Write up thesis

TABLE II
AVERAGE DEGREE OF AGREEMENT, %, BETWEEN THE PARTICIPANTS ’ AND
THE SYSTEM ’ S TENSION LABELLINGS IN THE CASE OF NON - MUSICIANS .
minimum medium maximum AVERAGE
harmonic 52% 40% 54% 49%
melodic 46% 50% 76% 57%
acc. mel. 60% 58% 78% 65%
AVERAGE 53% 49% 69%
C. Discussion
To test the randomness of the data, the probabilities of
matching the system’s tension evaluations by chance were
calculated using contingency tables. The degrees of agreement
proved to be above the by-chance probability threshold ex-
cept in one case. The only exception is the non-musicians’
harmonic phase, at medium tension. The probability of the
data being random in this case is 44%, whereas the empirical
agreement is 40%.
The average agreement in the case of accompanied melodies
is around 75% from musicians’ responses and 65% from non-
musicians’, which fall within the state of the art results in
similar generative tasks in the field of Music Computing.
In the case of musicians, the agreement for the harmonic
phase is 80%. However, it falls to nearly 60% in the melodic
phase. This might happen because of the possible harmonies
inferred by the participants when listening to the melodies.
This suggests there might be two different listening ap-
proaches, being non-musicians less focused on the flow of
the harmonic content. For instance, non-musicians’ melodic
degree of agreement is actually greater than their harmonic
degree of agreement, unlike in the case of musicians. Likewise,
the agreement between the model and participants is higher in
the case of musicians. This raises the question of how different
the perception of tension is between musicians and non-
musicians. It would be of interest in future projects to explore
different types of musical tension, such as those described by
Margulis [14].
The least matching phases correspond to the melodic mini-
mum and medium. All the generated melodic transitions were
annotated and analysed. The maximum case differed from the
rest by more than half of the transitions, but the minimum
and medium generated melodies used nearly the same melodic
transitions. To improve the melodic generation in the future,
looking into harmonic and melodic attraction [17] and the
differences between these and expectancy seems to be a good
starting point.
A harmonic rhythm transition analysis was also carried out,
similarly to the one concerning melodic transitions, but no
significant correlations were found among the disagreements
between participants and the system.
Most of the harmonic mislabellings could be explained as
a matter of separation from the current tonality, what suggests
the need of implementing a tonality-confirmation strategy.
D. Future work
Based on the steps shown in the methodology and the
findings of the empirical study, the next step on my PhD
plan concerns the improvement of the music generator. An
exploration of the defined parameters would be carried out,
as well as a refinement of the stochastic algorithms involved
in the selection of transitions. The improved system would
be tested again against listeners’ perceptions. Thinking of the
final application of the generator within video games, a new
experimental methodology should be used, such as asking
participants to annotate the tension they perceive while listen-
ing to music [17], [30], or getting participants’ physiological
response to music [31].
Once the final music generator is completed, a deeper
literature review would be carried out concerning Game Music,
focusing on the role of tension within game narratives. From
this literature review a theoretical framework would be de-
signed, exploring the relationship between game features and
tension in narratives. The framework would be qualitatively
tested based on the contributions of experts in the field of
game design and game scoring.
The final step of this research project would be the im-
plementation of both tension frameworks, that is the music
generator and the narratives analyser, into a video game. A
final experiment would be carried out to test the mapping
strategy, but also to test the influence of the generated game
music in terms of immersion. A summary of the future plan
in shown in Table III
V. C ONTRIBUTIONS
The research contributions to date include a critical review
of the literature concerning models of tonal tension, and the
music generative model. The data gathered in the empirical
study also constitute useful findings, by suggesting strategies
for future improvement.
According to the future plan, some expected contributions
of the PhD work would include the theoretical framework of
tension in games, the mapping strategy between the musical
and the game tension approaches, and a video game where
procedural music adapts in real-time to dynamically changing
narratives.
 R EFERENCES [27] C. Aspromallis, and N. E. Gold, “Form-Aware, Real-Time Adaptive
Music Generation for Interactive Experiences”, in Sound and Music
[1] D. Cope, “Position Paper for the Second Panel on Algorithmic Music”, Computing, 2016.
in Proceedings of the International Computer Music Conference, pp. [28] M. Scirea, J. Togelius, P. Eklund, and S. Risi, Metacompose: “A com-
23–28, 1993. positional evolutionary music composer”, in International Conference
[2] G. Papadopoulos, and G. Wiggins, “AI methods for algorithmic com- on Computational Intelligence in Music, Sound, Art and Design, pp.
position: A survey, a critical view and future prospects”, in AISB 202–217, March 2016.
Symposium on Musical Creativity, Vol. 124, pp. 110–117, April 1999. [29] A. Prechtl, “Adaptive music generation for computer games”, 2016.
[3] G. Nierhaus, G, “Algorithmic composition: paradigms of automated [30] M. M. Farbood, “A parametric, temporal model of musical tension”, in
music generation”, Springer Science & Business Media, 2009. Music Perception: An Interdisciplinary Journal, University of California
[4] J. D. Fernández, and F. Vico, F., “AI methods in algorithmic compo- Press Journals, 29(4), pp. 387–428, 2012.
sition: A comprehensive survey”, in Journal of Artificial Intelligence [31] C. L. Krumhansl, “An exploratory study of musical emotions and psy-
Research, 48, pp. 513–582, 2013. chophysiology”, in Canadian Journal of Experimental Psychology/Revue
[5] D. Herremans, C. H. Chuan, C. H., and Chew, E., “A functional canadienne de psychologie exprimentale, 51(4), 336, 1997.
taxonomy of music generation systems”, ACM Computing Surveys
(CSUR), 50(5), 69, 2017.
[6] K. Collins, K., “An introduction to procedural music in video games”,
in Contemporary Music Review, 28(1), pp. 5–15, 2009.
[7] J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne, “Search-
based procedural content generation: A taxonomy and survey”, in IEEE
Transactions on Computational Intelligence and AI in Games, 3(3), pp.
172–186, 2011.
[8] A. M. Smith, and M. and Mateas, “Variations forever: Flexibly gener-
ating rulesets from a sculptable design space of mini-games”, in Pro-
ceedings of the 2010 IEEE Conference on Computational Intelligence
and Games, pp. 273–280, August 2010.
[9] C. Remo, “MIGS: Far Cry 2s Guay on the importance of procedural
content”, Gamasutra, November 2008.
[10] E. Bigand, R. Parncutt, and F. Lerdahl, “Perception of musical tension
in short chord sequences: The influence of harmonic function, sensory
dissonance, horizontal motion, and musical training”, in Perception &
Psychophysics, 58(1), pp. 125–141, 1996.
[11] R. Y. Granot, and Z. Eitan, “Musical tension and the interaction of
dynamic auditory parameters”, in Music Perception: An Interdisciplinary
Journal, 28(3), pp. 219–246, 2011.
[12] D. Herremans, and E. Chew, “Tension ribbons: Quantifying and visual-
ising tonal tension”, 2016.
[13] F. Lerdahl, “Tonal pitch space”, in Music Perception: An Interdisci-
plinary Journal, 5(3), pp. 315–349, 1988.
[14] E. H. Margulis, “A model of melodic expectation”, in Music Perception:
An Interdisciplinary Journal, 22(4), pp. 663–714, 2005.
[15] H. Schenker, “1979. Free Composition”, in Trans. and ed. E. Oster. New
York, 1935.
[16] R. N. Shepard, “Geometrical approximations to the structure of musical
pitch”, in Psychological review, American Psychological Association,
89(4), 1982.
[17] F. Lerdahl, and C. L. Krumhansl, “Modeling tonal tension”, in Music
Perception: An Interdisciplinary Journal, University of California Press
Journals, 24(4), pp. 329–366, 2007.
[18] F. Lerdahl, and R. S. Jackendoff, “A generative theory of tonal music”,
MIT press, 1985.
[19] F. Lerdahl, “Tonal pitch space”, Oxford University Press, 2004.
[20] S. Koelsch, M. Rohrmeier, R. Torrecuso, and S. Jentschke, “Processing
of hierarchical syntactic structure in music”, in Proceedings of the
National Academy of Sciences, 110(38), pp. 15443-15448, 2013.
[21] J. P. Swain, “Dimensions of Harmonic Rhythm”, in Music Theory
Spectrum, University of California Press, 20(1), pp. 48–71, 1998.
[22] K. Collins, “An introduction to the participatory and non-linear aspects
of video games audio”, in Essays on sound and vision, pp. 263–298,
2007.
[23] J. Wingstedt, “Narrative functions of film music in a relational per-
spective”, in 26th International Society for Music Education World
Conference, 2004.
[24] A. K. Hoover, W. Cachia, A. Liapis, and G. N. Yannakakis, “Au-
dioinspace: Exploring the creative fusion of generative audio, visuals
and gameplay”, in International Conference on Evolutionary and Bio-
logically Inspired Music and Art, pp. 101–112, April 2015.
[25] P. Lopes, A. Liapis, and G. N. Yannakakis, “Sonancia: Sonification
of procedurally generated game levels”, in Proceedings of the 1st
computational creativity and games workshop, 2015.
[26] S. Engels, T. Tong, and F. Chan, “Automatic real-time music generation
for games”, in Eleventh Artificial Intelligence and Interactive Digital
Entertainment Conference, September 2015.