See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/306276194

Music in Multimodal Narratives. The role of the soundtrack in Digital Stories

Chapter · January 2016

CITATIONS READS

6 264

1 author:

Dolores Porto
University of Alcalá
29 PUBLICATIONS 151 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Emergent and Peripheral Discourses and their Social Projection: Critical and Socio-Cognitive Approach (DEEPS) ( FFI2016-77540-P) View project

'Polarization and Digital Discourses: Critical and Socio-Cognitive perspectives' (PID2020-119102RB-I00) View project

All content following this page was uploaded by Dolores Porto on 18 August 2016.

The user has requested enhancement of the downloaded file.

 Porto Requejo, M. Dolores (2016) "Music in Multimodal Narratives: The Role of the
Soundtrack in Digital Stories". In Jarmila Mildorf and Till Kinzel (eds) Audionarratology.
Interfaces of Sound and Narrative (29-46) Berlin: de Gruyter Mouton (ISBN: 978-3-11-
047275-2) p. 29-46.
https://www.degruyter.com/viewbooktoc/product/468357

Music in Multimodal Narratives. The role of the soundtrack in Digital

Stories

Abstract

This paper examines the function of background music and sounds in digital stories and their

contribution to the general structure, meaning and purpose of the story. Based on previous work

on the genre in the framework of a larger research project, the soundtracks of thirty digital

stories are first analyzed in relation with the structure of the stories. Next, theoretical issues

from different disciplines – narratology, cognitive linguistics, psychology, musicology – are

applied to the sample in order to examine the effects of the music in terms of attention

phenomena, emotional response and persuasive effects. The results evidence that music in

digital stories, far from merely supporting text, can serve a number of functions – structural,

attentional, evaluative and also persuasive –, that are essential for a thorough interpretation of

the story.

1. Introduction: Digital Stories

Digital stories are short multimodal narratives created by non-experts in literature nor

technologies that constitute a new emergent and rapidly expanding genre in the Internet.1 In line

1
This paper forms part of a research project on Discursive Strategies in English and Spanish. Socio-
cognitive and Functional Interactions, financed by the Spanish Ministry of Science and Innovation
(FFI2012-30790).
with the recent democratization of the World Wide Web, the practice of digital storytelling

allows ordinary people to narrate their own experiences and share them in the net. First

developed by the Center for Digital Storytelling (CDS) in Berkeley in the 1990s, this practice

has spread all through the world, mostly through workshops held by non-profit organizations

with educational purposes that intend to reach a widespread audience. However, there are also

more limited, local purposes, especially in schools and higher education institutions for the

teaching of specific contents, or with the aim of community engagement in residential areas

(Lambert 2002/2013). Thus, it is possible to find thousands of digital stories on the net with a

great variety of purposes, but typically, digital stories narrate highly emotional, personal

experiences of overcoming that intend to denounce social wrongs or support others who might

be suffering similar situations to those featuring in the stories.

Apart from this emotional, personal content, digital stories all share the same format which

consists on an multimodal narrative about 3 or 4 minutes long that matches the recorded voice

of the narrator with a set of images – personal photographs, drawings, symbols, short videos or

even generic pictures taken from digital repositories – and sometimes also background music or

special sound effects. The brevity of the stories combined with the emotional content and

advisory purpose of the narratives urge the narrator to compress the maximum of information in

a very short time span. Therefore, in a digital story there are no superfluous information, side

stories or ornamental effects; every aspect of it, textual, visual or acoustic, must be maximally

significant and contribute to the general purpose of the narrative.

Within the framework of a larger research project on Narrative and Cognition, we have worked

in a detailed analysis on the features of the genre, the structure of the stories, the kind of

information provided by each semiotic mode and how the whole meaning is constructed through

the integration of all these elements (Alonso Belmonte et al. 2013). On a second stage, we

focused on the specific function of images in the construction of the story (Porto and Alonso

Belmonte 2014) and concluded that they had an important “glocalising” function, i.e. they

provided local stories with a universal meaning. At this point, it is quite straightforward that the
next stage of the project should address the role of the third component in these multimodal

narratives, i.e. the music and sounds that accompany the narration. The main hypothesis in this

work is that music, as much as text and images, is not merely decorative in digital stories, and

plays an important role, even if optional, in the construction and interpretation of the story.

2. Sample and Methodology

As pointed out above, this study is part of a larger research project on digital stories.

Consequently, the present analysis on the music and sounds has been done on the same sample

collected by the team for that previous work.2 Rather than a formal corpus, it consists of a

sample of thirty stories taken from well-known, recognized nongovernmental organizations

devoted to the creation and publication of this kind of narratives to serve their purposes of

denouncing wrongs, supporting victims or building community, such as BBC Telling lives,

Creative Narrations, Engender Health, Silence Speaks, among others. Even if randomly

selected, we deliberately searched for a variety of cultures and continents represented and so, we

included stories narrated by people from South Africa, Namibia, India, Philippines, USA,

United Kingdom, Peru, Mexico… even if all of them are told in English or at least subtitled in

English. This apparent contradiction responds to the acknowledged circumstance that English is

the language of the Internet and any narrator that intends to reach a wide audience must tell their

stories in English. As for the topics represented in the sample, we also searched for a balance

and decided to include both big global issues – HIV, immigration, sexual discrimination, school

bullying, etc, as well as small stories of personal achievements and life experiences.3

The thirty stories were coded by the acronym of the organisation that promoted and published

the stories, e.g. Creative Nations CN, Bristol Stories BS and a number (See Appendix for a

complete list of the thirty stories). Next, they were transcribed in tables that matched every

2
The members of this research group, working as a sub-team in the project coordinated by Manuela
Romano, are Isabel Alonso Belmonte, Silvia Molina and myself.
3
As a matter of fact, we later found out that these “small stories” also intended a universal meaning for a
global audience from faraway countries and cultures. See Porto and Alonso Belmonte 2014.
fragment of text with the image simultaneously displayed and a short description of background

music. Allthis painstaking task allowed us to analyse in depth the way in which the meanings

conveyed through the three modes were integrated in digital stories.

For obvious reasons, in this paper I will also draw on the analysis of the structure of the thirty

stories performed in Porto and Alonso Belmonte (2014), where Labov’s schema of oral

narratives of personal accounts (Labov, 1972; Labov and Waletzky, 1967/1997) was applied to

the sample in order to identify the different parts of the narratives and the role of images in the

different sections. Thus, a comparison will be possible between the role of images and the role

of music in the digital stories analysed. Finally, some other theories and insights from fields

other than linguistics or narratology, such as psychology and musicology will be taken into

account as for the effects of sound and music in the audience.

3. Sounds and music in Digital Stories

Not all digital stories include music or sound effects. The essential features of the genre are the

recorded voice of the narrator and the images, whereas the musical background is optional. As a

matter of fact, manuals and guides for storytellers and workshop facilitators all warn of the

“power” and also of the “dangers” of the musical background. According to these

recommendations, music can set the tone of a story, enhance emotionality and add depth and

complexity to the narrative (Lambert 2013/2002: 64). However, musical soundtracks, especially

those with lyrics, can distract the audience and compete with the narrator’s voice, or else

provide “unintended meanings”. Moreover, in order to avoid conflicts derived from the use of

copyrighted material and music, some guides advise storytellers to use their own music, created

by themselves or a friend, or just whistling, humming, singing (Simon Turner’s guide at BBC

Wales). 4

4
http://www.bbc.co.uk/wales/audiovideo/sites/about/pages/recordingothers.shtml
In our sample, there are only five stories without any music. This can be due to technical

difficulties or lack of knowledge on the side of the story’s creator, which is a very plausible

assumption for those stories without any music that belong to the same website/workshop (DS1

and DS2 from Diversity Hub and TL2, TL3 from the Telling Lives program). However, the lack

of music can also be intentional, so as to provide more emphasis to the recorded voice of the

narrator, whose tone and rhythm can convey powerful meanings and emotions by itself

(Distance SC9). As for the rest of the sample, a great variety of kinds of music and

combinations can be found: pop, urban rap, gospel, folk, hymns… Some storytellers use only

one song or melody all along the story, but many others combine more than one with interesting

effects on the narrative structure and on the listener’s attention, as we will see.

In the analysis of the role and effects of music and background sounds in digital stories, the

following functions have been observed and identified: structural, evaluative, attentional and

persuasive. It goes without saying that this classification is merely methodological and that

those functions constantly mix and overlap in the narratives. Therefore, attracting the listener’s

attention to a particular point of the story can be both an evaluative and a persuasive device, as

much as marking the shifts from a section to the next in the story structure is aimed at guiding

and maintaining the listener’s attention on the narrative.

4. The structural function of music in digital stories

Following the same methodology as in the analysis of the function of images in digital stories

(Porto and Alonso Belmonte 2014), music reveals as a major structuring device, marking the

moves from a section to the next. Table 1 summarizes the Labovian schema of narratives of

personal accounts (Labov, 1972; Labov and Waletzky, 1967/1997). According to this schema,

this kind of narratives can be divided in three main sections, orientation, complication and

resolution, with clearly defined functions, and sometimes, preceded by a brief abstract and

completed with a coda.

Labov and Waletzky also take into account a sixth element, evaluation, that is not confined to a

specific part of the story, but can be found all through it in different elements of the narrative,

often overlapping other functions. Since evaluation cannot be regarded a distinct section, I will

not consider it at his point of the analysis of the structure. However, evaluation is an essential

element in digital stories that reveals the real purpose of telling it to a given audience and we

will see that the soundtrack also contributes to it in a significant way.

NARRATIVE NARRATIVE QUESTION NARRATIVE FUNCTION

CATEGORY
Abstract (optional) What was this about?
Orientation Who or what are involved in the
story and when and where did it
take place?
Signals that the story is about to begin and
draws attention from the listener.
The orientation sets the scene and thus helps
the listener to identify the time, place,
people, activity and situation of the story.

Complicating action Then what happened? Describes the action or events that occurred
in the story.
Resolution What finally happened? It explains the outcome of the story

Coda (optional) How does it all end? It brings the listener to the present time

Now, I will provide some examples of how music and sounds contribute to the different parts of

the story following this schema

4.1 Musical abstract

The abstract is an optional element in the narratives of personal experiences. Its function is to

signal that the story is about to begin and to draw the audience’s attention to it. In digital stories,

when present, abstracts can be textual, visual or musical, or a combination of these. Examples of

musical abstracts can be found in the stories in which the music itself sets the tone and

sometimes even a first clue on the topic of the narrative, even before any image is displayed or

the narrator starts speaking. Typically this is the case in digital stories that start with a song that
a part of the audience may already know and so activate certain associations, images and

feelings that will play a role in the understanding of the story.

Thus, for instance in Despite my fears (BS2), the first chords of a famous song by ABBA,

Dancing Queen, can be heard before the narrator starts her story. She tells her experience in an

amateur theatre company, so the song introduces the topic of acting, being part of a show, etc.

Also, To Every Child (CN3) starts with the instrumental beginning of a well-known gospel song

while a Bible quotation is displayed. This music sets the tone of the story, which deals with

inequality in education for black children in Boston and the narrator’s determination to fight

against it by becoming an educator herself.

There are also musical abstracts where it is the lyrics of the song that constitute a message to the

audience. This is the case of The Day I made Him Stop (UM1), a story about a school girl who

used to be beaten by his teacher, which starts with the music of a Christian hymn that the

narrator sings herself:

Jesus loves me, this I know,

for the Bible tells me so.
Little ones to Him belong;
they are weak but He is strong

After that, the narrator starts her story without any additional soundtrack. It can be considered a

musical abstract, even if it is combined with text, because the lyrics do not have a direct relation

with the story itself. It is only the fact that it is a religious hymn that makes the abstract.

Moreover, the feeling of closeness is strongly supported by the fact that it is voice of the

narrator that we can hear, without any instruments, instead of a commercial recording.

4.2 Musical orientation

Orientation is the section of the story that sets the scene in a specific time or place or introduces

the people involved in it. Music is a very basic device to convey this meaning, as it is so

culturally embedded. In stories like Sacrificios (SC2) or The Home Land (BS1), music instantly

contextualizes the narrative. In Sacrificios, a Spanish guitar serves as a musical abstract first,
and then as an orientation as it plays at the background while the narrator introduces his

grandparents both textually:

This is the story of my grandparents, Fernando and Emilia Sánchez

and visually by showing an old photograph. In The Home Land it is the sound of African drums

that accompanies the beginning of the narrative about the African origin of the narrator who

lives in England:

There was no particular time in my life when I thought to myself “I want to learn about

my African heritage.”

Folk music is a recurrent tool for the orientation section in the digital stories analysed, as it is a

straightforward way of setting the scene in a specific time and place. It can be found in many of

them (Mexican music in CN1, Chinese in IL1, African in SC5, etc.). It has a localising function,

comparable to that of images showing costumes, landscapes and people racial features.

It is also interesting to note that not only the music, but also the sound of the narrator’s voice

has an orientation function. Because all stories are told in the first person, the voice introduces

the main character in them: gender, age, accent…

4.3 Music and sounds in the complication

After the abstract and the orientation, the music usually turns down or even stops, so that the

voice of the storyteller can be clearly heard. However, as the story evolves, different musical

and sound effects are used by storytellers for various purposes, among them, in order to signal a

change in the action. Thus, in some stories, a change in the background music marks a shift in

the action. For example in Privilege (SC3) a soft music matches the account of the narrator’s

happy childhood in a big farm where his parents worked until the moment when the field

workers went on strike and were fired. At that point the music turns up and becomes a protest

song. In Rock Bottom (EH3), the narrator first introduces the story with an abstract without any

music, next he starts telling the audience about his childhood, with a tinkling happy musical

background. The music changes when he moves towards his days at university with an active
social life, rife with parties and women. After that, the strings of a contrabass signal the change

in his life when he meets his girlfriend. An electric guitar can be heard when he explains he

decided to change his life, and at the end, a complete melody is used as the background of the

resolution. Interestingly, in International Living-Southern China (IL1), several pieces of

Chinese bamboo flute music are used to accompany parts of the story, whereas in others it is

silenced so that the original sounds of the videos displayed can be heard – the sounds of traffic

nosises, street music, birds singing at an openmarket, children playing. This is a remarkable

strategy that transports the audience to those places and provides a more vivid account of the

experience narrated.

4.4 Music in the resolution

Typically, the music turns up towards the end of the story, which serves to warn the audience

that we are reaching the final part of the story and also connects this end with the beginning,

when the same music was heard (BS2, CN2, SC8, IL1…). It must be noted, though, that in

almost half of the stories in the sample, we can talk of a “non-resolution component” (Porto and

Alonso 2014: 6), as the experiences narrated often refer to a global issue (environmental justice,

school bullying, sexual discrimination) that they are denouncing and cannot be solved by the

individual. Even so, since the purpose of digital stories is that of advising and supporting,

resolutions or endings are always positive and encouraging. Consequently, the music that goes

with the resolution tends to be joyful and lively and the rising volume contributes to this effect.

In Nelao’s story (EH1), the narrator tells about her difficulties to find a partner because she is

HIV positive. However, at the end of her story she values other aspects in her life that make her

happy and is optimistic about the future. Whereas most of her story, except for the orientation,

goes without music, so increasing the strength and drama of her narration, in the final part, some

soft background music is heard accompanying the “non-resolution” of her narrative:

It’s the beginning of a new life. I’m young, beautiful and intelligent and I have a bright
future ahead of me…
In Privilege (SC3), the positive feeling at the end of the story provided the music is reinforced

by an upwards movement of the camera to focus the sky as the music volume also rises so that

we can hear the lyrics of the song.

4.5 Musical coda

Those narrators that use a song as the background tend to take advantage of its message and use

it as a musical coda. In these cases, after the narrator’s voice stops, the music becomes louder so

that we can hear the lyrics of the song. Thus, for example, in To Every Child (CN3), the chorus

of the gospel song that served as the background for the whole story constitutes the coda of a

story where the narrator exposes that she wants to change the future of black children by

becoming a teacher.

Imagine me, being free, trusting you totally, finally I can imagine me, I admit it was hard
to see you being in love with someone like me, Finally I can imagine me, imagine me.

A message of hope is also the musical coda of The Balcony (SC6), which goes without any

soundtrack at all until the story is completely finished and the following can be heard while the

slides of credits are shown: 5

Someday, when we are wiser, when the world’s older, when we have learned....
I pray someday we may yet, live to live and let live.

Particularly effective in this sense is the case of Memories of a Political Prisoner from

Worcester (SC5), where the background music is merely a humming during the narration, but

that is finally sung by the narrator at the end as an hymn for South Africa.

5. The soundtrack as an attentional marker to guide the listener

Attention is one of our main cognitive abilities: It plays a leading role in the way in which we

perceive and interpret the world around us by enabling us to focus on small parts of our

environment before we organize our knowledge and understand the whole.Accordingly,

5
From the song Someday, in the film The Hunchback of Notre Dame (Disney 1996).
attention is also reflected in language, in the way we speak about the world. Linguistic forms

can direct our attention in discourse at different levels – phonological, lexical, syntactical,

semantic, pragmatic… through various devices, all of which constitute what Talmy calls the

“Attentional System of Language” (Talmy 2007, 2008). In narratives, attention also plays a

leading part in their organization and interpretation (Romano and Porto 2013), since a narrative

rarely has the linear, well-organized structure that we often assume.

In purely textual narratives, discourse markers work as attention guiding mechanisms in this

sense, but in multimodal narratives, as it is the case with digital stories, various factors regularly

interact to produce attentional effects. Thus, textual strategies, i.e. linguistic and pragmatic

markers, are combined in several ways with visual and acoustic devices in order to catch the

audience’s attention and guide it to the most relevant parts of the narrative. Such visual and

acoustic strategies have much to do with perceptual salience, e.g. colour and salience in images

and volume or pitch in sounds, but also with the way in which these interact with structural,

emotional and cultural factors, as we will see.

So for instance, the musical abstracts described in the previous section have an obvious

attentional function overlapping the structural one, since catching the audience’s attention is

partly the aim of an abstract. When the ABBA song starts to play in Despite my Fears (BS2), or

the rhythmic beginning of a urban rock piece precedes the narrator’s account in Bad Choices

(CN2), listeners get ready to listen and watch and take notice of what is coming next. Similarly,

the changes in the music that signal transitions between different parts of the story also perform

both a structuring and an attentional function (SC1, SC3, EH3, EH8, IL1…). As a consequence,

every time the audience perceive a change in the music, they give ear to a possible change in the

story.

In addition, cultural identification overlaps these functions and also works as an attentional

marker. When storytellers use songs that can be easily identified by their potential audience,

they are more readily attentive as if caught by a familiar, well-known melody. It is the case of

Despite my Fears (BS2), To Every Child (CN3) or Memories of a Political Prisoner (SC5).
Perceptual factors in music are the most obvious devices for catching attention: if the volume

turns up or if, on the contrary, there is a sudden silence in the soundtrack, the listener will direct

his/her attention to that point of the narrative. The effect of changes in the volume can be

observed in most stories both in the abstract and the resolution. As already pointed out, music is

turned down when the narrator starts speaking after the abstract and it goes up again in the

resolution as if signalling that the end is getting closer. Even more interesting is the use of

silence as an attentional marker in digital stories. In Everyone Knows: Manoj’s story (EH6), the

narrator starts telling his story with some instrumental soft music at the background. He

introduces himself, his age, his city… until the moment he says he is HIV positive:

When I was 22, I donated blood for an operation. It was then that I found out that I had
HIV.

At that point, the music stops and the next words gain strength because of that sudden silence:

I felt isolated, scared, and so sad

After that, music is resumed and the story goes on. A similar strategy can be found in other

stories: Bad choices (CN2), Lillo’s Story (EH8), Nelao’s Story (EH1), Untitled (SC1)…

Perception is also the key to catch the listener’s attention when specific points of the story are

highlighted with particular sound effects. In Nelao’s Story (EH1), the sound of footsteps

running away accompanies the following text:

I have always had bad experiences with men who ran away when I disclosed my HIV
status to them.

And the sound of breaking glass matches the image of a broken heart, as the narrator explains

her feelings:

I’m torn apart. I want to love and be loved as well.

Such effects force the audience to have not only linguistic understanding, but also sight and

hearing focused on the same idea: men running away or a broken heart, which provides great

strength to the intended meanings and will constitute a persuasive element in the narration.
 6. Music and emotions: Evaluative function of the soundtrack in digital stories

It is a well known fact that music influences emotion and behaviour. Major keys when

combined with rapid tempos evoke feelings of joy and happiness in the listener, whereas minor

keys and slow tempos may arouse sadness and melancholy to a certain degree. Moreover,

dissonance consistently provokes negative reactions and when combined with rapid tempos it

may cause emotions like distress or fear (Krumhansl 1997). Empirical experiments have been

carried out (Sloboda 1992; Krumhansl 1997, 2000; Justin and Sloboda 2001; Zentner 2008) in

order to confirm these intuitions. These experiments have measured the emotional response to

music in terms of physiological responses, such as heart rate, skin response, breathing or

hormone secretion, as well as considering degrees of brain activation in those areas mostly

involved in emotional responses, i.e. hippocampus and amygdala, and expressive behaviour of

facial muscles. Anyway, everybody has ever experienced how music arouses different emotions

and, as a matter of fact, music is widely used for this purpose in films and advertising.6

The point is that the background music used in digital stories arouses different kinds of

emotions in their audience and that this emotional response is usually unconscious. After all, the

music is not the main focus of the listeners’ attention, so whereas the cortical areas of their brain

are busy interpreting the textual part and relating this with the images displayed, music is

working at sub-cortical levels, those in charge of emotions (Blood and Zaltorre 2001). It is in

this sense that background music and sounds can be said to have an evaluative function, since

they can elicit positive or negative feelings in the audience about the events narrated.

In My Iligan (MS1), the musical abstract is a piece of orchestral music in crescendo that

anticipates the intensity of the positive feelings of the narrator towards the city where he lives.

Similarly, in International Living-Southern China (IL1), the narrator tells his experience as a

leader of a group of students who went to China for the summer as part of the program World

Learning that promotes the knowledge of other countries in high school students. It is presented

6
Not so well known is the application of this knowledge about the effects of music on emotions to
different kinds of therapies for the treatment of emotional disorders or even to increase pain tolerance (see
Zentner et al. 2008 for a list of references on empirical research on these matters).
as a highly positive experience and this evaluation is provided not only by textual and visual

devices, but also, and probably at a less conscious level, by the music and sounds that can be

heard all through the narration. A lively, joyful music is at the background of the story and,

from time to time, also different real sounds can be heard which also evoke positive feelings:

people laughing, children playing, dancing in the street, an open market, etc…

In Rock Bottom: James’s Story (EH3), the changes in music at different sections match the

narrator’s evaluation for what is being told at that moment. For instance, the strings of a

contrabass go with the section where narrator tells about his girlfriend for the first time, evoking

a sense of sensuality that matches his words:

In 2004 I met my girlfriend [...] she was really beautiful, she always looked sexy.

This sensuality is mixed with a negative evaluation as the low pitch provides a cue that

something in that relationship is obscure and can go wrong. Also in this story, a negative feeling

is prompted when a strongly dissonant chord is heard at the moment when the narrator explains

that he had decided to change his life but his friends did not support this decision:

My friends though I was becoming a wimp.

In The Balcony (SC6), the song that serves as a coda (see above) comes from a Disney film and

is interpreted by a female singer. Therefore, apart from the message of hope conveyed by the

lyrics, also the soft music and the singer’s voice contribute to the positive evaluation of the

whole account of events narrated. Moreover, for a part of the audience who may know about the

source film of that music, a feeling of tenderness associated to childhood and innocence can be

prompted. In this sense, positive evaluation can also be activated by cultural conditioning

associated to specific songs or to certain kinds of music, such as religious hymns (WL1) or

political songs (SC2, SC5).

It is particularly interesting to observe that negative evaluation is usually associated with

silence. It is at those points in the story when something goes wrong that music generally stops.

In Manoj’s Story (EH6) when he finds out he is HIV positive (see above), in Bad Choices

(CN2), when the narrator and his brother are taken to Social Services,
 Next thing I know, we were sent to an apartment within the Social Service. We spent two
years in the System

Or in Rock Bottom (EH3) when he breaks up with his girlfriend:

But it was too late […] it was the beginning of the end and we broke up

In short, music and sound effects, as well as the lack of those, may have an evaluative function

in digital stories by arousing emotions and evoking positive or negative feelings associated to

the meanings that are conveyed through simultaneous texts and images.

7. Music and persuasion in digital stories

Digital stories also have a persuasive aim and music contributes to it. As a matter of fact, all

kinds of narrative, even fictional ones, have an element of persuasion: the idea, inherent to any

narrative, that the story is worth listening and remembering. Besides, all narratives are told with

a purpose, that of making the audience perceive the world differently. In the case of the digital

stories analysed in this work, which are fostered and published by non-profit organizations, this

persuasive function is quite straightforward, as they are created with the explicit intention of

changing attitudes and beliefs, for example about sexual discrimination, HIV, environmental

issues, etc.

Among the persuasive strategies that can be found in narratives, emotional involvement is one

of the most effective ones. The pathos, i.e. the appeal to the audience’s emotions, was already

considered by Aristotle one of the three main modes of persuasion and it is present in all kinds

of narratives, from “small stories” in our everyday life to films or literature. There is extensive

research on how this aim is achieved in narratives and the notions of identification (Cohen

2001), transportation (Gerrig 1993, Green and Brock 2000), absorption (Slater and Rouner

2002) and narrative engagement (Busselle and Bilandzic 2008; de Graaf et al. 2009) all

converge in the idea that readers/listeners feel “transported” into the narrative, identify with the

characters, adopt their personality and experience the same emotions as them. This emotional
involvement has been tested empirically in some experiments (Green and Brock 2000; Slater

and Rouner 2002; Busselle and Bilandzic 2009; de Graaf et al. 2009) that evidence how readers

are engaged in narratives.

As for multimodal narratives, it is quite straightforward that the combination of images and

sounds with the text strengthens the audience’s involvement in the story. Music, particularly,

reveals a powerful tool in order to achieve this narrative engagement that leads to persuasion

through several dimensions. One of them, emotional involvement, is easily inferred from the

potential of music to arouse emotions discussed in previous section. Several examples have

been provided of the way in which the music that accompanies text and images induce positive

or negative feelings and emotions of happiness or melancholy (IL1, SC6, MS1, SS1, etc.) with a

combination of different rhythms, pitch, tempos.

Another significant dimension of narrative engagement is attention. According to Green (2006),

when all mental resources are occupied with a narrative, there is no capacity left for critical

analysis of story content. This means that the attentional focus of the audience on the story leads

to a reduction of negative cognitive responses, increasing acceptance of beliefs implied by the

story (Green and Brock, 2000). Therefore, not only music is an effective way of catching and

maintaining the audience’s attention, as evidenced in section 5, but also the fact that it is

combined with text and images turns it into an outstanding persuasive device. In digital stories,

listeners strive to integrate the meanings provided by three different modes, that is, they have to

keep track on the narrator’s textual account, the series of images displayed simultaneously and

at the same time with the background sounds and music that go with them. Consequently, all

their senses are focused on the story, which necessarily reduces their critical capacity on what is

being told.

Finally, the identification of the audience with the narrator is another dimension of digital

stories to which music contributes and which leads to narrative engagement. Digital stories in

the sample are always narrated in the first person and it is allegedly the narrator’s own voice

that listeners can hear in them. This creates a feeling of closeness that makes the whole story

more credible and promotes a certain degree of personal identification with the narrator, no
matter if male or female. This feeling is reinforced when, instead of using commercial music or

pre-recorded one, storytellers make their own musical soundtrack, especially when the narrators

themselves sing the songs. This is the case of the Christian hymn in the abstract of The Day I

Made him Stop (WL1), or the humming of a political song at the background of Memories of a

Political Prisoner from Worcester (SC5), where the narrator finally sings the song at the end of

his narration. The narrator’s voice has a strong persuasive force both because of the intimacy it

creates with the audience and because of the emotions that it can convey.

Apart from this personal identification, it is also possible to speak of a cultural identification

when the music chosen as background has a strong cultural component. A Chinese bamboo flute

in IL1, African drums in BS1, a Spanish guitar in SC2, a Mexican trumpet in CN1, as well as

urban rock in CN2 or American gospel in CN3, they all situate the story from the very

beginning and the audience, even if from a different country, feel involved, as if transported into

that culture and get ready to experience the narrator’s emotions and thoughts in that context.

8. Conclusions

It must be noted that the analysis of the sample presented in this papers is merely a first

approach to the issue, as the number of stories collected and analysed is relatively small and not

representative enough. Besides, the qualitative analysis needs confirmation with a quantitative

one on a bigger corpus of narratives. Another important caveat must be noted as for the results

provided. As already stated, the creators of the digital stories in this sample are not experts, but

ordinary people who attended a workshop to learn how to tell a story and how to use the right

software to record their voices, edit images and sounds and put them all together. Therefore,

some of the effects pointed out in the previous sections, such as the persuasive effect of the

background humming of a melody instead of using commercial music, may be the consequence

of the lack of expertise of storytellers, or of copyright issues, as observed. Even so, the effects

on the audience are the same, independently of the real intentions and reasons of the narrator to
make it that way. Similarly, the creators of these digital stories are not likely to be

knowledgeable of the results of empirical tests on the effect of music on emotions that have

been exposed, but still they can intuitively choose the right music for the meanings they intend.

Much of this intuition derives from cultural contexts and from their experience with films, tv

commercials and the personal feelings and emotions evoked in them by different kinds of

music.

Nevertheless, and taking into account these limitations, the results of this first approach reveal

that background music and sounds in digital stories are an essential part of the meaning

construction in these narratives. Several functions have been identified – structural, attentional,

evaluative and persuasive – that belie the idea that it is merely an ornamental effect in

multimodal narratives. An integrative part of their whole meaning, music i) signals transitions

between the different segments that constitute the story, ii) directs the audience’s attention to the

most relevant parts or events, iii) provides an evaluative meaning that reveals the narrator’s and

guide the listener’s attitude towards the events narrated, and iv) provides the mechanisms that

can persuade the audience and produce a change in their attitudes and beliefs.

Music has revealed particularly effective for these purposes for several reasons. Firstly because

it is out of the focus of attention, as the listener’s cortical areas seem to be concentrated on the

textual and visual elements of the story, and so it tends to elicit an unconscious, emotional

response. Secondly, because it is culturally charged with positive and negative associations,

which provides an evaluative conditioning on what is being narrated. Thirdly, because it

contributes to supress the possible criticisms, as both hemispheres are busy with constructing

the meaning provided by text, images and sounds.

It goes without saying that most of these results could be extended to other kinds of multimodal

narratives, such as films. However, the digital stories selected for this paper constitute an

excellent object of analysis, because of their formal features. On the one side, the aim of these

narratives is quite straightforward and their persuasive intention is unmistakable. On the other

side, because of their brevity, the narrators are forced to compress the maximum amount of

information in a very short time span and for this reason they have to take the most out of
images and sounds. In more extensive multimodal narratives, the role of images and sounds are

allegedly the same but they may be less obvious. This could be an object for further research.

9. References

Alonso Belmonte, I; Molina, S. and Porto, M.D. (2013). “Multimodal Digital Storytelling:
Integrating Information, Emotion and Social Cognition”. Review of Cognitive
Linguistics 11(2): 369-385.
Blood, A. J., and Zatorre, R. J. (2001). Intensely pleasurable responses to music correlate with
activity in brain regions implicated in reward and emotion. Proceedings of National
Academy of Sciences, 98, 11818-11823.
Busselle, R. and Bilandzic, H. (2008). Fictionality and perceived realism in experiencing
stories: A model of narrative comprehension and engagement. Communication Theory,
18, 255-280.
Busselle, R. and Bilandzic, H. (2009) Measuring narrative engagement. Media Psychology 12
(4)
Cohen, J. (2001). Defining identification: A theoretical look at the identification of audiences
with media characters. Mass Communication and Society, 4(3), 245-264.
de Graaf A.,; Hoeken H.,; Sanders J.,; Beentjes H. (2009) The role of dimensions of narrative
engagement in narrative persuasion. Communications 34 (4) (2009), 385 405
Gerrig, R. J. (1993). Experiencing narrative worlds. New Haven, CT: Yale University Press
Green, M. C. and Brock, T. C. (2002). In the mind’s eye: Transportation-imagery model of
narrative persuasion. In T. C. Brock, J. J. Strange, and M. C. Green (Eds.), Narrative
impact: Social and cognitive foundations (pp. 315-341). Mahwah, NJ: Lawrence
Erlbaum.
Green, M. C. (2006). Narratives and cancer communication. Journal of Communication, 56(1),
163-183
Juslin P. and J. Sloboda (Eds.), (2001) Music and emotion: Theory and research. Oxford,
England: Oxford University Press.
Krumhansl C L, 1997 ``An exploratory study of musical emotions and psychophysiology''
Canadian Journal of Experimental Psychology 51 336 – 352
Krumhansl C L, 2000 ``Music and affect: Empirical and theoretical contributions from
experimental psychology'', in Musicology and Sister Disciplines: Past, Present, Future
Ed. D Greer (Oxford: Oxford University Press) pp 88 ^ 99
Labov, W. and Waletzky, J. (1967/1997). “Narrative Analysis: Oral Version of Personal
Experience”. Journal of Narrative and Life History, 7 (1-4): 3-38.
Labov, W. (1972). Language in the inner city. Philadelphia, University of Pennsylvania Press.
Lambert, Joe (2002/2013 4th edition) Digital Storytelling: Capturing Lives, Creating
Community. New York and London: Routledge
Langacker (2001) “Discourse in Cognitive Grammar” Cognitive Linguistics 12(2): 143-188.
Porto Requejo, M.D. and I. Alonso Belmonte (2014) From local to global: Visual strategies of
glocalisation in digital storytelling Language and Communication 39: 1–10
Romano, M. and M. D. Porto (2013) Emotion, attention and idiolectal variation in radio
narratives RESLA (Spanish Review of Applied Linguistics) Extra 1: 143-164
Slater, M. D., Rouner, D., and Long, M. (2006). Television dramas and support for
controversial public policies: Effects and mechanisms. Journal of Communication, 56,
235-252.
Sloboda, J. (1992). Empirical studies of emotional response to music. In M. R. Jones and S.
Holleran (Eds.), Cognitive bases of musical communication (pp. 33– 46). Washington,
DC: American Psychological Association.
Talmy (2007) “Attention Phenomena”. In Geraeerts and Cuyckens (eds) The Oxford Handbook
of Cognitive Linguistics. Oxford U. P. (264-293).
Talmy (2008) “Aspects of attention in language”. In Robinson and Ellis (eds) Handbook of
Cognitive Linguistics and Second Language Acquisition. London. Routledge (27-38)
Zentner, M., Grandjean, D., and Scherer, k (2008) Emotions Evoked by the Sound of Music:
Characterization, Classification, and Measurement. Emotion 8 (4): 494–521

Appendix
(Last accessed 15 July 2014)

BS1 The Home Land http://www.bristolstories.org/story/111

BS2 Despite my fears http://www.bristolstories.org/story/73

BW1 Never Too Old to Learn http://www.bbc.co.uk/wales/audiovideo/sites/yourvideo/page

s/catherine_collins_01.shtml
CN1 What the water gave me http://www.creativenarrations.net/node/16

CN2 Bad Choices http://www.creativenarrations.net/node/76

CN3 To every child http://www.creativenarrations.net/node/78

DS1 Stop Bullying http://www.youtube.com/watch?v=6YsFZjbH6_Y

DS2 Culture Clash http://www.youtube.com/watch?v=2wv37Co_aRU&list=PL

E2571E3579DEA1CF&feature=plcp
EH1 Nelao's Story (Stories from http://www.youtube.com/watch?v=ryAZlNjGot8
Namibia)
EH2 Ngamane's Story (Stories from http://www.youtube.com/watch?v=30UuSNy4-pY
Namibia)
EH3 James's Story Rock Bottom. http://www.youtube.com/watch?v=15Tb3vYgQ1U
(Stories from Namibia)
EH4 The Positive Way: Naresh's http://www.youtube.com/watch?v=rW4m8K--WS4
story. I'm not alone (Stories from
India)
 EH5 Men As Partners: Sibongiseni's http://www.youtube.com/watch?v=BI7DOgX6sAY
Story
EH6 The Positive Way: Manoj's http://www.youtube.com/watch?v=rRpxVwNHSp0
Story. (Stories from India)
EH7 Men As Partners: Gary's http://www.youtube.com/watch?v=mKHzMfhnpSw
Story. Mission
EH8 Men As Partners: Lillo's Story http://www.youtube.com/watch?v=Vs4PHB62TPs

IL1 International Living in Southern https://www.youtube.com/watch?v=pBR_gN87Wnc

China
MS1 My Iligan https://www.youtube.com/watch?feature=player_detailpage
&v=_aycmvgXp-0
SC1 Untitled- a transgender boy http://64.13.206.59/stories/index.php?cat=7

SC2 Sacrificios http://64.13.206.59/stories/index.php?cat=4

SC3 Privilege http://64.13.206.59/stories/index.php?cat=3

SC5 Memories of a Political Prisoner http://64.13.206.59/stories/index.php?cat=2

from Worcester
SC6 The Balcony http://64.13.206.59/stories/index.php?cat=2

SC7 Mixed Race Me http://64.13.206.59/stories/index.php?cat=7

SC8 My Shoes http://64.13.206.59/stories/index.php?cat=2

SC9 Distance https://www.youtube.com/watch?v=qhd9SDFjopU

SS1 A Struggle Within Reach http://www.youtube.com/watch?v=Z_D9arQ-4WU

TL2 Whatever happened to Miss http://www.bbc.co.uk/humber/telling_lives/humber_interme

Pears? diary4.shtml
UM1 Real Men Do Housework http://stories.umbc.edu/projects.php?movie=ELC054_S09By
ungchangKim.flv
WL1 The day I Made Him Stop http://www.youtube.com/watch?v=hneAZCEl5v4

View publication stats