Emotion Recognition in Comics:

The Effect of Visual Morphemes in Visual Narrative Contexts

Hyorim Han & Jiyoun Choi*

Department of Social Psychology,
Sookmyung Women’s University

Abstract
In comics, the term “visual morphemes” refers to one type
of graphic structure that can be combined with other graphic
elements to generate diverse meaning. For instance, the visual
morphemes of whirlwind-shaped lines indicate confusion if they
are placed above a person’s head. Prior empirical research has
shown that such emotive visual morphemes do in fact help comic
readers recognize the emotions of comic characters. However,
there has been little evidence of the effect of emotive visual
morphemes when emotion recognition is required in narrative
contexts where multiple images are arranged to form a story, as
opposed to when in solitary images of character-morpheme dyads.
This study thus examined how emotive visual morphemes affect
the identification of character’s emotions in narrative contexts
consisting of three image panels. Results showed that emotion
recognition was slower when the visual morphemes were not
corresponding to the emotions of characters than when they were
corresponding or when they were not provided at all. The findings
thus add to our understanding of visual morpheme processing by
providing empirical support for the emotive visual morpheme
effects in the visual narrative structure.

Key words: emotion recognition, comics, visual morpheme, visual

narrative, visual language

Received 07 September 2023; Revised 26 September 2023; Accepted 27 September 2023

* Correspondence: jiyoun.choi@sookmyung.ac.kr
Journal of Cognitive Science 24(3): 337-354 September 2023
©2023 Institute for Cognitive Science, Seoul National University
338 Hyorim Han & Jiyoun Choi

1. Introduction

Comics comprise sequential images to visually provide narratives. Comics

exist across cultures; for instance, webtoons, a type of digital comic where
images are arranged vertically, are popular in Korea. Drawing comics is not
arbitrary but is guided by a set of patterns shared by individuals, known
as visual languages (Cohn, 2018b). As spoken languages use phonology
(sound) to convey meaning, visual languages do with graphic structure
(drawn lines). Visual languages contain and utilize their visual vocabularies
also in a similar way as spoken languages do, such that, for instance, there
are some visual morphemes (e.g., speech balloon) that cannot stand alone
and must be attached to their sources (e.g., speaking person), similar to how
bound morphemes in spoken language must combine with a stem (Cohn &
Maher, 2015; Cohn, Murthy, & Foulsham, 2016; Cohn & Foulsham, 2022).
The visual morphemes, also known as “pictorial runes” (Forceville, 2005;
Kennedy, 1982) or “emanata” (Walker, 1980, as cited in Ojha, Forceville,
& Indurkhya, 2021), are one type of the meaning units in visual narratives
(Cohn, 2012), and they generate diverse meaning from their combinations
with other graphic units in different ways. For example, curving lines above
a mug of tea conventionally mean heat, straight lines (also called motion
lines) attached to a moving object indicate the movement paths, and vertical
whirlwind-shaped lines above a person’s head typically show confusion.
Several studies have conducted corpus analyses in visual languages and
documented the visual morpheme inventory and frequency in certain
comics (Abbott & Forceville, 2011; Cohn & Ehly, 2016; Forceville, 2011;
Forceville, Veale & Feyaerts, 2010), providing systematic insights into the
category and use of visual morphemes.
Recent studies have further examined how visual morphemes are
processed (Cohn et al., 2016; Cohn & Maher, 2015; Ojha et al., 2021). For
example, Ojha et al. (2021) examined the role of visual morphemes in
 Emotion Recognition in Comics: The Effect of Visual Morphemes in Visual Narrative Contexts 339

identifying the emotional state of comic characters. In Ojha et al. (2021),

participants were presented with individual panels of comic characters with
no or minimal facial expressions along with different conventional visual
morphemes around heads (e.g., multiple spikes, droplets), and they were
asked to choose the characters’ emotions from alternatives. The findings
indicated that certain emotive visual morphemes provoked biased responses
to particular emotions, demonstrating that visual morphemes provide
information about the emotions of characters even when only a single panel
is given.
What remains somewhat unclear, however, is to what degree visual
morphemes would affect the processing of emotions in narrative contexts
consisting of multiple panels (not in solitary images). This study, thus, set
out to examine the impact of emotional visual morphemes on emotion
recognition in the context of visual narratives. Specifically, we investigate
whether processing of emotional meaning derived from an entire narrative
may be affected by conventional emotive visual morphemes inserted in a
single panel.
To this end, we provide participants with three-panel visual narratives,
either with or without emotive visual morphemes, and ask them to infer
the emotion of a main character who is directly involved in the narrative’s
episodes and, as a result, who is anticipated to feel certain emotions. We
further manipulate the type of visual morphemes to be either congruent
or incongruent with the expected emotions of the main character (e.g.,
angry or happy visual morphemes inserted into narratives where the main
characters are expected to feel angry). By comparing those three conditions
(i.e., congruent, incongruent, and empty morpheme conditions), we
investigate whether the processing of emotion recognition will be facilitated
by congruent morphemes and if it will be hindered by incongruent
morphemes. Ratings on the overall comprehensibility and naturalness of
all stimuli as well as participants’ fluency with reading comics are also
340 Hyorim Han & Jiyoun Choi

obtained. The experimental work offers some important insights into the
processing of visual morphemes in visual narrative structures.

2. Method

2.1 Participants
Participants were 36 native speakers of Korean (Mage = 21.47, SDage = 0.17)
who had no difficulties in reading comic scripts on a screen. They were
recruited by flyers posted at Sookmyung Women’s University, Seoul and
were paid for participation. All participants have signed a consent form to
participant in the experiment.

2.2 Stimuli
2.2.1 Comic scripts drawing
A total of 90 webtoon-style comic scripts were created by the first author,
with 30 for each of anger, surprise, and embarrassment target emotion
conditions. The stimuli were created using Adobe Photoshop and Clip
Studio and were saved as PNG image files with a size of 1200 by 3200
pixels. Each comic script consists of three panels arranged vertically. It
follows the canonical comic narrative structure of Establisher-Initiation-
Peak, where the Establisher (the first panel) describes the background of
the story, the Initiation (the second panel) is the beginning of an event,
and the Peak (the last panel) describes the core of the event (Cohn, 2013,
2014). All stimuli scripts are about the stories of Korean college students’
daily life, with a main character who is directly involved in the events and
who, as a result, experiences the target emotions. Any visual morphemes
corresponding to the target emotions are not included in the stimuli, so
that the target emotions should be inferred by the events of the stories (see
Figure 1 for example scripts). The main character was colored orange, while
other characters were colored light yellow. Other objects or backgrounds
 the beginning of an event, and the Peak (the last panel) describes the core
of the event (Cohn, 2013, 2014). All stimuli scripts are about the stories
of Korean college students’ daily life, with a main character who is
directly involved in the events and who, as a result, experiences the target
emotions. Any visual morphemes corresponding to the target emotions
areEmotion
not included
Recognition in the stimuli,
in Comics: soVisual
The Effect of thatMorphemes
the target emotions
in Visual should be
Narrative Contexts 341
inferred by the events of the stories (see Figure 1 for example scripts).
The main character was colored orange, while other characters were
colored
were light
in black andyellow.
white, Other
and textsobjects
wereorin backgrounds
Korean. were in black and
white, and texts were in Korean.

Figure
Figure 1. Examples
1. Examples of of 3-panel
3-panel webtoon-style
webtoon-style comic
comic scriptsforfora)a)anger,
scripts anger,b)
b) embarrassment,
embarrassment, and c)and c) surprise.
surprise.

2.2.2 A judgement study for selection

To validate the quality and understandability of the comic scripts, a
judgment study was carried out with 12 participants (Mage = 23.08, SDage =
3.50) recruited from Sookmyung Women’s University in Seoul. None of
them participated in the main experiment. The participants were presented
with the 90 comic scripts one by one and were asked to read them with no
time limit. They were then asked to classify emotions that a main character
in the last panel would experience as one of three emotions ("anger",
"surprise", "embarrassment") or as "other emotions". After each classification,
participants had to indicate the understandability of the stories on a scale
of 1 (not very understandable) to 7 (very understandable). All stimuli and
response options were shown on a screen, and participant responses were
given by clicking their choice on the screen with a computer mouse.
342 Hyorim Han & Jiyoun Choi

We computed hit rates for each comic script as a measure of how well
the participants recognized the intended emotion. One-sample t-tests
were performed to compare each hit rate to the chance level of 0.25 (in
a 4-alternative forced choice task). Twenty-five stimuli with hit rates
above chance were selected. Additionally, 28 stimuli were further selected
(from the rest of 65 stimuli) on the basis of coherence in the participants’
classification. That is, in spite of the hit rates being lower than chance
(i.e., the failure to recognize the intended emotion), if the rates of what
most participants chose as their answer were significantly higher than
the chance level, the stimuli were included. The understanding ratings
for each of the selected 53 stimuli were computed and compared to the
midpoint of the scale (i.e., 4) using one-sample t-tests. A total of 10 stimuli
with an understanding rating significantly lower than 4 (i.e., not well-
comprehensible stories) were excluded, leaving 43 experimental stimuli:
22 for anger, 15 for surprise, and 6 for embarrassment target emotion
conditions.

2.2.3 Manipulation
The selected 43 stimuli were manipulated by inserting visual morphemes
above the main character’s head in the last panels. The visual morphemes
were either corresponded (congruent condition) or did not correspond to
the target emotions (incongruent condition). The original stimuli, which
had no visual morphemes for the target emotions, were also used for the
empty condition. For the congruent condition, three visual morphemes that
are conventionally used to express each target emotion were used (Cohn,
2018a; Cohn & Ehly, 2016; Ojha et al., 2021) (1) the shape of smoke clouds
for anger; (2) the shape of a spark for surprise; and (3) the shape of beads
of perspiration for embarrassment (see Figure 2). For the incongruent
condition, the visual morpheme of the heart shape that was widely used
to represent happiness was selected. See figure 3 for examples of the three
 visual morphemes that are conventionally used to express each target
emotion were used (Cohn, 2018a; Cohn & Ehly, 2016; Ojha et al., 2021)
(1) the shape of smoke clouds for anger; (2) the shape of a spark for
Emotion Recognition
surprise; and (3) thein Comics:
shape TheofEffect
beadsof Visual Morphemes in Visual
of perspiration for Narrative Contexts 343
embarrassment
(see Figure 2). For the incongruent condition, the visual morpheme of
the heart shape that was widely used to represent happiness was selected.
visual
Seemorpheme
figure 3 forconditions.
examples of the three visual morpheme conditions.

Figure 2. The visual morphemes to convey a) anger, b) embarrassment,

Figure 2. The and
c) surprise, visual morphemes to convey a) anger, b) embarrassment, c)
d) happiness.
surprise, and d) happiness.
Our visual morpheme manipulation entails that stimuli in the
incongruent condition always had the happiness visual morphemes (the
Our visual
heart morpheme
shape). manipulation
To distract entails we
from this pattern, thatincluded
stimuli in22 the incongruent
filler comic
scripts either with congruently inserted happiness visual morphemes
condition always had the happiness visual morphemes (the heart shape). (5
stimuli), with incongruently inserted anger, surprise, or embarrassment
To distract from this(4pattern,
visual morphemes we each),
stimuli for included 22 filler
or with comic
no visual scripts for
morpheme either
happiness-induced stories (5 stimuli). The filler comic scripts
with congruently inserted happiness visual morphemes (5 stimuli), with were
created by the first author, as were the experimental stimuli.
incongruently inserted anger, surprise, or embarrassment visual morphemes
(4 stimuli for each), or with no visual morpheme for happiness-induced
stories (5 stimuli). The filler comic scripts were created by the first author,
as were the experimental stimuli.

Figure Figure 3. Manipulation

3. Manipulation ofofcomic
comic scripts a) without
scripts visual morphemes,
a) without b)
visual morphemes,
with congruent or c) incongruent visual morphemes above the main
b) withcharacter’s
congruent orinc)theincongruent
head last panel. visual morphemes above the main
character’s head in the last panel.
2.3 Procedure
Participants were tested individually in an experimental room at
Sookmyung Women’s University. They were seated at a comfortable
viewing distance from a screen which showed the stimuli. Written
instructions were presented on a screen, asking them to read a 3-panel
webtoon stimulus, to press the space key on a keyboard when they
understood the stimulus, and to indicate the emotion they thought the
344 Hyorim Han & Jiyoun Choi

2.3 Procedure
Participants were tested individually in an experimental room at
Sookmyung Women’s University. They were seated at a comfortable
viewing distance from a screen which showed the stimuli. Written
instructions were presented on a screen, asking them to read a 3-panel
webtoon stimulus, to press the space key on a keyboard when they
understood the stimulus, and to indicate the emotion they thought the
main character, colored in orange, had experienced among anger, surprise,
embarrassment, happiness, and other emotions. The instructions further
asked them to rate how easy it was to understand the story in the stimulus
on a scale of 1 (not very understandable) to 7 (very understandable) and
how natural the drawing expressions of the stimulus were on a scale of 1
(very contrived) to 7 (very natural).
Task trials began with a fixation mark in center screen for 300 ms,
followed by a blank screen for 100 ms, and one stimulus comic script in
center screen until participant pressed the space key. Five response options
(Anger, Surprise, Embarrassment, Happiness, and Other emotions) were
then presented, and participants responded by pressing the corresponding
keys on a keyboard. After that, the 7-point understandability rating scale
and the 7-point drawing naturalness rating scale (with descriptions at
the end points) were displayed one by one until participants responded.
Responses were given by pressing the number keys on a keyboard.
A total of 65 stimuli, comprised of 43 experimental and 22 filler comic
scripts, were presented in random order. To avoid the effects of repetition,
the 43 experimental stimuli were divided into three sets of 14, 14, and 15
comic scripts (each set consisting of 5 for surprise, 2 for embarrassment,
and 7 or 8 for anger conditions), and each set was assigned to each of the
congruent, incongruent, and empty visual morpheme conditions. Thus,
every experimental comic script occurred only once in the congruent,
incongruent, or empty condition. The set assignment to the visual
 Emotion Recognition in Comics: The Effect of Visual Morphemes in Visual Narrative Contexts 345

morpheme conditions was counterbalanced. Six practice trials were given,

containing unique stimuli (i.e., not used in the main experiment). The
experiment took about 20 minutes. The experiment was implemented in
PsychoPy 2021.1.4 and conducted on a laboratory computer (Peirce, J. &
Open Science Tools Ltd., 15 April 2021; Peirce & MacAskill, 2018).
The participants next completed the Visual Language Fluency Index
(VLFI) questionnaire, which was translated into Korean, to assess their
fluency with reading comics (Cohn, 2014). The VLFI contains self-
assessment questions regarding (i) frequency of comic reading habits across
a variety of genres on a scale of 1 to 7 (1=never, 7=always), (ii) frequency
of drawing comics on the same 7-point scale, (iii) comic reading expertise
on a scale of 1 to 5 (1=below average, 5=above average), and (iv) drawing
ability on the same 5-point scale. Note that we modified the comic genres in
the first section of assessment by adding a common stylistic genre in Korea,
webcomics to the existing stylistic genres in the original questionnaire
(comic books and comic strips). The VLFI scores were calculated following
the formula provided in Cohn (2014); higher values indicate greater reading
comic fluency. The study was approved by the Sookmyung Women’s
University Institutional Review Board (approval number: SMWU-
2104-HR-021-03).

3. Results

3.1 Emotion inference task

Responses to the experimental stimuli were submitted for analysis. The
mean reaction times (RTs; measured from stimulus onset till the responses)
and mean proportions of correct responses were calculated for by-subject
repeated-measure ANOVAs, with a within-subject factor Visual morpheme
(empty, congruent, incongruent).
The ANOVA on accuracy showed no main effect of Visual morpheme.
 Responses to the experimental stimuli were submitted for analysis.
346
The mean reaction timesHyorim (RTs;Han & Jiyoun from
measured Choi stimulus onset till the
responses) and mean proportions of correct responses were calculated for
by-subject
This indicatesrepeated-measure
that accuracy isANOVAs,
comparablewith a within-subject
across factor
the visual morpheme
Visual morpheme (empty, congruent, incongruent).
conditions
The ANOVA (mean onaccuracy:
accuracy empty
showed= no 0.45, congruent
main effect of=Visual
0.44, incongruent
morpheme,
= 0.45). Ascomparable
indicating shown in Figure 4, the
accuracy patterns
across for RTs
the visual were different
morpheme from
conditions
(mean accuracy: empty = 0.45, congruent = 0.44, incongruent
those for accuracy such that RTs were noticeably slower in the incongruent= 0.45).
As shown in Figure 4, the patterns for RTs were different from those
condition than in both congruent and empty conditions. The analysis indeed
for accuracy such that RTs were noticeably slower in the incongruent
showed a significant
condition than in both maincongruent
effect of Visual morpheme
and empty (F(2, 70)
conditions. The= analysis
10.70, p
2
< .001, showed
indeed ηp = .234). Follow-upmain
a significant pairwise comparisons
effect (Bonferroni-corrected)
of Visual morpheme (F(2, 70) =
2
10.70, < .001, ηslower
revealedpsignificantly p = RTs .234). Follow-up
in the pairwise
incongruent comparisons
than congruent (t(35)
(Bonferroni-corrected)
= -4.92, p < .001) and empty conditions (t(35) = -2.59, p < .05), with the
revealed significantly slower RTs in no
incongruent than congruent (t(35) = -4.92, p < .001) and empty
difference between
conditions (t(35) =the-2.59,
congruent
p < and
.05),empty
with conditions (t(35)between
no difference = -1.72, pthe=
0.283).
congruent and empty conditions (t(35) = -1.72, p = 0.283).

Figure
Figure 4.
4. Mean
Mean emotion
emotion inference
inference response
response times
times (RTs)
(RTs) for
for three
three visual
visual
morpheme conditions. Error bars show standard errors. * p < 0.05;
morpheme conditions. Error bars show standard errors. * p < 0.05; **** p
p<
< 0.001.
0.001.

3.2 Understandability
3.2 Understandability rating
rating
The mean understandability ratings were calculated for correct
The mean understandability ratings were calculated for correct emotion
emotion inference responses, and they were analyzed in the same way as
inference responses, and they were analyzed in the same way as described
above. As shown in Figure 5a, the understandability ratings were higher
for the empty and congruent conditions than for the incongruent condition.
 Emotion Recognition in Comics: The Effect of Visual Morphemes in Visual Narrative Contexts 347

described above.
As shown ANOVA
A by-subject in Figureindeed
5a, theshowed
understandability
a significantratings were higher
main effect for
of Visual
the empty and congruent conditions than for the incongruent condition.
morpheme
A by-subject (F(2,
ANOVA70) =indeed
19.70,showed
p < .001, η p2= .360),main
a significant andeffect
the follow-up
of Visual
analyses (Bonferroni-corrected) revealed significantly
2 higher
morpheme (F(2, 70) = 19.70, p < .001, ηp = .360), and the follow-up ratings for the
analyses
empty and(Bonferroni-corrected)
congruent conditions thanrevealed
for thesignificantly
incongruenthigher ratings
condition for
(empty
the empty and congruent conditions than for the incongruent condition
vs. incongruent: t(35) = 4.893, p < .001; congruent vs. incongruent: t(35) =
(empty vs. incongruent: t(35) = 4.893, p < .001; congruent vs.
5.521, p < .001);
incongruent: there
t(35) was nopdifference
= 5.521, < .001); between
there wasthenoempty and congruent
difference between
conditions (t(35) = 0.164, p = 1.0).
the empty and congruent conditions (t(35) = 0.164, p = 1.0).

Figure
Figure5.5.a)a)Mean
Meanunderstandability ratings
understandability and b)and
ratings mean
b) naturalness ratings
mean naturalness
ratings
for threefor three
visual visual morpheme
morpheme conditions.conditions. Error standard
Error bars show bars show standard
errors. ** p
<errors.
0.001.** p < 0.001.

3.3Drawing
3.3 Drawingnaturalness
naturalness rating
rating
The mean ratings for drawing naturalness of
The mean ratings for drawing naturalness of correct
correct emotion inference
emotion inference
responses were analyzed in the same way as understandability ratings.
responses
Similar were
to theanalyzed in the same ratings,
understandability way as understandability ratings.
the ratings for drawing
Similar to thewere
naturalness understandability
higher for theratings,
empty the
andratings for drawing
congruent naturalness
conditions than for
the incongruent
were higher for condition
the empty(see andFigure 5b), conditions
congruent and there was thanindeed
for thea
significant
incongruent main effect of Visual morpheme (F(2, 70) = 25.10, p
condition (see Figure 5b), and there was indeed a significant < .001,
ηp2 = .418); follow-up analyses (Bonferroni-corrected) 2 showed
main effect ofhigher
significantly Visualratings
morpheme
for the(F(2, 70) and
empty = 25.10, p < .001,
congruent ηp = .418);
conditions than
follow-up analyses (Bonferroni-corrected)
for the incongruent showed significantly
condition (empty vs. incongruent: higherp
t(35) = 5.070,
< .001;for
ratings congruent
the empty vs.and
incongruent: t(35) = 5.353,
congruent conditions p < the
than for .001), with no
incongruent
condition (empty vs. incongruent: t(35) = 5.070, p < .001; congruent vs.
348 Hyorim Han & Jiyoun Choi

incongruent: t(35) = 5.353, p < .001), with no difference between the empty
and congruent conditions (t(35) = 0.483, p = 1.0).

3.4 Effect of fluency with reading comics

The mean VLFI score across participants was 11.19 (range = 5.50 – 23.04,
SD = 4.75). Correlations were computed between VLFI scores and the
dependent variables (i.e., emotion inference accuracy, emotion inference
RTs, understandability ratings, and drawing naturalness ratings). No
significant correlations were found.

4. Discussion

The current study investigated how conventionalized visual morphemes

conveying emotive meaning would affect the recognition of character’s
emotions in a visual narrative context. To this end, 3-panel comics in the
canonical narrative sequence (Establisher-Initial-Peak) were provided
to participants, and they were asked to infer the emotions of the main
character who was directly involved in episods that occurred in the
narratives. Our results showed that emotion recognition became slower
when the emotional visual morphemes that were provided in the last panels
were incongruent with the character’s emotions than when they were
congruent or when they were not given. The effect of visual morphemes
was also found in understandability rating and drawing naturalness rating,
such that narrative sequences including incongruent visual morphemes
were rated less understandable and less natural than those with congruent
or no morphemes. These findings thus indicate that the emotional visual
morphemes are processed by readers in such a way that the inappropriate
individual emotional morphemes in a single panel not only impede the
processing of emotive meaning that emerges from a whole narrative but
also have an impact on the comprehensibility and naturalness ratings of the
 Emotion Recognition in Comics: The Effect of Visual Morphemes in Visual Narrative Contexts 349

overall narratives.
The visual morphemes, however, had no effect on emotion inference
accuracy, as shown by comparable accuracy across the congruent,
incongruent, and empty visual morpheme conditions. This demonstrates
that congruent morphemes failed to provide additional benefits for
identifying emotive meanings in the narrative contexts and that incongruent
morphemes alone were unable to override the semantic information
retrieved from the narratives. This result differs from that of Cohn et al.
(2016), who showed the visual morpheme effect on the interpretations of
the overall meaning of morpheme-face dyads, with the interpretations being
more consistent under matching than mismatching morpheme conditions.
This discrepancy can be explained by the distinct methodologies applied
in the two studies. While Cohn et al. (2016) used an open-ended task with
no right or wrong answers and measured the consistency of participants’
responses, we employed a 5-alternative forced choice task and measured
the correctness. The type of stimulus also varied so that Cohn et al. (2016)
used independent 1-panel comic images, whereas 3-panel comic sequences
were employed in this study to provide the narrative contexts. These
methodological differences may have resulted in assessing distinct aspects
of visual morpheme comprehension processes, which in turn may have led
to the inconsistent findings.
Another interesting finding of the current study was that participants’
fluency with reading comics, which was measured by VLFI, did not
correlate with accuracy and RTs of emotion recognition or any ratings.
The results are in accordance with the pattern observed by earlier studies,
including those by Cohn & Foulsham (2022), who found no correlation
between VLFI scores with comprehension ratings at morpheme-face
dyads, as well as Ojha et al. (2021), who showed no difference in judgment
on comic characters’ emotions between groups that were familiar versus
unfamiliar with comics. In contrast, others found correlative effects of
350 Hyorim Han & Jiyoun Choi

comic fluency on visual morpheme processing (e.g., Cohn & Maher,

2015; Cohn et al., 2016). For instance, Cohn & Maher (2015) found that
participants who had the higher VLFI scores took longer time to process
panels with anomalous visual morphemes. The reason for this inconsistency
is not clear but it might be related to differences in the types or degrees of
task difficulty utilized in different studies. A further study with more focus
on the VLFI effects is therefore suggested.
The main goal of the current study was to investigate how emotional
visual morphemes function in the context of visual narratives. The most
significant finding to emerge from this study is that emotional visual
morphemes affected the speed of identifying characters’ emotions, with
faster emotion recognition for congruent and empty visual morpheme
conditions than for incongruent conditions. The majority of experimental
research on visual morpheme processing has used isolated images of a
single face with a visual morpheme that either matches or mismatches the
facial expression as stimuli (Cohn & Foulsham, 2022; Cohn et al., 2016;
Kendall, Raffaelli, Todd, Kingstone, & Cohn, 2020). Although these studies
have shown the congruency effects of visual morphemes on how the
morpheme-face dyads are processed, it has remained unclear if and to what
extent the congruency effect may be observed in the visual narrative context
comprising sequences of images. The findings of the current study thus
expand our understanding of visual morpheme processing by providing
new empirical evidence demonstrating the congruency effects of emotional
visual morphemes in the visual narrative structure. Some caution, however,
should be applied when generalizing our findings given the lack of male
participants in the sample. Further research with other sets of stimuli and
various tasks, as well as other samples of participants (e.g., gender-balanced
samples), is required to better understand how visual morphemes are
processed in visual narrative contexts and how comic reading proficiency
relates to this.
 Emotion Recognition in Comics: The Effect of Visual Morphemes in Visual Narrative Contexts 351

Acknowledgements A part of the results was presented to the Korean Society for
Cognitive & Biological Psychology Annual Conference 2022.

Funding This work was supported by Sookmyung Women's University Research Grants
1-2203-2020.

Declarations

Ethics Approval This study was approved by the Sookmyung Women's University
Institutional Review Board (approval number: SMWU-2104-HR-021-03).

Consent to Participate and Consent for Publication Written consent was obtained from
all participants.

Conflict of Interest The authors declare no conflict of interest.

References

Abbott, M., & Forceville, C. (2011). Visual representation of emotion in manga:

Loss of control is Loss of hands in Azumanga Daioh Volume 4. Language
and Literature: International Journal of Stylistics, 20(2), 91–112.
https://doi.org/10.1177/0963947011402182
Cohn, N. (2012). Comics, linguistics, and visual language: The past and future
of a field. In F. Bramlett (Ed.), Linguistics and the Study of Comics (pp.
92–118). New York: Palgrave MacMillan.
http://www.visuallanguagelab.com/P/NC_Comics&Linguistics.pdf
Cohn, N. (2013). Visual Narrative Structure. Cognitive Science, 37(3), 413–452.
https://doi.org/10.1111/cogs.12016
Cohn, N. (2014). You’re a good structure, Charlie Brown: the distribution of
narrative categories in comic strips. Cognitive Science, 38(7), 1317–1359.
https://doi.org/10.1111/cogs.12116
352 Hyorim Han & Jiyoun Choi

Cohn, N. (2018a). Combinatorial Morphology in Visual Languages. In G. Booij

(Ed.), Advances in Construction Morphology (pp. 175–199). London:
Springer. https://doi.org/10.1007/978-3-319-74394-3_7
Cohn, N. (2018b). Empirical Comics Research. In A. Dunst, J. Laubrock, & J.
Wildfeuer (Eds.), Empirical Approaches to Comics Research: Digital,
Multimodal, and Cognitive Methods (1st ed., pp. 305–328). New York:
Routledge. https://doi.org/10.4324/9781315185354
Cohn, N., & Ehly, S. (2016). The vocabulary of manga: Visual morphology in
dialects of Japanese Visual Language. Journal of Pragmatics, 92(2), 17–29.
https://doi.org/10.1016/j.pragma.2015.11.008
Cohn, N., & Foulsham, T. (2022). Meaning above (and in) the head:
Combinatorial visual morphology from comics and emoji. Memory &
Cognition, 50(7), 1381–1398. https://doi.org/10.3758/s13421-022-01294-2
Cohn, N., & Maher, S. (2015). The notion of the motion: the neurocognition of
motion lines in visual narratives. Brain Research, 1601, 73–84. https://doi.
org/10.1016/j.brainres.2015.01.018
Cohn, N., Murthy, B., & Foulsham, T. (2016). Meaning above the head:
combinatorial constraints on the visual vocabulary of comics. Journal of
Cognitive Psychology, 28(5), 559–574.
https://doi.org/10.1080/20445911.2016.1179314
Forceville, C. (2005). Visual representations of the idealized cognitive model of
anger in the Asterix album La Zizanie. Journal of Pragmatics, 37(1), 69-88.
Forceville, C. (2011). Pictorial runes in Tintin and the Picaros. Journal of
Pragmatics, 43(3), 875–890. https://doi.org/10.1016/j.pragma.2010.07.014
Forceville, C., Veale, T., & Feyaerts, K. (2010). Balloonics: The visuals of
balloons in comics. In J. Goggin & D. Hassler-Forest (Eds.), The rise and
reason of comics and graphic literature: Critical essays on the form (pp.
56-73). Jefferson NC: McFarland.
Kendall, L. N., Raffaelli, Q., Todd, R. M., Kingstone, A., & Cohn, N. (2020).
Show me how you feel: Iconicity and systematicity in visual morphology.
In P. Perniss, O. Fischer, & C. Ljungberg (Eds.), Operationalizing Iconicity
(Iconicity in Language and Literature series) (pp. 214–229). Amsterdam:
John Benjamins.
Kennedy, J. M. (1982). Metaphor in pictures. Perception, 11(5), 589–605. https://
doi.org/10.1068/p110589
 Emotion Recognition in Comics: The Effect of Visual Morphemes in Visual Narrative Contexts 353

Ojha, A., Forceville, C., & Indurkhya, B. (2021). An experimental study on the
effect of emotion lines in comics. Semiotica, 2021(243), 305–324. https://
doi.org/10.1515/sem-2019-0079
Peirce, J. W., & MacAskill, M. R. (2018). Building Experiments in PsychoPy.
London: Sage.
Peirce, J., & Open Science Tools Ltd. (15 April 2021). Standalone PsychoPy
2021.1.4 for 64bit Windows (using Python3.6) [Computer Software].
Retrieved from https://github.com/psychopy/psychopy/releases