Download as pdf or txt
Download as pdf or txt
You are on page 1of 6

Facially Expressed Emotions and Hedonic Liking on

2022 17th International Workshop on Semantic and Social Media Adaptation & Personalization (SMAP) | 978-1-6654-8727-6/22/$31.00 ©2022 IEEE | DOI: 10.1109/SMAP56125.2022.9942096

Social Media Food Marketing Campaigns:Comparing


Different Types of Products and Media Posts

Katerina Tzafilkou Foteini Rafailia Panavou Anastasios A. Economides


SMILE lab SMILE lab SMILE lab
University of Macedonia University of Macedonia University of Macedonia
Thessaloniki, Greece Thessaloniki, Greece Thessaloniki, Greece
tzafilkou@uom.edu.gr fotini138@hotmail.com economid@uom.gr
katerinatzaf@gmail.com

Abstract— When viewers watch food video campaigns in advertisements can influence the viewers’ food choices, this
social media, they are experiencing various emotions. This associates food advertisements with food consumption
study explores these viewers’ emotional states that can be decisions [2].
detected through facial expressions, as well as their perceived Face tracking is a popular research methodology for
hedonic liking of the product. The study consists of five emotion detection because facial expressions are essential to
experiments, one per product/stimulus including hedonic and the expression of emotions. Researchers agree that changes in
utilitarian foods. Seventy-six viewers successfully facial expressions can lead to new objective measures of the
participated in the tasks, and 164 valid video records were affective responses to foods, since the internal feelings are
analyzed by FaceReader Online. The results indicated that frequently accompanied by them [2]. Furthermore, facial
FaceReader Online can capture differences in emotional expressions might provide additional information on fast
responses elicited by different types of food and media posts changing emotions during food consumption. Several
in social media marketing campaigns. Sadness prevailed all previous studies examined food-evoked emotions through
other emotions throughout the campaigns, while arousal explicit dynamic approaches (e.g., Temporal Dominance of
remained at levels of inactivity. The responses in the hedonic Emotions) and implicit face tracking methodologies [3]-[7].
product campaign were significantly less negative (in terms of However, most of these studies included sensory attributes in
sadness and anger) than those in the campaigns of utilitarian physical human-product experiences (e.g., while tasting the
products. The hedonic liking ratings indicated significant product) and have not been conducted in the context of digital
differences among campaigns of similar content and media campaigns. In a face tracking study, Mahieu et al. [8] showed
characteristics, implying the determinant role of other factors, that watching an advertisement triggers stronger implicit
like individual product preferences and sensory expectations. emotions than smelling a perfume or tasting a chocolate. This
The results contribute to understanding consumer emotions finding is very contributing to our research since it confirms
during watching food related campaigns in social media. that strong implicit emotions can be elicited in exposure to
food advertisements, similarly to sensory based/embodied
Keywords— consumer emotions, face tracking, food experience.
marketing campaigns, social media campaigns, video ads Moreover, most studies in the field examine typically
hedonic products, while research on utilitarian products is
missing. Finally, although several studies examined the
I. INTRODUCTION characteristics of video campaigns that tend to retain user
attention and evoke emotional responses [9], the effects of
Measuring emotions can be crucial in the field of social social media image posts and video campaigns on food
media food market research since psychological and social elicited emotions and preferences is significantly under
factors can elicit food-related consumption behaviors. researched.
Emotions are also crucial determinants of selecting and Motivated by the above, the main purpose of this study is
consuming hedonic or utilitarian foods. Hedonic foods (e.g., to examine the viewers’ emotions that can be detected via face
ice cream) primarily serve to provide sensory pleasure, tracking software while they are watching social media
whereas utilitarian foods (e.g., vegetables, bread) typically campaigns of utilitarian and hedonic food products. The study
have a more instrumental value [1]. Since the exposure to food

978-1-6654-8727-6/22/$31.00 ©2022 IEEE

Authorized licensed use limited to: b-on: Universidade Catolica Portuguesa. Downloaded on December 28,2023 at 09:13:39 UTC from IEEE Xplore. Restrictions apply.
also examines the self-perceived hedonic liking of every TABLE I. PRODUCT AND MEDIA CHARACTERISTICS
product and explores the differences among the campaigns. Stimulus Characteristics
Experiment/
The research objectives (ROs) are the following: Product
Campaign Content and Tech
Media type Duration
RO1: To facially detect consumer emotions during name characteristics
watching social media food campaigns of different types of Category #1: Healthy Yohurt (video ads, utilitarian)
branded: yes,
products (hedonic, utilitarian) and media (video, image posts). Experiment 1/ ‘Yogurt soundtrack: yes, Video 15
RO2: To examine the differences in hedonic liking and Milk yogurt Milk’ human faces: yes (YouTube) seconds
emotional responses among different social media food
campaigns. branded: yes,
The experimental results are expected to shed light on the Experiment 2/ ‘Yogurt soundtrack: yes, Video 17
effectiveness and usefulness of applying face tracking to Almond yogurt Almond’ human faces: yes (YouTube) seconds
detect consumer emotions in social media marketing research.
The results provide the research community with richer Category #2: Nutritious Nutbutters (image posts, utilitarian)
insights on consumer emotions during watching food related Video-
Slideshow
video and image posts campaigns for hedonic and utilitarian Experiment 3/
branded: yes, (Facebook
products. Nut butter ‘Nutbutters 15
soundtrack: no, and
package and Only’ seconds
human faces: no Instagram
jars only
image
II. RESEARCH DESIGN posts)
Video-
As part of a larger study, participants completed one Slideshow
separate test for each product/stimulus, providing a final Experiment 4/
‘Nutbutters branded: yes, (Facebook
Nut butters 16
dataset of 164 face tracking records. All data were collected Hedonic soundtrack: no, and
combined with seconds
at the University of Macedonia (Greece). The experimental Use’ human faces: yes Instagram
hedonic food
image
protocol was approved by the ethical committee of the posts)
University of Macedonia. Category #3: Hedonic Ice cream (video ad, hedonic)
‘Hedonic Ad branded: yes,
Experiment 5/ Video 30
- Ice Cream’ soundtrack: yes,
A. Stimuli and Experimental Design Ice cream (YouTube) seconds
human faces: yes
Five commercially available products of branded milk
yogurt, almond yogurt, nut butters (hazelnuts, almond,
sesame, etc.), and ice cream, were used as stimuli. This B. Participants
research focused on healthy and nutritious utilitarian Seventy-six participants successfully completed at least one
products since they are considered more difficult to provoke of the five experimental tasks within a period of 60 days. There
strong food elicited emotions, compared to unhealthy and was an approximately equal gender distribution in the sample
hedonic products [1] [2]. (51% female, 49% male). The majority (64.5%) was in the age
A five-task experimental survey was designed in the group of 25-34, 18.4% were in the age group of 35-50, 14.5%
FaceReader Online platform, to individually track the facial were between 18 and 24 years old, while 2.6% were above 51
emotional expressions for each product stimulus. The first years old. Only 5 participants reported food allergies, two of
experiment included a YouTube video ad of a regular Greek whom concerned dairy products; hence these participants were
yogurt, while the second one included a YouTube video ad excluded from the yogurt experimental data. None of the
of an almond yogurt and related almond milk products of the participants was vegan, and 14.5% of the participants reported
same brand. The third experiment included a set of Facebook that they do not consume sugar.
image posts on a branded nut butter product, presenting only
the product and its package. The fourth experiment presented
the same nut butter highlighting their use in a hedonic C. Procedure and Measured Attributes
approach, e.g., through combinations with hedonic food (like An online invitation page including detailed instructions was
muffins and pancakes). The fifth experiment presented a distributed to all participants through email, describing the
video ad on a branded ice cream (hedonic product). Due to experimental steps and providing the corresponding FaceReader
the FaceReader restriction to upload stimuli in image Online and self-reported questionnaire links. Participants
formats, the tasks in experiments 3 and 4 were conducted completed each experiment separately on their desktop or laptop
through a video slideshow presenting the selected social devices. Due to technical limitations on mobile devices
media image posts. (especially on iOS smartphones), the participants were asked not
The characteristics of the selected products and media to use mobile devices for the experiment.
stimuli are presented in Table I. After watching a video ad, the participants filled in a
questionnaire about the hedonic liking of the promoted product.
A separate questionnaire link was provided in the instructions
page for every experimental task. After completing one
experimental task they could manually move to the next one.

Authorized licensed use limited to: b-on: Universidade Catolica Portuguesa. Downloaded on December 28,2023 at 09:13:39 UTC from IEEE Xplore. Restrictions apply.
All participants provided their formal consent to be facially TABLE II. DESCRIPTIVE STATISTICS FOR FXPERIMENT 1
tracked by FaceReader Online for emotion detection Std.
Emotion Min. Max. Mean [0,1]
purposes. Instructions were also provided on the position of Deviation
their head, the camera state, the room lighting, and the Happy .000 .409 .0467 .0921
potential removal of glasses and make up. Neutral .153 .989 .7060 .2170
The FaceReader Online™ software detects the six basic Sad .004 .929 .1202 .1739
emotions of Ekman [10] (happy, sad, angry, surprised, Angry .001 .748 .0789 .1290
Surprised .000 .131 .0153 .0219
scared, disgusted), a neutral state (neutral), as well as valence
Scared .000 .136 .0146 .0243
and arousal. The self-reported survey included a
Disgusted .000 .161 .0219 .0330
questionnaire item to measure the hedonic liking of the
Valence -.928 .357 -.1349 .2252
promoted product. The questionnaire was based on a five-
Arousal .189 .585 .3201 .0783
point hedonic scale: “I like the product [1= Extremely
Hedonic liking** 3.00 5.00 3.960 .767
Disagree; …; 5=Extremely Agree], according to relevant
literature in food science [11]. *Note: The mean value ranges between 0 and 1 for the FaceReader extracted
emotions, and between 1 and five for the self-reported hedonic liking item.
**Note. The value is self-reported
D. Data Processing and Analysis
Recording and analysis was completed successfully in
TABLE III. DESCRIPTIVE STATISTICS FOR FXPERIMENT 2
203 sessions. All analyzed results were exported by
FaceReader Online in csv format to be cleaned and further Emotion Min. Max. Mean [0,1]
Std.
processed. After removing all records of low recording Deviation
Happy .000 .469 .0391 .0959
quality (less than 4.0/10.0), a total of 164 valid records
Neutral .283 .985 .7347 .2035
composed the final dataset (Experiment 1: 45 sessions,
Sad .001 .593 .0989 .1234
Experiment 2: 37 sessions, Experiment 3: 41 sessions,
Angry .000 .425 .0553 .0994
Experiment 4: 41 sessions, Experiment 5: 13 sessions). The Surprised .000 .290 .0309 .0569
final records achieved an average recording quality score of Scared .000 .078 .0144 .0180
9.0. Recording quality in FaceReader is ranging from 1=very Disgusted .000 .404 .0286 .0684
poor to 10=ideal quality. The final data were imported in Valence -.589 .287 -.1054 .1791
SPSS to calculate the descriptive statistics and conduct the Arousal .183 .547 .3163 .0745
comparative analyses. Hedonic liking** 1.00 5.00 3.320 .973
The platform of FaceReader Online generated a list of
visualized graphs and reports, filtering out the records of *Note: The mean value ranges between 0 and 1 for the FaceReader extracted
quality lower than 4.0/10.0. The reports included time charts, emotions, and between 1 and five for the self-reported hedonic liking item.
**Note. The value is self-reported
graphs, and pies for every task, as well as comparative results
in tables and graphs between the experiments based on T-
Test analysis. TABLE IV. DESCRIPTIVE STATISTICS FOR FXPERIMENT 3
Std.
Emotion Min. Max. Mean [0,1]
Deviation
III. RESULTS
Happy .000 .457 .0392 .0923
Neutral .296 .981 .6946 .1981
Sad .001 .344 .0880 .0960
A. Descriptive Statistics
Angry .001 .572 .0917 .1367
Tables II-VI depict the descriptive statistics’ results that Surprised .000 .276 .0335 .0591
were calculated for each experiment separately, based on the Scared .000 .230 .0230 .0401
exported and processed datasets. The self-reported hedonic Disgusted .000 .330 .0401 .0728
liking value is integrated in every table to depict the hedonic Valence -.587 .403 -.1361 .1811
liking score for the promoted product in every experiment. Arousal .164 .481 .3082 .0652
As expected, the hedonic ad of ice cream received the Hedonic liking** 1.00 5.00 2.930 1.058
highest values of hedonic liking, while the lowest scores were
indicated in the nut butter product in the campaign *Note: The mean value ranges between 0 and 1 for the FaceReader extracted
‘Nutbutters only’. Interestingly, the same product in a emotions, and between 1 and five for the self-reported hedonic liking item.
different marketing campaign where its hedonic use was **Note. The value is self-reported
presented, reported higher hedonic liking rates among the
participants.

University of Macedonia, Greece.

Authorized licensed use limited to: b-on: Universidade Catolica Portuguesa. Downloaded on December 28,2023 at 09:13:39 UTC from IEEE Xplore. Restrictions apply.
TABLE V. DESCRIPTIVE STATISTICS FOR FXPERIMENT 4 average expressions of all participants. The time series data
Std. in the charts highlight the average levels (between 0 and 1 in
Emotion Min. Max. Mean [0,1]
Deviation axis y) of every measured emotion throughout the duration of
Happy .000 .207 .0260 .0482 every video campaign, separated in timeframes of 2 and 5
Neutral .313 .984 .7040 .1936 seconds (in axis x). The decreasing or increasing linear
Sad .001 .520 .1052 .1189 pattern in the prevailing emotion ‘sad’ is also depicted for
Angry .000 .314 .0680 .0849 each depicted campaign. The emotion ‘neutral’ was excluded
Surprised .000 .149 .0320 .0385
from the reports since its levels were significantly higher than
Scared .000 .123 .0188 .0230
all measured emotions (around 0.7/1.0), revealing low levels
Disgusted .000 .409 .0355 .0756
of emotional responses.
Valence -.517 .084 -.1370 .1350
Overall, the participants expressed similar facial emotional
Arousal .196 .542 .3143 .0677
states in all five experiments, where the emotion of sadness
Hedonic liking** 1.00 5.00 3.290 1.146
(followed by anger) was prevailing over the other emotions
*Note: The mean value ranges between 0 and 1 for the FaceReader extracted during the sessions. The expression of happiness remained at
emotions, and between 1 and five for the self-reported hedonic liking item. lower levels throughout the time frames. The trend lines of
**Note. The value is self-reported the prevailing emotion of sadness indicated a mild decrease
or increase during the campaign duration of the healthy and
TABLE VI. DESCRIPTIVE STATISTICS FOR FXPERIMENT 5 nutritious products, whereas in the case of hedonic product a
quite intensive decrease was observed in the first and last
Std. seconds where the emotion of surprise significantly
Emotion Min. Max. Mean [0,1]
Deviation
Happy .380 .962 .7974 .1719 approached sadness. No significant differences were
Neutral .004 .111 .0491 .0391 observed in the sad area sizes among the non-hedonic
Sad .001 .238 .0545 .0672 products. The results imply that although negative facially
Angry .001 .164 .0343 .0485 expressed emotions tend to change throughout the time, they
Surprised .000 .059 .0095 .0157 still prevail over the positive ones in all types of products.
Scared .000 .065 .0161 .0200 As regards the hedonic liking of the promoted products,
Disgusted -.225 -.009 -.0798 .0592 the participants rated the hedonic liking of the yogurt and nut
Valence .202 .357 .2739 .0457 butter products of medium levels (mean= 3.35), while the ice-
Arousal .380 .962 .7974 .1719 cream hedonic liking score was significantly higher (mean=
Hedonic liking** 2.00 5.00 4.160 0.90 4.16).
The reports on the circumplex model indicated
*Note: The mean value ranges between 0 and 1 for the FaceReader extracted approximately equal levels of arousal and valence in all the
emotions, and between 1 and five for the self-reported hedonic liking item. five experiments. Participants expressed a quite unpleasant
**Note. The value is self-reported
(valence < 0) and inactive (arousal < 0.5) emotional state on
their facial expressions during watching the campaigns. This
B. Temporal Average Expressions and Circumplex Model result can be explained by the participants’ directed attention
Figures 1 -3 depict three examples (one from each product on the provided stimuli, as explained in [11].
category) of the reported time charts about the temporal

0.2
Emotion al score ranges

0.15
0.1
[0,1]

0.05
0
00:00.0 00:02.0 00:04.0 00:06.0 00:08.0 00:10.0 00:12.0 00:14.0 00:15.0
Time frames 2 sec

Happy Sad Angry Surprised Scared Disgusted Linear (Sad)

Figure 1 Time chart Experiment 1 ‘Yogurt Milk’

Authorized licensed use limited to: b-on: Universidade Catolica Portuguesa. Downloaded on December 28,2023 at 09:13:39 UTC from IEEE Xplore. Restrictions apply.
Emotion al score ranges 0.15

0.1

0.05
[0,1]

0
00:00.0 00:02.0 00:04.0 00:06.0 00:08.0 00:10.0 00:12.0 00:14.0 00:15.9
Time frames 2 sec

Happy Sad Angry Surprised Scared Disgusted Linear (Sad)

Figure 2 Time chart Experiment 5 ‘Nutbutters Hedonic Use’

0.1
Emotion al score ranges

0.08
0.06
0.04
[0,1]

0.02
0
00:00.0 00:05.0 00:10.0 00:15.0 00:19.9
Time frames 5 sec

Happy Sad Angry Surprised Scared Disgusted Linear (Sad)

Figure 3 Time chart Experiment 5 ‘'Hedonic Ad Ice Cream

C. Comparative Analysis The results also extended previous research on face tracking
A comparative T-Test analysis with unequal sample size indicating that FaceReader Online can capture differences in
and unequal variance was conducted between pairs of stimuli. emotional responses elicited by different types of food in
Five significant emotional differences (p<0.05) were reported digital-social media marketing campaigns.
in the results, where four of them concerned differences A comparative analysis was also conducted on the hedonic
between the healthy/nutritious products and ice cream liking scores of the non-hedonic products. The one-way
(hedonic product). The detected differences emerged in ANOVA results indicated significant differences (p<0.05) in
emotions of surprise, sadness, anger, and valence. the hedonic liking among the four products. The participants
The following insights emerged in terms of emotions: rated lowest the product in Experiment 4 ‘Nutbutterss Only’,
• In Experiment 1 ‘Yogurt Milk’, the selected participants while the highest hedonic liking score was reported in
responded more surprised than in Experiment 4 Experiment 1 ‘YogurtMilk’.
‘Nutbutters Hedonic Use’. A bi-pair T-Test analysis with unequal sample size and
• In Experiment 5 ‘Hedonic Ad - Ice Cream’ the selected unequal variance also revealed significant differences between
participants responded less sad than Experiment 1 the two yogurt products, where the milk yogurt received
‘Yogurt Milk’, less sad than Experiment 4 ‘Nutbutters significantly higher scores than the almond yogurt (p<0.05).
Hedonic Use’, and less angry than Experiment 3 No significant differences were detected between the two
‘Nutbutters Only’. products of the nut butters campaigns. This result possibly is
The following insight emerged in terms of valence: explained by the fact that both campaigns promoted the same
product in a different marketing approach, and the participants
• In Experiment 5 ‘Hedonic Ad - Ice Cream’ the selected
were asked to rate the product itself. The hedonic ice cream
participants responded more positively than Experiment 4
product was excluded from the hedonic liking statistical
‘Nuts Hedonic Use’.
comparison since its value was significantly higher (p<0.05)
The above results extended previous research on chocolate
than all other products. These results suggest that the
versus bakery products, where chocolate products elicited
individual preferences on a product possibly prevail over the
stronger emotional emotions, in bodily sensing studies [1].
campaign design characteristics.

Authorized licensed use limited to: b-on: Universidade Catolica Portuguesa. Downloaded on December 28,2023 at 09:13:39 UTC from IEEE Xplore. Restrictions apply.
IV. CONCLUSIONS [5] W. He, S. Boesveldt, C. de Graaf, and R. A. de Wijk, “The relation
between continuous and discrete emotional responses to food odors with
This study describes a remote Face Tracking application facial expressions and non-verbal reports,” Food Qual. Prefer., vol. 48,
using real-time facial expression recognition in five pp. 130–137, 2016.
campaigns/stimuli, combined with a self-reported survey to [6] K. A. Leitch, S. E. Duncan, S. O’Keefe, R. Rudd, and D. L. Gallagher,
“Characterizing consumer emotional response to sweeteners using an
measure the product hedonic liking. emotion terminology questionnaire and facial expression analysis,”
The findings extend previous research in food-sensory Food Res. Int., vol. 76, no. P2, pp. 283–292, 2015.
experiences by suggesting that facial expressions of negative [7] R. van Bommel, M. Stieger, M. Visalli, R. de Wijk, and G. Jager, “Does
emotions (sad, angry, or surprised) can reveal significant the face show what the mind tells? A comparison between dynamic
differences in social media food campaigns. emotions obtained from facial expressions and Temporal Dominance of
Emotions (TDE),” Food Qual. Prefer., vol. 85, no. December 2019, p.
The FaceReader Online tool detected significantly lower 103976, 2020.
levels of negative emotional states in the hedonic product (ice [8] B. Mahieu, M. Visalli, P. Schlich, and A. Thomas, “Eating chocolate,
cream), compared to the rest campaigns of healthy and smelling perfume or watching video advertisement: Does it make any
nutritious products (yogurts and nut butters). This finding difference on emotional states measured at home using facial
comes in agreement with previous studies in sensory research expressions?,” Food Qual. Prefer., vol. 77, no. April, pp. 102–108,
2019.
where chocolate products elicited more intensive emotional
[9] J. L. Plass et al., “Emotional design for digital games for learning: The
responses compared to non-hedonic food products [1]. effect of expression, color, shape, and dimensionality on the affective
Significant differences were detected in surprise between quality of game characters,” Learn. Instr., vol. 70, no. February 2019, p.
the image post campaign of nut and the video campaign of 101194, 2020.
milk yogurt, implying that FaceReader Online can detect [10] P. Ekman, “Are There Basic Emotions?,” Psychological Review, vol.
99, no. 3. pp. 550–553, 1992.
differences in the emotional responses between different types
[11] S. Wichchukit and M. O’Mahony, “The 9-point hedonic scale and
of products and stimuli. hedonic ranking in food science: Some reappraisals and alternatives,” J.
The statistical analysis revealed significant emotional Sci. Food Agric., vol. 95, no. 11, pp. 2167–2178, 2015, doi:
differences among the healthy products’ campaigns’ hedonic 10.1002/jsfa.6993.
liking rates where the video ad of the branded milk yogurt was [12] S. Wiens and E. Syrjänen, “Directed attention reduces processing of
more positively indicated than the video ad of the branded emotional distracters irrespective of valence and arousal level,” Biol.
Psychol., vol. 94, no. 1, pp. 44–54, 2013, doi:
almond yogurt. Both video ads shared similar characteristics 10.1016/j.biopsycho.2013.05.001.
like short duration, the use of soundtrack and the use of human
faces consuming the product.
Some limitations of the study regard the analysis
methodology as regards the comparison of the hedonic liking
rates and the consumers’ emotions, since other factors have
possibly affected their ratings apart from the campaign
characteristics (e.g., their previous experience with the product
and the expected sensory appeal [4]).
Also, the findings rely on the FaceReader software and
provide mainly descriptive analysis. Finally, the experimental
methodology that instructed participants to direct their
attention on the given stimuli is possibly the reason for the
decreased emotional effects in all the examined campaigns
[11], hence more research (e.g., in the wild) is suggested. This
research, however, faces the challenge that any food related
social media campaign can reveal some emotional responses
at a relatively narrow level of arousal and valence.

REFERENCES
[1] T. Otterbring, M. Folwarczny, and K. Gidlöf, “Hunger effects on option
quality for hedonic and utilitarian food products,” Food Qual. Prefer.,
vol. 103, no. April 2022, 2023.
[2] G. Juodeikiene et al., “Effects of emotional responses to certain foods
on the prediction of consumer acceptance,” Food Res. Int., vol. 112, no.
May, pp. 361–368, 2018, doi: 10.1016/j.foodres.2018.06.064.S.
[3] L. Danner, S. Haindl, M. Joechl, and K. Duerrschmid, “Facial
expressions and autonomous nervous system responses elicited by
tasting different juices,” Food Res. Int., vol. 64, no. November 2018, pp.
81–90, 2014.
[4] D. Garcia-Burgos and M. C. Zamora, “Facial affective reactions to bitter-
tasting foods and body mass index in adults,” Appetite, vol. 71, pp. 178–
186, 2013.

Authorized licensed use limited to: b-on: Universidade Catolica Portuguesa. Downloaded on December 28,2023 at 09:13:39 UTC from IEEE Xplore. Restrictions apply.

You might also like