Introduction Introduction Introduction

Introduction
XI Peruvian Symposium on Artificial Introduction
Intelligence (SPIA)

Emotion recognition using Texture Maps and

Convolutional Neural Networks
Lourdes Ramírez Cerna*
Edwin J. Escobedo Cárdenas+
Guillermo Cámara Chávez+
*Department of Informatics
National University of Trujillo
Peru
+Department of Computer Science
Federal University of Ouro Preto
Brazil
Motivation Introduction Proposed Method Experiments Conclusion

Motivation

Why Facial Expressions?

• Facial expressions are one of the more important aspects of human

communication.

• The face is responsible for communicating not only thoughts or

ideas, but also emotions.

• Automatic facial expression recognition (FER) has recently

attracted increasing attention due to its wide range of applications.

2
Motivation Introduction Proposed Method Experiments Conclusion

Motivation

Why Facial Expressions?

3
Motivation Introduction Proposed Method Experiments Conclusion

Motivation

Why Facial Expressions?

4
Motivation Introduction Proposed Method Experiments Conclusion

Motivation

Sign language recognition

b) work without desire c) work happily

a) Work

5
Motivation Introduction Proposed Method Experiments Conclusion

Introduction

• Authors usually do not consider the face behavior and head motion of
videos.

begin end

Neutral The last N frames are considered as

samples for a class.

Fig. Image sequence of happy emotion.

6
Motivation Introduction Proposed Method Experiments Conclusion

Introduction

• The Facial Expressions are a secondary parameter for Sign Languages.

• The head motion is important to interpret a sign.

Fig. Sample of the affirmative sign in BSL.

Motivation Introduction Proposed Method Experiments Conclusion

Proposed Method

• This study considers the importance of face behavior and head motion to
recognize facial expressions.
• We propose to generate texture maps [1] to encode the face variations and
motion.
• We consider the Landmark [2] points to generate our texture maps.

Fig. Facial expression sequence begins with a neutral expression and proceeds to
target expression. For each frame, we compute its respective landmark points . 8
Motivation Introduction Proposed Method Experiments Conclusion

Proposed Method

Fig. Pipeline of our proposed model

9
Motivation Introduction Proposed Method Experiments Conclusion

Proposed Method

Grouping Points

10
Motivation Introduction Proposed Method Experiments Conclusion

Proposed Method

Texture Maps Generation (Ding et al. [1])

• We compute three different distances using the face landmarks
• Each distance is allocate into a different color channel.

11
Motivation Introduction Proposed Method Experiments Conclusion

Proposed Method

Texture Maps Generation R1 R2 R3

12
Motivation Introduction Proposed Method Experiments Conclusion

Proposed Method

Classification
• We use transfer-learning.
• We use the pre-trained ImageNet-vgg-f model.
• The images are resized to 256x256x3.

13
Motivation Introduction Proposed Method Experiments Conclusion

Experiments

The Extended Cohn-Kanade dataset (CK+) (Lucey et al. [3])

Anger Contempt Disgust Fear

Surprise Happiness Neutral Sadness

Fig. The CK+ dataset consists of 593 images sequences, 123 subjects
performed 8 facial expressions categories.

14
Motivation Introduction Proposed Method Experiments Conclusion

Experiments

Comparison with other methods

15
Motivation Introduction Proposed Method Experiments Conclusion

Conclusion
• It is necessary to combine facial changes with face motion and local information to
produce a robust method.

• In this work, we prove that using texture maps is a feasible way to encode the
temporal information.

• As future work, we will extend our CNN models to improve the achieving results
and test our method on a continuous dataset.

16
Motivation Introduction Proposed Method Experiments Conclusion

References
[1] Zewei Ding, Pichao Wang, Philip O Ogunbona, and Wanqing Li. Investigation of different
skeleton89features for cnn-based 3d action recognition. In2017 IEEE International Conference on
Multimedia90& Expo Workshops (ICMEW), pages 617–622. IEEE, 2017b.

[2] Yuval Nirkin, Iacopo Masi, Anh Tran Tuan, Tal Hassner, and Gerard Medioni. On face
segmentation,92face swapping, and face perception. In2018 13th IEEE International Conference on
Automatic93Face & Gesture Recognition (FG 2018), pages 98–105. IEEE, 2018.

[3] Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews.
The69extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified
ex-70pression. In2010 IEEE Computer Society Conference on Computer Vision and Pattern
Recognition-71Workshops, pages 94–101. IEEE, 2010.

[4] Hui Ding, Shaohua Kevin Zhou, and Rama Chellappa. Facenet2expnet: Regularizing a deep
face83recognition net for expression recognition. In2017 12th IEEE International Conference
on84Automatic Face & Gesture Recognition (FG 2017), pages 118–126. IEEE, 2017a.

[5] Nianyin Zeng, Hong Zhang, Baoye Song, Weibo Liu, Yurong Li, and Abdullah M Dobaie.
Facial86expression recognition via learning deep sparse autoencoders. Neurocomputing, 273:643–
649,872018.

[6] Mengyi Liu, Shaoxin Li, Shiguang Shan, and Xilin Chen. Au-aware deep networks for
facial97expression recognition. In2013 10th IEEE International Conference and Workshops on
Automatic98Face and Gesture Recognition (FG), pages 1–6. IEEE, 2013. 17
.