Facial Emotions Analysis Through Video and Artificial

Intelligence
John Paul F. Avila1, Charlene Dianne D. Nicomedes2, Abby Kate C. Orioque3

Abstract—In this work, the user's emotion using its facial expressions will be detected. These expressions can be derived from
the live feed via the system's camera or any pre-existing image available in the memory. Emotions possessed by humans can be
recognized and has a vast scope of study in the computer vision industry upon which several researchers have already been done.
The work has been implemented using Python (3.10), Open Source Computer Vision Library (OpenCV) and NumPy which run
on a low-powered single board microprocessor, the Lattepanda. The floating point numbers (testing dataset) is being compared to
training dataset and thus emotion is predicted. The objective of this paper is to develop a system which can analyze the video and
predict the expression of the person. The study proves that this procedure is workable and produces valid results.

Keywords: Python, OpenCV, Numpy, Lattepanda

Gradient Face, have a High discrimination power
1. Introduction illumination variation, they are nonetheless praised for their
In real-time applications nowadays, methods ability to capture facial expressions and age variation
based on human-computer interaction are utilized to situations.
quickly and precisely identify human activity in movies. Since the beginning of the last decade, a variety
Realizing and monitoring human face expressions and of techniques for recognizing human facial expressions and
emotions from streaming video with the aim of serving emotions using video or images have been developed. The
various purposes, such as physical weariness detection, is approaches and facial expression datasets employed by
one such topic. We initially describe the requirements of these methods vary. The two performance measures that
face emotion and expression identification in the following characterize the effectiveness and quality of a suggested
words before going into further detail. technique are detection accuracy and processing time,
In human-to-human communication, the sound of which are crucial for every method. Our aim in this
a person's mental, emotional, and even physical state is research is to investigate such modern video-based facial
used in conversations about significant information in emotion detection systems, together with their benefits and
addition to pronouncing a communication channel. Facial drawbacks.
expressions are the idea of a person's facial expressions in
their simplest form, which is a more subtle happy or angry 2. Background of the Study & Related
thoughts, feelings, or absorption of all speaker expectation Literature
from listeners, sympathy, or even what the speaker
detection technology that is based on the optical flow
vector speed infusion technology.The algorithm uses
segmented image frames and we provide a vector based on
their results that detects facial emotions to the strongest
degree of equality. Optical flow speed over time intervals
indicates image alterations. The algorithm of work (AU)
based on database operation combine coded facial
expression. This technique can be used to recognize which
facial expressions belong to the four different categories.
To identify the first type, use facial expression
that expresses emotion quickly. The third form of facial
expression is used to recognize the active shape model in
the second type of optical flow, which uses facial
expression to identify an image frame. The fourth type of
neural networks recognize faces in a complex,
multidimensional view by analyzing facial expression. It
takes a lot of effort to build a model for facial recognition.
Face detection is possible with a variety of distinct
condition databases (expression, lights, etc.) that each have
a unique face. There are various approaches to recognize
facial expressions, including non-monotonic illumination
variation, random noise, and variations in age, while
making it appear as though each was affected by the
method's drawbacks. Even while some techniques, like
 The desired reconfigurable hardware (FPGA)
structure must be taken into account while designing
algorithms for real-time adaptive face recognition. Because
earlier systems were not designed to adapt algorithms to the
target hardware resources, they did not provide results that
were acceptable. Since conventional facial recognition
systems have limitations including producing power
dissipation, slow response times, and a lack of adaptability,
a decade ago they were unable to address real-world
difficulties
Figure 1: Overview of a general face recognition
system (a) Training Phase (b) After the completion of
training.
Figure 2: Emotion Recognition System Using
Video Sequence
For the purpose of recognizing face expressions
in video sequences, the author presented spatiotemporal
feature extraction. With little computational effort, the
suggested spatiotemporal texture map (STTM) can capture
minute spatial and temporal fluctuations in facial emotions.
The Viola-Jones face detector is used to first identify the
face, and then the frames are cropped to eliminate
extraneous background. The suggested STTM, which
separates the spatiotemporal information from the three-
dimensional Harris corner function, is then used to
represent facial features. The dynamic features are
extracted using a block-based approach, and the
characteristics are displayed as histograms. The support
vector machine classifier then divides the characteristics
into categories of emotion and expressiveness. The
experimental results show that the proposed approach
outperforms state-of-the-art approaches, with average
recognition rates of 95.37, 98.56, and 84.52 percent,
respectively, on datasets containing posed expressions,
unforced microexpressions, and expressions that are similar
to those found in real-world situations.
Figure 3: shows the difference of two kinds of data set.
For the purpose of developing emotion
recognition models and assessing performances, several
data sets have been created. On other data sets, the same
approaches can produce radically different outcomes. The
variation in data sets is to blame for this. Lab data sets, like
the CohnKanade CK+ database, and wild datasets, such
Acted Facial Expressions in the Wild (AFEW), are two
forms of facial expression databases. When compared to
most lab datasets, the latter chooses facial expression
photos or videos from camera recordings that have
carefully controlled lighting, head attitude, and other
variables.
3. Methodology
a. Process
As stated, the research suggests an
algorithm to use the people's images in real-time films shot
to gather data about the individuals' immediate
psychological states. In order to do this, human faces in the
movies must first be identified before the data can be pre-
processed. The techniques used on facial photographs allow
for the detection of facial landmarks. These data points are
used to build a support vector machine algorithm, which
then estimates which emotion class (happy, sad, angry,
surprised, and normal) the emotion belongs to.
The
result
of all
experiments performed in this study was obtained by using
computer with the following properties and software
installed:
● Processor: Intel(R) Atom Cherrytrail + ATMega
MCU
● RAM 2GB.
● GPU 1.8GHz.
● 64-bit Operating System, X64 based processor.
Figure 3: System Hardware - Lattepanda
● Python (2020.2.3) is an object oriented,
interpretative, modular and interactive high level
programming language. Its modular structure
supports the system and any data field entry.
With Python, which can work on almost any
platform, software can be developed in many
areas such as system programming, user interface
programming, network programming, application
and database software programming.
● OpenCV (Open Computer Vision) is an open
source library created by Intel, composed in C
dialect that can run on Windows, Linux, MacOS
X, PSP (Playstation Versatile) stages.
● Numpy is a Python library used for working with
arrays. It also has functions for working in the
domain of linear algebra, fourier transform, and
matrices. NumPy was created in 2005 by Travis
Oliphant. It is an open source project and you can
use it freely. NumPy stands for Numerical
Python.
● TensorFlow is an end-to-end open source
platform for machine learning. It has a
comprehensive, flexible ecosystem of tools,
libraries and community resources that lets
researchers push the state-of-the-art in ML and
developers easily build and deploy ML powered
applications.
b. Data Set
Figure 4: Landmarks on face
For face detection and recognition, facial
landmarks are extremely important. In the case of phrases,
the same landmarks might be utilized. A 68 facial landmark
detector in the Dlib library provides the locations of 68
landmarks on the face.
All 68 landmarks are depicted on the face in
Figure 4. We may obtain the coordinates(x,y) of each face
point using the dlib package. These 68 points can be further
broken down into individual regions, such as the left and
right eyes, left and right brows, mouth, nose, and jaw.
4. Results and Discussion
At the end of this evaluation, it was seen that
different success rates were obtained on 5 emotional states
(happy, sad, surprised, neutral, angry, fear).
 We can see that there are not equal numbers of
courses, which could lead to certain classes being
misclassified. Additionally, the class can be viewed as an
outlier because there are fewer training examples available.
The algorithm may develop a bias toward the feeling of
surprise and begin to identify the majority of photos as
surprises.

5. APPLICATIONS AND FUTURE SCOPE

The field of computer vision is extremely broad and is still

Figure 5: Face detection through a video of user
The initial and most crucial step in the processing
pipeline was face detection. Even though our video only
contained frontal facial expression data, we still had to
identify the face before moving on to the next step of
processing. It was simpler to identify the region of interest
and extract features from the face after it had been
identified.
in its infancy. The pace of research in this area is
accelerating. Computer vision cannot exist without the
ability to recognize emotions. If one can become aware of
the complexities and limitless opportunities presented
under the field of emotion detection, a ton of jobs and
procedures can be carried out.
Some common and widely used applications of emotion
detection are:
a. App and product development

The optimization of different software

Table 1: Prediction and Accuracy of Emotions
Table 1 shows the correct prediction and accuracy values
in report format. From this, we can infer the correct number
of emotions accuracy for each class.
engineering processes, including testing the usability of a
product, can greatly benefit from emotion recognition. It
has long been known that how comfortable a person is
using various software products greatly depends on their
emotions. Human emotions can be changed by a product's
overall appearance and feel, which in turn affects whether
or not a person decides to purchase it. For those working in
the associated industries, learning about the various
emotional states of the human body and how they are
affected by the use of various products is therefore of
utmost importance.
b. Improved learning practices

1: Happy: 44/50 88% Evidence reveals that while certain emotional

2: Sad: 41/50 82% states aim to repress them, others support superior learning
3: Surprise: 35/50 70% techniques. It is uncommon to detect the difference
4: Neutral: 43/50 86% between the two groups' various emotional states. For
5: Angry: 22/50 44% instance, it is believed that pleasant emotions are harmful
6: Fear: 17/50 34% for learning while slightly negative emotions encourage
analytical thinking and are suitable for performing
important activities.

c. Improvised web development

Service providers are interested in gathering a ton

of data that may be retrieved from users due huge the
massive size at which the internet is increasing. All of the
information and adverts are accordingly played based on
the user's profile. Therefore, incorporating extensive
information about the various human emotions can offer far
more accurate behavioral models of various user types.
Figure 6: Accuracy Charrt of Facial Emotions Analysis

d. Immersive gaming

A sizable portion of the entertainment market is
made up of video games. Therefore, video game designers
center their research on many forms of common human
emotions in order to make these games much more intense
and intrusive. Video games are created in such a way that
they organically blend human emotions into their game
play in order to draw in an increasing number of players.
6. Conclusion and Future Work
International Journal of Pattern Recognition and Artificial
Intelligence.
[6] Cao, T. & Li, M. (2019). Facial expression recognition
algorithm based on the combination of CNN and K-Means.
11th International Conference on Machine Learning and
Computing.
[7] Oztel, I. (2018). Facial expression detection with
machine learning methods on partial and full face images.
PhD Thesis, Sakarya University.

[8] Tenekeci, M. E. , Gumuscu, A., Baytak, A., Aslan, E.

This paper's objective is to give a comparative (2014). Emotion analysis from image with OpenCV.
analysis of various methods for video-based facial and Academic Informatics'14 - XVI. Academic Informatics
emotional identification. In the past ten years, automatic Conference Proceedings.
facial expression and emotion identification has become
increasingly important in human-computer interaction [9] Ozmen, G. (2012). Facial expression recognition with
(HCI) systems, biometrics, security, and other areas of cubic bezier curves. Master's Thesis, Trakya University.
computer science. There has been a lot of thorough and
beneficial research done in this field in recent years. The [10] Martinez, J. C.
approaches from 2015 and 2016 under study, together with "https://livecodestream.dev/post/detectingface-features-
their benefits and drawbacks, were accuracy tested. with-python/ 21.02.2021".

Because so many emotions can be expressed on [11] Lyons. M. , Akamatsu, S. , Kamachi, M. & Gyoba, J.
the face at once and in remarkably similar ways, emotion (1998). Coding facial expressions with Gabor wavelets.
analysis is a challenging field. The current dataset's success Third IEEE International Conference on Automatic Face
rates are higher than the average values analyzed in the and Gesture Recognition, 200-205.
literature. As in every machine learning study, more data is
required to boost these rates. There are, however, a lot of
other limitations. However, it is very impossible to entirely
separate specific emotions because of the emotional
complexity and quick mood swings.
This document defines the current research
methodologies. Future studies should build new methods to
increase recognition's resilience, effectiveness, and
accuracy.

REFERENCE

[1] Romero, M. , Pears N. (2009). Landmark localisation in

3D face data. 6th IEEE International Conference on
Advanced Video and Signal Based Surveillance, AVSS
2009, 73-78.

[2] Valstar, M.F. & Pantic, M. (2012). Fully automatic

recognition of the temporal phases of facial actions. IEEE
Trans. Syst. Man Cybern. Part B Cybern. 42(1), 28-43. [3]
Gudi, A. , Tasli, H.E. , Den Uyl, T. M. & Maroulis A.
(2015). Deep learning based FACS action unit occurence
and intensity estimation.

[4] Khorromi, P.K. , Paine, T.L. , Brady, K. , Dagli, C. &

Huang, T.S. (2016). How deep neural networks can
improve emotion recognition on video data. 1-5.

[5] Nguyen's Yearn, HD., Lee, G-S. , Yang, H-J. , Na, L.

& Kim, H. (2018). Facial emotion recognition using an
ensemble of multilevel convolutional neural networks.