Facial Expression Detection

Anish Panda, Priyansh Sachidanand Singh, and Vikas?

VIT-AP University, anish.21bce8748, priyansh.21bce7306, vikas.21bce@vitapstudent.ac.in

Abstract – Our group designed a simple convolutional can empower the blind to better navigate social interactions,
neural network for the purpose of facial expression leading to more meaningful and enriching conversations.
detection based on the FER2013 Dataset. Keras was used Despite the compelling potential of facial expression
to make the neural network model. The model is first detection, challenges persist in achieving real-time accuracy
trained on the original dataset, and then on the under diverse conditions. Presently, computers' ability to
generated dataset to achieve low variance, the model predict facial expressions in real-world scenarios remains
reached an accuracy of 67.34% after the complete limited, impeding seamless human-computer
training. communication. Additionally, human observers themselves
only possess a moderate level of proficiency in recognizing
Index Terms – CNN, emotion detection, human expression others' expressions, leaving room for improvement in this
identification. field.
The fusion of artificial intelligence, computer vision,
I. INTRODUCTION and deep learning methodologies, particularly Convolutional
The advent of the digital age has ushered in a plethora of Neural Networks (CNNs), is instrumental in addressing
opportunities for technological advancements, and among these challenges. By leveraging vast datasets and
them, the ability of computers to discern and accurately sophisticated neural network architectures, researchers
interpret human emotions and facial expressions stands as a endeavor to train machines to emulate human-like perceptual
promising frontier. Such an ability holds immense potential abilities, bridging the communication gap between humans
for various applications, permeating domains like Virtual and computers and rendering interactions more effective and
Reality (VR) gaming, online education, driving assistance, intuitive.
medical care, and support for visually impaired individuals.
By enabling computers to comprehend human emotions, a II. LITERATURE SURVEY
new realm of interactive experiences can be unlocked,
revolutionizing the way we interact with machines and each All papers must use the following layout:
other.
In the realm of VR gaming, facial expression detection
can significantly enhance user immersion and engagement. TABLE I
Imagine a gaming environment that reacts dynamically to a COMPARING SIMILAR APPROACHES
player's emotional cues, adapting the gameplay to evoke Research Paper Name Method Used Accuracy
Facial Expression Recognition CNN 66.7
tailored responses. Likewise, in online education, facial Using CNN with Keras
expression detection can provide invaluable feedback to (Apeksha Khopkar, et al.)
educators, enabling them to gauge students' attentiveness Facial Emotion Recognition: VGGNet 73.28
and emotional states during remote learning sessions, State of the Art Performance on
FER2013 (Yousif Khaireddin,
leading to personalized teaching approaches for enhanced et al.)
comprehension and retention. Hybrid Facial Expression DCNN and Haar Cascade 70.04
The potential implications in the realm of road safety Recognition (FER2013) Model Deep learning architectures
cannot be understated. Advanced driver-assistance systems for Real-Time Emotion
Classification and Prediction
equipped with facial expression detection cameras can (Ozioma Collins Oguine, et al.) 91.76
monitor drivers in real time to detect signs of fatigue or Real Time Face Expression Cycle GAN
distraction, thereby reducing the risk of accidents and Recognition along with
ensuring safer journeys for all road users. Balanced FER2013 Dataset
using CycleGAN (F. Mazen, et 70.02
In the healthcare sector, facial expression detection can al)
aid in assessing patients' emotional well-being and pain Deep-Emotion: Facial Attentional convolution
levels. Physicians and caregivers can employ this technology Expression Recognition Using network
to provide better-tailored care and empathetic support, thus Attentional Convolutional
Network (Shervin Minaee, et 65.03
improving the overall patient experience. al.)
Furthermore, the positive impact of facial expression Facial Expression Recognition CNN Ensemble
detection extends to supporting the visually impaired. By with CNN Ensemble (K. Liu, et 75.42
providing insights into the emotions of others, computers al.)
Local Learning with Deep and Local learning, SVM and
Handcrafted Features for Facial CNN followed by the final SoftMax activation layer with the
Expression Recognition 66.4 number of neurons corresponding to the number of classes.
(Georgescu, M, et al.)
Going Deeper in Facial Deep neural network
Expression Recognition using
Deep Neural Networks (Ali -
Mollahosseini, et al.)
Learning Vision Transformer Learning Vision
with Squeeze and Excitation for Transformer
Facial Expression Recognition 67.34
(Mouath Aouayeb, et al.)
Our Approach CNN with Mixed Training

Set all these values using the “FILE” Menu. Select the
Page Setup – Margins tab option and click on portrait
orientation option. The above margin dimensions can then
be inserted into the Page Setup Window. Now select the
Paper tab and click on the paper size and select the letter
paper size. Next, select the Page Setup – Layout tab and set
the Header and Footer to 0.5 inches. To set the column
width, click anywhere in the document within the two
column format, then Select the Columns for the Format
menu. Click on the Equal Column Width and set the
spacing. Make sure you also apply these changes to “This
Section.”
When formatting your document, make consistent use
of punctuation marks and spelling. Either American or
British formatting is acceptable, but it must be consistent,
not a mix. For example,
 Putting all commas and periods either inside
(American) or outside (British) of quotation marks
 Use of single/double quotes, e.g. 'service
center' (British) rather than “service center” (American).
 Spellings such as grey and disc (British) vs.
gray and disk (American).
III. PROPOSED WORK
This study aims to develop an effective CNN-based model
for facial expression detection using the FER-2013 dataset,
comprising 30,000 training images and 3500 testing images,
with each image dimension being 48x48 pixels. The dataset
encompasses seven distinct emotions: 'Happy,' 'Sad,' 'Fear,'
'Disgust,' 'Anger,' 'Surprise,' and 'Neutral.' Preprocessing
steps have been performed on the dataset, including
normalization and resizing, to ensure optimal model training.
Subsequently, a two-step training approach was adopted,
which involved training the model on the original
preprocessed dataset and then on an augmented dataset.
The CNN model architecture was meticulously designed
to capture intricate features within facial expressions. The
model consists of multiple convolutional layers, each
followed by ReLU activation functions to introduce non-
linearity. Pooling layers, including MaxPooling2D, were
incorporated to down-sample the spatial dimensions and
retain important features. Dropout layers were strategically
added to mitigate overfitting by randomly deactivating
neurons during training. Furthermore, dense layers with
dropout regularization were appended to the network,
FIGURE I
OUR CNN MODEL ARCHITECTURE
The training phase involved two key steps. Initially, the
model was trained on the preprocessed dataset.
Subsequently, a second training phase was executed using
data augmentation. Augmentation techniques included
rotation, horizontal and vertical shifts, shear transformations,
and zooming, introducing diversity into the training set. The
augmentation process was facilitated using the
ImageDataGenerator module. For both training phases, the
model was trained with a batch size of 128 for 100 epochs.
Early stopping was implemented with a patience of 20 to
mitigate the risk of overfitting.
For evaluation, the model was subjected to the testing
dataset, and performance metrics were computed. These
metrics included accuracy, precision, recall, and F1-score,
which provide insights into the model's ability to accurately
classify different facial expressions. The model was also
evaluated on the ability to generalize, and the test accuracy
metric was reported.
The software ecosystem utilized for this research
included TensorFlow and its related libraries. The model
was constructed using the Sequential API, while layers such
as Conv2D, Dense, MaxPooling2D, and Dropout were
employed to build the neural architecture. The Adam
optimizer was selected for training, and early stopping was
implemented to halt training if validation loss plateaued.
Regularization in the form of L2 regularization was
incorporated into the dense layers to curb overfitting
tendencies.
 The presented methodology serves as a robust
framework to address the challenges associated with facial
expression detection and lays the groundwork for future
improvements, adaptations, and advancements in this
domain.
IV. RESULT
The developed CNN-based facial expression detection
model underwent comprehensive training and evaluation,
yielding insightful performance metrics. The model's
effectiveness was assessed through a series of experiments,
considering both the original preprocessed dataset and the
augmented dataset.
After training on the original preprocessed dataset, the
model achieved a training accuracy of approximately
63.90%. The validation accuracy reached 64.28%, reflecting
a relatively consistent performance. However, a potential
concern was observed, as the validation loss (0.9918) was
slightly higher than the training loss (0.9587), suggesting a
potential risk of overfitting.
To address this, the model was subjected to training
with the augmented dataset, incorporating diverse
transformations to enhance its ability to generalize.
Following the mixed training approach, the model
demonstrated improved training performance, with a training
accuracy of 76.91%. The validation accuracy exhibited a
more substantial increase, reaching 75.72%. Importantly, the
validation loss reduced to 0.7143, indicating a positive
reduction in overfitting tendencies.
FIGURE II
ACCURACY OF EACH RESPECTIVE EMOTION
The model's performance was further evaluated on the
testing dataset, demonstrating the generalization capabilities
acquired during training. The test accuracy reached 72.30%,
reaffirming the model's efficacy in accurately classifying
facial expressions on unseen data. Additionally, Confusion
metrics were computed for each emotion class, providing a
comprehensive understanding of the model's class-specific
performance.
FIGURE III
OUR CNN MODEL ARCHITECTURE
In summary, the results highlight the significance of the
mixed training approach, which involved training the model
on both the original preprocessed dataset and an augmented
dataset. This strategy contributed to a substantial reduction
in overfitting tendencies, resulting in improved validation
accuracy and loss. The achieved test accuracy underscores
the model's proficiency in facial expression detection.
Despite the success, avenues for further enhancement exist,
such as fine-tuning hyperparameters, exploring more
sophisticated architectures, and leveraging advanced
regularization techniques to achieve even more robust
performance.
The results affirm the feasibility of the proposed
methodology and pave the way for future research and
advancements in the domain of facial expression detection
using CNN-based models.
V. CONCLUSION
The importance and usefulness of accurately recognizing the
facial expressions of another human without the need of
another human is preceding to be more significant than ever,
and the need of it grows as time progresses and technology
advances. Our model provides a solution to the problem,
while being extremely simple and easy to improve upon.
We used 10 convolutional layers, with max pooling and
dropout in between, the model was first trained on the
original dataset and then on the generated dataset to avoid
overfitting.as mush as possible. Further, early callback
helped in the same, and as a result, we were able to achieve
67.34% accuracy on the testing dataset. Although the model
performed fairly well, the facial expressions in real life are
quite complex and normally cannot simply be put into 7
classes. Other than the complex human nature, light and
brightness plays a huge role in the model’s ability to
recognize, even then the captured image may also be
blocking one’s face or may be blurry, which may stop the
model from detecting altogether. We need to continue
working on these problems.
REFERENCES
Place references in a separate References section at the end
of the paper. Number the references sequentially by order of
appearance, not alphabetically. List up to three authors’
names in a reference; replace the others by “et al.”
 Reference text: 8 point, Times New Roman,
full justified, hanging .25”, no space between the
references, tab between right bracket and the start of the
reference
[1] “Today in Science History: Engineering Quotes.” 2012.
todayinsci.com/QuotationsCategories/E_Cat/Engineering-
Quotations.htm. Web. Accessed: April 9, 2012.
[2] Donohue, Susan K. and Richards, Larry G. October 2011. “P-12
Engineering Education: Using Engineering Teaching Kits to Address
Student Misconceptions in Science.” Proceedings of the 41st Frontiers
in Education Conference, Rapid City, SD, pp. F2A-1 – F2A-3.
[3] Dweck, Carol S. 2006. Mindset: The New Psychology of Success,
New York: Random House, Inc.
[4] Kaplan, Avi and Maehr, Martin L. June 2007. “The Contributions and
Prospects of Goal Orientation Theory.” Educational Psychology
Review 19(2), pp. 141 – 184.
[5] Dweck, Carol S. “Messages That Motivate: How Praise Molds
Students’ Beliefs, Motivation, and Performance (In Surprising
Ways).” In Aronson, Joshua (ed.), 2006, Improving Academic
Achievement: Impact of Psychological Factors on Education. New
York: Elsevier Science, pp. 37 – 60.
AUTHOR INFORMATION
An example: Jane Smith, Professor, Department of
Electrical and Computer Engineering, Big State University.