COUNTENANCE BASED HARMONY

A PROJECT REPORT

Submitted by

SIVA KARTHICK M (210416104101)

NARESH ANAND M (210416104065)

in partial fulfillment for the award of the degree

of

BACHELOR OF ENGINEERING

in

COMPUTER SCIENCE AND ENGINEERING

CHENNAI INSTITUTE OF TECHNOLOGY
CHENNAI 600 069

ANNA UNIVERSITY:: CHENNAI 600 025

APRIL 2020
 BONAFIDE CERTIFICATE

Certified that this Report titled “COUNTENANCE BASED HARMONY” is the bonafide
work of “SIVA KARTHICK M (210416104101), NARESH ANAND M
(210416104065)” who carried out the work under my supervision. Certified further that to
the best of my knowledge the work reported herein does not form part of any other thesis
or dissertation on the basis of which a degree or award was conferred on an earlier occasion
on this or any other candidate.

(SIGNATURE) (SIGNATURE)
Dr.B.Sundarambal., Dr.R.Janarthanan, M.Tech,MBA,P.hD.
HEAD OF THE DEPARTMENT SUPERVISOR
Professor Professor
Dept. of Computer Science and Dept. of Computer Science and
Engineering Engineering
Chennai Institute of Technology Chennai Institute of Technology
Kundrathur-600069 Kundrathur-600069

Submitted for the viva-voice held on ______________ at Chennai Institute of

Technology, Kundrathur.

INTERNAL EXAMINER EXTERNAL EXAMINER

i
 ACKNOWLEDGEMENT
We convey our profound thanks and gratitude to our honorable Chairman
Mr.P.Sriram, Chennai Institute of Technology, Chennai for providing an
excellent academic climate, which made this endeavor possible.
We also express our gratitude to our beloved Principal Dr.P.Parthiban., who
constantly nurtured our standard of education and devoted their precious time for
our needs.
We deeply indebted to pay our sincere thanks to our respectable Head of the
Department Dr.B.Sundarambal., Department of Computer Science and
Engineering and Supervisor Dr.R.Janarthanan.,M.Tech,MBA,Ph.D., Department
of Computer Science and Engineering for showing some exuberant consent for our
project and providing us with all the facilities in the department to complete the
project.
We thank all our department teaching and non-teaching staffs, who have
helped directly and indirectly to complete this project in time.
Last but not least we extend my deep gratitude to our beloved family members
for their moral coordination, encouragement and financial support to carry out this
project.
Finally we express our heartfelt and deep sense of gratitude to all faculty
members in our division and to our friends for their helping hands, valuable, support
and encouragement during the project work.

ii
 ABSTRACT

Facial expressions are a form of verbal communication where our project is

to introduce a system based on facial recognition. To identify the facial expression
and emotion of a person derived from live feed or static images. Using OpenCV
python library for facial object extraction and classification of expressions expressed
by the person. The emotion of a person can be analyzed and can be made stable
through elements such as music.

The user’s emotion using its facial expressions will be detected. These
expressions can be derived from the live feed via system’s camera or any pre-
existing image available in the memory. Emotions possessed by humans can be
recognized and has a vast scope of study in the computer vision industry upon which
several researches have already been done. The work has been implemented using
Python (3.6), Open Source Computer Vision Library (OpenCV) and NumPy. The
scanned image (testing dataset) is being compared to training dataset and thus
emotion is predicted. The objective of the system is to analyze the image and predict
the expression of the person.

The FisherFaces algorithm can be used to detect the face from the image
through average classification. The emotion can be detected by taking various
images of different emotions to train the data-set. The model can be trained for
improved accuracy and robust. Once the emotion is successfully detected with some
basic error reductions, the music can be played or suggested.

iii
 TABLE OF CONTENTS
CHAPTER TITLE PAGE NO
ABSTRACT iv
LIST OF FIGURES vii
LIST OF ABBREVIATIONS viii
1 INTRODUCTION 1
1.1 Objective 1
1.2 Definition of Image 2
1.3 Image Processing 3
1.4 Types of Image Processing 4
1.5 Application of Image Processing 8
1.6 Face Detection 14
1.7 Emotion 18
2 SYSTEM ANALYSIS 23
2.1 Existing System 23
2.2 Literature Survey 27
2.2 Proposed System 35
3 REQUIREMENTS SPECIFICATION 36
3.1 Hardware Specification 36
3.2 Software Specification 36
3.3 Technology Specification 36
4 SYSTEM DESIGN 42
4.1 System Layout 42
4.2 Use Case Diagram 43
4.3 Activity Diagram 44
4.4 Class Diagram 45
4.5 Data Flow Diagram 46
4.6 Sequence Diagram 47
iv
5 IMPLEMENTATION 48
5.1 Project Implementation 48
5.2 Modules 51
5.3 Algorithm Description 55
6 TESTING 58
6.1 Testing 58
7 CONCLUSION AND FUTURE WORKS 61
7.1 Conclusion 61
7.2 Future works 62
8 APPENDIX -- A 65
APPENDIX -- B 72
REFERENCE 73

v
 LIST OF FIGURES

FIG NO TITLE PAGE NO

1.1 Emotion classification process 1

1.2 Facial expressions 2
1.3 Image Processing 3
1.4 Image sharpening and restoration 9
1.5 UV Imaging 10
1.6 Transmission and encoding 11
1.7 Hurdle detection 12
1.8 Line follower robot 12
1.9 Video processing 13
1.10 Payment face detection 15
1.11 Face detection for access and security 16
1.12 Criminal identification 16
1.13 Healthcare using face detection 17
1.14 Face recognition in personalized car 19
1.15 Emotion detection in interviews 20
1.16 Emotion detection in video gaming 21
1.17 Facial recognition market 22
2.1 Emotion classification approach 23
3.1 Python 37
3.2 OpenCV 38

vi
3.3 Numpy 38
3.4 HTML,CSS,JS 41
4.1 System Layout 42
4.2 Use Case Diagram 43
4.3 Activity Diagram 44
4.4 Class Diagram 45
4.5 Data Flow Diagram 46
4.6 Sequence Diagram 47
5.1 Camera visualization 51
5.2 Face detection 52
5.3 Extracting and classifying faces 53
5.4 Feature extraction 54
5.5 Emotion recognition 54
5.6 Music suggestion 55
5.7 HAAR feature 57

vii
 LIST OF ABBREVISTIONS

UV Ultra Violet
RGB Red Green Blue
HSV Hue Saturation and Value
AI Artificial Intelligence
PET Positron Emission Tomography
fMRI functional Magnetic Resonance Imaging
HR Human Resources
API Application Program Interface
ATM Automated Teller Machine
SVM Support Vector Machine
ANN Artificial Neural Network
CNN Conventional Neural Network
LSTM Long Short-Term Memory
ELM Extreme Learning Machine
NLP Natural Language Processing
EEG Electroencephalography
ECG Electrocardiography
LBP Local Binary Pattern
GMM Gaussian Mixture Matrix
JAFFE Japanese Female Facial Expression
OpenCV Open Source Computer Vision
VGG Visual Geometry Group

viii
GPL General Public License
ONNX Open Neural Network Exchange
CUDA Compute Unified Device Architecture
GPU Graphics Processing Unit
GUI Graphical User Interface
WHATWG Web Hypertext Application Technology Working Group
SVG Scalable Vector Graphics
HTML Hyper Text Markup Language
CSS Cascading Style Sheet
DOM Document Object Model

ix
 CHAPTER 1

1.INTRODUCTION
1.1 OBJECTIVE:
Facial expressions are a form of verbal communication where our project is
to introduce a system based on facial recognition. To identify the facial
expression and emotion of a person derived from live feed or static images. Using
OpenCV python library for facial object extraction and classification of
expressions expressed by the person. The FisherFaces algorithm can be used to
detect the face from the image through average classification. Data Sampling is
performed by dividing the data into Training and Testing data. The emotion can
be detected by taking various images of different emotions to train the data-set.
The model can be trained for improved accuracy and robust. The emotion of a
person can be analyzed and can be made stable through elements such as music.
Once the emotion is successfully detected with some basic error reductions, the
music can be played or suggested.

Fig 1.1 Emotion classification process

1
1.2 DEFINITION OF IMAGE:
An image is an artefact that depicts visual perception, such as a photograph or
other two-dimensional picture, particularly one that resembles a subject (usually
a physical object). It is a data representing a two-dimensional scene such as a
photograph or screen display, or three-dimensional, such as a statue or hologram,
that is composed of pixels arranged in a rectangular array. Each pixel may
consists of one or more bits of information representing the brightness of the
image including colour information encoded as RGB triples.

In the context of signal processing, an image is a distributed amplitude of

colour(s). They may be captured by optical devices (such
as cameras, mirrors, lenses, telescopes, microscopes) and natural objects and
phenomena (such as the human eye or water).The word "image" is also used in
the broader sense of any two-dimensional figure such as a map, a graph, a pie
chart, a painting or a banner.

A volatile image is one that exists only for a short period of time. This may be
a reflection of an object by a mirror, a projection of a camera obscure, or a scene
displayed on a cathode ray tube. A fixed image, also called a hard copy, is one
that has been recorded on a material object, such as paper or textile by
photography or any other digital process. A mental image exists in an individual's
mind, as something one remembers or imagines. The subject of an image need
not be real; it may be an abstract concept, such as a graph, function, etc.

Fig 1.2 Facial Expressions

2
1.3 IMAGE PROCESSING:
Image processing is any form of processing for which the input is
an image or a series of images or videos, such as photographs or frames of video.

Fig 1.3 Image Processing

The output of an image processing can be either an image or a set of

characteristics or parameters related to the image. It also means "Analysing and
manipulating images with a computer".

Image processing basically includes the following three steps:

❖ Importing the image via image acquisition tools;

o First, import images with an optical devices like a scanner or

a camera, or make them by computer-generated imagery.

❖ Analysing and manipulating the image;

o Second, manipulate or analyse the images in some way. This step

can include image improvement and data summary, or the images
are analysed to find rules that aren't seen by the human eyes. For
example, meteorologists use this processing to analyse satellite
photographs.

3
 ❖ Output in which result can be altered image or report that is based on image
analysis.

o Last, output the result of image processing. The result might be the
image changed by some way or it might be a report based on analysis
or result of the images.

In the 21st century image processing is usually "digital", but optical and
analog image methods are still used. Image processing is a method to perform
some operations on an image, in order to get an enhanced image or to extract
some useful information from it. It is a type of signal processing in which input
is an image and output may be image or characteristics/features associated with
that image. Nowadays, image processing is among rapidly growing technologies.
It forms core research area within engineering and computer science disciplines
too.

1.4 TYPES OF IMAGE PROCESSING:

There are two types of methods used for image processing namely, analog and
digital image processing. Analog image processing can be used for the hard
copies like printouts and photographs. Image analysts use various fundamentals
of interpretation while using these visual techniques. Digital image processing
techniques help in manipulation of the digital images by using computers. The
three general phases that all types of data have to undergo while using digital
technique are pre-processing, enhancement, and display, information extraction.

1.4.1 ANALOG IMAGE PROCESSING:

Analog image, In remote sensing, an image in which continuous variation in
the scene being sensed is represented by continuous variation in image tone, such
as the image produced from photosensitive chemicals in a photographic film.
Analog image processing is any image processing task conducted on two-
dimensional analog signals by using analog means (as opposed to digital image

4
processing). The processing is applied on analog signals and it processes only
two-dimensional signals. The images are manipulated by electrical means by
varying the electrical signal. The common example include is the television
image. The images are manipulated by electrical signals. In analog image
processing, analog signals can be periodic or non-periodic.

1.4.2 DIGITAL IMAGE PROCESSING:

In computer science, digital image processing is the use of a digital
computer to process digital images through an algorithm. As a subcategory or
field of digital signal processing, digital image processing has many advantages
over analog image processing. It allows a much wider range of algorithms to be
applied to the input data and can avoid problems such as the build-up
of noise and distortion during processing. Since images are defined over two
dimensions digital image processing may be modelled in the form
of multidimensional systems. The generation and development of digital image
processing are mainly affected by three factors: first, the development of
computers; second, the development of mathematics (especially the creation and
improvement of discrete mathematics theory); third, the demand for a wide range
of applications in environment, agriculture, military, industry and medical
science has increased.

Filtering
Digital filters are used to blur and sharpen digital images. Filtering can be
performed by:

• convolution with specifically designed kernels (filter array) in the spatial

domain.
• masking specific frequency regions in the frequency (Fourier) domain

5
 The following examples show both methods:

Filter type Kernel or mask Example

Original
Image

Spatial
Lowpass

Spatial
Highpass

Pseudo-code:

Fourier image = checkerboard

Representation F = Fourier Transform of image
Show Image: log(1+Absolute
Value(F))

6
 Fourier
Lowpass

Fourier
Highpass

1.4.3 DIFFERENCE BETWEEN ANALOG AND DIGITAL IMAGE

PROCESSING:

ANALOG IMAGE PROCESSING DIGITAL IMAGE PROCESSING

The analog image processing is applied The digital image processing is applied to
on analog signals and it processes only digital signals that work on analysing and
two-dimensional signals. manipulating the images.

Analog signal is time-varying signals so It improves the digital quality of the image
the images formed under analog image and intensity distribution is perfect in it.
processing get varied.

Analog image processing is a slower and Digital image processing is a cheaper and
costlier process. fast image storage and retrieval process.

Analog signal is a real-world but not good It uses good image compression techniques
quality of images. that reduce the amount of data required and
produce good quality of images

7
It is generally continuous and not broken It uses an image segmentation technique
into tiny components. which is used to detect discontinuity which
occurs due to a broken connection path.

Analog image’s pixel value must be Digital image’s pixel value must be
continuous. discrete.

The amplitude of analog image is infinite. The amplitude of digital image is finite.

It is quite impossible to store all the pixels It is quite possible to store all the pixels of
of analog image. digital image.

1.5 APPLICATION OF IMAGE PROCESSING:

Some of the major fields in which digital image processing is widely used are
mentioned below

• Image sharpening and restoration

• Medical field

• Remote sensing

• Transmission and encoding

• Machine/Robot vision

• Color processing

• Pattern recognition

• Video processing

• Microscopic Imaging

• Others

8
Image sharpening and restoration
Image sharpening and restoration refers here to process images that have
been captured from the modern camera to make them a better image or to
manipulate those images in way to achieve desired result. It refers to do what
Photoshop usually does.

This includes Zooming, blurring , sharpening , gray scale to color

conversion, detecting edges and vice versa , Image retrieval and Image
recognition. The common examples are:

The original image The zoomed image Blurr image

Sharp image Edges

Fig 1.4 Image sharpening and restoration

9
Medical field
The common applications of DIP in the field of medical is
• Gamma ray imaging
• PET scan
• X Ray Imaging
• Medical CT
• UV imaging
UV imaging
In the field of remote sensing , the area of the earth is scanned by a satellite
or from a very high ground and then it is analyzed to obtain information about
it. One particular application of digital image processing in the field of remote
sensing is to detect infrastructure damages caused by an earthquake.

As it takes longer time to grasp damage, even if serious damages are

focused on. Since the area effected by the earthquake is sometimes so wide , that
it not possible to examine it with human eye in order to estimate damages. Even
if it is , then it is very hectic and time consuming procedure. So a solution to this
is found in digital image processing. An image of the effected area is captured
from the above ground and then it is analysed to detect the various types of
damage done by the earthquake.

Fig 1.5 UV Imaging

10
The key steps include in the analysis are
• The extraction of edges
• Analysis and enhancement of various types of edges
Transmission and encoding
The very first image that has been transmitted over the wire was from
London to New York via a submarine cable. The picture that was sent is shown
below.

Fig 1.6 Transmission and Encoding

The picture that was sent took three hours to reach from one place to
another. Now just imagine , that today we are able to see live video feed , or live
cctv footage from one continent to another with just a delay of seconds. It means
that a lot of work has been done in this field too. This field doesnot only focus
on transmission , but also on encoding. Many different formats have been
developed for high or low bandwith to encode photos and then stream it over the
internet or e.t.c.

Machine/Robot vision
Apart form the many challenges that a robot face today , one of the biggest
challenge still is to increase the vision of the robot. Make robot able to see things
, identify them , identify the hurdles e.t.c. Much work has been contributed by
this field and a complete other field of computer vision has been introduced to
work on it.

11
Hurdle detection
Hurdle detection is one of the common task that has been done through
image processing, by identifying different type of objects in the image and then
calculating the distance between robot and hurdles.

Fig 1.7 Hurdle Detection

Line follower robot
Most of the robots today work by following the line and thus are called
line follower robots. This help a robot to move on its path and perform some
tasks. This has also been achieved through image processing.

Fig 1.8 Line Follower Robot

Color processing
Color processing includes processing of colored images and different color
spaces that are used. For example RGB color model , YCbCr, HSV. It also
involves studying transmission , storage , and encoding of these color images.

12
Pattern recognition
Pattern recognition involves study from image processing and from
various other fields that includes machine learning ( a branch of artificial
intelligence). In pattern recognition , image processing is used for identifying the
objects in an images and then machine learning is used to train the system for
the change in pattern. Pattern recognition is used in computer aided diagnosis ,
recognition of handwriting , recognition of images ,etc.

Video processing
A video is nothing but just the very fast movement of pictures. The quality
of the video depends on the number of frames/pictures per minute and the quality
of each frame being used. Video processing involves noise reduction , detail
enhancement , motion detection , frame rate conversion , aspect ratio conversion
, color space conversion ,etc.

Fig 1.9 Video Processing

13
1.6 FACE DETECTION:
Face detection can be regarded as a specific case of object-class detection.
In object-class detection, the task is to find the locations and sizes of all objects
in an image that belong to a given class. Examples include upper torsos,
pedestrians, and cars. Face-detection algorithms focus on the detection of frontal
human faces. It is analogous to image detection in which the image of a person is
matched bit by bit. Image matches with the image stores in database. Any facial
feature changes in the database will invalidate the matching process. A reliable
face-detection approach based on the genetic algorithm and the eigen-
face technique:

Firstly, the possible human eye regions are detected by testing all the valley
regions in the gray-level image. Then the genetic algorithm is used to generate all
the possible face regions which include the eyebrows, the iris, the nostril and the
mouth corners.

Each possible face candidate is normalized to reduce both the lighting

effect, which is caused by uneven illumination; and the shirring effect, which is
due to head movement. The fitness value of each candidate is measured based on
its projection on the eigen-faces. After a number of iterations, all the face
candidates with a high fitness value are selected for further verification. At this
stage, the face symmetry is measured and the existence of the different facial
features is verified for each face candidate.

1.6.1 APPLICATIONS OF FACE DETECTION:

1. PAYMENTS :
It doesn’t take a genius to work out why businesses want payments to be
easy. Online shopping and contactless cards are just two examples that
demonstrate the seamlessness of postmodern purchases. With FaceTech,
however, customers wouldn’t even need their cards. In 2016, MasterCard

14
launched a new selfie pay app called MasterCard Identity Check. Customers open
the app to confirm a payment using their camera, and that’s that. Facial
recognition is already used in store and at ATMs, but the next step is to do the
same for online payments. Chinese ecommerce firm Alibaba and affiliate
payment software Alipay are planning to apply the software to purchases made
over the Internet.

Fig 1.10 Payments Face Detection

2. ACCESS AND SECURITY :

As well as verifying a payment, facial biometrics can be integrated with

physical devices and objects. Instead of using passcodes, mobile phones and other
consumer electronics will be accessed via owners’ facial features. Apple,
Samsung and Xiaomi Corp. have all installed FaceTech in their phones. This is
only a small scale example, though. In future, it looks like consumers will be able
to get into their cars, houses, and other secure physical locations simply by
looking at them. Jaguar is already working on walking gait ID – a potential
parallel to facial recognition technology. Other corporations are likely to take
advantage of this, too. Innovative facial security could be especially useful for a
company or organisation that handles sensitive data and needs to keep tight
controls on who enters their facilities.

15
 Fig 1.11 Face Detection for Access and Security
3. CRIMINAL IDENTIFICATION :

If FaceTech can be used to keep unauthorised people out of facilities,

surely it can be used to help put them firmly inside them. This is exactly what the
US Federal Bureau of Investigation is attempting to do by using a machine
learning algorithm to identify suspects from their driver’s licences. The FBI
currently have a database which includes half of the national population’s faces.
This is as useful as it is creepy, giving law enforcers another way of tracking
criminals across the country. AI equipped cameras have also been trialled in the
UK to identify those smuggling contraband into prisons.

Fig 1.12 Criminal Identification using Face Detection

4. ADVERTISING :

The ability to collect and collate masses of personal data has given
marketers and advertisers the chance to get closer than ever to their target

16
markets. FaceTech could do much the same, by allowing companies to recognise
certain demographics – for instance, if the customer is a male between the ages
of 12 and 21, the screen might show an ad for the latest FIFA game. Grocery giant
Tesco plans to install OptimEyes screens at 450 petrol stations in the UK to
deliver targeted ads to customers. According to company CEO Simon Sugar, the
cameras could change the face of British retail. Perhaps he’s right – but only if
the cameras can correctly identify customers. Being classified as the wrong age
or gender is far less amusing than having your name spelt wrong on a Starbucks
cup.

5. HEALTHCARE :

Instead of recognising an individual via FaceTech, medical professionals

could identify illnesses by looking at a patient’s features. This would alleviate the
ongoing strain on medical centres by slashing waiting lists and streamlining the
appointment process. The question is, would you really want to find out you had
a serious illness from a screen? If it’s a choice between a virtual consultation or
a month long wait for an appointment, then maybe so. Another application of
facial biometrics within healthcare is to secure patient data by using a unique
patient photo instead of passwords and usernames.

Fig 1.13 Healthcare using Face Detection

With a predicted worth of $15 billion by 2025, biometrics is an industry
worth watching. It’s clear that facial biometrics are a helpful tool for finance, law
enforcement, advertising and healthcare, as well as a solution to hacking and

17
identity theft. Of course, FaceTech is by no means fool proof. Gaining access to
possessions using physical traits could even be counterintuitive for security. A
face, as social robots like Nadine have shown us, is easily replicated. And when
it comes to public adoption, some people are reluctant to switch to contactless
cards, let alone abandon them completely. For the most part, though, facial
recognition technology seems to be encouraging a more seamless relationship
between people, payments and possessions.

1.7 EMOTION :
Emotions are biological states associated with the nervous system brought
on by neurophysiological changes variously associated with thoughts, feelings,
behavioural responses, and a degree of pleasure or displeasure. There is currently
no scientific consensus on a definition. Emotion is often intertwined
with mood, temperament, personality, disposition, creativity and motivation.
Research on emotion has increased significantly over the past two decades with
many fields contributing including psychology, neuroscience, affective,
neuroscience, endocrinology, medicine, history, sociology of emotions,
and computer science. The numerous theories that attempt to explain the origin,
neurobiology, experience, and function of emotions have only fostered more
intense research on this topic. Current areas of research in the concept of emotion
include the development of materials that stimulate and elicit emotion. In
addition PET scans and fMRI scans help study the affective picture processes in
the brain.

1.7.1 APPLICATION OF EMOTION DETECTION :

1. MAKING CARS SAFER AND PERSONALIZED :

Car manufacturers around the world are increasingly focusing on making
cars more personal and safe for us to drive. In their pursuit to build more smart
car features, it makes sense for makers to use AI to help them understand the

18
human emotions. Using facial emotion detection smart cars can alert the driver
when he is feeling drowsy.

Fig 1.14 Face and Emotion Recognition for Personalized car

The US Department of Transportation claims that driving-related errors

cause around 95% of fatal road accidents. Facial Emotion Detection can find
subtle changes in facial micro-expressions that precedes drowsiness and send
personalized alerts to the driver asking him to stop for a coffee break, change
music or temperature.

2. FACIAL EMOTION DETECTION IN INTERVIEWS :

A candidate-interviewer interaction is susceptible to many categories of
judgment and subjectivity. Such subjectivity makes it hard to determine whether
candidate's personality is a good fit for the job. Identifying what a candidate is
trying to say is out of our hands because of the multiple layers of language
interpretation, cognitive biases, and context that lie in between. That's where AI
comes in, which can measure candidate's facial expressions to capture their
moods and further assess their personality traits.

19
 Fig 1.15 Emotion Detection in Interviews

Notably, Unilever is already starting to incorporate this technology into

their recruitment process. With this technology, a recruiter will be able to know,
say, the overall confidence level of an interviewee and make a decision about
whether or not this candidate will be able to perform well at a client-facing job.
Similarly, it will be possible to find whether the candidate is honestly replying to
all the questions by measuring the change in emotions during his responses and
correlating it the vast amount of knowledge available in this area.

Employee morale can also be perceived using this technology by holding

and recording interactions on the job. As an HR tool, it can help not only in
devising recruiting strategies but also in designing HR policies that bring about
best performance from employees.

3. TESTING FOR VIDEO GAMES :

Video games are designed keeping in mind a specific target audience. Each
video game aims to evoke a particular behavior and set of emotions from the
users. During the testing phase, users are asked to play the game for a given period
and their feedback is incorporated to make the final product. Using facial emotion
detection can aid in understanding which emotions a user is going through in real-
time as he is playing without analyzing the complete video manually.

20
 Fig 1.16 Emotion Detection in Video Gaming

Such product feedback can be taken by analyzing a live feed of the user
and detecting his facial emotions. While feelings of frustration and anger are
commonly experienced in advanced video games, making use of facial emotion
detection will help understand which emotions are experienced at what points in
the game. It is also possible that some unexpected or undesirable emotions are
observed during the game. Taking feedback from the user has experienced the
game can be inefficient. This is because it can often be difficult to put an
experience into words. Moreover, users may be unable to remember what exactly
they went through emotionally across different parts of the game. Facial emotion
detection is a practical means of going beyond the spoken or written feedback
and appreciating what the user is experiencing. When feedback is taken in this
format, it becomes genuinely non-intrusive when it comes to user experience. At
the same time, such feedback is more reliable than other forms.

4. MARKET RESEARCH :
Traditional market research companies have employed verbal methods
mostly in the form of surveys to find the consumers wants and needs. However,
such methods assume that consumers can formulate their preferences verbally
and the stated preferences correspond to future actions which may not always be
right.
21
 Fig 1.17 Facial Recognition Market

Another popular approach in market research industry is to employ

behavioral methods that let users act while trying the product. Such methods are
considered more objective than verbal methods. Behavioral methods use video
feeds of a user interacting with the product, and the video is then analyzed
manually to observer user's reactions and emotions. This can quickly become
very labor intensive increasing the overall cost. Facial emotion recognition AI
can automatically detect facial expressions on user's faces and automate the video
analysis completely. Market research companies can use this technology to scale
the data collection efforts and rely on the technology to do analysis quickly.

Detecting emotions with technology is a challenging task, yet one where

machine learning algorithms have shown great promise. Using ParallelDots'
Facial Emotion Detection API, customers can process images, and videos in real-
time for monitoring video feeds or automating video analytics, thus saving costs
and making life better for their users.

22
 CHAPTER 2

2. SYSTEM ANALYSIS
2.1 Existing Solution:
This paper discusses the application of feature extraction of facial
expressions with combination of neural network for the recognition of
different facial emotions (happy, sad, angry, fear, surprised, neutral etc..).
Humans are capable of producing thousands of facial actions during
communication that vary in complexity, intensity, and meaning. In Emotion
recognition using brain activity the developer Robert Horlings has used brain
activities which is toughest task to do as it become expensive, complex and
also time consuming when we try to measure human brain with
Electroencephalography (eeg).

2.1.1 Approaches:
The accuracy of emotion recognition is usually improved when it combines
the analysis of human expressions from multimodal forms such as texts,
physiology, audio, or video. Different emotion types are detected through the
integration of information from facial expressions, body movement and gestures,
and speech. The technology is said to contribute in the emergence of the so-called
emotional or emotive Internet.

Fig 2.1 Emotion classification approach

23
 The existing approaches in emotion recognition to classify
certain emotion types can be generally classified into three main categories:
knowledge-based techniques, statistical methods, and hybrid approaches.

2.1.2 Knowledge-based techniques:

Knowledge-based techniques (sometimes referred to as lexicon-based
techniques), utilize domain knowledge and
the semantic and syntactic characteristics of language in order to detect
certain emotion types. In this approach, it is common to use knowledge-based
resources during the emotion classification process such as WordNet,
SenticNet, ConceptNet, and EmotiNet, to name a few. One of the advantages of
this approach is the accessibility and economy brought about by the large
availability of such knowledge-based resources. A limitation of this technique on
the other hand, is its inability to handle concept nuances and complex linguistic
rules.

Knowledge-based techniques can be mainly classified into two categories:

dictionary-based and corpus-based approaches. Dictionary-based approaches
find opinion or emotion seed words in a dictionary and search for
their synonyms and antonyms to expand the initial list of opinions
or emotions. Corpus-based approaches on the other hand, start with a seed list of
opinion or emotion words, and expand the database by finding other words with
context-specific characteristics in a large corpus. While corpus-based approaches
take into account context, their performance still vary in different domains since
a word in one domain can have a different orientation in another domain.

2.1.3 Statistical methods:

Statistical methods commonly involve the use of different
supervised machine earning algorithms in which a large set of annotated data is
fed into the algorithms for the system to learn and predict the
appropriate emotion types. Machine learning algorithms generally provide more

24
reasonable classification accuracy compared to other approaches, but one of the
challenges in achieving good results in the classification process, is the need to
have a sufficiently large training set.

Some of the most commonly used machine learning algorithms

include Support Vector Machines (SVM), Naive Bayes, and Maximum
Entropy. Deep learning, which is under the unsupervised family of machine
learning, is also widely employed in emotion recognition. Well-known deep
learning algorithms include different architectures of Artificial Neural Network
(ANN) such as Convolutional Neural Network (CNN), Long Short-term Memory
(LSTM), and Extreme Learning Machine (ELM). The popularity of deep
learning approaches in the domain of emotion recognition may be mainly
attributed to its success in related applications such as in computer vision, speech
recognition, and Natural Language Processing (NLP).

2.1.4 Hybrid approaches:

Hybrid approaches in emotion recognition are essentially a combination of

knowledge-based techniques and statistical methods, which exploit
complementary characteristics from both techniques. Some of the works that have
applied an ensemble of knowledge-driven linguistic elements and statistical
methods include sentic computing and iFeel, both of which have adopted the
concept-level knowledge-based resource SenticNet. The role of such knowledge-
based resources in the implementation of hybrid approaches is highly important
in the emotion classification process. Since hybrid techniques gain from the
benefits offered by both knowledge-based and statistical approaches, they tend to
have better classification performance as opposed to employing knowledge-based
or statistical methods independently. A downside of using hybrid techniques
however, is the computational complexity during the classification process.

25
2.1.5 Datasets:
Data is an integral part of the existing approaches in emotion recognition and
in most cases it is a challenge to obtain annotated data that is necessary to
train machine learning algorithms. For the task of classifying
different emotion types from multimodal sources in the form of texts, audio,
videos or physiological signals, the following datasets are available:

1. HUMAINE: provides natural clips with emotion words and context labels
in multiple modalities.
2. Belfast database: provides clips with a wide range of emotions from TV
programs and interview recordings.
3. SEMAINE: provides audiovisual recordings between a person and
a virtual agent and contains emotion annotations such as angry, happy,
fear, disgust, sadness, contempt, and amusement.
4. IEMOCAP: provides recordings of dyadic sessions between actors and
contains emotion annotations such as happiness, anger, sadness,
frustration, and neutral state.
5. eNTERFACE: provides audiovisual recordings of subjects from seven
nationalities and contains emotion annotations such as happiness, anger,
sadness, surprise, disgust, and fear.
6. DEAP:provides electroencephalography (EEG), electrocardiography (EC
G), and face video recordings, as well as emotion annotations in terms
of valence, arousal, and dominance of people watching film clips.
7. DREAMER: provides electroencephalography (EEG)
and electrocardiography (ECG) recordings, as well
as emotion annotations in terms of valence, arousal, and dominance of
people watching film clips.
8. MELD: is a multiparty conversational dataset where each utterance is
labeled with emotion and sentiment. MELD provides conversations in

26
 video format and hence suitable for multimodal emotion recognition
and sentiment analysis. MELD is useful for multimodal sentiment
analysis and emotion recognition, dialogue systems and emotion
recognition in conversations.

2.2 Literature Survey

1. An Emotion Recognition System for Mobile Applications

Author: M. Shamim Hossain

Published year: February 2017

Description:
In this paper, it proposes a high-performance emotion recognition system
for mobile applications. The embedded camera in a smart phone captures the
video of the user. The Bandlet transform is applied to some selective frames to
reduce the burden of processing to give some subband images. Local binary
patterns (LBP) histogram is calculated from the subband images which is a
powerful gray-level image descriptor which is used to describe the features of the
frames. A Gaussian mixture model (GMM) based classifier is used as a classifier
for training and testing. The proposed emotion recognition system is evaluated
using several databases for improved accuracy. An emotion recognition system
for mobile applications has been proposed. The emotions are recognized by face
images. The Bandlet transform and the LBP are used as features, which are then
selected by the KW feature selection method. The GMM based classifier is
applied to recognize the emotions. Two publicly available databases are used to
validate the system. The proposed system achieved 99.8% accuracy using the
JAFFE database, and 99.7% accuracy using the CK database. It takes less than
1.4 seconds to recognize one instance of emotion. The high performance and the
less time requirement of the system make it suitable to any emotion-aware mobile

27
applications. In a future study, we want to extend this work to incorporate
different input modalities of emotion.
In the proposed system, facial video is captured by an embedded camera
of a smart phone. Some representative frames are extracted from the video, and a
face detection module is applied to extract the face regions in the frames. The
Bandlet transform is realized on the face regions, and the resultant subband is
divided into non-overlapping blocks. Local binary patterns’ histograms are
calculated for each block, and then are concatenated over all the blocks. The
Kruskal–Wallis feature selection is applied to select the most dominant bins of
the concatenated histograms. The dominant bins are then fed into a Gaussian
mixture model-based classifier to classify the emotion. Experimental results show
that the proposed system achieves high recognition accuracy in a reasonable time.

Advantage:
• Provides an emotion recognition system for mobile application.
• It achieves an accuracy of about 99.8%.
• It takes less than 1.4 seconds to recognize one instance of emotion.
• Using histograms, only the representative frames are selected to reduce
the burden of processing.

2. An Emotion Recognition Model Based on Facial Recognition in

Virtual Learning Environment

Author: D. Yang , Abeer Alsadoon , P.W.C. Prasad , A. K. Singh,

A.Elchouemi

Published year: December 2017

Description:
In the field of education, existing virtual learning environments with
success simulate interaction at a psychological feature level throughout ancient
teaching processes. Students show completely different expressions in videos,
lecturers will determine whether or not the scholars perceive their teaching
28
content in line with student’s with completely different expressions, and might
modify their teaching programs. In virtual environments, student’s emotional
knowledge is obtained from a series of video frames. Therefore, to achieve real
time understanding of student’s emotions, accuracy and time must be balanced.
To improve accuracy, a lot of video frames square measure needed, increasing
computation time. This paper proposes a learning emotion recognition model,
which consists of three stages: Feature extraction, subset feature and emotion
classifier. A Haar Cascades method is used to detect the input image, a face, as
the basis for the extraction of eyes and mouth, and then through the Sobel edge
detection to obtain the characteristic value. Through Neural Network classifier
training, six kinds of different emotional categories are obtained. Experiments
using JAFFE database show that the proposed method has high classification
performance. Experimental results show that the model proposed in this paper is
consistent with the expressions from the learning situation of students in virtual
learning environments. This paper demonstrates that emotion recognition based
on facial expressions is feasible in distance education, permitting identification
of a student’s learning status in real time. Therefore, it can help teachers to change
teaching strategies in virtual learning environments according to the student’s
emotions.
Advantage:
• Solves the problems of emotion recognition based on facial recognition in
virtual learning environments.
• It can be applied for real distance education.
• Optimization of recognition of emotional changes in the facial features of
online learners.
Drawback:
• It does not involve the illumination and pose of the image, it is uncertain
how much these factors influence facial expressions.
• The accuracy of recognition of faces is comparatively less.

29
3.Emotion Detection using Image Processing in Python

Author: Raghav Puri, Archit Gupta, Manas Sikri, Mr. Mohit Tiwari, Dr. Nitish
Pathak, Dr. Shivendra Goel.

Published year: March 2018

Description:
It is a very efficient way of image processing through which an image can
be converted into its digital form subsequently performing various operations on
it. This is a technique similar to signal processing, in which the input given is a
2D image, which is a collection of numbers ranging from 0 to 255 which denotes
the corresponding pixel value. The image is converted into digital form for good
resolution and high quantization level for human perception of different shades.
The subject on each image is analyzed, converted to grayscale, cropped and saved
to a directory which can be compiled as training set and classification set. The
user’s emotion using its facial expressions will be detected. These expressions
can be derived from the live feed via system’s camera or any pre-existing image
available in the memory. Emotions possessed by humans can be recognized and
has a vast scope of study in the computer vision industry upon which several
researches have already been done. The work has been implemented using Python
(2.7), Open Source Computer Vision Library (OpenCV) and NumPy. The
scanned image (testing dataset) is being compared to training dataset and thus
emotion is predicted. The objective of this paper is to develop a system which can
analyze the image and predict the expression of the person. The study proves that
this procedure is workable and produces valid results.
Advantage:
• Artificial Intelligence is used to solve the intriguing tasks in emotion
detection.
• The image is converted into digital form to enhance for good resolution
and high quantization level.
• The input is a 2D image which consists of pixel values.
30
Drawback:
• The system detects only less than three emotions.
• The optimum accuracy is nearly 83% which is relatively less.
• The properties of the image can be affected due to converting it into
digital form.

4. Face Detection and Tracking Using OpenCV

Author: Kruti Goyal, Kartikey Agarwal, Rishi Kumar

Published year: April 2017

Description:
Face detection is the most popular area of research in the vision of
computer science. It is a computer technology which is being used in a variety of
applications that identifies human faces in digital images. The research under this
field is expanding in many areas of science such as psychology. Face detection is
one of the most talked about in technology. Localization of human faces is
considered as the primary and the initial stage in study of face detection . The
application is used for tracking and detecting faces in videos or still images
through cameras which can be used for multipurpose activities. The intention of
the paper is to deep study of face detection using OpenCV. A tabular comparison
is provided to understand the available algorithms in an easier manner. It talks
about various types of algorithms that exist like Adaboost, Haar cascades. The
paper aims to help in understanding the best prerequisites for face detection. Face
detection and tracking is used in many of the fields these days. It is collaborated
in every section of the trade and some are effectively and efficiently using it,
others still depend the on the old methods.

31
Advantage:
• It compares all the algorithm in terms of space and time paradigm.
• It talks about the difference in face detection via performing it with
OpenCV and performing it with matlab.
• OpenCV is fast, portable and free of cost.
• It provides an understanding of the best prerequisites for face detection.

Drawback:
• The feature extraction using HAAR Cascades is time consuming.
• It cannot process without the variable light source for face detection
through camera.

5. Facial emotion recognition in real-time and static images

Author: Shivam Gupta

Published year: January 2018

Description:
Facial expressions are a form of nonverbal communication. Various studies
have been done for the classification of these facial expressions. There is strong
evidence for the universal facial expressions of eight emotions which include:
neutral happy, sadness, anger, contempt, disgust, fear, and surprise. So it is very
important to detect these emotions on the face as it has wide applications in the
field of Computer Vision and Artificial Intelligence. These fields are researching
on the facial emotions to get the sentiments of the humans automatically. In
Robotics, emotions classification can be used to enhance human-robot
interactions since the robot is capable of interpreting a human reaction . In this
project, the images datasets were organized properly and then sorted accordingly.
Then the face was detected in all the images using the Haar filters in OpenCV as
it has few face recognized classes and then detect, crop and save faces. The faces

32
were detected in the frames according to the facial landmarks which will contain
the eyes, eyebrows, nose, mouth, corners of the face. The classification is done
using the supervised learning (SVM) in this project as it gave better accuracy.
The training and classification sets were created and we randomly sample and
train on 80% of the data and classify the remaining 20%. This whole work was
done on the Cohn-Kanade datasets of static images.
Advantage:
• Provides a fully automatic recognition of facial emotions using the
computer vision and machine learning algorithms.
• Provides an accuracy of 94.1%.
• The system can be extended to the deep learning system using the
Convolutional Neural networks with an accuracy of 99.5%.
• Each image sequence is processed for the development of an emotional
expression.
Drawback:
• The classifier needs to be trained in order to achieve accuracy in
extraction of emotion.
• Using SVM, it contained a little errors in the prediction which reduced
the accuracy of processing.

6. Facial Emotion Recognition in Real Time

Author: Dan Duncan, Gautam Shine, Chris English

Published Year: 2016

Description:
Emotions often mediate and facilitate interactions among human beings.
Thus, understanding emotion often brings context to seemingly bizarre and/or
complex social communication. Emotion can be recognized through a variety of
means such as voice intonation, body language, and more complex methods such

33
electroencephalography (EEG). However, the easier, more practical method is to
examine facial expressions. There are seven types of human emotions shown to
be universally recognizable across different cultures: anger, disgust, fear,
happiness, sadness, surprise, contempt. Interestingly, even for complex
expressions where a mixture of emotions could be used as descriptors, cross-
cultural agreement is still observed. The task of emotion recognition is
particularly difficult for two reasons: 1) There does not exist a large database of
training images and 2) classifying emotion can be difficult depending on whether
the input image is static or a transition frame into a facial expression. The latter
issue is particularly difficult for real-time detection where facial expressions vary
dynamically. The goal of this project was to implement real-time facial emotion
recognition. Using the custom trained VGG network with a face-detector
provided by OpenCV, we successfully implemented an application wherein an
emoji indicating one of six expressions (anger, fear, neutral, happy, sad, surprise)
is superimposed over a user’s face in real time. First, a much larger dataset should
be designed to improve the model’s generality. While it achieved > 90% accuracy
in laboratory conditions (perfect lighting, camera at eye level, subject facing
camera with an exaggerated expression), any deviation from that caused the
accuracy to fall significantly. In particular, any shadow on a subject’s face would
cause an incorrect classification of ‘angry’. Also, fully training a network other
than VGG might yield substantial improvements in computation speed, since
VGG is relatively slow.

Advantages:
• It is able to classify a subject’s emotion reliably at higher frame-rates.
• It achieves higher accuracy in laboratory conditions.
• The emotion detected by the CNN is indicated by the type of emoji
superimposed over the subject’s face.

34
2.3 Proposed Solution

Humans have always had the innate ability to recognize and distinguish
between faces from their emotions where now computers are able to do the same.
This opens up tons of applications. We have developed a system which can detect
about all the expressed emotions using OpenCV. The inbuilt camera that is
present in laptop, mobile devices, etc is accessed to capture the image of the
person. The image can be captured from live-feed or static images from the
camera. The face from the webcam is detected and pre-process the image of the
face using a pre-trained HAAR classifier supplied with OpenCV. The
FisherFaces algorithm is used to detect the face from the processed image by
converting the image into grayscale through average classification. In that sense,
an individual's components do not dominate over the others. The emotion is
detected by taking varied images of different emotions to train the data-set which
can be used to classify emotions expressed based on differences in patterns.
Humans are used to taking in non-verbal cues from facial emotions. This system
can detect four different human emotions - Happy, Sad, Neutral, Angry. The
trained model is capable to detect all the mentioned emotions in real-time and can
be trained for improved accuracy and robust. Based on the emotion extracted with
some basic error reductions, music is suggested to enhance the state of the person
through various genres of music. The system can auto-play music or suggest
preference of music from the infrastructure available on the target computers on
which the software will be run.

35
 CHAPTER 3

3. REQUIREMENT SPECIFICATION

3.1 Hardware Specification

❖ CPU(Desktop)
➢ Windows – x86 and 64 bit architecture
➢ Mac – x64 bit architecture
➢ Linux – x64 bit architecture

❖ RAM
➢ Desktop – Minimum 4GB

❖ Storage
➢ Desktop – Minimum 512 MB

❖ Camera
➢ 1.5 megapixel or higher

3.2 Software Specification

❖ Operating System(OS)
➢ Windows - Windows 7 and Windows 10
➢ MacOS - Sierra 10.12.6+
➢ Linux – Ubuntu 16.04, Ubuntu 18.04 and CentOS 7
❖ Python
➢ Version – 3.6

3.3 Technology Specification

❖ 3.3.1 Python
Python is a general-purpose interpreted, interactive, object-
oriented, and high-level programming language. It was created by Guido van
Rossum during 1985- 1990. Like Perl, Python source code is also available under
the GNU General Public License (GPL). Python is dynamically
typed and garbage-collected. It supports multiple programming paradigms,
including procedural, object-oriented, and functional programming. Python is

36
often described as a "batteries included" language due to its
comprehensive standard library. Python interpreters are available for
many operating systems. A global community of programmers develops and
maintains CPython, an open source reference implementation. A non-profit
organization, the Python Software Foundation, manages and directs resources for
Python and CPython development.

Fig 3.1 Python

❖ 3.3.2 OpenCV-3.4.9

OpenCV (Open source computer vision) is a library of programming

functions mainly aimed at real-time computer vision. Originally developed
by Intel, it was later supported by Willow Garage then Itseez (which was later
acquired by Intel). The library is cross-platform and free for use under the open-
source BSD license. OpenCV supports some models from deep
learning frameworks like TensorFlow, Torch, PyTorch (after converting to an
ONNX model) and Caffe according to a defined list of supported layers. It
promotes OpenVisionCapsules, which is a portable format, compatible with all
other formats. Currently OpenCV supports a wide variety of programming
languages like C++, Python, Java etc and is available on different platforms
including Windows, Linux, OS X, Android, iOS etc. Also, interfaces based on
CUDA and OpenCL are also under active development for high-speed GPU
operations.

37
 Fig 3.2 OpenCV

❖ 3.3.3 Numpy-1.18.1
NumPy is a library for the Python programming language, adding support
for large, multi-dimensional arrays and matrices, along with a large collection
of high-level mathematical functions to operate on these arrays. NumPy targets
the CPython reference implementation of Python, which is a non-
optimizing bytecode interpreter. Mathematical algorithms written for this
version of Python often run much slower than compiled equivalents. NumPy
addresses the slowness problem partly by providing multidimensional arrays and
functions and operators that operate efficiently on arrays, requiring rewriting
some code, mostly inner loops using NumPy. Array in Numpy is a table of
elements (usually numbers), all of the same type, indexed by a tuple of positive
integers. In Numpy, number of dimensions of the array is called rank of the
array.A tuple of integers giving the size of the array along each dimension is
known as shape of the array. An array class in Numpy is called as ndarray.
Elements in Numpy arrays are accessed by using square brackets and can be
initialized by using nested Python Lists.

Fig 3.3 NumPy

38
❖ 3.3.4 Eel-0.12.2
There are several options for making GUI apps in Python, but if you want
to use HTML/JS (in order to use jQueryUI or Bootstrap, for example) then
you generally have to write a lot of boilerplate code to communicate from the
Client (Javascript) side to the Server (Python) side.The closest Python
equivalent to Electron (to my knowledge) is cefpython. It is a bit heavy weight
for what I wanted. Eel is not as fully-fledged as Electron or cefpython - it is
probably not suitable for making full blown applications like Atom - but it is
very suitable for making the GUI equivalent of little utility scripts that you
use internally in your team. For some reason many of the best-in-class number
crunching and maths libraries are in Python (Tensorflow, Numpy, Scipy etc)
but many of the best visualization libraries are in Javascript (D3, THREE.js
etc). Hopefully Eel makes it easy to combine these into simple utility apps for
assisting your development. Install through pip : pip install eel
An Eel application will be split into a frontend consisting of various web-
technology files (.html, .js, .css) and a backend consisting of various Python
scripts.

❖ 3.3.5 HTML5
HTML5 is a markup language used for structuring and presenting content
on the World Wide Web. HTML5 was the fifth and last major version
of HTML that is a World Wide Web Consortium (W3C) recommendation.
The current specification is known as the HTML Living Standard and is
maintained by a consortium of the major browser vendors
(Apple, Google, Mozilla, and Microsoft), the Web Hypertext Application
Technology Working Group (WHATWG). HTML5 includes detailed
processing models to encourage more interoperable implementations; it
extends, improves and rationalizes the markup available for documents, and
introduces markup and application programming interfaces (APIs) for

39
 complex web applications.[6] For the same reasons, HTML5 is also a
candidate for cross-platform mobile applications, because it includes features
designed with low-powered devices in mind. Many new syntactic features are
included. To natively include and handle multimedia and graphical content,
the new <video> , <audio> and <canvas> elements were added, and support
for scalable vector graphics (SVG) content and MathML for mathematical
formulas. To enrich the semantic content of documents, new page structure
elements such as <main>, <section>, <article>, <header>, <footer>, <aside>,
<nav>, and <figure> are added. New attributes are introduced, some elements
and attributes have been removed, and others such as <a>, <cite>, and
<menu> have been changed, redefined, or standardized.

❖ 3.3.6 CSS3
Cascading Style Sheets (CSS) is a style sheet language used for
describing the presentation of a document written in a markup
language like HTML. CSS is a cornerstone technology of the World Wide
Web, alongside HTML and JavaScript. CSS is designed to enable the
separation of presentation and content, including layout, colors,
and fonts. This separation can improve content accessibility, provide more
flexibility and control in the specification of presentation characteristics,
enable multiple web pages to share formatting by specifying the relevant CSS
in a separate .css file, and reduce complexity and repetition in the structural
content. Separation of formatting and content also makes it feasible to present
the same markup page in different styles for different rendering methods, such
as on-screen, in print, by voice (via speech-based browser or screen reader),
and on Braille-based tactile devices. CSS also has rules for alternate
formatting if the content is accessed on a mobile device.

40
❖ 3.3.7 JavaScript
JavaScript, often abbreviated as JS, is a programming language that
conforms to the ECMAScript specification. JavaScript is high-level,
often just-in-time compiled, and multi-paradigm. It has curly-bracket
syntax, dynamic typing, prototype-based object-orientation, and first-class
functions. Alongside HTML and CSS, JavaScript is one of the core
technologies of the World Wide Web. JavaScript enables interactive web
pages and is an essential part of web applications. The vast majority
of websites use it for client-side page behavior and all major web
browsers have a dedicated JavaScript engine to execute it. As a multi-
paradigm language, JavaScript supports event-driven, functional,
and imperative programming styles. It has application programming
interfaces (APIs) for working with text, dates, regular expressions,
standard data structures, and the Document Object Model (DOM). However,
the language itself does not include any input/output (I/O), such
as networking, storage, or graphics facilities, as the host environment
(usually a web browser) provides those APIs. JavaScript engines were
originally used only in web browsers, but they are now embedded in server-
side website deployments, usually via Node.js. They are also embedded in a
variety of applications created with frameworks such
as Electron and Cordova. Although there are similarities between JavaScript
and Java, including language name, syntax, and respective standard libraries,
the two languages are distinct and differ greatly in design.

Fig 3.4 HTML,CSS,JS

41
 CHAPTER 4

4. SYSTEM DESIGN

4.1 System Layout

42
4.2 Use Case Diagram

43
4.3 Activity Diagram

44
4.4 Class Diagram

45
4.5 Data Flow Diagram

46
4.6 Sequence Diagram

47
 CHAPTER 5

5. IMPLEMENTATION
5.1 Project Implementation
Facial Detection — Ability to detect the location of face in any input image
or frame. The output is the bounding box coordinates of the detected faces
Emotion Detection — Classifying the emotion on the face as happy, angry,
sad, neutral, surprise, disgust or fear
Humans have always had the innate ability to recognize and distinguish
between faces from their emotions where now computers are able to do the same.
This opens up tons of applications. We have developed a system which can detect
about all the expressed emotions using OpenCV. The inbuilt camera that is
present in laptop, mobile devices, etc is accessed to capture the image of the
person. The image can be captured from live-feed or static images from the
camera. The face from the webcam is detected and pre-process the image of the
face using a pre-trained HAAR classifier supplied with OpenCV. The
FisherFaces algorithm is used to detect the face from the processed image by
converting the image into grayscale through average classification. In that sense,
an individual's components do not dominate over the others. The emotion is
detected by taking varied images of different emotions to train the data-set which
can be used to classify emotions expressed based on differences in patterns.
Humans are used to taking in non-verbal cues from facial emotions. This system
can detect four different human emotions - Happy, Sad, Neutral, Angry. The
trained model is capable to detect all the mentioned emotions in real-time and
can be trained for improved accuracy and robust. Based on the emotion extracted
with some basic error reductions, music is suggested to enhance the state of the
person through various genres of music. The system can auto-play music or
suggest preference of music from the infrastructure available on the target
computers on which the software will be run.

48
❖ Image Processing:
It consists of three basic steps :
1.) Scanning the image: a raw image is acquired which has to be
processed. It can be expressed in form of pixels as stated above.
The aim of this step is to extract information which is suitable for
computing.

2.) Processing and Enhancing it: -the image is converted into digital
form by using a digitizer which samples and quantizes the input
signals. The rate of sampling should be high for good resolution
and high quantization level for human perception of different
shades using different using gray-scale.

3.) The obtained result describes the property of the image and further
classifies the image.

❖ Facial detection is the first part of our pipeline. We have used the python
library Face Detection that we found easy to install and very accurate in detecting
faces. This library scans the input image and returns the bounding box coordinates
of all detected faces. The FisherFace algorithm of the python Face Detection
library is used to detect the face present in an image by using a classifier to obtain
the feature points of the image to detect and obtain the face accurately with the
objects present.

❖ Face recognition involves three basic steps which include face detection,
face recognition, and face extraction. Any system needs to encapsulate the image
and then manage as well as record the vital features to determine the location of
the face. For the recognition of the captured image it keeps records of various
features such as skin color, skin tone etc. Face recognition takes an image from

49
a video or a camera as input and outputs the identified image topic. Facial
features may include regions in the face, variations in the face structure, face cuts
and angles that have been formatted and styled. Face extraction includes
grabbing of the features from camera. Face detection includes removal of the
background and focusing on the foreground eliminating any other elements apart
from the face region.

❖ The classifier will work best if the training and classification images are
all of the same size and have (almost) only a face on them (no clutter). We need
to find the face on each image, convert to grayscale, crop it and save the image
to the dataset. We can use a HAAR filter from OpenCV to automate face finding.
Actually, OpenCV provides 4 pre-trained classifiers, so to be sure we detect as
many faces as possible let’s use all of them in sequence, and abort the face search
once we have found one. Get them from the OpenCV directory or from here and
extract to the same file you have your python files.
Create another folder called “dataset”, and in it create subfolders for each
emotion (“neutral”, “anger”, etc.).

❖ The dataset has been organised and is ready to be recognized, but first we
need to actually teach the classifier what certain emotions look like. The usual
approach is to split the complete dataset into a training set and a classification
set. We use the training set to teach the classifier to recognize the to-be-predicted
labels, and use the classification set to estimate the classifier performance.

❖ Based on the real-time extraction of facial expressions as well as extracting

features from the face to classify into a specific emotion which can be used to
generate a playlist automatically such that the computation cost is relatively low
and auto-play of suggestions.

50
5.2 Modules
5.2.1. Camera Visualization and Capture:
There is a rapidly growing demand for using smart cameras for various
biometric applications in surveillance. Although having a small form-factor,
most of these applications demand huge processing performance for real-time
processing. Face recognition is one of those applications. In this paper we show
that we can run face detection in real-time by implementing the algorithm for
face detection and capture from the image obtained. The inbuilt camera that is
present in laptop, mobile devices, etc. is accessed to capture the image of the
person. The image can be captured from live-feed or static images by enabling
the camera.

Fig 5.1 Camera visualization

5.2.2. Face Detection:

A face consists of some features on it which play an important role in the
detection of the emotions on it. The emotion recognition system is divided into 3
stages: face detection, feature extraction, and emotion classification. We have
encoded our eight emotions in the datasets as {0= happy, 1=sadness, 2=fear,
3=anger, 4=surprise, 5=disgust, 6=contempt, 7=neutral}. Initially, the faces are
detected in all the sequences using the FisherFace Algorithm. Each image
sequence consists of the development of an emotional expression, starting from
51
a neutral face and ending with some particular emotion. So, from each image
sequence, our focus is to extract images with an emotional expression in the
sequence. Through the feature detection by using the Principal Component
Analysis and Linear Discriminant Analysis on the data, the features can be
extracted accurately with minimal noise reduction. It usually doesn't focus on the
features that discriminate one individual from another. Instead, it concentrates on
the ones that represent all the faces of all the people in the data, as a whole.

Fig 5.2 Face detection

5.2.3. Extracting Faces :

The classifier will work properly if the images contain only the faces so
the images were processed accordingly for the detection of the faces and then
were converted to grayscale and were cropped and were stored in some specific
folder. We have used a HAAR filter from OpenCV for automatic detection of
faces. As OpenCV contains 4 pre-trained classifiers, so it is better we detect as
many faces as possible. Through object detection of HAAR classifier for each
feature, it finds the best threshold which will classify the faces to positive and
negative. But obviously, there will be errors or misclassifications. So it selects
the features with minimum error rate, which means they are the features that best
classifies the face and non-face images.

52
 Fig 5.3 Extracting and classifying faces

5.2.4. Extracting features from the faces :

In the feature extraction stage, the faces detected in the previous stage are
further processed for identification of eye, eyebrows, nose, corner of face and
mouth regions. Initially, the likely Y coordinates of the eyes were identified with
the use of the horizontal projection. Then the areas around the y coordinates were
processed to identify the exact regions of the features. Finally, a corner point
detection algorithm was used to obtain the required corner points from the feature
regions. This will result in a lot of dots on the faces in the webcam video outlining
the shape and all the “moveable parts”. Features are small information that is
used to describe the object or object state that we are trying to classify into
different categories. The facial landmarks from the image material tell about the
position of all the “moving parts” of the depicted face, the things we need to
express an emotion.

53
 Fig 5.4 Feature extraction
5.2.5. Emotion Recognition:
The dataset which has been organized and is ready to be recognized, but
initially, we need to actually train the classifier what particular emotions look
like. The approach we have used is to split the complete dataset into a training
set and a classification set. We use the training set to teach the classifier to
recognize the labels to be predicted, and used the classification set to estimate
the performance of the classifier. After creating the training and classification
set, we randomly sample and train on 80% of the data and classify the remaining
20%, and repeat the process 20 times. After training the fisher face classifier the
emotions were predicted.

Fig 5.5 Emotion recognition

54
5.2.6 Music Suggestion:
Music plays a very important role in human's daily life and in the modern
advanced technologies. Usually, the user has to face the task of manually
browsing through the playlist of songs to select. Here we are proposing an
efficient and accurate model, that would generate a playlist based on current
emotional state and behaviour of the user. Existing methods for automating the
playlist generation process are computationally slow, less accurate and
sometimes even require use of additional hardware like EEG or sensors. Speech
is the most ancient and natural way of expressing feelings, emotions and mood
and its and its processing requires high computational, time, and cost. This
proposed system based on real-time extraction of facial expressions to classify
into a specific emotion that will generate a playlist automatically such that the
computation cost is relatively low.

Fig 5.6 Music Suggestion

5.3 Algorithm Description

5.3.1 FisherFace Algorithm
Fisherface is one of the popular algorithms used in face recognition, and
is widely believed to be superior to other techniques, such as eigenface because
of the effort to maximize the separation between classes in the training process.
55
The purpose of this research is to establish a program of face recognition
application using fisherface method by utilizing GUI applications and databases
that are used in the form of a Papuan facial image. Image recognition using
fisherface method is based on the reduction of face space dimension using
Principal Component Analysis (PCA) method, then apply Fisher's Linear
Discriminant (FDL) method or also known as Linear Discriminant Analysis
(LDA) method to obtain feature of image characteristic. The algorithm used in
the process for image recognition is fisherfaces algorithm while for identification
or matching face image using minimum euclidean. The method used in this study
is literature study that is studying and reviewing various books or literature
related to mathematical concepts that underlies the formation of fisherface
algorithm to recognize the image of a person's face which is then applied in
programming language, especially programming language Matlab7.10. While in
the process of preprocessing used Adobe Photoshop CS4 application program,
its goal is to make the face image to be uniform in terms of size and format so
that the image is ready to be used by the system. The results show that for image
recognition where the image of testing is the same as the training image, the
percentage of program success is 100%, while for 73 facial test images with
various expressions and various positions, 70 faces are recognized correctly and
3 faces are recognized incorrectly, so the percentage of success is 93%.

5.3.2 Haar Classifier

Object Detection using Haar feature-based cascade classifiers is an
effective object detection method proposed by Paul Viola and Michael Jones in
their paper, “Rapid Object Detection using a Boosted Cascade of Simple
Features” in 2001. It is a machine learning based approach where a cascade
function is trained from a lot of positive and negative images. It is then used to
detect objects in other images. Initially, the algorithm needs a lot of positive
images (images of faces) and negative images (images without faces) to train the

56
classifier. Then we need to extract features from it. For this, haar features of fig
5.1 are used. They are just like our convolutional kernel. Each feature is a single
value obtained by subtracting sum of pixels under white rectangle from sum of
pixels under black rectangle.

Fig 5.7 HAAR Feature

Now all possible sizes and locations of each kernel is used to calculate
plenty of features. For each feature calculation, we need to find sum of pixels
under white and black rectangles. To solve this, they introduced the integral
images. It simplifies calculation of sum of pixels, how large may be the number
of pixels, to an operation involving just four pixels.
For each feature, it finds the best threshold which will classify the faces to
positive and negative. But obviously, there will be errors or misclassifications.
We select the features with minimum error rate, which means they are the
features that best classifies the face and non-face images. After each
classification, weights of misclassified images are increased. Then again same
process is done. New error rates are calculated. Also new weights. The process
is continued until required accuracy or error rate is achieved or required number
of features are found.

57
 CHAPTER 6

6. TESTING
6.1 Testing
In this project, different types of testing in different environments which are
given as follows:
6.1.1 Alpha Testing

• It is the most common type of testing used in the Software industry. The
objective of this testing is to identify all possible issues or defects before
releasing it into the market or to the user.
• Alpha Testing is carried out at the end of the software development phase
but before the Beta Testing. Still, minor design changes may be made as a
result of such testing.
• Alpha Testing is conducted at the developer’s site. In-house virtual user
environment can be created for this type of testing.

6.1.2 Beta Testing

• Beta Testing is a formal type of Software Testing which is carried out by

the customer. It is performed in the Real Environment before releasing the
product to the market for the actual end-users.
• Beta Testing is carried out to ensure that there are no major failures in the
software or product and it satisfies the business requirements from an end-
user perspective. Beta Testing is successful when the customer accepts the
software.
• Usually, this testing is typically done by end-users or others. It is the final
testing done before releasing an application for commercial purpose.
Usually, the Beta version of the software or product released is limited to
a certain number of users in a specific area.

58
 • So end-user actually uses the software and shares the feedback to the
company. Company then takes necessary action before releasing the
software to the worldwide.

6.1.3 Blackbox Testing

• Internal system design is not considered in this type of testing. Tests are
based on the requirements and functionality
• BLACK BOX TESTING is defined as a testing technique in which
functionality of the Application Under Test (AUT) is tested without
looking at the internal code structure, implementation details and
knowledge of internal paths of the software. This type of testing is based
entirely on software requirements and specifications. In Black Box Testing
we just focus on inputs and output of the software system without bothering
about internal knowledge of the software.

6.1.4 Whitebox Testing

• Input involves different types of requirements, functional specifications,

detailed designing of documents, proper source code and security
specifications. This is the preparation stage of white-box testing to lay out
all of the basic information.
• Processing involves performing risk analysis to guide whole testing
process, proper test plan, execute test cases and communicate results. This
is the phase of building test cases to make sure they thoroughly test the
application the given results are recorded accordingly.
• Output involves preparing final report that encompasses all of the above
preparations and results.

59
6.1.5 Integration Testing

• Testing of all integrated modules to verify the combined functionality after

integration is termed as Integration Testing.
• Modules are typically code modules, individual applications, client and
server applications on a network, etc. This type of testing is especially
relevant to client/server and distributed systems.
• INTEGRATION TESTING is a level of software testing where individual
units are combined and tested as a group. The purpose of this level of
testing is to expose faults in the interaction between integrated units. Test
drivers and test stubs are used to assist in Integration Testing.
• Testing performed to expose defects in the interfaces and in the interactions
between integrated components or systems

6.1.6 Performance Testing

• This term is often used interchangeably with ‘stress' and ‘load' testing.
Performance Testing is done to check whether the system meets the
performance requirements. Different performance and load tools are used
to do this testing.

6.1.7 Recovery Testing

• It is a type of testing which validates how well the application or system

recovers from crashes or disasters.
• Recovery Testing determines if the system is able to continue the operation
after a disaster. Assume that application is receiving data through the
network cable and suddenly that network cable has been unplugged.
• In software testing, recovery testing is the activity of testing how well an
application is able to recover from crashes, hardware failures and other
similar problems. Recovery testing is the forced failure of the software in
a variety of ways to verify that recovery is properly performed.

60
 CHAPTER 7

7. CONCLUSION AND FUTURE WORKS

7.1 Conclusion
The main purpose is to detect various emotions in a given sample image.
The most challenging part in this task was to determine the exact emotion when
two emotions look quite similar, for e.g. “Disgust” being classified as “Sadness”,
“Surprise” like “Happy” and so on. Now for four different categories, the result
was approximately 70% accurate which is quite well actually as our classifier
learned quite a bit. So, we must see how can we increase its efficiency and
accuracy. If we look at our emotions list we can find out that we have only limited
number of examples for “sad”, “fear” and “happy”. By increasing the number of
images for these emotions we can certainly increase optimization, or if we no
longer consider these emotions in the list then optimization can be increased more
than 80%. Artificial Intelligence can be used to solve intriguing tasks such as
emotion detection, although this task was quite convolute even more when using
a great number of images. We humans also sometimes make a mistake while
recognizing someone’s emotion so is our program. Using HAAR Cascades to
detect eyes and mouth and identify all kinds of emotion through the machine
learning algorithm method, the combination of efficiency and accuracy is
achieved. By detecting the emotions from an image, the relative genre of music
is played to the user on run-time. By providing the modes such as Queue mode
and Emotion mode where the user can choose to detect the emotion and play
music or the system can relatively act as a music player of the choice of music to
be played by the user.

61
7.2 Future Works
In this project, we have presented the fully automatic recognition of facial
emotions using the computer vision and machine learning algorithms which
classify these eight different emotions. We tried many algorithms for the
classification but the best which came out of the results was the support vectors
machines with the accuracy of around 94.1%. Our results imply that user
independent, fully automatic real-time coding of facial expressions in the
continuous video stream is an achievable goal with present power of the
computer, at least for applications in which frontal views can be assumed using
the webcam. This machine learning based system for the emotion recognition can
be extended to the deep learning system using the Convolutional Neural networks
which will have many layers and the chances of getting much higher accuracy is
there around 99.5%. This project can be extended in which it will detect as many
emotions of different people in one frame in the real-time videos. Emotion
recognition is going to be very useful in the near future in the research field of
robotics and artificial Intelligence for example if a robot can sense the sentiment
of any human and that robot can act accordingly without any intervention of any
other humans. This automatic machine learning system for emotion recognition
can also be extended with the detection of mixed emotions other than these eight
universal emotions.

Emotion detection is an inseparable part of computer vision. Loads of tasks

and processes can be performed if one can become aware about the intricacies
and endless possibilities offered under the field of emotion detection. Some
common and widely used applications of emotion detection are:
❖ App and product development
Emotion recognition can play a huge role in optimizing various software
engineering processes which comprises of testing of ease with which a product
can be used. It’s a long-established fact that level of comfort with different

62
software products depends hugely upon human emotions. A products overall look
and feel can also alter human feelings which in turn makes the person buy the
product or not. Thus, researching about different emotional states of a human
body and how it is influenced by usage of different products is a matter of prime
importance for related industry professionals.

❖ Improved learning practices

Evidence suggests that part of emotional states vouches for better learning
practices while the other part try to suppress them. The difference between the
two groups possessing different emotional states is not so common to find. For
example, positive emotional state is thought to be bad for learning purposes while
slightly negative emotional state fosters analytical thinking and is also
appropriate for carrying out critical tasks.

❖ Improvised web development

With the mammoth scale at which the internet is expanding, service
providers are interested in collecting tons and tons of data which can be extracted
from the users. Correspondingly, all the content and advertisements are played
based on the users’ profile. Subsequently, adding intricate details about the
different human emotions can provide much more precise behavioral models of
different types of users.

❖ Immersive gaming
Video games constitute a large chunk of entertainment industry. Thus, in
order to make these games much more intensive and intrusive, video game
creators base their research on different types of human emotions commonly
found. In order to allure more and more players, video games are made in such a
way that they incorporate human emotions naturally into their game play.
63
❖ Safe Driving
Being able to recognize those actions for drivers or passengers can help
smart cars to better understand what’s going on inside the car. Then based on
those observations, control can pass between the smart car and driver - or triggers
can be put in place, where an alarm goes off to the driver if he becomes drowsy
before he drifts from his lane. These features not only can enhance the road safety
but it will enhance the driving experience of people for next generations of family
cars and public transportation.

❖ Social Media
Social media like Twitter and Facebook is full of emotions, feelings and
opinions of people all over the world. However, analysing and classifying text on
the basis of emotions is a big challenge and can be considered as an advanced
form of Sentiment Analysis. Analysing the emotion through camera images is
also a big challenge and can be expanded for future enhancement in detecting
emotions while texting.

64
 APPENDIX

APPENDIX – A: SAMPLE CODE

Capture.py

def crop(clahe_image, face):

for (x, y, w, h) in face:
faceslice = clahe_image[y:y+h, x:x+w]
faceslice = cv2.resize(faceslice, (350, 350))
facedict["face%s" %(len(facedict)+1)]=faceslice
return faceslice

def grab_face():
ret, frame = video_capture.read()
cv2.imshow("Video", frame)
cv2.imwrite('test.jpg', frame)
cv2.waitKey()
cv2.imwrite("images/main%s.jpg" %count, frame)
gray = cv2.imread('test.jpg',0)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
clahe_image = clahe.apply(gray)
return clahe_image

def detect_face():
clahe_image=grab_face()
face = facecascade.detectMultiScale(clahe_image, scaleFactor=1.1, minNeigh
bors=15, minSize=(10, 10), flags=cv2.CASCADE_SCALE_IMAGE)
if len(face) >= 1:
faceslice=crop(clahe_image, face)
#return faceslice
else:
print("No/Multiple faces detected!!, passing over the frame")

def save_face(emotion):
print("\n\nLook "+emotion+" untill the timer expires and keep the same emo
tion for some time.")
#winsound.Beep(frequency, duration)
print('\a')

for i in range(0, 5):

print(5-i)
time.sleep(1)

while len(facedict.keys())<16:
detect_face()

65
 for i in facedict.keys():
path, dirs, files = next(os.walk("dataset/%s" %emotion))
file_count = len(files)+1
cv2.imwrite("dataset/%s/%s.jpg" %(emotion, (file_count)), facedict[i])
facedict.clear()

def update_model(emotions):
print("Update mode for model is ready")
checkForFolders(emotions)

for i in range(0, len(emotions)):

save_face(emotions[i])
print("Collected the images, looking nice! Now updating the model...")
Update_Model.update(emotions)
print("Model train successful!!")
def identify_emotions():
prediction = []
confidence = []

for i in facedict.keys():
pred, conf = fishface.predict(facedict[i])
cv2.imwrite("images/%s.jpg" %i, facedict[i])
prediction.append(pred)
confidence.append(conf)
output = emotions[max(set(prediction), key=prediction.count)]
print("You seem to be %s" %output)
facedict.clear()
return output;

Display.py

def show_frame():
_, frame = cap.read()
frame = cv2.flip(frame, 1)
cv2image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA)
img = Image.fromarray(cv2image)
imgtk = ImageTk.PhotoImage(image=img)
lmain.imgtk = imgtk
lmain.configure(image=imgtk)
lmain.after(10, show_frame)

66
Update_Model.py

def update(emotions):
run_recognizer(emotions)
print("Saving model...")
fishface.save("model.xml")
print("Model saved!!")

def make_sets(emotions):
training_data=[]
training_label=[]

for emotion in emotions:

training=training=sorted(glob.glob("dataset/%s/*" %emotion))
for item in training:
image=cv2.imread(item)
gray=cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
training_data.append(gray)
training_label.append(emotions.index(emotion))
return training_data, training_label

def run_recognizer(emotions):
training_data, training_label=make_sets(emotions)
print("Training model...")
print("The size of the dataset is "+str(len(training_data))+" images")
fishface.train(training_data, np.asarray(training_label))

model.xml

<?xml version="1.0"?>
<opencv_storage>
<opencv_fisherfaces>
<threshold>1.7976931348623157e+308</threshold>
<num_components>3</num_components>
<mean type_id="opencv-matrix">
<rows>1</rows>
<cols>122500</cols>
<dt>d</dt>
<data>
1.2516740088105728e+02 1.2442511013215861e+02
1.2354405286343614e+02 1.2265859030837005e+02
1.2179735682819384e+02 1.2093171806167402e+02
1.1991850220264318e+02 1.1878414096916300e+02
1.1750881057268724e+02 1.1566960352422909e+02
1.1364977973568283e+02 1.1172246696035243e+02
1.0980176211453745e+02 1.0812555066079297e+02
1.0634801762114539e+02 1.0465638766519825e+02

67
 1.0307709251101322e+02 1.0179074889867843e+02
1.0057929515418503e+02 9.9590308370044056e+01
9.8638766519823804e+01 9.7905286343612346e+01
9.7017621145374463e+01 9.6066079295154196e+01
</data>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4271991251107886e+02 -3.3896220482578474e+02
-3.6573086542596559e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4273866263198272e+02 -3.3899534241133790e+02
-3.6619194488891287e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4273509221926548e+02 -3.3891537879463488e+02
-3.6688125336462372e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4274420341218968e+02 -3.3964100091790311e+02
-3.6714775703502420e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4275364862484702e+02 -3.3999607975979859e+02
-3.6699728036397511e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4279189729893000e+02 -3.4136939981878021e+02
-3.6957102621035199e+02</data></_>
<_ type_id="opencv-matrix">

68
 <rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4276148305262836e+02 -3.3933532551921923e+02
-3.6923730039222505e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4275579744005893e+02 -3.3982389303019039e+02
-3.6665379210691754e+02</data></_>
<_ type_id="opencv-matrix">
<rows>1</rows>
<cols>3</cols>
<dt>d</dt>
<data>
-4.4276253030710495e+02 -3.4151185469706076e+02
-3.6853576120018147e+02</data></_></projections>
<labels type_id="opencv-matrix">
<rows>454</rows>
<cols>1</cols>
<dt>i</dt>
<data>
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3</data></labels>
<labelsInfo>
</labelsInfo></opencv_fisherfaces>
</opencv_storage>

69
Code.js

function addq(elem) {
console.log(elem.id);
var x = elem.id.charAt(1);
if (!songrun) {
var z = songs[x][0];
document.getElementById("sname").innerHTML = sname[x];
document.getElementById("sel").src = z;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + son
gs[x][3] + "')";
songrun = true;
return;
}
if (bool[x] == true)
return;

bool[x] = true;
var l = document.createElement("label");
l.id = "e" + eqc;
l.name = x;
l.innerHTML = sname[x] + "<br>";
//var text=document.createTextNode(sname[x]+"<br>");
//l.appendChild(text);
document.getElementById("queue").appendChild(l);
eqc = eqc + 1;
}

function nextsong() {
if (sqc == eqc) {
alert("Queue is empty.");
return;
}
var elem = document.getElementById("e" + sqc);
var xa = elem.name;
var pa = songs[xa][0];
bool[xa] = false;
document.getElementById("sname").innerHTML = sname[xa];
document.getElementById("sel").src = pa;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + songs[x
a][3] + "')";

songrun = true;
document.getElementById("queue").removeChild(elem);

70
 sqc = sqc + 1;

function next_in_Q() {
songrun = false;
if (sqc == eqc) {
alert("Queue is empty.");
return;
}
var elem = document.getElementById("e" + sqc);
var xa = elem.name;
var pa = songs[xa][0];
document.getElementById("sname").innerHTML = sname[xa];
document.getElementById("sel").src = pa;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + songs[x
a][3] + "')";
songrun = true;
document.getElementById("queue").removeChild(elem);
sqc = sqc + 1;
}

function rand_play() {
var index = Math.random() * path.length;
index = parseInt(index);
var pa = songs[index][0];
document.getElementById("sname").innerHTML = sname[index];
document.getElementById("sel").src = pa;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + songs[i
ndex][3] + "')";
songrun = true;

function moody(val) {
var index = Math.random() * mood[val].length;
index = parseInt(index);
var pa = songs[mood[val][index] - 1][0];
document.getElementById("sname").innerHTML = sname[mood[val][index] - 1];
document.getElementById("sel").src = pa;
document.getElementById("main_slider").load();
document.getElementById("main_slider").play();
document.getElementById("emoji").style.backgroundImage = "url('" + songs[m
ood[val][index] - 1][3] + "')";

71
 songrun = true;
}

APPENDIX – B: SAMPLE OUTPUT

72
REFERENCE

[1] M. Chen, Y. Zhang, Y. Li, S. Mao, and V. C. M. Leung, ‘‘EMC:

Emotionaware mobile cloud computing in 5G,’’ IEEE Netw., vol. 29, no. 2,
pp. 32–38, Mar./Apr. 2015.
[2] C. Shan, S. Gong, and P. W. McOwan, ‘‘Facial expression recognition based
on local binary patterns: A comprehensive study,’’ Image Vis. Comput., vol.
27, no. 6, pp. 803–816, 2009.
[3] K. K. Rachuri, M. Musolesi, C. Mascolo, P. J. Rentfrow, C. Longworth, and
A. Aucinas, ‘‘EmotionSense: A mobile phones based adaptive platform for
experimental social psychology research,’’ in Proc. 12th ACM Int. Conf.
Ubiquitous Comput. (UbiComp), Copenhagen, Denmark, Sep. 2010, pp. 281–
290.
[4] J. W. Ragsdale, R. Van Deusen, D. Rubio, and C. Spagnoletti, ‘‘Recognizing
patients emotions: Teaching health care providers to interpret facial
expressions,’’ Acad. Med., vol. 91, no. 9, pp. 1270–1275, Sep. 2016.
[5] J. C. Castillo et al., ‘‘Software architecture for smart emotion recognition
and regulation of the ageing adult,’’ Cognit. Comput., vol. 8, no. 2, pp. 357–
367, Apr. 2016.
[6] R. Nielek and A. Wierzbicki, ‘‘Emotion aware mobile application,’’ in
Computational Collective Intelligence. Technologies and Applications

73
 (Lecture Notes in Computer Science), Berlin, Heidelberg, Springer: vol. 6422.
2010, pp. 122–131.
[7] Shivam Gupta, ‘‘Facial emotion recognition in real-time and static images,’’,
2018.

[8] T. Jabid, M. H. Kabir, and O. Chae, ‘‘Robust facial expression recognition

based on local directional pattern,’’ ETRI J., vol. 32, no. 5, pp. 784–
794, 2010.
[9] X. Zhao and S. Zhang, ‘‘Facial expression recognition based on local binary
patterns and kernel discriminant isomap,’’ Sensors, vol. 11, no. 10, pp. 9573–
9588, 2011.
[10] Y. Rahulamathavan, R. C.-W. Phan, J. A. Chambers, and D. J. Parish,
‘‘Facial expression recognition in the encrypted domain based on local Fisher
discriminant analysis,’’ IEEE Trans. Affect. Comput., vol. 4, no. 1, pp. 83–
92, Jan./Mar. 2013.
[11] L. Chao, J. Tao, M. Yang, Y. Li, and Z. Wen, ‘‘Multi-scale temporal
modeling for dimensional emotion recognition in video,’’ in Proc. 4th Int.
Workshop Audio/Vis. Emotion Challenge (AVEC), Orlando, FL, USA, Nov.
2014, pp. 11–18.
[12] G. Muhammad and M. F. Alhamid, ‘‘User emotion recognition from a
larger pool of social network data using active learning,’’ Multimedia Tools
Appl., accessed on Sep. 19, 2016, doi: 10.1007/s11042-016-3912-2.
[13] Kim & Joo, Young Hoon & Bae Park, Jin. (2015). Emotion Detection
Algorithm Using Frontal Face Image.
[14] Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min
Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann and Shrikanth
Narayanan (2004). Analysis of Emotion Recognition using Facial
Expressions, Speech and Multimodal Information.
[15] M. Shamim Hossain , Ghulam Muhammad (2017), An Emotion
Recognition System for Mobile Applications.

74