CISC 6080 Capstone Project in Data Science

Yepu Wang
Abstract: With the rapid development of science and technology, face recognition expression
recognition is applied in many intelligent fields. The scientific basis for rapid and accurate
identification of a person's identity through computer operation is the uniqueness of each
person's facial expression. Some scientists pointed out that there are seven kinds of facial
expressions: happy, sad, angry, disgusted 【 disˈɡəstəd 】 , surprised and afraid neutral 。
Facial expression recognition technology can better realize the communication between
intelligent machine and people. The realization of computer understanding and recognition of
facial expression can greatly change the relationship between human and computer, so that the
computer can better serve human beings, so as to achieve better human-computer interaction.In
application, facial expression recognition technology has great potential application value in
psychology[saɪˈkɑːlədʒi] , intelligent robot, intelligent monitoring, virtual reality and synthetic [sɪn
ˈθetɪk] animation.For example, in the retail industry, we can identify the customer's expression to
obtain his preference for goods, or we can intelligently recommend appropriate advertisements
through expression recognition to achieve precision marketing. For my project, I plan to design a
programmer based on CNN to do facial expression recognition. Similar to face recognition in
application, facial expression recognition also needs to go through these three steps: face image
acquisition[ˌækwɪˈzɪʃn] and preprocessing, expression feature extraction and expression
classification. The selection of convolution kernels and the choosing of hyperparameters[ pə
ˈramədər] are very important to the recognition accuracy. The main expected outcome of this
project is to recognize facial expression with high accuracy.

I plan to use JAFFE dataset and CK+ dataset.

Keras(kai ruosi)
The face region is cut out to remove the redundant[rɪˈdʌndənt] interference[ˌɪntərˈfɪrəns] , and
then the face region is scaled to a 48 * 48 image [ˈɪmɪdʒ] , which is stored in. CSV format

data set
For the already ffinish work ,
Actually now I have already written [ˈrɪtn] a code to do fac for JAFFEdataset
I dived the dataset 100 epoch [ˈepək]

[48×48]⇀conv2d ［ 3×3 ］ ×128⇀pooling⇀conv2d[5×5]×64⇀pooling⇀conv2d[3×3]×32⇀pooling

⇀fc[6×6×32]×200⇀dropout⇀softmax[200×7]
micro-expression [ˈmaɪkroʊ] recognition ,

Project Description:
1. Main algorithm for this project
The main algorithm for this project is convolutional neural network. CNN is a class of deep
neural network, most commonly applied to analyze visual imagery. A convolutional neural
network consists of an input layer, hidden layers and an output layer. In any feed-forward neural
network, any middle layers are called hidden because their inputs and outputs are masked by the
activation function and final convolution. In a convolutional neural network, the hidden layers
include layers that perform convolutions. Typically this includes a layer that performs a dot
product of the convolution kernel with the layer's input matrix. This product is usually the
Frobenius inner product, and its activation function is commonly ReLU. As the convolution kernel
slides along the input matrix for the layer, the convolution operation generates a feature map,
which in turn contributes to the input of the next layer. This is followed by other layers such as
pooling layers, fully connected layers, and normalization layers.
In the project, different convolution kernels and hyperparameters will be tried to make compare
and improve the accuracy. (Number of filters, Filter size, Pooling type and size and so on.)

2. Data sets for this project

Number
Resolutio
Databas Number of of Gray/Co Ground
Facial expression n, Frame Type
e Subjects images/v lor truth
rate
ideos
593 Facial
image expression
Extende neutral, sadness, sequence labels and
Posed;
d Cohn- surprise, s (327 FACS (AU
Mostly spontan
Kanade happiness, fear, 123 sequence 640* 490 label for
gray eous
Dataset anger, contempt s having final frame
smiles
(CK+)[5] and disgust discrete in each
emotion image
labels) sequence)
Japanese
Female neutral, sadness,
Facial surprise, Facial
213 static
Expressi happiness, fear, 10 Gray 256* 256 expression Posed
images
ons anger, and label
(JAFFE) disgust
[6]

it would be interesting to see how race and/or gender impacts the ability to predict the
emotional state. That could be assessed by building a model only using data for one
race/gender, or by using a mixed dataset and then seeing how it performs on individuals, but
then aggregate the performance results based on race and gender.

3. The main data mining/data science task.

The main data science task is machine learning/deep learning. The main algorithm for this
project is CNN.

4. Learning tools/toolkits for this project.

Python, TensorFlow,Keras. OpenVC may will also be used, but not sure.

5. Steps for project.

Facial expression recognition also needs to go through these three steps: face
image acquisition ˌakwəˈziSH(ə)n and preprocessing, expression feature extraction
and expression classification. Then optmization will be done, like making
adjustment for hyperparameters. Finally, I will write my paper for this project.
Timeline:
6/20-7/15 Finishing my code which can achieve expression classification.
7/15-7/20 Doing optmization to improve accuracy.
7/20-7/25 Writing paper and making PPT.

6. How will you assess/evaluate your results and how will you determine the utility of these
results?
The result can be assessed according to the accuracy of facial expression recognition.

7. Are there other papers/results that you can compare them against, or is this a novel problem.
Yes, there are many papers/ results that I can compare them against. All articles / projects about
image recognition, face recognition and facial expression recognition can be referred to.

8.A short rationale for why the project is sufficiently complex for a capstone project.

CNN is a very important algorithm in data science. Facial expression recognition is widely used in
reality. Coding, model design and parameter adjustment all take a lot of time.

REFERENCES
[1] Stephen Balaban Deep learning and face recognition: the state of the art 2015
[3]WANG Xuejiao ， ZHI Min. Human motion recognition based on deformable convolutional
neural network［J］.Computer Engineering and Science，2021，49（1）：105-111
[4]SIMONYAN K ， ZISSERMAN A. Very deep convolutional networks for large-scale image
recognition ［ EB/OL ］ ［ . 2019-10-17 ］ . https ： //www. researchgate.
net/publication/265385906_
Very_Deep_Convolutional_Networks_for_Large-Scale_Image_Recognition
[5] Kim, B. K., Roh, J., Dong, S. Y., & Lee, S. Y. (2016). Hierarchical committee of deep
convolutional neural networks for robust facial expression recognition.Journal on Multimodal
User Interfaces,10(2), 1-17.