VISVESVARAYA TECHNOLOGICAL UNIVERSITY

“Jnana Sangama”, Belagavi-590018

A
Project Work Phase – II
Report On
“Facial Image Captioning using DNN”
SUBMITTED IN PARTIAL FULFILLMENT FOR 8TH SEMESTER
BACHELOR OF ENGINEERING
IN
COMPUTER SCIENCE AND ENGINEERING
SUBMITTED BY

Animesh Anand (1JB20CS011)

Biswadeep Bhagat (1JB20CS022)
Darshan Raje Urs (1JB21CS404)
Harish Kumar K (1JB21CS407)

Under the Guidance of

Mrs. Vijayalakshmi B
Assistant professor

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

SJB INSTITUTE OF TECHNOLOGY
No.67, BGS Health & Education City, Dr.Vishnuvardhan Rd, Kengeri, Bengaluru, Karnataka 560060
Approved by AICTE - New Delhi, Accredited by NAAC A+, Accredited by NBA

2023 - 2024
 || Jai Sri Gurudev ||
Sri Adichunchanagiri Shikshana Trust ®
SJB INSTITUTE OF TECHNOLOGY
No.67, BGS Health & Education City, Dr.Vishnuvardhan Rd, Kengeri, Bengaluru, Karnataka 560060
Approved by AICTE - New Delhi, Accredited by NAAC A+, Accredited by NBA

Department of Computer Science and Engineering

CERTIFICATE

Certified that the Project Work Phase - II entitled “Facial Image Captioning using DNN”
carried out by Mr. Animesh Anand bearing USN 1JB20CS011, Mr. Biswadeep Bhagat bearing USN
1JB20CS022, Mr. Darshan Raje Urs bearing USN 1J B21CS404 and Mr. Harish Kumar K bearing
USN 1JB21CS407 are bonafide students of SJB Institute of Technology in partial fulfilment
for 7th semester of BACHELOR OF ENGINEERING in COMPUTER SCIENCE AND
ENGINEERING of the Visvesvaraya Technological University, Belagavi during the academic year
2023-24. It is certified that all corrections/suggestions indicated for Internal Assessment have been
incorporated in the Report deposited in the Departmental library. The project report has been approved as it
satisfies the academic requirements in respect of Project work phase-1 prescribed for the said Degree.

Signature of Guide Signature of HOD

Mrs. Vijayalakshmi B Dr. Krishna A N

Assistant professor Professor & Head
Dept. of CSE, SJBIT Dept. of CSE, SJBIT
 ACKNOWLEDGEMENT
We would like to express our profound grateful to His Divine Soul Jagadguru Padmabhushan
Sri Sri Sri Dr. Balagangadharanatha Mahaswamiji and His Holiness Jagadguru Sri Sri Sri
Dr. Nirmalanandanatha Mahaswamiji for providing us an opportunity to complete our
academics in this esteemed institution.

We would also like to express our profound thanks to Revered Sri Sri Dr. Prakashnath
Swamiji, Managing Director, SJB Institute of Technology, for his continuous support in
providing amenities to carry out this Project Phase - II in this admired institution.

We express our gratitude to Dr. K. V. Mahendra Prashanth, Principal, SJB Institute of

Technology, for providing us an excellent facilities and academic ambience; which have helped
us in satisfactory completion of Project Phase - II work.

We extend our sincere thanks to Dr. Krishan A N, Head of the Department, Computer Science
and Engineering for providing us an invaluable support throughout the period of our Project
Phase - I work.

We wish to express our heartfelt gratitude to our guide Mrs. Vijayalakshmi B, for her valuable
guidance, suggestions and cheerful encouragement during the entire period of our Project Phase -
I work. We express our truthful thanks to Dr. Veena H N, Project Coordinator, Department of
CSE, for her valuable support.

Finally, we take this opportunity to extend our earnest gratitude and respect to our parents,
Teaching & Non-teaching staffs of the department, the library staff and all our friends, who have
directly or indirectly supported us during the period of our Project Phase - II work.

Regards,
Animesh Anand (1JB20CS011)
Biswadeep Bhagat (1JB20CS022)
Darshan Raje Urs (1JB21CS404)
Harish Kumar K (1JB21CS407)
 ABSTRACT

In recent years, the surge in facial recognition studies has been driven by its
pivotal role in enhancing human-computer interaction. With the advent of challenging
datasets, the integration of deep learning methodologies has become imperative. In this
research endeavor, we present a Python script for real-time emotion, age, and gender
detection using computer vision technologies. Leveraging Open CV for face detection,
the 'dnn' module for age and gender identification, and a meticulously trained deep
learning model for emotion recognition, our script harnesses the power of these tools to
capture video from the default camera (webcam). It seamlessly detects faces within each
frame, delivering precise predictions for the age, gender, and emotion of individuals in
each facial region. The results are exhibited in real-time, with informative labels for age,
gender, and emotion gracefully superimposed onto the live video feed. Our script is
further fortified by the incorporation of pre-trained models for face detection, age and
gender recognition, and emotion analysis, which collectively work harmoniously to
enrich the video feed with invaluable insights.
 REFERENCES

1) Vo, T. H., Lee, G. S., Yang, H. J. & Kim, S. H. Pyramid with super resolution for the
wild facial expression recognition. IEEE Access 8, 131988–132001 (2020).
2) Mehrabian, A. Nonverbal communication (Aldine Transaction, 2007).
3) Ekman, P. Darwin, deception, and facial expression. Ann. N. Y. Acad. Sci. 1000, 205–2
(Kortli & Jridi, 2020) (2006).
4) Farzaneh, A. H. & Qi, X. Facial expression recognition in the wild via deep attentive
center loss 2021 IEEE winter conference on applications computer vision (WACV)
2401–2410 (IEEE, 2021).
5) Alnuaim, A. A. et al. Human computer interaction for recognizing speech emotions using
multilayer perceptron classifier. J. Healthc. Eng. 2022, 600 5446 (2022).
6) Li, S. & Deng, W. Deep facial expression recognition: A survey. IEEE Trans Affect.
Comput. 13, 1195–1215 (2022) 13 Canal, F. Z. et al. A survey on facial emotion
recognition techniques: A state oft heart literature review. Inf.Sci.582 593–617 (2022).
7) He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition
in 2016 IEEE conference on computer vision and pattern recognition imaging a
(CVPR)\770–778 (IEEE, 2016).
8) Mollahosseini, A., Hasani, B. & Mahoor, M. H. AffectNet: A database for facial
expression, valence, and arousal computing in the wild. IEEE Trans. Affect, Comput. 10,
18–31 (2019).
9) Schoneveld, L. & Othmani, A. Towards a general deep feature extractor for the facial
expression recognition in 2021 IEEE international conference on image a processing
(ICIP) 2339–2342 (IEEE, 2021).

10) Rajan, V., Brutti, A. & Cavallaro, A. Is cross attention preferable to self-attention for
multimodal emotion recognition in ICASSP 2022–2022 IEEE international conference
on acoustics, speech and signal processing (ICASSP) 4693–4697 (IEEE, 2022).
11) Zhuang, X., Liu, F., Hou, J., Hao, J. & Cai, X. Transformer based interactive multimodal
attention network for video sentiment detection. Neural Process. Lett. 54, 1943–1960
(2022).
12) Zhang, Y., Wang, C., Ling, X. & Deng, W. Learn from all: Erasing attention consistency
for noisy label facial expression recognition in computer science (eds. Avidan, S.,
Brostow, G., Cissé, M., Farinella, G. M. & Hassner T.) 418–434 (Springer,2022).
13) Savchenko, A. V., Savchenko, L. V. & Makarov, I. Classifying emotions and engagement
in online learning based on a single facial expression recognition neural network. IEEE
Trans. Affect. Comput. 13, 2132–2143 (2022).
14) Fan, Y., Lam, J. C. K. & Li, V. O. K. Multi region ensemble convolutional neural network
for facial expression recognition in Artificial neural networks and machine learning—
ICANN 2018 (eds. Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L. &
Maglogiannis, I.) 84–94 (Springer International Publishing, 2018).
15) Wang, Z., Zeng, F., Liu, S. & Zeng, B. OAENet: Oriented attention ensemble for
accurate facial expression recognition. Pattern Recognit. 112, 107694 (2021).
16) Schoneveld, L., Othmani, A. & Abdelkawy, H. Leveraging recent advances in deep
learning for audio Visual emotion recognition. Pattern Recognit. Lett. 146, 1–7 (2021).