F

REAL TIME FACE

DETECTION AND
RECOGNITION
Under the guidance of Dr. Amita jain
 Real Time Face Detection and Recognition

MAJOR PROJECT REPORT

Submitted in partial fulfilment of the requirements for the award of the degree

of

BACHELOR OF TECHNOLOGY

in

COMPUTER SCIENCE ENGINEERING

by

Name of the Student: Amit Dhyani Name of the Student: Shashank Bhardwaj
Enrollment No:03410102714 Enrollment No: 04610102714

Guided by

Dr. Amita Jain

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

AMBEDKAR INSTITUTE OF ADVANCED COMMUNICATION TECHNOLOGIES
& RESEARCH (GOVT.OF NCT OF DELHI)
GEETA COLONY, DELHI-110031, INDIA
(AFFILIATED TO GURU GOBIND SINGH INDRAPRASTHA UNIVERSITY,
DELHI)

MAY 2018

1|Page
 CANDIDATE’S DECLARATION

It is hereby certified that the work which is being presented in the B. Tech Major Project Report
entitled "REAL TIME FACE DETECTION AND RECOGNITION" in partial fulfilment
of the requirements for the award of the degree of Bachelor of Technology and submitted in
the Department of Computer Science of, AMBEDKAR INSTITUTE OF ADVANCED
COMMUNICATION TECHNOLOGIES & RESEARCH GEETA COLONY, DELHI-
110031, New Delhi (Affiliated to Guru Gobind Singh Indraprastha University, Delhi) is
an authentic record of our own work carried out during a period from January 2018 to May
2018 under the guidance of Dr. Amita Jain, Asst. Prof.. The matter presented in the B. Tech
Major Project Report has not been submitted by me for the award of any other degree of this
or any other Institute.

(Amit Dhyani) (Shashank Bhardwaj)

(En. No: 03410102714) (En. No: 04610102714)

This is to certify that the above statement made by the candidate is correct to the best of my
knowledge. They are permitted to appear in the External Major Project Examination

(Dr. Amita Jain)

Asst. Prof.

2|Page
 ABSTRACT

Face detection is the major step in face recognition. Which is one of the most classic and
representative application in computer vision. Face is one of the physiological biometric based
on the stable feature. Face detection is challenging mission because face in image are all
uncontrolled, E.g. illumination condition, vary pose, different facial expression.

Four types of face detection are developed. They are knowledge based methods, template
matching, invariant feature methods and learning based methods. In this project, learning based
method are deployed: AdaBoost Method

Adaboost is a machine learning algorithm which if formed from a sequence of weak classifier.
Each of weak classifier form a cascade and contributes a string classifier in this project as a
face detector.

Identifying a person with an image has been popularised through the mass media. However, it
is less robust to fingerprint or retina scanning. This report describes the face detection and
recognition major-project undertaken at AIACT&R. It reports the technologies available in the
Open-Computer-Vision (OpenCV) library and methodology to implement them using Python.
For face detection, Haar-Cascades were used and for face recognition Eigenfaces, Fisherfaces
and Local binary pattern. Next, the results are shown including screen-shots followed by a
discussion of encountered challenges.

3|Page
 ACKNOWLEDGEMENT

We express our deep gratitude to Dr. Amita Jain, Designation, Department Computer and
science for her valuable guidance and suggestion throughout my project work

Sign Sign
(Amit Dhyani) (Shashank Bhardwaj)
(En. No:03410102714) (En. No: 04610102714)

4|Page
 TABLE OF CONTENTS

CANDIDATE DECLARATION 2
ABSTRACT 3
ACKNOWLEDGEMENT 4
TABLE OF CONTENTS 5-6
LIST OF FIGURES 7
LIST OF ABBREVIATIONS 8

Contents
1. INTRODUCTION ...................................................................................................................... 9
Computer vision:........................................................................................................................... 9
Face detection:................................................................................................................................. 11
OBJECTIVES ................................................................................................................................. 12
Hardware and Software Requirements ........................................................................................ 12
Hardware ..................................................................................................................................... 12
SOFTWARE................................................................................................................................ 12
Chapter Summary .......................................................................................................................... 13
Literature Review ............................................................................................................................... 14
Introduction ..................................................................................................................................... 14
History of Face Detection: .............................................................................................................. 15
Importance for face Detection and Recognition: ......................................................................... 16
Method Approaches for Face Detection........................................................................................ 17
Major challenges: ............................................................................................................................ 19
Performance Evaluation ................................................................................................................. 22
Current status of overseas research .............................................................................................. 24
Example of Face Detection ............................................................................................................. 25
Chapter Summary .......................................................................................................................... 27
Methodology for Project..................................................................................................................... 28
Introduction ..................................................................................................................................... 28
Pre-processing ................................................................................................................................. 29
Grayscale conversion .................................................................................................................. 29
Algorithm of Face Detection: ......................................................................................................... 30
Haar classifier (Viola and Jones algorithm): ............................................................................ 30
INTEGRAL IMAGE: ................................................................................................................. 31

5|Page
 ADABOOST: ............................................................................................................................... 33
Pseudocode for Adaboost ........................................................................................................... 35
Cascading ..................................................................................................................................... 36
Why AdaBoost is used in this project ?......................................................................................... 37
Face Recognition ............................................................................................................................. 38
Local Binary Pattern Histogram ............................................................................................... 38
Chapter Summary .......................................................................................................................... 39
Testing and Implementation .............................................................................................................. 40
Overview .......................................................................................................................................... 40
Testing objective.............................................................................................................................. 40
Tools in Testing ............................................................................................................................... 40
Hardware Configuration:........................................................................................................... 40
Software configuration: .............................................................................................................. 40
Face Database: ................................................................................................................................ 41
Test case summary:......................................................................................................................... 42
Chapter Summary: ......................................................................................................................... 42
CONCLUSION ................................................................................................................................... 43
FUTURE WORK:............................................................................................................................... 45
Appendix: ............................................................................................................................................ 46
Result of test case 001: .................................................................................................................... 46
EXAMPLE SCREENSHOT OF FACE DETECTION ............................................................... 47
Example of Face Recognition ......................................................................................................... 48
References ............................................................................................................................................ 49

6|Page
LIST OF FIGURES:
1.1…………….Example of Object Detection

2.1…………….Sample for template Matching

2.2…………….Pose Variation
2.3…………….Presence/Absence of structuring elements/occlusions
2.4…………….Facial Expression Change
2.5…………….Aging
2.6…………….Varying illumination Condition
2.7…………….An example of face Detection
2.8…………….camera example
2.9…………….camera example

3.1…………….Overview of FDS
3.2……………..Grayscale Conversion
3.3……………..Haar features
3.4……………..Integral Image
3.5……………..Integral image
3.6………………Adaboost
3.7……………..Ada boost
3.8…………….Strong classifier from weak classifier
3.9 ……………..Pseudocode of Adaboost
3.10……………..Cascading
3.11……………Local binary pattern histogram
3.12……………results when brightness are changed

4.1……………..BioID database

List of Tables:

1.1………………Haar vs LBP
2.1……………….Application of Face recognition
4.1………………..Test case summary

7|Page
List of abbreviation

CCTV - Closed Circuit Television

FDS - Face detection system

GUI - Graphical user interface

NN - Neural Network

RGB - RGB color model

SVM - Support vector machine

PDF - probability density function

8|Page
 1. INTRODUCTION
Computer vision:

First of all to understand my project you have to understand computer

vision. Humans use their eyes and their brains to see and visually sense the world around them.
Computer vision is the science that aims to give a similar, if not better, capability to a machine
or computer. Computer vision is concerned with the automatic extraction, analysis and
understanding of useful information from a single image or a sequence of images.
The applications of computer vision are numerous and include:

 face recognition
 gesture analysis
 pollution monitoring
 process control

 remote sensing
 robotics
 security and surveillance

Face Recognition is one of the more representative and classic application in computer vision.
Face Detection is the most important step of face recognition. Not only the face recognition,
face detection also a first step in Human computer interaction system (HCI).It’s conveyed
information from physics world into logical thinking to control the computer system.
In additional, face detection is one of an object detection which to classify the desired object
from the given images/video and located it. License plate Detection is one of examples about
object detection.

9|Page
 Figure-1.1 Example of object detection

In biometric approaches, human faces are unique object like fingerprints, Iris which widely
used in security issues. Many types of personal authentication system have been developed
related this approaches which takes advantage of unique and special characteristics. System
can be searched effectively to screen out useful information (face) from dozens of video media
or photos from internets.

For example, Video surveillance in UK, there is one CCTV cameras for every 14 people. They
also need to analyse all these video, which use face detection to extract any useful information
and store it for further used.

Human faces are non-rigid objects and appeared in different scales, pose, angle and facial
expression. Human faces always have variation for example, glasses. In additional, the image
have different brightness contrast. These result are the challenge of face detection.

10 | P a g e
 Face detection:

Face detection is a computer technology being used in a variety of applications that identifies
human faces in digital images. Face detection also refers to the psychological process by which
humans locate and attend to faces in a visual scene.
Face detection can be regarded as a specific case of object-class detection. In object-class
detection, the task is to find the locations and sizes of all objects in an image that belong to a
given class. Examples include upper torsos, pedestrians, and cars.
Face-detection algorithms focus on the detection of frontal human faces. It is analogous to
image detection in which the image of a person is matched bit by bit. Image matches with the
image stores in database. Any facial feature changes in the database will invalidate the
matching process.
A computer program that decides whether an image is a positive image (face image) or negative
image (non-face image) is called a classifier. A classifier is trained on hundreds of thousands
of face and non-face images to learn how to classify a new image correctly. OpenCV provides
us with two pre-trained and ready to be used for face detection classifiers:

1. Haar Classifier
2. LBP Classifier

Haar vs LBP which is better:

Table-1.1 Haar vs LBP

11 | P a g e
OBJECTIVES

The aim of this project is to develop and propose a system to detect human faces in digital
images effectively, no matters what person’s ethnic, pose. Input images may be varied with
face size, complex of background and illumination condition.

Hardware and Software Requirements

Hardware

This system has been run on following configuration. In order to get bettr performance, update
RAM to 2GB or more is recommended.

Here is list of minimum hardware requirement:

1. Intel Pentium Processor or any other compatible Processor, 1 GHz or greater.
2. Minimum 128 MB of RAM capacity or more
3. Minimum of 32 MB Graphic Card RAM capacities or more
4. Recommended hard disk space of 1 GB or more.

SOFTWARE

This system was developed by python and Opencv and along with that we take help of some
package provided by python pip which are numpy, matplotlib, PIL.etc

OpenCV

OpenCV means Intel® Open Source Computer Vision Library. It is a

collection of C functions and a few C++ classes that implement some
popular Image Processing and Computer Vision algorithms. OpenCV has cross-platform
middle-to-high level API that consists of a few hundreds C functions. It does not rely on
external libraries, though it can use some when it is possible. OpenCV is free for both non-
commercial and commercial use

Python

Python is high level programming language like a perl, Ruby. Which are used as a scripting
language. It was conceived by Guido van Rossum in 1989. Also python is one of three “official
languages” in Google which means that more application in Google was deployed this
language, E.g Google App Engine SDK. Here are the points that make python is selected:

12 | P a g e
 1. Free, Python is product of open source. People allow to use it in business or commercial
without any charge.
2. Easy to read, syntax in Python is clear and readable. Beginner can be easily to read and
handle Python’s coding very well.
3. Rapid development, it is because it likes pseudo code. Everything coding in Python is
direct result.
4. Highly portability, Python is working on different platforms, because of python is
written portable ANSI C.
5. Reusability, Python is easily reused modules and packages. People can be developed
their own library and reused it later project.
6. Object-Oriented Programming. Unlike scripting language, python is designed to be
object-oriented. OO programming means you can implemented using ideas of
inheritance and polymorphis

Chapter Summary

This chapter is presented the background of computer vision to address the important of face
detection in our real life. Objective of project is being presented and let the people know the
basic requirement of hardware and software which tools are used to implement.

13 | P a g e
Literature Review
In this section, face detection methods will be introduced with short overview to give any
general idea about the history of face detection and future approaches of it.

Introduction

Face Detection is a technology to determine human face in videos and arbitrary images. The
aim of face detection is detect faces in any images or videos, if yes it’s reported the location on
the images or videos. It just only concern facial feature in testing image. In other word face
detection can be regarded as a pattern classification. Face detection is the first part of face
recognition, it is because we need to clarify facial feature before doing recognition step.

Face detection can be regarded as a specific case of object-class-detection. For the detection is
to locate the face in digital images/video stream, no matter what the pose, scale, facial
expression. In other words, face detection algorithm to handle pattern classification. It task to
identify a given image to decide it face or not.

The following items are the difference between face detection and other face processing:

1. Face detection: To determine any face in given image/video.

2. Face localization: To locate the location of the face in given image/video.
3. Face recognition: To recognize faces in given images/video.

14 | P a g e
History of Face Detection:

In the early stage, face detection algorithm mainly focused to detect the frontal human face.
However newer algorithms try to consider the different view of face as a core of face detection.

The first of face detection system has been developed since in early 1970’s Due to limitation
of computation, system can’t be satisfied the requirement of users, which is identify passport
photograph real time.

At the beginning of 1990’s techniques are proposed focused on the face recognition on and
increase the need of face detection. Many system were constructed to deal with video
streaming. In the past few years, lots of methods are developed at least more than 150 methods

During 1964 and 1965, Bledsoe, along with Helen Chan and Charles Bisson, worked on using
the computer to recognize human faces (Bledsoe 1966a, 1966b; Bledsoe and Chan 1965). He
was proud of this work, but because the funding was provided by an unnamed intelligence
agency that did not allow much publicity, little of the work was published. Given a large
database of images (in effect, a book of mug shots) and a photograph, the problem was to select
from the database a small set of records such that one of the image records matched the
photograph. The success of the method could be measured in terms of the ratio of the answer
list to the number of records in the database.
U.S. Government-sponsored evaluations and challenge problems have helped spur over two
orders-of-magnitude in face-recognition system performance. Since 1993, the error rate of
automatic face-recognition systems has decreased by a factor of 272. The reduction applies to
systems that match people with face images captured in studio or mugshot environments.
In Moore’s law terms, the error rate decreased by one-half every two years.

15 | P a g e
Importance for face Detection and Recognition:

1. In order to prevent the frauds of ATM in India, it is recommended to prepare the

database of all ATM customers with the bank in india and deployment of high
resolution camera and face recognition software at all ATMs, So whenever user will
enter in ATM his photograph will be taken to permit the access after it is being matched
with stored from the database.
2. Duplicate voter are being reported in India. To prevent this, a database of all voters, of
course, is recommended to prepared. Then at the time of voting the resolution camera
and face recognition equipped of voting site will accept a subject face 100% and
generates the recognition for voting if match is found.
3. Passport and visa verification can also be done using face recognition technology as
explained above.
4. Driving license verification can also be exercised face recognition technology as
mentioned earlier.
5. To identify and verify terrorists at airports, railway stations and malls the face
recognition technology will be the best choice in india as compared with other biometric
technologies since other technologies cannot be helpful in crowdy places
6. It can also be deployed in police station to identify and verify the criminals.
7. It can also be deployed vaults and lockers in banks for access control verification and
identification of authentic users.
8. Present barcode system could be completely replaced with the face recognition
technology as it is better choice for access and security since barcode could be stolen
by anybody else.

The following tables are the example application use of the face detection:

Table-2.1 Application of face detection

16 | P a g e
Method Approaches for Face Detection

In general, FD can be implemented by four methods: knowledge based methods, template

matching, invariant feature methods and learning based methods. These methods will be
introduced with the following.

1. Knowledge based methods: The models are used human knowledge to find face
patterns from testing images. Based on the nature of human faces, algorithms scan the
image from top to bottom and left to right order to find facial feature. For instance, face
should be including two eyes and mouth.

Pros: Easy applied into simple rules

Cons: difficult to detect in invariant background, such as different pose, uncontrolled

illumination. Well results based on well- defined rules. This algorithm does not work
on the pose.

2. Template matching: The model is used several templates to find out the face class
and extract facial features. Rules are pre-defined and decide whether there is
face in the image. For instance, using filters to extract the contours of the face shape.

Figure-2.1 Sample for template matching

Pros: Simple to apply the method

Cons: similar to knowledge based method, hard to detect face in poses.

17 | P a g e
 3. Invariant feature methods: The methods is bottom-up approaches and used to
find a facial feature, even in the presence of composition, perspective vary, so it is
difficult to find a face real time using this method. Statistical models are developed to
determine the faces. Facial features of human faces are: shape, texture, skin.

Pros: Unlike knowledge base method, it is invariant to pose and expression.

Cons: not suitable to detect facial features from uncontrolled background, time
consuming algorithms. Detection rate is not accuracy, because of need to combine
different feature and processing it.

4. Learning Based Methods: The models are trained from set of a training set before
doing detection. For the large amount of training data, it can be provided high accuracy
recognition rate to resist variation, expression and pose of faces images. For instance,
many of non-face and face images import into the system. Machine learning techniques
are employed to train the system based on the statistical properties and probability
distribution function. Principle Component Analysis (PCA), Support Vector Machine
(SVM), Naïve Bayes Classifier, Hidden Markov model, Neural Network and adaboost
are all well-known classifiers to use for face detection.

Pros: fast to detect face. Can be detected different faces pose and orientation if have
enough training set. Showed a good empirical results.

Cons: need more and more non-face and face sample for training, need to scan different
scale.

18 | P a g e
Major challenges:

Although many of the algorithms have been proposed and improved within a few years. Face
detection are still limited by uncontrolled factors. E.g. variant pose, illumination condition,
expression, hairstyle, and different camera equipment. Either one of the problems can decrease
the performance of face detection. At the worst case, it should be the combination of above
problem. However, this situation often happens in our real worlds. E.g. surveillance with low
quality.
In addition, occlusion is another of factor to effect the result of the FD rate. E.g. make-up
sunglasses. In other words, occlusion sometime will be greatly changed the appearances of the
faces.
The following items are the summary of the main challenge in face detection:

1. Variant pose: Variant pose is occurred because of people not always orient to the
camera.
2. Illumination condition: Different lighting and the quality of camera directly affect the
quality of the face. Sometimes it can be varied greater than the facial expression and
occlusion.
3. Facial Expression: Different expression in the faces is presented different information
to the machine. Face is non-rigid objects which is changed by different expression.
4. Occlusion: Face detection not only deals with different faces, however, it need deal
with any optical object. E.g. Hairstyle, sunglasses are the example of occlusion in face
detection. For global feature, occlusion is one of the main feature in face detection.
5. Uncontrolled background: Face detection systems are not only detected faces on
uniform environment. In reality, people are always located on complex background
with different texture and object. These thing are the major factors to effect the
performance of face detection system.
The following photos are example of uncollected factors

1. Pose variations

Figure -2.2 Pose variation

19 | P a g e
 2. Presence/absence of structuring elements/occlusions

Figure-2.3 Presence/absence of structuring elements/occlusions

3. Facial expression changes

Figure-2.4: Facial expression change

4. Ageing of the face

Figure-2.5 : Aging

20 | P a g e
 5. Varying illumination condition

Figure-2.6 : Varying illumination codition

21 | P a g e
Performance Evaluation

The result of evaluation of face detection system is varied from different face database. In order
to get better evaluation of face detection and recognition, regular bases should be applied. The
result of the or analysis will played certain role for future work.

For face detection, two parameters are introduced to evaluate the performance:
1. Detection rate: It is the ratio to find the face in the given images/ video.
2. False positive rate: It is the rate of non-face objects are reported as positive.
Here is the equation of false positive rate.
𝑓𝑎𝑙𝑠𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑟𝑎𝑡𝑒 = (𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑓𝑎𝑙𝑠𝑒 𝑝𝑜𝑠𝑖𝑡𝑣𝑒𝑠)/
(𝑡𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑐𝑡𝑢𝑎𝑙 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠

If we want to achieve a high detection rate, then the false positive rate will tend to be higher.
In other words, if you want to achieve low false positive rate, then the detection rate would be
under the expectation.

In perfect algorithm of face detection, it must be 100% of detection rate and 0% false positive
rate. However, at this stage only the human brain can do this.

We limit consideration here to three questions: Under what conditions does an algorithm
perform well/poorly, how much performance variation is attributable to specific covariates,
and what are the covariate effects? Fundamentally, this is a predictor/response investigation.
5.1. Outcomes Our unit of analysis was a match pair, and our response variable was I(sA(q, t)
> τ ) where τ is the threshold yielding a false acceptance rate of 0.001 (i.e., the 0.999 percentile
of the non-match scores) and I(X) is 1 if X is true and 0 otherwise. The alternative response
variable was the similarity score, sA. For modern face recognition algorithm, this is a poor
option. Some algorithms produce similarity scores that are a mix of ratings and continuous
numbers: for example there might be categories of ‘terrible’, ‘bad’, ‘possible’, and numerical
scores above that. Second, similarity scores have very different distributions depending on
whether the image pair comprises two images of the same person (a ‘match score’) or two
different people (a ‘non-match score’). Because of these reasons, similarity scores between
algorithms are incomparable. Non-match scores tend to have broad and very heavy-tailed
distributions compared to match scores. Moreover, distributions of both types of scores can be
markedly skewed, especially for no matches. Third, because the calculation of similarity scores
can be highly esoteric, one should have no confidence that some sort of score normalization
would produce values for which incremental changes scale in any sensible way with algorithm
performance. Our approach of limiting consideration to trials on match pairs discards the vast
majority of similarity scores; however, it allows us to pose interpretations in terms of the
estimated probability of successful verification, which is directly relevant for algorithm

22 | P a g e
evaluators and end users. 5.2. Covariates Two sets of covariates are associated with the
similarity matrix. The first set contains measurements associated with a single image, such as
imaging setting (an environment variable), gender (subject), or focus (image). The second set
contains covariates associated with image pairs, such as setting change (e.g., between outdoor
and indoor), time lapse between images, and comparative focus. The covariates available in
our dataset are listed in Table 2. The lighting variable is an estimate of how the face is
illuminated using models that connect the theory of illumination of convex objects with the
empirical representation of non-linear manifolds (Beveridge et al., 2010). Location is the exact
location where images were acquired: one of 10 possible sites used by the University of Notre
Dame to collect these data. Edge density is the average edge magnitude of a Sobel operator
applied to the chip; this and the focus measure are reviewed by Beveridge et al. (2009b, 2010).
The remaining covariates are mostly self-explanatory. 5.3. The Good, Bad and Ugly partition
labels As noted above, the dataset is partitioned into three portions. In a rough sense, the image
pairs are grouped in a manner related to ease of verification. This may be a concern because
the partition label is a covariate that seems to be a virtual recoding of the response variable.
There are three reasons for proceeding despite this concern. The first is pragmatic. The
biometric algorithm development community needs to improve performance on difficult
problems while at the same time not sacrificing performance on easier problems for which
performance is already good. This is best done with a graduated challenge problem. The second
is also pragmatic, but having more directly to do with choosing between two modelling options.
Concern about the GBU covariate could be alleviated by breaking the analysis apart and
treating each partition as its own unique study. While cleaner, such an approach denies us the
opportunity to directly quantify interactions between the partitions and other covariates. A final
reason is based on how GBU should be interpreted. For each person separately, the person’s
image pairs were sorted by similarity score, with the very best being placed in the Good
partition, the very worst in the Ugly partition, and a part of the remainder placed in the Bad
partition. It is critical to note that the GBU factor level for a match pair is a person specific
measure of the extent to which that image pair approaches the subject’s maximum potential
match ability.

A face recognition algorithm A may be defined as a mapping from a pair of face images to a
real number sA representing some measure of the similarity (equivalently, distance) between
those images. Formally, A : IT × IQ → ℜ (1) G.H. Givens et al. / Computational Statistics and
Data Analysis 67 (2013) 236–247 239 where it is common to describe the first image as being
drawn from a set of target images IT and the second from a set of query images IQ . This
definition gives rise to a data structure called the similarity matrix: SA = {A(t, q) ∀t ∈ IT , q ∈
IQ } = [sA(t, q)]. (2) A similarity score sA(t, q) is said to be a match score when the same
person is pictured in images q and t and a non-match score when the pictures are of different
people. For most typical evaluation tasks, an algorithm is run on all pairwise combinations of
images from IT and IQ , and all subsequent questions about algorithm performance are
addressed by studying the similarity matrices. Recognition, and hence performance evaluation,
may be expressed in terms of two distinct tasks: identification and verification. In an
identification task, a query image q ∈ IQ is compared to a set of (target) images in a gallery IT

23 | P a g e
. The gallery is then sorted by similarity with respect to q and either the most similar or a small
set of the most similar target images are returned as a presumed match or match ranking for q.
In a verification task, a person presents themselves to a system that already has a stored (target)
image of his/her face. The system acquires a new (query) image of the person and compares
the resulting similarity sA(t, q)to an acceptance threshold. If the similarity exceeds that
threshold then the system confirms that the person is who s/he claims; otherwise the person’s
claimed identity is rejected. The acceptance threshold is chosen using a training dataset where
true identities are known. The threshold is usually set to achieve a pre-determined false
verification (i.e., false acceptance) rate. The rate of correct verification will depend on the
extent to which the distributions of match and non-match scores overlap. Most of the major
face recognition evaluations carried out in the past decade have concentrated on the verification
task. One reason is that the outcome of a verification test depends only on the similarity score
sA(t, q) and the threshold for acceptance. In contrast, identification performance depends upon
the other people and images in the gallery. Consequently, results for small galleries typically
do not scale to problems involving more people and images. Accordingly, the development of
sophisticated statistical methods for evaluation of identification performance is an open field
for new research.

Current status of overseas research

The face detection algorithm is well classified with different methods. These algorithms are
used base on which condition are employed. Well-know university research institutes,
enterprise institute have established a research group associated with face related project. They
are CMU, MIT, Michigan State university,UCLA… and so on.

In the good controlled conditions, the detection rate can be achieved at least 95% from the case
about 1000 people. However, in the environment is relatively poor circumstances, the detection
rate just got 80% or below.

24 | P a g e
Example of Face Detection

Nowadays, the technology of face detection is widely applied within different field of business,
Photo-taking, search engine, biometrics and so on. These technologies become popularity
because of increase humean interest in life, as a well-known mobile phone manufactures said:
connecting people.

The following is an example of face detection:

1. Search engine-Microsoft:

The search result of ‘bill gate’ is shown below.

Figure-2.6 An example of face detection

25 | P a g e
 2. Camera Manufacturer-Nikon:
Pictures from advisement of Nikon are shown below

Figure-2.8 Nikon camera example

Figure-2.9 : Nikon camera example

26 | P a g e
Chapter Summary

This chapter is presented about the introduction of face detection, the major challenges of face
detection. In addition, four face detection methods are introduced to give general idea to user
about the approaches of research area. Also the performance evaluation is addressed to show
how is the successful detection system should be achieved.

27 | P a g e
Methodology for Project
Introduction

Each input image is processed with resizing and converting to grey color for further detection.
When given image input to system, it will be experienced with number of classifier (cacascade)

The face detector was proposed by viola and jones. Adaboost used as a classifier which is
trained with a few hundred sample images. It’s included with positive sample (faces) and
negative sample (non faces). Each trained samples were resize and scaled into same size.

Face detection in this project can be regarded as two parts and illustrated as follows:
1. Pre-processing: This stage include image processing to enhance the quality of input
image.
2. Face detector stage: The core section in face detection includes Adaboost filter.

Figure-3.1 : Overview of FDS

28 | P a g e
Pre-processing

System input is color image which included images of human face or not, output is the human
face which is extracted from original images, In order to get the better result of detection, pre-
processing is essential. In this section pre-processing is addressed with giving detail
description.

Grayscale conversion

For getting to reduce the information of image, images should be done a converting a grayscale.
Each color image (RGB images) are composed of 3 channels to present red,green,blue
components in RGB space. Below is the example to giving idea of RGB color image

Figure-3.2 : Grayscale conversion

Image is defined with grayscale level, which means pixels in images are stored 8 bit integer to
represent color from black to white. In addition grayscale images are sufficient enough to
process face detection.

29 | P a g e
Algorithm of Face Detection:

Haar classifier (Viola and Jones algorithm):

It is a machine learning based approach where a cascade function is trained from a lot of
positive and negative images. It is then used to detect objects in other images.

Here we will work with face detection. Initially, the algorithm needs a lot of positive images
(images of faces) and negative images (images without faces) to train the classifier. Then we
need to extract features from it. For this, Haar features shown in the below image are used.
They are just like our convolutional kernel. Each feature is a single value obtained by
subtracting sum of pixels under the white rectangle from sum of pixels under the black
rectangle.

Feature= (sum of pixel under black rectangle) - (sum of pixel under white

Region)

Figure-3.3 : Haar feature

It is a common observation that among all faces the region of the eyes is darker than the region
of the cheeks. Therefore a common Haar feature for face detection is a set of two adjacent
rectangles that lie above the eye and the cheek region. The position of these rectangles is
defined relative to a detection window that acts like a bounding box to the target object (the
face in this case).

30 | P a g e
In the detection phase of the Viola–Jones object detection framework, a window of the target
size is moved over the input image, and for each subsection of the image the Haar-like feature
is calculated. This difference is then compared to a learned threshold that separates non-objects
from objects. Because such a Haar-like feature is only a weak learner or classifier (its detection
quality is slightly better than random guessing) a large number of Haar-like features are
necessary to describe an object with sufficient accuracy. In the Viola–Jones object detection
framework, the Haar-like features are therefore organized in something called a classifier
cascade to form a strong learner or classifier.

INTEGRAL IMAGE:

The Integral Image is used as a quick and effective way of calculating the sum of values (pixel
values) in a given image – or a rectangular subset of a grid (the given image).
Now, all possible sizes and locations of each kernel are used to calculate lots of features. (Just
imagine how much computation it needs? Even a 24x24 window results over 160000 features).
For each feature calculation, we need to find the sum of the pixels under white and black
rectangles. To solve this, they introduced the integral image. However large your image, it
reduces the calculations for a given pixel to an operation involving just four pixels. Nice, isn't
it? It makes things super-fast.

Figure-3.4: Integral Image

Well, once you have used the equation to calculate and fill up your Summed Area Table, the
the task of calculating the sum of pixels in some rectangle which is a subset of the original
image can be done in constant time. Yes, that’s right, in O(1) complexity!
In order to do this we only need to use 4 values from the Summed Area Table, that is, 4 array
references into the Summed Area Table. With these 4 values, we then add or subtract them for
the correct value of the sum of the pixels within that region. To do this, we use this equation:

31 | P a g e
 Figure-3.5 : Integral image

32 | P a g e
ADABOOST:

AdaBoost is a short form of Adaptive Boosting.AdaBoost is machine learning algorithm. It

pick up a few thousand features and assign weights to each one based on a set of training
images. The aim of Adaboost is to assign each weak classifier with best combining weights.

Figure-3.6 : Adaboost

But among all these features we calculated, most of them are irrelevant. For example, consider
the image below. The top row shows two good features. The first feature selected seems to
focus on the property that the region of the eyes is often darker than the region of the nose and
cheeks. The second feature selected relies on the property that the eyes are darker than the
bridge of the nose. But the same windows applied to cheeks or any other place is irrelevant. So
how do we select the best features out of 160000+ features? It is achieved by Adaboost.

Figure-3.7: AdaBoost

For this, we apply each and every feature on all the training images. For each feature, it finds
the best threshold which will classify the faces to positive and negative. Obviously, there will

33 | P a g e
be errors or misclassifications. We select the features with minimum error rate, which means
they are the features that most accurately classify the face and non-face images. (The process
is not as simple as this. Each image is given an equal weight in the beginning. After each
classification, weights of misclassified images are increased. Then the same process is done.
New error rates are calculated. Also new weights. The process is continued until the required
accuracy or error rate is achieved or the required number of features are found).

Figure-3.8: Strong classifier from weak classifier

The final classifier is a weighted sum of these weak classifiers. It is called weak because it
alone can't classify the image, but together with others forms a strong classifier. The paper says
even 200 features provide detection with 95% accuracy. Their final setup had around 6000
features

34 | P a g e
Pseudocode for Adaboost

An important part of the modified AdaBoost algorithm is the determination of the best feature,
polarity and threshold. There seems to be no smart solution to this problem and Viola-Jones
suggest a simple brute force method. This means that the determination of each new weak
classifier involves evaluating each feature on all the training examples in order to find the best
performing feature. This is expected to be the most time consuming part of the training
procedure. The best performing feature is chosen based on the weighted error it produces. This
weighted error is a function of the weights belonging to the training examples. As seen in
Figure 5 part 4) the weight of a correctly classified example is decreased and the weight of a
misclassified example is kept constant. As a result it is more ‘expensive’ for the second feature
(in the final classifier) to misclassify an example also misclassified by the first feature, than an
example classified correctly. An alternative interpretation is that the second feature is forced
to focus harder on the examples misclassified by the first. The point being that the weights are
a vital part of the mechanics of the AdaBoost algorithm.

Figure-3.9 pseudocode of Adaboost

35 | P a g e
Cascading

Instead of applying all 6000 features on a window, the features are grouped into different stages
of classifiers and applied one-by-one. (Normally the first few stages will contain very many
fewer features). If a window fails the first stage, discard it. We don't consider the remaining
features on it. If it passes, apply the second stage of features and continue the process. The
window which passes all stages is a face region.
The authors' detector had 6000+ features with 38 stages with 1, 10, 25, 25 and 50 features in
the first five stages. (The two features in the above image are actually obtained as the best two
features from Adaboost). According to the authors, on average 10 features out of 6000+ are
evaluated per sub-window.

Figure-3.10: Cascading

36 | P a g e
Why AdaBoost is used in this project ?

AdaBoost algorithm is core methods in this project, the following are the advantages:

1. Simple to implementation: As you know, AdaBoost is a machine learning which

only input two set of data. One is training dataset, other is a sequence of weak classifier.
System is no need to concern about the facial structure.

2. Adaptive algorithm: AdaBoost is a short name of adaptive boosting, which means

each of weak classifier will be adjusted their own weight when import positive and
negative samples in learning stage of each iteration.

3. Fast Detection Method: In other research, AdaBoost got 15 frames per second of
face detection with the size of resolution is 384*288 pixel

37 | P a g e
Face Recognition
The following sections describe the face recognition algorithm Local binary pattern histogram
and how they are implemented in OpenCV.

Local Binary Pattern Histogram

Local binary patterns were proposed as classifiers in computer vision and in 1990 By Li Wang
[4]. The combination of LBP with histogram oriented gradients was introduced in 2009 that
increased its performance in certain datasets [5]. For feature encoding, the image is divided
into cells (4 x 4 pixels). Using a clockwise or counter-clockwise direction surrounding pixel
values are compared with the central as shown in figure 6. The value of intensity or luminosity
of each neighbors is compared with the center pixel. Depending if the difference is higher or
lower than 0, a 1 or a 0 is assigned to the location. The result provides an 8-bit value to the cell.
The advantage of this technique is even if the luminosity of the image

Figure 3.11: Local binary pattern histogram generating 8-bit number

is changed as in figure 7, the result is the same as before. Histograms are used in larger cells to
find the frequency of occurrences of values making process faster. By analysing the results in
the cell, edges can be detected as the values change. By computing the values of all cells and
concatenating the histograms, feature vectors can be obtained. Images can be classified by
processing with an ID attached. Input images are classified using the same process and
compared with the dataset and distance is obtained. By setting up a threshold, it can be
identified if it is a known or unknown face. Eigenface and Fisherface compute the dominant
features of the whole training set while LBPH analyse them individually.

Figure 3.12: The results are same even if brightness is changed

38 | P a g e
Chapter Summary

This section present the methodology used in the project. They are included two parts. The first
part is pre-processing section, which can be divided into “Grayscale conversion” and “Image
resizing”, Second part is algorithm section which are “Integral image”, “AdaBoost” and
“cascade”

39 | P a g e
Testing and Implementation

Overview

This section is pointed out the test case and test documentation. The aim of this section is to
make sure ‘FDS’ to achieve the objective without any error. The details information is shown
below.

The main testing is defined the ability of face detection. Testing will be focused on the pose,
illuminated condition. These testing will be tested with different face database to get the face
detection rate of the system.

Testing objective

1. Make sure 100% correct code.

2. Meet the functional requirement of the FDS

Tools in Testing

Hardware Configuration:

- CPU: Intel 4.00 GHz

- Ram : 8.00GB

Software configuration:

- Window 10
- Python 3.0
- Opencv 1.0

40 | P a g e
Face Database:

BioID is a test set, it has larger vary of illumination, complex of background and different face
size of each picture. The condition of environment in the set like an indoor condition. In
addition each picture is involved one person.

Figure-4.1 BioID database

41 | P a g e
Test case summary:

Table-4.1 Test case result

Chapter Summary:

This section is given more detail information for test case. Everything in test cases is under
controlled condition to determine the function of system is worked correctly. Test cases are
focused on detection rate the system with face database. The general face detection rate is over
80% with variation with different face size.

42 | P a g e
CONCLUSION

Face recognition is one of the several techniques for recognising people.

There are several methods that can be used for that purpose. Some of the most common are
using PCA or

Eigen faces. Thought there are other new techniques more simple to understand use and
implement but also with very good performance. The algorithm has one of those algorithms.
As we show it has very good performance and is a very accurate especially if we use a
feedforward neural network.

Face recognition technology has come a long way in the last twenty years. Today, machines
are able to automatically verify identity information for secure transactions, for surveillance
and security tasks, and for access control to buildings. These applications usually work in
controlled environments and recognition algorithms that can take advantage of the
environmental constraints to obtain high recognition accuracy. However, next generation face
recognition systems are going to have widespread application in smart environments, where
computers and machines are more like helpful assistants. A major factor of that evolution is
the use of neural networks in face recognition. A different filed of science that also is very fast
becoming more and more efficient, popular and helpful to other applications.

The combination of these two fields of science manage to achieve the goal of computers to be
able to reliably identify nearby people in a manner that fits naturally within the pattern of
normal human interactions. "They must not require special interactions and must conform to
human intuitions about when recognition is likely. This implies that future smart environments
should use the same modalities as humans, and have approximately the same limitations.
"These goals now appear in reach however, substantial research remains to be done in making
person recognition technology work reliably, in widely varying conditions using information
from single or multiple modalities."

The results presented here illustrate how statistical models can be applied in a manner to help
algorithm developers and users understand – and hopefully improve – recognition
performance. It is clear that the statistical toolbox contains many more applicable methods for
development and evaluation. Performance evaluation is a particularly fertile topic for statistical
attention due to burgeoning interest from the biometric community and the fact that a
predictor–response formulation is one natural way to pose key questions using available
datasets. More broadly, opportunities for statistical research in face recognition are abundant
because many aspects of recognition algorithms, the structure of the identification and
verification tasks they face, and their performance have only recently begun to be posed
explicitly in a statistical context.

43 | P a g e
The face is one of the most useful keys to identifying a person. Thus far many algorithms have
been developed for automatic face recognition, namely PCA, LDA, SVM, LFA, neural network
[1, 2]. Some of them have demonstrated excellent results. But there are still many technical
limitations coming from the variety of illumination, expression, pose and accessory.
Understanding the limitations of the current face recognition technology is a key to developing
successful face recognition systems. This has led to the development of systematic tests and
image databases.
While the attention of face recognition researchers has largely been focused on methods for
distinguishing between face images, robust face recognition will not be attained until
researchers also learn how to deal with the environmental variations that are encountered in
real-world applications. To accomplish this it is necessary to create face databases with
carefully calibrated environmental variations that can be used to challenge existing face
recognition algorithms, and expose the specific weaknesses that are inherent in each. Such
databases can also be useful in validating new approaches to face recognition, and in comparing
the performance of different face recognition algorithms. The methodology described in this
paper provides one possible answer to the question of how to build such a database.

44 | P a g e
FUTURE WORK:

At this time, researchers are still figure out the difficulty of face detection: vary illumination,
occlusion on the face, pose variation. The algorithm of face detection are still improved by
1. To increase face detection rate.
2. Developing rapid object detection system in real time.
For the increasing the face detection rate, some techniques do a great job here, Eye detector or
skin detector are example of this improvement. New methods are added, as a result, it
eliminated uncontrolled factors. For example, skin detector is applied in the step of pre-
processing. System just considered a small part of area in the image.

The general experimental evaluation of the face expressional system guarantees better face
recognition rates. Having examined techniques to cope with expression variation, in future it
may be investigated in more depth about the face classification problem and optimal fusion of
color and depth information. Further study can be laid down in the direction of allele of gene
matching to the geometric factors of the facial expressions. The genetic property evolution
framework for facial expressional system can be studied to suit the requirement of different
security models such as criminal detection, governmental confidential security breaches etc.

Robots with facial recognition technology, in the long run, can help cut costs, be
assigned tasks that are otherwise difficult or impractical for humans or in areas with
a clear deficit of human resource. They will also play an important role in servic es
that require a high level of accuracy. Mobile robots like those by big robotics
firms can be used in restricted areas for patrolling. If the system recognise a human
in the PTZ (Pan-Tilt-Zoom) camera image, it transmits an alarm signal to a guard
station. After analysing the video image, an operator can make a decision to reset an
alarm, turn on a siren, or a turn on a strobe on the robot and send out security personnel
to deal with the intrusion. In large facilities and factories, such robot surveillance can
add an extra layer of security

45 | P a g e
Appendix:

Result of test case 001:

Figure-23 Result of test case 001

46 | P a g e
EXAMPLE SCREENSHOT OF FACE DETECTION

Figure-24 Screenshot of face detection

Figure-25 Screenshot of face detection

47 | P a g e
Example of Face Recognition

Figure-26 screen shot of face recognition

48 | P a g e
References
[1] Takeo Kanade. Computer recognition of human faces, volume 47. Birkh¨auser Basel,
1977.

[2] Lawrence Sirovich and Michael Kirby. Low-dimensional procedure for the
characterization of human faces. Josa a, 4(3):519–524, 1987.

[3] M. Turk and A. Pentland. Eigenfaces for recognition. Journal of Cognitive

Neuroscience, 3(1):71–86 , Jan 1991.

[4] Dong chen He and Li Wang. Texture unit, texture spectrum, and texture analysis.
IEEE Transactions on Geoscience and Remote Sensing, 28(4):509–512, Jul 1990.

[5] X. Wang, T. X. Han, and S. Yan. An hog-lbp human detector with partial occlusion
handling. In 2009 IEEE 12th International Conference on Computer Vision, pages
32–39, Sept 2009.

[6] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman. Eigenfaces vs. fisherfaces:

recognition using class specific linear projection. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 19(7):711–720, Jul 1997.

[7] P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple
features. In Proceedings of the 2001 IEEE Computer Society Conference on
Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages I–511–I–
518 vol.1, 2001.

[8] John G Daugman. Uncertainty relation for resolution in space, spatial frequency, and
orientation optimized by two-dimensional visual cortical filters. JOSA A, 2(7):1160–
1169, 1985.

[9] S Marˆcelja. Mathematical description of the responses of simple cortical cells. JOSA,
70(11):1297 – 1300, 1980.

[10] Rainer Lienhart and Jochen Maydt. An extended set of haar-like features for rapid
object detection. In Image Processing. 2002. Proceedings. 2002 International
Conference on, volume 1, pages I–I. IEEE, 2002.

[11] John P Lewis. Fast template matching. In Vision interface, volume 95, pages 15–19,
1995.

49 | P a g e