2019 4th International Conference on Pattern Recognition and Image Analysis (IPRIA), 6 and 7 March, Tehran, Iran
Partially Covered Face Detection in Presence of
Headscarf for Surveillance Applications
Hoda Qezavati
Department of Computer Engineering Department of Computer Engineering
Khatam University Khatam University
Tehran, Iran
h.qezavati@khatam.ac.ir b.majidi@khatam.ac.ir
Abstract— In the past few years, the application of surveillance
for security and smart cities are growing rapidly. The human
detection based on the surveillance videos is a complex task and
traditional clothing such as headscarf makes this task even more
difficult. The surveillance systems designed for many countries are
required to be able to recognize the people with these traditional
clothing. In this paper, a computer vision system for partially
covered face detection in low resolution surveillance videos
containing traditional Middle Eastern clothing including the
headscarf is presented. The proposed framework uses a
combination of Haar cascade and Locally Binary Patterns
Histogram (LBPH) for feature extraction and the Support Vector
Machine (SVM) algorithm for face classification. A large dataset
of a crowded office environment in Middle East is collected and
used for evaluation of the proposed model. The experimental
results show that the proposed method has acceptable results for
face detection in complex surveillance scenarios.
Index Terms— face detection, surveillance, Haar cascade,
LBPH, SVM.
I. INTRODUCTION
Face detection and detection in real-world surveillance
videos is a challenging task as faces might be influenced in the
video streams by variations in both the illumination and the
posture. Furthermore, sometimes input images might contain
some interference factors like noise, which complicates the face
detection process considerably. However, face detection is an
important task especially for security monitoring purposes and
it is necessary for many smart city applications. Face detection
plays a major role in personal identification, crowd analysis,
biometrics, database establishment, access control, general
identity verification and abnormal event detection [1]. Such
technologies are invaluable particularly to the police force for
tracking criminals. Since surveillance videos are used in many
applications, these cameras have been installed in many public
places such as airports, shopping malls, streets and etc. Manual
security monitoring is both unreliable and expensive and using
artificial intelligence to detect, recognize and identify humans
improves the efficiency and robustness substantially.
Different approaches have been presented for face detection
[2][3]. Generally, face detection methods which have been
proposed can be separated into two main categories: the ones
that use visual characteristics and methods that extract features
using machine learning. The former methods utilize the intuitive
978-1-7281-1621-1/19/$31.00 ©2019 IEEE
Authorized licensed use limited to: KDU - Gen. Sir John Kotelawala Defence University. Downloaded on December 30,2021 at 19:54:35 UTC from IEEE Xplore. Restrictions apply.
Babak Majidi Mohammad Taghi Manzuri
Department of Computer Engineering
Sharif University of Technology
Tehran, Iran Tehran, Iran
manzuri@sharif.edu
features from faces in order to detect faces. The latter methods
extract the features from sample faces and then use them in a
machine learning algorithm for detecting faces.
There are many face detection methods based on skin [4, 5],
but because of various lighting conditions and many existing
interference areas, skin-based face detection sometimes cannot
attain acceptable outcome from surveillance video sequences. In
addition, some approaches use feature extraction and machine
learning to detect faces. The best algorithm in this category is
Viola and Jones's approach [6]. In this approach, an AdaBoost
Cascaded Classifier and Haar Features is used and achieves
good results in the field of face detection. On the other hand,
these approaches need to design features manually and the
classifiers of conventional machine learning have not acquired
good enough outcome when it comes to detecting unconstrained
faces in the video streams.
Machine learning and image processing are able to provide
reliable solution for provision of autonomous solutions in
various smart environments [7-13]. Currently, the detection and
recognition of faces which are partially covered is an issue. In
many Middle Eastern countries traditional clothing dictates
partial coverage of the faces. Clothing such as headscarves and
similar items will partially to fully cover the head and a part of
the face. This problem is aggravated by low resolution
surveillance cameras. In this paper, an algorithm for face
detection in partially covered faces in low resolution
surveillance cameras is proposed. The proposed framework uses
a combination of Haar Cascade and Locally Binary Patterns
Histogram (LBPH) for feature extraction and the Support Vector
Machine (SVM) algorithm for classification. A large dataset of
a crowded office environment in Middle East is collected and
used for evaluation of the proposed model. The experimental
results show that the proposed method has acceptable results for
person identification in low resolution surveillance cameras in
presence of partially covered faces.
The rest of this paper is organized as follows. The proposed
face detection framework is detailed in Section II. The
experimental design and the simulation scenarios are discussed
in Section III. Finally, Section IV concludes the paper.
195
 II. PARTIALLY COVERED FACE DETECTION IN PRESENCE OF by training the SVM model on the dataset. So, the training part
HEADSCARF involves the minimization of the error function as follows:
A number of researches have been performed on face ே
detection and recognition during the past few years. Some ͳ ௧
‫ ݓ ݓ‬൅ ‫ ܥ‬෍ ߙ௜ (1)
repeatedly used methods in face detection combining synthetic ʹ
design and feature extraction with machine learning. Further ௜ୀଵ
extension of Viola and Jones's algorithm is a topic of research LBPH gains a local structure of a given image by evaluating
and their method is improved to some extent [14]. New features every pixel with the neighborhood of that pixel. A pixel is
which could benefit detection of faces such as Local Binary chosen as center and the comparison is started. If the value of
Patterns (LBP) [15], Locally Assembled Binary (LAB) [16], the center is smaller it will take the 0 value and 1 value
Multi-Block LBP (MB-LBP) [17] and Normalized Pixel otherwise. Finally, we will have a binary matrix of a given
Difference (NPD)[18] are also used. image. The formal definition of LBP operator is:
There are some other approaches in the field of face ௣ିଵ
detection that have tried other machine learning classifiers [19]. ‫ܲܤܮ‬ሺ‫ݔ‬௖ ǡ ‫ݕ‬௖ ሻ ൌ ෍ ʹ௣ ‫ݏ‬ሺ‫ݐ‬௣ െ݅௖ ሻ (2)
However, in terms of both accuracy and speed these methods are
not good enough in comparison with Viola and Jones’s method. ௣ୀ଴
Moreover, neither of these approaches can acquire good The algorithm of video frame selection is as follows: The
outcome when working with unconstrained faces. proposed method employs a new kind of frame selection
Some of the approaches mentioned to address the face algorithm in videos. In this algorithm, the even frames are
detection and detection problem can only detect one single face selected to do the detection part. With the help of this algorithm
in total of face dataset. Most of the algorithms are not we can experience higher speed which is important for real
appropriate for images which include several faces. Moreover, security tasks. Finally, we can have a list of frames according to
they do not perform well in the existence of blur noise. In this the results of detection to be used as a flag for every video track.
paper, a multiple face detection algorithm using skin color is To improve the precision of the proposed approach, CalcHist
proposed. It is evaluated in the presence of noise on occluded function was used to benefit from RGB colors gained from the
and partially covered faces. The proposed approach has three cropped frames. Amongst all colors obtained with the use of
main steps: 1) human detection using Histogram of Oriented CalcHist, the color we needed was the skin color. Therefore, we
Gradients (HOG) and SVM. In order to reduce false positive omitted other colors and the only color we kept was skin tone
cases, non-maximum suppression is used in terms of maximum related color. Then, with this feature added to the algorithm the
overlapping criteria. accuracy of the proposed algorithm increased substantially.
2) Face detection using Haar cascade classifier. The detected Figure 1 presents the results of this operation.
faces can be transferred to a different window. It is of
importance that detection of faces in this paper is completely
independent from that of human detection. Because sometimes
it happens that only some parts of human body is shown in video
streams [20]. Also, we defined a function to look for faces only
in the regions where a human was detected. Consequently,
number of false faces which are detected and false positive rate
is decreased.
3) Using the trained model, the algorithm recognizes the
people in the video. This face detection step works with Haar
cascade and LBPH. In order to recognize an individual, we have
to train the machine from specific images of the same individual.
The values for HOG parameters are as follows: 1) derivative
mask [-1, 0, +1] centered in 1D. 2) the size of detection window
is 64×128. 3) the size of the cell is 8×8. 4) the size of the block
is 16×16.
Haar cascade is a method based on machine learning where
a cascade function from various positive and negative images
can be trained to detect objects in other images and videos. This
algorithm has four key steps: 1) the selection of Haar features 2)
Figure 1: Using histogram to extract color feature
making integral images 3) training of Adaboost 4) cascading
classifiers.
The facial detection process in the proposed method is
We used SVM for classification of the input data. The task presented in Figure2.
here was to find a functional form for SVM to predict new cases

196

Authorized licensed use limited to: KDU - Gen. Sir John Kotelawala Defence University. Downloaded on December 30,2021 at 19:54:35 UTC from IEEE Xplore. Restrictions apply.
 Is the face
Start detected

Input video Draw a rectangle Input images data set

around the faces and Import Haar
features and face
detection

Setting HOG with pre- Importing .yaml file LBPH to train

trained SVM to detect containing LBPH features Machine and Save
human of training images LBPH features

Face recognizing
Is the human
using LBPH
detected?
recognizer

Draw a rectangle
around them Is the person
recognized?

Using Non-maximum
Suppression and
Loading Haar features of
face detection in
memory
Output name of
person recognized which
was learned from dataset

End

Figure 2: Facial detection process using the proposed method

Figure 3: Facial images cropped from the surveillance videos

197

Authorized licensed use limited to: KDU - Gen. Sir John Kotelawala Defence University. Downloaded on December 30,2021 at 19:54:35 UTC from IEEE Xplore. Restrictions apply.
 III. EXPERIMENTAL RESULTS is that the background in our videos were not homogeneous on
the contrary to the background that was used in other methods.
In order to simulated the proposed algorithm a dataset in a
crowded office environment from the surveillance videos is For this paper, we carried out comparative experiment upon
collected. The dataset contains five different video streams from different datasets. Another contribution is adding another
five different surveillance cameras captured in Khatam feature for skin color which as the result the precision increased
University, Tehran, Iran. The duration of the mentioned indoor as presented in Table 1. In the Table 2 there are differences
videos is about 20 hours (2 working days) and they were among the collected datasets and the dataset used before
captured from different cameras installed in hallways, main improvement [21]. The experimental results show that the
entrance and the restaurant. Furthermore, the resolution of the proposed algorithm is exceptionally capable of detecting
videos is quite low. Therefore, the process of face detection is partially covered faces with headscarves and similar clothing.
considerably difficult as fewer features could be used for The figure 4 shows some cases of detected faces.
detection. The faces are in a variety of positions and usually far
Table 1: The experimental result table
from cameras. Another challenging factor is that from ceiling
cameras mostly we can only get side view faces as opposed to No of
frontal faces which makes this process more difficult and results FP TP
Faces
in lack of accuracy. Proposed
78 9 72
We managed to employ Haar cascade classifier to detect Method
faces from all five videos and then cropped color images [14] 78 14 69
containing faces in order to create the dataset which consists of
around 10,000 detected face images. We used 5,000 facial Cascade 78 32 61
images out of 10,000 images cropped from surveillance videos.
One of the challenges compared to other face detection datasets
Table 2: The differences between datasets

No of images Grayscale images surveillance No of subjects Low resolution

New dataset 5,000 No Yes 20 Yes

[14] 1,000 Yes No 2 No

and reduce false positive results, we managed to use CalcHist

function in order to extract RGB colors and utilized obtained
features to improve the accuracy of face detection. A large
dataset of a crowded office environment in Middle East is
collected and used for evaluation of the proposed model. The
experimental results show that the proposed method has
acceptable results.
REFERENCES
[1] J. Li, S. Karmoshi, and M. Zhu, "Unconstrained face detection based
on cascaded Convolutional Neural Networks in surveillance video,"
in ICIVC, 2017.
[2] J. Rajeshwari, K. Karibasappa, and M. T. Gopalkrishna, "Adaboost
modular tensor locality preservative projection: face detection in
video using Adaboost modular-based tensor locality preservative
projections," IET Computer Vision, vol. 10, pp. 670-678, 2016.
[3] P. F. De Carrera and I. Marques, "Face detection algorithms,"
Master's thesis in Computer Science, Universidad Euskal Herriko,
2010.
[4] J. Rajeshwari, K. Karibasappa, and M. GopalKrishna, "Survey on
Figure 4: the result of face detection and detection skin based face detection on different illumination, poses and
occlusion," in IC3I, 2014.
CONCLUSION [5] S. L. Phung, A. Bouzerdoum, and D. Chai, "A novel skin color model
in ycbcr color space and its application to human face detection," in
In this paper, a novel computer vision system for face Image Processing, 2002.
detection in low resolution surveillance videos containing [6] P. Viola and M. Jones, "Rapid object detection using a boosted
traditional Middle Eastern clothing including the headscarf is cascade of simple features," in Computer Vision and Pattern
presented. In order to improve accuracy of facial detection

198

Authorized licensed use limited to: KDU - Gen. Sir John Kotelawala Defence University. Downloaded on December 30,2021 at 19:54:35 UTC from IEEE Xplore. Restrictions apply.
 Detection, 2001. CVPR 2001. Proceedings of the 2001 IEEE
Computer Society Conference on, 2001, pp. I-I.
[7] B. Majidi and A. Bab-Hadiashar, "Land Cover Boundary Extraction
in Rural Aerial Videos," in MVA, 2007, pp. 311-314.
[8] B. Majidi, J. C. Patra, and J. Zheng, "Modular interpretation of low
altitude aerial images of non-urban environment," Digital Signal
Processing, vol. 26, pp. 127-141, 2014.
[9] A. Shamisa, B. Majidi, and J. C. Patra, "Sliding-Window-Based
Real-Time Model Order Reduction for Stability Prediction in Smart
Grid," IEEE Transactions on Power Systems, vol. 34, no. 1, pp. 326-
337, 2019.
[10] A. Mansouri, B. Majidi, and A. Shamisa, "Metaheuristic Neural
Networks for Anomaly Detection in Industrial Sensor Networks with
Packet Latency and Jitter for Smart Infrastructures," International
Journal of Computers and Applications, 2019.
[11] B. Majidi and B. Moshiri, "Industrial assessment of horticultural
products' quality using image data fusion," in Proceedings of the
International Conference on Information Fusion, 2003, pp. 868-873.
[12] S. Sanaei, B. Majidi, and E. Akhtarkavan, "Deep Multisensor
Dashboard for Composition Layer of Web of Things in the Smart
City," in 2018 9th International Symposium on Telecommunications
(IST), 2018, pp. 211-215.
[13] D. Nedaei, P. Khanzadi, B. Majidi, and A. Movaghar, "Inbound e-
marketing using neural network based visual and phonetic user
199
Authorized licensed use limited to: KDU - Gen. Sir John Kotelawala Defence University. Downloaded on December 30,2021 at 19:54:35 UTC from IEEE Xplore. Restrictions apply.
experience analytics," in 2018 4th International Conference on Web
Research (ICWR), 2018, pp. 12-18.
[14] T. Mita, T. Kaneko, and O. Hori, "Joint haar-like features for face
detection," in ICCV, 2005.
[15] H. Jin, Q. Liu, H. Lu, and X. Tong, "Face detection using improved
LBP under Bayesian framework," in ICIG'04, Third International
Conference on, 2004.
[16] S. Yan, S. Shan, X. Chen, and W. Gao, "Locally assembled binary
(LAB) feature with feature-centric cascade for fast and accurate face
detection," in Computer Vision and Pattern Detection, 2008. CVPR
2008. IEEE Conference on, 2008, pp. 1-7.
[17] L. Zhang, R. Chu, S. Xiang, S. Liao, and S. Z. Li, "Face detection
based on multi-block lbp representation," in International conference
on biometrics, 2007, pp. 11-18.
[18] S. Liao, A. K. Jain, and S. Z. Li, "A fast and accurate unconstrained
face detector," IEEE transactions on pattern analysis and machine
intelligence, vol. 38, pp. 211-223, 2016.
[19] S. C. Brubaker, J. Wu, J. Sun, M. D. Mullin, and J. M. Rehg, "On the
design of cascades of boosted ensembles for face detection,"
International Journal of Computer Vision, vol. 77, pp. 65-86, 2008.
[20] https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients
[21] https://github.com/ITCoders/Human-detection-and-
Tracking/raw/master/results/HUMAN%20DETECTION%20ANDa
DETECTION.pdf