See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/360884792

Hybrid Based Model Face Shape Classification Using Ensemble Method for
Hairstyle Recommender System

Chapter · January 2022

DOI: 10.1007/978-981-16-9480-6_7

CITATIONS READS

3 484

6 authors, including:

Mr Abdullah Ali Hussain

James Cook University Brisbane Inje University
15 PUBLICATIONS 75 CITATIONS 26 PUBLICATIONS 223 CITATIONS

SEE PROFILE SEE PROFILE

Sikandar Ali Shigri Hee-Cheol Kim

Inje University Inje University
24 PUBLICATIONS 167 CITATIONS 176 PUBLICATIONS 2,079 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Mr Abdullah on 29 May 2022.

The user has requested enhancement of the downloaded file.

Hybrid based model face shape classification using
Ensemble method for hairstyle recommender system

Abdullah1 ,Ali Hussain1 , Ali Sikandar1, Hee-Cheol Kim2, Mangal Sain3,*

Satyabrata Aich4,*
1Institute of Digital Anti-Aging Healthcare, Inje University, Gimhae 50834, Korea;
Abdullahbaltistani@gmail.com

2
College of AI Convergence/Institute of Digital Anti-Aging Healthcare/u-HARC,
Inje University, Gimhae 50834, Korea;heeki@inje.ac.kr

3 Division of Computer Engineering, Dongseo University, Busan,

Republic of Korea

4Wellmatix Co., Ltd., Changwon 51395, Korea; satyabrataaich@gmail.com

Abstract:
The face shape classification has a great impact on fashion industry, it helps to select the
appropriate hairstyle, makeup style and eyewear frames for an individual on the base of their face
shape and looks. Computer-aid face shape classification system help to enhance the beauty and
attractiveness of system, furthermore it minimizes the cost and time spent by experts. In this
paper, we have present machine learning and engineering-based method to classify the face shape
to recommend hairstyle. The method proposed combines facial landmarks, the geometrical, and
the pre-train model features to classify facial shapes into four categories. The method can extract
wide range of characteristics and select the most effective attributes to be evaluated using distinct
method of machine learning. The overall accuracy of proposed method is 90%. This is an
innovative method of face shape classification, which shows good performance against state of
the art.

Keywords: Face shape, Facial feature, pre-train model, Facial landmark, Feature extraction

Introduction:
Identification and recognition of face to study facial attribute is an essential part of
personality development. Face shapes have an important role in fashion and cosmetic
industry to choose right products for an individual according to their shape. Face shape
type have great impact, when we have to select attractive hairstyle and eyewear glass
frame to enhance the beauty of attractiveness. The facial attribute is also used to study
about the health and emotion recognition of a person. Beauty expert divide face shape
into five categories namely round shape, oval shape, square shape, oblong shape, and
heart shape [1]. The manually process to identifying and classifying the face shape
using image processing is time consuming and tedious task, however automatic
computer-aid designed system save the time, effort, and expenses of expert. In this
work, we present a face shape classification based on features extracted manually as
well as self learned by machine and used Boruta algorithm to select valuable features,
furthermore, we merged theses feature to train the model. According to beauty expert
every face is distinctive and is defined based on the dimension of forehead, jaw
cheekbones, length ratio between major axis and minor axis. Transfer learning method
is adopted to perform experiment, in which already pre-train model is used to extract
feature and machine learning technique is used for classification. The rest of paper is
organized as followed. Section 2 illustrates the related work, which shows previous
technique used by people to solve this problem. Section 3 explain the method, we have
adopted in this work. Section 4 illustrate the result performance and discussion. Section
5 describe the conclusion and future work.

Related work:
The human face is the major and important part of a whole body. Beauty is a part of
human nature, and this beauty and attractiveness definitely depends on individual face.
Over the last decade face has evolved into an extremely active research area which
focuses on the detection and tracking of faces such as facial feature detection and
recognition, similarly facial expression, and emotional recognition. The development
of advanced machine learning methods and statistical machine learning techniques have
made a numerous application, including image retrieval, biometrics, surveillance
virtual speech recognition and online marketing and many more [2].

Facial image analysis is extensively studied in literature to recognized face, age, and
gender classification, face shape and to build recommender system for attractiveness.
The knowledge of face shape is important to choose appropriate hairstyle, to analyse
face shape the width of cheekbone and jawline, similarly length of a face is important
to determine its proportions [3]. Facial look highly influences by environmental
circumstances like used of cosmetics, lightning condition, and head pose, furthermore
aesthetics surgery or long-term effect like as weight gaining, and aging process have a
major impact on the appearance of our face [4]. Active Appearance Model (AAM) is
an effective method for extraction of statistical features. The model is extensively
utilized in facial shape identification to identify facial expressions. Fashion industry
has developed a several products to enhance the attractiveness of various parts of a face.
Mostly of the people adopt the similar fashion method to analyse the face, feature
extraction and classification [5]. Fisher face approach` that is used for face detection
and recognition. This model is basically developed for the application of mobile face
recognition [6], In addition, another technique based on cloud environment using
eigenface help the people to pick an appropriate hair style, eyebrows and glassware
based on their face type [7]. Female hairstyle recommender system is studied [8] on a
small dataset and used different machine learning algorithm for comparison.
 In the last year deep learning model, specifically Convolution neural network (CNN)
have get more attention in image processing and computer vision field. CNN is
powerful technique to extract various kind of feature from raw image data [9]. The
CNN based different architecture is also existed to handle various kind of problem. In
addition, facial attribute extraction using deep learning technique is split holistic as well
as part-based techniques. The holistic -based approach make used of relationships
between features that can be help in finding more discriminative signals, however the
part-based method assists in detecting localization features [10].

Recently Ling and wang [11] conducted experiment by considering gender, age,
occupation, customer rating and skin colour to developed automatic recommender
system. It is important to know any beautician about the client face before to apply
make-up or hairstyle. The facial feature like face shape, size and colour help to choose
best make-up, eyelashes, and hair-cut style for an individual face. In this work, we have
proposed computer-based face shape classification for hairstyle recommender system
using hybrid model. To build good system, it should be ensured to consider maximum
features to get better result, however; various types of facial, mouth, nose and eyes
attribute are essential to determine, which hairstyle would be suitable for specific face
type. We try to consider all possible features to build robust and accurate system on a
custom dataset. The motivation behind this work is to developed computer-based
identification and recognition of face, furthermore, processed it and recommend
suitable hairstyle according to there look.

3. Methodology:
Face is most prominent and eye-catching part in our body. Its gives lot of information
about heath, emotion, attractiveness, gender, age, personality and many more. To
analyse face characteristic from different perspective is gaining more attention now-a-
days. The method, we have choose to analyse the shape of face in personality
development perspective. We used two feature extraction method and merged both
features, furthermore, we used feature selection method to refine best feature to trained
model.

Fig. 1 workflow of proposed model

3.1 Data pre-processing :

In our dataset four types of face shape are identified by facial expert. They define
few rules and criteria, on that basis the face is categorized into multiclass. To perform
experiment the raw image data is cropped using frontal face Harcasacde [12] OpenCV
library, which identify the corner of face and make rectangular bounding box,
furthermore it removes redundant part from the image. In addition, we resize the whole
images 256 x 256 for further processing. The training dataset consist of 400 images of
both male and female as shown in fig. 2.

(a) Round-shape face (b) Oblong-shape face (c) Square-shape face (d) Oval-shape face

Fig. 2 Face type define by expert

3.2 Hand-craft facial features extraction:
To extract feature from face manually, the facial landmarks (68 points) [13] is used,
the main features which includes length and width ration of face, length of cheekbone
and forehead, angle between different facial landmark point. To determine the
geometric shape of the face and facial part of each organ needs to be taken into
consideration. The 17 facial landmark point are representing boundary of face
sssimilarly 12 landmark points are considered to address eye feature. Additionally, the
landmarks are unable to determine the area of the forehead, so we employ the k-mean
clustering algorithms. It makes differentiation based on colours of skins and hair.
Recently hybrid-based face shape classification [14] uses 19 geometrical landmark
features, so we add up more feature to get more accurate result. In this way we have
calculated overall 35 features to train model. Fig. 3 demonstrate the (a) original image,
(b) 68 points landmark detection and (c) feature extraction.

(a) Original image (b) Face detection (c) Features

and landmark

Fig. 3 Handcraft feature extraction

3.3: Eye feature extraction:

 Eye features have an importance due to its application in cosmetic industry, eye wear
glass frame and makeup style.

Fig.4 Eye landmark

The OpenCv library provide eye landmark to extract eye features. The 12-point
landmark shown in fig.4 gives deep feature of both eyes. So to calculate these feature,
we used below equations.

𝑑(𝑙1 ,𝑙4 ) + 𝑑(𝑙5 ,𝑙6 )

𝑅= (1)
2𝑑(𝑙1 , 𝑙4 )

𝑑(𝑙1 ,𝑙4 )
𝑐1 = (2)
2

𝑑(𝑙7 ,𝑙10 )
𝑐2 = (3)
2

𝑐 + 𝑐
𝑅= 1 2 (4)
2
Equation (2) and (3) gives the center of the individual eyes, furthermore, equation (4)
represents the center point, which is equal distance from both eyes. In this work, we
have considered all theses kind of feature to improve output performance

3.4. Feature extraction based on pre-train model:

The deep learning model particularly CNN based model is widely utilized to tackle
the problem of image classification. In this study, Vgg19 is used as a pre-train model
to identify facial characteristics to improve the performance of shape classification.
This model is already trained on the dataset of ImageNet for 1000 classes, we used our
custom dataset to extract features, in consequence, it generates a lot of features which
are refined by using feature selection method Boruta and keep only useful attributes to
trained model.

3.5. Classification:
The pre-train model and handcraft feature are combined into a single feature vector,
then it splits into training and testing set. In this study we have used multiple classifiers
to analyse the output performance. We also used ensemble method to get final output.
The different machine learning classifier namely decision Tree (DT), random forest
(RF), gradient-boosting machine (GBM), extreme gradient boosting (XGB) and
multilayer perceptron (MLP) are used to classify the face shape into four classes,
furthermore the individual output results are ensemble to get stable and more accurate
output. The proposed model architecture and entire process is shown in fig.1.

4. Result and discussion:

The experimental result reveal that the hand pick feature and machine self-learn
feature are merged to give effective result for face classification. In our method, we
extract more features as compared to existing literature and we used ensemble method.
These improvements enhance our model performance. The confusion matrix of final
output of our model is mentioned in table-I.

Table 2. shows the result of several models. In the integrated multimodal eye and
face shape attribute [15] and model inception v3 [16] both gives good output
performance, which is 85% and 84.4% respectively. The inception V3 used similar
strategy with a smaller number of futures, they used histogram-oriented gradient (HOG)
and 68-point landmarks features but output performance of our proposed model is 2%
higher than inception v3 model. In our model, we used VGG19 instead of inception v3,
similarly we used har-cascade instead of HOG. The overall result of our model is 86.5%
which is quite good as compared to currently other existing model This system is more
reliable and robust for hair stye recommender system.

Table 1. Confusion matrix of ensemble method

Table 2. The different method used for face shape classification with their accuracy.

Methods Accuracy
Hybrid approach VGG and SVM [14] 70.3%
Active Apparency Model (AAM), segmentation and SVM [2] 72%
Geometric Features [17] 80%
Corelation and similarity-based region and fractal parameter [18] 80%
Model inception V3 [16] 84.4%
Integrated multimodal face shape and eye attribute [15] 85%
Proposed Model 86.5%

The various machine learning classifier is used to evaluate our custom data set. Table
3. illustrate the accuracy against each classifier. The table shows the higher accuracy of
RF, XGB and GBM are 85%, 86% and 87.5% respectively. We used soft voting
ensemble method to get stable and convenient output result.
 Table 3. Accuracy of different classifier on our model

Classifier Accuracy
Decision Tree 72%
Random forest (RF) 85%
Multilayer perceptron (MLP) 83%
Gradient boosting machine (GBM) 87.5%
eXtreme Gradient Boosting (XGB) 86%
Ensemble Method 86.5%

5. Conclusion:
In this paper, face shape classification based on a hybrid model is presented to developed
hairstyle recommender system. The proposed model reveals the efficient performance in
the identification and classification of face shapes. It is a tedious and time-consuming task
to identify face shapes by any experts. The development of this efficient framework helps
individuals to select any hairstyle based on face shape without any facial beauty expert.

The approach, we adopted in this researched is different from traditional method because I
this work, we used 35 features extracted by using feature engineering technique and 466
features from automatically learned based method, furthermore, we have used vgg19 to get
facial feature and we have get 86.5% output result, which is 2 % higher score than the other
state-of-art models. In this way, our model result and performance are better to classify face
shape, whereas it is more suitable for hairstyle recommender system. Furthermore, it is still
challenging task due to lack of training size and low-quality data. We can increase it
performance by increasing quality and quantity of training data.
6. Future work:
In future, this work can be extended by developing graphical user interface and make
supportable for mobiles phones. To improve the result and model, we can increase the
number of training dataset to improve its performance. Furthermore, used an artificial
neural network (ANN) instead of using a simple machine learning classifier.
References

1. Derrick, J. Is Your Face Round, Square, Long, Heart or Oval Shaped? Available
online: https://www.byrdie.com/is-your-face-round-square-long-heart-or-oval-
shaped-345761 (accessed on 1 June 2019)
2. Garcia, C., Ostermann, J. and Cootes, T., 2008. Facial image processing. EURASIP
Journal on Image and Video Processing, 2007, p.070872.

3. Brewer, T. The Most Flattering Haircuts for Your Face Shape. Available
online: https://www.thetrendspotter.net/haircuts-for-face-shape/ (accessed on 1 March
2020).
4. Eidinger, E., Enbar, R. and Hassner, T., 2014. Age and gender estimation of unfiltered
faces. IEEE Transactions on Information Forensics and Security, 9(12), pp.2170-2179.
5. Jain, A.K. and Li, S.Z., 2011. Handbook of face recognition (Vol. 1). New Yor
k: springer.
6. Siregar, S.T.M., Syahputra, M.F. and Rahmat, R.F., 2018, February. Human fa
ce recognition using eigenface in cloud computing environment. In IOP Confer
ence Series: Materials Science and Engineering (Vol. 308, No. 1, p. 012013).
IOP Publishing.
7. Pasupa, K., Sunhem, W. and Loo, C.K., 2019. A hybrid approach to building
face shape classifier for hairstyle recommender system. Expert Systems with Ap
plications, 120, pp.14-32.
8. Wu, Y. and Ji, Q., 2019. Facial landmark detection: A literature survey. Intern
ational Journal of Computer Vision, 127(2), pp.115-142.
9. Sunhem, W., Pasupa, K. and Jansiripitikul, P., 2016, May. Hairstyle recommendation
system for women. In 2016 Fifth ICT International Student Project Conference (ICT-
ISPC) (pp. 166-169). IEEE.
10. Zheng, X., Guo, Y., Huang, H., Li, Y. and He, R., 2020. A survey of deep facial
attribute analysis. International Journal of Computer Vision, 128(8), pp.2002-2034.
11. Liang, Z. and Wang, F., 2016. Research on the Personalized Recommendation
Algorithm for Hairdressers. Open Journal of Modelling and Simulation, 4(3), pp
.102-108.
12. Wilson, P.I. and Fernandez, J., 2006. Facial feature detection using Haar
classifiers. Journal of Computing Sciences in Colleges, 21(4), pp.127-133.
13. Facial Landmark Detection Simplified with OpenCv. Available online :
https://www.analyticsvidhya.com/blog/2021/07/facial-landmark-detection-simplified-
with-opencv/ (accessed on 6 July 2021).
 14. Alzahrani, T., Al-Nuaimy, W. and Al-Bander, B., 2019, December. Hybrid feature
learning and engineering based approach for face shape classification. In 2019
International Conference on Intelligent Systems and Advanced Computing Sciences
(ISACS) (pp. 1-4). IEEE.
15. Alzahrani, T., Al-Nuaimy, W. and Al-Bander, B., 2021. Integrated Multi-Model Face
Shape and Eye Attributes Identification for Hair Style and Eyelashes
Recommendation. Computation, 9(5), p.54.
16. Tio, A.E., 2019. Face shape classification using inception v3. arXiv preprint
arXiv:1911.07916.
17. Jeong, K.M. and Kim, J.H., 2012. Face classification and analysis based on geometrical
feature of face. Journal of the Korea Institute of Information and Communication
Engineering, 16(7), pp.1495-1504.
18. Bansode, N.K. and Sinha, P.K., 2016. Face shape classification based on region
similarity, correlation and fractal dimensions. International Journal of Computer
Science Issues (IJCSI), 13(1), p.24.

View publication stats