An Improved Method for Facial Features Extraction in Images

Renn Contreras, Oleg Starostenko, Leticia Flores Pulido

CENTIA, Computer Science Department, Universidad de las Amricas Puebla,
Cholula, Puebla, 72820, Mxico,
renan.contrerasgz@udlap.mx, oleg.starostenko@udlap.mx, leticia.florespo@udlap.mx

Abstract
When it is necessary to analyze a persons face,
whether it is for recognizing pathologies, emotions, or
states of mind, it becomes necessary to obtain a maximum
of information of the facial characteristics that especially
reflect these aspects. These characteristics are
principally, the mouth, eyes and eyebrows. The start of
the analytic process, once the face and the facial feature
to be analyzed have been divided into segments, consists
in applying an algorithm to said feature for the detection
of its edges. The most used edges algorithms are; SUSAN,
Canny, Sobel, and Roberts, among others. These
algorithms function excellently when the edges are fairly
well defined, but in the case of faces in color, where the
transitions are not clearly marked, and when there are
many imperfections and shadings, the above mentioned
algorithms generate incomplete contours in a great
number of cases, leading to errors in high level analysis.
This article presents a new methodology based on the
Canny algorithm which allows us to obtain the edges of
the mouth with much greater information than the other
above mentioned algorithms, which makes it more
adequate for the originally stated objective. The method
has been tested detecting the outline contour of the mouth
and using the databases of facial images, MMI Facial
Expression Database compiled by M. Pantic & M. F.
Valstar" and "A.M. Martinez and R. Benavente. The AR
Face Database. CVC Technical Report #24, June 1998".

1. Introduction
When it is necessary to obtain the contour of an object
in a certain image that has been previously segmented
using one of the existing methods [8], [9], for later
description and analysis, a edge algorithm is generally
employed, which can be based on derivatives, such as the
denominated methods Canny [1], Sobel [2], Laplacian
[3], or based on the internal area of a nucleus such as the
SUSAN method [4]. These algorithms function
exceptionally well when the contours are reasonably
defined, i.e., when the intensity difference between the
object and its background is prominent. Fig. 1 shows the

excellent quality of the contours obtained using two of

these algorithms (Canny and SUSAN), when they are
applied to a test image. As can be observed in this figure,
the contours obtained are clearly defined, but even so,
note the noise generated around some of the figures.

a)

b)

c)

Figure 1. a) Original image, b) Contour image

obtained with Canny, c) Contours with SUSAN

However, in certain applications, such as facial

analysis, for the purpose of detecting emotions,
pathologies or state of mind, [5], [6], [10], [11], using
analysis of the deformation or movement of the main
elements of the face, (mouth, eyes and eyebrows), the
afore mentioned algorithms in many instances generate
incomplete contours which do not contain all the relevant
information for the analysis to be performed. It is
important to mention that the face, as well as its principal
components, should be previously segmented [12]. Fig. 2
shows the contours obtained from a previously segmented
characteristic from a facial image.

chosen for the image to be analyzed, in this case the

mouth. This algorithm utilizes two thresholds and
generates well defined contours if the transitions between
the background and the object contain elevated gradients;
if this is not the case, the contour will not be detected,
such as can be seen in the upper right hand contour and in
the lower contour of the lips. Fig. 2a shows another
example of a contour obtained with the Canny algorithm,
observe that even with good results, there exists partial
information for the lower lip.

a)
a)
b)
b)
Figure 2a. a) Original image, b) Contour obtained with
the Canny algorithm.

c)

d)

e)

Figure 2. a) Original image, b) Contour obtained with the

Sobel algorithm, c) Contour obtained with the Roberts
algorithm, d) Contour obtained with SUSAN, e) Contour
obtained with Canny.
As it can be observed, from the results shown in Fig. 2,
the Sobel, Roberts and SUSAN algorithms generate
poorly defined contours of little utility. The best
functioning algorithm is the Canny with parameters

As has been mentioned, in applications where

information of the principal facial components is
required, whether for detection of emotions, pathologies
or states of mind, in many instances the algorithms
presented give partial information which upon processing
leads to errors in high level interpretation.
The purpose of this article is to present a new
methodology for obtaining the outline edges of the
principal characteristics of a face, which will be the base
for its later representation. The methodology presented is
hybrid, is based on the Canny algorithm, is applicable to
color images and improves the contours obtained using
the modification of the histogram and the emphasis of the
RGB components and the black and white image. The
contour of these emphasized images is obtained and these
are added to the contour of the original image thus
obtaining the final representation.
The algorithm has been tested utilizing images of men
and women with manual segmentation of the
characteristics, obtained from the databases of facial
images [7], [13].

2. Proposed Methodology
A key aspect of the methodology is the fact that the
histogram analysis of the diverse images of mouths
manually segmented, tells us that the relevant information
is located between 60% and 90% of the intensity values,

so that the algorithm processes and emphasizes this range

for each individual color component. The average
intensity histogram of the images is shown in Fig. 3.
The fundamental idea behind the method consists in
obtaining, in individual form and through use of the
Canny algorithm, the edges of the RGB components of
the original image. Previously, these components have
their histogram expanded and then a gamma correction is
performed on them with values of 0.4 and 0.8 plus
compressing the histogram by 80%. Finally, the edges
obtained are added to the edges of the original image
transformed to gray scale from which the histogram is
expanded to 100%.
The algorithm basically consists of the following
steps:
a). Conversion of the color image to gray scale
(ImRGB2Gray) and smoothing using a Gaussian mask
filter of 20 and a deviation of 0.1.
b). Given that the analysis of the histogram, performed
on 20 images from the database [7], Fig. 3, shows that
the majority of the pertinent information is to be found
between 60% and 90% of the intensity values, this
range of the histogram of ImRGB2Gray is expanded to
100%, thus obtaining the ImBWHE image.

fundamentally improves the transitions between the

background and the object, allowing better edges
detection. The images obtained are denominated
ImadjRCH8, ImadjRCH4, ImadjGCH8, ImadjGCH4,
ImadjBCH8 and ImadjBCH4. The process described is
shown in Fig. 5.
d) The edges of the images ImRGB2Gray, ImBWHE,
ImadjRCH8, ImadjRCH4, ImadjGCH8, ImadjGCH4,
ImadjBCH8 and ImadjBCH4 are obtained and added
using the Canny algorithm for edges, and thus
obtaining an improved contour.
e) Finally the image is cleaning up using algorithms
for contour cleaning or for component length. This
step may not be required for the applications we are
dealing with given that in many cases the additional
information of the contour corresponds to wrinkles
that can be considered part of the expression and
improve the facial analysis.

Figure 4. Gamma correction curve for emphasizing

images R, G and B.

b)

a)
Figure 3. An average histogram of 20 images where it
is shown that the relevant information is located between
60% and 90% of the intensity values.
c). The R, G and B components are obtained from the
original image, ImR, ImG and ImB and then their
histogram is expanded to 100%, obtaining the images
ImRHE, ImGHE and ImBHE.
d). A double gamma correction is performed on the
ImRHE, ImGHE and ImBHE images, Fig. 4, with
parameters of 0.4 and 0.8 and a compression of the
histogram in the range of 60% to 80%. The absolute
range values to which the histogram is compressed are
not significant, although the range of 20% between
them is. This compression value was obtained
experimentally. The gamma adjustment performed

c)

d)

Figure 5. a) Original histogram component G. b)

Histogram expanded to 100%. c) Histogram compressed
with gamma = 0.8. d) Histogram compressed with
gamma = 0.4

3. Obtained Results

a)

The results obtained with various images are presented

below, in Fig. 6, 7, 8, and 9, resulting from the
segmentation of faces from the database. These results are
compared with the results obtained utilizing the Canny
algorithm, since for this application this algorithm gives
better results than the SUSAN, Roberts and others.

b)

c)
a)
Figure 8. a) Original object, b) Contour obtained with
Canny, c) Proposed contour algorithm.

b)

a)
c)

Figure 6. a) Original object, b) Contour obtained with

Canny, c) Proposed contour algorithm.

a)

b)

c)
Figure 9. a) Original object, b) Contour obtained with
Canny, c) Proposed contour algorithm.

b)

c)

Figure 7. a) Original object, b) Contour obtained with

Canny, c) Proposed contour algorithm.

The examples shown indicate that in the majority of

cases the proposed algorithm generates greater
information about the facial characteristic than with just
the Canny algorithm. This is of great utility in
applications where a study of the deformation of the
object is required.
Finally, Fig. 10 shows the contour obtained by
overlying the original image. In this figure you can
observe the adequateness of the contour generated by the
proposed algorithm.

Figure 10. Contour obtained by overlying the original

image.

4. Evaluation and discussion

As can be observed in the examples shown, the
proposed algorithm generates a contour of the object
much more complete than the algorithms of the derivative
type such as the Robert, Sobel or Canny, or of the nucleus
evaluation type such as the SUSAN. Although it can be
seen that the proposed method generates additional
contours, these do not constitute a problem when the
contour is used in applications such as facial analysis
where it requires a study of the deformation of the facial
characteristics for diverse ends in a sequence of images.
The performance of the algorithm was tested on
images expressing different emotions, taken from the
database compiled by M. Pantic & M. F. Valstar [7] and
segmenting manually the pertinent facial characteristic. In
all of the cases the result is either better or equal to the
result obtained by using only the Canny algorithm. In no
case did the algorithm obtain a lower quality contour.
The algorithm can be evaluated visually or by relating
the sum of the length of the detected contours to the sum
total of the real contours of the object traced manually.
This is expressed in the following equation:

Length of detected segments

Effectiveness =

(1)
Length of real segments

For the algorithm shown, the effectiveness is superior

to 0.94 while for the Canny algorithm, this falls in some
cases to below 0.80, as can be observed in the example in
Figure 8.

4. Conclusions
In certain very specific applications, such as facial
analysis, focused on determining emotions, pathologies
or state of mind, where it requires an analysis of the
deformation of the main facial characteristics, it is
particularly important to obtain the greatest amount of

information possible on the characteristics to be analyzed,

to subsequently obtain a representation that will serve as
a base for high level analysis. Obtaining the contour of
the required characteristics using the algorithms such as
Susan, Roberts, Canny and others, many times generate
incomplete and inadequate contours for the application.
In this article, we present a hybrid methodology to obtain
an improved contour of a color image of the mouth which
obtains more detailed characteristics than those provided
by derivative algorithms and nucleus algorithms such as
the SUSAN. This method uses the Canny contour
algorithm and the emphasis of the RGB color
components of the image.
The results obtained, using the databases, MMI Facial
Expression Database [7], and The AR Face Database
[11], show a significant improvement in the
representation.

5. References
[1] Canny, J., A Computational Approach To Edge Detection,
IEEE Trans. Pattern Analysis and Machine Intelligence, 8:679714, 1986.
[2] Sobel, I., Feldman,G., "A 3x3 Isotropic Gradient Operator
for Image Processing", presented at a talk at the Stanford
Artificial Project in 1968, unpublished but often cited, orig. in
Pattern Classification and Scene Analysis, Duda,R. and Hart,P.,
John Wiley and Sons,'73, pp271-2
[3] D. Marr, E. Hildreth, Theory of Edge Detection,
Proceedings of the Royal Society of London. Series B,
Biological Sciences, Vol. 207, No. 1167 (Feb. 29, 1980), pp.
187-217
[4] S.M. Smith and M. Brady, SUSAN - a new approach to
low level image processing, International Journal of Computer
Vision, Vol. 23(1), 45-78, 1997.
[5] Jeffrey F. Cohn, Takeo Kanade, Use of Automated Facial
Image Analysis for Measurement of Emotion Expression, The
handbook of emotion elicitation and assessment, J. A. Coan & J.
B. Allen (Eds.), Oxford University Press Series in Affective
Science. New York: Oxford.
[6] Carlos Busso, Zhigang Deng , Serdar Yildirim, Murtaza
Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich
Neumann, Shrikanth Narayanan, Analysis of Emotion
Recognition using Facial Expressions, Speech and Multimodal
Information, Proc. of ACM 6th International Conference on
Mutlmodal Interfaces (ICMI 2004), State College, PA, Oct
2004.
[7] M. Pantic, M.F. Valstar, R. Rademaker and L. Maat, "Webbased Database for Facial Expression Analysis", Proc. IEEE
Int'l Conf. Multmedia and Expo (ICME'05), Amsterdam, The
Netherlands, July 2005.
[8] Dzung L. Pham, Chenyang Xu, and Jerry L. Prince (2000):
Current Methods in Medical Image Segmentation, Annual
Review of Biomedical Engineering, volume 2, pp 315-337.

[9] N. R. Pal, S.K. Pal, A review on image segmentation

techniques, Pattern Recognition, vol. 26, pp. 1277-1291, Sep.
1993.
[10] Maja Pantic & Leon Rothkrantz, Automatic Recognition of
Facial Expressions and Human Emotions. Knowledge Based
Systems Section, Faculty of Technical Mathematics and
Informatics, Delft University of Technology.
[11] Hatice Gunes, Massimo Piccardi, Tony Jan. Face and Body
Gesture Recognition for a Vision-Based Multimodal Analyzer.
the Pan-Sydney Area Workshop on Visual Information
Processing (VIP2003), Sydney. Conferences in Research and
Practice in Information Technology, Vol. 36.
[12] Hjelms, E., Face Detection: A Survey Computer Vision
and Image Understanding 86, 236-274, 2001
[13] "A.M. Martinez and R. Benavente. The AR Face Database.
CVC Technical Report #24, June 1998".