Fractals, Vol. 10, No.

1 (2002) 47–52
c World Scientific Publishing Company

FRACTALS IN EMOTIONAL FACIAL

EXPRESSION RECOGNITION

TAKUMA TAKEHARA
Department of Psychology, Doshisha University
Kyoto 602-8580, Japan
takehara@psychology.doshisha.ac.jp
FUMIO OCHIAI
Department of Human and Cultural Studies
Tezukayama University, Nara 631-8501, Japan
NAOTO SUZUKI
Department of Psychology, Doshisha University
Kyoto 602-8580, Japan

Received June 5, 2001; Accepted September 7, 2001

Abstract
Following the Mandelbrot’s theory of fractals, many shapes and phenomena in nature have been
suggested to be fractal. Even animal behavior and human physiological responses can also be
represented as fractal. Here, we show the evidence that it is possible to apply the concept of
fractals even to the facial expression recognition, which is one of the most important parts of
human recognition. Rating data derived from judging morphed facial images were represented
in the two-dimensional psychological space by multidimensional scaling of four different scales.
The resultant perimeter of the structure of the emotion circumplex was fluctuated and was
judged to have a fractal dimension of 1.18. The smaller the unit of measurement, the longer
the length of the perimeter of the circumplex. In this study, we provide interdisciplinarily
important evidence of fractality through its application to facial expression recognition.

Keywords: Facial Expression Recognition; Morphing; Multidimensional Scaling.

47
48 T. Takehara et al.
1. INTRODUCTION
Many studies have demonstrated that the structure
of emotional facial expression recognition can be
represented as a circumplex in the two-dimensional
psychological space, consisting of valence (pleasure-
displeasure) as one dimension and the level of
arousal as the other.1 – 7 Further experiments on the
circumplex structure of facial expression recogni-
tion showed a more detailed structure than before.6
According to the experiments, morphed facial im-
ages, lying between two prototype facial expressions
(e.g. Happiness and Surprise) and prototype faces
were represented in the two-dimensional psycholog-
ical space in a roughly circular order. Interestingly,
complex morphed facial images with more than one
emotional component showed a rather complex dis-
tribution rather than simply between their proto-
types corresponding to their morphing proportions.
Although the structure of emotional facial
expression recognition has been represented in the
two-dimensional space, it was interesting to note
that facial images formed a roughly circular struc-
ture. In other words, these experiments showed a
somewhat different distribution of facial images. In
addition, results from Takehara and Suzuki (1997)6
showed complex and wiggled arrangements between
two prototype faces on a circumplex rather than
smooth and continuous distribution. Their results
also showed that prototypes and morphed facial
images were located in somewhat different loca-
tions. The fearful face, for example, was located
near the surprised face in one experiment and was
not in the other experiment even though the two
experimental conditions were the same. If the two-
dimensional psychological space is continuous and
homogeneous, one would expect that the changes in
the representative trajectory running from one pro-
totype to another around a circumplex would be
smooth, and geometric relationships between the
prototype faces would be mapped in the continuous
space (see Fig. 1). But, why did we get such com-
plex and fluctuated representations? This kind of
fluctuation may mask the possible fractality of the
shape.
Fractals are a concern of new geometry, whose
object is to describe a great variety of natural
structures that have irregularities of various sizes.
Until recently, various shapes or surfaces in nature
have been explained as fractal in terms of their self-
similarity.8 – 12 Furthermore, it has been reported
that the feeding behavioral pattern of Drosophila
Prototype Prototype
face face
Morphed
face
Fig. 1 Schematic representation of a trajectory in a con-
tinuum running from one prototype to another depicted in
the previous studies. The change is fluctuating rather than
continuous.
melanogaster13 and the diving duration of whales14
both showed a temporal fractal. In human phys-
iology, the brain wave is proved to have a fractal
dimension,15 and moreover, the heart rate spectrum
also shows a fractal.16 However, to our best knowl-
edge, we have not seen any studies that discussed
fractals in the complex world of human recognition.
It is natural to assume that the human recognition
system has a fractal structure, because behavior or
a physiological response, which we can explain as
fractal, can be modified by the recognition system.
The complex shape in the representation of facial
images in the two-dimensional space might be in-
fluenced by fractals. The purpose of this study,
therefore, is to examine the possibility of a fractal
structure underlying the facial expression recogni-
tion which is one of the most important parts of the
human recognition system. If the representation of
the perimeter of an emotion circumplex is fluctu-
ated by a fractal, it would have no characteristic
length scale and it must, therefore, have a fractal
dimension of D.
2. METHOD
2.1 Experimental Conditions
Considering that eight facial expressions of Hap-
piness, Calm, Sleepiness, Sadness, Anger, Fear,
Surprise and Excitement were typical prototypes,
we designed the perimeter of a circumplex varying
clockwise as shown in Fig. 2. To prepare different
scales, the following four experimental conditions
were set: (i) 32 divided, (ii) 40 divided, (iii) 48
divided, and (iv) 64 divided conditions.
2.2 Participants
The participants for each of the four experimental
conditions were all undergraduate students: 23 for
 Facial Expression Recognition 49

Surprise

Fear Excitement

Anger Happiness

Sadness Calm

Sleepiness
(a)

Happiness Calm

(b)

Fig. 2 (a) Schematic representation of eight prototype facial expressions on the perimeter of a circumplex. Note that the
distances between prototypes are not exactly the same. (b) An example of Happiness-Calm pair of morphing process and its
morphed images in condition (i). Physical increment is kept constant. The other pairs were carried out in the same way.

condition (i), 19 for (ii), 31 for (iii), and 28 for (iv).
They volunteered to be research participants to get
a credit for a psychology course. All had normal or
corrected-to-normal vision.
2.3 Stimuli
Based on the arrangement illustrated in Fig. 2,
morphed facial images were generated between all
adjacent prototype face pairs positioned on the
perimeter in each of the four different scales using a
morphing software (Morpher for WindowsTM ). In
condition (i), each interval between adjacent pro-
totypes was divided into four segments and three
morphed facial images were generated. In total,
32 facial images (eight prototypes and 24 morphs)
were generated around a circumplex for condi-
tion (i). Similar to this condition, morphed facial
50 T. Takehara et al.
images were generated for the other three con-
ditions: 40 facial images (eight prototypes and
32 morphs) for condition (ii), 48 facial images
(eight prototypes and 40 morphs) for (iii), and 64
facial images (eight prototypes and 56 morphs) for
(iv). The particular advantage of the morphing is
that it is possible to create highly objective facial
images and easy to control the amount of physi-
cal changes. Since the morphing has been used in
many studies,17 – 20 its procedure would provide an
extremely useful tool to investigate the underlying
structure of facial expression recognition.
2.4 Procedure
For each of the four experimental conditions,
participants were asked to rate all facial images on
a six-point Likert-type scale (from “not at all” to
“very so much”) about each item of the eight pro-
totype emotions of Happiness, Calm, Sleepiness,
Sadness, Anger, Fear, Surprise and Excitement.
In rating, facial stimuli were presented one at a
time in a random order on a 14.1-inch LCD with
240×320 pixels. The distance between a participant
and a display was about 50 cm. Data were collected
by a Compaq personal computer (DESKPRO;
Pentium II Processor 350 MHz) with a two-button
mouse and by a rating software made by an Inprise
C++ compiler (Borland C++ Builder Version 4).
No feedback was given.
3. RESULTS
In order to visualize the assumed two-dimensional
psychological space framework, we need to spec-
ify the relative positions of faces within the psy-
chological space. This can be accomplished with
multidimensional scaling (MDS), which derives a
spatial representation of the stimuli from psycholog-
ical data, such as ratings of perceived similarity.21
For each of the four experimental conditions, rating
data were used to calculate a matrix of dissimilari-
ties among eight facial expression categories, which
was then subjected to MDS to find coordinates of
all facial images in two-dimensional space. For the
two-dimensional configuration, all RSQ values were
sufficiently high: 0.978 for condition (i), 0.971 for
(ii), 0.976 for (iii), and 0.977 for (iv). RSQ is a mea-
sure of the proportion of the variance in the data
accounted for by the MDS solution. Similar to the
results of many previous studies, the positions of the
1.5
Excitement
Surprise
1.0
Fear
0.5
Happiness
0.0
Anger
-0.5
Calm
Sadness
-1.0
-1.5
Sleepiness
-2.0
-2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5
Fig. 3 Configuration of prototype and morphed facial
images in the two-dimensional psychological space. This ex-
ample illustrates a resultant configuration of condition (ii).
Black rectangles indicate the prototypes and white rectan-
gles morphs. Adjacent faces are tied with a straight line.
All morphs are arranged between their prototypes, but the
perimeter of a circumplex seems to be complex and wiggled
rather than smooth. The other conditions showed the same
tendency.
prototypes fell in a roughly circular order and were
rated in the same order under all conditions,3,4,6,7,22
and all morphs were plotted between their parent
prototypes corresponding to their morphing pro-
portions. The resultant configuration in condition
(ii) is shown in Fig. 3. Roughly speaking, each
configuration may appear to form a circle, but no-
tice that the perimeter is complex and somewhat
wiggled rather than smooth. The distance of the
perimeter of a circumplex is defined here as the to-
tal distance, which is the sum of distances between
adjacent facial images.
If the distance of the perimeter were exactly the
same for the four conditions despite the different
scales, it would not have fractal properties, suggest-
ing that the perimeter has a fixed and characteristic
length scale like a complete circle. On the contrary,
if it were revealed that as the scale was made smaller
and smaller, the distance of the perimeter tended to
increase steadily without bounds, we would obtain
a fractal dimension, indicating that the perimeter
had no characteristic length scale, and was fluc-
tuated by a fractal dimension. To test this, we

1.9
1.8
Log number of steps
1.7
1.6
1.5 D = 1.18
-0.8 -0.7 -0.6 -0.5
Log step-length
Fig. 4 Linear regression of log distance of the perimeter
on log step-length. The slope of the straight line gives −D,
from which we find a fractal dimension D = 1.18. R2 value
is 0.998.
calculated the length of the perimeter for each of the
four experimental conditions using the coordinates
in two dimensions.
The relationship between fractal dimension D,
step-length r and the perceived length of the line
l(r) measured with step r is given by the following:
l(r) ∝ r (1−D) .
According to the definition by Mandelbrot (1982),10
fractal dimension D must lie in the range 1 ≤ D ≤ 2
for a line. The slope of the resulting line on the
log/log plot shown in Fig. 4 was estimated by re-
gression analysis, then a fractal dimension of 1.18
was derived with an R2 value of 0.998, indicat-
ing that a circumplex did not have a characteris-
tic length scale. The fact that the estimation of
D is a fraction, 1.18, is highly impressive because
the perimeter of the emotion circumplex in the psy-
chological space has not been considered to have a
fractional dimension, or an integer dimension (two-
or three-dimension) has been major in psychology.
Furthermore, a fractional dimension itself never has
been conceptualized in the domain of the human
recognition, either.
4. DISCUSSION
Many studies have suggested that the representa-
tion of the structure of emotional facial expression
Facial Expression Recognition 51
recognition was a circumplex,1 – 7,22 but no study has
stated that the perimeter was complex and wiggled
rather than smooth. Focusing on the subtle fluc-
tuation around the perimeter of a circumplex, we
examined whether the emotion circumplex had a
fractal structure or not in four different scales. The
perimeter of a circumplex represented in the two-
dimensional psychological space showed a fractal
structure, indicating that human facial expression
recognition, in some sense, had fractal properties
and that the two-dimensional space itself was not
homogeneous everywhere. Why is the perimeter of
a circumplex fluctuated? Let us discuss possible
reasons and the future application.
In relation to the complex fluctuation of the
-0.4 perimeter around a circumplex, can we say that the
fluctuation occurs in a random fashion? The answer
is no, because if the wiggled perimeter were due to
the randomness, the log/log plot in Fig. 4 could not
be linear. The log/log plot in this study showed a
linear relation. It was, therefore, suggested that the
fluctuated perimeter was not due to the randomness
but to a fractal. The perimeter, however, seemed
to be wiggled irregularly, the fractal regularity was
hidden in the structure.
One reasonable interpretation for the fluctuated
perimeter is that it may enhance the recognition
efficiency for complex facial expressions which have
various emotional components. For example, con-
(1) sider one continuum around the perimeter run-
ning from Happiness to Excitement. These two
emotion categories occupy the similar region in the
two-dimensional space. In addition, the morphs
generated by interpolation of these two prototypes
are complex facial expressions which have at least
more than two emotional components — Happiness
and Excitement. The fact that the trajectory of
the continuum was not linear but fractal might in-
dicate the enhancement of the recognition efficiency
for complex and subtle facial expressions through
some fluctuations.
The fact that the recognition system has
fractal properties can perhaps be applied not only
to psychology but also to other various domains
of research dealing with recognition. To name a
few: clinical psychology, developmental psychology,
artificial intelligence, human communication, brain
research and recognition engineering. In the future,
it is hoped that theoretical and empirical verifica-
tions will be carried out in an interdisciplinary man-
ner since the topic has important implications to
many disciplines.
52 T. Takehara et al.

ACKNOWLEDGMENTS 11. B. B. Mandelbrot, D. E. Passoja and A. J. Paullay,

Nature 308, 721 (1984).
We thank Professor James A. Russell for provid- 12. D. R. Morse, J. H. Lawton, M. M. Dodson and M.
ing us the facial stimuli and Shizuhiko Nishisato, H. Williamson, Nature 314, 731 (1985).
Kazuyasu Shigemoto and Hiroshi Watanabe for 13. I. Shimada, Y. Minesaki and H. Hara, J. Ethol. 13,
useful commentary. 153 (1995).
14. B. J. West, Fractal Physiology and Chaos in
Medicine (World Scientific, Singapore, 1990).
REFERENCES 15. Y. Shinagawa, K. Kawano, H. Matsuda, H. Seno and
H. Koito, Forma. 6, 205 (1991).
1. R. Adolphs, D. Tranel, H. Damasio and A. Damasio,
16. Y. Yamamoto and R. L. Hughson, Physica D68, 250
Nature 372, 669 (1994).
(1993).
2. R. Adolphs, D. Tranel, H. Damasio and A. Damasio,
17. A. J. Calder, A. W. Young, D. I. Perrett, N. L. Etcoff
J. Neurosci. 15, 5879 (1995).
and D. Rowland, Vistual Cog. 3, 81 (1996).
3. J. A. Russell and M. Bullock, J. Pers. Soc. Psychol.
18. A. J. Calder, D. Rowland, A. W. Young, I. Nimmo-
48, 1290 (1985).
Smith, J. Keane and D. I. Perrett, Cognition 76, 105
4. J. A. Russell and M. Bullock, Dev. Psychol. 22, 97
(2000).
(1986).
19. N. L. Etcoff and J. J. Magee, Cognition 44, 227
5. H. Schlosberg, J. Exp. Psychol. 44, 229 (1952).
(1992).
6. T. Takehara and N. Suzuki, Percept. Mot. Skills 85,
20. A. W. Young, D. Rowland, A. J. Calder, N. L.
1003 (1997).
Etcoff, A. Seth and D. I. Perrett, Cognition 63, 271
7. T. Takehara and N. Suzuki, North Am. J. Psychol.
(1997).
3, 217 (2001).
21. R. N. Shepard, Science 210, 390 (1980).
8. D. Avnir and D. Farin, Nature 308, 261 (1984).
22. M. Katsikitis, Perception 26, 613 (1997).
9. B. B. Mandelbrot, Science 156, 636 (1967).
10. B. B. Mandelbrot, The Fractal Geometry of Nature
(Freeman, New York, 1982).