Chapter 11: Advanced Scoring

Techniques

This chapter discusses techniques that are important for scoring the face in a productive research environment. The sub-
jects of research are often speaking, and you need to know how to distinguish the movements that are part of the speech
production process from those AUs that must be scored. Motion records of behavior have little of the discrete qualities of
the samples of behavior contained in the Manual as examples and practice. Instead, behavior flows in a continuous stream.
You need to learn how to segment this flow and the AUs that occur in it into chunks that can be analyzed. Facial muscles
are bilateral, but there can be many differences in the action of the muscle strands on the two sides of the face. The final
part of this chapter shows how to score unilateral actions and presents a method for scoring the asymmetry of AUs.

Scoring AUs During Speech

Speech production involves movements of the lips, jaw, tongue, and other structures that produce appearance changes in
the lips, mouth, cheeks, chin, and neck. Many of these appearance changes are produced by the same muscles that are the
basis for AUs. Users of FACS are most likely to be interested in measuring movements that are different from those of
speech, and these are the movements that FACS is designed to represent. The subtle movements of the lips and other
movements involved in speech production cannot be measured by FACS. Distinguishing movements that are part of
speech production from functionally different movements remains an issue for FACS users. A goal in FACS scoring is to
disregard movements that function only to produce speech and score the other movements that co-occur with speech.

Some AUs occur in areas of the face that are not involved in speech production (e.g., the forehead and eyes). Such AUs
are identified and scored the same way as if there were no speech. Other AUs occur in areas of the face that are changing
in appearance because of the speech production process (e.g., the mouth, lips, and cheeks). Some of these actions are
obviously unnecessary for the production of speech and are scored as though there were no speech, but the determination
of the specific AUs that are used in speech might depend upon the language spoken. What we describe is based on our
knowledge of English speakers; it is possible that in other languages, different AUS are required by speech. For example,
AU 12 is unnecessary for speech in the English language and should always be detected and scored whether or not there is
speech. The movements that produce speech might complicate detecting the beginning or end of these AUs or even mask
an AU’s presence, but this is a separate problem that is similar to other co-occurrence problems. Other AUs that occur in
the speech-affected facial areas might be part of the speech process or some different process or both. Such AUs most
commonly are those that move the lips and include 10, 14, 16, 17, 18, 20, 22, 23, 24, 25, 26, and 28. These AUs are the
most problematic to score during speech, but there are differences between movements that serve speech production func-
tions and movements that might have other functions that enable distinguishing them.

Almost all movements that serve only speech production are low intensity movements (A level). Sometimes, such an AU,
which is required to produce the speech, is accentuated beyond what is necessary for the speech sound itself. This incon-
gruous intensity is one clue that such an AU should be scored. Another clue is that the timing of an AU and its action rel-
ative to the sounds and other movements of speech appears quite different when the movement is part of speech
production versus some other process. For example, an AU that begins before it should in respect to the speech, or is held
on the face for longer than needed for speech should be scored. Another possibility is that the movement occurs during
pauses in the speech, a clear sign that it should be scored. Looking closely, you will see many actions that do not fit the

Facial Action Coding System

357