Accid. Anal. and Prev., Vol. 26, No. 5, pp.

571-581, 1994
Pergsmon Copyright 6 1994 Elsevier Science Ltd
Printed in the USA. All rights reserved
MW-4.575194 $6.00 + .OO
OOOl-4575(94)EOOl7-F

EVALUATION OF DRIVER DROWSINESS BY

TRAINED RATERS

Vehicle Analysis and Simulation Laboratory, Virginia PoIytechnic Institute and State University,
Blacksburg, VA, U.S.A.

(Accepted 11January 1994)

Abstract-Drowsiness of vehicle operators is a major hazard in transportation systems, and methods need
to be developed for practical evaluation of drowsiness level. One suggested approach is observer rating.
Accordingly, an experiment was carried out using trained observer-raters to evaluate the levels of drowsiness
of drivers, the drivers’ faces were recorded on videotape. Videotaped segments of drivers at various stages
of drowsiness were presented in two sessions separated by a time interval of one week. The experiment was
directed at determining test-retest reliability, interrater reliability, intrarater re~iab~ity, and consistency.
Results indicate that such ratings are reliable and consjstent. A subsequent experiment shows that ratings
covary with other known indicators of drowsiness.

Keywords-Alertness, Driver performance, Drowsiness, Fatigue

INTRODUCTION observer ratings of drowsiness would be assessed.

Drowsiness of vehide operators is believed to be a
serious problem in transportation systems and a di-
rect or contributing cause of many accidents. Tiiley,
Erwin, and Gianturco (19731, for example, found,
by means of questionnaires administered to 1,500
drivers receiving license renewals, that 64% had at
one time or another become drowsy while driving.
Seven percent of the drivers in the survey indicated
that they had had an accident involving drowsiness,
and another seven percent indicated they had had
a near miss resulting from drowsiness. Because of
the hazard that drowsiness presents in transporta-
tion systems, methods need to be developed for op-
erationaliy assessing and counteracting its effects.
‘This paper describes a method of assessing the
level of drowsiness based on a rating system. The
input info~ation for the rating is a video image of
the vehicle operator’s face. Recently, while per-
forming simulator experiments on sleep-deprived
drivers, it was found that the experimenters con-
ducting the experiments could estimate the level of
drowsiness of the drivers based on characteristics
such as facial tone, slow eyelid closure, and manner-
isms (rubbing, yawning, nodding, etc.). Of course,
such an approach is anecdotal and requires scientific
examination. Sufficient promise existed that it was
decided to conduct a formal experiment in which
571
The observer-rating approach appears to have been
largely overlooked by researchers.
Before describing the experiment conducted, it
is worthwhile to discuss how observer ratings of
drowsiness could be used, assuming they are scien-
ti~cal~y worthy. There are several potential applica-
tions. First, there is the possibility of a “black box”
recorder in which a videotape is made of the opera-
tor’s face. Modern cameras can operate at very low
light levels, so that darkness does not cause diffi-
culties. In addition, modern cameras can be made
very small so that they are unobtrusive. If an acci-
dent occurs, the tape could be retrieved and assessed
for drowsiness. Such an approach could be used in
gaining accurate records of accident causes and in
developing work schedules and other countermea-
sures that minimize the Iike~ihood of such accidents.
fn another application, e.g. in transportation sys-
tems of limited range, such as subways, video im-
ages could be sent by land line to an observer station.
The trained observer could then re-alert any one of
several operators who appears to be having drowsi-
ness difficulties, or, if necessary, take orher action
to minimize the chances of an accident. Yet another
very important application is that of serving as an
independent assessment variable. The development
of any drowsiness detection device or system re-
quires an independent variable that assesses level
512 W. W. WIERWILLE and L. A. ELLSWORTH
of drowsiness and against which the device or sys-
tem can be checked. Traditionally, EEG (electroen-
cephalogram) waveforms and estimation of sleep
stages have been employed for this purpose. While
the EEG approach is worthwhile, it is usually diffi-
cult to use in practice.
Because of the potential applications of ob-
server ratings of drowsiness, an experiment was
conducted to determine whether the concept has
scientific merit. This paper describes the experiment
and the results in detail. A subsequent experiment
was also run. Its purpose was to determine the de-
gree of correlation of observer ratings with other
supposed indicators of drowsiness. High correla-
tions would represent a form of validity of observer
ratings as a measure of drowsiness. This latter exper-
iment is summarized in the present paper.
RESEARCH LITERATURE
Most of the relevant literature has been con-
cerned with ratings by the subjects themselves (Ogil-
vie, Wilkinson, and Allison 1989; Yoshitake 1971,
1978; Rosa et al. 1985). These studies presented the
subjects with scales or questionnaires to fill out ei-
ther during or after completion of an experiment.
The issue addressed was the feeling of fatigue at a
given time during the experiment. Most of the stud-
ies have used the Stanford Sleepiness Scale (which
is a 7-point Likert-type scale) or the fatigue rating
scale developed by Yoshitake (1971, 1978). The
study by Ogilvie et al. (1989) found that there was
a high correlation between the subjective index (the
Stanford Sleepiness Scale) and behavioral and phys-
iological variables. The study further suggests that
the subjective index would be a useful identifier of
progress towards early phases of sleep (Ogilvie et
al. 1989). Another study found a high correlation
between subjective ratings (using the Stanford
Sleepiness Scale and questionnaires) and perfor-
mance measures (Rosa et al. 1985). Two types of
cognitive tasks were used as indicators of perfor-
mance: grammatical reasoning and digit addition.
Again, these studies used ratings given by the sub-
jects as opposed to using ratings given by observers.
One study was found in which observer rating
of drowsiness was studied (Carroll, Bliwise, and
Dement 1989). In particular, interrater reliability of
observations was examined. This study used two
individuals to observe the disturbances of the sleep-
wake cycle in 39 nursing home residents. The obser-
vations were made during daytime and nighttime.
The results exhibited a high interrater reliability
when observing the sleep-wake cycle. The study
results suggest that use of behavioral observations
is an appropriate approach to studying the sleep-
wake cycle in nursing home settings.
There has been one known previous application
of “facial expression” to drowsiness research. In
their paper on drowsiness warning devices, Yabuta
et al. (1985) used a three-level ordinal scale of facial
expression as one component in assessing alertness.
The other components were brain-wave evaluation
and amount of blinking. The combined assessment,
called an alertness index, was used as an indepen-
dent variable in simulator tests directed at devising
a drowsiness detection system. It appears that Ya-
buta et al. (198.5) did not evaluate their facial expres-
sion ratings in terms of reliability.
The literature does thus suggest the notion of
using observers for assessment of drowsiness-
related decrements, and, indeed, one study has used
a coarse scale of facial expression. However, none
of the previous studies provides an adequate basis
for assessing the reliability and consistency of ob-
server ratings of drowsiness for use in transportation
systems.
METHOD
Raters
Six individuals (three males and three females)
volunteered to participate in this study. They were
graduate students in the Human Factors Engineering
program at Virginia Polytechnic Institute and State
University. These students were chosen because of
their familiarity with rating procedures and human
factors methodology. (It was assumed that persons
performing drowsiness evaluations in research or in
applications would have received behavioral train-
ing.) Each individual participated in two separate
sessions, each lasting approximately one hour.
Apparatus
Previous experiments involving drowsy drivers
had been performed in the Vehicle Analysis and
Simulation Laboratory, in which low-light level
video recordings of the drivers’ faces had been
made. The videotapes were retained for archival
purposes and were available for use in the present
study. The tapes showed drivers driving a computer-
controlled, moving-base driving simulator, and con-
tained episodes of a variety of levels of apparent
drowsiness. Thus, segments of the tapes could be
transferred to new master tapes for use in the present
experiment. Two such tapes were made for the ex-
periment. The tapes were played on a high-quality
video recorder and 20-inch monitor. (The video im-
age was monochrome.)
A new rating scale was developed for this exper-
 Driver drowsiness
Segment #
I I I I
Not Slightty
Drowsy Drowsy
Fig. 1. Rating scale used in the experiment.
iment. It was a form of the Likert (descriptive graph-
ics) scale. The continuous scale contained five
descriptors: Not Drowsy, Slightly Drowsy, Mod-
erately Drowsy, Very Drowsy, and Extremely
Drowsy. One scale was used for each segment to
be evaluated, totaling 48 scales (24 scales for each
session). The scale is shown in Fig. 1. Each copy
of the scale was provided on a separate slip of paper,
approximately 22 cm wide and 7 cm high, to mini-
mize influences from previous ratings.
A Macintosh5 II personal computer was used
to analyze the data from this experiment. The Super
ANOVA computer software package, version 1.11
(Abacus Concepts, Inc., Berkeley, CA), and the EX-
CEL computer software package, version 3.0 (Mi-
crosoft Corporation, Redmond, WA), were used to
perform statistical analyses of the resulting data.
The experimentai design used in this study was
a single factor complete factorial design. The single
factor was rater. This main factor (with six levels)
was treated as the independent variable. By treating
rater as the independent variable, each cell of the
experimental design contained 48 replications of the
rating task. In this experimental design, rater was
a within-rating-task variable rather than rating-task
being a within-rater variable. The dependent vari-
ables were the raw-error-rating-scores. In both
cases, errors were defined as differences from the
mean across raters. There were 48 scores per experi-
mental cell, giving a total of 288 data points.
The 48 segments to be rated were divided into
two groups and then recorded onto two video tapes
(24 segments per tape). The segments represented
various levels of alertness/drowsiness and were as-
signed to a location on the tape. One tape was pre-
sented during the first session, and the other tape
was presented during the second session, which oc-
curred approximately one week after the first ses-
evaluation 573
I I I I I
Moderately Very Extremely
Drowsy Drowsy Drowsy
sion. A counterbalanced design was used in which
half the raters received Tape A first, followed by
Tape B, while the other half of the raters received
Tape B first, followed by Tape A. Raters 1, 2, and
3 (two males and one female) received Tape A, then
Tape B, while Raters 4, 5, and 6 (two females and
one male) received Tape B, then Tape A. The pur-
pose in using two video tapes, separated by one
week, was to allow the determination of test-retest
reliability.
htrarrater reliability. On each of the tapes,
three of the segments were repeated on that same
tape. On Tape A, Segments 3, 8, and 10 were re-
peated as Segments 23, 18, and 19, respectively.
Likewise, on Tape B, Segments 3, 8, and 10 were
repeated as Segments 23, 18, and 19, respectively.
The three segments repeated on Tape A were differ-
ent from those repeated on Tape B. During a particu-
lar session, the rater was thus exposed to the three
segments twice. The repetition of segments gave six
sets (pairs) of scores for each rater which were used
to determine intrarrater reliability.
Test-retest reliability. In addition to the repeti-
tion of three of the segments within each session,
three different segments from the first session were
repeated in the second session. Segments 5, 12, and
20 from Tape A were repeated as Segments 20, 12,
and 5, respectively, on Tape B. Therefore, each
rater was exposed to these three segments a second
time during Session 2. This procedure of repeating
segments gave three pairs of scores per rater for use
in determining test-retest reliability.
Interrater reliability. To determine interrater
reliability, all repeated segments and the first seg-
ment (the practice segment) were temporarily de-
leted from the data, Only those segments that were
not repeated were used in the statistical analysis.
After deleting the repeated segments, there re-
mained 28 segments per rater (14 segments from
each session).
 W. W. WIERWILLEand L. A. ELLSWORTH
I m Rater 1 0 Rater 2
E 90
B
d 80
E
,e 70
DA
f n
1 3 5 7
Fig. 2. Raw rating scores as a function of segment mean rank.
Procedure
On the first day of the experiment, the rater
was asked to read the general instructions for the
experiment. These instructions described the nature
of the experiment, the tasks to be performed, and
the approximate length and timing of the two ses-
sions. The instructions made it clear that the rating
scale was a continuous one and that the rater could
place a rating anywhere on the scale, not just at one
of the descriptors. Once the instructions were read,
the rater was asked to read the informed consent
form and sign the form if he or she agreed to the
conditions of the study. Any questions concerning
the instructions, the informed consent form, or the
experiment in general were answered. The rater was
then seated in front of the videorecorder and moni-
tor. At this point the experimenter reviewed the
instructions and gave additional instructions. These
additional instructions included showing the rater
the rating forms, giving examples of how to mark
the scales correctly, and providing the rater with the
“Description of Drowsiness Continuum” form. This
form contained a description of the various levels
of drowsiness and gave an idea of the characteristics
to look for when rating the segments. This form is
shown in Appendix A. The rater read the description
form before the experiment began and was also al-
lowed to refer to the description form during the
experiment. Once all questions had been answered,
the first experimental session began.
0 Rater 3 0 Rater 4 ARater 5 0 Rater ~
6 I
9 11 13 15 17 19 21 23 25 27
Rank of Segment Means
When the rater returned after approximately
one week for the second session, the written instruc-
tions were offered for review. Once the instructions
were reviewed, the experimenter asked the rater to
review the Description of Drowsiness Continuum
form.
Experimental task procedures. The rating task
consisted of viewing 24 segments of different drivers
at various levels of drowsiness for each session and
subjectively rating each segment on its correspond-
ing rating-scale form. When the experimental ses-
sion began, the first videotaped image appeared on
the screen. A short time thereafter, a recorded voice
instructed the rater to begin the evaluation for that
segment. For example, at the beginning of Segment
1, the rater heard “Begin, Segment 1.” This com-
mand informed the rater that the evaluation period
for segment 1 had begun. The rater observed the
videotaped driver until a second voice command,
“End, Segment I,” was given. (The length of the
evaluation period was one minute.) The “End” com-
mand informed the rater that the evaluation period
was over and that a rating on the scale was to be
provided. The rater observed the beginning of the
videotaped image prior to the “Begin, Seg-
ment- ” command, but was instructed to only
rate the interval between the “Begin” and “End”
commands.
After the “End” command was given, the seg-
ment continued for 15 seconds, but the rater did
 Driver drowsiness
80
0 10 20 30
Fig. 3. Intrarrater reliability (first exposure vs. second exposure).
not evaluate this section. Once the 15 seconds had
elapsed, the screen went blank for 10 seconds before
the next segment appeared. The rater used the last
15 seconds of the segment and the 10 seconds of
blank screen between the segments (totaling 25 sec-
onds) to provide a rating. If this amount of time was
insufficient, the rater asked the experimenter (who
sat behind the rater) to pause the tape until the evalu-
ation was completed. This pausing technique al-
lowed the rater to refer to the Description of Drowsi-
ness Continuum sheet. Once the rating was
accomplished, the experimenter restarted the tape.
The rater could also change an answer if desired,
but only if the rating was changed before the next
segment started. (Only the current segment rating
could be changed.) The rater was not permitted to
go back to a previous segment to change a rating.
The sequence continued until all 24 segments had
been rated.
RESULTS
The first step in analyzing the data was to con-
vert the ratings on the rating scales to numerical
values. This task was accomplished by converting
the scale to a IOO-point scale and then measuring
the location of each given rating. Zero was at the
left end of the scale and 100 was at the right end
(Fig. 1). The second step was to pair the repeated
segments to perform the correlations on these data,
The third step was to separate the paired data points
evaluation 575
40 50 60 70 80 90 100
First Exposure
and the practice segment from the original data set.
The final step was to convert the original data scores
into raw error rating scores and absolute error rating
scores. The mean of each segment (across raters)
was determined and ranked from low to high, Each
rater’s raw data were then plotted against this rank-
ing, as shown in Fig. 2. The graph suggests that
there was little, if any, error of central tendency in
the experiment. It can be seen from the graph that
raters rated at both the low end and the high end of
the scale as well as near the middle.
The raw error scores were obtained by sub-
tracting each segment’s mean score from the score
given by each rater. Because there is no numerical
(or objective) definition of drowsiness, an indepen-
dent variable did not exist for this experiment. (This
point is discussed in greater detail later in the section
entitled “Indications of Validity.” Therefore the rat-
er’s scores were compared to the mean segment
score to determine consistency of the scores. The
absolute error scores were obtained by taking the
absolute value of the raw error scores. Thus, 28
raw error scores and 28 absolute error scores were
derived for each rater resulting in a total of 168
raw and absolute error scores across the six raters.
Because of the way the error scores were calculated,
a positive error score indicated that the segment was
overrated compared to the segment mean rating and
a negative error score indicated that the segment
was underrated compared to the mean rating for that
segment.
5% W. W. WIERWILLE and L. A. ELLSWORTH
Four different correlations and four paired
t-tests were performed on the data. The criterion
for “acceptability” for the correlations was 0.80.
The first correlation compared first exposure to sec-
ond exposure for the segments that were repeated
within a session. As mentioned, three segments were
repeated in Session 1 and three segments were re-
peated in Session 2. Therefore, six pairs of data
points per rater existed, giving a total of 36 data
pairs with which to perform the correlation. This
correlation was used to determine if raters were con-
sistent within themselves when scoring segments
during the same time period. The measurement is
an indication of intrarrater reliability. A paired t-
test was also performed.
The second correlation was performed to deter-
mine the relationship between first exposure ratings
and second exposure ratings from Session 1 only.
There were three data pairs per rater (a total of 18
pairs) for this co~elation. The third correlation was
also used to determine the relationship between first
exposure ratings and second exposure ratings, but
these data came from Session 2. Again, three data
pairs per rater, giving a total of 18 pairs of data,
were used to calculate the correlation. Both of these
correlations are indications of intrarrater reliability,
but the sessions were analyzed separately to deter-
mine whether a fatigue/learning effect existed. Once
computed, the correlations were compared to one
another to determine whether a significant difference
existed. Separate f-tests were performed on each of
the two sets of data.
The fourth correlation was performed to deter-
mine test-retest reliability. The three segments from
Session 1 were paired with the corresponding re-
peated segments from Session 2. The three pairs per
rater gave a total of 18 pairs with which to determine
if raters consistently rated segments at two different
points in time (i.e. over a week’s period). A fourth
t-test was performed on the data to determine
whether the difference between pairs was signifi-
cantly different from zero.
An additional correlation analysis was per-
formed as part of the interrater reliability analysis.
It involved correlating the raw ratings of each rater
with every other rater, as a means of quantitatively
assessing consistency.
Two analyses of variance (ANOVAs) were con-
ducted on the data. The first ANOVA compared the
raw error rating scores in each experimental cell to
determine whether there were any biases in rater
ratings. Positive biases are indicative of a tendency
to overrate segments as compared to the mean rating
for that segment, while negative biases are indicative
of a tendency to underrate segments as compared
to the mean rating for that segment. In short, this
analysis was used to determine whether the raters
rated the segments consistently from one rater to the
next. The analysis gave an indication of interrater
reliability.
The second ANOVA compared the absolute
values of the raw error rating scores in each cell
to the mean of the segment. This measure made it
possible to determine each rater’s score deviation
from the mean segment rating. Clearly, absolute er-
ror rating scores that are close {or equal) to zero
indicate accuracy of rating with respect to the mean,
whereas absolute error rating scores that are greater
than zero signify less accurate ratings with respect
to the mean. Thus, the ANOVA indicated whether
a difference existed across raters, in score deviations
from the mean.
Post-hoc analyses of significant main effects
were performed using the Newman-Keuls pairwise
comparison technique. This procedure was used to
determine exactly which raters were significantly
different from one another on the rating task.
Anulysis of intrurruter reliability
The Pearson r correlation procedure gave a cor-
relation value of 0.88 (t = 10.92, d.f. = 34) for intrar-
rater reliability. This value was significant (p <
0.001). The data used for this procedure are depicted
in Fig. 3 and indicate that raters consistently rated
the level of drowsiness when asked to rate the same
segment twice. The paired t-test gave a value t =
0.032 (d.f. = 3.5); p > 0.20. Thus, ~if~ere~~~s within
pairs (i.e. differences between the first and second
members of each pair) of data were not significantly
different from zero.
Session 1 correlation versus Session 2 correlation
The comparison between the two correlations
did not show a significant difference. The correlation
value for Session 1 was 0.93 and the correlation
value for Session 2 was 0.85. Both of these correla-
tions are sign~icant (p < 0.001; t = 9.98, d.$ = 16
for Session 1 and t = 6.35, d.f. = 16 for Session 2).
Comparison of the two values indicated that they
did not differ significantly from one another (p =
0.1335). This comparison suggests there was no reli-
able learning effect from Session 1 to Session 2. The
t-tests performed on these data were not significant
(p > 0.20; t = -0.46, d.f = 17 for Session 1 and
t = 0.33, df. = 17 for Session 2). The results of the
t-tests indicate that the differences within data pairs
in both sets of data were not significantly different
from zero.
 Driver drowsiness evaluation 577

Analysis of test-retest reliability Table 1. Interrater correlation matrix

The correlation value for test-retest reliability Rater number
as determined by the Pearson r correlation proce- Rater
number 2 3 4 5 6
dure was 0.81 (t < 5.45, d.J = 16). This value was
significant (p < 0.001) and indicates that raters con- 1 0.87 0.81 0.68 0.72 0.81
sistently rated the level of drowsiness when asked 2 0.85 0.79 0.85 0.91
3 0.84 0.80 0.86
to rate the same segment twice with a given period 4 (F=0.811 0.84 0.76
of time (i.e. one week) separating the two exposures. 5 0.80
Figure 4 shows a plot of data. The paired t-test
yielded a value t = 0.66 (d.J = 17) which is not
significant (p > 0.20), indicating that the differences demonstrate differential biases when rating the level
within pairs of data were not significantly different of alertness/drowsiness. Raters 1, 2, and 5 tended
from zero. to underrate the level of drowsiness with respect to
the mean and Raters 3, 4, and 6 tended to overrate
Correlations of non repeated segments the level of drowsiness with respect to the mean
(interrater reliability) (Fig. 5). Post-hoc analysis using the Newman-Keuls
The overall correlation value for interrater relia- technique ((Y = 0.05) revealed which raters were
bility as determined by averaging the individual significantly different from one another. Rater 3
Pearson r correlations was 0.81 (t = 3.707, d.f. = (mean = 8.17) rated significantly different from
26). This value was significant at a p < 0.001. The Raters 1 (mean = -6.90), 2 (mean = -3.94), and
individual paired rater correlations as well as the 5 (mean = -4.65). Rater 3 tended to overrate as
average correlation are shown in Table 1. The indi- compared to the mean while Raters 1,2, and 5 tended
vidual rater correlations range from 0.68 to 0.91. to underrate. Rater 4 (mean = 3.78) rated signifi-
These correlations and the average correlation indi- cantly higher as compared to the mean than Rater
cate that ratings tend to be consistent between rat- 1 (mean = -6.90). Rater 6 (mean = 3.53) rated
ers. In other words, the ratings from different raters significantly higher than Rater 1 (mean = -6.90) and
tend to follow the same trend. 2 (mean = -3.94).
Surprisingly, the Newman-Keuls post-hoc test
Analysis of raw error scores did not show a significant difference between Raters
The ANOVA performed on the raw error scores 6 and 5. The test also did not show a significant
revealed a significant main effect of Rater (F = difference of Rater 4 and Raters 2 or 5. The differ-
5.159, p = 0.001). This effect indicates that raters ences in the means of these raters are greater than

100

80

0
0 10 20 30 40 50 60 70 80 90 loo
First Exposure

Fig. 4. Test-retest reliability (first exposure vs. second exposure).

578 W. W. WIERWILLE
10 -
8.17
8 --
6 --
3.78
L
3.53
3
t- 17
A
4
I
A
6
-4 --
-6 --
-8 -
Fig. 5. Mean rating error as a function of rater. (Mean ratings having common letters do not differ signi~cantly, a = 0.05.)
other differences in means, which are significant,
and seem to be an artifact of the Newman-Keuls
post-hoc test. If one rater is removed and the test
is readministered using only 5 raters, then the above
mentioned nonsignificant differences become sig-
nificant. For example, if Rater 2 is removed from
the data, then Rater 4 is significantly different from
Rater 5. If Rater 5 is removed, Rater 4 becomes
significantly different from Rater 2. And, if the test
is performed after removing Rater 4, Rater 6 is sig-
nificantly different from Rater 5. These observations
indicate that the above-mentioned raters should
probably be considered as significantly different
from one another. Accordingly, the raters then fall
into two groups that are significantly different from
one another. In Fig. 5 the first group is designated
by the letter A and the second by the letter B.
Descriptively speaking, the means ranged from
-6.90 to 8.17. The average standard deviation
across all raters’ scores was 12.5. This value is an
indication of the spread in bias of ratings that can
be expected when informed raters perform ratings
on the level of drowsiness.
Analysis of absolute error scores
The ANOVA performed on the absolute error
scores revealed no significant effect of Rater,
F(.5,135) = 1.537, p = 0.1929. This result suggests
that raters do not differ significantly in their accuracy
when it comes to rating the level of drowsiness. The
mean absolute rating error for each rater is depicted
and L. A. ELLSWORTH
t
B
2
,
B
5
1
B
-3.94
-4.65
-6.90
Rater Number
graphically in Fig. 6. (The figure is included because
it gives an indication of the spread in accuracy of
the ratings. The reader is cautioned, however, that
differences are not statistically significant.) The av-
erage mean absolute rating error across all raters
was 10.85. This value is an indication of the expected
magnitude of error (from the mean) for a given rater.
DISCUSSION
Interpretation of results
The correlation values for intrarrater reliability
and for test-retest reliability were greater than 0.80
indicating that raters tended to be consistent within
themselves. Intrarrater reliability correlation (0.88)
was slightly higher than test-retest reliability correla-
tion (0.81) and suggests that the raters may lose a
small amount of consistency over time. However,
according to the statistical test to determine if a
learnin~/fatigue effect existed between Session 1
and Session 2, the two correlations were not signifi-
cantly different.
It is not surprising to find that there was a sig-
nificant rater main effect in the ANOVA that was
performed on the raw error rating scores. However,
the previously mentioned study by Carroll et al.
(1989) indicated that interrater reliability was high
for observers studying the disturbances of the sleep-
wake cycle. The rater main effect indicates that rat-
ers demonstrate differential biases when rating the
level of drowsiness. Variability between raters can
 Driver drowsiness
13.16
11.85
11.14
+ L
+ t
6 1 5 4
Rater Number
Fig. 6. Mean absolute rating error as a function of rater. (Differences are not statistically significant, (Y= 0.05.)
most likely be attributed to differences in individual
definitions of drowsiness. Even though each rater
was provided with the same Description of Drowsi-
ness Continuum form, the interpretations of these
descriptions may vary across raters.
According to the analysis of variance performed
on the absolute rating error scores, raters’ absolute
error rating scores were not significantly different
from one another. This result suggests that informed
raters tend to display the same accuracy when it
comes to rating the level of drowsiness.
CONCLUSIONS OF THE STUDY
The results from this study indicate that there
is a good degree of consistency among and within
raters when rating the level of drowsiness using vid-
eotaped segments of drivers’ faces. The intrarrater
reliability and the test-retest reliability indicate that
raters are consistent within themselves. Even
though the ANOVA of the raw error rating scores
showed a significant effect of rater, suggesting that
inconsistent biases in ratings exist between raters,
one must look at the spread of the means of raw
error rating scores as compared to the scale used.
The means range from -6.9 to 8.17 giving a spread
of approximately 15 points. Very small increments
were used for the divisions on the scale used to
convert the ratings to numerical values. The distance
between any two descriptors on the scale was 25
points. The 15 point spread of means constitutes
only 3/5 of the distance between one descriptor and
evaluation 579
10.78
10.45
7.72
+ 1 -I
3 2
the next. Furthermore, the ANOVA performed on
absolute ratings as a function of rater was not sig-
nificant. Therefore, a good degree of consistency
is present between raters when rating the level of
drowsiness in this study.
Finally, it is clear that the raters in this study
were willing to use the entire scale. They ascribed
widely different values to what they observed in the
various videotaped segments. These findings, along
with the reliability findings, suggest that ratings of
drowsiness by informed raters do consistently dis-
criminate between presented conditions.
INDICATIONS OF VALIDITY
The previously described experiment shows
that there is consistency and reliability in the ratings
produced. However, the experiment does not and
cannot indicate the extent to which the raters are
rating the “true drowsiness level,” since drowsiness
is not a precisely or numerically defined quantity.
It will be recalled that individual rating errors had
to be defined in terms of deviations from the mean
of all the raters, because there is no universal defini-
tion of drowsiness or drowsiness level. How, then,
does one determine the validity of a drowsiness as-
sessment procedure, such as that obtained from the
rating process described in this paper? Or, in short,
how does one establish validity?
There are several approaches to validity. One
approach is to apply the rating procedure to an actual
or operational situation and determine whether the
580 W. W. WIERWKLE
Table 2. Correlations of rater drowsiness ratings with other indicators (after Ellsworth et al. 1993)
PERCLOS AVECLOS EYEMEAS SUBRATE RTMTHCOR RTLTCOR
0.71 I 0.91 I 0.875 0.833 0.322 0.547
Note: Indkuior RIY~IIWS
PERCLOS: percentage of time that the eye? were more than 80% closed
AVECLOS: mean percent eye cloture
EYEMEAS: mean square of percent eye closure
SUHRATE: subject onlme rstmg of drowsiness using an adjustable bar-knob control
KTMTHCOR: mean time to correct ietponsein the math task
RTLTCOR: mean time to cwrect response in the letter search task
MNALPHA: mean amplitude of the EEG alpha wave, measured at the occipital lobe
MNTHETA: mean ampiltude of the EEG theta wake. measured at the occipital lobe
/‘&RATIO: ratio of MNALPHA to mean amplitude of the EEG beta wave, measured at the occipital lobe
THREOG: percent time that the electroculogram was above a set threshold (indicating eye blink or eye roll or both)
MNHRT: mean of instantaneous pulse rate
MNSQHRT: mean square of instantaneous pulse rate
procedure “measures what it is supposed to mea-
sure” (Ghiselli 1964). This is an application-oriented
approach. Another possible approach is to compare
the rating procedure to other supposed indicators of
drowsiness in a controlled experiment. Such indica-
tors might be physiological, performance based, or
subjective. If it can be shown that the candidate
assessment method provides results that covary
with a variety of other known indicators, then the
new method reflects changes associated with the
common independent variables.
To provide answers to questions about validity,
an additiona1, new experiment was conducted.
Briefly, the experiment involved having sleep-
deprived subjects perform alternating letter search
and arithmetic tasks on a computer screen while a
variety of measures were taken (Ellsworth, Wreggit,
and Wierwille 1993). The various measures were
then correlated with informed-rater drowsiness rat-
ings using a procedure identical to that described in
this paper.
Typical results are shown in Table 2, which is
derived from Table 4 of Ellsworth et al. (1993). As
can be seen. correIations of rater ratings with eye
ciosure and subject ratings are high, and correlations
with physiological and performance measures are
moderate. There are results for eight subjects and
four raters. Three of the eight subjects did not exhibit
any signs of drowsiness whatsoever. When they
were eliminated from the data analysis, correlations
increased to values greater than those shown in Ta-
ble I (Ellsworth et al. 1993). The results taken to-
gether support the validity of rater assessment of
drowsiness and suggest that rater assessment is a
viable method of drowsiness assessment when a
video image of the vehicle operator is obtainable.
Arknow/ed~ernents-This research was supported under a coop-
erative agreement with the National Highway Traffic Safety Ad-
ministration, Office of Crash Avoidance Research. The authors
would like to thank Steven Wreggit who helped in preparing the
videotapes and who made suggestions in the design of the study.
The authors wish to thank Ron Knipling. Bob Clarke, Mike Perel,
and L. A. ELLSWORTH
MNALPHA MNTHETA ABRATIO THREOG MNHRT MNSQHRT
0.568 0.567 0.468 0.483 -0.547 -0.525
and Mike Goodman of NHTSA for helpful suggestions. The opin-
ions expressed are those of the authors.
REFERENCES
Carroll, J. S.; Bliwise, D. L.; Dement, W. C. A method
for checking interobserver reliability in observational
sleep studies. Sleep 12:363-367; 1989.
Ellsworth, L. A.; Wreggit, S. S.; Wierwiile, W. W. Re-
search on vehicle-based driver status/performance
monitoring; Third semi-annual research report, Sep-
tember, 1992 to March, 1993. Dept. of Industrial and
Systems Engineering Report No. 93-02. Blacksburg,
Virginia: Virginia Polytechnic Institute and State Uni-
versity; April 1993.
Ghiselli, E. E. Theory of psychological measurement.
New York: McGraw-Hill; 1964.
Ogilvie, IX.D.; Wilkinson, R.T.; Allison, S. The detection
of sleep onset: Behavioral, physiological, and subjec-
tive convergence. Sleep 12: 458-474; 1989.
Rosa, R. R.; Wheeler, D. D.: Warm, J. S.; Colligan,
M. 3. Extended workdays: Effects on performance and
ratings of fatigue and alertness. Behavior Research
Methods, Instruments, & Computers 17 (1): 6-15; 1985.
Tilley, D. H.; Erwin, C. W.; Gianturco, D. T. Drowsiness
and driving: Preliminary report of a population survey.
Paper No. 73012 1. (Presented at the International Auto-
motive Engineering Congress, Detroit, MI, January
1973.) Warrendaie, PA: Society of Automotive Engi-
neers; 1973.
Yoshitake, H. Relations between the symptoms and the
feeling of fatigue. Ergonomics 14: 175-186; 1971.
Yoshitake, H. Three characteristic patterns of subjective
fatigue symptoms. Ergonomics 2113): 231-233; 1978.
Yabuta, K.; lizuka, H.; Yanagishima, T.; Kataoka, Y.:
Seno, T. (i985). The development of drowsiness warn-
ing devices. In: Proceedings of the Tenth International
Technical Conference on Experimental Safety Vehi-
cles. Washington, DC.: U.S. Department of Transpor-
tation; 198.5:282-288.
APPENDIX
Description of Drowsiness Continuum
A person who is not drowsy while driving will exhibit
behaviors such that appearance of alertness wilt be pres-
ent. For exampfe, normal facial tone, normal fast eye
 Driver drowsiness
blinks, and short ordinary glances may be observed. Occa-
sional body movements or gestures may occur.
As an individual becomes drowsy, various behaviors
may be exhibited. These behaviors, called mannerisms,
may include rubbing the face or eyes, scratching, facial
contortions, and moving restlessly in the seat, among oth-
ers. These actions can be thought of as countermeasures
to drowsiness. They occur during the intermediate stages
of drowsiness.
Not all individuals exhibit mannerisms during inter-
mediate stages. Some individuals appear more subdued,
they may have slower eyelid closures, their facial tone
may decrease, they may have a glassy-eyed appearance,
and they may stare at a fixed position.
As an individual becomes very drowsy eyelid closures
of say 2 to 3 seconds or longer usually occur. This is often
accompanied by a rolling upward or a sideways movement
of the eyes themselves. The individual may also appear
not to be focusing the eyes properly, or may exhibit a
evaluation 581
cross-eyed (lack of proper vergence) look. Facial tone will
probably have decreased. Very drowsy drivers may also
exhibit a lack of apparent activity and there may be large
isolated (or punctuating) movements, such as providing a
large correction to steering or reorienting the head from
a leaning or tilting position.
Drivers who are extremely drowsy are falling asleep
and usually exhibit prolonged eyelid closures (4 seconds
or more) and similar prolonged periods of lack of actvitity.
There may be large punctuated movements as they transi-
tion in and out of intervals of dozing.
Please try to rate each videotape segment you will
be viewing by taking into account this description of the
drowsiness continuum. However, if you feel that the
above description overlooks something of importance or
does not properly describe what you are viewing, then
supplement the description with your own best judgment
in making your rating.