Audio-Video Correspondence and
Its Role in Altenfion and Memory
[] Tom Grimes
Tom Grimes is Assistant Professor and Research
Associate, School of Journalism and Mass
Communication, University of Wisconsin-
Madison, Madison, WI 53706.
This study measured the effect of audio-
video channel correspondence on attention
and memory. Three versions of four TV news
stories were the stimuli. The high-
correspondence version employed a specific
audio-video semantic match, the medium-
correspondence version a less specific match,
and the no-correspondence version no
match. High channel correspondence
promoted the most efficient division of
attention and the best memory scores on
visual and factual recognition measures.
The medium-correspondence version yielded
significantly worse visual memory and
attention scores, but factual attention and
memory scores equal to those for the high-
correspondence version. The no-
correspondence version yielded high visual
recognition scores, but the worst visual
attention scores, suggesting that attentional
capacity boundaries were overwhelmed by
the stimulus. The results across all
conditions suggest that when attentional
capacity is exceeded, memory of the overall
story is degraded.
ETR&D.Vol, 38, No. 3, pp. 15-25 ISSN1042-t629
[] A good deal of the research in mass com-
munication (e.g., Graber, 1988; Gunter, 1987;
R o b i n s o n & Levy, 1986) is d e v o t e d to an-
swering the question, W h y do people seem
to learn so little from television news? In-
deed, field surveys suggest that people who
watch TV news remember very little of what
they see and hear (Katz, Adoni, & Parness,
1977; Neuman, 1976; Robinson & Levy, 1986,
pp. 107-132; Stauffer, Frost, & Rybolt, 1983).
The question extends beyond TV news; it
e x t e n d s to all m u l t i m e d i a presentations.
Reeves (1989) sees n e w s as a " m e r e con-
tainer" for the psychological phenomena that
mass media researchers examine, no differ-
ent from TV advertisements, sitcoms, or, for
that matter, educational presentations. The
reason is that the processes being evaluated
are so basic to h u m a n cognition that a mes-
sage's program title should have no effect on
the cognitive processes under study (Reeves,
1989, p. 194).
If Reeves is correct, then TV news may
serve as the paradigm for a range of similar
audiovisual presentations. Understanding
which variables influence learning from TV
news s h o u l d help in u n d e r s t a n d i n g h o w
learning is accomplished across several types
of audiovisual presentations.
In considering those "learning variables"
that are inherent in all audiovisual presen-
tations, one deserves special attention. It is
channel correspondenceithe semantic
match or mismatch between the audio and
video messages in the presentation. Channel
correspondence should be singled out be-
cause it is a troublesome variable for all au-
t5
'~
diovisual presentations. In TV news, the pic-
tures and words seldom match exactly. TV
news, by its nature, is a hurry-up profession
in which loose ends mar many newscasts.
One of those loose ends is audio-video cor-
respondence.
In most audiovisual presentations, tight
coordination between what is seen and what
is heard is inconstant. In an effort to better
understand how channel correspondence
mediates message understanding, Drew and
Grimes (1987) created three correspondence
conditions. They used the same seven-story
series in each condition, but varied the de-
gree to which the video matched the audio.
The high-correspondence condition featured
matching audio and video messages. In the
no-correspondence condition, there was no
semantic relationship between the audio and
video information channels. The third con-
dition was one of medium correspondence,
with half high-correspondence and half no-
correspondence messages. Drew and Grimes
found that in the optimum viewing condi-
t i o n - h i g h channel correspondefice--audio
recall and story understanding were the
highest, and video recall the lowest. In the
least optimum viewing condition--no chan-
nel correspondence--video recall was the
highest, audio recall was the lowest, and
story understanding was the worst.
Unfortunately, Drew and Grimes (1987)
created a medium-correspondence condition
that was less a condition than a combination
of conditions. What is called for is a true me-
dium match between the audio and the
video, with the two channels neither unre-
lated nor tightly bound together semantically.
It is, in fact, this medium-correspondence
condition that should be the most interesting,
because it best resembles the condition of au-
dio-video correspondence in most television
news reports.
Additionally, Drew and Grimes never
clearly defined which cognitive processes
should be affected by channel correspon-
dence. Simple measures of recognition do not
tell us enough about the cognitive processes
that ultimately blend the two information
channels into what we call "the story." With-
out additional information, memory and at-
ETR&D,Vol. 38, No. 3
tention--two of the most salient processes in
audiovisual message reception--are lumped
together into simple measures of recognition.
What is called for, then, is a test that uses
a valid medium-correspondence Condition
and separates memory from attentional pro-
cesses. Consequently, a study was designed
that made several assumptions about the way
people learn from TV news and other audio-
visual presentations, as discussed below.
Channel correspondence probably affects
the way in which attention is divided be-
tween the auditory and visual channels. Be-
cause attention is a limited c o m m o d i t y
(Kahneman, 1973), more attentional capacity
is probably required to encode and integrate
in m e m o r y the two information channels
when those two channels convey slightly to
moderately different messages. With enough
difference in messages (e.g., a medium-cor-
respondence condition), attention's capacity
threshold will probably be a p p r o a c h e d ,
maybe even exceeded, and viewers may try
to concentrate on the auditory channel--the
who, what, when, where, why channel--at
the expense of the less specific, and less use-
ful, visual channel.
On the other hand, when the message
channels correspond more than they differ,
attention should be more efficiently distrib-
uted between channels, which would require
less capacity. (This represents the high-cor-
respondence condition.) This is because high
channel correspondence should allow the
viewer to monitor the auditory channel for
the who, what, when, and where of the story
and to shadow the visual channel for infor-
mation that supplements the narration and
can be integrated into the information sup-
plied by it. Thus, attention to, and memory
for, the visual channel should be greater in
this condition than in the medium-corre-
spondence condition.
However, if the conflict between the two
information channels is great enough (e.g.,
the no-correspondence condition), attention
to the visual channel will override attention
to the auditory channel. This idea is sup-
ported not only by the no-correspondence
data from the Drew and Grimes study, but
also by the work of Colavita (1974) and Pos-
AUDIO-VIDEO CORRESPONDENCE
net, Nissen, and Klein (1976). They report
that visual stimuli dominate auditory stimuli
when pictures and words have no relation-
ship to one another. This may be due to the
fact that pictures access semantic codes more
quickly than do words (Dhawan & Pelle-
grino, 1977; Pellegrino, Rosinski, Chiesi, &
Siegel, 1977; Smith & Magee, 1980). That is,
visual signals appear to go through fewer ex-
trapolations than auditory signals before
they result in memory codes (Cohen, 1973).
When people are presented pictures and
words that have no semantic relationship, the
pictures require less effort to encode and
store. Thus, there should be less attention
applied to the visual channel because less is
required; the visual channel should be very
easy to encode, and visual memory should
be high because encoding is so easy.
To test subject response to news stories in
the three correspondence conditions, two de-
pendent measures were created. One meas-
ured allocation of attention between chan-
nels, and the other measured visual and
factual recognition.
The dependent measure that tracked at-
tention employed secondary task methodol-
ogy, which has been used in earlier commu-
nication research (e.g., Thorson, Reeves, &
Schleuder, 1985, 1986) and in cognitive psy-
chology (e.g., Britton, Muth, & Glynn, 1986).
Secondary task methodology makes three
assumptions. First, attention is limited. This
limited resource cannot be distributed evenly
among competing demands when those de-
mands require more resource than is avail-
able (Kahneman, 1973; Norman & Bobrow,
1975). Second, if competing demands require
more resource than is available, some de-
mands will use more of the resource than
other demands, creating an unequal distri-
bution of the resource. Third, this inequality
can be measured.
Thorson, Reeves, and Schleuder (1985) il-
lustrated this phenomenon by using the anal-
ogy of driving a car (p. 430). It is more diffi-
cult to answer questions (the secondary task)
when traffic is heavy and the driver must
devote most of his or her attention to the act
of driving (the primary task). Likewise, sub-
ject response to the presence of a light probe
17
should be slower if the subject is devoting a
lot of attention to the visual channel. On the
other hand, if the subject is not devoting
much attention to visual messages, then re-
s p o n s e to the p r e s e n c e of a light p r o b e
should be fast, because visual messages are
not competing with the probe for entry into,
and analysis by, the human information-pro-
cessing system. The same logic holds for tone
probes.
The point is that attending and respond-
ing to a probe requires additional attentional
capacity not devoted to encoding and assim-
ilating the messages within which the probe
is embedded. That is why reaction times to a
probe should be slow when the subject is
attending to the messages containing the
probe: The subject must "disengage" from
the messages in order to respond to the pres-
ence of the probe. On the other hand, re-
sponding to the presence of the probe should
take much less time when the subject is not
trying to simultaneously encode complex
messages conveyed within the sam e channel
as the probe appears. The amount of time
these processes take is measured in millisec-
onds.
In summary, this study tracked the locus
of attention as it was split between the audi-
tory and visual information channels. The
purpose was to discover which channel, if
either, received less attention due to dimin-
ishing channel correspondence. If attentional
measures shifted among the three correspon-
dence conditions, visual and factual recog-
nition measures were taken to track the ef-
fects on memory of these shifts.
METHOD
Supers
The subjects were 202 undergraduate com-
munication majors ranging from sophomore
to senior classification who were recruited
from two different classes. The subjects were
split among six cells in a 3 x 2 factorial de-
sign ( A u d i o - V i d e o C o r r e s p o n d e n c e x
Probe).
,~
Stimulus Materials
Four fictitious but realistic television news
stories, each lasting 1 minute 30 seconds,
were used. The video for each was obtained
from one of three sources: the CBS Evening
News with Dan Rather, ABC World News
Tonight with Peter Jennings, or the local news
of the CBS affiliate. The narrations were cre-
ated by the author. Each story dealt with con-
crete, nonabstract ideas.
Each of the correspondent and partially
c o r r e s p o n d e n t stories was evaluated by a
panel of professional broadcast journalists
(two local n e w s directors, one n e w s pro-
ducer, a n d one a n c h o r m a n ) . T h e y were
asked to determine whether each of the sto-
ries was suitable for airing on their news
broadcasts. All four correspondent stories
were judged suitable for airing, as were all
four partially c o r r e s p o n d e n t stories. This
evaluation was conducted in order to pro-
duce an ecologically valid stimulus, one that
would persuade practitioners of the validity
of the result.
Story Versions
Three versions of each story were created:
high correspondence, m e d i u m correspon-
dence, and no correspondence.
HIGH-CORRESPONDENCE. In the high-cor-
respondence version, there was a "tight" pic-
t u r e - w o r d match. For instance, w h e n the
narration in one of the stories stated, " A
needle the size of a screwdriver is inserted
into the herniated disk [in the spine]," the
accompanying video showed a needle, the
size and shape of a screwdriver, about to be
inserted into the spine of a patient. When the
script referred to more abstract concepts,
such as "[Patients] have to remain in the hos-
pital u p to t h r e e - a n d - a - h a l f w e e k s , " the
video was more general but related (in this
instance, people in wheelchairs were shown
in a hospital corridor). In other words, there
were some places in each script where there
was no specific visual correlation between
the audio and video. However, where con-
ETR&D,k,~ol.38, No. 3
crete, nonabstract ideas were conveyed, there
was audio-video correspondence.
MEDIuM-CORRESPONDENCE. In the me-
dium-correspondence version, the two infor-
mation channels were thematically related
but the individual audio and video messages
were not correspondent. This was achieved
by accompanying a specific reference in the
narration, for example, "A needle the size of
a screwdriver is inserted into the herniated
disk [in the spine]," with highly general
video that was not semantically correspond-
ent with the reference, for example, "needle"
and "the size of a screwdriver." In this in-
stance, a patient was shown being wheeled
into surgery as the reporter referred to the
surgical tool.
No-CORRESPONDENCE. In the no-corre-
spondence version, the video did not match
the audio. Unlike the low-correspondence
condition in Drew and Grimes' (1987) study,
the n o - c o r r e s p o n d e n c e condition in this
study did not employ coherent video. The
visual track for each of the four stories em-
ployed shots of events that were mixed to-
gether in a kind of visual potpourri. For in-
s t a n c e , as the n a r r a t o r d e s c r i b e d h o w
herniated disks are repaired, the viewer saw
a barge loaded with g a r b a g e a n c h o r e d in
New York Harbor. W h e n the shot changed,
it s h o w e d celebrants of the m o s t recent
Chinese New Year, while the narration con-
tinued to discuss surgery. The purpose of
this condition was to maximize channel con-
flict between the audio and video as a basis
of comparison with the other extreme con-
dition, tight audio-video correspondence.
Story narrations were not varied; they were
the same across all conditions.
Visual and Auditory Probes
Subjects were e x p o s e d to visual or tone
probes embedded within each story for the
p u r p o s e of measuring subject attention to
either the visual or auditory channel. Half
the subjects were exposed to seven visual
probes nested w i t h i n each story, and the
other half were exposed to analogous tone
probes.
AUDIO-VIDEOCORRESPONDENCE
Visual probes consisted of National Tele-
vision Standards Committee Split Field Color
Bars lasting 33 milliseconds. These color bars
filled the entire screen when they appeared.
In the high-correspondence condition, the
visual probes were placed at points where
word and picture messages were presented
simultaneously. In the other two conditions,
the probes were placed at the same points
where they appeared in the high-correspon-
dence condition.
The tone probes were analogous to the
visual probes in number and placement.
Where color bars appeared in the visual
probe condition, tones appeared in the au-
ditory probe condition. Tones were generated
by a Hewlett-Packard Wide Range Oscillator
and were set at 10,000 Hz.
In both probe conditions, subjects were
told to push a telegraph key as soon as they
heard a tone or saw a color bar. The amount
of time it took for them to press the key from
the onset of the probe--measured in milli-
seconds-provided the estimate used to infer
attentional locus. The longer it took, the more
attention was assumed to be devoted to the
channel within which the probe appeared.
The faster the response to a probe's pres-
ence, the less attention was assumed to be
devoted to the channel within which the
probe appeared.
Recognition Tests
Subjects were given factual memory and vis-
ual memory tests after each story. The factual
memory test was a 53-question, multiple-
choice examination that probed for memory
of facts presented in the narration--facts that
could not be inferred from information pre-
sented in the visual channel.
In the visual memory test, subjects were
shown freeze frames that included frames
taken from the visual channel of each story
and frames taken from distractor material de-
signed to look very much like material that
appeared in the story. Distractors, in fact,
came from "outtakes," that is, video that
originally appeared when each story aired
but was edited out by the author when each
t9
story was being prepared for use in this
study. A total of 36 frames were shown to
subjects, with subjects making yes/no deci-
sions as to whether each freeze frame ap-
peared in one of the four stories. Freeze
frames were clustered by story.
Procedure
Subjects were run individually by 1 of 12
trained experimenters. A diary was kept by
each experimenter. After data were collected,
each subject was asked whether it was easy
or difficult to attend to the narration and
video. Subjects were also asked whether the
probes to which they were exposed were dis-
tracting. Any other observations subjects
wanted to offer were recorded as well.
Prior to viewing, subjects were read in-
structions that told them they were partici-
pating in a memory test designed to find out
what sort of television news stories are best
remembered. They were told to remember as
much as possible of what they were about to
see and hear. Subjects were intentionally
misled into believing that if they did not score
well enough on either the auditory or visual
test, they would have to return to take the
tests again. This was done to emphasize the
importance of remembering the contents of
the auditory and visual channels (the pri-
mary task).
To control for an order effect, four orders
of the four stories were employed. The topics
of the stories were maple syrup production,
a new technique for repairing herniated disks
in the spine, a new method for scoring tennis
games, and tagging of hibernating bears in
northern Wisconsin.
Data Analysis
All reaction-time outliers for tone and color
bar reaction times were removed by calculat-
ing each subject's mean time and then elim-
inating times that fell more than three stan-
d a r d d e v i a t i o n s either side of t h a t
individual's mean. Outliers are always pres-
ent in reaction-time data because of differ-
20 ETR&D, Vol. 38, No. 3

460 -

449
440,

O3

v 420,
~J VISUAL PROBE

Z 400. 406

~) AUDITORY PROBE
<
~J 380

360

353
340
HIGH
' MEDIUM
' N'O

CORRESPONDENCE
FIGURE 1 [] ReactionTimes for Visual and Auditory Probes

ences among subjects' motor skills. Recent
use of alcohol or drugs or simple fatigue can
greatly affect reaction time scores for any
given subject.
Analysis of variance was performed on
the data using Newman-Kuels post hoc tests
where appropriate.
RESULTS
Results f r o m the r e a c t i o n - t i m e data are
shown in Figure 1. There was no statistically
significant difference a m o n g the auditory
probe conditions, F(2, 104) = 1.010, p < .368.
The reaction-time results for the visual
probe conditions (Figure 1) show a main ef-
fect, F(2, 96) = 3.833, p < .025. The slowest
mean reaction time was in the high-corre-
spondence condition, and the fastest reaction
times were in the medium-correspondence
and no-correspondence conditions. Reaction
time was significantly slower in the high-cor-
respondence condition (M = 449 millisec-
onds [msecs]) than in the m e d i u m - c o r r e -
spondence condition (M = 406 msecs), p <
.005. Reaction time was significantly slower
in the high-correspondence condition (M =
449 msecs) than in the no-correspondence
condition (M = 416 msecs), p < .02. No dif-
ference was found between the medium-cor-
respondence and no-correspondence times,
p < .25.
There was no interaction effect between
probe type and level of correspondence, F(2,
200) = 0.836, p < .435.
Visual and Factual Memory Scores
Figure 2 plots visual and factual m e m o r y
scores not as mean number correct, but as
AUDIO-VIDEOCORRESPONDENCE
7O
66%
U
~ 60
0
U 54%
50,
Z
40
30 ! !
MEDIUM
CORRESPONDENCE
FIGURE 2 [ ] Visual and Factual Memory Recognition Scores
percentage of answers correct. This was done
so that visual and factual m e m o r y scores
could be compared.
There was a main effect a m o n g factual
memory scores, F(2, 185) = 87.467, p < .001.
Mean recognition scores were 29.76 (out of 53
possible), or 56% correct, for the high-corre-
spondence condition, 28.44 (54% correct) for
the medium-correspondence condition, and
18.35 (35% correct) for the n o - c o r r e s p o n -
dence condition. The scores of subjects in the
high-correspondence and medium-corre-
spondence conditions did not differ signifi-
cantly, p < .10, but the scores of those in both
the high-correspondence and medium-cor-
respondence conditions differed from the
scores in the no-correspondence condition at
the .001 level of significance.
With r e g a r d to the visual recognition
scores, there was a main effect, F(2, 198) =
49.436, p < .001. The mean number correct
for the high-correspondence condition was
23.851 (out of 36 possible), or 66% correct.
For the medium-correspondence condition,
2t
64%
VISUAL F~COG
---- FACTUALREGOG
NO
the mean number correct was 19.712 (55%
correct). The two conditions differed from
each other at the .001 level of significance.
The mean number correct for the medium-
correspondence condition and the no-corre-
spondence condition (23.104, or 64% correct)
differed at the .001 significance level. There
was no statistically significant difference be-
tween scores in the high- and low-correspon-
dence conditions.
Pearson's p r o d u c t - m o m e n t correlations
were performed on all reaction-time and rec-
ognition data. All significant relationships
were o b t a i n e d u s i n g B o n f e r r o n i - a d j u s t e d
probabilities that guarantee that family-wise
comparisons were not larger than an alpha
set at 0.05. All probabilities were based on
one-tailed tests.
In the high-correspondence condition, a
positive correlation between audio reaction
times and visual recognition scores obtained
(r = .387, r a = .150, p < .025). That is, longer
audio reaction times were associated with
higher visual recognition scores. There were
22
no significant correlations in the medium-
correspondence conditions.
There were two significant correlations in
the no-correspondence condition. Subjects
exposed to tone probes exhibited a negative
correlation between audio reaction times and
visual recognition scores (r = -.310, ra =
.096, p < .045). There was a similar negative
correlation between reaction times and visual
recognition scores for subjects in the visual
probe version of the no-correspondence con-
dition (r = -.308, r 2 = .095, p < .047).
Post-Experiment Interviews
As described earlier in the Procedure section,
each of the 12 experimenters who collected
data kept a diary and asked questions of the
subjects. The questions revolved around
channel correspondence, attention, and
memory: Were the visual and auditory chan-
nels easy to attend to? Did the stories make
sense? Were the probes bothersome? Were
the stories easy to remember? Some subjects
were more articulate than others, and some
had difficulty understanding such concepts
as attention and channel correspondence. To
complicate matters, self-reports on attention
are always suspect. Thus, there is no formal
analysis of the data; however, some refer-
ences to the data appear in the Discussion
section where appropriate.
DISCUSSION
The data patterns generally support the as-
sumptions stated earlier about the way peo-
ple watch TV news. TV news viewers seem
able to split attention between the visual and
auditory channels more efficiently when au-
dio-video correspondence is high than when
correspondence is not as high, or when there
is no correspondence. Efficiently split atten-
tion, in this study, means that the highest
visual and factual recognition scores were re-
corded in the high-correspondence condi-
tion. In other words, attention and memory
seem to be positively correlated when corre-
spondence is high.
ETR&D,Vol. 38, No. 3
However, when the two channels exhibit
some audio-video dissonance, as repre-
sented by the medium-correspondence con-
dition, attentional capacity limitations be-
come manifest. Capacity limitation, in this
study, is expressed as a decrement in visual
memory. The decrement in visual memory,
associated with a decline in visual attention,
suggests that attention and memory are pos-
itively correlated. In TV news stories, the
narration usually contains the who, what,
when, where, and why of a story. When ca-
pacity limits were reached, the attentional
shift away from the video of a news story may
have helped maintain coherency in message
reception. This may be an artifact of the
"graceful degradation" phenomenon de-
scribed by Norman and Bobrow (1975), the
attempt by the human information-process-
ing system to continue functioning, even if
performance is degraded by a limitation in
capacity.
The following four dependent measures
provide support for these conclusions.
Reaction Times
If years of watching television have had an
effect on these subjects, perhaps it explains
the statistically flat auditory probe reaction
times. There was no statistically significant
difference among the three conditions on this
measure. This suggests that the subjects
were monitoring the narration with consis-
tency across conditions. Perhaps it was be-
cause the narration contains the who, what,
when, where, and why of the story. Even in
the no-correspondence condition, in which
auditory factual recognition scores were very
low, auditory probe reaction times were not
statistically different from those in the other
two conditions. Indeed, all subjects in the no-
correspondence condition said, in postexper-
iment interviews, they had tried to listen to
the narration but found it either difficult or
impossible.
The times of reaction to visual probes sug-
gest that subjects invested more attentional
capacity in the visual channel when it corre-
sponded to the narration than when it did
AUDIO-VIDEOCORRESPONDENCE '~3

not. The section below, entitled Reaction tween visual recognition scores for those two
Time/Test Score Relationships, should make conditions.
it more clear why the visual probe reaction R e m e m b e r that professional broadcast
times yielded this pattern. journalists had judged stories in the me-
d i u m - c o r r e s p o n d e n c e condition to be of
broadcast quality. Professionally at least--
and even aesthetically--these medium-cor-
Factual Memory
respondence stories were judged acceptable.
Yet they uniformly produced significantly
There was no statistically significant differ-
lower visual memory scores. This result sug-
ence between factual memory scores in the
gests that professional judgments about what
high- and medium-correspondence condi-
is acceptable audio-video correspondence
tions. This, along with the auditory-reaction
may not necessarily lead to optimal message
times for these two correspondence condi-
memory. These data suggest that there is a
tions, suggests that subjects were closely
measurable difference between what profes-
monitoring the auditory channel.
sionals believe is acceptable audio-video cor-
However, in the no-correspondence con-
respondence and what the human informa-
dition, factual recognition scores declined to
tion-processing system will tolerate.
their lowest level, but auditory probe reaction
As for the high visual recognition scores
times remained unchanged. This suggests
in the no-correspondence condition, it is nec-
that subjects were paying attention to the au-
essary to interpret them via the attention
ditory channel, but that they did not remem-
scores. In addition to the extreme audio-
ber much of the factual information con-
video conflict, which severs all semantic links
tained within.
between the two information channels, the
The recognition data for the no-corre-
shots in the no-correspondence condition
spondence condition are consistent with the
had no relationship to one another. This pro-
observations of Drew and Grimes (1987), and
moted viewing that was not systematic or co-
with similar extreme auditory-visual con-
herent, but rather viewing that was very
flicts observed by Cohen (1973), Posner et al.
much like watching a kaleidoscope of uncon-
(1976), and Colavita (1974). Apparently, if un-
nected visual stimuli. The relatively fast re-
related auditory and visual stimuli are com-
action times, then, imply that very little at-
peting for simultaneous processing, then the
tention was being applied to the visual task;
visual encoding primacy Cohen described
indeed, little was needed. Cohen (1973) has
may be particularly noticeable: Factual infor-
argued that accurate visual recognition can
mation may never get enough attentional ef-
be accomplished with very little cognitive ef-
fort to instantiate most factual messages into
fort, in other words, with an expenditure of
memory when visual and factual messages
very little attention. Thus, high recognition
are competing for entry into the human in-
scores and fast reaction times to visual
formation-processing system. Visual mes-
probes suggest that the visuals in this con-
sages do not need as much attention to be
dition were processed only to the point of
efficiently encoded, and therefore get priority
recognition. Had there been coherent visual
entry, of sorts, into the human information-
messages in this condition--as in the other
processing system.
two c o r r e s p o n d e n c e c o n d i t i o n s - - t h e n
greater attention would probably have been
required to piece the shots together in order
Visual Memory to infer a unitary visual message(s). But rec-
ognition scores, like those in the high-corre-
Even though the correspondence manipula- spondence condition, would have remained
tion between the high- and medium-corre- high. In other words, the attention data are
spondence conditions was very subtle, there useful in interpreting the visual recognition
was a statistically significant difference be- scores across conditions.
24
Reaction Time/'l'est Score Relationships
High message-channel correspondence ap-
pears associated with a more thorough pro-
cessing of the visual channel. The high visual
recognition scores and slow visual probe re-
action times in the high-correspondence con-
dition suggest that subjects were investing
more attention in the visual channel com-
pared with the other two conditions. This
conclusion is supported by the lowest corre-
lation between visual reaction times and vis-
ual recognition scores (r = .038, r 2 = .001, p
< .425) among the various correspondence
and probe conditions. A scatter plot analysis
shows reaction times concentrated in a nar-
row band between 350 and 500 msecs. This
explains why there is a weak correlation and
a low probability level: Subjects in this con-
dition were uniform in their heightened at-
tention to the visual channel. Little variation
in reaction times produced a very low prod-
uct-moment correlation.
Reaction times and test scores of subjects
in the high-correspondence, auditory probe
condition provide corroboration. In this con-
dition, there was enough variation between
reaction times and visual recognition scores
to produce a correlation: As times of reaction
to tone probes got slower, visual recognition
scores got better (r = .387, r ~ = .150, p <
.025). This suggests that as subjects paid
more attention to the narration, they were
able to process the video better, and so they
produced better visual recognition scores. In
other words, as subjects paid more attention
to what was said, they used their remaining
attentional capacity to integrate what they
saw with what they heard. This result sug-
gests that when the two information chan-
nels are c o r r e s p o n d e n t , the n a r r a t i o n
"drives" the processing of the visuals.
IMPLICATIONS FOR
EDUCATIONAL MEDIA
Reeves' (1989) observations about the gener-
alizability of fundamental cognitive re-
sponses to all audiovisual presentations is
ETR&D,Vol.38, No. 3
appealing. These cognitive processes, he has
argued, are inexorable and will proceed re-
gardless of message type. If he is correct,
then the data and conclusions from this ex-
periment should apply to similar audiovisual
presentations that are formally defined as
"'educational."
It would seem unwise to apply conclu-
sions inferred from this study to media pre-
sentations that differ from the audiovisual
paradigm tested here. For instance, presen-
tations employing overhead or slide projec-
tion, and scripted or ad libbed commentary
presented live, vary in significant ways. It is
impossible to calculate the effect a live pre-
senter might have in further confusing, or
clarifying, an audiovisual presentation.
Different formats notwithstanding, per-
haps the most unforgiving mandate for tight
audio-video correspondence occurs when a
narration excerpts a document, which is, it-
sell excerpted visually. For instance, Reese
(1982) has demonstrated that when docu-
ments are summarized verbally and simulta-
neously shown verbatim, viewers inevitably
read the document and ignore the verbal
summary. This appears to be an involuntary
response. Subjects report they "can't help"
but read the accompanying document. Only
exact duplicates on screen of what is being
said in the narration appear to lead to un-
impeded processing of the intended mes-
sage.
This might be extended to include any oral
presentation in which an accompanying doc-
ument departs from what is being said. Peo-
ple do not look at the written word without
reading it. Reading is automatic. Using legi-
ble documents as visual support for points
being made in a narrative when the narrative,
not the document used as visual support, is
the principal message, invites channel con-
flict.
This study made ample use of psycholog-
ical models, models built from highly puri-
fied stimuli. These stimuli were designed to
give psychologists maximum experimental
control. Although we in educational psychol-
ogy and mass communication research envy
that e x p e r i m e n t a l control, we are bur-
dened--or blessed--with much more noisy
AUDIO-VIDEOCORRESPONDENCE
stimuli that, at times, seem to defy control.
They invite myriad alternative, and irksome,
hypotheses.
Yet these stimuli yield information that
more simple stimulus sets might not yield.
For instance, note how, in the no-correspon-
dence condition, the visuals a p p e a r to con-
tact the mental lexicon much more efficiently
than words do, just as psychological models
have suggested would be the case (Cohen,
1973; Dhawan & Pellegrino, 1977; Pellegrino
et al., 1977; Smith & Magee, 1980). But then
note how, w h e n visuals are paired with se-
mantically relevant words, visual facts are no
better r e m e m b e r e d than narrative facts, a re-
sult that seems to impose a caveat on visual-
processing models.
As both we and cognitive psychologists
are discovering, the h u m a n information-pro-
cessing system apparently reshapes itself to
conform to the goals i m p o s e d u p o n it, some-
thing that makes model building more diffi-
cult. O u r noisy, hard-to-control stimuli may
actually place in high relief the versatility of
the h u m a n information-processing system.
Further research, then, might focus not on
conforming to models of attention and mem-
ory, but rather on teasing out permutations.
In that way, a mosaic of attentional and mem-
ory patterns may emerge that better explain
how we think and learn. []
The author gratefully acknowledges the
theoretical and methodological contributions of
Professors Arthur Glenberg and John Theios,
Department of Psychology, University of
Wisconsin-Madison.
REFERENCES
Britton, B. K., Muth, K. D., & Glynn, S. (1986).
Effects of text organization on memory: Test of
a cognitive effort hypothesis with limited expo-
sure time. Discourse Processes, 9, 475-487.
Cohen, G. (1973). How are pictures registered in
memory? Quarterly Journal of Experimental Psy-
chology, 25, 557-564.
~
Colavita, E B. (1974). Human sensory dominance.
Perception & Psychophysics, 16, 409-412.
Dhawan, M., & Pellegrino, J. W. (1977). Acoustic
and semantic inference effects in words and pic-
tures. Memory & Cognition, 5, 340-346.
Drew, D. G., & Grimes, T. (1987). Audio-visual
redundancy and TV news recall. Communication
Research, •4(4), 452-461.
Graber, D. (1988). Processing the news: How people
tame the information tide. New York: Longman.
Gunter, B. (1987). Poor reception: Misunderstanding
and forgetting broadcast news. Hillsdale, NJ: Eft-
baum.
Kahneman, D. (1973). Attention and effort. New
York: Holt, Rinehart & Winston.
Katz, E., Adoni, H., & Parness, P. (1977). Remem-
bering the news: What pictures add to recall.
Journalism Quarterly, 54, 231-239.
Neuman, R. (1976). Patterns of recall among tele-
vision news viewers. Public Opinion Quarterly, 40,
115-121.
Norman, D. A., & Bobrow, D. G. (1975). On data-
limited and resource-limited processes. Cogni-
tive Psychology, 7, 44-64.
Pellegrino, J. W., Rosinski, R. B., Chiesi, H. L., &
Siegel, A. (1977). Picture-word differences in de-
cision latency: An analysis of single and dual
memory models. Memory & Cognition, 5(4), 383-
396.
Posner, M. I., Nissen, M. J., & Klein, R. M. (1976).
Visual dominance: An information-processing
account of its origins and significance. Psycholog-
ical Review, 83(2), 157-171.
Reese, S. D. (1982). Multi-channel redundancy effects
on television news learning. Unpublished doctoral
dissertation, University of Wisconsin.
Reeves, B. (1989). Theories about news and theo-
ries about cognition: Arguments for a more rad-
ical separation. American Behavioral Scientist,
33(2), 191-198.
Robinson, J. E, & Levy, M. R. (1986). The main
source: Learning from television news. Beverly Hills,
CA: Sage Publications.
Smith, M. C., & Magee, L. E. (1980). Tracing the
time course of picture-word processing. Journal
of Experimental Psychology: General, 109, 373-392.
Stauffer, J., Frost, R., & Rybolt, W. (1983). The
attention factor in recalling network television
news. Journal of Communication, 33, 29-37.
Thorson, E., Reeves, B., & Schleuder, J. (1985).
Message complexity and attention to television.
Communication Research, 12(4), 427-454.
Thorson, E., Reeves, B., & Schleuder, J. (1986).
Attention to local and global complexity in tele-
vision messages. In M. L. McLaughlin (Ed.),
Communication Yearbook 10 (pp. 366-383). New-
bury Park, CA: Sage Publications.