Research Article

Psychological Science

Detection of Audiovisual Speech 24(4) 423­–431

© The Author(s) 2013
Reprints and permission:
Correspondences Without Visual Awareness sagepub.com/journalsPermissions.nav
DOI: 10.1177/0956797612457378
http://pss.sagepub.com

Agnès Alsius and Kevin G. Munhall

Queen’s University

Abstract
Mounting physiological and behavioral evidence has shown that the detectability of a visual stimulus can be enhanced by a
simultaneously presented sound. The mechanisms underlying these cross-sensory effects, however, remain largely unknown.
Using continuous flash suppression (CFS), we rendered a complex, dynamic visual stimulus (i.e., a talking face) consciously
invisible to participants. We presented the visual stimulus together with a suprathreshold auditory stimulus (i.e., a voice
speaking a sentence) that either matched or mismatched the lip movements of the talking face. We compared how long
it took for the talking face to overcome interocular suppression and become visible to participants in the matched and
mismatched conditions. Our results showed that the detection of the face was facilitated by the presentation of a matching
auditory sentence, in comparison with the presentation of a mismatching sentence.This finding indicates that the registration of
audiovisual correspondences occurs at an early stage of processing, even when the visual information is blocked from conscious
awareness.

Keywords
speech perception, consciousness
Received 1/22/12; Revision accepted 6/14/12

The ability to combine information presented in different sen-
sory modalities into unified percepts is an important aspect of
an organism’s successful interaction with the external world.
Psychophysical studies have demonstrated that multisensory
integration enhances perceptual sensitivity, reduces ambiguity,
and shortens reaction time (RT; Bolognini, Frassinetti, Serino,
& Ladavas, 2005; Frassinetti, Bolognini, & Ladavas, 2002).
For instance, a synchronous sound can improve detection of a
flash (McDonald & Ward, 2000; Noesselt, Bergmann, Hake,
Heinze, & Fendrich, 2008), increase the perceived luminance
of a foveal light (Stein, London, Wilkinson, & Price, 1996),
and influence the perceptual organization of simple naturalis-
tic visual scenes (Sekuler, Sekuler, & Lau, 1997; Shams,
Iwaki, Chawala, & Bhattacharya, 2005; Vroomen & de Gelder,
2000) and complex naturalistic visual scenes (Arrighi, Marini,
& Burr, 2009; Kim, Kroos, & Davis, 2010).
Despite the many studies illustrating the consequences of
multisensory integration at the behavioral and neural levels,
the actual mechanisms of such cross-modal enhancement and
the stages of processing at which it occurs remain largely
unknown. In our experiments reported here, we explored
whether conscious awareness of visual sensory signals is
required for audiovisual integration to occur, or whether
the visual and auditory signals interact to some extent at a
preconscious level (i.e., prior to conscious visual sensory
processing).
Some recent studies have indicated that when an audiovi-
sual speech stimulus is presented, the observer’s awareness of
the visual input (Munhall, ten Hove, Brammer, & Paré, 2009)
or acoustic input (Tuomainen, Andersen, Tiippana, & Sams,
2005) is a key factor influencing audiovisual integration. These
studies have involved the McGurk illusion (McGurk & Mac-
Donald, 1976), in which the perception of an otherwise intelli-
gible speech sound is dramatically altered if the observer is
watching a phonologically conflicting visual (lip movement)
utterance. To study whether audiovisual integration arises only
when the visual stimulus is consciously processed as a face,
Munhall et al. (2009) used a bistable percept, a dynamic ver-
sion of Rubin’s (1915) classic face-vase illusion in which the
face moved and seemingly articulated a syllable. The results
showed that a simultaneous sound was more likely to be inte-
grated with the visual speech stimulus when the face, rather
than the vase, was consciously perceived as the dominant
figure. Tuomainen et al. (2005) combined pseudowords syn-
thesized as sine-wave speech with visual information from
speaking lips. In sine-wave speech, the formants of the
original speech are replaced with time-varying sinusoids; only
Corresponding Author:
Agnès Alsius, Department of Psychology, Queen’s University, Humphrey
Hall, 62 Arch St., Kingston, Ontario, Canada K7L 3N6
E-mail: aalsius@gmail.com
424 Alsius, Munhall
observers informed of its speechlike nature perceive a sine-
wave stimulus as speech. Tuomainen et al. found that incongru-
ence between the visual and auditory information influenced
categorization of the sine-wave speech (the McGurk effect)
only when the sounds were heard as actual speech, whereas the
visual influence was negligible when participants were not
aware of the speechlike nature of the stimuli.
In contrast to the results of these studies, mounting neural
evidence indicates that audiovisual interactions can take place
at very early stages of processing, in primary sensory cortices
(Shams et al., 2005; Watkins, Shams, Tanaka, Haynes, &
Rees, 2006) or even subcortically (see Driver & Noesselt,
2008, for a review). Given that such cross-modal effects arise
early (Klucharev, Möttönen, & Sams, 2003), it seems possi-
ble that at least some intersensory interactions occur at a pre-
conscious level of processing. The detectability of simple
visual signals can be enhanced by the presence of simultane-
ous task-irrelevant sounds (e.g., Bolognini et al., 2005). Audi-
tory information also heightens perceptual awareness of
visual stimuli (Frassinetti, Bolognini, Bottari, Bonora, &
Ladavas, 2005) by, for instance, preventing adaptation to the
visual information (Sheth & Shimojo, 2004) or extending the
duration of a congruent visual scene’s dominance in binocular
rivalry (Alais, van Boxtel, Parker, & van Ee, 2010; Chen,
Yeh, & Spence, 2011; Conrad, Bartels, Kleiner, & Noppeney,
2010; van Ee, van Boxtel, Parker, & Alais, 2009; see Schwartz,
Grimault, Hupé, Moore, & Pressnitzer, 2012, for a recent
review on multistable perception and cross-modal binding).
Finally, information perceived without awareness in one
modality can sometimes bias the perceptual experience of
sensory information presented in another modality, leading to
some classic multisensory illusions (Bertelson, Francesco,
Ladavas, Vroomen, & de Gelder, 2000; Dufour, Touzalin,
Moessinger, Brochard, & Després, 2008; Leo, Bolognini,
Passamonti, Stein, & Ladavas, 2008). Together, these find-
ings suggest that intersensory interactions, at least for simple
audiovisual pairings, can occur at early levels of processing,
even when the visual or auditory modality is blocked from
awareness.
Simple multisensory stimuli commonly used in multisen-
sory experiments, such as beeps and light flashes, differ from
naturally occurring events in a number of ways; for instance,
they are brief and have minimal internal structure. The
observer combines such stimuli on the basis of their similar
onset timing, close proximity, or both, creating an artificial
event. More-complex natural stimuli have inherent dynamics
in each modality that reflect the source of the stimuli, and
these dynamics must be matched for multisensory integration
to occur. It is notable that for natural stimuli, such as spoken
language, variables such as onset timing and location are of
limited importance for integration (Munhall & Vatikiotis-
Bateson, 2004).
The question of whether conscious awareness of sensory
signals is necessary for the integration of complex natural sig-
nals remains unresolved. Within the speech domain, a few
studies have shown that audiovisual conditions lower the
detection threshold for masked auditory stimuli. For instance,
Grant and Seitz (2000) showed that the ability to detect the
presence of auditory speech in a background of noise was
improved when observers also saw a face articulating the same
utterance. Kim and Davis (2004) partially replicated this find-
ing with a larger stimulus sample. However, both of these
studies had limitations. Grant and Seitz used only three sen-
tences repeatedly in their entire experiment, and Kim and
Davis’s study contained no audiovisual control condition in
which the auditory and visual information were incongruent.
The aim of the present study was to investigate whether the
detectability of a complex visual signal, preconsciously pro-
cessed, can be improved by a concurrent acoustic stream. The
use of a visual detection task rather than an auditory detection
task is advantageous because the visual signal in an audiovi-
sual event characteristically precedes the auditory signal in
time and is thought to have a priming effect on auditory pro-
cessing (Sánchez-García, Alsius, Enns, & Soto-Faraco, 2011).
Thus, local expectancy effects were minimized in our design.
An interocular suppression technique called continuous flash
suppression (CFS; Tsuchiya & Koch, 2005) was used to keep
the visual stimulus (a talking face) outside of participants’
conscious awareness. In CFS, arrays of random overlapping
rectangles of varying sizes and colors are presented succes-
sively to one eye at a fast pace, reliably suppressing from con-
sciousness an image presented to the corresponding location
of the other eye.
Several studies using the CFS paradigm have demonstrated
that both low-level properties of visual stimuli, such as coher-
ence in motion (Conrad et al., 2010; Yamada & Kawabe,
2011), and high-level aspects of visual stimuli can be pro-
cessed during interocular suppression (Jiang, Costello, & He,
2007). For instance, the right fusiform face area is activated
even when faces are rendered consciously invisible through
CFS (Jiang & He, 2006).
In the current experiments, the articulatory movements of
the suppressed talking face were either matched or mismatched
with an auditory sentence. In Experiment 1, we compared how
long it took for the talking face to overcome interocular sup-
pression and become visible to participants in the matching
and mismatching conditions (see Mudrik, Breska, Lamy, &
Deouell, 2011, and Yang, Zald, & Blake, 2007, for similar
paradigms). Differences between RTs in the two conditions
would suggest that audiovisual correspondences can be
detected before visual information is consciously perceived.
In Experiment 2, to avoid anticipatory responses and ensure
that participants were following the instructions, we followed
the same procedure but included some catch trials in which the
face was not presented. Finally, in a control experiment
(Experiment 3), we excluded the possibility that any differ-
ence in performance between the matching and mismatching
conditions was explained by representational dissociations
occurring at the first stages of partial awareness of the stimuli
(i.e., when the talking face started emerging into dominance,
Audiovisual Integration Without Visual Awareness 425
rather than during effective suppression). If this were the case,
faster RTs in one condition, compared with the other, could be
attributed to differences in recognition speeds, motor
responses, or response criteria (Kouider & Dehaene, 2007).
Thus, in Experiment 3, we did not use interocular suppression;
instead, the visual stimuli were digitally edited, and a gradual
transition between the visual noise and the talking face (mim-
icking perceptions in Experiment 2) was presented.
Method
Participants
Experiment 1 had 16 participants (13 females, 3 males; mean
age = 19.8 years), Experiment 2 had 37 participants (29 females,
8 males; mean age = 19.6 years), and Experiment 3 had 37 par-
ticipants (28 females, 9 males; mean age = 20 years). All sub-
jects were native speakers of English and reported having
normal vision and no speech or hearing difficulties. The project
was approved by the local ethics committee. Participants
received monetary compensation.
Apparatus and data acquisition
The study was conducted in a dark sound booth. Stimuli were
presented in stereo using a head-mounted display (HMD) with
integrated headphones (nVisor SX60, NVIS, Reston, VA). The
HMD contained two independent LCD screens, one for each
eye, with a simultaneous refresh rate of 60 Hz. To calibrate the
two displays, we measured the voltage drop across a photo
resistor taped alternately onto each eyepiece; 16 levels of
luminance equally spaced across the luminance range were
individually adjusted until the voltage drops matched. The
same photo resistor was used for all measurements, and the
results were confirmed through several readings, with eye-
pieces alternated between measurements. The HMD was
attached to a PC (Intel Core, Santa Clara, CA) running custom
software that presented the stimuli synchronously to the two
eyes. Video was controlled by a GeForce 8400 GS graphic
card (NVIDIA, Santa Clara, CA). Screen tearing was avoided
by rendering frames for both eyes to a back buffer and then
presenting them upon screen refresh (i.e., page flipping).
Responses were obtained with a response box connected to the
PC. RTs (time from stimulus onset to when participants indi-
cated they saw the face) were independently computed for
each trial. Auditory speech stimuli were delivered through the
headphones at an intensity of 73 dB(A) SPL.
Stimuli
The suppressors used in Experiments 1 and 2 were colorful,
dynamic noise masks (i.e., overlapping rectangles, each mea-
suring 0.83° × 0.83°1) presented at a frequency of approxi-
mately 30 Hz. For all three experiments, audiovisual stimuli
were prepared from digital video recordings of a female
speaker (showing the lower part of the face) articulating a set
of 84 English sentences. The recorded utterances were edited
to last 6 s using Adobe Premiere 6.0 software. The videos
(720 × 576 pixels at 29.97 frames/s; played using the Cine-
pack codec, Microsoft, Redmond, WA) were then converted to
gray scale.
To ensure effective suppression of the face at the beginning
of each trial, we reduced the contrast of the face by 84%,2
centered on the middle value of the luminance space. The con-
trast ramping during the trials was accomplished by comput-
ing the luminance of each pixel in each frame and scaling it by
the contrast adjustment percentage (0.13%) away from the
center of the luminance space. Therefore, in each frame,
the contrast increased by 0.13%, so that at the end of the trial,
the contrast of the face was 60% relative to the contrast of the
original image.
The suppressor was first presented at full contrast; after the
initial 1,000 ms, the contrast was decreased linearly by −0.6%
per frame, so that by the end of the trial, the contrast had been
reduced by 90%.3 The eye viewing the face and the eye view-
ing the suppressor were counterbalanced across trials and ran-
domly intermixed throughout the experiment. Both the image
of the talking face and the suppressor were circumscribed by a
black rectangular border (19° × 16°).
In the trials that presented matching audiovisual informa-
tion (50%), the visual sentence was presented together with its
original auditory recording. To equate the amount of visual
information in the matching and mismatching trials, and to
ensure that any difference between conditions was not
explained by low-level stimulus differences (such as motion
rate or local contrast differences), we used the same visual
sentences in both conditions. The mismatching stimuli (50%
of trials) were created by taking an original audiovisual stimu-
lus and replacing the auditory sentence with another sentence
from the pool of 84 sentences. Each auditory sentence was
presented only once to each participant.
For the catch trials in Experiments 2 and 3, we created a
solid-gray video frame by computing the average luminance
of the individual pixels that made up the face stimulus; the
frame was presented instead of the face. Catch trials (22% of
the trials in Experiments 2 and 3, 16 sentences) were randomly
intermixed with genuine trials.
Experiment 3 was designed to simulate the visual stimuli in
Experiment 2 but did not involve interocular suppression.
Instead, the video recordings of talking faces used in Experi-
ment 2 (including the contrast ramping) were digitally blended
into the visual noise mask using the cross-dissolve transition
function4 of Adobe Premiere 6.0, and the output displays were
presented to both eyes (thus, on each trial, participants saw a
smooth transition between the visual noise and the face). This
condition has been used in previous binocular-rivalry studies
to control for partial awareness (Jiang et al., 2007; Mudrik et
al., 2011), but it does not completely match the perceptual
experience of rivalry in every aspect. Thus, the absolute mag-
nitude of the RT is not expected to be the same in this condi-
tion as in the binocular-rivalry condition. Jiang et al. adjusted
426 Alsius, Munhall

the contrast’s rate of change for their control condition to bring
the RT in that condition artificially in line with the RT for their
binocular-rivalry condition. Because absolute RT was not our
concern, we did not make such adjustments. Partial awareness
was possible, in principle, with the superimposed stimuli in
this control condition. Thus, Experiment 3 allowed us to rule
out partial awareness as an explanation for our results.5
If congruency effects were the result of processing occur-
ring after the visual stimuli had already begun to overcome
suppression, similar differences in RTs between the matching
and mismatching trials would be expected in the CFS experi-
ments (Experiments 1 and 2) and the control experiment
(Experiment 3). Any effects occurring at the first stages of par-
tial awareness of the face were expected to affect RTs equally
in all experiments. Finding differences between the matching
and mismatching trials only in the CFS experiments would
allow us to reject the partial-awareness interpretation.
Procedure
Before we started the experimental sessions, we adjusted the
HMD individually for each participant to account for interocu-
lar differences and to ensure correct visibility of both screens.
Participants were instructed to close one eye at a time while
reading a short sentence and to verbally report any difference in
visual clarity between their two eyes. After this calibration
phase, participants first completed eight practice trials (four
matching, four mismatching) to familiarize themselves with the
experimental paradigm. None of the auditory or visual sen-
tences presented in the practice trials were used in the experi-
mental trials. Effective suppression was assessed after the
practice trials by participants’ subjective reports and was con-
firmed later with RT data. Participants who consistently saw the
face at the beginning of the trials were excluded from analyses.
Figure 1 shows a schematic representation of the audiovi-
sual stimuli used in the three experiments. Participants were
instructed to maintain stable fixation at the center of the screen
throughout the trials (no fixation point was presented) and to
press a button on the response box as soon as they saw any part
of the face. They were also instructed to ignore the auditory
information. The recorded RT served as an index of the time
required to bring the face into conscious perception. Partici-
pants were instructed to not press the button if they did not see
a face (catch trials). After each trial, the experimenter trig-
gered the onset of the following trial by pressing a key on the
computer keyboard.

Time (6 s)
CFS (Exp. 1, 2)

(e.g., More people would ride bikes if they didn’t have to worry about weather, traffic, and hills.)
Video

Control (Exp. 3)

(e.g., More people would ride bikes if they didn’t have to worry about weather, traffic, and hills.)

Matching
Audio

(e.g., More people would ride bikes if they didn’t have to worry about weather, traffic, and hills.)

Mismatching

(e.g., She hadn’t been concerned about her studies before, but she is now that the final exam is getting close.)

Fig. 1. Illustration of the stimulus sequence in the three experiments. Experiments 1 and 2 used continuous flash
suppression (CFS): Participants were presented with a dynamic, colorful noise mask to one eye and a video of a talking face
to the other eye; the contrast of the mask decreased across frames, while the contrast of the face increased. In Experiment
3, a control experiment, the visual stimuli from Experiment 2 were digitally blended into the visual noise mask, and the
output display was presented to both eyes, so participants saw a smooth transition between the visual noise and the face.
Participants were instructed to press a button as soon as any part of the face became visible. In matching trials, the visual
sentence stimulus was accompanied by the auditory recording of the same sentence. In mismatching trials, the visual
stimulus was accompanied by the auditory recording of another sentence, selected randomly.
Audiovisual Integration Without Visual Awareness 427
Experiment 1 included 56 genuine trials (no catch trials),
divided into two blocks of 28 trials; Experiments 2 and 3 each
included 56 genuine trials plus 16 catch trials, divided into two
blocks of 36 trials.
Results
Figure 2 summarizes overall RTs as a function of stimulus
type (matching, mismatching) in the three experiments. RTs
longer than 6 s (0.02%, 0.008%, and 0.002% of trials in
Experiments 1, 2, and 3, respectively) were excluded from
further analyses. In Experiment 1, a two-tailed paired t test
revealed a significant effect of stimulus type on RT, t(15) =
−2.83, p = .01, d = −0.708; average RT was shorter in the
matching condition (M = 2,629 ms), compared with the mis-
matching condition (M = 2,764 ms).6 Experiment 2 repli-
cated this effect: Even though RTs were longer overall than
in Experiment 1 (possibly because participants performed an
extraperceptual visual verification process to avoid false
alarms in catch trials), there was still a reliable difference
between the matching (M = 2,898 ms) and mismatching (M =
2,988 ms) conditions, t(36) = −2.10, p < .05, d = 2.12. The
fact that congruency between auditory and visual informa-
tion modulated RTs suggests that audiovisual correspon-
dence is detected even when the visual information is
suppressed and therefore preconsciously processed.
The analysis of participants’ RTs revealed substantial
individual differences across participants, with some par-
ticipants detecting the face early in the sequence and others
much later. Note, however, that 12 (75%) of the 16 subjects
in Experiment 1 and 24 (65%) of the 37 subjects in Experi-
ment 2 had a shorter RT for the matching stimuli, compared
with the mismatching stimuli, and there was no systematic
pattern relating this difference in RT to how early the rec-
ognition of the face occurred (i.e., how early or late partici-
pants detected the face did not predict the difference
Matching
Mismatching
5,000
4,500
Reaction Time (ms)
4,000
3,500
* et al., 2002; Frassinetti et al., 2005; Sheth & Shimojo, 2004;
*
3,000
2,500
2,000
Experiment 1 Experiment 2 Experiment 3
Fig. 2.  Mean reaction time in Experiments 1, 2, and 3 as a function of stimulus
type (matching, mismatching). Asterisks indicate significant differences
between conditions (p < .05). Error bars represent 1 SEM.
between their RTs on matching trials and their RTs on mis-
matching trials).
In contrast to the results for the CFS experiments, the
results for Experiment 3, the control experiment, did not reveal
differences in RTs between the matching (M = 4,550 ms)
and mismatching (M = 4,547 ms) conditions, t(36) = 0.30, p =
.766, d = 0.313. Furthermore, an analysis of variance with
experiment (Experiment 2, Experiment 3) as a between-partic-
ipants factor and stimulus type (matching, mismatching) as a
within-participants factor revealed a significant Experiment ×
Stimulus Type interaction, F(1, 72) = 4.49, p < .05, thus rein-
forcing the finding that between-condition differences in per-
formance in the CFS experiments are not likely to have been
caused by partial awareness. In Experiments 2 and 3, subjects
were very accurate in withholding responses in the catch trials
(M = 97% and 95%, respectively), F(1, 73) = 2.76, p = .10,
d = 0.386.
Discussion
The main finding of these three experiments is that a com-
plex visual stimulus rendered invisible by interocular sup-
pression can emerge into conscious perception more quickly
when it is combined with a correlated acoustic signal. That
is, congruency between auditory and visual information
affected the time the visual stimulus remained suppressed,
with matching visual speech stimuli consistently breaking
through interocular suppression more quickly than mis-
matching stimuli did. The effect cannot be attributed to dif-
ferences between the matching and mismatching visual
stimuli (e.g., differences in contrast or motion speed) because
the same stimuli were used in the two conditions. Further-
more, the effect is unlikely to be accounted for by partial
awareness because no differences in RTs between the match-
ing and mismatching trials were found in the control experi-
ment (Experiment 3).
Together, our results extend to the cross-modal binding
domain previous findings showing that visual grouping can
occur preconsciously (e.g., Mitroff & Scholl, 2005). Our
results also suggest that audiovisual interactions transpire at
relatively early levels of processing, where neural information
associated with the suppressed visual stimuli is still registered.
These findings add to growing evidence that the detectability
of a visual stimulus undergoing conscious suppression can be
enhanced by a concurrent sound (Alais et al., 2010; Bolognini
et al., 2005; Chen et al., 2011; Conrad et al., 2010; Frassinetti
van Ee et al., 2009) and show that such cross-modal gains can
generalize to real-world, naturalistic stimuli that change
dynamically over time.
Previous studies examining whether awareness is a critical
factor for the audiovisual integration of speech stimuli have
yielded mixed results (Grant & Seitz, 2000; Kim & Davis,
2004; Munhall et al., 2009; Tuomainen et al., 2005), possibly
because of differences in tasks (e.g., identification of phonetic
428 Alsius, Munhall
content vs. detection of the presence of speech). Whereas stud-
ies that required phonetic categorization of the speech stimuli
(Munhall et al., 2009; Tuomainen et al., 2005) have pointed to
a crucial role of consciousness in audiovisual integration,
studies that involved detection tasks (including the present
study) have provided evidence for preconscious binding of
correlated cross-modal information.
These differing results are not necessarily incompatible,
however. The auditory and visual sensory systems interact at
multiple levels of processing during speech perception (see
Eskelund, Tuomainen, & Andersen, 2011; Hertrich, Mathiak,
Lutzenberger, Menning, & Ackermann, 2007; Klucharev et
al., 2003). It is possible, therefore, that the encoding of pho-
netic information requires integration over a longer time scale
than the initial interaction of vision and audition requires
(Munhall et al., 2009). Indeed, electroencephalography studies
have shown that the earliest interactions between the auditory
and visual properties of an audiovisual stimulus occur approx-
imately 85 ms after stimulus onset, whereas the first phonetic
audiovisual interactions are observed 155 ms after onset of an
auditory stimulus (Klucharev et al., 2003). Further evidence
for the multistage account of audiovisual speech integration
has come from studies showing that brain areas responsive to
the processing of cross-modal coincidence of physical events
do not overlap with those involved in cross-modality fusion
(Miller & D’Esposito, 2005). Still more evidence has come
from studies showing that the stage underlying the audiovisual
detection advantage is not specific to speech, whereas the
stage in which the phonetic information of an audiovisual
stimulus is encoded (e.g., the McGurk illusion) is specific to
speech (Eskelund et al., 2011). Therefore, we believe it is pos-
sible that early audiovisual interactions (e.g., early detection
of audiovisual spatiotemporal correspondences) occur prior to
conscious visual awareness, but that audiovisual interactions
affect phonetic categorization only after the conscious
processing of sensory signals (see Munhall et al., 2009, and
Tuomainen et al., 2005).
The observed detection benefits for audiovisual stimuli
could be derived from structural coupling between auditory
and visual events resulting from the inherent relationship
between acoustic speech sounds and their visible generators
(Fowler, 2004). That is, in addition to matching in phonologi-
cal content, the concurrent streams of auditory and visual
speech information are usually invariant in their onsets and
offsets, rhythmic patterning, duration, intensity, and overall
amplitude contour (Chandrasekaran, Trubanova, Stillittano,
Caplier, & Ghazanfar, 2009). Mouth movements (e.g., peak
opening and closing) correlate with peaks in the acoustic
waveform, and the visual kinematics correspond with the tim-
ing of changes in the auditory envelope in great detail (see
Yehia, Rubin, & Vatikiotis-Bateson, 1998). This detailed cor-
respondence appears to be important for audiovisual speech
integration.
Although in our study, the visual stimuli were identical in
the matching and mismatching audiovisual conditions,
variations in the timing of changes in the mouth aperture and
in the onsets and offsets of movement were temporally related
to the patterning of the acoustical energy in the matching con-
dition but not in the mismatching condition. It is plausible,
therefore, that such comodulation of the auditory and visual
signals in the matching condition was registered by systems
responsible for establishing cross-modal correspondence
(Bernstein, Auer, & Takayanagi, 2004; Eskelund et al., 2011;
Grant & Seitz, 2000; Kim & Davis, 2004; Kim et al., 2010).
Increased brain activations in response to temporally corre-
sponding audiovisual streams have been found in a broadly
distributed neural system encompassing the superior collicu-
lus, fusiform gyrus, lateral occipital cortex, extrastriate visual
cortex, and superior temporal regions (Stevenson, Altieri,
Kim, Pisoni, & James, 2010). The results of the present study
suggest that at least some of these areas may be activated even
when visual information is preconsciously processed. The
activation in these areas would subsequently amplify the neu-
ral response in visual regions responsible for the perceptual
representation of the face, enhancing the signal-to-noise ratio
derived from visual rivalry. Such amplification of the neuronal
response in visual cortices is possibly mediated by bidirec-
tional cortical interactions between higher-order association
regions and early sensory areas (Schroeder & Foxe, 2002) or
by lateral circuitry between the auditory and visual regions
(Falchier, Clavagnier, Barone, & Kennedy, 2002).
Because the auditory stimuli were comprehensible to the
participants in our study, perhaps they used high-level linguis-
tic information (semantic, syntactic, lexical, or phonological)
to generate probabilistic hypotheses and thus anticipate repre-
sentations of the visual words in advance of their appearance
(Sánchez-García et al., 2011). Participants may also have used
visual imagery to predict the upcoming visual signal (see Pear-
son, Clifford, & Tong, 2008). Chen et al. (2011), for example,
showed that when subjects were instructed to attend to an
audio soundtrack, the soundtrack’s semantic content influ-
enced their visual awareness of static pictures in a binocular-
rivalry paradigm.
However, we do not believe that the effects found in our
studies can be explained solely by such high-level expectan-
cies. First, participants were explicitly instructed to ignore the
auditory information, and the auditory sentence in a given trial
was not predictive of the type of trial being presented (22% of
the trials were catch trials, and the remaining trials were
equally split between the matching and mismatching condi-
tions). However, it is possible that participants in fact did not
ignore the irrelevant auditory information and used it to antici-
pate the upcoming mouth movements. Note, however, that if
this were the case, the same expectancy effects likely would
have resulted in faster responses in the matching condition
than in the mismatching condition of the control experiment,
but they did not.
Second, it has been shown that even in the case of a language
that participants do not know, visually guided enhancements of
auditory speech perception can occur when the masked auditory
Audiovisual Integration Without Visual Awareness 429
signals are accompanied by visual sentences (Kim & Davis,
2003); presumably, such visual enhancements of auditory stim-
uli are due to local correspondences between the two signals.
Further research examining whether an effect similar to the one
reported here arises when the suppressed speech is in an
unknown language will allow researchers to determine whether
the enhancement of the visual signal occurs at a prephonologi-
cal level or at higher levels of linguistic processing.
In conclusion, our findings constitute novel evidence that
a temporally dynamic, complex visual speech stimulus that
is only preconsciously processed can be integrated with supra-
threshold auditory information, and that such integration
heightens the robustness of residual signals associated with the
suppressed visual speech stimulus. Although the neural mecha-
nism that mediates the enhancement of the visual signal remains
open for investigation, this behavioral finding suggests that
acoustic signals modulate brain activity at an early stage of
visual processing, possibly by amplifying the visual signal and
thus facilitating its conscious processing (Shams et al., 2005).
Declaration of Conflicting Interests
The authors declared that they had no conflicts of interest with
respect to their authorship or the publication of this article.
Funding
This research was supported by the Natural Sciences and Engineering
Research Council of Canada. A. A. was supported by a postdoctoral
fellowship from Spain’s Ministry of Education.
Notes
1. All visual-angle measurements in this article are based on the
specifications of the product and varied slightly among participants
because of individual adjustments.
2.  Image manipulations were based on the luminance of the original
images, and therefore only relative values (i.e., percentages) can be
provided here. The specific ramping percentages we used were estab-
lished on the basis of previous pilot data regarding the threshold at
which the flash stimulus could reliably suppress the face at the begin-
ning of the trial and the contrast at which the face would become
visible before the end of the trial.
3.  For the visual noise ramping, the same algorithm was applied to
reduce the contrast over the course of the trial; the contrast-adjust-
ment percentage (0.6%) reduced the contrast range toward the center
of the luminance range. The contrast adjustment for any given frame
of a visual sequence can be computed by multiplying the time (in
seconds) by the frame duration (29.97 ms) by the step size (0.13 for
the face, 0.6 for the noise) and adding or subtracting (as appropriate)
the obtained value from the initial contrast adjustment of that stimu-
lus (84% for the face, 0% for the visual noise).
4.  The weight of each pixel was gradually changed from 100% of its
weight in the visual noise stream (initial frame) to 100% of its weight
in the talking-face stream (final frame).
5.  This interpretation, however, rests on the assumption that the time
window over which the partial-awareness effects would be observed
(the near-threshold stimulation) was similar in the control and the
CFS conditions. Note, however, that this may not necessarily have
been the case. The rate of transition between dominance of the flash
stimulus and the face could have been shorter in the control condition
because of the very straightforward changes in stimulus contrast that
occurred; in comparison, the transitions in the CFS condition were
much more variable.
6. All d values were corrected for dependence between means, using
Morris and DeShon’s (2002) Equation 8.
References
Alais, D., van Boxtel, J. J., Parker, A., & van Ee, R. (2010). Attending
to auditory signals slows visual alternations in binocular rivalry.
Vision Research, 50, 929–935.
Arrighi, R., Marini, F., & Burr, D. (2009). Meaningful auditory
information enhances perception of visual biological motion.
Journal of Vision, 9(4), Article 25. Retrieved from http://www
.journalofvision.org/content/9/4/25.full
Bernstein, L. E., Auer, E. T., Jr., & Takayanagi, S. (2004). Auditory
speech detection in noise is enhanced by lipreading. Speech Com-
munication, 44, 5–18.
Bertelson, P., Francesco, P., Ladavas, E., Vroomen, J., & de Gelder, B.
(2000). Ventriloquism in patients with unilateral visual neglect.
Neuropsychologia, 38, 1634–1642.
Bolognini, N., Frassinetti, F., Serino, A., & Ladavas, E. (2005).
“Acoustical vision” of below threshold stimuli: Interaction
among spatially converging audiovisual inputs. Experimental
Brain Research, 160, 273–282.
Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A., &
Ghazanfar, A. A. (2009). The natural statistics of audiovisual
speech. PLoS Computational Biology, 5, e1000436. Retrieved
from http://www.ploscompbiol.org/article/info%3Adoi%2F10
.1371%2Fjournal.pcbi.1000436
Chen, Y.-C., Yeh, S.-L., & Spence, C. (2011). Crossmodal constraints
on human perceptual awareness: Auditory semantic modulation
of binocular rivalry. Frontiers in Psychology, 2, 1–12. Retrieved
from http://www.frontiersin.org/Perception_Science/10.3389/fpsyg
.2011.00212/full
Conrad, V., Bartels, A., Kleiner, M., & Noppeney, U. (2010). Audio-
visual interactions in binocular rivalry. Journal of Vision, 10(10),
Article 27. Retrieved from http://www.journalofvision.org/
content/10/10/27/F1.expansion.html
Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals
crossmodal influences on “sensory-specific” brain regions, neu-
ral responses, and judgments. Neuron, 57, 11–23.
Dufour, A., Touzalin, P., Moessinger, M., Brochard, R., & Després,
O. (2008). Visual motion disambiguation by a subliminal sound.
Consciousness and Cognition, 17, 790–797.
Eskelund, K., Tuomainen, J., & Andersen, T. S. (2011). Multistage
audiovisual integration of speech: Dissociating identification and
detection. Experimental Brain Research, 208, 447–457.
Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Ana-
tomical evidence of multimodal integration in primate striate cor-
tex. Journal of Neuroscience, 22, 5749–5759.
Fowler, C. A. (2004). Speech as a supramodal or amodal phe-
nomenon. In G. Calvert, C. Spence, & B. E. Stein (Eds.), The
430 Alsius, Munhall
handbook of multisensory processes (pp. 189–202). Cambridge,
MA: MIT Press.
Frassinetti, F., Bolognini, N., Bottari, D., Bonora, A., & Ladavas, E.
(2005). Audio-visual integration in patients with visual deficit.
Journal of Cognitive Neuroscience, 17, 1442–1452.
Frassinetti, F., Bolognini, N., & Ladavas, E. (2002). Enhancement
of visual perception by crossmodal visuo-auditory interaction.
Experimental Brain Research, 147, 332–343.
Grant, K. W., & Seitz, P. (2000). The use of visible speech cues for
improving auditory detection of spoken sentences. Journal of the
Acoustical Society of America, 108, 1197–1208.
Hertrich, I., Mathiak, K., Lutzenberger, W., Menning, H., &
Ackermann, H. (2007). Sequential audiovisual interactions dur-
ing speech perception: A whole-head MEG study. Neuropsycho-
logia, 45, 1342–1354.
Jiang, Y., Costello, P., & He, S. (2007). Processing of invisible stimuli
advantage of upright faces and recognizable words in overcom-
ing interocular suppression. Psychological Science, 18, 349–355.
Jiang, Y., & He, S. (2006). Cortical responses to invisible faces: Dis-
sociating subsystems for facial-information processing. Current
Biology, 16, 2023–2029.
Kim, J., & Davis, C. (2003). Hearing foreign voices: Does knowing
what is said affect masked visual speech detection? Perception,
32, 111–120.
Kim, J., & Davis, C. (2004). Investigating the audio-visual speech
detection advantage. Speech Communication, 44, 19–30.
Kim, J., Kroos, C., & Davis, C. (2010). Hearing a point-light talker:
An auditory influence on a visual detection task. Perception, 39,
407–416.
Klucharev, V., Möttönen, R., & Sams, M. (2003). Electrophysiologi-
cal indicators of phonetic and non-phonetic multisensory inter-
actions during audiovisual speech perception. Cognitive Brain
Research, 18, 65–75.
Kouider, S., & Dehaene, S. (2007). Levels of processing during
non-conscious perception: A critical review of visual masking.
Philosophical Transactions of the Royal Society B: Biological
Sciences, 362, 857–875.
Leo, F., Bolognini, N., Passamonti, C., Stein, B. E., & Ladavas, E.
(2008). Cross-modal localization in hemianopia: New insights on
multisensory integration. Brain, 131, 855–865.
McDonald, J. J., & Ward, L. M. (2000). Involuntary listening aids
seeing: Evidence from human electrophysiology. Psychological
Science, 11, 167–171.
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing
voices. Nature, 265, 746–748.
Miller, L. M., & D’Esposito, M. (2005). Perceptual fusion and stimu-
lus coincidence in the cross-modal integration of speech. Journal
of Neuroscience, 25, 5884–5893.
Mitroff, S. R., & Scholl, B. J. (2005). Forming and updating object
representations without awareness: Evidence from motion-
induced blindness. Vision Research, 45, 961–967.
Morris, S. B., & DeShon, R. P. (2002). Combining effect size esti-
mates in meta-analysis with repeated measures and independent-
groups designs. Psychological Methods, 7, 105–125.
Mudrik, L., Breska, A., Lamy, D., & Deouell, L. (2011). Integration
without awareness: Expanding the limits of unconscious process-
ing. Psychological Science, 22, 764–770.
Munhall, K., ten Hove, M., Brammer, M., & Pare, M. (2009). Audio-
visual integration of speech in a bistable illusion. Current Biol-
ogy, 19, 735–739.
Munhall, K. G., & Vatikiotis-Bateson, E. (2004). Spatial and tempo-
ral constraints on audiovisual speech perception. In G. A. Calvert,
C. Spence, & B. E. Stein (Eds.), The handbook of multisensory
processing (pp. 177–188). Cambridge, MA: MIT Press.
Noesselt, T., Bergmann, D., Hake, M., Heinze, H. J., & Fendrich,
R. (2008). Sound increases the saliency of visual events. Brain
Research, 1220, 157–163.
Pearson, J., Clifford, C. W., & Tong, F. (2008). The functional impact
of mental imagery on conscious perception. Current Biology, 18,
982–986.
Rubin, E. (1915). Synsoplevede Figurer [Visual figures]. Copenha-
gen, Denmark: Gyldenhal.
Sánchez-García, C., Alsius, A., Enns, J. T., & Soto-Faraco, S. (2011).
Cross-modal prediction in speech perception. PLoS ONE,
6(10), e25198. Retrieved from http://www.plosone.org/article/
info%3Adoi%2F10.1371%2Fjournal.pone.0025198
Schroeder, C., & Foxe, J. (2002). The timing and laminar profile of
converging inputs to multisensory areas of the macaque neocor-
tex. Cognitive Brain Research, 14, 187–198.
Schwartz, J. L., Grimault, N., Hupé, J. M., Moore, B. C., & Press-
nitzer, D. (2012). Multistability in perception: Binding sensory
modalities, an overview. Philosophical Transactions of the Royal
Society B: Biological Sciences, 367, 896–905.
Sekuler, R., Sekuler, A. B., & Lau, R. (1997). Sound alters visual
motion perception. Nature, 385, 308. doi:10.1038/385308a0
Shams, L., Iwaki, S., Chawala, A., & Bhattacharya, J. (2005). Early
modulation of visual cortex by sound: An MEG study. Neurosci-
ence Letters, 378, 76–81.
Sheth, B. R., & Shimojo, S. (2004). Sound-aided recovery from and
persistence against visual filling-in. Vision Research, 44, 1907–
1917.
Stein, B. E., London, N., Wilkinson, L., & Price, D. (1996). Enhance-
ment of perceived visual intensity by auditory stimuli: A psycho-
physical analysis. Journal of Cognitive Neuroscience, 8, 497–506.
Stevenson, R. A., Altieri, N. A., Kim, S., Pisoni, D. B., & James,
T. W. (2010). Neural processing of asynchronous audiovisual
speech perception. NeuroImage, 49, 3308–3318.
Tsuchiya, N., & Koch, C. (2005). Continuous flash suppression reduces
negative afterimages. Nature Neuroscience, 8, 1096–1101.
Tuomainen, J., Andersen, T., Tiippana, K., & Sams, M. (2005). Audio-
visual speech perception is special. Cognition, 96, B13–B22.
van Ee, R., van Boxtel, J. J., Parker, A. L., & Alais, D. (2009). Mul-
tisensory congruency as a mechanism for attentional control over
perceptual selection. Journal of Neuroscience, 29, 11641–11649.
Vroomen, J., & de Gelder, B. (2000). Sound enhances visual per-
ception: Cross-modal effects of auditory organization on vision.
Journal of Experimental Psychology: Human Perception and
Performance, 26, 1583–1590.
Audiovisual Integration Without Visual Awareness 431

Watkins, S., Shams, L., Tanaka, S., Haynes, J.-D., & Rees, G. (2006).
Sound alters activity in human V1 in association with illusory
visual perception. NeuroImage, 31, 1247–1256.
Yamada, Y., & Kawabe, T. (2012). Illusory line motion and transfor-
mational apparent motion during continuous flash suppression.
Japanese Psychological Research, 54, 348–359.
Yang, E., Zald, D. H., & Blake, R. (2007). Fearful expressions gain
preferential access to awareness during continuous flash suppres-
sion. Emotion, 7, 882–886.
Yehia, H., Rubin, P., & Vatikiotis-Bateson, E. (1998). Quantitative
association of vocal-tract and facial behaviour. Speech Commu-
nication, 26, 23–43.
Copyright of Psychological Science (Sage Publications Inc.) is the property of Sage Publications Inc. and its
content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's
express written permission. However, users may print, download, or email articles for individual use.