McGurk Effect: Do We Hear What We See?

Dyantie Nur Azizah1

1
International Program on Science Education, Universitas Pendidikan
Indonesia, Jl. Dr. Setiabudhi No. 229, Bandung 40145, Indonesia

*dyantieazizah@student.upi.edu

Abstract. McGurk effect is known as human’s respond to the conflicting information both
visually and auditory. In this case, human’s brain can be tricked to misheard what is being said.
The common example of McGurk effect can be seen in a dubbed video. This experiment is
intended to view McGurk effect in real life application by showing a clip to the respondents
(naturally spoken and manipulated) which then resulted in various heard word while the spoken
word is the same.

1. Introduction
If we have ever been watching a dubbed television program or music video,
we may be able to notice on how audio and video tracks were slightly out of sync [1].
In other words, when our vision and hearing are being stimulated at the same time
while watching something, our brains do its job of picking up on which lip movements
go with which speed sounds [2]. Speaking of, brain can also be fooled by watching
person’s lips that would trick it to misheard what is being said – this is known as
McGurk Effect [2].
McGurk Effect was firstly explained in an experiment by psychologists Harry
McGurk and John McDonald in 1976 [2]. McGurk effect refers to what takes place
inside the brain when receiving conflicting visual and auditory information. The
movements of someone’s mouth and lips which person hears would result in a different
message perception [1][2]. The best-known case is when dubbing a voice saying onto
a face articulating results in hearing which caused fusion since the percept differs from
the sound and its visual components [3]. The McGurk effect is a powerful,
multisensory illusion as when people are having a face-to-face conversation, the brain
is engaged in complicated activity as it tries to decide how to put lip movements
together with the speech sounds that are heard. Furthermore, brain is attempting to
resolve what it thinks it is hearing with a sound closer to what it visually sees [2].
2. Method
The research method applied is quantitative in which the data are collected
through questionnaire gained after conducted a simple experiment to prove the
McGurk Effect. The details procedures are then broken down into lists as followed:
1) Review the literature about McGurk Effect
2) Set up the hypothesis, such as:
a. H0 = There is no McGurk Effect taking place during watching
un-sync video.
b. H1 = There is McGurk Effect taking place during watching un-
sync video.
3) Make an online questionnaire to be answered by the respondents.
4) Set up 31 respondents to watch a video with same sound, different visual,
and different speed. Then, ask respondents to fill up the questionnaire on
what they heard. The sample of the video can be seen in Figure 1 below.

Figure 1. McGurk Effect video

(Source: BBC Horizon Clip)

5) The experiment is conducted twice. First with eyes opened and second
with eyes closed.
6) Analyze and conclude the gained data in relation to McGurk Effect.

3. Result and Discussion

a. Result
After the experiment, the data was collected and shown in the charts below:
1) First Attempt
 In the first attempt, respondents were asked to watch a clip of someone
saying “Ba”. The first clip – named as natural, is displaying someone saying
“Ba” with “Ba” voice. However, for the second clip – named as manipulated,
is displaying someone saying “Ba” but with manipulated lip movement which
similarly looks like “Va”.
The results are shown in Charts 1 below. The result for natural clip
being watched with opened eyes shows that majority heard “Ba”. On the other
hand, the result for manipulated clip being watched with opened eyes shows
the majority heard Va” instead of “Ba”.

Charts 1. First Attempt Data

2) Second Attempt
In the second attempt in the experiment, the respondents are asked to
hear the exact same video without being able to see. The respondents were
ought to directly fill up the questionnaire as soon as they heard the voice. The
whole results are displayed in Charts 2 below.
For the natural clip, the respondents were mostly still heard “Ba”.
Then for the manipulated clip, the results are becoming various in term of
different speed. But mostly respondents heard “Ba” even in certain speed
(0.90x), those heard “Va” is still dominant with its 13 respondents.
 Charts 2. Second Attempt Data

b. Discussion
The experiment is conducted twice as aimed to gain different data with eyes
opened and eyes closed to make sure McGurk effect has real effect related to visual
and auditory. Besides, the speed of the video is also considered to further examine
the various results.
Analyzing from the first attempt of the experiment (eyes opened), for natural
clip has a good data which showed on how mostly respondents heard “Ba”. The
rest might hear another sound which might happen because of the quality and
stability sound from the speaker or due to the respondents’ distance to the speaker
and projector where the video is displayed. However, for the manipulated clip, the
result is quite interesting since the respondents are already tricked by the displayed
video so that the resulted data mostly heard someone saying “Va” instead of “Ba”
whereas the real sound of both clips is actually “Ba”.
Devolving to the second attempt (eyes closed), the voice heard from the natural
clip is shown in Charts 2 which can be interpreted that mostly respondents heard
“Ba” but those who answered “Va” are also increasing. This case might happen
due to its time spare from doing the first attempt and second attempt that is too
close which may distract respondents’ respond according to the sound and the
video. Furthermore, for the data of manipulated clip in second attempt is enticing
on how respondents were mostly heard “Ba” in normal speed and 0.60x speed
while in 0.90x speed mostly still heard “Va”. The case might probably similar to
the previous one. But from this second attempt, it can be clearly seen that the data
is similar to the theory of McGurk effect especially in terms of fusion due to the
video’s conflicting information of visual and auditory. Moreover, it resulted in the
different perception of information reception.

3) Conclusion
From the experiment, it was found that McGurk has a real effect
towards individual’s perception in receiving information in terms of visual and
auditory. It means that hypothesis (H1) is evident. Speaking further, McGurk
effect would be an excellent tool to investigate multisensory integration in
speech perception. McGurk effect can be referred to a change in auditory
perception due to the unsynchronized visual speech that the observers hear
other sound than what is actually delivered. However, the experiment should
be done to more respondents with taking care of every details of technical and
timing issues to get more valid data.

4) Acknowledgments
Authors wishing gratitude for the assistance and encouragement from
colleagues, and lecturers of Science in Daily Life Course.

5) References
[1] Cowen, Amy. (September 1, 2016). Putting the McGurk Effect to the Test.
Retrieved from: https://www.sciencebuddies.org/blog/putting-the-
mcgurk-effect-to-the-test
[2] Nierenberg, Cari. (February 28, 2017). The Strange ‘McGurk’ Effect: How
Your Eyes Can Affect What You Hear. Retrieved from:
https://www.livescience.com/58047-mcgurk-effect-weird-way-eyes-
trick-brain.html
[3] Tiippana, Kaisa. (July 10, 2014). What is the McGurk effect?. Retrieved
from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4091305/