Computers in Human Behavior 27 (2011) 11–15

Contents lists available at ScienceDirect

Computers in Human Behavior

journal homepage: www.elsevier.com/locate/comphumbeh

Effectiveness of on-screen pinyin in learning Chinese: An expertise reversal

for multimedia redundancy effect
Chee Ha Lee *, Slava Kalyuga
School of Education, The University of New South Wales, NSW, Australia

a r t i c l e i n f o a b s t r a c t

Article history: The reported study is aimed at developing effective techniques for reducing learner cognitive overload
Available online 31 May 2010 while using pinyin (a phonetic system) to learn the Chinese language. According to the multimedia
redundancy effect, commonly used concurrent visual presentations of characters and pinyin, as well as
Keywords: their auditory pronunciations might impose an unnecessary extraneous load on learners and thus elim-
Cognitive load inate any potential benefits of using pinyin. In this study, learning effects of three computer-based pre-
Classic Chinese sentation techniques (full on-screen pinyin transcription, partial on-screen pinyin transcription, and no
Pinyin
on-screen pinyin) were compared for learners with higher and lower levels of prior language proficiency
Multimedia redundancy effect
Expertise reversal effect
and pinyin knowledge. Results demonstrated the superiority of the partial pinyin condition over the two
other conditions for more experienced learners. No differences between the three experimental groups
were found for learners with lower levels of prior knowledge. The effectiveness of different pinyin pre-
sentation techniques depended on levels of learner prior knowledge (an expertise reversal effect).
Ó 2010 Elsevier Ltd. All rights reserved.

1. Introduction
The basic unit of the Chinese written language is a character.
Because of the ideographic nature of the written language, pronun-
ciation information conveyed by characters is very limited. Pinyin
represents a modern phonic transcription system that assists in
learning pronunciation of characters in Putonghua (the official spo-
ken language of China, also called Mandarin). Pinyin shares the
alphabet with English, and diacritical signs are marked above the
vowel of a pinyin to indicate the tone of the word. A syllable pro-
nounced in different tones would represent different characters
and result in different meaning in the Chinese language (e.g., /
mā/, /má/, /mă/, /mà/).
In China, students are usually well trained in reading and pro-
nouncing pinyin before they start learning characters. Pinyin is rec-
ommended for assisting students to read independently at early
stages of learning. Therefore, characters in the Chinese language
textbooks (and more recently, in computer-based materials) for
early stage learning are usually transcribed with pinyin and only
new words are transcribed with pinyin for later stages (Zhang &
Zhu, 2007).
It is assumed that young readers could comprehend text com-
posed of a large amount of unfamiliar characters with the assis-
* Corresponding author. Address: School of Education, The University of New
South Wales, Sydney, NSW 2052, Australia. Tel.: +61 2 9385 1958; fax: +61 2 9385
1946.
E-mail address: cheelee.hilda@gmail.com (C.H. Lee).
tance of the pinyin transcription. For example, Shu, Zeng, and
Chen (1993) demonstrated that junior primary students performed
better on a vocabulary meaning test (involving words familiar to
participants in their spoken language) in a characters-with-pinyin
experimental condition than in a characters-only condition. Shu
and Liu (1994) also found the advantage of using pinyin to help
young children in sentence reading. However, in both studies,
the pinyin condition did not demonstrate any advantage in test
questions that included vocabulary items that were not familiar
to the participants in their spoken language. While the pinyin tran-
scription for the words that were familiar to participants in their
spoken language allowed them to comprehend the written text,
it was not the case for unfamiliar words. Limitations of using pin-
yin for learners who do not use Mandarin as a daily spoken lan-
guage (i.e., learners with a Chinese dialect background) were also
demonstrated by Xu, Bao, and Xu (1997).
In practice, pinyin is often used simultaneously with its audi-
tory pronunciation. Chung (2003) compared a visual pinyin pre-
sentation with an auditory pronunciation condition (without
pinyin transcription) and demonstrated superiority of the visual
pinyin on a pronunciation recall test. This was a surprising result
because it had been expected that encoding activities could be
enhanced using a dual-modality instructional format (a character
with its verbal pronunciation). According to the modality effect
in cognitive load theory, presenting a picture (character) with
written explanations (pinyin) could be less effective than pre-
senting it with spoken explanations (pronunciation). Chung
(2003) suggested that the visual pinyin might have allowed the

0747-5632/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved.
doi:10.1016/j.chb.2010.05.005
12 C.H. Lee, S. Kalyuga / Computers in Human Behavior 27 (2011) 11–15
rehearsal of its pronunciation in working memory, while the ver-
bal pronunciation vanished quickly and was difficult to recall. In
regards to the effects of these instructional conditions on the
meaning recall post-tests, no significant differences were ob-
served. Based on the description of the experimental procedure,
it is possible to speculate that participants’ prior skills in reading
pinyin could have reduced working memory load and contrib-
uted to these results.
The study described in this paper investigated the effects of
simultaneously presenting visual pinyin and its auditory pronunci-
ation compared to an auditory-only presentation (no-pinyin condi-
tion) using relatively more complex learning materials (classical
Chinese texts) in a computer-based learning environment. The
instruction with visual pinyin included two forms corresponding
to different levels of pinyin presence: a full pinyin condition in
which all the characters were transcribed with pinyin and a partial
pinyin condition in which only potentially new or key characters
were transcribed with pinyin.
Classical Chinese is the language of ancient Chinese historical,
philosophical and poetic texts. Relative to modern Chinese, it is
characterized by using significantly fewer characters for express-
ing ideas. Also, grammar rules and vocabulary in classical Chi-
nese are different from modern Chinese (Faurot, 1995). From
the perspective of cognitive load theory, classical Chinese is
characterized by high levels of element interactivity for begin-
ners. For students who are not familiar with Mandarin (including
participants in this study), the level of intrinsic cognitive load
would be further increased. The majority of Hong Kong students
use Cantonese (one of the Chinese dialects) for daily communi-
cation. Learning Mandarin is recommended for all students from
Year 1 to Year 9 levels, and pinyin is used conventionally to
accompany characters in Mandarin textbooks and computer-
based materials to assist in learning pronunciation. Since Hong
Kong students are familiar with English, distinguishing English
and pinyin may impose an additional load before students be-
come fluent pinyin readers. McGinnis (1997) suggested the term
the can effect to describe the situations when pinyin spellings
could be very easily confused with pronunciation of English
(e.g., can is pronounced as /tshan/ in pinyin). In addition, stu-
dents may also require cognitive resources to cross-reference
the Mandarin pronunciation of characters with their Cantonese
pronunciation.
Thus, if pinyin is used with complex learning materials like
classical Chinese texts, the level of intrinsic cognitive load could
be very high. In this situation, according to the multimedia redun-
dancy effect (Kalyuga, 2000; Kalyuga, Chandler, & Sweller, 1999,
2004; Mayer, Heiser, & Lonn, 2001), a concurrent presentation
of Chinese characters (pictorial representations), pinyin (visual
verbal representations), and its auditory pronunciation (spoken
verbal representations) may impose an unnecessary extraneous
load and thus eliminate any potential benefits of using pinyin.
The visual pinyin or its auditory equivalent could be regarded
as a redundant source of information. Since the auditory equiva-
lent is essential for learners who may not be skillful in reading
pinyin during early stages of learning, it is expected that visual
pinyin may inhibit learning, especially for learners who are not
familiar with spoken Mandarin and not experienced in reading
pinyin.
However, according to the expertise reversal effect (Kalyuga,
2007; Kalyuga, Ayres, Chandler, & Sweller, 2003), levels of learner
prior experience could affect the effectiveness of an instructional
technique. It could be predicted that the effectiveness of pinyin
might depend on levels of learner prior language proficiency and
experience with pinyin. Therefore, possible interactions between
instructional formats and learner levels of expertise were also
investigated in this study.
2. Methods
2.1. Participants
Six classes of 240 Year 8 students (113 females and 127 males;
average age 14.04 years, SD = 0.68) from two Hong Kong subsi-
dized secondary schools participated in the experiment. At each
school, three classes were selected randomly.
2.2. Materials and procedure
The experiment included a pretest phase, a learning phase, and
a post-test phase.
2.2.1. Pretest phase
The pretest phase was conducted in a normal classroom setting
in a single session of 25 min. Participants were first asked to com-
plete a brief questionnaire to collect demographic information
including family language background. The pretest was used to
evaluate the overall level of participants’ combined prior knowl-
edge of pinyin and Mandarin language. The paper-based, multi-
ple-choice pretest consisted of three questions each of which
included 12 items, and participants were instructed by a pre-re-
corded voice message. The first two questions were used to evalu-
ate familiarity with pinyin and ability to distinguish four tones of
Mandarin. Students were requested to match a written pinyin with
a pronunciation they heard. The last question evaluated students’
familiarity with pronunciations of some of the characters selected
from the learning materials. Students were requested to judge
whether a pronunciation they heard corresponding to a character
was correct or not. Each response was scored 1 (correct) or 0
(incorrect). The maximum score of 36 could be obtained if all items
were answered correctly. The answer options ‘I don’t know if it is
correct or incorrect’ or ‘I can’t make a judgment’ were included
in all items to prevent participants from guessing their answers.
Cronbach’s a of 0.65 (for a modified sample N = 120, see Section 3)
indicated a sufficiently high level of internal consistency for the
three pretest questions. Following the pretest phase, participants
in each school were listed in ascending order according to their
pretest scores and alternatively allocated to three experimental
conditions (i.e., full pinyin transcription, partial pinyin transcrip-
tion, and no-pinyin condition).
2.2.2. Learning phase
The learning and post-test phases were conducted 1 week later
in the language laboratory in a single session of 40 min. Twelve
sentences were selected from several classic Chinese text. Using
a pre-recorded voice message, the participants were instructed to
read and comprehend 12 sentences shown on the computer screen
and learn the pronunciation of the characters in Mandarin. They
were also told that a short test would immediately follow the
learning phase. The learning materials were presented on the com-
puter screen using PowerPoint slides. The presentation used tradi-
tional characters and a common Chinese font Kaiti ( ). The visual
pinyin transcription in the corresponding experimental condition
was presented just above the characters (see Fig. 1). This presenta-
tion format corresponded to the format in the Mandarin textbooks
and materials used in the two participating schools.
The distances between the Chinese characters and pinyin were
the same across different PowerPoint slides. The levels of difficulty
of the sentences were judged by four experienced Chinese lan-
guage teachers from the participating schools as appropriate for
their Year 8 students (the content and structure of the sentences
were not too hard or too easy to comprehend). The time spent
on reading each sentence was equalized throughout the three
 C.H. Lee, S. Kalyuga / Computers in Human Behavior 27 (2011) 11–15
Fig. 1. Examples of experimental formats used in the learning phase.
instructional conditions. The learning procedure and the sentence
presentation order were also the same. Each sentence (with full,
partial, or none pinyin transcription added depending on the
experimental condition) was first shown on the computer screen
for 3–7 s depending on the length of the sentence. It was then fol-
lowed by a voice-over pronouncing the sentence three times with
4 s intervals between each reading (the sentence remained on the
screen). The PowerPoint slides changed automatically, and the
word (The end) was shown on the last slide.
2.2.3. Post-test phase
A multiple-choice, paper-based test consisted of three sets of
items measuring sentence comprehension, recognition of charac-
ters, and pronunciation, respectively. The options ‘I don’t know if
it is correct or incorrect’ or ‘I can’t make a judgment’ were provided
13
for sentence comprehension and pronunciation items to prevent
participants from making random choices. The post-test was
accompanied by pre-recorded instructions to ensure that all partic-
ipants were allocated the same amount of time for the test. They
were not allowed to go back to the previous questions. The first
set of five items was designed to assess comprehension of the
learning materials. The measures were based on evaluating the
adequacy of the learner constructed text representations after
reading the learning materials. In each item, participants were gi-
ven a sentence written in modern Chinese with meaning presum-
ably corresponding to the meaning conveyed by the learning
materials and were asked to judge whether the statement was cor-
rect or not. Statements were written according to possible correct
or incorrect representations. The adequacy of learner representa-
tions was judged by their ability to verify the correctness of the
presented statements. The second set of six items was designed
to evaluate learners’ recognition of characters. Corresponding six
sentences were selected from the learning materials with some
characters modified (usually a homophone or a character that
shares a similar graphic structure with the correct original one).
Participants were asked to circle a wrongly used character (if
any) and write down the correct one if applicable. A voice-over
pronouncing the sentences was provided. The third set of seven
items was designed to evaluate learners’ pronunciation perfor-
mance. For these items, sentences or phrases were selected from
the learning materials. For each item, students were required to
judge whether the pronunciation of the target character (presented
with a bracket) was correct or not. For the first and third sets of
items, each response was scored 1 (correct) or 0 (incorrect). For
the second set of items, each response was scored 1 (correct), 0.5
(the correct character was not provided for a correctly identified
modified character) or 0 (incorrect). The maximum scores of 5, 6,
and 7 could be obtained for measures of sentence comprehension,
recognition, and pronunciation correspondingly if all items were
answered correctly.
3. Results
The dependent variables under analysis were post-test scores
for sentence comprehension, recognition of characters, and pro-
nunciation, while the independent variables were instructional for-
mats (full pinyin, partial pinyin, or no pinyin) and levels of learner
expertise (lower and higher level).
In order to exclude students with a strong Mandarin speaking
background, the participants who had indicated in the question-
naire that they studied in Mainland China and communicated with
their family members (native Mandarin speakers) in Mandarin
were eliminated from the analysis. Pretest scores for the remaining
178 students were used as indicators of levels of learner prior
experience with pinyin and Mandarin language. In order to obtain
two groups of learners with distinctively different levels of prior
knowledge, the pretest scores were arranged in an ascending order
and a median range of about one third of participants (58 students
with pretest scores between 21 and 25) was excluded from the
analyses. Subsequently, the total number of participants in the
modified experimental groups was 120.
With the three modified groups (n = 43 for the pinyin group,
n = 39 for the partial pinyin group, and n = 38 for the no-pinyin
group), analysis of variance (ANOVA) indicated a highly non-signif-
icant difference for the pretest scores, F(2, 117) = 0.78, MSE = 36.40
(M = 22.53, SD = 6.54 for the pinyin group; M = 22.44, SD = 5.68 for
the partial pinyin group; and M = 22.95, SD = 5.78 for the no-pinyin
group; .05 significance level was used throughout the manuscript).
Thus, the three groups of participants were at the same level of
14 C.H. Lee, S. Kalyuga / Computers in Human Behavior 27 (2011) 11–15

7 auditory pronunciation, the results demonstrated an interaction

between instructional conditions and levels of learner expertise
full pinyin (an expertise reversal effect). The partial pinyin condition outper-
6.5
partial pinyin formed the no-pinyin conditions for more knowledgeable learners,
no pinyin however this superiority was found in learning pronunciation but
6 not in sentence comprehension and recognition of characters. For
less knowledgeable learners, there were no differences among
5.5 the three conditions.

5 4. Discussion

This paper investigated cognitive load aspects of learning the

4.5
4 is conventionally taken for granted that pinyin is always useful
3.5
3 concurrently.
2.5
2 might need to cross-reference the spoken and visual sources of
lower prior higher prior
knowledge level knowledge level
Fig. 2. Interaction between instructional formats and levels of learner expertise for
post-test scores measuring learning of pronunciation.
prior familiarity with pinyin and Mandarin language, and none of
the groups had an initial advantage in the experiment.
MANOVA for the three dependent variables indicated no signif-
icant main effect for experimental conditions, H = 0.43,
F(3, 113) = 1.62, g2p = 0.04. Expectedly, there was a significant main
effect for the expertise levels, H = 0.22, F(3, 112) = 8.23, g2p = 0.18,
p < .01. Most importantly, MANOVA indicated a significant interac-
tion between instructional conditions and levels of learner exper-
tise, H = 0.08, F(3, 113) = 3.03, g2p = 0.07, p < .05 (see Fig. 2).
Means and standard deviations are provided in Table 1.
Separate univariate ANOVAs for interactions between instruc-
tional conditions and levels of learner expertise for each dependent
variable revealed a significant effect for measures of pronunciation
learning, F(2, 114) = 4.04, MSE = 2.19, p < .05, g2p = 0.07. There were
no significant effects for measures of sentence comprehension,
F(2, 114) = 0.20, MSE = 0.94, and recognition of characters,
F(2, 114) = 1.97, MSE = 0.99.
In regards to simple effects for learning pronunciation, ANOVAs
indicated a significant difference between the three instructional
formats for higher knowledge level learners, F(2, 54) = 3.67,
MSE = 2.69, p < .05, g2p = 0.12, with no simple effect for lower knowl-
edge level learners, F(2, 60) = 0.65, MSE = 1.73. According to Tukey
HSD post hoc test, the partial pinyin format outperformed the no-
pinyin format for the higher knowledge level learners (p < .05).
Thus, although no overall multimedia redundancy effect was
found in using visual pinyin transcription simultaneously with its
Chinese language using a phonic transcription system called pin-
yin. Because of the ideographic nature of the Chinese language, it
in providing pronunciation information for Chinese characters
and should accompany characters in learning materials for begin-
ner learners. It is also a common practice to present characters
with both pinyin and verbal pronunciation instructions
The multimedia redundancy effect generated by cognitive load
theory may render the presentation of pinyin accompanied by its
verbal pronunciation an ineffective instructional method. Learners
essentially the same information. Mental processing of duplicate
pronunciation information might therefore leave insufficient cog-
nitive resources for learning schemas required for comprehension,
characters recognition, and learning pronunciation. It is suggested
that an instructional format without visual pinyin (i.e., presenting
characters and their verbal pronunciation) could be a more effec-
tive alternative.
The results of the described experiment indicated no overall
multimedia redundancy effect. However, an expertise reversal ef-
fect was found demonstrating a significant interaction between
the effectiveness of alternative instructional techniques in learning
pronunciation and levels of learner prior experience. For less expe-
rienced learners, no differences were found between the full pinyin
transcription condition, partial pinyin transcription condition, and
no-pinyin condition. Insufficient knowledge in learner long-term
memory may have resulted in a very high level of intrinsic cogni-
tive load that overloaded working memory. For more experienced
learners, the partial pinyin condition outperformed the no-pinyin
instructional format in learning pronunciation, with no differences
between other conditions. The available knowledge structures may
have reduced the level of intrinsic load sufficiently to allow pro-
cessing of the redundant visual pinyin.
Mayer and Johnson (2008) have recently demonstrated that the
multimedia redundancy principle (according to which graphics
and narration are superior to graphics, narration, and on-screen
text) may not apply when the on-screen text is a short version of
the narration that highlights its key points and is placed next to
the corresponding portion of the graphic. The pinyin transcription
in the partial pinyin condition was provided only next to poten-
tially new (key) characters to draw the learner’s attention and
could be regarded as exactly this case.
The above advantage of partial pinyin was not demonstrated for
the sentence comprehension and character recognition test tasks

Table 1
Means and standard deviations for post-test scores measuring learning pronunciation for different experimental conditions and levels of learner prior knowledge.

Lower prior knowledge level Higher prior knowledge level

Full pinyin Partial pinyin No pinyin Full pinyin Partial pinyin No pinyin
M 3.50 3.09 3.47 4.52 5.47 4.00
SD 1.10 1.02 1.78 1.89 1.23 1.67
 C.H. Lee, S. Kalyuga / Computers in Human Behavior 27 (2011) 11–15
for which no differences were found between the three experimen-
tal conditions. This result is in correspondence with some previous
studies (Shu et al., 1993; Shu & Liu, 1994) that have suggested that
even though pinyin enhanced independent reading and character
recognition for young Chinese children (native speakers), it may
not be beneficial for children with dialect background when deal-
ing with new vocabulary items that are unfamiliar to learners in
their spoken language (Xu et al., 1997). This is the category of par-
ticipants used in this study. The benefits of pinyin were demon-
strated only in learning pronunciation.
The conventional believe that pinyin should necessarily benefit
learning the Chinese language may not be always correct. The re-
ported study indicates that the effectiveness of pinyin as a means
of instructional support may depend on learner prior language pro-
ficiency and experience with pinyin, as well as on the level of the
provided support (full or partial support). The effect may also apply
only to certain aspect of language learning (learning pronunciation
of characters). The results highlight the importance of assessing ac-
tual levels of learner prior knowledge and tailoring accordingly the
levels of pinyin support and presentation formats. The effective-
ness of such adaptive computer-based learning environments
(including instructional procedures based on learner control over
required levels of pinyin support) needs to be investigated in fu-
ture studies.
Acknowledgments
The authors would like to thank Yin Ping Chiu, Yee Mei Wan,
and the Principals of the participating schools for their support in
this study. The authors also thank the technicians and the students
for their cooperation.
15
References
Chung, K. K. H. (2003). Effects of pinyin and first language words in learning of
Chinese characters as a second language. Journal of Behavioral Education, 12,
207–223.
Faurot, J. L. (1995). Gateway to the Chinese Classics: A practical guide to literary
Chinese. San Francisco, CA: China Books and Periodicals.
Kalyuga, S. (2000). When using sound with a text or picture is not beneficial for
learning. Australian Journal of Educational Technology, 16, 161–172.
Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored
instruction. Educational Psychology Review, 16, 161–172.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). Expertise reversal effect.
Educational Psychologist, 38, 23–31.
Kalyuga, S., Chandler, P., & Sweller, J. (1999). Managing split-attention and
redundancy in multimedia instruction. Applied Cognitive Psychology, 13,
351–371.
Kalyuga, S., Chandler, P., & Sweller, J. (2004). When redundant on-screen text in
multimedia technical instruction can interfere with learning. Human Factors, 46,
567–581.
Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia
learning: When presenting more material results in less understanding. Journal
of Educational Psychology, 93, 187–198.
Mayer, R. E., & Johnson, C. I. (2008). Revising the redundancy principle in
multimedia learning. Journal of Educational Psychology, 100, 380–386.
McGinnis, S. (1997). Tonal spelling versus diacritics for teaching pronunciation of
Mandarin Chinese. The Modern Language Journal, 81, 228–236.
Shu, H., & Liu, B. X. (1994). Pinyin zai xiaoxue dinianji ertong zaoqi yuedu zhong
zuoyong de yanjiu (The role of pinyin in early reading for Chinese junior
primary school children). Psychological Development and Education, 3, 11–15.
Shu, H., Zeng, H. M., & Chen, Z. (1993). Xiaoxue dinianji ertong liyong pinyin xuexi
shengzici de shiyan yanjiu (Research on using pinyin in learning vocabulary by
junior primary school children). Psychological Development and Education, 1,
18–22.
Xu, F., Bao, X. H., & Xu, M. (1997). Pinyin zai fangyan ertong zici xuexi zhong de
zuoyong (The role of pinyin in learning vocabulary for Chinese children with
dialect background). Chinese Journal of Applied Psychology, 3, 39–44.
Zhang, Q., & Zhu, J. L. (Eds.). (2007). Yu Wen (Language). Year 1–Year 6. Nanjing,
China: Jiangsu Education Publishing House.