Does TOEIC measure

Running Head: DOES TOEIC MEASURE THE TEST-TAKERS OVERALL

ENGLISH COMMUNICATION SKILLS?

Does TOEIC measure the test-takers overall English Communication

Skills?
Kiyoung Sohn
California State University Monterey Bay

IST 595 Graduate Writing Workshop

Professor Karen Wisdom

Does TOEIC measure

August 4, 2009

Does TOEIC measure

The Test of English for International Communication (TOEIC) has been widely used in
many non-English speaking countries, as a standardized test that measures English
communication skills primarily in a business setting. According to Education Test Service (ETS),
over the last three decades, more than 8,000 businesses, governments, and organizations in more
than 90 countries use the TOEIC test as a selection tool for qualified employees in the
international market place. The TOEIC is also used in academic settings, in over 9,000 schools
across the world. The test is used in the process of recruiting high-achieving students, evaluating
students progress, documenting program and curricula effectiveness, and preparing students for
the future job market.
It may be easily assumed that, with the popular use of the test, both in academic and
employment contexts, TOEIC is a valid measure for English communication skills of test-takers.
While this fact strongly suggests the practicality of the test, this is by no means the legitimate
reasoning to come to a quick conclusion that the TOEIC perfectly serves its intended purpose.
Considering that the TOEIC is such a high-stake test which has an impact on test-takers lives, it
is only reasonable to empirically prove its validity as a standardized test for the intended
purpose. To investigate this legitimacy of the TOEIC, many researchers and scholars have
conducted considerable studies about the validity and reliability of the TOEIC, both within and
outside of the ETS. The original TOEIC test was seemingly limited to fulfill its intended purpose
in that the test directly measures only receptive skills, listening and reading. However, there has
been considerable research that suggests it is a valid assessment criterion for evaluating the testtakers overall English communication skills.
This seemingly limited characteristic of the original TOEIC called for a more complete
assessment of test-takers English communication skills. In an attempt to address this problem

Does TOEIC measure

and directly assess English speaking and writing skills in an employment setting, speaking and
writing sections were added to the original TOEIC. The validity of this new TOEIC should also
be evaluated through carefully designed and comprehensive research and study.
To evaluate whether the TOEIC is a legitimate assessment for English communication
competence for its intended purpose, this paper will discuss the test more in-depth in the
following sections. In order to give the readers a better and clear idea of the test, a summary of
the test will be provided in terms of its population, format, scoring process, and finally its main
use. Then, in the analysis section, the author will discuss whether or not the TOEIC measures
test-takers English skills effectively. The benefits to the test-takers and test-users will be
discussed as well as the limitations of the test in accomplishing its intended purpose. This paper
concludes that, after the discussion of the test, its validity and reliability, the TOEIC offers value
to both test-takers and test-users alike in spite of identified restriction.

The TOEIC was first developed by ETS in 1979 to measure the test-takers ability to
listen and read in English primarily in the international business setting. Since its development,
the TOEIC has been widely used across the world in both employment and academic settings.
Every year, more than two million people take the TOEIC (ETS, n.d.). The majority of people
taking the exam reside in Asian countries such as Japan, Korea and Thailand. According to the
TOEIC Report on Test-Takers Worldwide, 1997-98, 63% of the TOEIC scores were used in
Japan, 29% were used in Korea, and 8% in other countries around the world. More recent
statistics from ETS (2005, p.4) reports, Japan and Korea accounted for approximately 81% of
test takers worldwide.

Does TOEIC measure

The majority of the tests taking population in non-English speaking countries are students
in higher education, future employees, and professional workers who want to advance their
education or career. Many schools and institutions use the TOEIC scores for entrance
instruments, placement tests, and as a graduation requirement. Expanding numbers of companies
and corporations require the TOEIC score from perspective employees in recruitment as well as
current employees in determining promotion and overseas assignments.
The TOEIC was originally developed to measure listening and reading skills in English
as well as speaking and writing skills indirectly (Ito, Kawaguchi, & Ohta 2005). Recently, in
response to growing needs for a competent English speaking workforce in a multinational work
setting, speaking and writing sections were added to the TOEIC, so that it can better serve as a
full assessment of all four language skills. The test is designed to assess the examinees ability to
carry out every day English tasks using real-life situations and context.
The assessment approaches used in the TOEIC should be addressed in two parts: 1)
Listening and Reading, 2) Speaking and Writing, since these two are distinctly different in their
formats.The Listening and Reading test is a paper-based assessment. The test takes about two
and half hours with two sections of 100 multiple-choice questions. The listening section is
composed of four sections: Photographs; Question-Response; Short Conversation; and Short
Talks. The reading section consists of three parts: Incomplete Sentences (40 questions), Error
Recognition (20 questions), and Reading Comprehension (40 questions) with one and a half
hours to complete. The new TOEIC Speaking and Writing tests are taken through the Internet
worldwide. The TOEIC Speaking Test consists of 11 questions, with the time duration of about
20 minutes. The TOEIC Writing Test has 8 tasks to complete in about an hour.

Does TOEIC measure

In the United States, TOEIC Listening and Reading tests are proctored at government
agencies, schools, and corporations for their current or potential employees or students of
English-language programs under the Institutional Testing Program. Also, organizations can send
test takers to open sessions rather than testing them on their own sites. Outside the United States,
the TOEIC Listening and Reading tests are administered on scheduled dates at various locations
through the Public Testing Program.
The scoring process of the TOEIC seems to relatively open and straightforward.
According to ETS (2009), the TOEIC Listening and Reading tests are scored by the number of
correct answers and then are converted to a scaled score. In the score report, test takers receive
listening, reading as well as total scaled scores. The total scaled score is derived from adding the
two scaled scores together. A total score ranges from 10 to 990 with each section of the test
ranging from 5 to 495. In addition, the ETS TOEIC website provides the TOEIC Listening and
Reading Score Descriptors, which summarizes a test takers strengths and weaknesses by scores
in increments of 50, and suggests the areas needing improvement. As for the rating process for
the TOEIC Speaking and Writing tests, test-takers' responses are sent to ETS's Online Scoring
Network (OSN), where certified ETS test raters score them. The score ranges from 0 to 200 for
Speaking and Writing tests respectively. The ETS website provides a simple guideline for
scoring as well as Speaking and Writing proficiency level descriptors for more detailed
information.
With regard to the tests validity and reliability, Perkins (1987, p. 82, as cited in Wilson,
2001), in his review of the test, asserts the TOEIC is a standardized, highly reliable and valid
measure of English, specifically designed to assess real-life reading and listening skills of
candidates who will use English in a work context. Zhang (2006) seems to be along the same

Does TOEIC measure

line with Perkins (1987). In her study, Investigating the relative effects of persons, items, sections,
and languages on TOEIC score dependability, Zhang (2006) investigated two language groups,

Korean and Japanese to conduct analysis and reported that there is highly reliable score
dependability, suggesting that the TOEIC scores are reliable and can be generalized.
The TOEIC test and the scores are used by a variety of organizations and educational
institutions. From small businesses and multinational organizations to government agencies, a
large number of organizations use the TOEIC test scores as an assessment tool in recruiting,
promoting, job training, and deploying employees on overseas assignments. Also, language
schools use the test as a class placement test in placing incoming and continuing students into
appropriate class levels, tracking student progress, and diagnosing the effectiveness of the
school's program by comparing the test scores at the beginning and end of the course. Many
universities and institutions of higher education often use the TOEIC score as a graduation
requirement to indicate the students English language ability to perform well in the work place
after they graduate.
Based on the summary of the TOEIC, the rest of the paper analyzes the test in terms of
whether or not it meets its intended purpose. The TOEIC is reported to be a relevant measure of
the language ability of test-takers according to the intended purpose of the test. Having in mind
that the test was developed to diagnose the test-takers English knowledge, skills, and abilities in
international business settings, it is acknowledged that ETS explicitly indicates that the tests use
common everyday business phrases, key expressions, and vocabulary. Also,

the content of the test is

from work related situations and settings including business meetings and appointments, which
is consistent with its purpose.

Does TOEIC measure

In order to evaluate whether TOEIC measures test-takers English skills effectively, many
researchers have designed and conducted numerous studies. Both the original TOEIC with
Listening and Reading tests and the new TOEIC have been reported as an adequate measure of
the test-takers ability in English and as an effective indicator of the test-takers knowledge,
skills and abilities. Nagara, Wilson, and Woodhead (2004) reported that scoresontheTOEIC
havebeenfoundtobestronglyrelatedtotheperformanceontheLanguageProficiencyInterview
(LPI)procedure,whichisadirectmeasureofspeakingproficiency.Theauthorsobservedsample
groupsofnativespeakersofJapanese,Korean,French,Spanish,andArabictocomparethe
performanceintheTOEICandLPI.Thestrongcoefficientthroughoutallthesubgroupsinthe
studysuggeststhattheTOEICactuallydeterminestesttakersabilityaccurately.As for the new
TOEIC speakingandwritingtests,testtakersperceptionwereexaminedtoevaluatewhetherthe
testisagoodpredictoroftheirsuccessinthefuture(Powers, Kim, Yu, Weng, & VanWinkle,
2009).AselfassessmentinventoryadministeredtoTOEICexamineesinJapanandKorea
revealedthatthereisarelativelystrongrelationbetweenTOEICscoresandtesttakersself
reportsforbothspeakingandwritingtasks (Powers, Kim, Yu, Weng, & VanWinkle, 2009). In the
study, the authors investigated thetesttakersperceptionsoftheirperformanceinvarious
everydayEnglishlanguagetasks.TheyshowedthatexamineesateachhigherTOEICscorelevel
weremorelikelytoreportthattheycouldsuccessfullyaccomplisheachoftheeverydaylanguage
tasksinEnglish.
As the study by Powers, Kim, Yu, Weng, and VanWinkle (2009) suggested, being
successful in the test appears to transfer to the test-takers performance in real life. Not only do
test-takers perceive it as personal success, they are also rewarded by obtaining what they want to
gain, promotion, due to their test performance. Higher TOEIC scorers insure better chances to be

Does TOEIC measure

admitted to schools and institutions as well as a wider range of job opportunities. Currently,
working individuals can also benefit from the high scores with better career opportunities
through promotions and overseas assignments. In addition, test-takers who take the test regularly
can check their progress in everyday English performance and improve their English proficiency.
While the test appears to be serving the intended purpose appropriately, there seems to be
limitations. Although it is claimed that the test is equitable and appropriate for the test-takers
worldwide, the test materials were based on North American standards only (Suzuki, & Daza,
2004). This limitation was, however, addressed in the revision made in July 2005 by adopting
different varieties of English outside North America, such as Australia (Powers, Kim, Yu, &
Weng, 2008). Also, Buck (2001, as cited in Suzuki & Daza, 2004, p.21) points out that the
TOEIC is an effective tool that provides people with an accurate idea of a persons English
proficiency level. to be practical and accurate, especially when used for screening purposes in
the intermediate levels of English proficiency. Suzuki and Daza (2004) echoed this position,
noting that the most targeted test-takers are limited to lower intermediate level. This limited
target audience may affect the scope of the test practicality.
However, the test seems well designed and reliable to minimize any possible testing bias.
Takada (2007) asserts that, although his discussion is limited to Listening and Reading tests, the
TOEIC has low probability of causing problems in terms of the test setting, rubric, input and the
expected response. As he points out, there is no special kind of equipment required to administer
the test other than audiotapes and paper and pencil format. It doesnt require any special kind of
equipment. Also, the test rubric is described as simple and direct, so it is unlikely that it causes
different test takers to perform differently (Takada, 2007, p. 159). Provided examples before the

Does TOEIC measure

10

listening test was found to be helpful for the test-takers to have a clear idea of what is expected
in each part of the test.
Lastly, the intended purpose of the test and the actual use of test scores discussed above
leads us to believe that the test results are used appropriately. That is, the fact that the TOEIC
aims to measure test-takers English ability in everyday work places. This is reflected in the use
of the test results in assessing this ability in the screening process of entering and exiting schools,
and hiring and promoting within work places. Thus, the test results appear to be used according
to its intended purpose.
While there will always be more room for better designed future research and study for
any kind of academia and fields, it appears that the TOEIC is aligned with its intended purpose
and current needs of language skills and abilities in this globalization era. Examining existing
research and studies, it was suggested that the TOEIC is a legitimate diagnostic tool for testtakers English ability to be used in both employment and academic settings. The content of the
test is proved to be consistent with its intended purpose by measuring the English proficiency,
using real-life situations in work environments appropriately. In addition, the test seems to serve
as a good indicator for test-takers future success. Although there are some limitations to the test,
the benefits of the test out rule them. For schools and business organizations, the test offers a
reliable assessment tool of students and employees English proficiency needed in their
organization. For test-takers, the test can serve as a tool for advancing their careers as well as
diagnosing individual skills.

Does TOEIC measure

11

References:
Buck. G. (2001). Assessing listening. Cambridge, UK: Cambridge University Press.
Educational Testing Service. (2005). New TOEIC test official workbook. Tokyo: The Institute for
International Business Communication.
ETS (n.d). TOEIC can-do guide: Linking TOEIC scores to activities performed in English.
Princeton, NJ: Author.
ETS TOEIC Home, Retrieved in July 20, 2009
http://www.ets.org/portal/site/ets/menuitem.fab2360b1645a1de9b3a0779f1751509/?
vgnextoid=06cfd898c84f4010VgnVCM10000022f95190RCRD
Ito, T., Kawaguchi, K., & Ohta, R. (2005). A study of the relationship between TOEIC scores and
functional job performance: Self-assessment of foreign language proficiency (TOEIC
Research Rep. No. 1). Tokyo: Institute for International Business Communication.
Nagara, S., Wilson, K., & Woodhead, R. (2004). The TOEIC/LPI Relationships in Academic and
Employment Contexts in Thailand. (ETS-RR-04-16), Princeton, NJ: ETS.
Retrieved from ETS TOEIC Home, in June 20, 2009
http://www.ets.org/Media/Research/pdf/RR-04-16.pdf
Perkins, K. (1987). Test of English for International Communication. In J.C. Alderson, K.J.
Krahnke, & C.W. Standsfield (Eds.), Reviews of English language proficiency tests (pp.
81-83). Washington, DC: Teachers of English to Speakers of Other Languages.
Powers, D., Kim, H-J, Yu, F., Weng, V., & VanWinkle, W. (2009). The TOEIC Speaking and
Writing Tests: Relations to Test-Taker Perceptions of Proficiency in English. (ETS- RR09-18), Princeton, New Jersey.

Does TOEIC measure

Powers, D., Kim, H-J, Yu, F., & Weng, V. (2008). The redesigned TOEIC (Listening and
Reading) Test: relations to test-taker percptions of peroficiency in English, Princeton,
New Jersey.
Suzuki, M., & Daza, C. (2004). A review of the reading section of the TOEIC. TESL Canada
journal, 22 (1).
Takada, K. (2007). Evaluation on the listening comprehension section of Test of English for
International Communication (TOEIC), :

6(1) pp.157-168, Tohoky Bunka Gakuen University.

Wilson, K. M., & Stupak. S. S. (2001). Overestimation of LPI rating in the TOEIC Testing
context in Korea: Search for explanation (ETS RR-01-15). Princeton, NJ: ETS.
Zhang, S. (2006). Investigating the Relative Effects of Persons, Items, Sections, and Languages on
TOEIC Score Dependability. Language Testing; 23 (3) p. 351-369

12