Running head: EVALUATING CLOZE TESTING 1

Practical Research Application:

Evaluating Combined Single-Word Cloze and Phrase-Cloze Testing

Salam Ali

St. Cloud University

Author Note

Class

Professor

Date
EVALUATING CLOZE TESTING 2

Practical Research Application:

Evaluating Combined Single-Word and Phrase-Cloze Testing

Multiple skills must be acquired before an individual is proficient in learning a language.

As a language learner develops, tests are typically administered that allow the instructor to

monitor progress. One of the most widely tested skills is reading, since it is often considered

easier to test than other skills, such as writing or speech. Nonetheless, there are difficulties in

testing reading proficiency, especially in the context of learning English as a foreign or second

language (EFL/ESL). Teachers need the ability to test each student’s level of reading ability so

they can provide the necessary instruction. Learning to read well in a foreign language is

challenging and so the ESL instructor often utilizes reading comprehension tests comprised of a

series of related items based on a common passage of reading. It is difficult to imagine teaching

the required skill of reading English to ESL students apart from the ability to properly test the

students’ reading level, and this logically calls for effective tests that may be easily administered

to students of various levels of comprehension.

The importance of reading comprehension cannot be overstated, especially in regards to

ESL students. Williams, Ari and Santamaria (2011) underscored the “urgent need to improve the

ability to measure reading comprehension ability” (p. 216) so ESL students can attain success. It

is understood, however, that accurately measuring reading comprehension is not a simple matter.

Experts affirm that assessing reading comprehension is complex and involves various elements,

including decoding, fluency, prior knowledge, vocabulary and more. Research is extensive on

this topic but there is no consensus regarding the most effective methods of measuring reading

comprehension of ESL learners.

I. Background
EVALUATING CLOZE TESTING 3

Numerous methods are utilized among ESL teachers but one of the more popular is called

cloze testing. The word “cloze” was first mentioned in the context of language learning in the

early 1950s by Wilson Taylor who developed a unique form of testing for language

comprehension. Specifically, Taylor (1953) defined a cloze unit as “any single occurrence of a

successful attempt to reproduce accurately a part deleted from a message (any language product),

by deciding from the content that remains, what the missing part should be” (p. 416). In its

earliest manifestation, cloze testing was not designed for ESL students but was eventually

adapted for that purpose. Conceptually, this cloze process refers to the human tendency to

complete patterns that seem familiar but are incomplete. Cloze testing involves deleting certain

words from a passage of reading. In most cases, words are removed based on specific criterion,

such as deleting every fifth word, seventh word, or a similar pattern consistently throughout the

passage. The ESL student test taker fills in the blanks with the suitable words (or phrases). In

theory, these tests help measure a student’s reading comprehension by assessing whether he or

she properly understands the context of the passage. There are, however, a variety of cloze tests

used to assess ESL learners as well as much debate surrounding their usage as an effective

method to measure reading comprehension.

Initially, the cloze process was simply used as a tool for assessing the readability of text

passages, but was rapidly adopted as a useful method for testing reading ability among ESL

students. A cloze test is essentially based on the law of “closure”, an essential concept in Gestalt

psychology which asserts that individuals naturally complete patterns as long as they understand

that pattern’s significance (Atef-Vahid, Kashani, & Haddadi, 2011). This concept considers

phenomena holistically instead of simply examining or assessing each individual detail of

something. The application is compatible with language learning by ESL students.

EVALUATING CLOZE TESTING 4

Cloze tests are extremely useful in assessing ESL students’ proficiency since they are

based on an established standard of testing. Moreover, they are practical since

they can take a relatively short amount of time to complete (e.g., 15-35 minutes,

depending on the difficulty level of the test); their flexible format (choice of text, length

of text, word deletion ratio, scoring method, etc.) makes it possible to target a particular

range of proficiency levels; and they are easy to create and are easy to score if clear

scoring criteria are established. (Atef-Vahid et al., 2011, p. 153)

The use of cloze tests has therefore been widely researched and there is subsequently a great deal

of information attesting to the reliability and validity of cloze testing in the context of ESL

students.

Importantly for this proposed testing procedure, cloze tests are viewed as internally

consistent, a quality that is known to remain stable regardless of what type of cloze test format is

implemented (Atef-Vahid et al., 2011). Interestingly, while a cloze test is able to identify ESL

learners at varying levels of proficiency, it is also possible to adjust the level of difficulty for a

cloze test so that it is neither too complicated nor too undemanding for the test takers’ ability

level (Atef-Vahid et al., 2011). Cloze tests appear to be most effective in distinguishing between

the proficiency levels of lower to middle range ESL learners, but not as effective in attempting to

provide adequate testing for learners that are more widely separated in level of proficiency (such

as low and high skill levels). Therefore, cloze tests are most suitable for use in classrooms that

are comprised of ESL students grouped more closely to a norm rather than in settings where a

wider variation of proficiency exists.

In general, cloze tests fall into one of two categories—tests that require open-ended

answers and tests that utilize a multiple choice format. The addition of open-ended responses is
EVALUATING CLOZE TESTING 5

an attempt to improve the validity of cloze tests that were viewed as less reliable (Sadeghi,

2014). Some believe that the deletion of every fifth or seventh word from a passage is not a

proper method for determining comprehension. This line of reasoning asserts that specific

elements of speech should be deleted to provide a truer sense of the ESL students’ reading

comprehension. Accordingly, this research application is based on the assumption that there is

merit in both lines of reasoning and that a form of cloze testing that employs both a standard

every seventh word deletion sequence (single-word) cloze test and a multiple choice phrase-

cloze test (deleting specifically targeted phrases) mat be beneficial in assessing reading

comprehension.

II. Proposed Solution

The purpose of this assessment is to develop a hybrid version of the cloze test. The test

will combine two variations of cloze testing—single-word and phrase-cloze tests—to determine

if combining these two variations can improve reading comprehension in ESL students and the

assessment of reading by ESL teachers. The single-word portion of the cloze test will consist of a

standard every seventh word deletion sequence. The phrase-cloze test will consist of the deletion

of key phrases (2-4 words in length) that provide meaning to the passage. In both cases, the

student will have to choose from a list of words or phrases to complete the sentence.

III. Test Specifications

Description of test: The passages used for the test will be taken from common reading

material at the attributed reading level of the students. The text for the phrase-cloze test was

taken from the study conducted by Sadeghi (2014). The test will combine two variations of cloze

testing—single-word and phrase-cloze tests to determine if combining these two variations can
EVALUATING CLOZE TESTING 6

improve reading comprehension in ESL students. The first and last sentences of each passage

will be left intact to provide context.

Test audience: Adult students of ESL.

Test scheduling: The test will be administered following the completion of a unit on

reading comprehension. And will take place during regular class hours.

It is well known that accurately measuring reading

PROBLEM STATEMENT
comprehension among EFL learners is not a simple matter.
Assessing reading comprehension is complex and involves
many elements, including decoding, fluency, prior
knowledge, vocabulary and more. Research is extensive on
this topic and there is no consensus regarding the most
effective methods of measuring reading comprehension.

The purpose of this assessment is to develop a hybrid version

of the cloze test. The test will combine two variations of
cloze testing—single-word and phrase-cloze tests—to
determine if combining these two variations can improve
reading comprehension in ESL students.
During the data collection, the purpose of the tests will be
explained to students in each class orally as well as
appearing on the covering letter given to each student with
the test materials. Each student will receive an answer sheet
where they will give their name (optional), student number,
age, sex, languages spoken and whether they attend private
1. CONTENT
English classes.

The phrase cloze tests were administered first. The time

given for each of the cloze tests will be 30 minutes.
Instructions will also be given to the students to write all the
answers on the answer sheet.
The single-word cloze test required the proper placing of 34
words into sentences (based on an every-7th word deletion
method)

The phrase-cloze test required the selection of the correct

2. STRUCTURE
phrase from a choice of three for each of 37 phrases deleted
from the text.

Students were allotted 30 minutes for the single-word test

and 45 minutes for the phrase-cloze test.
3. TECHNIQUES Techniques used include filling in blanks with the proper
EVALUATING CLOZE TESTING 7

4. CRITICAL LEVELS OF
PERFORMANCE
word or phrase.
The students will be unfamiliar with the text provided in
both tests prior to taking the test. A response is marked as
correct if it matches exactly the deleted word or phrase. The
difficulty of a passage is determined by the mean of the
students’ percentage scores on the tests. These will be
calculated separately for both tests.
The highest score possible for both tests is 100% and a
passing grade is 75% on both tests.

5. SCORING A student’s scores from both tests will then be combined and
PROCEDURES the mean used to determine an overall score.

Correlation Coefficients between the phrase-cloze test and

single-word cloze test will also be determined.
One test may properly replace another and still be considered
a valid measure of that test if all the following conditions are
met:
a) If both tests are of the same nature (i.e., these two cloze
tests);
b) If both tests are intended for the same purpose (i.e., for
6. VALIDATION OF testing reading comprehension); and
TESTS c) If the correlation between the two tests is not only
significant but also very high and near +1.00.

The goal is to determine if combining these two forms of

cloze tests (in this case, two variables) will improve the
ability to assess reading comprehension in adult ESL
students.

IV. Sample Test items

Instructions will clearly describe what is required. The students are provided a list of

words to choose from for the single-word cloze portion and a multiple-choice list of phrases for

the phrase-cloze portion. The phrase lists will consist of three phrases consisting of dissimilar

grammatical structure so that only one phrase correctly corresponds to the sentence context.

Single-Word Cloze Test (Source—Essa Al-Dhaheri, 2008):

EVALUATING CLOZE TESTING 8

WORDLIST: What never actions I’ll He something a to as learned father late very

father’s Later I the my father’s same father’s at time life my made Denmark my to

watch Then from fun Yes picking silver of this at Uncle to It’s And silver

Time is very important in our lives. It organizes our everyday moments. However,

______ never had any importance in my ______ until I received a watch from _____ father that

organized my life and ______ me more responsible.

It came ______ Denmark to the UAE jewelry shop ______ a gray box. It weighs 8 ______. It's

round in the center with ______ silver bands that go around my ______. And all of it is made

______ silver. This object tells me the ______ of time in my life.

______ received this gift on a gray-sky ______. I had to go to the ______ at 9:00 AM to

pick up ______ Uncle Ali and take him to ______ father's house. However, I was late ______ I

was hanging out with my _______. Later on that day, around 11:00 ______, I remembered my

uncle, but I ______ very late for him. He had ______ the airport and taken a taxi ______ my

father's house.

I got ______ my father's house at 2:00 PM ______ the same day and looked at ______

angry father's face. I felt ashamed ______ myself at that moment. After I ______ hi to my angry

father and ______ uncle, my father asked me to ______ next to him where he handed ______

this watch which was a gift ______ him. Then he said, "Essa did ______ have fun with your

friends today?" ______ answered, "Yes father, and I'm sorry ______ not picking up my Uncle

Ali." ______ said, "What you did was not ______ nice and you should be sorry ______ your

actions." I was ashamed and ______, "Father I’ll never do it again. ______ promise." He said, "I

hope today ______ learned something important, and this watch ______ be a reminder for you."

He ______ me to take this watch and ______ it as an organizer of my ______.

EVALUATING CLOZE TESTING 9

I learned a very important lesson ______ my father: to respect time and ______ be late to

get someone. This watch is important to me, not because of its price, but because of the lesson

that I learned from it.

Phrase-Cloze Test (Source—Sadeghi, 2014):

The paper label must be one of the most abundant of art forms and, taking into account

the number of people it reaches, one of the most influential. Here is a brief look at the history

and functions of the labels on everyday products. Manufacturers have become increasingly

aware of …3… a part the label plays in the sales of …4…. Considerable attention is now given

to the visual impact …5…, particularly its overall colour, this being the first point of …6….

In the early days, brands were sometimes asked …7… only, to such an extent that the

company would incorporate the colour into …9…. Thus, there was ‘Green Label’ chutney and

‘Red Label’ tea. Manufacturers ought to remember that the label can become so opulent in

comparison with …10… that the customer’s expectations are dashed when the contents are

revealed. In 1888, when …11… in the US was moving towards ever more lavish labels, an

article in the New York Sun commented, ‘The label is often …12… the cigar…’

Since label design plays such an important role in a product’s continuity, it is surprising

that manufacturers today allow labels to be vandalised by a clutter of competition

announcements and …13…. One can only suppose that the public are not offended by these

recent design changes, which are now commonplace …14… – ‘the gallery of commercial art’ –

where some 2,000 designs are displayed. The variety is certainly incredible. There are those …

15… follow the latest fashion, those which are modern but not ‘over the top’, those which retain

much of …16… previous generations yet are still smart, and …17… try to look nostalgic.
EVALUATING CLOZE TESTING 10

Design apart, the label itself has remained unchanged in its primary function, and the

container to which it is attached – matchbox or tin can – …18… since the 1850s, which is

remarkable when so many extraordinary changes have happened elsewhere …19…. What were

the original functions of the label? …20…, there was the simple need to say what was …21….

The shape of the bottle or might already suggest this, but the word ‘Strawberry’, for example,

impressed on the …22… would identify the …23… inside it, or the word ‘Burgess’ …24….

Probably the most necessary labelling was for medicines, where it was vital that the contents and

directions for use …25… plainly.

The second aim was to glamorise the pack. …26… or pictorial image, particularly if

hand-coloured, would instantly enliven …27…. Some designs were relevant, depicting a scene

in which the product was used; some were purely …28…, using devices such as a beautiful girl.

…29… colour printing, decorative labels could be mass-produced in a variety of sizes. They

could be stuck onto, say, a box of gloves, which instantly transformed it into …30….

Manufacturers soon noticed that their products …31… if they had an element of prestige …32…

on the label. The presence of the royal coat of arms, a string of medals won at …33…, or a

testimonial from a respected analyst as to …34…, gave customers confidence in …35… what

they were buying.

By the 1950s, a further sales device was in general use – the direct incentive. Incentives

had, indeed, been …36… since the 1880s when, on Sunlight soap boxes, …37… was offered to

anyone who could find any impurity in the product. Other inducements were offered at that time,

including the pack that …38… after the contents had been consumed, and …39… that could be

saved and stuck into a scrapbook. By the end of the 1950s, the promotional pack bearing details

of a competition, free gift or price reduction was commonplace.

EVALUATING CLOZE TESTING 11

List of Phrases (for multiple choice):

3 took into account; takes into account; taking into account

4 one of most; one of the most; one of a most

5 how important; the important; an important

6 its products; their products; them products

7 visual contact; visually contact; visually contacted

8 for their colour; with their colour; by their colour

9 these brand name; the brand name; them brand name

10 them container’s contents; these container’s contents; the container’s contents

11 these cigar matket; that cigar market; the cigar market

12 best than; better than; gooder than

13 price reductions; price reduces; price reduced

14 on the supermarket; by the supermarket; in the supermarket

15 designs which; designs where; designs what

16 these tradition of; them tradition of; the tradition of

17 them which; that which; those which

18 have altered little; had altered little; has altered little

19 in daily life; with daily life; here daily life

20 By the first instance; With the first instance; In the first instance

21 inside them pack; inside the pack; inside those pack

22 side on a jar; side of a jar; side in a jar

23 type or jam; type of jam; type the jam

24 these manufacturer; them manufacturer; the manufacturer

EVALUATING CLOZE TESTING 12

25 should be saw; should be seen; should seen

26 Them design; Those design; A design

27 the overall effect; these overall effect; them overall effect

28 to gained attention; to gaining attention; to gain attention

29 Among the arrival of; With the arrival of; And the arrival of

30 an desirable present; a desirable present; the desirable present

31 sold gooder; sold better; sold good

32 attached to that; attached to these; attached to them

33 traded exhibitions; trade exhibitions; trader exhibitions

34 a product’s purity; an product’s purity; these product’s purity

35 the quality am; the quality are; the quality of

36 on existence; of existence; in existence

37 an handsome reward; a handsome reward; the handsome reward

38 have a function; has a function; had a function

39 these label; the label; them label

EVALUATING CLOZE TESTING 13

References

Atef-Vahid, S., Kashani, A. F., & Haddadi, M. (2011). The relationship between level of

ambiguity tolerance and cloze test performance of Iranian EFL learners. LiBRI. Linguistic

and Literary Broad Research and Innovation, 2(2), 149-169.

Sadeghi, K. (2014). Phrase cloze: A better measure of reading? The Reading Matrix, 14(1), 76-

94.

Taylor, W. L. (1953). Cloze procedure – A new tool for measuring readability. Journalism

Quarterly, 30, 415-433.

Williams, R. S., Ari, O., & Santamaria, C. N. (2011). Measuring college students’ reading

comprehension ability using cloze tests. Journal of Research in Reading 34(2), 215-231.
EVALUATING CLOZE TESTING 14

APPENDIX A: SINGLE-WORD CLOZE TEST FULL TEXT

The Hours That Count in My Life by Essa Al-Dhaheri

Time is very important in our lives. It organizes our everyday moments. However, time never
had any importance in my life until I received a watch from my father that organized my life and
made me more responsible.

It came from Denmark to the U.A.E. jewelry shop in a gray box. It weighs 8 ounces. It's round
in the center with two silver bands that go around my wrist. And all of it is made of silver. This
object tells me the importance of time in my life.

I received this gift on a gray-sky day. I had to go to the airport at 9:00 AM to pick up my Uncle
Ali and take him to my father's house. However, I was late because I was hanging out with my
friends. Later on that day, around 11:00 AM, I remembered my uncle, but I was very late for
him. He had left the airport and taken a taxi to my father's house.

I got to my father's house at 2:00 PM on the same day and looked at my angry father's face. I
felt ashamed of myself at that moment. After I said hi to my angry father and tired uncle, my
father asked me to sit next to him where he handed me this watch which was a gift from him.
Then he said, "Essa did you have fun with your friends today?" I answered, "Yes father, and I'm
sorry about not picking up my Uncle Ali." He said, "What you did was not very nice and you
should be sorry for your actions." I was ashamed and said, "Father I'll never do it again. I
promise." He said, "I hope today you learned something important, and this watch will be a
reminder for you." He told me to take this watch and use it as an organizer of my life.

I learned a very important lesson from my father: to respect time and never be late to get
someone. This watch is important to me, not because of its price, but because of the lesson that I
learned from it.
EVALUATING CLOZE TESTING 15

APPENDIX B: PHRASE-CLOZE TEST FULL TEXT

The paper label must be one of the most abundant of art forms and, taking into account the
number of people it reaches, one of the most influential. Here is a brief look at the history and
functions of the labels on everyday products. Manufacturers have become increasingly aware of
how important a part the label plays in the sales of their products. Considerable attention is
now given to the visual impact of the label, particularly its overall colour, this being the first
point of visual contact. In the early days, brands were sometimes asked for their colour only, to
such an extent that the company would incorporate the colour into the brand name. Thus, there
was ‘Green Label’ chutney and ‘Red Label’ tea. Manufacturers ought to remember that the label
can become so opulent in comparison with the container’s contents that the customer’s
expectations are dashed when the contents are revealed. In 1888, when the cigar market in the
US was moving towards ever more lavish labels, an article in the New York Sun commented,
‘The label is often better than the cigar…’ Since label design plays such an important role in a
product’s continuity, it is surprising that manufacturers today allow labels to be vandalised by a
clutter of competition announcements and price reductions. One can only suppose that the
public are not offended by these recent design changes, which are now commonplace in the
supermarket – ‘the gallery of commercial art’ – where some 2,000 designs are displayed. The
variety is certainly incredible. There are those designs which follow the latest fashion, those
which are modern but not ‘over the top’, those which retain much of the tradition of previous
generations yet are still smart, and those which try to look nostalgic. Design apart, the label
itself has remained unchanged in its primary function, and the container to which it is attached –
matchbox or tin can – has altered little since the 1850s, which is remarkable when so many
extraordinary changes have happened elsewhere in daily life. What were the original functions
of the label? In the first instance, there was the simple need to say what was inside the pack.
The shape of the bottle or jar might already suggest this, but the word ‘Strawberry’, for example,
impressed on the side of a jar would identify the type of jam inside it, or the word ‘Burgess’ the
manufacturer. Probably the most necessary labelling was for medicines, where it was vital that
the contents and directions for use should be seen plainly. The second aim was to glamorise the
pack. A design or pictorial image, particularly if hand-coloured, would instantly enliven the
overall effect. Some designs were relevant, depicting a scene in which the product was used;
some were purely to gain attention, using devices such as a beautiful girl. With the arrival of
colour printing, decorative labels could be mass-produced in a variety of sizes. They could be
stuck onto, say, a box of gloves, which instantly transformed it into a desirable present.
Manufacturers soon noticed that their products sold better if they had an element of prestige
attached to them on the label. The presence of the royal coat of arms, a string of medals won at
trade exhibitions, or a testimonial from a respected analyst as to a product’s purity, gave
customers confidence in the quality of what they were buying. By the 1950s, a further sales
device was in general use – the direct incentive. Incentives had, indeed, been in existence since
the 1880s when, on Sunlight soapboxes, a handsome reward was offered to anyone who could
find any impurity in the product. Other inducements were offered at that time, including the pack
that had a function after the contents had been consumed, and the label that could be saved and
stuck into a scrapbook. By the end of the 1950s, the promotional pack bearing details of a
competition, free gift or price reduction was commonplace.