See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/271670146

Is What You See What You Really Get? Comparison of Scoring Techniques in the
Assessment of Real-World Divergent Thinking

Article in Creativity Research Journal · May 2014

DOI: 10.1080/10400419.2014.901023

CITATIONS READS

79 324

3 authors, including:

Jonathan Plucker Meihua Qian

Johns Hopkins University Clemson University
276 PUBLICATIONS 11,133 CITATIONS 15 PUBLICATIONS 1,293 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Jonathan Plucker on 06 February 2015.

The user has requested enhancement of the downloaded file.

This article was downloaded by: [University of Connecticut], [Jonathan A. Plucker]
On: 08 May 2014, At: 12:29
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK

Creativity Research Journal

Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/hcrj20

Is What You See What You Really Get? Comparison of

Scoring Techniques in the Assessment of Real-World
Divergent Thinking
a b b
Jonathan A. Plucker , Meihua Qian & Stephanie L. Schmalensee
a
University of Connecticut
b
Indiana University
Published online: 08 May 2014.

To cite this article: Jonathan A. Plucker , Meihua Qian & Stephanie L. Schmalensee (2014) Is What You See What You Really
Get? Comparison of Scoring Techniques in the Assessment of Real-World Divergent Thinking, Creativity Research Journal, 26:2,
135-143

To link to this article: http://dx.doi.org/10.1080/10400419.2014.901023

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the
Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and
should be independently verified with primary sources of information. Taylor and Francis shall not be liable for
any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of
the Content.
This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
 CREATIVITY RESEARCH JOURNAL, 26(2), 135–143, 2014
Copyright # Taylor & Francis Group, LLC
ISSN: 1040-0419 print=1532-6934 online
DOI: 10.1080/10400419.2014.901023

Is What You See What You Really Get? Comparison

of Scoring Techniques in the Assessment of Real-World
Divergent Thinking
Jonathan A. Plucker
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014

University of Connecticut

Meihua Qian and Stephanie L. Schmalensee

Indiana University

In recent years, the social sciences have seen a resurgence in the study of divergent
thinking (DT) measures. However, many of these recent advances have focused on
abstract, decontextualized DT tasks (e.g., list as many things as you can think of that
have wheels). This study provides a new perspective by exploring the reliability and val-
idity evidence for several methods for scoring real-world DT tasks (i.e., tasks situated
within a real-world problem or situation). The results suggest a combination of
objective and subjective scoring methods may be optimal for originality scoring for
contextualized DT tasks, which stands in contrast to recent research suggesting the
objective, percentage scoring technique may be optimal for scoring originality on
abstract tasks.

Divergent thinking (DT) tasks remain a popular form of
creativity assessment, with some evidence that they are
increasing in use in both empirical and practical appli-
cations (Kaufman, Plucker, & Baer, 2008; Plucker &
Makel, 2010). This phenomenon is probably due to
several factors, including the growth in concern about
creativity in many countries and a lack of creativity
assessments other than DT measures that can be easily
administered to large groups (Plucker & Makel, 2010).
DT tests are open-ended tasks measuring ideational
fluency, the ability to create a list of responses that are
The research reported in this article was supported, in part, by a
grant to the first author from the Kempf Assessment Fund at the Indi-
ana University School of Education. We appreciate the input and
encouragement provided by James Kaufman and Paul Silvia through-
out this line of research. A version of this article was presented at the
2011 conference of the National Association for Gifted Children in
New Orleans, LA, and at a special symposium in 2011 at East China
Normal University in Shanghai, China.
Correspondence should be sent to Jonathan A. Plucker, Neag
School of Education, University of Connecticut, 2131 Hillside Road
U-3007, Storrs, CT 06269–3007. E-mail: jonathan.plucker@uconn.edu
Color versions of one or more of the figures in the article can be
found online at www.tandfonline.com/hcrj.
original (Runco, 1991, 1992). However, DT tasks allow
for numerous scoring options (Silvia, 2011). Responses
are usually scored for originality (statistical infrequency),
fluency (number of responses), and flexibility (number of
distinct categories of response), although there are a
range of strategies for deriving such scores. Because of
the increased attention with regards to DT tests, their
popularity and ease regarding large group administration
within educational settings, as well as the recognition of
the multitude of scoring possibilities, DT assessments
have undergone significant scrutiny. However, such scru-
tiny is based on the recognition that blanket approval or
disapproval of DT measures is misguided: They have
strengths and weaknesses, just like any other class of
assessments (Kaufman, Plucker, & Baer, 2008).
One of the weaknesses of DT tests regarding scoring
techniques is fluency contamination of the scoring of
originality. Fluency is conventionally calculated by
tallying the total number of given responses, and orig-
inality is derived by determining the statistical infre-
quency of the responses (Plucker, Qian, & Wang, 2011).
However, research has documented that highly fluent
respondents generally have a higher chance of
 136 PLUCKER, QIAN, SCHMALENSEE
earning a high originality score, thus leading to fluency
contamination on originality scores (Hocevar, 1979a,
1979b; Hocevar & Michael, 1979; Plucker et al., 2011;
Silvia, 2008; Silvia et al., 2008; Torrance, 1988). Silvia,
Martin, and Nusbaum (2009) noted that the scoring of
fluency and the number of unique responses have tra-
ditionally been highly correlated, begging the question:
Do they actually express discrete pieces of assessment
information?
The issue of fluency contamination is also important
due to controversies over the psychometric quality of
DT assessments. For example, several leading research-
ers have found that reliability evidence for originality
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014
scores is often poor if fluency effects are not controlled
(Clark & Mirels, 1970; Hocevar, 1979c), and that orig-
inality scores are associated with improved reliability
estimates when fluency is controlled (Runco & Albert,
1985). Due to the recognition of weaknesses within
scoring techniques in the research community, many
scholars have been analyzing various scoring models
for DT assessments. This line of research has significant
historical precedents. As Silvia et al. (2009) noted,
Guilford and his colleagues, as part of their seminal
work with DT assessment, experimented with various
scoring techniques (e.g., Wilson, Guilford, & Christen-
sen, 1953), and several other leading creativity scholars
have performed similar studies (Milgram & Milgram,
1978; Runco & Charles, 1993; Runco & Mraz, 1992).
This research is distinct from, but runs parallel to,
research on scoring techniques with creativity product
assessments, such as the work of Caroff and Besançon
(2008), Kaufman, Baer, Cole, and Sexton (2008), and
Plucker, Kaufman, Temple, and Qian (2009), among
many others.
These different DT scoring strategies have included,
but are not limited to, objective and subjective methods,
uniqueness scoring, snapshot scoring, percentage scor-
ing, having participants rate their own originality, and
controlling the number of responses from each partici-
pant, in addition to traditional originality, fluency, and
flexibility calculations (Clark & Mirels, 1970; Hocevar,
1979a; Hocevar & Michael, 1979; Plucker & Runco,
1998; Runco & Mraz, 1992; Runco & Smith, 1992;
Silvia, 2011; Silvia et al., 2008; Silvia et al., 2009). Many
of these methods are time-consuming for researchers
and educators, leading to a situation in which one’s
choice of scoring technique may impact the student’s
score and create a tangible opportunity cost in lost time
and resources.
In addition, the impact of the various techniques on
different types of validity evidence has not been exam-
ined. DT tests are often associated with convincing
evidence of reliability and validity, although the evi-
dence is admittedly inconsistent (Davis, 1989; Hocevar
& Bachelor, 1989; Kaufman et al., 2008; Plucker &
Makel, 2010; Plucker & Runco, 1998; Runco, 1991,
1999; Runco, Okuda, & Thurston, 1987). Predictive
and concurrent validity evidence is relatively strong
(e.g., Hong, Milgram, & Gorsky, 1995; Milgram &
Hong, 1993; Okuda, Runco, & Berger, 1991; Plucker,
1999a; Sawyers & Canestaro, 1989), and discriminant
validity evidence is growing (Kaufman et al., 2008;
Plucker & Makel, 2010), but researchers have largely
overlooked issues of convergent validity1 (i.e., How do
the various techniques for controlling for fluency impact
correlations between DT originality scores and external
criteria of creativity?) and the educational outcomes of
test use (i.e., How does use of a specific scoring tech-
nique impact who is admitted to a gifted program?).
Plucker et al. (2011) attempted to answer these ques-
tions, but they—like many others before them—used
abstract DT tasks, by far the most common form of
DT assessment. For example, in a recent study of
alternative scoring techniques for DT tasks, Silvia
(2011) utilized three common types of DT prompts:
‘‘two Unusual Uses tasks (unusual uses for a brick and
for a knife), two Instances tasks (instances of things that
are round and that make a noise), and two Consequences
tasks (consequences of not needing to sleep and of every-
one shrinking to 12 in. in height)’’ (p. 26). Although these
abstract DT tasks are important to understanding scoring
methods, they are lacking in terms of practicality and
meaningfulness to respondents. For example, Kaufman
et al. (2008) noted that other forms of validity evidence
are much stronger for applied DT tasks than abstract
items, suggesting that the lack of research on applied
DT is a major limitation on the assessment of creativity.
THE PRESENT STUDY
In this study, various techniques were used to score
student responses to the real-world DT assessments that
employ real-world problems in a replication and exten-
sion of the work of several researchers, including
Plucker et al. (2011) and Silvia (2011; Silvia et al.,
2008). Due to the implications of DT assessment use
(i.e., educational placements), this study serves as an
additional piece of information in practitioners’ efforts
to assess student originality while controlling for pot-
ential fluency contamination effects.
Using the abstract Wheels and Noise tests (Wallach
& Kogan, 1965), Plucker et al. (2011) compared seven
scoring methods, including objective and subjective
techniques, and found evidence that the traditional
scoring method (i.e., counting the number of responses
provided by less than 20% of the sample) as well as
the percentage scoring method (i.e., dividing originality
1
See Runco (1984, 1985) for exceptions.

scores by fluency scores) for originality are promising,
with the latter probably being the most appropriate
scoring strategy.
In addition, although Silvia et al. (2008) and Hocevar
(1981) found somewhat impressive results for subjective
scoring methods, Plucker et al. (2011) did not find that
subjective scoring methods performed well in controlling
fluency effects. This inconsistency may reflect that sub-
jective scoring techniques can be unstable due to the
characteristics of raters (Kaufman et al., 2008; Runco,
2008) or other factors such as the use of different cri-
terion measures and different DT tasks. However, will
the conclusions from previous research hold for real-
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014
world DT tasks? In this study, we apply techniques used
in previous studies to the scoring of two real-world
problems.
Specifically, the following issues are explored in this
study:
1. Because responses to DT tests are usually scored
for originality, fluency, and flexibility, but fluency
can be a contaminating influence on originality
scores, fluency will be treated as one unique
method for scoring DT tests, and the correspond-
ing evidence of reliability as well as validity will
be examined in this study.
2. In contrast with fluency, several methods for
scoring originality proposed in previous studies
will also be examined regarding evidence of
reliability and validity.
3. By correlating fluency and various originality
scores and taking into account the findings from
the first two goals of our study, the results should
suggest which scoring methods most effectively
control fluency effects on originality.
METHOD
Participants
A total of 148 adolescents (grades 8–10) attending a
summer, residential program for high ability students
volunteered to participate in this study (43% girls, 57%
boys). Seventh-grade students in the program’s region
of the country scoring at or above the 97th percentile
on their school-administered achievement test were
invited to participate in the talent search, which involved
an invitation to take the SAT or ACT out-of-level. Stu-
dents achieving specific SAT (or ACT) score criteria
become eligible for the summer residential programs.
Although racial information was not collected for stu-
dents in this sample, the sample was similar to that used
in previous studies, where the samples included roughly
75% Caucasian, 15% Asian American, 5% African
American, and 5% Hispanic students.
SCORING REAL-WORLD DT TASKS 137
Measures
Participants completed two real-world DT tasks, the Pat
and Kelly problems (Appendix A; Chand & Runco,
1992; Runco, Illies, & Eisenman, 2005; Runco & Okuda,
1988), and a battery of creativity assessments: the
Creative Personality Scale (CPS; Gough, 1979), the
Creative Behavior Inventory (CBI; Hocevar, 1981),
and one performance measure of creativity. To avoid
any presentation bias, instruments and activities were
presented to students in a random order.
Established protocols were used to score these
creativity assessments (Amabile, 1982, 1996; Hocevar,
1979c; Kaufman et al., 2008; Plucker & Runco, 1998;
Runco, 1987). Specifically, the CPS consisted of 18 posi-
tive and 12 negative items. A participant scored 1 by
endorsing a positive item and scored
1 by endorsing
a negative item. If an item was not checked, it was coded
as 0. In our sample, the internal consistency estimate of
the CPS was acceptable (Cronbach’s a ¼ .70). The CBI
(Hocevar, 1981) contained 90 items and required parti-
cipants to indicate their involvement in a range of
creative activities on a 4-point scale (i.e., never, 1–2
times, 3–4 times, or more than five times). Plucker
(1999b) found that a one-factor solution best represents
the structure of CBI data. Internal consistency estimates
were excellent (Cronbach’s a ¼ .96). The performance
measure required students to produce a story based on
the prompt, ‘‘Write an original story of what it would
be like to learn on the moon.’’ Students were given
5 min to write their stories. The participants were ver-
bally reminded to be creative when writing the story.
Two raters, both graduate students in educational
psychology, evaluated the creativity of each student’s
story. The correlation between the raters’ scores was .89.
Participants were given 5 min to complete each of the
two real-world DT tasks. The following six methods
were employed to score each student’s responses to the
DT tests in terms of originality. Fluency scores were also
calculated by counting the number of responses.
. Traditional scoring of originality a (i.e., count the
number of responses provided by less than 10% of
the sample; Method 1)
. Traditional scoring of originality b (i.e., count the
number of responses provided by less than 20% of
the sample; Method 2)
. Percentage scoring formula (i.e., originality a div-
ided by fluency using the entire ideational pool;
Method 3)
. Percentage scoring formula (i.e., originality b div-
ided by fluency using the entire ideational pool;
Method 4)
. Scoring of originality using external raters’ ratings
of the entire ideational pool (Method 5)
 138 PLUCKER, QIAN, SCHMALENSEE

. Scoring of originality c using external raters’ Validity Evidence

ratings of the answers provided by less than 20%
of the sample (Method 6)
. Percentage scoring formula (i.e., originality c
obtained through Method 6 divided by fluency
using the entire ideational pool) (Method 7)
For Methods 5–6, two raters evaluated the creativity
of the responses from each student. Each item was
evaluated separately on a 0–10 scale; evaluating them
collectively would have increased the likelihood of flu-
ency effects. The correlations between raters were .90
for Pat item evaluations and .91 for Kelly item evalua-
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014
Convergent and criterion-related validity. Because
the CPS assesses participants’ creative personality and
DT tasks estimate an aspect of their creative potential,
the correlation between them provides evidence of each
scoring method’s convergent validity. Correlations
between DT tasks and the CBI, which assesses parti-
cipants’ actual creative achievements, and the perfor-
mance measure of creativity which is used to measure
subjects’ creative product will provide evidence of DT
tasks’ predictive and concurrent validity, respectively.
Table 4 includes correlations between the various
originality scores and the criterion measures. The data

tions for Method 5.
RESULTS
Tables 1 and 2 include the descriptive statistics of these
DT task scores. For the Pat problem, all the scores
approximated a normal distribution. However, with
regard to the Kelly problem, the distributions of orig-
inality scores varied. Specifically, the objective scoring
methods (i.e., Methods 3, 4 and 7) produced similar dis-
tributions that were close to normal; the other objective
methods (i.e., Methods 1 and 2) and the subjective scor-
ing methods (i.e., Methods 5 and 6) produced another
cluster of similar distributions, which were all positively
skewed and peaked to various degrees.
Reliability Evidence
Intraclass correlations were used to provide evidence of
reliability for the DT test scores based on each of the
scoring methods (see Table 3). With the exception of
Methods 3 and 4, reliability estimates did not vary
dramatically. Estimates, although not overwhelmingly
positive, are in the expected range for assessments using
only two tasks.
suggest that almost every scoring method significantly
correlated with CPS (providing evidence of convergent
validity) and CBI (evidence of predictive validity), with
the evidence of validity most convincing for Methods
4, 6, and 7. However, in all cases, the magnitude of cor-
relations was not impressive. In addition, few of the
originality scores were significantly correlated with the
performance measures of creativity (concurrent valid-
ity), and those that were corresponding to one objective
scoring (i.e., Method 4) and the combination of subjec-
tive and objective scoring of originality (i.e., Method 7)
were relatively small in magnitude. Fluency scores signi-
ficantly correlated with some of the criterion measures,
but the magnitude of the correlations was also small.
Social consequences of test interpretation and
use. As noted by Messick (1989, 1995), various types
of validity evidence (e.g., content validity, predictive
and concurrent validity, convergent and discriminant
validity) should be considered when examining the
psychometric quality of data produced by a particular
measure. Also, test validation involves not only score
meaning but also value implications and action out-
comes, especially for applied decision making. The
social consequences of test interpretation and use consti-
tute an important aspect of validity (Kane, 2008), which
has received little attention within creativity research.

TABLE 1
Descriptive Statistics: Pat Problem

Scoring Methods Mean SD N Skew (SE) Kurtosis (SE)

Fluency 8.58 3.70 148 .56 (.20) .17 (.40)

1: Traditional scoring method @ 10%(Originality a) 2.09 1.91 148 1.21 (.20) 1.13 (.40)
2: Traditional scoring method @ 20% (Originality b) 5.50 3.52 148 .87 (.20) .65 (.40)
3: Originality a=fluency .22 .16 148 .50 (.20) –.19 (.40)
4: Originality b=fluency .60 .22 148 –.44 (.20) –.04 (.40)
5: Scoring of originality using external raters’ ratings of the entire ideational pool 25.48 11.73 148 .65 (.20) .59 (.40)
6: Scoring of originality using external raters’ ratings of the answers provided by 15.66 10.37 148 .92 (.20) .90 (.40)
less than 20% of the sample
7: Originality via Method 6=fluency 1.70 .65 148 –.35 (.20) –.13 (.40)
 SCORING REAL-WORLD DT TASKS 139

TABLE 2
Descriptive Statistics: Kelly Problem

Scoring Methods Mean SD N Skew (SE) Kurtosis (SE)

Fluency 5.22 2.32 143 .92 (.20) 1.98 (.40)

1: Traditional scoring method @ 10% (Originality a) 1.32 1.41 142 1.50 (.20) 3.14 (.40)
2: Traditional scoring method @ 20% (Originality b) 3.21 2.03 142 1.14 (.20) 2.17 (.40)
3: Originality a=fluency .24 .23 142 .76 (.20) .16 (.40)
4: Originality b=fluency .60 .24 143 –.47 (.20) .37 (.40)
5: Scoring of originality using external raters’ ratings of the entire ideational pool 14.76 7.28 142 1.53 (.20) 5.93 (.40)
6: Scoring of originality using external raters’ ratings of the answers provided by 9.61 6.00 136 2.25 (.20) 10.30 (.40)
less than 20% of the sample
7: Originality via Method 6= fluency 1.71 .81 143 .44 (.20) .83 (.40)
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014

they were used to rank students. Figure 1 depicts how

TABLE 3
Reliability Estimates of the Various Scoring DT Tests
Scoring Methods
Fluency
1: Traditional scoring method @ 10%(Originality a)
2: Traditional scoring method @ 20% (Originality b)
3: Originality a=fluency
4: Originality b=fluency
5: Scoring of originality using external raters’ ratings of the
entire ideational pool
6: Scoring of originality using external raters’ ratings of the
answers provided by less than 20% of the sample
7: Originality via Method 6=fluency
Note. ICC: intraclass correlation coefficient.
Given the frequency with which educators use DT
test scores to make important placement decisions, such
as identification for accelerated learning programs, we
used ranks to illustrate the influence of different scoring
methods on decision making in real educational settings.
In this study, as the objective scoring (i.e., Method 4) as
well as the combination of objective and subjective scor-
ing (Method 7) appear to be relatively more promising,
student ranks for originality differ according to the
two scoring methods. If only minor differences exist,
ICC the scatterplot would depict points clustered along a
.65
straight line moving from the bottom left to the top
.57 right, with few data points in the upper left and lower
.62 right quadrants, and the correlation among the ranks
.43 would be large and statistically significant. The data in
.37 Figure 1 suggest that, for the majority of students, the
.70
ranks are similar. However, the number of students
.67 who ranked high according to one method but low on
the other is striking; this phenomenon appears to be
.53 especially marked for students who scored high using
Method 7.
The scatterplot data can be interpreted in several
ways, but one interpretation appears to be most plaus-
ible. Method 4 controlled the impact of fluency through
dividing originality scores obtained via Method 2 by flu-
ency scores, but the data suggest that fluency is still a
contaminating factor in the calculation of some orig-
inality scores because some students who ranked very
high according to Method 7 (dividing originality scores
obtained via Method 6 by fluency scores) ended up
with very low ranks according to Method 4. Hence,

TABLE 4
Correlations Between Pat and Kelly Problem Scores and Criterion Measures

Creative Personality Creative Behavior Performance Measure

Scale Inventory of Creativity

Scoring Methods r tau r tau r tau

Fluency .186 .142 .324 .216 .071 .068

1: Traditional scoring method @ 10% (Originality a) .260 .219 .356 .268 .094 .127
2: Traditional scoring method @ 20% (Originality b) .294 .237 .426 .291 .125 .132
3: Originality a=fluency .294 .203 .330 .238 .164 .138
4: Originality b=fluency .386 .260 .407 .290 .197 .118
5: Scoring of originality using external raters’ ratings of the entire ideational pool .211 .141 .312 .211 .077 .080
6: Scoring of originality using external raters’ ratings of the answers provided by .328 .232 .425 .298 .058 .086
less than 20% of the sample
7: Originality via Method 6=fluency .400 .275 .360 .249 .193 .121

Note. r ¼Pearson product-moment correlations. tau ¼Kendall rank order correlations.


p < .05.  p < .01.
 140 PLUCKER, QIAN, SCHMALENSEE
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014
FIGURE 1 Scatterplot of ranks based on the Pat and Kelly problems.
alternative methods for scoring originality and control-
ling for fluency effects are critical in applied settings.
Relationship Between Fluency and Originality
To further explore the relationship between fluency
and originality scores, Table 5 displays the correlations
between fluency and various originality scores. Appar-
ently, Method 5 was influenced the most by fluency.
However, the magnitude of correlations between fluency
and other methods was also generally large. Considering
the nature of traditional scoring methods for originality
TABLE 5
Correlations Between Fluency and Seven Scoring Methods
Fluency
Scoring Methods for Originality r tau
1: Traditional scoring method @ 10% (Originality a) .71
2: Traditional scoring method @ 20% (Originality b) .92
3: Originality a=fluency .30
4: Originality b=fluency .32
5: Scoring of originality using external raters’ .95
ratings of the entire ideational pool
6: Scoring of originality c using external raters’ .88
ratings of the answers provided by less than 20%
of the sample
7: Originality via Method 6=fluency .33
Note. r ¼Pearson product-moment correlations. tau ¼Kendall rank
order correlations.

p < .05.  p < .01.
(i.e., originality was calculated by counting number of
responses provided by less than 20% of the sample),
these results are not surprising.
DISCUSSION
Reliability
In this study, two real-world problems were used as the
DT tasks, and four objective and one subjective method,
as well as two methods involving both subjective and
objective scoring, were employed to score participants’
responses for originality. Previous creativity researchers
(e.g., Hocevar, 1981; Silvia et al., 2008) concluded that
subjective scoring methods of DT tasks are more appro-
priate from a conceptual and psychometric perspective.
As one finding of our study indicates, Method 5 (scoring
of originality using external raters’ ratings of the entire
.54 ideational pool) did produce the most convincing evi-
.77 dence of reliability among the seven methods. At the
.24 same time, the results in Table 5 suggest that Method
.28 5 is the subjective measure most likely to be influenced
.82
by fluency effects. These findings were consistent with
.69 the results in Plucker et al. (2011). In other words, the
content of DT tasks does not seem to have a significant
impact on the effectiveness of various scoring methods
.26 in terms of reliability.
In addition, with regard to the traditional objective
scoring for originality, some researchers have suggested
assigning one point for responses given by fewer than

5% or 10% of the sample (e.g., Milgram & Milgram,
1976), although a more common practice is to count
number of responses provided by less than 20% of the
sample (e.g., Plucker et al., 2011). According to Table 3
in this study, there is no obvious difference in reliability
estimates between Method 1 (i.e., 10%) and Method 2
(i.e., 20%), or between Method 3 and Method 4. This
indicates that the degree of infrequency does not greatly
influence reliability evidence.
Validity
According to the results in Plucker et al. (2011), fluency
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014
was not associated with evidence of validity. However,
in this study, fluency scores were significantly correlated
with some of the criterion measures, but the magnitude
was relatively small. For originality, each scoring
method was associated with considerable evidence of
validity, although the levels of evidence varied. Com-
pared to the findings in Plucker et al. (2011), the tra-
ditional scoring method (Method 2) and percentage
scoring method (Method 4) still performed well in terms
of convergent validity and predictive validity, but the
combinations of subjective and objective scoring (i.e.,
Method 6 or 7) also looked promising. The magnitude
of all correlations was considerably larger than those
in the previous study, suggesting that the content of
DT tasks matters, given that this study used two
real-world prompts and Plucker et al. (2011) used two
abstract prompts (i.e., the Wheels and Noise tests).
In addition, according to Table 4, there are only
minor differences in validity indices between Method 1
(i.e., 10%) and Method 2 (i.e., 20%), or between Method
3 (10%=fluency) and Method 4 (20%=fluency). These
results provide evidence that different cut-off points do
not play a dominant role in the effectiveness of various
objective scoring methods in terms of convergent and
predictive validity evidence.
With respect to concurrent validity, none of the
methods performed impressively. This confirms the find-
ings in Plucker et al. (2011), but stands in contrast to the
work of Silvia et al. (2008), who found somewhat
impressive results for subjective scoring methods. This
inconsistency may reflect that completely subjective
scoring techniques can be unstable due to the character-
istics of raters (Kaufman et al., 2008; Runco, 2008) or
other factors such as the use of different criterion mea-
sures (e.g., the Big Five Personality Test in Kaufman
et al. versus the performance measure of creativity in
the present study) and different DT tests.
One striking finding of this study is the evidence of
social consequences of test use. Rank-order correlations
were used to illustrate the influence of various scoring
methods on educational practice. Figure 1 demonstrates
that the ranks based on Method 4 (originality via
SCORING REAL-WORLD DT TASKS 141
Method 2=fluency) and Method 7 (originality via Method
6=fluency) were far from identical. For example, a student
received a rank in the top 30 according to Method 4, but
the same student was placed over 100th when Method 7
was used. These findings suggest that even small differ-
ences in scoring techniques can have a disproportionate
impact on score rankings for some students.
Given that the purpose of this study was to explore
the impact of different methods for controlling fluency,
it seems reasonable to conclude that using alternative,
fluency-controlling methods for calculating DT origin-
ality does not have a significant impact on the majority
of participants; but for a small subsample, controlling
for fluency does lower originality scores substantially
relative to their peers.
Relationship Between Fluency and Originality
The correlations between fluency and creativity criterion
measures were low (Table 4), which implies that fluency
should not play the same role as originality in scoring
students’ responses to DT tests. However, fluency sig-
nificantly correlated with most of the scoring methods.
In other words, fluency is, indeed, a contaminating
factor in scoring originality, and the techniques used
in this study to control for this contaminating influence
were only partially successful.
Future Research
Improving the precision and psychometric quality of
measures, especially the most widely used measures, will
continue to be a critical contribution to building the
foundations of the field of creativity research. Although
this study suggests that a combination of subjective and
objective scoring techniques may be good candidates for
scoring DT tests for originality, more research is needed
to examine the generalizability of this conclusion and
its psychometric implications, especially in the areas of
concurrent validity and social consequences of test use.
For example, examining both originality and appropri-
ateness or utility of responses (e.g., Runco & Charles,
1993), especially when real-world DT tasks are used, is
appropriate and in line with recent conceptualizations
of creativity (Plucker, Beghetto, & Dow, 2004).
REFERENCES
Amabile, T. M. (1982). Social psychology of creativity: A consensual
assessment technique. Journal of Personality and Social Psychology,
43, 997–1013.
Amabile, T. M. (1996). Creativity in context: Update to ‘‘the social
psychology of creativity.’’ Boulder, CO: Westview Press.
Caroff, X., & Besançon, M. (2008). Variability of creativity judgments.
Learning and Individual Differences, 18, 367–371.
 142 PLUCKER, QIAN, SCHMALENSEE
Chand, I., & Runco, M. A. (1992). Problem finding skills as
components in the creative process. Personality and Individual
Differences, 14, 155–162.
Clark, P. M., & Mirels, H. L. (1970). Fluency as a pervasive element in the
measurement of creativity. Journal of Educational Measurement, 7, 83–86.
Davis, G. A. (1989). Testing for creative potential. Contemporary
Educational Psychology, 14, 257–274.
Gough, H. G. (1979). A creative personality scale for the Adjective Check
List. Journal of Personality and Social Psychology, 37, 1398–1405.
Hocevar, D. (1979a). A comparison of statistical infrequency and
subjective judgment as criteria in the measurement of originality.
Journal of Personality Assessment, 43, 297–299.
Hocevar, D. (1979b). Ideational fluency as a confounding factor in the
measurement of originality. Journal of Educational Psychology, 71,
191–196.
Hocevar, D. (1979c). The unidimensional nature of creative thinking in
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014
fifth grade children. Child Study Journal, 9, 273–278.
Hocevar, D. (1981). Measurement of creativity: Review and critique.
Journal of Personality Assessment, 45, 450–464.
Hocevar, D., & Bachelor, P. (1989). A taxonomy and critique of
measurements used in the study of creativity. In J. A. Glover,
R. R. Ronning & C. R. Reynolds (Eds.), Handbook of creativity
(pp. 53–75). New York, NY: Plenum Press.
Hocevar, D., & Michael, W. B. (1979). The effects of scoring formulas
on the discriminant validity of tests of divergent thinking. Educa-
tional and Psychological Measurement, 39, 917–921.
Hong, E., Milgram, R. M., & Gorsky, H. (1995). Original thinking as a
predictor of creative performance in young children. Roeper Review,
18, 147–149.
Kane, M. T. (2008). Terminology, emphasis, and utility in validation.
Educational Researcher, 37, 76–82.
Kaufman, J. C., Baer, J., Cole, J. C., & Sexton, J. D. (2008). A
comparison of expert and nonexpert raters using the consensual
assessment technique. Creativity Research Journal, 20, 171–178.
Kaufman, J. C., Plucker, J. A., & Baer, J. (2008). Essentials of creativ-
ity assessment. New York, NY: Wiley.
Messick, S. (1989). Meaning and values in test validation: The science
and ethics of assessment. Educational Researcher, 18, 5–11.
Messick, S. (1995). Validity of psychological assessment: Validation of
inferences from persons’ responses and performances as scientific
inquiry into score meaning. American Psychologist, 50, 741–749.
Milgram, R. M., & Hong, E. (1993). Creative thinking and creative per-
formance in adolescents as predictors of creative attainments in
adults: A follow-up study after 18 years. Roeper Review, 15, 135–139.
Milgram, R. M., & Milgram, N. A. (1976). Creative thinking and
creative performance in Israeli students. Journal of Educational
Psychology, 68, 255–259.
Milgram, R. M., & Milgram, N. A. (1978). Quality and quantity of
creative thinking in children and adolescents. Child Development,
49, 385–388.
Okuda, S. M., Runco, M. A., & Berger, D. E. (1991). Creativity and
the finding and solving or real-world problems. Journal of Psycho-
educational Assessment, 9, 45–53.
Plucker, J. A. (1999a). Is the proof in the pudding? Reanalyses of
Torrance’s (1958 to Present) longitudinal data. Creativity Research
Journal, 12, 103–114.
Plucker, J. A. (1999b). Reanalyses of student responses to creativity
checklists: Evidence of content generality. Journal of Creative
Behavior, 33, 126–137.
Plucker, J., Beghetto, R. A., & Dow, G. (2004). Why isn’t creativity more
important to educational psychologists? Potential, pitfalls, and future
directions in creativity research. Educational Psychologist, 39, 83–96.
Plucker, J., Kaufman, J. C., Temple, J. S., & Qian, M. (2009). Do
experts and novices evaluate movies the same way? Psychology &
Marketing, 26, 470–478.
Plucker, J., & Makel, M. (2010). Assessment of creativity. In R. J.
Sternberg & J. C. Kaufman (Eds.), The Cambridge handbook of
creativity (pp. 48–73). New York, NY: Cambridge.
Plucker, J., Qian, M., & Wang, S. (2011). Is originality in the eye of the
beholder? Comparison of scoring techniques in the assessment of
divergent thinking. Journal of Creative Behavior, 45, 1–22.
Plucker, J. A., & Runco, M. (1998). The death of creativity measurement
has been greatly exaggerated: Current issues, recent advances, and future
directions in creativity assessment. Roeper Review, 21, 36–39.
Runco, M. A. (1984). Teachers’ judgments of creativity and social
validation of divergent thinking tests. Perceptual and Motor Skills,
59, 711–717.
Runco, M. A. (1985). Reliability and convergent validity of ideational
flexibility as a function of academic achievement. Perceptual and
Motor Skills, 61, 1075–1081.
Runco, M. A. (1987). Interrater agreement on a socially valid measure
of students’ creativity. Psychological Reports, 61, 1009–1010.
Runco, M. A. (1991). Divergent thinking. Norwood, NJ: Ablex.
Runco, M. A. (1992). Children’s divergent thinking and creative
ideation. Developmental Review, 12, 233–264.
Runco, M. A. (1999). Divergent and creative thinking. Cresskill, NJ:
Hampton Press.
Runco, M. A. (2008). Commentary: Divergent thinking is not synony-
mous with creativity. Psychology of Aesthetics, Creativity, and the
Arts, 2, 93–96.
Runco, M. A., & Albert, R. S. (1985). The reliability and validity of
ideational originality in the divergent thinking of academically
gifted and nongifted children. Educational and Psychological
Measurement, 45, 483–501.
Runco, M. A., & Charles, R. E. (1993). Judgments of originality
and appropriateness as predictors of creativity. Personality and
Individual Differences, 15, 537–546.
Runco, M. A., Illies, J. J., & Eisenman, R. (2005). Creativity, orig-
inality, and appropriateness: What do explicit instructions tell us
about their relationships? Journal of Creative Behavior, 39, 137–148.
Runco, M. A., & Mraz, W. (1992). Scoring divergent thinking tests
using total ideational output and a creativity index. Educational
and Psychological Measurement, 52, 213–221.
Runco, M. A., & Okuda, S. M. (1988). Problem-discovery, divergent
thinking, and the creative process. Journal of Youth and Adoles-
cence, 17, 211–220.
Runco, M. A., Okuda, S. M., & Thurston, B. J. (1987). The psycho-
metric properties of four systems for scoring divergent thinking
tests. Journal of Psychoeducational Assessment, 5, 149–156.
Runco, M. A., & Smith, W. R. (1992). Interpersonal and intrapersonal
evaluations of creative ideas. Personality and Individual Differences,
13, 295–302.
Sawyers, J. K., & Canestaro, N. C. (1989). Creativity and achievement
in design coursework. Creativity Research Journal, 2, 126–133.
Silvia, P. J. (2008). Creativity and intelligence revisited: A latent
variable analysis of Wallach and Kogan (1965). Creativity Research
Journal, 20, 34–39.
Silvia, P. J. (2011). Subjective scoring of divergent thinking: Examining
the reliability of unusual uses, instances, and consequences tasks.
Thinking Skills and Creativity, 6, 24–30.
Silvia, P. J., Martin, C., & Nusbaum, E. C. (2009). A snapshot of crea-
tivity: Evaluating a quick and simple method for assessing divergent
thinking. Thinking Skills and Creativity, 4, 79–85.
Silvia, P. J., Winterstein, B. P., Willse, J. T., Barona, C. M., Cram,
J. T., Hess, K. I., . . . Richard, C. A. (2008). Assessing creativity with
divergent thinking tasks: Exploring the reliability and validity of new
subjective scoring methods. Psychology of Aesthetics, Creativity, and
the Arts, 2, 68–85.
Torrance, E. P. (1988). The nature of creativity as manifest in its test-
ing. In R. J. Sternberg (Ed.), The nature of creativity: Contemporary

psychological perspectives (pp. 43–75). New York, NY: Cambridge
University Press.
Wallach, M. A., & Kogan, N. (1965). Modes of thinking in young
children: A study of the creativity-intelligence distinction. New York,
NY: Holt, Rinehart and Winston.
Wilson, R. C., Guilford, J. P., & Christensen, P. R. (1953). The
measurement of individual differences in originality. Psychological
Bulletin, 50, 362–370.
APPENDIX A: REAL-WORLD DIVERGENT
THINKING TASKS
Pat problem: Your friend Pat sits next to you in class.
Pat really likes to talk to you and often bothers you
Downloaded by [University of Connecticut], [Jonathan A. Plucker] at 12:29 08 May 2014
View publication stats
SCORING REAL-WORLD DT TASKS 143
while you are doing your work. Sometimes Pat distracts
you and you miss an important part of the lecture, and
many times you don’t finish your work because Pat is
bothering you. What should you do? How would you
solve this problem? Remember to list as many original
ideas and solutions as you can.
Kelly problem: It’s a great day for sailing and
your friend, Kelly, calls and asks you if you want
to go sailing. Unfortunately, you have a big project
due tomorrow, and it requires a full day to
complete. You would rather be sailing. What are
you going to do? Think of as many original ideas as
you can.