A Guide for Instrulllent Developlllent

and Validat·on

(instrument development. occupational therapy, test construction)

Jeri Benson Florence Clark

As occupationaltherapzsts become planned, developed, and ualzdated, n 1976, I-bsselkus and Safrit (I)
increasingly concerned with A
is
sequential
illustrated
step-by-step
with a
process
flowchart and of
claimed
measuremeill
I
that lOO often principles
theory were not
accountabilzty, the paucity of ade­
quate instru,mentation available applied in the hypothetzcal con­ applied in the construction of the
for documentzng therapeutic effec­structzan of an allztude scale to
tiveness surfaces as a major prob­assess school administrators' valu­
lem, Therapzsts will need to con­ ing of the role of occu,patzonal
struct new or refine exzstzng therapists in the schools This Jeri Benson, Ph,D" is Assistant

example is provided to show how

instruments to satisfy the requzre­ Professor, School of Education,

ments of third-party payment, The general psychometric principles Department of Occupational

purpose of thzs paper is to illus­are applied wzthin an occupa­ Therapy, Unzverszty of Southern

trate how a new instrument is tzonaltherapy contex/' California, Los Angeles,

Callfornia,

Florence Clark, Ph,D" 0 TR,

FA 0 T A, is Assistant Professor,

Department of Occupatzonal

Therapy, University of Southern

California, Los Angeles,

California,

The Amencan Journal of Occupational Therapy 789

Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
 Figure 1
Flowchart for Instrument Development

Phase I
Phase II
Planning
Construction

Review Literature
on Construct or
Variable of Develop Table of Develop New
Interest Specifications or Revise
Items

(1) (2) (3) (4) (5) (6)

Phase III Phase IV

Quantitative Evaluation Validation

Prepare Instru­ Revise Instrument Second Pilot Begin Validation

ment for First Prepare for Second Administration
Pilot Testing Pilot Testing

First Pilot
Administration

(7) (8) (9) (10) (11 ) (12) (13)

measurement tools occupational
therapists used to assess the existing
and potential functional capabili­
ties of clients. Assuming that this
si tuation was caused in pan by a
lack of familiarity with measure­
ment theory, these authors pro­
ceeded to summarize basic measure­
ment theory and selected procedures
for estimating reliability and valid­
ity within an occupational therapy
context. The purpose in publishing
their article was to enable therapists
to construct valid and reliable assess­
ments capable of yielding data of
scientific value.
Since then, a number of instru­
ments have been designed by occu­
pational therapists as clinical or re­
search tools possessing varying
degrees of adherence to the require­
men ts for adeq uate instru men ta tion
(2-10). Not only does the construc­
tion of these instruments signify
therapists' dissatisfaction with de­
pending exclusively on subjective
clinical judgment, but it also testi­
fies to their increased sophistication
regarding measurement concepts.
To avoid becoming overly optimis­
tic and so as not to exaggerate the
extent to which this trend has had
an impact upon the larger occupa­
tional therapy community, it is
i mponant to recognize that many of
these instruments were constructed
by graduate students in conjunc­
tion with requirements for the mas­
ter's degree. The students probably
had access to the consultation of
psychometricians or other faculty
with expertise in measurement.
Other therapists who do not have
this access may need to develop
instruments that are scientifically
sound, but may be precl uded from
doing so because they do not know
the steps necessary to proceed sys­

790 December 1982, Volume 36, Number 12

Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
 tematically and correctly with test
development and validation.
Numerous textbooks on measure­
ment and research design (11-13)
include chapters on instrument
reliability, validity, and item analy­
sis; summaries of this content have
been published in AjOT (14, 15).
However, no one source provides a
step-by-step guide to instrument
development and validation. The
purpose of this paper, therefore, is
to outline the step-by-step process
through which a new occupational
therapy instrument can be planned,
developed, and validated.
The Steps Toward In trument
Development and Validation
he flowchart shown in Figure I
illustrates the general step-by-step
process involved in planning, con­
structing, and validating a ne,·\!
instrument.
Although the illustration is gen­
eral in scope, specificity within a
chosen domain of interest to occu­
pational therapists is gained
through consultation with a psy­
chometrician or reference to the
psychometric literature.
Each of the four phases in Figure
1 contains several steps: l. planning
(steps 1-2), 2. constructIOn (steps 3­
6),3. quantitative evaluation (steps
7-11), and 4. validation (steps 12­
13). The phase and its correspond­
ing steps will be illustrated by an
example of how one would embark
upon the development and valida­
tion of a "hypothetical" attitude
scale designed for the purpose of
assessing school administrators'
attitudes toward the inclusion of
occu pational therapy in the schools.
The steps described will apply in
the construction of a wide range of
instruments including those of the
developmental, cognitive, motor,
affective, or functional assessment
variety.
Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
Plannmg Phase: Steps 1-2. This
most important phase begins with
the formulation of a statement of
the purpose of the intended instru­
ment(Step I). Thestatementshould
include a specification of the do­
main (content area) or construct
(abstract psychological trait) to be
measured and the target group for
which the instrument is intended.
In the hypothetical example the
statement of purpose might read:
"The instrument to be constructed
will assess the receptivity of school
administrators toward the inclu­
sion of occupational therapy in the
schools, where receptivity is the
construct to be measured and the
target group is school administra­
tors. "
Step 2 requires more time than
Step I since it involves a review of
the related literature. This step
ensures that an appropriate, reli­
able, and valid instrument does not
already exist. Critical reviews of
existing tests can be found in the
Mental Measurement Yearbooks
edited by Buros (16), and in selected
journals such as the journal oj
EducatIOnal Aleasurement, Applied
Psychological Measurement, and
iVleasurement and Evaluation Guid­
ance. Second, the review of the liter­
ature aids in formulating an opera­
tional definition for the construct to
be measured, an idea difficult to
grasp. When a construct is opera­
tionalized, the components neces­
sary to measure it are spelled out
(13). Classically, for example, the
operational definition of intelli­
gence (a construct) has been a score
on a standardized inteliigence test
such as the Stanford-Binet. In the
example of the school administra­
tors' "receptivity" scale, the con­
struct recepti vi ty is opera tionall y
defined as a score on the scale to be
constructed. An operational defini­
tion does not imply that the con­
The American journal oj Occupational Therapy
struct is being measured well. In
Step 2, therefore, the review of the
literature helps to identify the types
of items likely to assess the con­
struct as accurately and meaning­
flllly as possible. For the school
administrators' scale, the test devel­
oper might, for example, review the
recently published set of manuals
on occupational therapy in the
schools in order to identify the phi­
losophy of occupational therapy
educational management, the objec­
tives of occupational therapy man­
agement, and the role of the occupa­
tional therapist lD the schools (17).
Measurement of school administra­
tors' attitudes toward issues in each
of these domains migh t a ppropria te­
ly assess "receptivity" to the inclu­
sion of occupational therapy in the
schools.
The major product of Step 2 is the
formulation of open-ended ques­
tions that are then presented to
representatives of the target group.
Th is process usually el ucidates areas
of inquiry related to the construct
being measured that were not re­
vealed in the literature review and
sensitizes the test constructor to the
concerns of the terminology cus­
tomarily used by the target group.
The following are examples of
open-ended questions that might be
presented to the school adminis­
trator:
1. ,. As a school adminis tra tor I
believe an occupational therapist
could assist teachers by _
2. "The differences between the
special educator and occupational
therapist are _
Knowledge gam red from the re­
view of the literature and the re­
sponst's to the open-ended questions
are synthesized, concluding Step 2,
and the construction of the instru­
ment itself begins.
Construction Phase: Steps 3-6.
The construction of the instrument
791
 begins with listing the specific
objectives of the instrument that
pinpoint the purpose of the instru­
ment and indicate the content areas
to be assessed.
For example, for the school ad­
ministrator's receptivity scale, ob­
jectives might address the philos­
ophy of occupational therapy edu­
cational manag'ement, the objectives
of occupational therapy educational
management, and the role of the
occupational therapist in the
schools. (These three content areas
were identified in the literature
review undertaken in Step I as those
with which the instrument should
deal. )
Specifying the content areas to be
addressed by the objectives is neces­
sary but not sufficient for the for­
mulation of the objectives. In addi­
tion, a process dimension, which
hierarchica II y reflects va rious types
of human responses or reactions to
the content included on the ques­
tionnaire, typicall y is adopted for
formulating the objectives of the
instrument. In the field of educa­
tion, formally constructed process
taxonomies in the psychomotor
(18), cognitive (19), and affective
domains (20) exist and, while these
are used more commonly in the
preparation of educational objec­
tives, they can also be used to for­
mulate the objectives of an instru­
ment. In the example of the school
administrator's attitude survey, the
affective taxonomy developed by
Krathwohl, Bloom, and Bertram
(20) could be employed as a guide
for classifying the types of responses
school administrators might dem­
onstrate. In this taxonomy, affective
reactions are hierarchically arranged:
receiving, responding, valuing, or­
ganizing, and value complex. At the
most basic level, an individual
simply demonstrates awareness of a
phenomenon, whereas at the high­
792 December 1982, Volume 36, Number 12
Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
est level the individual not only
holds particular value toward the
phenomenon, but the individual
can also be characterized by a set of
values related to the phenomenon.
Objectives that reflect a feeling tone
or the degree of acceptance or rejec­
tion the administrators hold toward
occupational therapy in school can
be formulated by using the taxon­
omy for the scale. Objectives so con­
structed, each of which addresses
one of the specified content areas of
the instrument, might read:
Objective 1: The instrument will
assess a wareness of the role of the
occu pationa I thera pist in the schools.
Objective 2: The instrument will
assess the valuing of the objectives
of occupa tional therapy educational
ma na gemen t.
ObJectzve 3: The instrument will
assess internalization of the philos­
ophy of occupational therapy edu­
cational management.
When the objectives of the in­
strument have been constructed in
this manner, they readily lend them­
selves to the next step, preparing the
table of specifications (Step 4). The
purpose of the table of specifica­
tions is to delineate, as clearly as
possible, the scope and emphasis of
the test by relating items to objec­
tives. It ensures that the items are
derived from and appropriate to the
objectives of the instrument. Most
tables of specifications are two-way
grids with a number of cells. The
horizontal axis lists content areas;
the vertical axis lists the categories
in the process hierarchy used in the
construction of the objecti ves. Each
cell, therefore, represents the inter­
section of a content category and a
level of the process hierarchy. Be­
cause each objecti ve was constructed
with only one content area and one
level of the process hierarchy in
mind, it should "fit" in only one
cell of the table of specifications.
The relative emphasis the test will
give to each objective is then deter­
mined by the number of items the
test constructor apportions to each
cell.
An abbreviated "sample" table of
specifications is presented in Table
I that could be used for the devel­
opment of the proposed attitude
scale, The ojectives formulated in
Step 3 are placed in their corre­
sponding cells. Also, the number of
items that will be constructed for
each objective is indicated. The ex­
tent to which school administra­
tors have internalized the philoso­
phv of occupational therapy educa­
tional management is shown in the
table to be the most heavily weigh­
ted area on the scale.
With the objectives of the instru­
ment clearly stated, and the table of
specifications completed, the test
constructor begins to write items,
Step 4 on the flowchart (Figure I).
This step involves several facets.
First, the item format must be se­
lected. The choice of which format
best suits the intended respondent
in relation to age and ability is
important. Research in the cogni­
tivedomain, forexample, indicated
that item format may confound
what is actually being measured
(21-22). For example, multiple­
choice questions on social studies
content can, in part, be measuring
reading ability rather than knowl­
edge of social studies if the question
is written using a high-level vocab­
ulary. The choice of item formats
differs according to the type of scale
being designed. Multiple-choice,
true-false, or matching items are
typically used on instruments that
measure cognitive functions. Likert
(23), Thurstone (24), or Guttman
(25) scales are generally selected for
the development of affective instru­
ments. Psychomotor tests involve a
variety of item formats ra nging from
 Table 1
Table of Specifications for Attitude Scale
Content Domain
Philosophy
Item Level 01 OT Ed
(Process) Management
Awareness
(2 items)
Valuing
(3 items)
Internalization Objective 3
(5 items)
Tolal 10 items
simple imitative lasks to those in
which higher levels of skill must be
demonstrated. Greenstein (26) and
HarrO\v (18) provide discussions on
taxonomies for constructing psy­
chomotor objectives. These taxon­
omies may be useful for developing
iterm for psychomotor tests.
Since the Likert format is the
most commonly used in the del'el­
oprnent of attitude scales in the
social sciences, this format will be
chosen for the hypothetical attitude
survey. Likert scales typically h;:1\e
five options for responses rangillg
from strongly agree to strongly dis­
agree. The response, strongly agree,
might be assigned a value of 5. and
strongl}' disagree assigned a value
of I; thus the higher the score the
more positil'e the administrator's
attltude. The importance of the
num ber and nature of response op­
tions has been discussed by numer­
ous authors (27-30). The following
,lte examples of items correspond­
ing to objectives 1-3 that might be
included on the attitude scale for
admin is tra tors:
Circle the response that most closely matches your feeling regarding the following
statements There are no right or wrong answers Key: strongly agree = 5: agree = 4;
neutral =3: disagree = 2; strongly disagree = 1
1 Students who need occupational
therapy are attending public
schools . .
2. I endorse the integration of occupa­
tional therapy principles into class­
room curricula .
3. I regard the human as an active
being who is positively influenced
by purposefUl activity.
Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
Objectives Role 01
01 OT Ed OT in
Management Schools Total
Objective 1
(1 item) (1 item) 4 items
Objective 2
(4 items) (3 items) 10 items
(1 item) (1 item) 7 items
6 items 5 items 21 items
Often. the item l\"fiting is not per­
formed exclusilely by the tcst dc­
I·eloper. Experts in the field can be
"ssembled and trained for this pur­
pose (Step 4), especially when fund­
ing is al·aibble. Writing items can
be a long and tedious proces.; If it is
done soleh by one person, espe­
cially since it is strongly recom­
mended lh;il a pool of i telll s be delTl­
oped containing 111'1((' as Illdnl'
items as lI'i11 be needed for the final
form of the instrument. Thus. for
the attitude scale at least ,12 instead
of 21 items correctly apportioned to
the 9 cells in Table I will be needed.
for efficiency. each item should be
l\Titten on a separ:Clle note card to
enable rt'\'ising, shufflIng fur place­
ment on the test form, and keyll1g to
the table of speofications. Once the
pool of items has been written. care­
ful review by the writers is necessary
to determine insofar as possible
\I'!lcther: l. the ilems arc clearly
stated. 2. the items conform to a
selected format, 3. the rt'sponse op­
tions for each item are plausible,
,wd 4. the wording is familiar to the
.. 5 4 3 2 1
.. 54321
.. .. 54321
The Aml'1'l(an journal of OcrupallOnal Therapy
target group. At this point, review
of the item pool has just begun.
Steps to follow will include content
lalldaLion and further qualitative
elaluaLion in which the qualitv of
the items is assessed in relation to
the target group (Step 5).
In the content I'alidation session.
the 42 items typed as they would
appear on the final form o{ the test
dne! the table of specifications with­
out the items represented in thecells
all' gil('n lO colleagut'S or othn ex­
pcrts In the lield [or tllcil' IClit'II·.
This panel is then asked to match
Lhe items vI'ith the appropriate cell
on the table of specifications When
absolute agrefment is not re,lched
upon the placement of an item, that
item must be rel'ised until a consen­
sus on llS placement is reached.
Obl'iously, some items II·i11 nevtT
reach this standard despite seleral
reI isions 3nd therefore must be dis­
carded The final product should be
an I nSlru ment in which ,11 I items
1;111 be classified into thc ;lppIO­
priatt' It'll of the table of "pclIiII;I­
llOns. An instrument is considered
to be content valid when the ilems
adeq ua tel y reflect the process and
content dimensions of the specified
objectil'es of the instrument as de­
termined by expert opinion.
Following the content nllidation
proce"" ,1 sccond qu;llitati,c el ;Ii­
u,ttion sessIon is conl'ened, using
subjects considered to be rejJresen­
tatile of the target group. for ex­
ample, school administrators lI'ould
be assembled for the qualitative
assessment of the hypothetical atti­
tude scale. The participants would
be asked to critique the instrument,
but, unlike the panel of experts in
the con ten t I'alida t ion se5S ion, the
school administrators would be
asked to actually take the scale to
determine the time it takes for com­
pletion, and then judge the clarity
of the items and directlOns, and
793
 Child Number: _
PRESCHOOL
Session Number: _
Observer's Initials: _

0·1 year 1-2 years 2-3 years

Gross motor actIvity: reaches; plays

Gross motor activity: stands unsup­ Gross motor activity: beginning
~ with hands and feet; touches hands
ported; sits down; bends and recov­ integration of entire body in
Z to feet; crawls; sits with balance;
ers balance; walks and runs-wide activities-concentrates on complex
W
w:E pulls to stand; moves to continue
stance; climbs low objects; broad movements (i.e., throwing, jumping,
Ow pleasant sensation
movements involving large muscle climbing); pedals tricycle
«"
Q.c(
I1l
groups; rides kiddie car
z
c(
Territory: crib; playpen; house Territory: home; immediate surrounds Territory: outside; short excursions
:E Exploration: of self and objects within Exploration: of all unfamiliar things; Exploration: Increased exploration of
reach oblivious to hazards all unfamiliar objects; very curious
Comments: Comments: Comments:

ManipUlation: predominant-handles, Manipulation: predominant-throws; Manipulation: remains predominant­

mouths toys; brings two objects
together; picks up; hits; bangs;
~
Z shakes
w Construcllon: not evident
inserts; pushes; pulls; carries; feels; pats; dumps; squeezes; fills
pounds
Construction: little attempt to make Construction: manipUlation

:E product: relates two objects appro­ predominates; scribbles; strings

w
"«z priately (i.e., lid on pot); stacks;
takes apart; puts together
beads; puzzles 4-5 pieces

c( Interest: people; gazes at faces; Interest: movement of self-explores Interest: explores new movement

:E follows movements; attends to various kinesthetic and propriocep­ patterns (i.e., jumping); toys with

..J
~ voices and sounds tive sensations; moving objects (i.e., moving parts (i.e., dump trucks,

II: balls, trucks, pull toys) jointed dolls); makes messes

W
~ Purpose: sensation or function-uses Purpose: experiments in movement­ Purpose: process important-less
c( materials to see, touch, hear, smell, interest in finished product (ie.,
practices basic movement patterns
:E mouth (i.e., rattles, teething rings, (i.e., rock, walk, run); process scribbles, squeezes play dough);
colored objects) important repetition of gross motor skills
Attention: follows moving objects Attenllon: rapid shifts Attention: intense interest; quiet play
with eyes up to 15 minutes; plays with single
object or theme 5-10 minutes
Comments: Comments: Comments:

Imitation: of observed facial expres­
sions and physical movement (i.e.,
smiling, pat-a-cake); emotions (hugs
toys)
Imagination: not evident
Dramatization: not evident
Music: attends to sounds
Books: pats; strokes; picks at pictures
Comments:
ImItation: of simple actions; present
events and adults-self-related mim­
icry (i.e., feeds self with spoon)
Imagination: imaginary objects (i.e.,
pretend food on spoon)
Dramatization: not evident
MusIc: sways; listens
Books: handles; points to pictures;
begins to name pictures
Comments:
Imitation: of adult routines with toy­
related mimicry (i.e., child feeding
doll); toys as agents (i.e., doll feeds
self)
Imagination: personifies dolls, stuffed
animals; starts having imaginary
friends (i.e., animals, persons)
Dramatization: portrays single
character
MusIc: responds to music with whole
body (i.e., marching, twirling)
Books: likes familiar stories; fills in
words and phrases
Comments:

Type: solitary play (no effort to
interact with other children or
choose similar activities)
z Cooperation: demands personal
o Cooperation: possessive (much
j: attention; simple give and take inter­
«
c- action with immediate family or
O caretaker (i.e., tickling, peek-a-boo);
j: 7-10 months-initiates games rather
II: than follows
c(
0.. Language: attends to sounds and
voices; babbles: uses razzing sounds
Comments:
Type: combination of solitary,
onlooker play (watches others­
speaking but not entering their play)
Cooperation: more complex games
with a variety of adults (i.e., hide and
seek, chasing); offers toys but
somewhat possessive; persistent
Language: jabbers during play-talks
to self, often in sing-song rhythm;
uses gestures and words to com­
municate wants; labels objects
Comments:
Type: parallel play (plays beside
others, play remains independent,
but child situates self among
others, enjoys their presence)
snatch and grab, hoarding, no shar­
ing, resists toys being taken away);
independent (does not ask for help,
initiates own play)
Language: talkative-very little
jabber; begins to use words to com­
municate ideas, information
Comments:

'Revised from Knox, Susan. A Play Scale. In Playas Exploratory Learning, M. Reilly, Editor. Beverly Hills: Sage Publications, Inc.,
1974. - Jayne Shepherd, Nancy Bledsoe, 1981.

794 December 1982, Volume 36, Number 12

Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms

 PLAY SCALE' The following to be filled in after observations are Space Management: Imitation:
recorded: Mean ages for dimensions:
Material Management: Participation:
Play Age (mean of all dimensions):

3-4 years 4-5 years 5-6 years

Gross motor activity: more coordi­ Gross motor activity: increased Gross motor activity: more sedate;
nated body movements, smoother activity level: can concentrate on good muscle control and balance; ....Z
walking, jumping, climbing, running goal instead of movement; ease of hops on one foot; skips; somer­ w
(accelerates, decelerates) gross motor ability allows stunts, saults; skates: lifts self off ground w::E
tests of strength, exaggerated Ow
movements; clambers ~"
ll.~
IIl
Territory: home; immed. Territory: likes to be up off ground ze(
Territory: neighborhood
neighborhood ::E
Explorallon: interest in new experi­ Exploration: anticipates trips, likes Exploration: plans and enjoys
ences, places, animals, nature change of pace excursions and trips
Comments: Comments: Comments:

Manlpulallon: small muscle activity­
hammers, sorts, inserts small objects
(i.e., peg boards); cuts
Construction: makes simple products
(i.e., blocks, crayons, clay); com­
bines play materials; takes apart;
arranges in spatial dimension­
design is evident
Interest: anything new: fine motor
manipulation of play materials
Manipulation: Increasing fine motor
control allows quick movements,
force, pulling
Construcllon: predominates-makes
products, specific designs evident,
builds complex structures; puzzles
10 pieces
Interest: takes pride in work (i.e.,
shows and talks about products,
Manipulation: uses tools to make
things (i.e., cuts more precisely);
copies; traces; combines various
types of material
Construction: predominates-makes
recognizable products; likes small I­
construction, attends to detail (i.e., Z
w
eyes, nose, fingers apparent in ::E
drawings); uses products in play
w
Interest: in reality-manipulation of
real life situations (i.e., miniature
"

c(
Z
c(
compares with friends, likes pictures things); making something useful­ ::E
displayed); complex ideas props for play; permanence of ...I
c(
products; toys that "really work"
iX
w
Purpose: beginning to show interest Purpose: product very important­ Purpose: replicate reality
l­
in result or finished product use to express self; exaggerates e(
Atlentlon: longer span-around 30 Atlentlon: amuses self up to one Atlentlon: concentration for long ::E
minutes; plays with single object or hour; plays with single object or period of time: plays with single
theme 5-10 minutes theme 10-15 minutes object ortheme 10-15 minutes
Comments: Comments: Comments:

Imitation: more complex imitation of
real world-part of dramatization
Imagination: assumes familiar roles­
domestic themes, past experiences
Dramatlzallon: imitates simple action
and reaction episodes-mi rrors
experience, emphasis on domestic
and animals; portrays multiple char­
acters with feelings (mostly anger
and crying); little interest in
costumes
Music: sings simple songs-not
necessarily on pitch; plays
instruments
Books: new or information books;
pictures important; relates own
experiences to story
Comments:
Imitation: more complex imitation of
real world as part of dramatization
Imagination: prominent-able to use
familiar knowledge to construct a
novel situation (i.e., expanding on
the theme of a story or TV show)
Dramatization: role playing for or with
others; portrays more complex emo­
tions; sequences stories-themes
from domestic to magic; enjoys
dress-ups
Music: sings whole songs on pitch;
musical games (ie., Farmer in the
Dell); good rhyth m
Books: listens better-doesn't need
physical contact with book; looks at
books independently-repeats
familiar stories
Comments:
ImltalJon: more complex imitation of
real world as part of dramatization
Imaglnallon: prominent-continues to
construct new themes but emphasis
on reality-reconstruction of real
world
Dramalizallon: sequences stories­
emphasis on copying what occurs in
real world; costumes important; z
props; puppets o
~
<
!::
~
Music: meaning of songs important;
enjoys catchy tunes, songs that tell
stories; dances reflect interpretation
of music
Books: looks at books independently
or with peer; describes picture to tell
story; must be credible
Comments:

Type: associative play (similar
activities with groups of 2-3, no
organization to reach a common
goal, more interest in peers than
activity)
Cooperalion: limited-some turn
taking: asks for things rather than
grabbing; little attempt to control
others
Language: uses words to communi­
cate with peers, interest in new
words (repeats them, asks their
meaning)
Comments:
Type: cooperative (groups of 2-3 Type: cooperative (groups of 2-5,
organized to achieve a goal, i.e., organization of more complex
assigns roles for pretend play) games and dramatic play)
z
o
Cooperation: takes turns: attempts to Cooperation: social give and take ~
c(
control the activities of others (often evident (i.e., compromises to facili­ ll.
self-centered, bossy) tate group play); rivalry seen in U
competitive games ~
Language: very talkative-plays with Language: very prominent in socio­ a:
~
words; fabricates-capable of long dramatic play (uses words as part of ll.
narratives; questions persistently; playas well as to organize play);
communicates with peers to interest in present; relevant how,
organize activities what for questions
Comments: Comments:

The Amencan Journal of Occupational Therapy 795

Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
 make any other recommendation'>
rele\'ant to the overall quality of the Table 2

instrument. This Jspect of instru­ Methods of Estimating Reliability

ment development is crucial if the

response format is new to the target Type of
Types of
Accepted
Reliability
Instruments
Procedures Values
group. In Step 6, the items are
revised and. where indicated, new Stability (test-retest) Instruments used to Give the same test .60 or greater
items are written. It is often easier to predict or select­ to the same group
aptitude tests, at two different
improwa bad ite.m than to develop psychomotor tests, times, correlate the
a new one (12), therefore, do not personality invento­ two scores using
discaru any item too quickly. ries, neuro­ the PPM.
psychological tests.
Quantitative Evaluation Phase:
Steps 7-Il. Quantitative evaluation Equivalence Any tests that have Give Form 1 .80 or greater
of the instrument is substantially (parallel form) alternate forms immediately fol­
accomplished in the first pilot test­ lowed by Form 2;
correlate the two
ing. Prior to the pilot test, the 112 scores, use the PPM
re\ised items must be typed to pro­
duce the test form. Then, in the Internal Instruments used to For tests with .80 or greater
pilot test (Step 7). the scale is Jd­ Consistency infer an underlying dichotomously
construct-aptitude scored items use
lnlnistered under optimal condi­ tests, attitude KR20, for all other
tions to a group that is represellla­ scales, psychomo­ tests use Coefficient
tl\e of the target group. Ideally, the tor tests, personality Alpha
inventories,
group should be large so that the neu ropsycholog ical
responses of a fe\·,· subjects do not tests.
distort the results of the quantita­
tiw evaluation. It is generally L1se­ Note:
PPM = Pearson Product-Moment Correlation Coefficient.

ful to follow the pilot test with KR20 and Coefficient Alpha formulas can be found in Mehrens and Lehmann.

another qualitative e\'aluation de­

briefing session in which the sub­
jeers who participated ill the pilot
study are asked to provide feedback
regarding the clarity of each item,
are informed of the purpose of the
instrument, and are asked to offer
an\' additional comments. As sub­
jects in the pilot study, this group
has responded to twice as many
llems as will comprise the final
instrument.
The pilot study provides the dara
from which the initial reliability
estimates are calculated (Item An­
alysis. Step 8). Reliability is defined
as the consistency of the measure­
ment over time or the precision of
measurement. It reflects the confi­
dence gi ven to the observed score as
being a reflection of what a person
really knows, believes, or is able to
do. The reliability codficiclll can
range from a low of zero (negative
values are considered zero) to a high
of J .00, The difference between the
observed reliobility coefficient and
1.00 is attributed to error. Thus, if
the observed reliability coefficient
was .75, then .25 represents the
degree of inconsistency in the mea­
surement. The above coefficients
would be interpreted as follows: 75
percent of the variance in the test
'A'as measuring the subject's actual
ability, achie\'ement, attitude, or
personality, and 25 percent was clue
to chance or random error.
Se\ eral methods are '1\'ailable for
e,tim<1ting the reliability of an in­
strument. The choice of method is
dependent upon the intended pur­
pose of the instrumfnl. Table 2 dis­
ploys the three methods of estimat­
ing reliability, the types of instru­
ments for which each reliability
coefficient is required, the proce­
dure to follow in calculating the
reliability coefficient, and the gen­
erally accepted value recommended
by these writers for the obscn'ed
coefficient. If decisions are to be
made about indi\'iduals, rather than
groups, these reliability coefficients
should be considered a minimum.
In the example of the attitude
scale fOJ school administrators, s('\­
eral types of reliability estimates
might be neecled. For example, an
in lerna I consistency esti ma te usi ng
coefficient alpha would be required.
This type of reliability is important
because it ensures that all the items
on the scale are measuring the same

796 DecembN 1982. Va/lime 36, Number /2

Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
 trait, in this case "responsivity" to
occupational therapy in the schools.
In addition, if one assumes that the
scale is to be used to predict future
behavior, a measure of stability is
required. Although the coefficient
alpha esti mate can be obtained from
the pilot study data, calculation of
the stability coefficient will necessi­
tate that the test in its final form be
given twice to the same group of
people at two different times and
that these scores be correlated.
each item together with reliability
ReliabilIty to a large extent re­
estimates for the total instrument.
flects the adequacy of the items
since its calculation is dependent
upon the scores achieved by thesub­
jects on the items in the pilot study.
Items can often be improved through
Item analySIS, which provides two
types of quantitative indices of the
adequacy of each item: the item dif­
ficulty index and the item discrimi­
nation index provide a measure of
the difficu It y of respective items and
are calculated as follows.
:'\umber of people correctly
answering the item
p =- - - - - - - - - - ­
Total number of people
responding to the item
Obviously, the only types of items
for which this statistic is relevant
are those for which a correct answer
can be keyed, typically cognitive or
psychomotor items. A p value re­
flecting moderate item difficulty
ranging from .30 to. 70 is considered
optimal for most instruments. ThiS
stallstic would not be calculated for
the hypothetical scale.
The item discriminatIOn index,
denoted by the letter D, reflects the
correlation of the item with the
total test score and provides infor­
mation about how well the item
discriminates among respondents
of different types. When items have
Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
only one correct answer, typical of
those inc! uded on psychomotor and
cognitive ll1struments, D is calcu­
lated by first dividing the respon­
dents into two halves, putting those
with the highest scores in one group
and those with the lowest scores in
the other. These groups serve as
comparative groups in the calcula­
tion of D for each item. The for­
mula for obtaining Dis:
Number of persons in the
UPPER group correctly
Number of persons in the
LOWER group correctly
answering the item
0=
Total number of persons ­ tained, revised, or discarded. The
in the upper group
answering the time
Total number of persons
In the lower group
For affecti ve instru ments, the dis­
crimination index is calculated by
correlating the score each person
received on each item with his or
her total score. Regardless of the
type of instrument, item analysis is
tedious and is best handled by com­
puter, for which there are readily
a\'ailable reliability and item analy­
sis programs (31). In their absence,
Item analysis can be obtained by
using the formulas already pre­
sen ted.
Interpretation of the statistics ob­
tained in the item analysis results in
retaining only the best items for the
final, reduced form of the instru­
ment. Some rules of thumb do exist
for making decisions in this area:
any item discrimination index that
is nega ti ve is considered to be zero.
To produce a test with high relia­
The American Journal of Occupational Therapy
bility, the researcher must select
items that have high discrimination
indices. Ebel (II) suggested retain­
ing items whose discrimination
va Iue is .30 or grea ter. If the item
shows a discrimination index of .20
to .29, it should be revised. If the
discrimination index is .19 or below
(including negative values), the
item should be eliminated.
In summary, then, the first pilot
study provides quantitative data on
When these data are interpreted
with the information gathered in
the debriefi ng and qua Iita ti ve as­
sessment sessions, the test construc­
tor can make sound judgments re­
garding which items should be re­
end product of Steps 5 through 8 is a
revised instrument with the number
of items reduced to that which was
originally planned and which con­
form to the table of specifications.
In rare instances, all or most of the
items may be considered adequate.
The test constructor then may ran­
domly choose the number of items
needed and reserve the remainder
for an alternative test form. The
new red uced form of the ins tru men t
must next be pilot tested (Step 9). As
in the first pilot testing, the items
are administered to a representative
target group comprising individu­
als other than those who were in the
first pilot testing, then analyzed,
and new reliability estimates are
obtained for the reduced form of the
instrument. The second pilot test­
ing (Step 10) usually results in a
final form of the instrument and
esta blishes that the reliability of the
instrument is acceptable. Step 11 is
necessary only if the revised instru­
ment needs improvement. After Step
10, or Step 11 when it is needed, test
validation can begin.
Validation Phase: Steps 12-13.
797
 well a client may perform in the
future, or 3. infer the degree to
Table 3 which a client possesses some
Methods of Estimating Validity amount of a directly nonobservable
Type of

hypothetical trait, a construct, for

Type of Use of the Procedure
Validity Instrument In8trument
example, sensory integration. Each
of these purposes requires a differ­
Content To determine how Provide an expert Achievement or
with a copy of the physical perfor­ ent method of validation. In Table
well an individual
performs at one objectives, table of mance tests 3, procedures forestablishing valid­
point in ti me for a specifications and ity corresponding to each of the
given content the instrument; the
expert judges
three purposes of testing are de­
domain
whether the content scri bed and the types of instrumen ts
domain has ade­ for which each is appropriate are
quately been
assessed.
listed. Logically then, Steps 12 and
13, the valida tion process, assu me
that the purpose of the test was well
defi ned a t the ou tset (Steps I and 2).
Criterion-Related To predict future Give the test and Tests used to
performance correlate the results select or classify,
Quite often a test will have multiple
with the criterion e.g., aptitude or purposes, in which case several types
variable. The criteri­ developmental of validation may be needed.
on may be obtained scales
concurrently or at
Content validity, defined by Cron­
some time in the bach (32), pertains to whether the
future. set of items adequately covers the
content domain of interest as well as
the set of behaviors implied by the
Construct To infer some Based upon a the­ Any test that pur­ test score. The "behaviors implied
amount of a hypo­ ory underlying the ports to measure a
by the test score" relate to the con­
thetical trait trait, hypotheses are hypothetical trait,
set up and tested such as personal­ text in which the items are pre­
regarding the ity or altitude sented (22). For example, in the
behavior of persons scales, aptitude
who possess large tests or develop­
illustration of the attitude scale for
or small amounts of mental scales school administTators, vocabulary
the trait. level of tes t items shou Id not be so
difficult or technical that the scores
reflect in part lack of familiarity
with occupational therapy termi­
nology. Attitude is what the in­
strument is intended to measure.
For this instrument to have content
validity, measurement of attitude
should not be confounded with vo­
The validation of a newly devel­ ing. Since there are several methods cabulary level or some other uni­
oped instrument is almost never [or establishing the validity of a dentified variable.
accomplished through one study or given instrument (14), the selection Procedures for the preliminary
by one researcher. Instead, it re­ of a method is dependent upon the establishment of content validity
quires numerous research efforts intended use of the instrument. For were described in Step 5. When that
and, for this reason, must beconsid­ example, a therapist may want to: I. step is properly undertaken, the
ered an ongoing process. Despite its determine how well a client per­ content validation of the final form
time-consuming demands, valida­ forms in a designated content do­ is easil y accompl ished. Essentiall y,
tion is essential because validity main at a particular point in time, Step 5 is repeated but now with only
allows one to be confident of what for example, performance in activi­ the items on the final test form (Step
the instrument is actually measur- ties in daily living, 2. predict how 12). The table of specifications is

798 December 1982, Volume 36, Number 12

Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms

 presented to experts who are asked
to classify the final items into the
cells. The percentage of items as­
signed a ppropria tel y reflects the ex­
tent of the test's content validity.
Criterion-related validity is need­
ed when the new test is to be used to
classify individuals or predict their
future performance. In brief, the
esta blish men t of cri terion-related
validity involves correlating scores
obtained on the new test with those
obtained on an older, already vali­
dated test. Although this procedure
appears to be rather simple, in real­
ity it has inherent difficulties. The
major problem is locating an ade­
quate criterion, which must be either
a well-established test or highly
respected clinical ratings. For ex­
ample, the Slosson Intelligence Test.
a group-administered paper-and­
pencil intelligence test that is a
quick and efficient measure of intel­
ligence, was validated by compar­
ing scores on it with those obtained
by the same individuals on the Stan­
ford-Binet. In this instance, the
Stanford-Binet served as the crite­
rion. The acceptability of the Slos­
son Intelligence Test as a measure
of intelligence is dependent upon
the adequacy and acceptability of
the Stanford-Binet as a criterion;
that the criterion must have ade­
quate reliability and validity in its
own right is a given.
The two types of criterion-related
validity are concurrent and predic­
tive. The former is obtained by con­
currently testing a group of subjects
on the new test and the criterion.
The latter is established by first test­
ing a group of subjects with the new
test and then later with the crite­
rion, The selection of concurrent or
predictive validity is based upon the
intended purpose of the new in­
strument.
The third type of validity, con­
struct validity, is the most difficult
Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms
and perhaps the most important
form to obtain. It is needed when
test scores are used to infer the pres­
ence of some underlying hypotheti­
cal trait or construct. The term con­
struct validity was first discussed in
the now classic article by Cronbach
and Meehl (33). The general proce­
dure typically used to assess con­
struct validity of a new scale is to
define the trait operationally and
set up a hypothesis or hypotheses
about how individuals who possess
varying degrees of the trait will
behave. In the example used
throughout the paper, it might be
hypothesized that school adminis­
trators who agree with the idea of
employing occupational therapists
in public school systems will have
high scores on the attitude scale
and, conversely, individuals not re­
ceptive to this innovation will have
low scores on the attitude scale.
Once this hypothesis is formulated.
It needs to be tested using any of
three gener;] I methods: I. the known
groups procedure, 2. factor analy­
sis, and 3. the multitrait-multi­
method procedure.
The known groups procedure re­
quires that a known group be ob­
tdined that already possesses the
type of attitudes just described; for
example, those who agree and those
who disagree that occupational
therapists should be delivering ser­
vices in the schools. The intent is to
verify that the scale does measure
what it was intended to measure
and the known group acts as a cri­
terion for verification. If the scale is
valid, those "known" to be recep­
tive should achieve high scores;
those nonreceptive should obtain
low ones.
Thesecond procedure, factor anal­
ysis, requires that the researcher
hypothesize the nature and the num­
ber of factors underlying a scale. In
this instance, the term factor refers
The Amerzcan journal of Occupational Therapy
to a theoretic variable derived from
analyzing the intercorrelations of
test items. For example, on the
school administrators' scale, it
would be reasonable to hypothesize
that, in factor analysis. separate fac­
tors relating to each of the content
areas or process levels would emerge.
If this hypothesis is confirmed, the
test constructor can be reasonably
confident that the scale is actually
measuring the intended trait, "re­
ceptivity." Unfortunately, one fac­
tor analytic study never suffices­
generally a series is conducted. Fac­
tor analysis requires a very large
sample size and a general rule is to
use ten people for every item (13).
Several factor analytic solutions are
often necessary tOvalidate the under­
lying structure of the new scale
where each subsequent solution
must be obtained on a new sample.
Th us. factor analytic validation pre­
sents a formidable challenge and
probably cannot be undertaken in
occupational therapy unless sam­
ples of sufficient size are available.
For test developers who do not
have accessibility to large samples,
it IS recommended that they use the
multitrait-multimethod procedure
of establishing construct validity
(34). Campbell and Fiske (34) ex­
plain the procedure in detail. Essen­
tially, the procedure allows for de­
termination of the extent to which
tests purported to measure the same
trait with dIfferent methods or for­
mats are correlated. These correla­
tions are compared with those ob­
tai ned bet ween tes ts thought to
measure different traits with similar
formats. If the former set of correla­
tions is not higher than the latter,
this is taken as evidence that the
tests are not valid for measuring the
construct. Campbell and Fiske re­
port specific criteria for making the
judgment. The advantage of this
procedure over factor analysis is
799
 that it requires fewer subjects; the
disadvantage is that available sub­
jects are required to respond to two
or more scales under two or more
formats, which can be time con­
suming. Moreover, it is sometimes
difficult to locate sets of instruments
having the appropriate trait and
format characteristics.
Validation of a new scale requ ires
much time and funding. Several
studies across different populations
and across different times are usu­
ally needed to ensure that the new
instrument is valid for all individu-
als with whom it is to be used. Vali­
dation is a continual process (Step
13), one in which an end point is
rarel y ach ieved, bu t is only succes­
sivelyapproximated.
Conclusion
Test planning, construction, and
validation require time and pa­
tience. The compressed guide to
instrument development and vali­
dation presented in this paper pro­
vides an overview of the steps re­
quired and is useful as a method­
ological "road map" for the pro­
specti ve test constructor. I tis hel pfu I
for therapists who plan to embark
upon instrument development to
consult the psychometric literature
and seek out the expenise of a psy­
chometrician, Instruments devel­
oped within the occupational ther­
apy field according to the guidelines
presen ted here are not on Iy likel y to
satisfy the requirement of third­
pany payment but will also be ap­
propriate for research purposes. In­
struments of caliber are needed in
all areas of occupational therapy
practice.

tion on a sample of adults with head jective test performance: A question of

REFERENCES trauma. Master's Thesis, University of validity, Educ Psychol Measure 39:

1,Hasselkus BR, Safrit MJ: Measurement

Southern California, Los Angeles, Cali­ 381-387, 1979

in occupational therapy. Am J Occup

fornia, June 1981 22.Benson J: A redefinition of content

Ther 30: 429-436, 1976

11, Ebel R: Essentials of Educational Mea­
validity. Educ Psychol Measure 41'

2.Cermak SA, Coster W, Drake C: Rep­

surement (3rd Edition), Englewood
793-802, 1981

resentational and nonrepresentational

Cliffs, NJ Prentice Hall, 1979
23,Likert R: A technique forthe measure­

gestures in boys with learning disabili­

12,Mehrens W, Lehmann I: Measurement
ment of attitudes. Arch Psychol 140:

ties. Am J Occup Ther 34: 19-26, 1980

and Evaluation in Education and Psy­
52, 1932

3, Lindquist JC: A study of physical inac­ chology (2nd Edition), New York: Holt
24. Thurstone L: Attitudes can be mea­

tivity and its relationship to vestibular Rinehart & Winston, 1973

sured, Am Sociol Rev 9: 139-150, 1944

function in chronic schizophrenia. 13, Kerlinger F: Foundations of Behavioral

25. Guttman L: A basis for scaling qualita­

Master's Thesis, University of South­ Research (2nd Edition), New York: Holt
tive data. Am Sociol Rev 9: 139-150,

ern California, Los Angeles, California, Rinehart & Winston, 1973

1944

June 1981 14,Crocker LM: Validity of certification

26,Greenstein L: Psychomotor objectives

4.Drucker B: An imitation gesture test for measures for occupational therapists,

in occupational therapy education, Am

adolescents, Master's Thesis, Univer­ Am J Occup Ther 30: 229-233, 1976

J Occup Ther 30: 301-357, 1976

sity of Southern California, Los An­ 15,Crocker LM: Linking research to prac­
27.Edwards AL: Techniques of Attitude

geles, California, June 1981 tice: suggestions for reading a research

Scale Construction, New York: Apple­

5.Koomar JA, Cermak SA: Reliability of

article. Am J Occup Ther 31: 34-39,
ton-Century-Croft, 1957

dichotic listening using two stimulus

1977
28.Kamorita SS, Graham WK: Number of

formats with normal and learning-dis­

16.Buros OK (Editor): Mental Measure­
scale points and the reliability of scales.

abled children, Am J Occup Ther 35:

ment Yearbook (Vols 1-8). Highland
Educ Psychol Measure 4:987-995, 1965

456-463, 1981
Park, NJ: Gryphon Press, 1972
29, Lissitz R, Green S: The effect of the
6.Yerxa EJ, Burnett SE, Stocking S, Azen 17.Gilfoyle EM (Editor): Training: Occu­
number of scale points on reliability: A
SP: Development of the satisfaction pational Therapy Education Manage­
Monte Carlo approach. J App Psychol
with performance scaled questionnaire ment in Schools (Vols 1-4). Rockville,
60: 10-13, 1975

(SPSQ). Unpublished manuscript, Uni­ MD: American Occupational Therapy

30,Frisbie 0, Brandenburg 0: Equivalence

versity of Southern California, Los An­ Association, 1978

of questionnaire items with varying

geles, California 18.Harrow A: A Taxonomy 01 Psychomo­

response formats. J Educ Measure 16:

7.De Gangi GA, Berk RA, Larsen LA: The

tor Domain, New York: David McKay,
43-48, 1979

measurement of vestibular-based func­

1972
31.Hull CH, Nie NH: SPSS Update, New

tions in pre-school children, Am J

19.Bloom B (Editor): Taxonomyof Educa­
York: McGraw Hill, 1979

Occup Ther 34: 452-459, 1980

tional Objectives: The Classification of
32,Cronbach L: Test validation. In Educa­
8,Gilligan MB, Mayberry W, Stewart L,
Educational Goals, Handbook I: Cog­
tional Measurement (2nd Edition), RL

Kenyon P, Gaebler C: Measurement of

nitive Domain, New York. David McKay,
Thorndike, Editor. Washington, DC:

ocular pursuits in normal children. Am

1972
American Council on Education, 1971

J Occup Ther 35: 250-255, 1981

20.Krathwohl 0, Bloom B, Bertram M:
33,Cronbach L, Meehl P: Construct valid­

9.Harris NP: Duration and quality of the

Taxonomy of Educational Objectives:
ity in psychological tests. Psychol Bull

prone extension posture in four-, six­

The Classification of Educational
52 281-302, 1955

an eight-year-old normal children.

Goals: Handbook 1/: Affective Domain,
34.Campbell D, Fiske 0: Convergent and

Am J Occup Ther 35: 26-30, 1981

New York: David McKay, 1972
divergent validation by multitrait-multi­

10,Baum B: The establishment of reliabil­ 21 Benson J, Crocker L: The effects of method matrix. Psychol Bull 56: 81­

ity and validity of a perceptual evalua­ item format and reading ability on ob­ 105, 1959

800 December 1982, Volume 36, Number 12

Downloaded From: http://ajot.aota.org/pdfaccess.ashx?url=/data/journals/ajot/930542/ on 09/19/2018 Terms of Use: http://AOTA.org/terms