Practice: Get quality data into your evaluators hands

Key Action: Use technique to ensure valid and reliable data

TOOL: Making Sure Data Are Valid and elia!le
Purpose: Before you can confidently interpret and analyze your evaluation data, you
must ensure that the data you collect are valid and reliable. Otherwise, they
wont adequately support your outcomes. Use the questions and suggestions
in this table to ensure the data you collect are valid and reliable.
Instructions: . !eview the "#uestions to $onsider% in the table to ensure that the data
you collect for your evaluation are valid and reliable, and that your sample
size is adequate.
&. $onsider the suggestions or proactive measures you might ta'e to ensure
valid and reliable data.
(. Based on what you learn, note specific actions you might use in your
district to ensure that your evaluation generates valid and reliable data.

Practice: Get quality data into your evaluators hands

Key Action: Use technique to ensure valid and reliable data
Making Sure Data Are Valid and elia!le
"uestions to consider Suggestions Actions #ell consider
Do the data re$resent the
outco%es that the instru%ent
is su$$osed to %easure& Are
the data valid&
) valid measure assesses what
it is designed to measure, which
allows for comparison of results
across studies.
*irst ma'e sure you select instruments that measure the
'inds of outcomes your magnet program is e+pected to
produce. ,ou can increase measurement validity by using
field-tested instruments that have demonstrated reliability
and validity.
.ote: /tate assessments have been recalibrated or revised
during the years of your study, so you may not be able to
compare data from year to year.
'ave #e ensured that our
%easures are relia!le&
) reliable measure produces
stable responses regardless of
the data collector. )n unreliable
measure will yield varied
responses depending on
differences between interviewers
or data collectors.
0iscuss with your evaluator how you will test for reliability of
your instruments. ,ou may want to borrow from e+isting
instruments, have an e+pert panel review and react to new
instruments, or pilot test the instruments in real settings and
among members of your target audience. Build in time and
resources to test for reliability and calculate reliability
coefficients so that you can assure sta'eholders that you
have strong instruments.
.ote that qualitative data collection 1e.g., observations,
open-ended interviews2 poses different validity challenges.
3deally, the instruments you create for these purposes will
require low levels of inference. 4rovide time and resources
for researcher training in use of the instruments to minimize
differences in participant responses across data collectors.
Do #e have the right data&
Do #e have enough data&
,ou will need an adequate sample size to ensure your data
are valid. 5a'e sure you begin with large enough numbers
of students and schools in your evaluation study, ta'ing
&
Practice: Get quality data into your evaluators hands
Key Action: Use technique to ensure valid and reliable data
"uestions to consider Suggestions Actions #ell consider
6o be valid for decision-ma'ing,
data must answer your
questions about program
outcomes and include a
sufficient number of participants
to be representative of your
target population and its various
subgroups. 7alidity of data is
improved when data collection is
"triangulated,% for e+ample,
when various methodologies are
used to measure the same
phenomenon, or multiple
researchers conduct a
structured observation of the
same phenomenon.
pro8ected attrition into account. 6hen, as data collection
begins, chec' your data to ma'e sure that subgroup data
are appropriately coded and that sufficient numbers of
students in subgroups have ta'en the tests as planned.
Otherwise you may not have enough students in particular
subgroups to ma'e those data valid.
(