PSYCHOLOGICAL ASSESSMENT | CHAPTER 5 – VALIDITY
DEFINING VALIDITY
VALIDITY – defined as the agreement between a test score or measure
and the quality it is believed to measure.
 Test measures what it is supposed to measure.
 1985 – AERA, APA, NCME published a booklet entitled
Standards for Educational and Psychological Testing.
 Standards were revised in 1999 and 2014
 Latest version of standards is organized in three sections:
foundations, operations, and applications.
o Foundations is similar to our principles and focuses on
basic psychometric concepts such as validity and
reliability.
o Operations considers how tests are designed and built and
how they are administered, score and reported.
o Applications takes on a wide range of issues, ranging
from training required to administer and interpret tests.
 Validity is the evidence for inference made about a test score.
ASPECTS OF VALIDITY
1. FACE VALIDITY – is the mere appearance that a measure has
validity.
 A test has face validity if the items seem to be reasonably
related to the perceived purpose of the test.
 Example: a scale to measure anxiety might include such as
“My stomach gets upset when I think about taking tests” and
“my heart starts pounding fast whenever I think about all of the
things I need to get done”. Can we conclude that the person is
anxious? In this case, we can only conclude that the person
answers these two items in a particular way.
 Face validity is really not validity at all because it does not
offer evidence to support conclusions drawn from test scores.
 in many settings, it is crucial to have a test that “looks like” it is
valid. These appearances can help motivate test takers because
they can see that the test is relevant.
2. CONTENT-RELATED EVIDENCE FOR VALIDITY – considers the
adequacy of representation of the conceptual domain the test is designed
to cover.
 Example: if you are being tested on the first six chapter of this
book, then the content to the test is related to the information in
the chapters.
 Content validity evidence offers some unique features, such as it
is the only type of evidence besides face validity that is logical
rather than statistical.
 Two new concepts relevant to content validity evidence:
1. Construct underrepresentation – describes the
failure to capture important components of a
construct.
2. Construct-irrelevant variance – occurs when scores
are influenced by factors irrelevant to the construct.
3. CRITERION-RELATED EVIDENCE FOR VALIDITY – tells us
how well a test corresponds with a particular criterion.
 A criterion is the standard against which the test is compared.
 The reason for gathering criterion validity evidence is that the
test or measure is to serve as a “stand-in” for the measure we are
really interested in.
TYPES OF EVIDENCE FOR CRITERION VALIDITY ARE:
1. PREDICTIVE VALIDITY EVIDENCE – forecasting function of tests.
 Example: the SAT Critical Reading Test serves as predictive
validity evidence as a college admissions test if it accurately
forecasts how well high-school students will do in their college
studies.
o Predictor variable – quantitative and verbal tests
o Criterion – college GPA
2. CONCURRENT VALIDITY EVIDENCE – comes from assessments
of the simultaneous relationship between the test and the criterion.
 Example: learning disability and school performance.
 VALIDITY COEFFICIENT – relationship between a test and
a criterion is usually expressed as a correlation. This coefficient
tells the extent to which the test is valid for making statements
about the criterion.
EVALUATING VALIDITY COEFFICIENTS
1. LOOK FOR CHANGES IN THE CAUSE OF RELATIONSHIPS
 conditions of a validity study are never exactly reproduced.
 The logic of criterion validation presumes that the causes of the
relationship between the test and the criterion will still exist
when the test is in use. However, there may be circumstances
under which the relationship changes.
o Example: A test might be used and shown to be valid for
selecting supervisor in the industry; however, the validity study
may have been done at a time when all the employees were men,
making the test valid for selection supervisors for male
employees. If the company hires female employees, then the
test may no longer valid for selecting supervisors because it may
not consider the abilities necessary to supervise a sexually
mixed group o employees.
2. WHAT DOES THE CRITERION MEAN?
CRITERION-RELATED VALIDITY
 Studies mean nothing at all unless the criterion is valid &
reliable
 Constructors attempt to correlate their tests with other tests that
have unknown validity.
 Meaningless group of items that correlates well with another
meaningless group remains meaningless
REVIEW THE SUBJECT POPULATION IN THE VALIDITY
STUDY
 Validity study might have been done on a population that does
not represent the group to which inferences will be made
 Example: Some researchers have debated whether validity
coefficients for intelligence and personnel tests that are based
primarily on white samples are accurate when used to test
African American students
 Example: Industrial settings, attrition can seriously jeopardize
validity studies.
 Those who do poorly on the job either drop out or get fired and
thus cannot be studied when it comes time to do the job
assessment.
 If there was a group that did well on the test but failed on the
job, then it might not be represented and could be systematically
eliminated from the study because the workers were already off
the job by the time the assessment came around.
3. BE SURE THE SAMPLE SIZE WAS ADEQUATE
 Problem to look for is a validity coefficient that is based on a
small number of cases.
 Sometimes a proper validity study cannot be done because
there are too few people to study
 Cannot depend on a correlation obtained from a small sample,
particularly for multiple correlation and multiple regression
 Such a study can be quite misleading.
o CROSS VALIDATION STUDY - good validity study will
present some evidence for this; assesses how well the test
actually forecasts performance for an independent group of
subjects
 INITIAL VALIDITY STUDY: assesses the relationship
between the test and the criterion
 CROSS VALIDATION STUDY: checks how well this
relationship holds for an independent group of subjects.