PSYCHOLOGICAL MEASUREMENTS : Reliability estimate

Reliability, Validity and Utility

RELIABILITY - Synonym for dependability or
consistency, refers to the consistency of measurement.
● reliability coefficient – an index that indicates
the ratio between the true score variance on a
test and total variance.
● “reliability estimates” in the range of .70 and
.80 are good enough for most purposes in basic
research.
THE CONCEPT OF RELIABILITY
CLASSICAL TEST THEORY – That a score on a ability
test is presumed to reflect not only the test takers true
score on the ability being measures but also error.
Observe Score = True ability + Random Error
Measurement error – Are all of the factors associated
with the process of measuring some variable, other than
the variable being measured.
1. Test Retest Reliability – Also known as
time-sampling reliability because it uses the same
instrument to measure the same thing at two points in
time.
2. Parallel forms and Alternate forms reliability-
Compares two equivalent forms of a test that measure
the same attribute.
3. Internal Consistency – assesses the correlation
between multiple items in a test that are intended to
measure the same construct.
Ways to compute internal consistency
● Split–half reliability – correlating two pairs of
scores obtained from equivalent halves of a single test
administered once.
● Kr 20 formula or Kuder-Richardson formula 2
- A measure of internal consistency reliability for
measures with dichotomous choices.

Two forms of measurement error

1. Random Error - this source of error fluctuates from
one testing situation to another with no discernible ● Cronbach’s Alpha – the most famous and
pattern that would systematically raise or lower scores. commonly used among reliability coefficients because it
The main reason for random error are limitation of requires only one administration of the test.
instruments, environmental factors, and slight variation
in procedure.
2. Systematic error - this source of error is predictable
and either constant or else proportional to the
measurement. Systematic error affect validity. Typically ● Average proportional distance ( APD ) – as a
causes observational error, imperfect instrument measure used to evaluate the internal consistency of a
calibration and sampling bias. test that focuses on the degree of differences that exists
between item scores.
ERROR VARIANCE - The element of variability in a
score that is produced by extraneous factors, such as
measurement imprecision, and is not attributable in
independent variable, or other controlled experimental WHAT TO DO ABOUT LOW RELIABILITY
manipulation.
1. Item sampling or content sampling – refers to 1. Increase the number of items – the larger the
variation among items within a test as well as to variation sample the more likely that the test will represent the
among items between tests. true characteristic.
2. Administration- Test environment: the room
temperature, level of lighting, and the amount of 2. Factor and number analysis – test are most reliable
ventilation and noise. if they are unidimensional ; measuring a single ability,
● Test taker variables – emotional attribute, construct or skill.
problems, physical discomfort, lack of
sleep
● Scoring & interpretation – despite
rigorous scoring criteria set forth in
many of the better known tests of
intelligence, the scorer or rater can
be a source of error variance.

CHAPTER 7: VALIDITY
VALIDITY
validity is a term used in conjunction with the
meaningfulness of a test score what the test score
truly means.
CONCEPT OF VALIDITY
Validity, as applied to a test, is a judgment or
estimate of how well a test measures what it
purports to measure in a particular context. More
specifically, it is a judgment based on evidence
about the appropriateness of inferences drawn from
test scores.(An inference is a Logical result or
deduction.) Characterizations of the validity of tests
and test scores are frequently phrased in terms
such as “acceptable” or “weak.” These terms reflect
a judgment about how adequately the test
measures what it purports to measure.
- How useful the instrument is for a particular
purpose with a particular population of people
(“valid test.”)
- test has been shown to be valid for a particular
use with a particular population of testtakers at a
particular time
- No test or measurement technique is
“universally valid” for all time, for all uses, with all
types of testtaker populations.
- Further, to the extent that the validity of a test
may diminish as the culture or the times change,
the validity of a test may have to be re-established
with the same as well as other testtaker
populations.
Validation is the process of gathering and
evaluating evidence about validity. Both the test
developer and the test user may play a role in the
validation of a test for a specific Purpose. It is the
test developer’s responsibility to supply validity
evidence in the test manual.
Validation in therapy include active listening,
mindfully responding, and other validation skills that
encourage the understanding and acceptance of
the client's experiences. This helps clients feel
heard and understood, contributing to the
therapeutic outcome
It may sometimes be appropriate for
test users to conduct their own validation
studies with their own groups of testtakers.
Such local validation studies may yield
insights regarding a particular population of
testtakers as compared to the norming
sample described in a test manual. Local
validation studies are absolutely necessary
when the test user plans to alter in some
way the format, instructions, language, or
content of the test.
Example:
Adolescent Depression Inventory (ADI) in
identifying symptoms of depression
Local validation studies would also be
necessary if a test user sought to use a test with a
population of testtakers that differed in some
significant way from the population on which the
test was standardized.
One way measurement specialists have
traditionally conceptualized validity is according to
three categories:
1. Content validity. This is a measure of validity
based on an evaluation of the subjects, topics, or
content covered by the items in the test.
2. Criterion-related validity. This is a measure
of validity obtained by evaluating the relationship of
scores obtained on the test to scores on other tests
or measures
3. Construct validity. This is a measure of
validity that is arrived at by executing a
comprehensive analysis of
a. how scores on the test relate to other test
scores and measures, and
b. how scores on the test can be understood
within some theoretical framework for
understanding the construct that the test was
designed to measure.
- Trinitarian approaches to validity
assessment are not mutually exclusive
- all three types of validity evidence
contribute to a unified picture of a test’s validity
- A test user may not need to know about
all three
- Depending on the use to which a test is
being put, one type of validity evidence may be
more relevant than another.
Ecological validity refers
to a judgment regarding how well a test
measures what it purports to measure at the time
and place that the variable being measured
(typically a behavior, cognition, or emotion) is
actually emitted. In essence, the greater the
ecological validity of a test or other measurement
procedure, the greater the generalizability of the
measurement results to particular real-life
circumstances.
FACE VALIDITY
- Face validity relates more to what a test
appears to measure to the person being tested
than to what the test actually measures.
- if a test definitely appears to measure what it
purports to measure “on the face of it,” then it could
be said to be high in face validity.
- A test’s lack of face validity could contribute to
a lack of confidence in the perceived effectiveness
of the test—with a consequential decrease in the
testtaker’s cooperation or motivation to do his or
her best.
- In a corporate environment, lack of face validity
may lead to unwillingness of administrators or
managers to “buy-in” to the use of a particular test
- In a similar vein, parents may object to having
their children tested with instruments that lack
ostensible validity. Such concern might stem from a
belief that the use of such tests will result in invalid
conclusions.
- In reality, a test that lacks face validity may still
be relevant and useful. However, if the test is not
perceived as relevant and useful by testtakers,
parents, legislators, and others, then negative
consequences may result. These consequences
may range from poor testtaker attitude to lawsuits
filed by disgruntled parties against a test user and
test publisher. Ultimately, face validity may be more
a matter of public relations than psychometric
soundness. Still, it is important nonetheless, and
(much like Rodney Dangerfield) deserving of
respect.
CONTENT VALIDITY
Content Validity describes a judgment of how
adequately a test samples behavior representative
of the universe of behavior that the test was
designed to sample/ refers to the extent to which a
test or assessment instrument evaluates all aspects
of the topic, construct, or behavior that it is
designed to measure.
CRITERION-RELATED VALIDITY
is an assessment of how well a test score can
accurately predict or reflect an individual's likely
performance on a specific measure of interest,
which is referred to as the criterion.
Concurrent validity checks how well a test score
aligns with a criterion measured at the same time.
Predictive validity, on the other hand, measures
how accurately a test score can predict future
performance on a criterion.
An adequate criterion measure must also be valid
for the purpose for which it is being used. If one test
(X) is being used as the criterion to validate a
second test (Y), then evidence should exist that test
X is valid. If the criterion used is a rating made by a
judge or a panel, then evidence should exist that
the rating is valid.
criterion must not be contaminated
When criterion contamination does occur, the
results of the validation study cannot be taken
seriously. There are no methods or statistics to
gauge the extent to which criterion contamination
has taken place, and there are no methods or
statistics to correct for such contamination.
Concurrent Validity
If test scores are obtained at about the same time
as the criterion measures are obtained, measures
of the relationship between the test scores and the
criterion provide evidence of concurrent validity.
Statements of concurrent validity indicate the extent
to which test scores may be used to estimate an
individual’s present standing on a criterion.
Predictive Validity
Test scores may be obtained at one time and the
criterion measures obtained at a future time,
usually after some intervening event has taken
place. The intervening event may take varied
forms, such as training, experience, therapy,
medication, or simply the passage of time.
Measures of the relationship between the test
scores and a criterion measure obtained at a future
time provide an indication of the predictive validity
of the test; that is, how accurately scores on the
test predict some criterion measure.
base rate - is the extent to which a particular trait,
behavior, characteristic, or attribute exists in the
population (expressed as a proportion)
hit rate may be defined as the proportion of people
a test accurately identifies as possessing or
exhibiting a particular trait, behavior, characteristic,
or attribute.
ex: Imagine you have a test designed to identify
individuals with high creativity. You administer the
test to a group of people, and the hit rate is the
proportion of those correctly identified by the test as
highly creative among all the truly creative
individuals in the group. For example, if the test
correctly identifies 80 out of 100 genuinely creative
individuals, the hit rate would be 80%.
miss rate may be defined as the proportion of
people the test fails to identify as having, or not
having, a particular characteristic or attribute:
false positive, false negative
Construct Validity
Is a judgment about the appropriateness of
inferences drawn from test scores regarding
individual standings on a variable called a
construct. A construct is an informed, scientific idea
developed or hypothesized to describe or explain
behavior.
Constructs are unobservable, presupposed
(underlying) traits that a test developer may invoke
to describe test behavior or criterion performance.
- hypotheses about the expected behavior of
high scorers and low scorers on the test
- hypotheses give rise to a tentative theory about
the nature of the construct the test was designed to
measure.
- If the test is a valid measure of the construct,
then high scorers and low scorers will behave as
predicted by the theory.
- investigator will need to reexamine the nature
of the construct itself or hypotheses made about it
- the test simply does not measure the construct.
- One procedure may have been more
appropriate than another, given the particular
assumptions.
Although confirming evidence contributes to a
judgment that a test is a valid measure of a
construct, evidence to the contrary can also be
useful.
- Contrary evidence can provide a stimulus for
the discovery of new facets of the construct as well
as alternative methods of measurement.
Evidence of Construct Validity
■ the test is homogeneous, measuring a single
construct;
■ test scores increase or decrease as a function of
age, the passage of time, or an experimental
manipulation as theoretically predicted;
■ test scores obtained after some event or the
mere passage of time (or, posttest scores)
differ from pretest scores as theoretically predicted;
■ test scores obtained by people from distinct
groups vary as predicted by the theory;
■ test scores correlate with scores on other tests in
accordance with what would be predicted from a
theory that covers the manifestation of the
construct in question.
Validity, Bias, and Fairness Test Bias
bias is a factor inherent in a test that systematically
prevents accurate, impartial measurement.
Test bias can arise more from the study design than
the test itself
Rating error A rating is a numerical or verbal
judgment (or both) that places a person or an
attribute along a continuum identified by a scale of
numerical or word descriptors known as a rating
scale. Simply stated, a rating error is a judgment
resulting from the intentional or unintentional
misuse of a rating scale. Thus, for example, a
leniency error (also known as a generosity error) is,
as its name implies, an error in rating that arises
from the tendency on the part of the rater to be
lenient in scoring, marking, and/or grading.
Severity error is a type of rating error in which the
ratings are consistently overly negative, particularly
with regard to the performance or ability of the
participants. It is a type of error that can occur in
psychometric assessments. Severity error is the
opposite of leniency error, which is a type of rating
error in which the ratings are consistently overly
positive.
central tendency error
Here the rater, for whatever reason, exhibits a
general and systematic reluctance to giving ratings
at either the positive or the negative extreme.
Consequently, all of this rater’s ratings would tend
to cluster in the middle of the rating continuum.
solution:
use rankings
Halo effect describes the fact that, for some raters,
some ratees can do no wrong. More specifically, a
halo effect may also be defined as a tendency to
give a particular ratee a higher rating than he or
she objectively deserves because of the rater’s
failure to discriminate among conceptually distinct
and potentially independent aspects of a ratee’s
behavior.
Test Fairness
Unlike the technically complex nature of test bias,
concerns about test fairness are often tied to
values. While test bias can be addressed with
precision, fairness is subjective and can lead to
ongoing debates among people with different
viewpoints. In the context of psychometrics,
fairness is defined as the degree to which a test is
employed impartially, justly, and equitably.
CHAPTER 7: UTILITY
UTILITY: usefulness or practical value of testing to
improve efficiency.
FACTORS THAT AFFECT A TEST’S UTILITY
• Psychometric Soundness
o Reliability and validity of a test
o Gives us the practical value of both the
scores
(reliability and validity)
o They tell us whether decisions are
cost-effective
o A valid test is not always a useful test
especially if
test takers do not follow test directions
• Costs
o Economic and non economic
o (e.g.) using a less expensive and therefore
less
stringent application process for airline
personnel.
• Benefits
o Profits, gains, advantages
o (e.g.) more stringent hiring policy more
productive
employees
o (e.g.) maintaining successful and academic
environment of university
UTILITY ANALYSIS
What is Utility Analysis?
- a family of techniques that entail a cost-benefit
analysis designed to yield information relevant to a
division about the usefulness and/or practical value
of a tool of assessment.
How Is a Utility Analysis Conducted?
objective: dictate what sort of information will be
required as well as the specific methods to be used
Expectancy Data
o Expectancy table provides indication of the
likelihood that a test taker will score within some
interval of scores on a criterion measure
o Used to measure costs vs. benefits
Brogden-Cronbach-Gleser formula
o Utility gain: estimate of the benefit of using a
particular test or selection method
o Most simply is benefits-cost
o Productivity gain: estimated increase in work
output
SOME PRACTICAL CONSIDERATIONS
The Pool of Job Applicants
o There is rarely a limitless supply of
potential employees
o Dependent on many factors, including
economic environment
o We assume that top scoring individuals
will accept the job, but those individuals
are more likely to be the ones being
offered higher positions
The Complexity Of The Job
o It is questionable whether the same
utility analysis methods can be used to
measure the eligibility of varying
complexities of jobs
The Cut Score In Use
o Relative cut score: may be defined as a
reference point.
- Based on norm-related
considerations rather than on the
relationship of test scores to a
criterion
- Also called norm-referenced cut
score.
- (e.g.) top 10% of test scores get
A’s
o Fixed cut score: set with reference to a
judgment concerning a minimum level of
proficiency required to be included in a
particular classification.
- Also called absolute cut scores
o Multiple cut scores: using two or more
cut scores with reference to one predictor
for the purpose of categorizing test taker
- (e.g.) having cut score that marks
an A, B, C etc. all measuring
same predictor
o Multiple hurdles: for success, requires
one individual to complete many tasks,
with elimination at each level
- (e.g.) written application -> group
interview -> personal interview
etc.
o Compensatory model of selection:
assumption is made that high scores on
one attribute can compensate for low
scores on another attribute.
Methods for Setting Cut Scores
The Angoff Method
Judgments of experts are averaged
The Known Groups Method
Collection of data on the predictor of
interest from a group known to possess
and not to possess trait, attribute, or
ability. Cut score based on which test best
discriminates the two groups performance
IRT-Based Method
Based on test taker’s performance across
all items on a test
Some portion of the total number of items
on the test must be scored “correct” order
for the test taker to “pass” the test
Item-mapping method: determining
difficulty level reflected by cut score
Book-Mark method: test items are listed,
one per page, in ascending level of
difficulty. An expert places a bookmark to
mark the divide which separates test
takers who have acquired minimal
knowledge, skills, or abilities and those
that have not.
Problems include training of experts,
possible floor and ceiling effects, and the
optimal length of item booklets
Other Methods
-discriminant analysis: family of statistical
techniques used to shed light on the
relationship between certain variables and
two or more naturally occurring groups
-(e.g.) the relationships between scores of
tests and ppl judged to besuccessful or
unsuccessful at job