Chapter 2: Statistics Refresher (ex: Celsius, Fahrenheit)

Test scores are frequently expressed as numbers, and 4. Ratio Scale: has all the three properties (Kelvin)
statistical tools are used to describe, make inferences from,
and draw conclusions about numbers. Type of Scale Magnitude Equal Absolute 0
Intervals
Why we need statistics? Nominal No No No
Statistical methods serve two important purposes in the
Ordinal Yes No No
quest for scientific understanding.
1. Statistics are used for purposes of description. Interval Yes Yes No
2. We can use statistics to make inferences, which are Ratio Yes Yes Yes
logical deductions about events that cannot be observed
directly. Frequency Distrubutions
A distribution of scores summarizes the scores for a
Descriptive Statistics are methods used to provide a group of individuals.
concise description of a collection of quantitative The frequency distribution displays scores on a
information. variable or a measure to reflect how frequently each
Inferential Statistics are methods used to make value was obtained; defines all the possible scores and
inferences from observations of a sample to population. determines how many people obtained each of those
scores.
Scales of Measurement
Measurement: application of rules for assigning
numbers to objects.
The rules are the specific procedures used to transform
qualities of attributes into numbers.

Continuous Scale: measuring

Discrete Scale: counting or categorizing; countable

Properties of Scales TEST IS DIFFICULT TEST IS EASY

Three important properties make scales of measurement
different from one another:
1. Magnitude: property of “moreness”; if a particular
instance of the attribute represents more, less, or
equal amounts of the given quantity than does
another instance.
2. Equal Interval: if the difference between two points
at any place on the scale has the same meaning as
the difference between two other points that differ
by the same number of scale units.
3. Absolute 0: when nothing of the property being
measured exists; for many psychological qualities, it
is extremely difficult, if not impossible, to define an
absolute 0 point
Skewness: extent w/c symmetry is absent
 Positively skewed: frequent scores are clustered
at the lower end and the tail points toward higher or
more positive scores.
 Negatively skewed: frequent scores are clustered
at the higher end and the tail points toward lower or
more negative scores.
Kurtosis: degree to which score cluster at the ends of
distribution; the tails; steepness
 Leptokurtic- pointy positive kurtosis
 Mesokurtic- normal distribution
 Platykurtic- flat negative kurtosis

Types of Scales [NOIR]

1. Nominal Scales: classification or categorization
based on one or more distinguishing characteristics.
(ex: DSM, yes/no scale, sex, hair color, marital etc.)
2. Ordinal Scales: rank ordering characteristics;
compared / other & assigned a rank
(ex: board exam topnotchers, likert scale, level of
agreement)
3. Interval Scale: has the properties of magnitude and
equal intervals but not absolute 0
Percentiles
The specific scores or points within a distribution; it
divides the total frequency for a set of observations into
hundredths; indicates the particular score, below which a
defined percentage of scores falls.
Percentile Ranks replace simple ranks to adjust
the number of scores in a group.
What percent of the scores fall below a particular score?
 Comparative (difference, effect)
no. of times DV no. of groups
measured
no. of beaten examinees T-test independent 1 2
X 100
total no. of scores means
T- test dependent 2 1
interval,
means
Measures of Central Tendency ratio
Anova one way 1 >2
Mean: average of a data set/ arithmetic average score in Anova repeated >2 1
measures
a distribution Mann Whitney U test 1 2
Median: middle of the set of numbers (organizing the Wilcoxon Signed 2 1
scores in ascending/descending order) ordinal Rank test
Kruskal Wallis test 1 >2
Mode: most common number/score in a data set; 2 Friedman test >2 1
scores/modes—bimodal Anova two way 2 IV, 2 levels
MANOVA many measure-
Standard Deviation: an approximation of the average ments of DV
deviation around the mean; measure of how spread-out
numbers are Parametric/ Non-Parametric
Variance: average of the squared differences from the  Dependent T-test → Wilcoxon Signed Rank Test
mean  Independent T-test → Mann Whitney U Test
 Repeated Measures Anova → Friedman Test
Z score
 One way/Two way Anova → Kruskal Wallis Test
It transforms data into standardized units that are easier
to interpret; the difference between a score and the
Norms And A Good Test
mean, divided by the standard deviation.
Norms: performances by defined groups on particular
T score (McCall's T)
tests; standard which results will be compared; used to
It refers to a standardized score system that can be
give information about performance relative to what has
obtained from a simple linear transformation of Z scores.
been observed in a standardization sample
Mean = 50
Standard Deviation = 10
Age-related Norms
- Certain tests have different normative groups for
Standard Scores
particular age groups.
MEAN SD
- ex: SB-IQ test were obtained from various age
Z scores 0 1
groups.
T scores 50 10
Norm-referenced Tests
Stanine scores 5 2
Sten scores 5.5 2 - Determine how a test taker compares with other or
IQ 100 15 each person with a norm.
CEE 500 100 Criterion-referenced Tests
- Describes the specific types of skills, tasks, or
Variability knowledge a test taker can demonstrate.
 Range: High score- Low score
 Quartile: division of frequency, normal distribution Norming
divided into 4 parts Standardization: process of administering a test to a
 Interquartile: Q3-Q1 representative sample of test takers for the purpose of
establishing norms
 Semi-interquartile: Interquartile divided by 2
Sample: a portion of the universe of people that
represents the whole population
Sampling: process of selecting sample

Probability Sampling
- Simple Random Sampling: everyone has an equal
chance of being selected as part of the sample
- Systematic Sampling: every nth item or person after
is picked
- Stratified Sampling: random selection within
predefined groups
- Cluster Sampling: groups rather than individual units
of the target population are selected randomly
Non-Probability Sampling
- Convenience Sampling: selected based on their
availability
- Quota Sampling: specifying who should be recruited
for a survey according to certain groups or criteria
- Purposive Sampling: chosen consciously based on
their knowledge and understanding of the research
question
- Snowball or Referral Sampling: people recruited to
be part of a sample are asked to invite those they
know to take part
2.1. Reliability
- pertains to the consistency of test measurement
- can be estimated from the correlation of the
observed tests score with the true score
- a test may be reliable in one context and unreliable
in other
In psychological testing, the word error does not imply
that a mistake has been made. error implies that there
will always be some inaccuracy in our measurements.
In other words, tests that are relatively free of
measurement error are reliable, and tests that have so
much measurement error are considered unreliable.
History & Theory of Reliability
 Abraham De Moivre introduced the basic notion of
sampling error.
 Karl Pearson developed the product moment
correlation.
 Charles Spearman worked out most of the basics of
contemporary reliability theory with 1904 article
entitled "The Proof and Measurement of Association
between Two Things."
testers use “rubber yardsticks” to estimate measurements
Basics of Test Score Theory
The difference between the score we obtain and the
score we are really interested is the error of
measurement – X-T=E
Classical Test Score Theory: assumes that each
person has a true score that would be obtained if there
were no errors in measurement ( X=T+E)
- The difference between the true score and the
observed score results from measurement error
Standard error of measurement (SEm)
estimates how repeated measures of a person on the
same instrument tend to be distributed around his “true”
score; which the true score is always unknown because
no measure can be constructed that provides a perfect
reflection of the true score.
Domain Sampling Theory
another central concept in CTT; assumes that the items
that have been selected for any one test are just a
sample of items from an infinite domain of potential
items; considers the problems created by using a limited
number of items to represent a larger and more
complicated construct.
- as the sample gets larger, it represents the domain
more and more accurately.
- the greater the number of items, the higher the
reliability.
Item Response Theory (IRT)
A way to analyze responses to tests or questionnaires
with the goal of improving measurement accuracy and
reliability; used to focus on the range of item difficulty
that helps assess an individual's ability level.
item difficulty = item easiness
Sources of Error Variance
 Test Construction: item sampling or content
sampling, terms that refer to a variation among
items within a test as well as variation among items
between tests.
 Test Administration: may influence the test taker's
attention or motivation.
 Test Scoring and Interpretation: individual
administered test still require scoring by trained
personnel.
Test Re-Test Reliability
1 group, 1 test, 2 administration
2 weeks – 6months
o used to evaluate the error associated with
administering a test at two different times.
o It is appropriate when evaluating the reliability of a
test that purports to measure something that is
relatively stable over time, ex: personality trait.
o Carryover Effect: occurs when the first testing
session influences the scores from the old session.
(short span of interval)
o Practice Effect: the test-retest correlation usually
overestimates the true reliability. (test-taker's
concern)
Test-Retest Procedure
 Sample population = (Test A) [1] + (Test A) [2]
 Administer the psychological test
 Get the test result
 Wait for the interval (time gap)
 Re-administer the psychological test
 Get the test result
 Correlate the results
Disadvantages of Test-Retest
 Possible better performance
 Checking / knowing the answers
 Practice effect
coefficient of stability: the estimate of test-retest
reliability, when the interval between testing is greater
than six months
Parallel-Forms & Alternate- Forms Reliability
1 group, 2 test, 1 administration
o compares two equivalent forms of test that measure
the same attribute. The two forms use different
items; however the rules used to select items of a
particular difficulty level are the same
o two forms are administered to the same group of
people on the same day. Pearson's R product-
moment correlation is used to estimate the reliability
o alternate forms are designed to be equivalent with
respect to variables such as content and level of
difficulty.
o coefficient of equivalence- measures the same
attributes.
Procedure
 Administer the first test
 Administer the Alternate form test
 Score both tests
 Correlate
Disadvantages of using the Alternate Form Method
 Hard to develop or construct
 Time consuming
Split Half Reliability
1 group, 2 test, 2 administration
o obtained by correlating two pairs of scores obtained
from equivalent halves of a single test administered
once.
o test is given and divided into halves that are scored
separately.
o results of one half of the test are then compared
with the results of the other.
o if the test is long, the best method is to divide the
items randomly into two halves.
o to shorten the items, odd-even system or top
bottom method can be used.
- top bottom method: 1st half (25 items), 2nd
half (25 items)
- odd even system: 1st half (odd), 2nd half
(even)
o to adjust the half-test reliability, use the Spearman
Brown Formula.
o allows a test developer or user to estimate internal
consistency reliability from correlation of two halves
of a test.
o Spearman-Brown formula: r = 2r/1+r
where, (r) is the estimated correlation between the two
halves of the test (if each had had the total number of
items) and r is the correlation between the two halves of
the test
As a Psychometrician in a clinic, the psychologist instructed you to
develop a test that measure emotional stability of suicidal patients. You
decided to use split-half to establish its reliability. The correlation
between two halves of the test is .78. according to the spearman-
brown formula, the estimated reliability would be .876
Computation: r = 2 (.78) / 1 + .78
Kuder-Richardson Formula (KR20)
KR20 is (Kuder & Richardson, 1937) developed this
formula to measure the reliability estimate of Split-half.
The formula is applicable to items that are
DICHOTOMOUS, scored 0 or 1 (usually right or wrong).
o used when the items varies the level of difficulty.
KR21: used when the items have same level of difficulty
Coefficient Alpha
o Cronbach's Alpha (Cronbach, 1951) sometimes
called coefficient alpha (a) is the most common
measure of internal consistency (reliability).
o Cronbach's Alpha is most commonly used when you
have multiple Likert questions in a
survey/questionnaire that form a scale and you wish
to determine if the scale is reliable.
o Items are not scored as 0 or 1. Applicable for
personality and attitude scales.
How reliable is reliability?
- In research setting, it has been suggested that a
reliability estimates in the range of .70 and .80 are
 good enough for most purposes in basic research
(Kaplan & Saccuzzo, 2005)
- In clinical setting, high reliability is extremely
important. When tests are used to make important
decisions about someone's future, a reliability of .90
to .95 might be good enough
- If the test is unreliable, information obtained with it is
of little or no value.
- Psychometricians: .70
What to do with low reliability?
1. Increase the number of items
2. Factor and item analysis
3. Correction of attenuation
2.2. Validity
Validity, as applied to a test, is a judgment or estimate of
how well a test measures what it purports to measure in
a particular context.
Validity indicated what the test aims or purports to
measure.
It answer the question, “Does the test measure what it
is supposed to measure?”
Face Validity
Face validity is not really a validity at all because it does
not offer evidence to support conclusions drawn from
test score.
Face validity relates more to what a test appears to
measure to the person being tested than to what the test
actually measures.
Face validity only establishes the presentation, physical
appearance of the psychological test (Is the test
presentable to the test takers?)
Three Categories of Validity
Content Validity
 explores the appropriateness of test items of a
psychological test.
 It means that the test covers the content what is
supposed to cover. This describes a judgment of
how adequately a test samples behavior
representative of what the test was designed to
sample.
 test blueprint for the "structure" of the test
evaluation. A plan regarding the types of information
to be covered by the items
Criterion-related Validity
 criterion: a standard on which a judgment or
decision may be based. The standard against which
a test or a test score is evaluated
 refers to the ability to draw accurate inferences from
test scores to a related behavioral criterion of
interest
 the extent to which a measure is related to an
outcome.
- Predictive Validity; Concurrent Validity
Predictive Validity
- how well a certain measure can predict future
behavior
Concurrent Validity
- extent to which test scores may be used to estimate
an individual’s present standing on a criterion
Construct Validity
 judgment about appropriateness of inferences
drawn from test scores regarding individual
standings on a variable called construct.
 it arrived at by executing a comprehensive analysis
of:
a. How scores on the test relate to other test
scores and measures, and
b. How scores on the test can be understood
within some theoretical framework for
understanding the construct that the test was
designed to measure.
Convergent Validity
- when a measure correlates well with other tests
believed to measure the same construct.
- ex: correlate assessment scores for a math ability
test with scores obtained from other math ability
tests.
Discriminant Validity
- a construct measure diverges from other measures
that should be measuring different things.
- ex: correlate assessment scores for a math ability
test with scores obtained from a verbal ability test.
Practicality of a Test
 A test must be usable
 Selection of the test should also be based on:
- Effort
- Affordability
- Time frame
 Test requires simple directions
 Easy administration and scoring