10/12/13

Three segments
•  Reliability
Statistics One •  Validity
•  Sampling
Lecture 6
Measurement

1 2

Reliability
•  Important concepts & topics
–  Classical test theory
Lecture 6 ~ Segment 1 –  Reliability estimates
Reliability

3 4

1

 10/12/13

Reliability Reliability
•  Classical test theory •  Classical test theory
–  Raw scores (X) are not perfect –  In a perfect world, it would be possible to
–  They are influenced by bias and chance error obtain a “true score” rather than a “raw
•  For example, measurement of body temperature score” (X)
•  X = true score + bias + error

•  This is also known as “true score theory”

5 6

Reliability Reliability
•  A measure (X) is considered to be reliable •  Methods to estimate reliability
as it approaches the true score –  Test / re-test
–  The problem is we don’t know the true score –  Parallel tests
–  So, we estimate reliability –  Inter-item estimates

7 8

2

 10/12/13

Reliability Body temperature

•  Example: Body temperature
–  Orally
–  Internally
–  Infrared thermometer: “The wand”

9 10

Body temperature F° Body temperature C°

11 12

3

 10/12/13

Body temperature F°: Biased Body temperature C°: Biased

13 14

Reliability Reliability
•  Test / re-test •  Test / re-test
–  Measure everyone twice –  The correlation between X1 and X2 is an
•  X1 estimate of reliability
•  X2 •  However, if the bias is uniform then we won’t detect
it with the test / re-test method

15 16

4

 10/12/13

Reliability
•  Parallel tests
–  Measure body temperature with the wand
(X1) and with an oral thermometer (X2)
–  The correlation between X1 and X2 is an
estimate of reliability
•  AND, now the bias of the wand will be revealed
Reliability
•  Inter-item
–  Inter-item is the most commonly used method
in the social sciences
•  Test / re-test and parallel tests are time consuming
•  Inter-item is therefore more cost efficient

17 18

Reliability Segment summary

•  Inter-item •  Classical test theory (true score theory)
–  For example, suppose a 20-item survey is •  Reliability estimates
designed to measure extraversion –  Test / re-test
•  Randomly select 10 items to get sub-set A (X1) –  Parallel tests
•  The other 10 items become sub-set B (X2)
–  Inter-item estimates
•  The correlation between X1 and X2 is an estimate
of reliability

19 20

5

 10/12/13

END SEGMENT Lecture 6 ~ Segment 2

Validity

21 22

Validity Validity
•  What is a construct? •  What is a construct?
–  How to operationalize a construct –  An ideal “object” that is not directly observable
–  Construct validity •  As opposed to “real” observable objects
•  Content validity
•  Convergent validity •  For example, “intelligence” is a construct
•  Divergent validity
•  Nomological validity

23 24

6

 10/12/13

Validity Validity
•  How do we operationalize a construct? •  Construct validity
–  The process of defining a construct to make it –  Content validity
observable and quantifiable –  Convergent validity
•  For example, intelligence tests –  Divergent validity
–  Nomological validity

25 26

Validity Validity
•  An example: •  How to operationalize?
–  Construct: Verbal ability in children –  A vocabulary test

27 28

7

 10/12/13

Validity Validity
•  Construct validity •  Construct validity
–  Content validity –  Convergent validity
•  Does the test consist of words that children in the •  Does the test correlate with other, established
population and sample should know? measures of verbal ability?
–  For example, reading comprehension

29 30

Validity Validity
•  Construct validity •  Construct validity
–  Divergent validity –  Nomological validity
•  Does the test correlate less well with measures •  Are scores on the test consistent with more general
designed to test a different type of ability? theories, for example, of child development and
–  For example, spatial ability neuroscience
–  For example, a child with neural damage or disease to
brain regions associated with language development
should score lower on the test

31 32

8

 10/12/13

Reliability & Validity: Review

•  Important concepts & topics
–  Classical test theory
–  Reliability estimates
END SEGMENT
–  Construct validity

33 34

Sampling
•  Important concepts & topics
–  Random and representative sampling
Lecture 6 ~ Segment 3 –  Sampling error
Sampling –  Standard error

35 36

9

 10/12/13

Sampling Illustration
•  Random and representative
•  Recall the color wheel from Lecture 1

37 38

Random Not random

39 40

10

 10/12/13

Sampling error Sampling error

•  Sampling error: The difference between •  PROBLEM!
the population and the sample –  We typically don’t know the population
–  Notice that even the “random” histogram is parameters
not “perfectly” random –  So, how do we estimate sampling error?
–  There is some fluctuation due to sampling
error

41 42

Sampling error Sampling error

•  Sampling error mainly depends on the size of •  Assume 6 samples from a normal
the sample, relative to the size of the population
population •  N = 10
–  As sample size increases, sampling error •  N = 20
decreases •  N = 50
•  N = 100
•  N = 200
•  It also depends on the variance in the •  N = 1000
population 43 44

–  As variance increases, sampling error increases

11

 10/12/13

Sampling error Sampling error

45 46

Sampling error Standard error

•  Sampling error is estimated from the size •  Standard error is an estimate of amount of
of the sample and the variance in the sampling error
sample •  SE = SD / SQRT(N)
–  Under the assumption that the sample is
•  SE: Standard error
random and representative of the population •  SD: Standard deviation of the sample
•  N: Size of the sample

47 48

12

 10/12/13

Segment Summary
•  Important concepts & topics
–  Random and representative sampling
–  Sampling error
END SEGMENT
–  Standard error

49 50

END LECTURE 6

51

13