Supplement 1: Measuring Outcomes and Impacts

Patterns of Causality in Time Series

16 14 Probable and 12 10 Durable Probable and

Policy Intervention
8 6 4

Non-Durable Improbable Regression to Mean Improbable Constant Change

2 0 1 2 3 4 5 6 7 8 9

Improbale Non-Linear Change

Sequence number

Conditions Required to Make Causal Inferences

Condition X precedes condition Y in time X O

Condition X is correlated with condition Y rx.y > 0 (+/-) Conditions other than X do not affect condition Y r x.y = rx.y|z

Research Designs Help Make Plausible Causal Inferences

I II III IV V VI VII R R R R R R Yt Yb Yb X X ~X X ~X X ~ Xi Ya Ya Ya Ya Ya YT YT,C

Strengths of Quasi-Experimental Designs

Activity theory of causation Recognition of systemic complexity Financial, political, and ethical feasibility Rarity of true experiments Availability of resources (www.economagic.com, www.fedstats.gov, www.census.gov, www.eurostat, Statisticki Godisnjak/ Bilten)

Extended Time Series

I O1 O2 O3 O4 O5 O6 O7

Interrupted Time Series

I O1 O2 O3 X O4 O5 O6 O7

Control Series
I II O1 O1 O2 O2 O3 O3 X ~X O4 O4 O5 O5 O6 O6 O7 O7

Problems with Interrupted Time Series

Incremental diffusion of programs with no sharp cutting points

Multiple programs operating at same time

Lack of detailed knowledge of program activities Insufficient observations in time series Unknown time intervals due to delays in implementing programs

Multiple rival explanations of outcomes

Interrupted Time-Series Analysis Helps Detect Causality

16 14 Probable and 12 10 Durable Probable and

Policy Intervention
8 6 4

Non-Durable Improbable Regression to Mean Improbable Constant Change

2 0 1 2 3 4 5 6 7 8 9

Improbale Non-Linear Change

Sequence number

Some Outcome Indicators

HousingArea per person (square meters) Average Life Expectancy Quality Adjusted Life Years Persons Below Poverty Line Income Distribution (Gini Index) Air Pollution Index (parts per million) Lead Concentration Index (blood concentration)

Persons in Mental Hospitals

Average Test Scores Sales or Market Share Votes Cast for Candidates Foreign Direct Investment in MKD Number of newly licensed foreign companies

The Odds Ratio Measures Effect Size

Example--It is believed that more highly educated voters tend to vote for Democratic candidates in the U.S. Here is a sample of voters who voted in the 1992 Presidential Election. How would a policy of producing more Masters and Ph.D.graduates affect the outcome of elections?
Clinton Less than Masters
P

Bush and Perot 857 (0.52)

1-P

797 (0.48)

1,654 (1.0)
P / 1-P = 0.92

Masters or Ph.D. Degree

Q

111 (0.58)
1-Q

82 (0.42)

193 (1.0)
Q / 1-Q = 1.38

ODDS RATIO = 1.38 / 0.92 = 1.5

The Standardized Mean Difference Measures Effect Size

ExampleBetween 1987 ands 1989 the maximum speed limit in 40 of the 50 states of the U.S. was increased from 55mph to 65mph. The paired t-test, which involves a change in means from t0 to t+1 (Note: Observations in any time series are not independent), was used to test the null hypothesis that there is no statistically significant difference (p = 0.05) between traffic fatalities before (1987) and after (1989) the speed limit was raised to 65 mph in 40 states. The speed limit was kept at 55 mph for 10 states. What does the following test show about the effects of removing the old (55mph) policy?

texp = mean fatality rate after the policy - mean fatality rate before the policy / pooled standard deviation = -0.23 / 0.35 = -0.66 tcon = mean fatality rate after the policy mean fatality rate before the policy / pooled standard deviation = -0.07 / 0.10 = -0.70

Guidelines for Interpreting Standardized Effect Sizes

0.80-0.99 strong 0.60-0.79 moderate to strong 0.40-0.59 moderate 0.20-0.39 weak to moderate 0.00-0.19 negligible to weak

NOTE: The practical significance of an effect size depends on the social costs of being wrong.

Other Measures of Effect Size

Identical Units of Measure. Benefits and costs in constant value of a currency, unemployment rates, percent of budget variance, performance appraisal scale. Established Norms. Dietary intake of vitamins compared with minimum (RDA) required daily amount, international test scores, percent above poverty line, percent below a living wage. Average Effect Sizes. Average correlations in political science and sociology range from r = 0.20 to r = 0.30. Average internal consistency reliabilities for mental health inventories, placement examinations, and other instruments involving high risk of being wrong are r > 0.95. Coefficient of Variation. The standard deviation divided by the mean (CV = s / m) . This is the percent variability divided by the mean. A standard deviation, s, of 16 with a mean of 100 is the same as a standard deviation, s, of 96 with a mean of 600. The variability of large municipal budgets can be compared with smaller ones.

Pooled t-Test. The outcome mean after the intervention subtracted from the outcome mean before the intervention, divided by the pooled standard deviation. NOTE: The observations before and after the intervention are not independent and therefore the pooed t-test must be used. x2 x1 / sqrt [Sp (1/n1) + (1/n2)] Standard (z) Scores. An individual score subtracted from the mean of the distribution divided by the standard deviation. z = x - m / s . An individual score of 116 from a distribution with a mean of 100 and a standard deviation of 16 is the same as an individual score of 348 from a distribution with a mean of 300 and a standard deviation of 48. Individual scores measured with two different scales, or from two different distributions, can be directly compared.

Interrupted Time-Series With Two Observations

Fig. 8.1. Connecticut traffic fatalities, 1955-56
330

320

FATALITIES

310

300

290

280 1954

1955

1956

1957

YEAR

Interrupted Time-Series With Three Observations

56000 56000 54000 54000

52000

52000

FATALITIES

50000

50000

48000

48000

46000

46000

44000 42000 1971 1972 1973 1974

44000 42000 1975

YEAR

Extended Time-Series With Interruption

Fig. 8.2. Connecticut traffic fatalities, 1951-59
340 320

FATALITIES

300

280

260

240 220 . 1951 1952 1953 1954 1955 1956 1957 1958 1959 .

YEAR

Extended Time-Series With Interruption

600 00

FATALITIES

500 00

400 00

300 00 196 6 197 0 197 4 197 8 198 2 198 6 199 0 199 4 199 8 200 0 196 8 197 2 197 6 198 0 198 4 198 8 199 2 199 6

YEAR

Control Series With Interruption

Fig. 8.3. Connecticut and control states traffic fatalities, 1951-59
340

320

300

FATALITIES

280

260

240

Control States Connecticut

. 1951 1952 1953 1954 1955 1956 1957 1958 1959 .

220

YEAR

Changes in Fatalities Per Mile Correlated with Economic Factors

.2 .1

0.0

-.1 Fatalities -.2 1968 1972 1976 1980 1984 1988 1992 1996 2000 1970 1974 1978 1982 1986 1990 1994 1998 Economic Index

YEAR
Transforms: natural log, difference (1)

Control Series With Interruption: Fatality Rates in Europe and the US

28 26

24

22

20 US 18 EURCON 16 70 71 72 73 74 75 76 EUREXP

YEAR

Annual Changes in Fatality Rate and Miles Driven, 1913-2000

.4 .2

0.0

-.2

Billion Miles Driven -.4 Traffic Fatalities

19

96 19 0 9 19 4 8 19 8 7 19 2 7 19 6 6 19 0 6 19 4 5 19 8 4 19 2 4 19 6 3 19 0 3 19 4 2 19 8 1 19 2 1 19 6 0

Year

Group Problem
Examine the extended time-series graphs showing the observed fatality rate, the predicted fatality rate, and European Commission target for 2010.
1. Explain how interrupted time-series analysis might result in a different predicted fatality rate. Is the observed fatality rate a valid predictor of fatalities in future?
2. Explain how control-series analysis might change the Commissions 2010 target fatality rate. Is the target realistic?

ForecastEU Fatality Rate by 2010

f atalities per billion veh-km

40 30 20 10 0 1980

1985

1990 fatality rate

1995 model

2000

2005

2010

ETSC forecast

European Commission Proposed Target: 50% Reduction Between 2000 and 2010
80000 70000 60000
Fatalities per year

50000 40000 30000 20000 10000 0 1970

1975

1980

1985

1990

1995

2000

2005

2010