Duality of Error: Public Administration and Policy PAD634 Judgment and Decision Making Behavior

Public Administration and Policy
PAD634 Judgment and Decision Making Behavior
Duality of error
Thomas R. Stewart, Ph.D.
Center for Policy Research
Rockefeller College of Public Affairs and Policy
University at Albany
State University of New York
T.STEWART@ALBANY.EDU
Copyright © Thomas R. Stewart

Example of
a False
Positive
(New York
Times)
duality-of-error.ppt 2
Decision table terminology: Data for an imperfect
categorical forecast over 100 days (uncertainty)
Base rate = 20/100 = .20 Forecast for tomorrow’s

weather
Tomorrow’s No rain for Rain for

actual tomorrow tomorrow
weather (negative (positive
forecast) forecast)
Rain 6 14
(positive) (false (true
negative) positive)
No rain 71 9
(negative) (true (false
negative) positive)
Threshold model
Even if a decision or forecast is categorical, it is based on a

degree of belief or judgment
Threshold 1 Threshold 2
Decision A Decision B Decision C
Low High
Judgment
Uncertainty, Judgment, Decision, Error
• Taylor-Russell diagram
– Decision cutoff
– Criterion cutoff (linked to base rate)
– Correlation (uncertainty)
– Errors
• False positives (false alarms)
• False negatives (misses)
Decision threshold
r = .50
100
Don’t Act
"Truth"
50
Act
0
0 50 100
Judgment
r = .50
100
Action is appropriate
"Truth"
50 Criterion
Action is inappropriate threshold
0
0 50 100
Judgment
Taylor-Russell Decision threshold
diagram r = .50
100
False
negatives True
positives
"Truth"
50 Criterion
threshold
True False
negatives positives
0
0 50 100
Judgment
diagram r = .50
100
False
negatives True
positives
"Truth"
50 Criterion
threshold
True False
negatives positives
0
0 50 100
Judgment
diagram r = .50
100
False
negatives True
positives
"Truth"
50 Criterion
threshold
True False
negatives positives
0
0 50 100
Judgment
diagram r = .95
100
False True
negatives positives
"Truth"
50 Criterion
threshold
True False
negatives positives
0
0 50 100
Judgment
Taylor-Russell diagram
Tradeoff between false positives and false negatives
Uncertainty, Judgment, Decision, Error
• Another view: ROC analysis

– Decision cutoff
– False positive proportion
– True positive proportion
– Az measures forecast quality
ROC Curve
Problem: Optimal decision cutoff
• Given that it is not possible to eliminate both

false positives and false negatives, what
decision cutoff gives the best compromise?
– Depends on values
– Depends on uncertainty
– Depends on base rate
• Decision analysis is one optimization method.

Example: Weather forecaster’s decision to
warn the public about an approaching storm
Decision tree (for tornado warning example)
Expected value
Expected Value = P(O1)V(O1) +

P(O2)V(O2) +
P(O3)V(O3) +
P(O4)V(O4)
P(Oi) is the probability of outcome i
V(Oi) is the value of outcome i
Expected value
• One of many possible decision making

rules
• Used here for illustration because it’s
the basis for decision analysis
• Intended to illustrate principles
Expected value
Assign each point a number representing its value.

The expected value is the average (mean)
of those
values.
Assume all points in

a quadrant have the
same value.
Expected value
False negatives True positives

10
Action is 10
appropriate 10
0 10
0 10
Expected value
80
100 80 = 690/14
100 = 49.3
Action is not 80
100
appropriate
100
True negatives False positives

Don’t Act Act
Expected value
False negatives True positives

10
Action is 10
appropriate 10
10 10
0 10
Expected value
80
80 80 = 680/14
100 = 48.6
Action is not 80
appropriate
100 (.7 less than
100 previously)
True negatives False positives
Don’t Act Act
Where do the values come from?
Values depend on many factors
• Event
– Hail
– Tornado • Location
• Time – Population
– Season – Highways
– Day of week
– Time of day
Descriptions of outcomes
• True positive (hit--a warning is issued and the storm

occurs as predicted)
– Damage occurs, but people have a chance to prepare.
Some property and lives are saved, but probably not all.
• False positive (false alarm--a warning is issued but
no storm occurs)
– No damage or lives lost, but people are concerned and
prepare unnecessarily, incurring psychological and economic
costs. Furthermore, they may not respond to the next
warning.
Descriptions of outcomes (cont.)
• False negative (miss--no warning is issued, but the

storm occurs)
– People do not have time to prepare and property and lives
are lost. NWS is blamed.
• True negative (no warning is issued and storm
occurs)
– No damage or lives lost. No unnecessary concern about the
storm.
Values depend on your perspective
• Forecaster
• Emergency manager
• Public official
• Property owner
• Business owner
• Many others...
Which is the best outcome?
•True positive?
•False positive?
•False negative?
•True negative?
Give the best outcome a value of 100.
Which is the worst outcome?
•True positive?
•False positive?
•False negative?
•True negative?
Give the worst outcome a value of 0.
Rate the remaining two outcomes
•True positive?
•False positive?
•False negative?
•True negative?
Rate them relative to the worst (0) and

the best (100)
Interpreting values
Compare pairs where the weather is the same but
the forecast is different.
Weather is not severe

True negative - False positive = penalty for false alarm
Weather is severe
True positive - False negative = benefit of correct forecast
Interpreting values
Decision
Event Don’t warn Warn
False Negative True positive TP - 0 =
Tornado benefit of
0 TP warning
True negative False positive 100 - FP =

No tornado penalty for
100 FP false alarm
Measuring values
Values reflect different perspectives
Perspective
1 2 3
True positive? 40 90 80
False positive? 50 80 98
False negative? 0 0 0
True negative? 100 100 100
Expected value
Expected Value = P(O1)V(O1) +

P(O2)V(O2) +
P(O3)V(O3) +
P(O4)V(O4)
P(Oi) is the probability of outcome i
V(Oi) is the value of outcome i
Expected value depends on the decision
threshold
85
80
75
70
65
E.V.
60
55
50
45
40
Threshold
Expected value depends on the value
perspective
Value differences can lead to
disagreement and conflict
False negative True positive
Best threshold for

person A
Truth
Best threshold for

person B
True negative False positive
duality-of-error.ppt Judgment 38
Whose values?
• Forecasting weather is a technical

problem.
• Issuing a warning to the public is a
social act.
• Any warning policy has an implicit set of
values.
• Should those values be made explicit
and subject to public scrutiny?
Conclusion
• Choosing the cutoff

– Value tradeoffs are unavoidable.
– Decisions are based on values that should be
critically examined.
Injustice (welfare problem, see Hammond, p. 56)
Injustice to
individuals
The deserving
The Injustice to
undeserving society
Rejected Rewarded
Uncertainty
Judgment
Error
Injustice
Conflict
Implications and predictions
• Implications:
– Tradeoff between different kinds of errors is inevitable.
– Science can’t dictate the correct decision.
– Explicit consideration of values is important, but is rarely done.
– Solving one kind of problem creates another.
– Detecting rare events means many false alarms.
– Small gains in predictability can have large benefits.
• Predictions:
– For rare events, reaction to high visibility false (negative/positive)
is likely to create a large number of less visible false
(positives/negatives).
– Decision thresholds are likely to cycle.
Major points to remember
• Uncertainty creates the need for judgment

• Separate judgmental accuracy from the
decision cutoff
• Uncertainty leads to two kinds of error
–False positive/false negative tradeoff
• Error creates injustice
• Under uncertainty, value differences lead to
conflict
• Reducing uncertainty reduces conflict
Disposition Decisions in
Psychiatric Emergency Rooms
• False negatives: Inappropriate releases
– Occasionally lead to violence against others
– Increase the risk of suicide
– Increase the risk of injury or death due to
accidents
– Place stress and extra burdens on community
support systems
– Aggravate psychiatric symptoms
– Patient does not obtain proper treatment
• False positives: Inappropriate
admissions
– Can be disruptive and stigmatizing
– May lead to the loss of jobs, housing, and
child custody
– Average inpatient admission costs nearly
$10,000
• Taylor-Russell analysis
– Base rate
– Selection rate
– Judgmental accuracy
– Costs and benefits of outcomes
No policies regarding psychiatric emergency room
admissions can be meaningfully evaluated without
simultaneously considering all four factors.
Unfortunately, few public policy discussions discuss
all four factors. This means that implicit assumptions
about omitted factors have been made. These buried
assumptions may give rise to debates and disputes
that will be difficult to resolve, unless they are brought
to the surface and explicated.
Psychiatric ERs
Base rate
• What percentage of persons who

present at psychiatric ERs would benefit
from in-patient treatment and, thus,
"ought" to be admitted?
– Difficult to determine
– No “gold standard”
– Initial assumption: 50%
– Requires sensitivity analysis
Psychiatric ERs
Selection rate
• Varies substantially across sites

• Initial assumption: 50%
• Approximates the average rate found in
research to date
Psychiatric ERs
Judgmental accuracy
• No data due to absence of a “gold

standard”
• Study by Bruce Way found that the
correlation among psychiatrists
recommended dispositions was .34.
• If this is an estimate of reliability, then
accuracy can be no higher than the
square root of .34 = .58
Psychiatric ERs
Cost and benefits of outcomes
Rather than trying to develop monetary

estimates, the present analysis relies on
a decision analytic approach, in which
each possible outcome is assigned a
score from 0 to 100, reflecting its
relative desirability.
Psychiatric ERs
Which is the best outcome?
•True positive?
•False positive?
•False negative?
•True negative?
Give the best outcome a value of 100.
Psychiatric ERs
Which is the worst outcome?
•True positive?
•False positive?
•False negative?
•True negative?
Give the worst outcome a value of 0.
Psychiatric ERs
Rate the remaining two outcomes
•True positive?
•False positive?
•False negative?
•True negative?
Rate them relative to the worst (0) and

the best (100)
Psychiatric ERs
Value perspectives
Perspective
1 2 3
True positive 100 100 67
False positive 33 50 33
True negative 67 75 100
False negative 0 0 0
Psychiatric ERs
Taylor-Russell analysis
If the assumptions regarding the underlying base rate, the

payoff function, and the degree of predictive accuracy
are approximately correct, is the admission rate of 50%
optimal, in terms of maximizing total value? In light of
the substantial variation across institutions in observed
admission rates – from less than 10% to more than 90%
– this is an extremely pertinent question, with
substantial potential policy implications. Left to their
own devices, different institutions have come up with
quite different answers about what percentage of
potential patients is appropriate to admit.
Psychiatric ERs
Taylor-Russell analysis
• Injustice
– To individuals
– To society
• Cycles of differential injustice?
• Optimal cutoff and admission rate
• Sensitivity to base rate
• Improving judgmental accuracy
Psychiatric ERs
Rationing or quotas
• What happens if there are only a limited

number of beds to be filled?
• The cutoff is determined by the number
of beds available.
• Resource constraints dictate the value
tradeoffs
Left out of Taylor-Russell
• Creating new alternatives that may eliminate some of the tough
tradeoffs.
• Design and planning
• Dynamic properties of decision or environments
• The potential effects of testing and cutoffs and standards on the
points in the graphs (e.g., measures designed to increase airline
security have a deterrent effect. Also, potential terrorists develop
countermeasures)
• Implementation issues
• Cost of decision processes
• Amount of information -- how much is enough?
• Cue intercorrelations, causal texture
• Outcomes in the same quadrant may have different values
• Multidimensional nature of outcomes

Duality of Error: Public Administration and Policy PAD634 Judgment and Decision Making Behavior

Uploaded by

Copyright:

Available Formats

You might also like

Duality of Error: Public Administration and Policy PAD634 Judgment and Decision Making Behavior

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Duality of Error: Public Administration and Policy PAD634 Judgment and Decision Making Behavior

Uploaded by

Copyright:

Available Formats

Public Administration and Policy

PAD634 Judgment and Decision Making Behavior

Copyright © Thomas R. Stewart

Base rate = 20/100 = .20 Forecast for tomorrow’s

Tomorrow’s No rain for Rain for

Even if a decision or forecast is categorical, it is based on a

Decision A Decision B Decision C

Action is inappropriate threshold

• Another view: ROC analysis

• Given that it is not possible to eliminate both

• Decision analysis is one optimization method.

Expected Value = P(O1)V(O1) +

• One of many possible decision making

Assign each point a number representing its value.

Assume all points in

False negatives True positives

True negatives False positives

False negatives True positives

• True positive (hit--a warning is issued and the storm

• False negative (miss--no warning is issued, but the

Give the best outcome a value of 100.

Give the worst outcome a value of 0.

Rate them relative to the worst (0) and

Weather is not severe

True negative False positive 100 - FP =

Values reflect different perspectives

True negative? 100 100 100

Expected Value = P(O1)V(O1) +

Best threshold for

Best threshold for

True negative False positive

• Forecasting weather is a technical

• Choosing the cutoff

• Uncertainty creates the need for judgment

• What percentage of persons who

• Varies substantially across sites

• No data due to absence of a “gold

Cost and benefits of outcomes

Rather than trying to develop monetary

Which is the best outcome?

Give the best outcome a value of 100.

Which is the worst outcome?

Give the worst outcome a value of 0.

Rate the remaining two outcomes

Rate them relative to the worst (0) and

True negative 67 75 100

If the assumptions regarding the underlying base rate, the

• What happens if there are only a limited

You might also like