Dunn William

Public Policy Analyasis: an introduction

Open Access Resource

Public policy analysis : an introduction : Dunn, William N : Free Download, Borrow, and
Streaming : Internet Archive

Selective Texts

Techniques for monitoring (p. 302)

Monitoring, unlike other policy methods, does not involve procedures that are clearly associated
with alternative approaches. Many of the same techniques are therefore appropriate for each of
the four approaches: social systems accounting, social auditing, social experimentation, and social
research cumulation

Graphic displays

Much information about policy outcomes is presented in the form of graphic displays,
which are pictorial representations of the values of one or more action or outcome variables. A
graphic display can be used to depict a single variable at one or more points in time, or to
summarize the relationship between two variables. In each case a graph displays o set of data
points, each of which is defined by the coordinates of two numerical scales. The horizontal scale
of a graph is called the abscissa and the vertical scale is called the ordinate. When a graph is used
to display a causal relationship, the horizontal axis is reserved for the independent variable (X)
and is called the X-axis, while the vertical axis is used to display the dependent variable (Y) and
called the Y axis.

One of the most simple and useful kinds of graph is the time-series graph, which displays the
outcome variable on the vertical axis and time on the horizontal axis.

Graphs may also be used to depict relationships between two or more variables at one point in
time. Graphs of this kind, called scatter diagrams, were discussed in Ch. 4…..

If one of the variables precedes the other in time… or if there is a persuasive theory that explains
the correlated variables, then we may be warranted in claiming that there is a causal relationship
between the variables. Otherwise, variables are simply correlated - ….- and one variable cannot
be assumed to be the cause of the other.
A common problem in using graphs is spurious interpretation, a situation where two variables
that appear to be correlated are actually each correlated with some other variable.

Another means for displaying data on policy outcomes is the bar graph, a pictorial representation
that presents data in the form of separated parallel bars arranged along the horizontal (or vertical)
axis.

A histogram is a kind of bar graph that organizes information about a grouped frequency
distribution of an action or outcome variable at one point in time. In a histogram the width of the
bars along the horizontal axis is equal, and there is no space between the bars.. The height of the
bars in a histogram shows the frequency of occurrence in each group (called a class interval.)

Another way to display information about policy outcomes is the cumulative frequency
polygon, a graph that shows the cumulative frequencies of a distribution along the vertical
axis. As one moves from left to right along the horizontal axis, the frequency of persons …
in the first group is plotted on the vertical axis; the frequency of persons in the first and
second groups is then plotted; and so on until we reach the end of the horizontal scale,
which is the sum of all frequencies.

In monitoring the effects of policies designed to alleviate poverty, a useful type of

cumulative frequency polygon is the Lorenz curve, which can be used to display the
distribution of income, population, or residential segregation in a given population. For
example, a Lorenz curve enables us to compare the share of total income earned by each
successive percentage group in the population. These percentage groups are called quintiles
or deciles, which are groups comprised of one-fifth or one-tenth of the population
respectively.

The Gini index

Lorenz curves may also be used to display the distribution of population or certain types of
activities.

Lorenz curves can also be expressed in the form of the Gini concentration ratio (sometimes called
simply the Gini index) which measures the proportion of the total area under the diagonal that lies
in the area between the diagonal and the Lorenz curve. The formula for calculating this
proportion is

The Gini index shows the proportion of the area under the diagonal that lies between the diagonal
and the Lorenz curve.

It can be used to depict the concentration of wealth, income, population, ethnic groups, urban
residents, crime and other social conditions.

The advantage of Gini index is that it is a precise measure of concentration that supplies much
more information than the pictorial representation provided by the Lorenz curve.

Further, the Gini index ranges from zero (no concentration at all) to 1.0 (maximum concentration)

Tabular displays

Another useful way to monitor policy outcomes is to construct tabular displays. A tabular display
is a rectangular array used to summarize the key features of one or more variables.

The simplest form of tabular display is the one-dimensional table which presents information
about policy outcomes in terms of a single dimension of interest, for example, age, income,
region, or time.

Information may also be arranged in table with two dimensions, for example, levels of education
by income, target groups by level of education, or the income of target groups by time periods.

Another type of two-dimensional table involves the analysis of two or more groups by levels of
some outcome variable, which might be employment, services received, or earnings. For
example, social experiments are based on comparisons og groups that have received a particular
type of treatment (experimental group) and groups which have not (control group).

In monitoring policy outcomes data may also be arranged in three-dimensional tables.

Index numbers

A useful way to monitor changes in outcome variables over time is to construct index numbers.

Index numbers are measure of how much the value of an indicator or set of indicators changes
over time relative to a base period. Base periods are arbitrarily defined as having a value of 100,
which serves as the standard for comparing subsequent changes in the indicators of interest.

Many index numbers are used in public policy analysis. These include index numbers used to
monitor changes in consumer prices, industrial production, crime severity, pollution, health care,
quality of life, and other important policy outcomes.

Index numbers differ in their focus, complexity, and degree of explicitness.

Index numbers may focus on changes in prices, quantities or values. ….. Whether they focus on
price, quantity, or value, index numbers may be simple or composite.

Simple index numbers are those that have only one indicator (e.g., the quantity of crimes per
100,000 persons), while composite index numbers include many different indicators…..

Index numbers may be implicitily weighted or explicitly weighted.

In the former case indicators are combined with no explicit procedure for establishing their value.

There are two general procedures for constructing index numbers: aggregation and averaging.

Aggregative indices are constructed by summarizing the values of indicators … for given periods.
Averaging procedures (or the so-called average of relatives method) require the calculation of
average changes in the value of an indicator over time, while aggregative procedures do not.

A useful and simple aggregative price index is the purchasing power index, which measures the
real value of earnings in successive periods.

A purchasing power index may be used to monitor the impact of wage agreements on the real
income of employees as part of public sector collective bargaining activities.
A purchasing power index is constructing by using values of the Consumer Price Index. This is
done by locating the year for which we wish to determine the purchasing power of wages and
converting this value into a price relative, that is, a value that expresses the price of goods and
services in the year of interest relative to a base year (e.g. 1967). If we then divide this price
relative into one (this is called a reciprocal), we will obtain the purchasing power of wages in
1967.

The weakness of this implicitly weighted aggregative-quantity index of pollution is that it does
not take into account variations in the concentrations of pollutants or their relative damage to
health.

Dunn., p. p. 317

Index numbers have similar limitations.

First, explicit weighting procedures often lack precision.

Second, it is difficult to obtain sample data for indexes that are equally meaningful for all groups
in society.

Finally, index numbers do not always reflect qualitative changes in the meaning of significance of
index items over time.

Index numbers may be used to monitor a wide variety of changes. Yet these techniques provide
no systematic way of relating chanhes in policy outcomes to prior policy actions.

We now consider three sets of techniques that permit analysts to make systematic judgments
about the effects of policy actions on policy outcomes: interrupted time-series analyses, control-
series analyses, and regression-discontinuity analysis.

Thes procedures are used in conjunction with graphic displays and based on correlation and
regression techniques discussed in the chapter of forecasting.

Although two of these procedures (interrupted time-series and control-series analysis) are equally
applicable to social systems accounting, social auditing, and social experimentation, one of them
(regression discontinuity analysis) is exclusively applicable to social experiments.
Interrupted time-series analyses is a set of procedures for displaying in graphic and statistical
form the effects of policy actions on policy outcomes. Interrupted series analysis is appropriate
for problems where an agency initiates some action that is put into effect across an entire
jurisdiction or target group, for example, in a particular state or among all families below the
poverty threshold. Because policy actions are limited to persons in the jurisdiction or target
group, there is no opportunity for comparing policy outcomes across different jurisdictions or
among different categories of the target group. In this situation the only basis of comparison is the
record of outcomes for previous years.

Interrupted time-series analysis is particularly appropriate for those approaches to social

experimentation that are called “cvasi-experimental” because they lack one or more
characteristics of classical experiments (i. e. random selection of participants, random assignment
to experimental and control groups, and measurement before and after the experimental
treatment)

The main advantage of interrupted time-series analysis is that it enables us to consider

systematically various threats to the validity of causal inferences about policy outcomes.

The importance of these threats to validity can be easily seen when we compare policy outcomes

Dunn., p. 322

Control series Analyses involves the addition of one or more control groups to an interrupted
time-series design in order to determine whether characteristics of different groups exert an
independent effect on policy outcomes, apart from the original policy action.

The logic of control-series analysis is the same as interrupted time-series analyses. The only
difference is that a group or groups that were not exposed to the policy actions in question are
added to the graph.

Control-series analysis help further to scrutinize the validity of claims about the effects of policy
actions on policy outcomes.

Dunn, p. 358
Type of Question Illustrative
criterion criteria
Effectiveness Has a Units of
valued service
outcome
been
achieved?
Efficiency How Unit cost
much Net
effort benefits
was Cost-
required benefit
to ratio
achieve a
valued
outcome?

Dunn, p. 361

Varieties of Formal Evaluation

Developmental evaluation refers to evaluation activities that are explicitly designed to serve the
day-to-day needs of program staff. Developmental evaluation is useful for alerting the staff to
incipient weaknesses or unintended failures of a program and for insuring proper operation by
those responsabile for its operation.

Retrospective process evaluation involves the monitoring and evaluation of programs after they
have been in place for some time. …. does not permit the direct manipulation of inputs (e.g..,
expenditures) and processes. Rather it relies on expost facto (retrospective) descriptions of
ongoing program activities, which are subsequently related to outputs and impacts.

Retrospective process evaluation requires a well-established internal reporting system which

permits the continuous generation of program-related information…

Management information in public agencies sometimes permit retrospective process evaluations,

provided they contain information on processes as well as outcomes.

Experimental evaluation involves the monitoring and evaluation of outcomes under conditions of
direct controls over policy inputs and processes. The ideal of experimental evaluation has
generally been the controlled scientific experiment, where all factors which might influence
policy outcomes except one – that is, a particular input or process variable are controlled, held
constant, or treated as plausible rival hypothese. ………………
 Experimental evaluations must meet rather severe requirements before they can be carried out:

(1) a clearly defined and directly manipulable set of “treatment” variables which are specified in
operational terms<

(2) an evaluation strategy which permits maximum generalizability of conclusions about

performance to many similar target groups or settings (external validity);

(3) An evaluation strategy which permits minimum error in interpreting policy performance as the
actual result of manipulated policy inputs and processes (internal validity);

(4) A monitoring system which produces reliable data on complex interrelationships among
preconditions, unforseen events, inputs, processes, outputs, impacts, and side effects and
spillovers.
Since these demanding methodological requirements are rarely met, experimental evaluations
typically fall short of the true controlled experiment, and are referred to as „quasi-experimental”.
(5) Retrospective outcomes evaluations also involve the monitoring and evaluation of outcomes but
with no direct control over manipulable policy inputs and processes. At best controls are indirect
or statistical – that is, the evaluator attemps to isolate the effects of many different factors by
using quantitative methods.
(6) In general, there re two main variants of retrospective process evaluation: cross-sctional and
longitudinal studies. Longitudinal studies are those evaluate changes in the outcomes of one,
several, or many programs at two or more points in time. Many longitudinal studies have been
carried out in the area of family planning, where fertility rates and changes in the acceptance of
contraceptive devices are monitored and evaluated over reasonably long periods of time (five to
twenty years).
Cross-sectional studies, by contrast, seek to monitor and evaluate multiple programs at one point
in time. The goal of cross-sectional study is to discover whether the outputs and impacts of
various programs are significantly different from one another, and if so, what particular actions,
preconditions, or unforeseen events might explain the difference.
(7) Decision-theoretic evaluation is an approach which uses descriptive methods to produce reliable
and valid information about policy outcomes that are explicitly valued by multiple stakeholders.
(8) The key difference between decision-theoretic avaluation, on one hand, and pseudo- and formal
evaluation on the other, is that the decision-theoretic evaluation attempts to surface and make
explicit the latent as well as manifest goals and objectives of stakeholders. This means that
formally announced goals and objectives …. Are but one source of value).

Dunn. p. 363

Decision-theoretic evaluation is a way to overcome several deficiencies of pseudo-evaluation and

formal evaluation:
1. Underutilization and nonutilization of performance information. Much of the information
generated through evaluations is underutilized or never used to improve policy making. In part,
this is because evaluations are not sufficiently responsive to the goals and objectives who have a
stake in formulating and implementing policies and programs.
2. Ambiguity of performance goals. Many goals of public policies and programs are vague. This
means that the same general goal – for example, to improve health or encourage better
conservation of energy – can and do result in specific objectives that conflict wth one other. This
is evident when we consider that the same goal may be operationalized in terms of at least sic
types of evaluation criteria: effectiveness, efficiency, adequacy, equity, responsiveness, and
appropriateness. One of the purposes of decision-theorethic evaluation is to reduce the ambiguity
of goals and make conflicting objectives explicit.
3. Multiple conficting objectives. The goals and objectives of public policies and programs cannot
be satisfactory established by focusing on the values of one or several parties. In fact, multiple
stakeholders with conflicting goals and objectives are present in most situations requiring
evaluation

One of the main purposes of decision-theoretic evaluation is to link information about policy
outcomes with the values of multiple stakeholders. …. The two major forms of decision theoretic
evaluation are evaluability assessment and multiattribute utility analysis, both of which attempt to
link information about policy outcomes.

Dunn, p. 364

Evaluability assessment is a set of procedures designed to analyse the decision-making system

that is supposed to benefit from performance information and to clarify the goals, objectives, and
assumptions against which performance is to be measured. The basic question in evaluability
assessment is whether a policy or program can be evaluated at all. For a policy or program to be
evaluable, at least three conditions must be present: a clearly articulated policy or program;
clearly specified goals and/or consequencies; and a set of explicit assumptions that link policy
actions to goals and/or consequencies.

Dunn, p. 365

Multiattribute utility analysis is a set of procedures designed to elicit from multiple stakeholders
subjective judgments about the probabibilty of occurrence and value of policy outcomes.

The steps in conducting a multiattribute utility analysis are the following:

Stakeholders’ identification. Identify the parties who affect and are affected by a policy or
program. Each of these stakeholders will have goals and objectives that they wish to maximize.

Specification of relevant decision issues. Specify the courses of action or inaction about which
there is a disagreement among stakeholders. In the simplest case there will be two courses of
action: the status quo and some new iniative.
 Specification of policy outcomes. Specify the range of consequences that may follow each course
of action. Outcomes may be arranged in a hierarchy where one action has several consequencies,
each of which itself has further consequences. A hierarchy of outcomes is similar to an objective
tree, except that outcomes are not objectives until they have been explicitly valued.

Identification of attributes of outcomes. Here the task is to identify all relevant attributes that
make outcomes worthy or valuable. For example, each outcome may have different types of
benefits and costs to different target groups and beneficiaries.

Attribute rankings. Rank each value attribute in order of importance. …

Attribute scaling. Scale attributes that have been ranked in order of importance. To do so,
arbitrarily assign the least important attribute a value of ten. Proceed to the next most important
attribute, answering the question: how many timesmoreimportant is this attribute than the next-
least-important one? Continue this scaling procedure untilthe most important attribute has been
compared with all others. Note that the most important attribute may have a scale value 10, 20,
30, or more times that of the least important attribute.

Scale standardization. The attributes that have been scaled will have different maximum values
for different stakeholders…..sum all original values for each scale, divide each original value by
its respective sum, and multiply by 100. This results in separate scales whose component values
sum to 100.

Outcome measurement. Measure the degree that each outcome is likely to result in the attainment
of each attribute. The maximum probability should be given a value of 100; the minimum
probability should be given a value of 0.

Utility calculation. Calculate the utility (value) of each outcome by using the formula:

U= Σ wu

Where Ui = the aggregate utility (value) of the ith outcome

Wi= the standardized value of the ith attribute

Ui=the probability of occurrence of the ith outcome of the jth attribute.

1. Evaluation and presentation Specify the policy outcome with the greatest overall performance,
and present the information to relevant decision makers.

The strength of multiattribute utility analysis is that it enables analysts to deal systematically with
conflicting objectives of multiple stakeholders. This is possible, however, only when the steps
just described are carried out as part of a group process involving relevant stakeholders. Hence,
the essential requirement of multi-attribute utility analysis is that stakeholders who affect and are
affected by a policy or program are active participants in the evaluation of policy performance.