Policy Studies Review, Summer 1988
Vol. 7, No. 4, pp. 720-737
METHODS OF THE SECOND TYPE: COPING WITH THE
WILDERNESS OF CONVENTIONAL POLICY ANALYSIS
William N. Dunn
INTRODUCTION
In the past twenty years we have made considerable progress in develop-
ing new and more appropriate theories, models, and methods for public
policy analysis. Policy analysis has moved well beyond descriptive theories
of the policy-making process into realms of normative decision theory such
as cost-benefit analysis, input-output analysis, linear programming, delphi
technique and a host of additional methodologies that help fashion poten-
tial solutions to public policy problems. Indeed, it is no exaggeration to say
that policy analysis is among the most powerful problem-solving
methodologies available to the applied political scientist.
Despite these notable achievements, the field of public policy analysis
has not acknowledged and forthrightly faced a major methodological
problem: Methods employed by most policy analysts are not appropriate for
structuring policy problems as a prelude to their possible solution. This
methodological deficit is often fatal, judging by the large number of sys-
tematic studies, case materials, and records of experience showing that
policy analysts fail more often because they formulate the wrong problem
than because they choose the wrong solution (see, e.g., Ackoff, 1974b;
Fischhoff, 1977, 1986; Hogwood & Peters, 1985; Mitroff, Mason & Barabba,
1983; Sieber, 1987). To paraphrase John Tukey, we sorely lack methods
that yield "an approximate answer to the right question, which is often
vague, [rather] than an exact answer to the wrong question, which can al-
ways be made precise” (quoted in Rose, 1977, p.23).
This methodological deficit creates enormous difficulties for practicing
policy analysts. In the absence of appropriate problem-structuring
methods, how can we expect to formulate problems that encompass the
proper elements, for example, the proper policy objectives, alternatives,
and expected outcomes? Given a particular problem formulation, how do
we know whether all important elements have been included in the set? In
short, how do we know when we have formulated an approximate solution
to the right problem, as distinguished from an exact solution to the wrong
problem?
In addressing these questions, policy analysts face a situation similar to
that of the homesteaders described by Kline (1981). The homesteaders,
while clearing their land, are aware that enemies lurk in the wilderness,
which lies beyond the clearing. To increase their security, the homes-
teaders, clear a larger and larger area, but never feel completely safe. They
frequently must decide whether to clear more land or attend to their crops
and domesticated animals within the perimeter. The homesteaders do
Earlier versions of this paper benefited from comments and criticisms offered by Ralph
Banks, Eugene Bazan, Ronald Brunner, Anthony Cahill, Bahaman Fozouni, Burkhart
Holzner, Stephen Linder, Iran Mitroff, Guy Peters, Jerome Ravetz, and three anonymous
reviewers.
720Dunn: Methods of the Second Type 721
their best to push back the wilderness, knowing full well that the wilder-
ness is always there, and that one day enemies may surprise and destroy
them. The homesteaders also are aware that there are enemies within,
enemies that may undermine and distort their judgment. They hope they
will not choose to tend the crops and livestock when they should have
chosen to clear more land.
THE STRUCTURE OF POLICY PROBLEMS
Policy analysts, like homesteaders, require methods that enable them to
know when they have cleared enough land, that is, to know when they have
approximated the proper boundaries of a problem. The bulk of policy
analytic methods available today, however, assume that the boundaries of
policy problems are well defined. Writing more than twenty years ago,
Zeckhauser and Shaefer (1968, p. 29) provided a view of policy problems
that accurately describes much current analytic thinking. Here, the view
is that problems come in relatively well-bounded packages involving an ex-
plicit statement of a decision maker’s preferences, a careful exposition of
the alternative actions available, and a model that relates these alterna-
tives to the stated preferences in a manner that permits an efficient choice
among the alternatives (see also Stokey & Zeckhauser, 1978; Nagel, 1982,
pp.4-5).
This view of policy problems assumes that the boundaries, which cir-
cumscribe preferences, alternatives, and their relationships are relatively
well-defined. In effect, the analyst is faced with what Mitroff (1974; also
see Mitroff & Blankenship, 1973) defines as a structured decision problem,
where the relationships between decision makers (Dj), preferences or
utilities (Uy), alternatives (Aj), outcomes (Oj), and states of nature (Sj) are
certain, probabilistic, or uncertain. Structured problems “are problems
about which enough is known so that problems can be formulated in ways
that are susceptible to precise analytic methods of attack" (Mitroff, 1974,
p. 224).
Structured decision problems are properly contrasted with problems
that are ill-structured (Simon, 1973), squishy (Strauch, 1976), or messy
(Ackoff, 1974b). Formally, an ill- structured problem is one for which
decision makers (Dj), preferences or utilities (Uij), alternatives (Aj), out-
comes (Oj), or states of nature (Sj) are unknown or equivocal. In practice,
ill-structured problems (cf. Harmon & King, 1985, p. 28) have a number of
common properties:
*Policy Goals. The goals of policy are ambiguous or unknown, so that
determining what goals to achieve is part of the problem. "Our problem
is not to do what is right," stated Lyndon Johnson. "Our problem is to
know what is right" (quoted by Wood, 1968, p. v).
*Policy Phases. The phases through which policy goals are to be
achieved are indeterminate. Since linkages among phases involve feed-
back and feed-forward loops which may occur at any time, the pattern of
phases is more like a tangled river network (Beer, 1981, p. 30) than an
assembly line, tree, or cycle (cf. Jones, 1977). There is no assurance that
success at one phase will lead to success at another, for example, that722 Policy Studies Review, Summer 1988, 7:4
adopting an optimal policy alternative will lead to its successful im-
plementation.
*Policy Instruments. The policy instruments required to achieve
goals are ambiguous or unknown. Knowledge about what policy instru-
ments work best under which conditions is often rudimentary or simply
unavailable (Linder & Peters, 1985, 1987). Even when analysts use ad-
vanced decision technologies (e.g., computer-designed event and fault
trees) they frequently overlook instruments which are critical to the suc-
cess of policies. For example, nuclear near-disasters such as the Brown’s
Ferry fire "was started by a technician checking for an air leak with a
candle, in direct violation of standard operation procedures” (Fischhoff,
1977, p. 181).
*Policy Problem Domain. The domain of potentially relevant goals,
phases, and instruments is unbounded. No exhaustive or even ap-
proximate set of goals, phases, and instruments is available to the policy
analyst. The problem domain thus appears to be unmanageably huge,
with policy analysts engaged in what Dery (1984, p. 6) represents as "a
never-ending discourse with reality.”
Characterized in this fashion, ill-structured problems are not uncom-
mon; they are pervasive (Simon, 1973, p. 186). The pervasiveness of ill-
structured problems is a consequence of the fact that conflicting
representations of problems are continuously created, maintained, and
changed by stakeholders who affect and are affected by the policy-making
processes of modern governments.
If one or a few policy stakeholders were harmoniously engaged in the for-
mulation of problems, then policy analysts would spend most of their time
solving relatively well-structured problems. The typical case, however, is
one in which practicing analysts expend large amounts of time and energy
investigating the conflicting problem definitions of large numbers of policy
stakeholders. Such is the case with many health and educational polices,
which Sieber (1981) characterizes as "fatal remedies," which arise as a con-
sequence of faculty problems definitions. The "policy pathologies” docu-
mented by Hogwood and Peters (1985) also involve ill-structured problems
that inadvertently were defined as well-structured problems, thus promot-
ing apparently correct solutions to the wrong problems. Relatedly, in legis-
lative policy analysis, it is often the case that policy analysts not only must
address the problem definitions of legislators, but also those of legislative
staff, executive agency personnel, and representatives of a multitude of
public interest groups. Seasoned analysts know well that the process of for-
mulation problems occurs throughout the policy-making process; it invol-
ves legislators and other immediate clients of policy analysis as well as
street-level bureaucrats (Lipsky, 1971) and ordinary citizens situated at the
"periphery" of the policy-making process (see Sabatier & Mazmanian, 1983,
pp. 149-151). Thus, the process of formulating problems is not confined,
temporally or spatially, to those phases of policy making conventionally
labeled problem formulation or agenda setting. Competing problem
formulations are distributed throughout the policy-making process, creat-
ing a plethora of ill-structured problems.Dunn: Methods of the Second Type 723
Stakeholders situated at various locations in the policy-making process
actively construct, on the basis of their own experiences, different repre-
sentations of problems. In effect, stakeholders create ill-structured
problems by bringing to the policy-making process competing sets of as-
sumptions about those external events, which John Dewey liked to call a
"problem situation." These sets of assumptions, variously described as con-
ceptual models (Allison, 1971), systems of interpretation (Heclo, 1976), cog-
nitive maps (Axelrod, 1976), schema (Taylor & Crocker, 1980), frames of
reference (Holzner & Marz, 1979), and construction systems (Dunn, Cahill,
Dukes, & Ginsberg, 1986), vary among policy stakeholders who construct
for their own interest and survival competing representations of the same
problem situation. Yet critical elements of a problem situation may lie out-
side the boundaries of an individual's construction system; what is unrecog-
nized and unknown cannot be understood or anticipated. Inadequately
trained technicians at nuclear power facilities may endanger millions of
citizens by searching for air leaks with candles (Fischhoff, 1977), warnings
placed on cigarette packages may preclude future opportunities to deal with
public health problems (Sieber, 1981), and the institutionalization of
pretrial release actually may increase the jail population (Nagel & Neef,
1976).
A major challenge for policy analysis is to deal with the complexity aris-
ing from the mutual construction of policy problems. Policy analysts work-
ing in real-life policy contexts, as distinguished from analysts working in
the rarified atmosphere of academic publishing, typically are not faced with
a single, well-formulated problem. Rather they are faced with many
problems, which are distributed throughout the policy-making process,
owned by diverse stakeholders whose perspectives and actions are inter-
dependent, and continuously altered as a consequence of incremental policy
learning. Under these circumstances the practicing policy analyst appears
as a modern Diogenes engaged in a never-ending search for the right
problem (Dery, 1984, pp. 6-7; see also Wildavsky, 1979).
Thus, practicing analysts normally face a large, tangled network of com-
peting problem formulations that are socially constructed, distributed, and
dynamic. In these circumstances, practicing analysts are faced with a meta-
problem (Dror, 1971), a problem-of-problems that is ill-structured because
the domain of relevant policy goals, phases, and instruments is unbounded;
that is, unmanageably huge. Here, the central task of policy analysis is to
structure a problem-of-problems, a second-order entity that may be defined
as the class of all first-order problems, which are its members. Unless these
two levels of problems are clearly distinguished, analysts run the risk of for-
mulating the wrong problem by confusing member and class, thus ignoring
the rule that "whatever involves all of a collection must not be one of the
collection" (Whitehead & Russell, 1910, p. 37; also see Watzlawick, Weak-
land, & Fisch, 1974, p. 6).
THE PRINCIPLE OF METHODOLOGICAL CONGRUENCE
The distinction between member and class, between second-order and
first-order problems, provides a basis for assessing the appropriateness of
different methods now available to the practicing policy analyst. Methods
of policy analysis can be assessed according to what might be called the prin-724 Policy Studies Review, Summer 1988, 7:4
ciple of methodological congruence: The appropriateness of a particular
type of method is a function of its congruence with the type of problem under
investigation. The principle of methodological congruence is similar to a
more general principle which, originally stated by L.A. Zadeh, asserts that
conventional scientific methodologies are incompatible with social
problems that have exceeded a given threshold of complexity (see Brewer
& de Leon, 1983, p. 125).
The bulk of methods available to practicing policy analysts are methods
of the first type, that is, methods appropriate to relatively well-structured
problems where relationships between decision maker (D)), utilities (Ujj),
alternatives (Aj), outcomes (0j), and states of nature (Sj) are certain or prob-
abilistic. These first-order methods, well-known to users of standard policy
analysis textbooks, include a range of methodologies for policy modeling:
input-output analysis, linear and nonlinear programming, macroeconomic
and microeconomic analysis, land-use analysis, game theory, and system
dynamics (see, e.g., Greenberger, Crenson, & Crissey, 1976; White, 1983).
Yet the principle of methodological congruence asserts that these first-
order methods are inappropriate, and, therefore, should not be used to ad-
dress those second-order problems that have been characterized as squishy,
messy, or ill-structured. Cost-benefit analysis, for example, is an ap-
propriate and useful first-order method that enables calculations of net
benefits or benefit-cost ratios for a set of well-bounded alternatives. But
cost-benefit analysis does not and cannot generate the policy goals, phases,
and instruments that should be included in the set. The generation of
goals, phases, and instruments is a second-order task that cannot be ac-
complished with methods of the first type.
The principle of methodological congruence is generally ignored by those
who continue to approach relatively ill-structured problems with methods
suitable for relatively well-structured problems. Although many observers
contend that the complexity of problems requires multiple methods (e.g.,
Brewer & de Leon, 1983, pp. 137-38; Quade, 1985), these observers do not
clearly distinguish between first-order and second-order methods; nor do
they acknowledge that first-order methods, whether used singly or in mul-
tiples, are inappropriate for most second-order problems most of the time.
When second-order methods are presented for consideration, it is more
than a little doubtful whether the resultant methodological advice repre-
sents much more than a set of general recipes, for example, “the analyst
should prepare a menu of policy alternatives that cover a range of ap-
propriate, possible, and feasible solutions to the problem" (Brewer & de
Leon, 1983, p. 65). Such recipes, as Linder and Peters (1985, p. 240) cor-
rectly observe, typically represent "advice on what not to do and what pit-
falls to avoid in applying one’s intuitions ... an interactive and creative
process with few rules and guidelines ...”
The principle of methodological congruence, in addition to specifying the
conditions under which distinct levels of methods are appropriate, helps
remove the ambiguities evident in discussions of Type III errors in policy
analysis. In contrast to Type I and Type II statistical errors, Type III er-
rors are typically defined in somewhat vague terms as conceptualizing, for-
malizing, or solving the "wrong" or "less appropriate” problems (see, e.g.
Dunn 1981, pp. 109-110; Mitroff & Featheringham, 1976; Raiffa, 1968, p-
264). By drawing on the distinction between member and class we may for-Dunn: Methods of the Second Type 725
mulate a more concrete definition of Type III errors in policy analysis: Solu-
ing the wrong problem by employing a method whose level is incongruent
with that of the problem under investigation.
The practical consequences of methodological congruence and incon-
gruence are displayed below in Figure 1, which supplies several much-
needed distinctions among key methodological aspects of the process of
policy analysis.
PROBL
(DISSOLVING
Figure 1726 Policy Studies Review, Summer 1988, 7:4
*Problem Sensing and Problem Structuring. The process of policy
analysis does not begin with clearly articulated problems, but a sense of
diffuse worries and inchoate signs of stress (Rein & White, 1977, p. 262).
These diffuse worries and inchoate signs of stress are not problems, but
problem situations. Policy problems "are products of thought acting on
environments; they are elements of problem situations that are
abstracted from these situations by analysis. What we experience,
therefore, are problem situatuions, not problems which, like atoms or
cells, are conceptual constructs" (Ackoff, 1974b, p. 21).
*Problem Structuring and Problem Solving. Policy analysis is a
multilevel process that includes first-order methods of problem solving
as well as second-order methods of problem structuring. This multilevel
process is what others (e.g., Linder & Peters, 1985; Dryzek, 1983; Miller,
1985) describe as policy design or design science. Methods of problem
structuring are meta-methods, in the sense that they are “about” and
“come before” methods of problem solving. The principle of methodologi-
cal congruence dictates that first-order methods of problem solving (e.g.,
cost-benefit analysis) are rarely if ever appropriate for structuring
second-order problems that are ill-structured. When practitioners apply
first-order methods to second-order problems they typically make "er-
rors of a third kind: solving the wrong problem" (Raiffa, 1968, p. 264).
*Problem Resolving, Problem Unsolving, and Problem Dissolv-
ing. The terms problem resolving, problem unsolving, and problem dis-
solving refer to three kinds of error correcting processes in policy
analysis (see Simon, 1973, Ackoff, 1974a; Dunn, 1981, p. 99). Although
the terms "resolving," "unsolving," and "dissolving" stem from the same
root (L. solvere = to solve or dissolve), these error correcting processes
occur at distinct levels (see Figure 1). Problem resolving involves the
reanalysis of a correctly structured problem in order to reduce calibra-
tional errors: for example, errors involving the rejection of the null
hypothesis when it is true (Type I error) or accepting the null hypothesis
when it is false (Type II error). Problem unsolving, by contrast, invol-
ves the abandonment of a solution based on the wrong problem and a
return to problem structuring as a means to formulate the right
problem. Finally, problem dissolving involves the abandonment of an in-
correctly formulated problem prior to any effort to solve it.
In summary, policy analysis is a dynamic, multi-level process in which
different methods perform distinctly different functions. Because methods
appropriate at one level are inappropriate at the next level, questions of ap-
propriateness cannot be satisfactorily resolved without first considering
the level of the problem to which a method is applied. When policy analysts
violate the principle of methodological congruence they are likely to solve
the wrong problem, thus committing a Type III error.
METHODS OF THE SECOND TYPE
Methods of the second type, although generally available to the policy
analysis community, are seldom included as an integral part of training inDunn: Methods of the Second Type rer
policy analysis. Textbooks, manuals, and handbooks of policy analysis rare-
ly cover a range of second-order methods that have been expressly designed
for structuring those second-order problems variously described as squishy,
messy, or ill-structured. For example, authoritative sources such as the En-
cyclopedia of Policy Studies (Nagel, 1983) contain almost no references to
methods of the second type such as brainstorming (Osborn, 1948), synectics
(Gordon, 1961), policy capturing (Hammond, 1980), the analytic-hierarchy
process (Saaty, 1980), interpretive structural modeling (Warfield, 1976),
multiple perspective analysis (Linstone et al., 1981), and strategic assump-
tion surfacing and testing (Mason & Mitroff, 1981). Apart from one
reference to the analytic hierarchy process in a chapter on policy analysis
and management science (White, 1983a), the majority of references to these
methods appear in syntheses of literature in areas of technology assessment
(Porter & Rossini, 1983) and ethics (Dunn, 1983), areas which typically are
regarded as peripheral to mainstream policy analysis.
‘Among the many hypotheses that might be advanced to explain the lack
of attention to methods of the second type, several appear most plausible.
First of all, it bears notice that those who write most about methods of
policy analysis have far fewer opportunities to work in complex, real-life
contexts than practitioners who are the presumed beneficiaries of these
methods. Perhaps in response to the rarified incentive systems of academic
tenure review and professional publishing, those who write about methods
of policy analysis simply may be unaware that practicing policy analysts
spend most of their time structuring policy problems, not solving them.
Consequently, there is a considerable observed time lag between the emer-
gence of practical needs and efforts to satisfy them through the develop-
ment of appropriate new methodologies.
A second plausible hypothesis has to do with the vague or ambiguous
character of many methods of the second type. Although there is some
recognition that competing perspectives of a problem may assist in the dis-
covery of otherwise hidden policy goals, phases, and alternatives, the avail-
able guidelines for conducting brainstorming sessions (Osborn, 1948),
continuous decision seminars (Lasswell, 1960), or multiple perspective
analyses (Linstone et al., 1981) represent general heuristics, which cannot
be easily replicated or evaluated by two or more analysts. Even when
methods of the second type are well specified and replicable, as is the case
with policy capturing (Adelman, Stewart, & Hammond, 1975), the analytic
hierarchy process (Saaty, 1980), and strategic assumption surfacing and
testing (Mason & Mitroff, 1981), standards for evaluating the extent to
which an analyst has performed well in structuring a problem are am-
biguous or simply unavailable. As Mitroff and Mason acknowledge (1981,
pp. 73-86), there is no test that guarantees the completeness of a set of
problem representations. Saaty (1980, p. 14) makes much the same point
when he acknowledges that "there is no set procedure for generating objec-
tives, criteria, and activities to be included in a hierarchy or even a more
general system." The ambiguity or outright absence of performance criteria
is closely related to problems of interpreting the results of laboratory and
field research on the efficacy of methods of the second type. Understandab-
ly, research on the performance of these methods in producing improved
problem representations has yielded equivocal or conflicting findings (see,
e.g., Cosier, 1978, 1981; Cosier, Ruble, & Aplin, 1978; Mason, 1969; Mitroff,728 Policy Studies Review, Summer 1988, 7:4
Barabba, & Kilmann, 1977; Mitroff & Mason, 1981a; Schwenk & Cosier,
1980).
The further development and application of methods of the second type
presuppose the identification of explicit criteria for assessing the perfor-
mance of analysts in structuring problems. A first step in this process is to
recognize that all methods of the second type are heuristics that aim at the
discovery of elements that define a problem and/or one or more of their
general relationships. Apart from this broad aim, methods of the second
type perform functions that are constitutive and regulative. Methods
designed to discover the elements that define a problem are constitutive,
since they answer the question: What elements constitute the problem?
Regulative methods, by contrast, aim at the discovery of patterned relations
among these elements. Here the question is: How are the elements that
define the problem regulated? The distinction between constitutive and
regulative methods parallels the conclusion that problem definition in
policy analysis is based on what philosopher John Searle calls constitutive
rules, not regulative ones (Dery, 1984, p. 5). The distinction also
punctuates the fact that the process of structuring policy problems requires
that we test our explanation or comprehension of the patterns believed to
regulate a problem (regulative rules) as well as our definition of its con-
stituent elements (constitutive rules).
Methods of the second type also differ in terms of their replicability.
Methods with low replicability involve general and vague guidelines, while
methods with high replicability involve specific and readily comprehensible
prescriptions for carrying out a defined sequence of operations. As instru-
ments of discovery, such highly replicable methods have been characterized
by Landa (1984, p. 39) as algorithmic-heuristics and contrasted with non-
algorithmic heuristics that are based on imprecise and vague methodologi-
cal prescriptions; for example, the prescription to examine multiple
perspectives of a problem in order to discover its true nature and potential
solutions.
Methods of the second type may be sorted into four distinct categories
(see Figure 2) formed by the intersection of types of purpose that are con-
stitutive or regulative, and degrees of replicability that are high or low. In
one category are methods the primary purpose of which is to enumerate ele-
ments believed to constitute the boundaries of a problem. Although they
represent a powerful medium for generating potential policy goals and al-
ternatives, these methods are sufficiently ambiguous, imprecise, and
FUNCTIONS OF METHODS OF THE SECOND TYPE
PURPOSE OF METHOD
REPLICABILITY OF METHOD Constitutive Regulative
Element Enumeration Pattern Enumeration
Low (Multiple Perspective (Interpretive Structural
Analysis) Modeling)
Element Estimation Pattern Estimation
High (Policy Grid Analysis) (Strategic Assumption
Surfacing and Testing)
Figure 2.Dunn: Methods of the Second Type 729
general that they cannot be easily replicated by two or more analysts.
Methods of the second type such as brainstorming (Osborn, 1948), con-
tinuous decision seminars (Lasswell, 1960), and multiple perspective
analysis (Linstone et al., 1981) appear to belong in this category.
In a second category are methods the main purpose of which is to
enumerate patterns, which regulate elements within the boundaries of a
previously defined problem space. Although they represent a useful
medium for representing a variety of patterned relations among policy goals
and alternatives--for example, patterns represented by physical and biologi-
cal metaphors and analogies such as trees, rivers, waterfalls, or epidemics-
-these methods are again so imprecise, ambiguous, and general that they
cannot be replicated with the confidence that two or more analysts will
reach the same conclusions. Synectics (Gordon, 1961) and other forms of
analogical reasoning (see, e.g., Rein, 1976; Schon, 1983) appear to belong in
this category of methods of the second type.
A third category of methods of the second type includes procedures the
main purpose of which is to estimate elements that constitute the boun-
daries of a problem. These methods are highly replicable by virtue of their
relative specificity, precision, and comprehensibility. Presently, there are
few methods in this category, which means that the process of defining
policy problems still resembles what Dery (1984, pp. 6-7) appropriately
characterizes as "a never-ending discourse with reality, to discover yet more
facets, more dimensions of action, more opportunities for improvement
li.e., problems]." Although the absence of boundary-approximating rules or
tests frequently makes it impossible to know whether we have defined the
right problem (see Saaty, 1980, p. 14; Mitroff & Mason, 1981b, pp. 73-86),
methods such as policy grid analysis (Dunn, Cahill, Dukes, & Ginsberg,
1986; Dunn & Ginsberg, 1986; also see Kelly, 1955) appear to provide
replicable estimation procedures.
In the fourth category are methods of the second type that aim primari-
ly at estimating patterns believed to regulate relationships between pre-
viously defined elements, for example, patterns of conflict and
cohesiveness, distance and proximity, or consistency and inconsistency
among policy actors, policy goals, policy alternatives, and policy outcomes.
In this category are highly replicable methods that include policy capturing
(Adelman, Stewart, & Hammond, 1975), interpretive structural modeling
(Warfield, 1976), Q-methodology (Brown, 1980), the analytic hierarchy
process (Saaty, 1980), and strategic assumption surfacing and testing
(Mason & Mitroff, 1981). Singly and in combination, these methods are a
powerful medium for creating spatial, geometric, and quantitative repre-
sentations of the structure of problems. In addition to their replicability,
these methods provide tests for estimating the ecological validity (Adel-
man, Stewart, & Hammond, 1975), consistency (Saaty, 1980), and
plausibility (Mason & Mitroff, 1981) of patterns believed to represent the
structure of policy problems. Nevertheless, these methods do not permit
estimates of the boundaries of a problem.
PERFORMANCE CRITERIA
_ Our confidence in methods of the second type would be enhanced con-
siderably if explicit criteria were available to assess the performance of730 Policy Studies Review, Summer 1988, 7:4
policy analysts in structuring ill-structured problems. In this context, Res-
cher (1980) has formulated an integrated set of criteria for assessing the
plausibility of inductive estimates made under conditions where deter-
ministic and probabilistic conclusions are not possible. These criteria are
suitable for assessing the performance of policy analysts in using methods
of the second type.
Methods of the second type, as we saw above, are appropriate under
those numerous real-life conditions where the analyst’s problem is relative-
ly ill-structured, that is, where the analyst does not know and therefore
must discover the appropriate decision makers (Dj), utilities (Uij), alterna-
tives (Aj), outcomes (O)), and states of nature (Sj). The function of methods
of the second type is to produce plausible estimates of the elements that
constitute the problem as well as the patterns regulating these elements
(see Figure 2). The plausibility of these estimates--as opposed to their
statistical probability or deductive certainty--depends on the extent to
which they satisfy requirements of inductive estimates in general (see Res-
cher, 1980, pp. 24-26):
*Character. An estimate (P*) of problem (P) must have the same
character as its estimanda. For example, an estimate of a length must
be a length, not a temperature. In estimating the elements that con-
stitute a problem we typically must estimate elements which are subjec-
tive in character, since policy problems are products of thoughts acting
on environments. In turn, estimates of patterns that regulate relation-
ships among elements typically must be systemic, since policy problems
are second-order entities composed of the diverse problem formulations
of many interdependent stake-holders. In short, any estimate P* of a
problem P must have the character of a representation that is subjec-
tively meaningful to a complex system of policy stakeholders.
*Replicability. The process of producing an estimate (P*) of a problem
(P) must be replicable, thus maximizing the likelihood that two or more
analysts can obtain similar results in similar circumstances. Without
replicable methods policy analysts are little more than intelligent
clinicians who must rely on their own wits and whatever anecdotal
knowledge they have acquired through trial and error in the field
(Fischhoff, 1986, p. 112). The replicability of methods of the second type
is a function of their specificity, precision, and comprehensibility to
analysts faced with ill structured problems.
*Coordination. An estimate (P*) of a problem (P) must coordinate
with the elements of that problem and their relationships. The closer
the process of coordination, the more accurate the estimate. Estimates
based on observations of the ways that policy stakeholders actually con-
strue problems should coordinate more closely than estimates based on
the specialized constructs of analysts with a problem, for example, con-
structs such as "rationality," “utility,” or "revealed preference.”
*Cost-Effectiveness. The cost of producing an estimate (P*) of 4
problem (P) must be reasonable. A method that yields inaccurate es-
timates at a low cost may be far less cost-effective than a method thatDunn: Methods of the Second Type 731
yields more accurate estimates at a higher cost, particularly in cases
where an inaccurate estimate obscures externalities, sleeper effects,
spillovers and other negative unanticipated consequences. In short, the
opportunity costs of defining the wrong problem may be enormous.
*Correctness-in-the-Limit. When the information on which an es-
timate is based becomes increasingly complete, the estimate (P*) of a
problem (P) eventually should approximate the true answer being es-
timated. Although the "true" definition and structure of a problem can-
not be known with certainty, increasingly complete information should
produce increasingly accurate estimates. In the limit, an estimate (P*)
should converge on the true but unknown value of the problem (P).
Methods of the second type now available to practicing policy analysts
satisfy one or several, but not all, of these criteria. For example, methods
such as multiple perspective analysis and synectics, while they satisfy the
criterion of character, do not fare well on criteria of replicability, coordina-
tion, and correctness-in-the-limit. Methods such as policy capturing, Q-
methodology, interpretive structural modeling, the analytic hierarchy
process, and strategic assumption surfacing and testing satisfy in different
ways criteria of character, replicability, and coordination. As we have seen,
however, these methods of the second type do not provide any rule or test
that permits analysts to assess the completeness of a set of elements defin-
ing a problem. In effect, these methods do not satisfy the correctness-in-
the-limit criterion and, as such, do not provide plausible estimates (P*) of
the elements that constitute a problem (P).
In contrast to these methods of the second type, policy grid analysis
(Dunn, Cahill, Dukes, & Ginsberg, 1986; Dunn & Ginsberg, 1986; also see
Bannister & Mair, 1968) satisfies a performance criterion on which all other
methods are lacking: correctness-in-the-limit. Apart from its meeting the
criteria of character, replicability, coordination, and cost-effectiveness, the
unique advantage of policy grid analysis is its capacity to estimate the boun-
daries of a second-order problem composed of the competing problem rep-
resentations of policy stakeholders.
The method of estimation is precise and readily comprehensible, since it
involves a series of three simple steps (Dunn, Cahill, Dukes, & Ginsberg,
1986). The first step is saturation sampling, whereby the analyst generates
a snowball sample of stakeholders who affect or are affected by an existing
or proposed policy. Each person is an initial set of stakeholders, selected
to maximize differences in issue position and influence, is asked to name
others who agree and disagree with one or more existing or proposed
policies. The stakeholders so named are asked to name others, who in turn
provide additional names. The process is continued until no new
stakeholders are named, at which point "there is no sampling variance, be-
cause the total universe has been surveyed, unless one considers the group
as some subsample from a super population" (Sudman, 1976, p. 211).
The second step involves the elicitation of constructs used by
stakeholders to represent a given problem situation. These constructs are
the "ideas, basic paradigms, dominant metaphors, standard operating pro-
cedures, or whatever else we choose to call the systems of interpretation
by which we attach meaning to events" (Heclo, 1976, pp. 253-254). Although732 Policy Studies Review, Summer 1988, 7:4
there are many ways to elicit policy-relevant constructs (e.g., face-to-face
interviews, telephone interviews, mailed questionnaires, computer con-
ferencing), the basic procedure is to ask stakeholders to contrast a range of
policies and then supply the constructs in terms of which the contrasts were
made. At that point when stakeholders can supply no new constructs, the
analyst assumes that an accurate estimate (P*) of the boundaries of the
stakeholder’s problem (P) has been made. The constructs that form the
stakeholder’s representation of the problem situation can be subjected to
various forms of statistical analysis (see Bannister & Mair, 1968; Dunn,
Cahill, & Kearns, 1985; Slater, 1977), although this is not a necessary fea-
ture of the process.
The third step in policy grid analysis is second-order boundary estima-
tion. Here the analyst constructs a cumulative frequency distribution of all
unique constructs elicited from stakeholders at various stages in the selec-
tion of the snowball sample. The various stakeholders are plotted on the
horizontal axis; the number on nonduplicative and therefore new con-
structs is plotted on the vertical axis (Figure 3). As information about the
constructs is of a gradually expanding sample of stakeholders becomes more
complete, estimates of the elements that define the systemic problem space
become more accurate. These improvements in the accuracy of estimates
represent a continuous transformation of individual problem repre-
sentations into a systemic or second-order problem representation.
30
Pa = dx/dy = 0
0 3 10 15 20
SOURCE: Dunn, Cahill, and Dukes (1986), p. 372.
Figure 3.
The cumulative frequency distribution displayed in Figure 3, based on
data from a project in criminal justice policy, illustrates how the boundaries
of an ill-structured problem may be estimated. As the analyst plots the non-
duplicative and therefore new constructs of each successive stakeholder,
the slope of the curve displays different rates of change. Initially, there is
a rapid rate of change that decreased to the point of stagnation, which repDunn: Methods of the Second Type 733
resents the true but unknown boundary of the problem, as estimated by the
rule P* = dx/dy = zero. This rule (the first derivative of a function) is one
among several conceivable ways for estimating the boundaries of the
problem and, at the same time, reaching the entirely credible conclusion
that the collection of additional information from policy stakeholders is un-
likely to improve the accuracy of the problem representation.
The estimation procedures outlined above would appear to satisfy re-
quirements of inductive estimates in general: character, coordination, cost-
effectiveness, and correctness-in-the-limit. Previous applications of
similar procedures in contexts, which are at least as complex as public policy
(see, e.g., Stabell, 1978), suggest that systemic problem representations
normally may be estimated on the basis of a small number of interviews,
perhaps as few as twenty, which means that the universe of available
problem representations is limited and exhaustible. These estimation pro-
cedures permit policy analysts to transform an otherwise unmanageably
huge set of individual problem representations into a second-order problem
representation that satisfies the rule: "whatever involves all of a collection
must not be one of the collection" (Whitehead & Russell, 1910, p. 37).
CONCLUSION
Policy analysts, like allegorical homesteaders, face the seemingly un-
manageable task of coping with the wilderness of ill-structured problems,
while concurrently attending to well-structured problems that can be ad-
dressed with conventional methods of policy analysis. What has appeared
thus far as an unmanageable task may become manageable once analysts
acknowledge the principle of methodological congruence: The appropriate-
ness of a particular type of method is a function of its congruence with the
type of problem under investigation. In practical terms this means that
policy analysts who wish to improve the problem-solving capacities of
governments can employ methods that do not ignore the political, social,
cultural, and psychological complexity of the policy-making process.
To be sure, conventional methods of the first type are appropriate and
useful for solving first-order problems that are relatively well-structured.
Contexts of practice, however, are pervaded by second-order problems that,
variously described as squishy, messy, or ill structured, are the class of all
first-order problems. Just as methods of the first type are congruent with
the analytic demands of first-order problems, so are methods of the second-
type congruent with the complex analytic demands of the second-order
problems. When policy analysts fail to observe this principle of
methodological congruence they are likely to solve the wrong problem.
Methods of the second type are not limited to general heuristics, but in-
clude specific and readily comprehensible procedures for estimating the
boundaries and structure of ill-structured problems. Since these estima-
tion procedures appear to satisfy requirements of inductive estimates in
general, policy analysts can assess their own performance in providing ap-
proximate answers to the right question, thus coping with the enemies who
lurk in the wilderness of conventional policy analysis.734 Policy Studies Review, Summer 1988, 7:4
REFERENCES
Ackoff, R.L. (1974a). Beyond problem solving. General Systems, XIX, 237-
239.
Ackoff, R.L. (1974b). Redesigning the future: A system approach to societal
problems. New York: John Wiley.
Adelman, L., Stewart, T.R., & Hammond, K.R. (1975). A case history of the
application of social judgment theory to policy formulation. Policy Scien-
ces, 6, 137-159.
Allison, G.T. (1971). Essence of decision: Explaining the Cuban missile
crisis, Boston: Little, Brown.
Axelrod, R.A. (1976). Structure of decision. Cambridge, MA: Harvard
University Press.
Bannister, D., & Mair, J.M.M. (1968). The evaluation of personal con-
structs. New York: Academic Press.
Beer, S. (1981). The brain of the firm. New York: John Wiley.
Brewer, G.D., & de Leon, P. (1983). The foundations of policy analysis.
Homewood: The Dorsey Press.
Brown, S.R. (1980). Political subjectivity: Applications of Q-methodology
in political science. New Haven: Yale University Press.
Cosier, R.A. (1981). Dialectical inquiry in strategic planning: A case of
premature acceptance. Academy of Management Review, 6, 643-648.
Cosier, R.A. (1978). The effects of three potential aids for making strategic
decisions on prediction accuracy. Organizational Behavior and Human
Performance, 22, 295-306.
Cosier, R.A., Ruble, T.L., & Aplin, J.C. (1978). An evaluation of the effec-
tiveness of dialectical inquiry systems. Management Science, 24, 1483-
1490.
Dery, D. (1984). Problem definition in policy analysis. Lawrence, KS:
University Press of Kansas.
Dror, Y. (1971). Design for policy sciences. New York:
Elsevier.
Dryzek, J. (1983). Don’t toss coins into garbage cans: A prologue to policy
design. Journal of Public Policy, 3 345-367.
Dunn, W.N. (1983). Values, ethics, and standards in policy analysis. InS.S.
Nagel (Ed.), Encyclopedia of policy studies. New York: Marcel Dekker.
Dunn, W.N., Cahill, A.G., Dukes, M.J., & Ginsberg, A. (1986). The policy
grid: A cognitive methodology for assessing change dynamics. In W.N.
Dunn (Ed.), Policy analysis: Perspectives, concepts, and methods. Green-
wich, CT: JAI Press.
Dunn, W.N., Cahill, A.G., & Kearnes, K.P. (1984). Applications of the policy
grid to work-related frames of reference. (Working paper). Pittsburgh:
University of Pittsburgh, University Program for the Study of
Knowledge Use.
Dunn, W.N., & Ginsberg, A. (1986). A sociocognitive approach to organiza-
tional analysis. Human Relations, 39(11), 955-975. .
Fischhoff, B. (1986). Clinical policy analysis. In W.N. Dunn (Ed.), Policy
analysis: Perspectives, concepts, and methods (pp. 111-128). Greenwich,
CT: JAI Press. ;
Fischhoff, B. (1977). Cost benefit analysis and the art of motorcycle main-
tenance. Policy Sciences, 8, 177-202.Dunn: Methods of the Second Type 735
Gordon, W.J. (1961). Synectics. New York: Harper and Row.
Greenberger, M., Crenson, M.A., & Crissey, B.L. (1976). Models in the
policy process. New York: Russell Sage Foundation.
Hammond, K.R. (1980). Introduction to Brunswikian theory and methods.
New Directions for Methodology of Social and Behavioral Science, 3, 1-
12.
Harmon, P., & King, D. (1985), Expert systems: Artificial intelligence in
business. New York: John Wiley.
Helco, H.H. (1976). Policy dynamics. In R. Rose (Ed.), The dynamics of
public policy. Beverly Hills: Sage Publications.
Hofstadter, R. (1979). Godel, Escher, Bach. New York: Random House.
Hogwood, B.W., & Peters, B.G. (1985). The pathology of public policy. Ox-
ford: Clarendon Press.
Holzner, B., & Marx, J. (1979). Knowledge application: The knowledge sys-
tem in society. Boston: Allyn and Bacon.
Jones, C.0. (1977). An introduction to the study of public policy (2nd ed.).
North Scituate, MA: Duxbury Press.
Kelly, G. (1955). The psychology of personal constructs. New York: M.W.
Norton.
Kline, M. (1980). Mathematics: The loss of certainty. New York: Oxford
University Press.
Landa, L.N. (1984). Algorithmic-heuristic theory. In R.J. Corsini, & B.D.
Ozak (Eds.), Encyclopedia of psychology. New York: John Wiley.
Lasswell, H.D. (1960). Technique of decision seminars. Midwest Journal
of Political Science, 4(2), 213-226.
Linder, S.H., & Peters, B.G. (1985). From social theory to policy design.
Journal of Public Policy, 4(3), 237-259.
Linder, S.H., & Peters, B.G. (1987). A design perspective on policy im-
plementation: The fallacies of misplaced prescription. Policy Studies
Review, 6(3), 459-475.
Linstone, H.A., et al. (1981). The multiple perspective concept: With ap-
plications to technology assessment and other decision areas. Tech-
nological Forecasting and Social Change, 20, 275-325.
Lipsky, M. (1971). Street-level bureaucracy and the analysis of urban
reform. Urban Affairs Quarterly, 6, 391-409.
Mason, R.O. (1969). A dialectical approach to strategic planning. Manage-
ment Science, 15, 403-414.
Mason, R.O., & Mitroff, I.I. (1981). Challenging strategic planning as-
sumptions: Concepts, techniques, and methods. New York: John Wiley.
Miller, T.C. (1985). Conclusion: A design science perspective. In T.G.
Miller (Ed.), Public sector performance: A conceptual turning point. Bal-
timore: Johns Hopkins University Press.
Mitroff, II. (1974). The subjective side of science. New York: Elsevier.
Mitroff, I.I., Barabba, V.P., & Kilmann, R.H. (1977). The application of be-
havioral and philosophical technologies to strategic planning: A case
study of a large federal agency. Management Science, 24, 44-58.
Mitroff, I.I., & Blankenship, L.V. (1973). On the methodology of the holis-
tic experiment: An approach to the conceptualization of large-scale so-
cial experiments. Technological Forecasting and Social Change, 4,
339-353.736 Policy Studies Review, Summer 1988, 7:4
Mitroff, I.I., Featheringham, T. (1976). Towards a behavioral theory of sys-
temic hypothesis-testing and the error of the third kind. Theory and
Decision, 7, 205-220.
Mitroff, I.1., & Mason, R.O. (1981). Creating a dialectical social science:
Concepts, methods, and models. Dordrecht: D. Reidel.
Mitroff, LI., Mason, R.O., & Barabba, V.P. (1983). The 1980 census:
Policymaking amid turbulence. Lexington, MA: D.C. Heath.
Nagel, S.S. (1982). Policy evaluation. New York: Praeger.
Nagel, S.S. (1983). Encyclopedia of policy studies. New York: Marcel Dek-
ker.
Nagel, S.S., & Neef, M. (1976). Two examples from the legal process.
Policy Analysis, 2(2), 356-57.
Osborn, A.F. (1948). Your creative mind. New York: Charles Scribner.
Porter, A.L., & Rossini, F.A. (1983). Technological innovation and its as-
sessment. In S.S. Nagel (Ed.), Encyclopedia of policy studies. New York:
Marcel Dekker.
Quade, E.S. (1985). Analysis for public decisions (2nd ed.). New York: El-
sevier.
Raiffa, H. (1968). Decision analysis. Reading, MA: Addison-Wesley.
Rescher, N. (1980). Induction. Pittsburgh: University of Pittsburgh Press.
Rein, M. (1976). Social science and public policy. Baltimore: Penguin
Books.
Rein, M., & White, S.H. (1977). Policy research: Belief and doubt. Policy
Analysis, 3(2), 239-272.
Rose, R. (1977). Disciplined research and undisciplined problems. In C.H.
Weiss (Ed.), Using social research in public policy making (pp. 23-36).
Lexington, MA: D.C. Heath.
Saaty, T.L. (1980). The analytic hierarchy process. New York: McGraw-
Hill.
Sabatier, P.A., & Mazmanian, D.A. (1983). Policy implementation. In S.S.
Nagel (Ed.), Encyclopedia of policy studies. New York: Marcel Dekker.
Schon, D.A. (1983). The reflective practitioner. New York: Basic Books.
Schwenk, C.R., & Cosier, R.A. (1980). Effects of the expert, devil’s advo-
cate, and dialectical inquiry methods on prediction performance. Or-
ganizational Behavior and Human Performance. 26, 409-424.
Sieber, S.D. (1981). Fatal remedies. New York: Plenum.
Simon, H.A. (1973). The structure of ill-structured problems. Artificial
Intelligence, 4, 181-201.
Slater, P. (1927). Dimensions of intrapersonal space. London: Wiley.
Stabell, C.B. (1978. Integrative complexity of information environment
perception and information use: An empirical investigation. Organiza-
tional Behavior and Human Performance, 22, 116-142.
Stokey, E., & Zeckhauser, R. (1978). A primer of policy analysis. New York:
Norton
Strauch, R.E. (1976). A critical look at quantitative methodology. Policy
Analysis, 2, 121-144,
Sudman, S. (1976). Applied sampling. New York: Academic Press.
Taylor, S.E., & Crocker, J. (1980). Schematic bases of social information
processing. In E. Higgins (Ed.), Social cognition. Hillsdale, NJ:
Erlbaum.Dunn: Methods of the Second Type 737
Warfield, J.N. (1976). Societal systems: Planning, policy, and complexity.
New York: John Wiley.
Watzlawick, P., Weakland, J., & Fisch, R. (1974). Change: Principles of
problem formation and problem resolution. New York: W.W. Norton.
White, M.J. (1983). Policy analysis and management science. In S.S. Nagel
(Ed.), Encyclopedia of policy studies. New York: Marcel Dekker.
Whitehead, A.N., & Russell, B. (1910). Principia Mathematica, 1.
Cambridge: Cambridge University Press.
Wildavsky, A. (1979). Speaking truth to power. Boston: Little, Brown.
Wood, R.C. (1968). Foreword. In R.A. Bauer & K.J.Gergen (Eds.), The
study of policy formation. New York: The Free Press.
Zeckhauser, R., & Shaefer, E. (1968). Public policy and normative
economic theory. In R.A. Bauer & K.J. Gergen (Eds.), The study of policy
formation. New York: Free Press.Copyright of Policy Studies Review is the property of Policy Studies Organization and
its content may not be copied or emailed to multiple sites or posted to a listserv
without the copyright holder's express written permission. However, users may print,
download, or email articles for individual use.