POLICY

EVALUATION

TPL-5 MBM 1
 Policy Evaluation
a. Policy evaluation uses a range of research methods to
systematically investigate the effectiveness of policy
interventions, implementation and processes, and to
determine their merit, worth, or value in terms of
improving the social and economic conditions of
different stakeholders.
 Policy evaluation uses quantitative and qualitative
methods, experimental and non-experimental designs,
descriptive and experiential methods, theory based
approaches, research synthesis methods, and
economic evaluation methods.
 Policy evaluation for government privileges no single
method of inquiry and acknowledges the
complementary potential of different research methods.

TPL-5 MBM 2
Cont..

 The methods used in government

evaluation and analysis are usually
driven by the substantive issues at hand
rather than a priori preferences

TPL-5 MBM 3
 Types of evaluations
a. Summative evaluation (Sometimes called impact
evaluation) - asks questions about the impact of a policy,
programme or intervention on specific outcomes and for
different groups of people. Summative evaluation seeks
estimates of the effects of a policy either in terms of what
was expected of it at the outset, or compared with some
other intervention, or with doing nothing at all (i.e. the
counterfactual).
b. Formative evaluation (sometimes referred to as process
evaluation) - asks how, why, and under what conditions
does a policy intervention work, or fail to work? Formative
evaluations are important for determining the effective
implementation and delivery of policies, programmes or
projects.
TPL-5 MBM 4
 Formative evaluation typically seeks
information on the contextual factors,
mechanisms and processes underlying a
policy’s success or failure. This often
involves addressing questions such as
for whom a policy has worked (or not
worked), and why.

TPL-5 MBM 5
 Carry out the evaluation

 Organizer
 Purpose
 Key strengths
 Key weaknesses

TPL-5 MBM 6
 Summary of approaches for
conducting evaluations in policy
 Broad issues
 Identify and assess potential costs &
benefits of competing policies.
 Provide general direction for broadly
focused actions.
 Often corrupted or subverted by politically
motivated actions of participants.

TPL-5 MBM 7
Cont..

 Policy studies provide general guidance

and direction on broad issues by
identifying and assessing potential costs
and benefits of competing policies
 The drawback is these studies can be
corrupted or subverted by the politically
motivated actions of the participants.

TPL-5 MBM 8
 Evaluation methods and
techniques
 The analycentric approach focuses on individual
problems and its solutions; its scope is the micro-scale
and its problem interpretation is usually of a technical
nature. The primary aim is to identify the most effective
and efficient solution in technical and economic terms
(e.g. the most efficient allocation of resources).
 The policy process approach puts its focal point onto
political processes and involved stakeholders; its scope is
the meso-scale and its problem interpretation is usually of
a political nature. It aims at determining what processes
and means are used and tries to explain the role and
influence of stakeholders within the policy process. By
changing the relative power and influence of certain
groups (e.g enhancing public participation and
consultation),
TPL-5 solutions to problems
MBM may be identified. 9
Cont..
 The meta-policy approach is a systems and
context approach, i.e. its scope is the macro-
scale and its problem interpretation is usually
of a structural nature. It aims at explaining the
contextual factors of the policy process, i.e.
what are the political, economic and socio-
cultural factors influencing it. As problems may
result because of structural factors, e.g. a
certain economic system or certain political
institutions, solutions may include the change
of the structure itself.

TPL-5 MBM 10
 Methodology
 Policy analysis is methodologically diverse
using both qualitative methods and
quantitative method, including case studies,
survey research, statistical analysis, and
model building among others.
 One common methodology is to define the
problem and evaluation criteria; identify all
alternatives; evaluate them; and
recommend the best policy option.

TPL-5 MBM 11
Models of policy analysis
 Institutional model
 Public policy is determined by political institutions, which
give policy legitimacy. Government universally applies
policy to all citizens of society and monopolizes the use of
force in applying policy.
 Process model
 Policy creation is a process following these steps:
 Identification of a problem and demand for government
action.
 Formulation of policy proposals by various parties (e.g,
congressional committees, think tanks, interest groups).
 Selection and enactment of policy; this is known as Policy
Legitimation.
 Evaluation of policy.
TPL-5 MBM 12
 Rational model
 Policy is intended to achieve maximum social gain.
Rationally, the policy that maximizes benefits while
minimizing costs is the best policy. It is a part of
rational choice theory.
 Incremental model
 Policy is a continuation of previous government activity,
with minimal changes made to previous policy.
 Group model
 The political system's role is to establish and enforce
compromise between various, conflicting interests in
society.
 Elite model
 Policy is a reflection of the interests of those individuals
within a society that have the most power, rather than
the demands of the masses
TPL-5 MBM 13
Impact of policy
 Intended Effects
 The goals of policy may vary widely according to the
organization and the context in which they are made.
Broadly, policies are typically instituted in order to avoid
some negative effect that has been noticed in the
organization, or to seek some positive benefit.
 Corporate purchasing policies provide an example of
how organizations attempt to avoid negative effects.
Many large companies have policies that all purchases
above a certain value must be performed through a
purchasing process. By requiring this standard
purchasing process through policy, the organization can
limit waste and standardize the way purchasing is done.

TPL-5 MBM 14
Cont..
 The State of California provides an example of benefit-
seeking policy. In recent years, the numbers of hybrid
vehicles in California has increased dramatically, in
part because of policy changes that provide USD
$1,500 in tax credits as well as the use of high-
occupancy vehicle lanes to hybrid owners. In this case,
the organization (state and/or federal government)
created a positive effect (increased ownership and use
of hybrid cars) through policy (tax breaks, benefits).


TPL-5 MBM 15
 Cont..
Unintended Effects
 Policies frequently have side effects or unintended
consequences . Because the environments that policies
seek to influence or manipulate are typically complex
adaptive systems (e.g. governments, societies, large
companies), making a policy change can have
counterintuitive results.
 For example, a government may make a policy decision to
raise taxes, in hopes of increasing overall tax revenue.
Depending on the size of the tax increase, this may have
the overall effect of reducing tax revenue by causing
capital flight or by creating a rate so high, citizens are
disincentives to earn the money that is taxed.

TPL-5 MBM 16
Cont..
 The policy formulation process typically
includes an attempt to assess as many areas
of potential policy impact as possible, to lessen
the chances that a given policy will have
unexpected or unintended consequences.
Because of the nature of some complex
adaptive systems such as societies and
governments, it may not be possible to assess
all possible impacts of a given policy.

TPL-5 MBM 17
 Policy content
Policies are typically promulgated through official written
documents. Such documents have standard formats that
are particular to the organization issuing the policy. While
such formats differ in terms of their form, policy documents
usually contain certain standard components including:
 A purpose statement, outlining why the organization is
issuing the policy, and what its desired effect is.
 A applicability and scope statement, describing who the
policy affects and which actions are impacted by the
policy. The applicability and scope may expressly exclude
certain people, organizations, or actions from the policy
requirements
 An effective date which indicates when the policy comes
into force. Retroactive policies are rare, but can be found.

TPL-5 MBM 18
Cont..

 A responsibilities section,
indicating which parties and
organizations are responsible for
carrying out individual policy
statements. These responsibilities
may include identification of
oversight and/or governance
structures.
TPL-5 MBM 19
 Cont..
 Policy statements indicating the specific
regulations, requirements, or modifications to
organizational behavior that the policy is
creating.
 Some policies may contain additional sections,
including
 Background indicating the reasons and history
that led to the creation of the policy, which may
be listed as motivating factors
 Definitions, providing clear and unambiguous
definitions for terms and concepts found in the
policy document.
TPL-5 MBM 20
Projects to be evaluated?
These includes on the interventions and the target
passengers, markets, vehicles, infrastructures are
clear and identifiable;

a. The outcomes are clear, specific and measurable

b. That an appropriate evaluation design can be

implemented

TPL-5 MBM 21
Guiding Principles for evaluators
 The order of these principles does not imply priority
among them; priority will vary by situation and evaluator
role. The principles run as follows:
 Systematic Inquiry: Evaluators conduct systematic,
data-based inquiries about whatever is being evaluated.
 Competence: Evaluators provide competent
performance to stakeholders.
 Integrity / Honesty: Evaluators ensure the honesty and
integrity of the entire evaluation process.
 Respect for People; Evaluators respect the security,
dignity and self-worth of the respondents, program
participant, clients, and other stakeholders with whom
they interact.
 Responsibilities for General and Public welfare

TPL-5 MBM 22
Cont..

 Welfare: Evaluators articulate and take

into account the diversity of interests and
values that may be related to the general
and public welfare

TPL-5 MBM 23
 Policy studies

 Broad issues
 Identify and assess potential costs &
benefits of competing policies.
 Provide general direction for broadly
focused actions.
 Often corrupted or subverted by politically
motivated actions of participants.

TPL-5 MBM 24
Evaluation methods and techniques
 Evaluation is methodologically diverse using both
qualitative methods and quantitative methods,
including case studies, survey research,
statistical analysis, and model building among
others. A more detailed list of methods,
techniques and approaches for conducting
evaluations would include the following:
 Accelerated aging
 Action research
 Advanced Product Quality Planning
 Alternative assessment
 Appreciative Inquiry
 Assessment
TPL-5 MBM 25
Cont..
 Axiomatic design
 Benchmarking
 Case study
 Change management
 Clinical trial
 Cohort study
 Competitor analysis
 Consensus decision-making
 Consensus-seeking decision-making
 Content analysis

TPL-5 MBM 26
Cont..
 Conversation analysis
 Cost-benefit analysis
 Course evaluation
 Data mining
 Delphi Technique
 Discourse analysis
 Electronic portfolio
 Environmental scanning
 Ethnography
 Experiment
 Experimental techniques
 Factor analysis

TPL-5 MBM 27
 Factorial experiment
 Feasibility study
 Field experiment
 Fixtureless In-Circuit Test
 Focus group
 Force field analysis
 Game theory
 Grading
 Historical method
 Inquiry
 Interview
 Marketing research
 Meta-analysis
 Metrics
 Most significant change
TPL-5 MBM 28
 Multivariate statistics
 Naturalistic observation
 Observational techniques
 Opinion polling
 Organizational learning
 Participant observation
 Policy analysis
 Process improvement
 Project management
 Qualitative research
TPL-5 MBM 29
 Quality audit
Rubric
Sampling
School accreditation
Self-assessment
Six Sigma
standardized testing
Statistical process control
Statistical survey
Statistics
Strategic planning
Structured interviewing
Systems theory
Student testing
Total Quality Management
Triangulation

TPL-5 MBM 30
 Qualitative Evaluations
 The Qualitative evaluations are designed to "permit the
evaluator to study selected issues in depth and detail"
Patton (1990).It also helps the policy makers in
identifying and solving problems, differentiating good
or bad and also the risks related on policy. All these
covers the following outcomes:
a. Such depth and detail is usually necessary to
determine the appropriate questions to ask in an
evaluation, and to identify the situational and
contextual conditions under which a policy, programme
or project works or fails to work.
b. Qualitative methods of evaluation are particularly
important for formative evaluation, though they are
also used in summative evaluations.

TPL-5 MBM 31
Cont..

• Qualitative evaluation uses a range of

methods including in-depth interviews,
case studies, consultative methods,
focus groups, ethnography,
observational and participant-
observational studies, and conversation
and discourse analysis.

TPL-5 MBM 32
 Economic evaluation
methods
 Cost-effectiveness analysis
 Cost-benefit analysis
 Cost utility analysis
 Opportunity cost appraisal
 Deadweight and counterfactual appraisal

TPL-5 MBM 33
 EVALUATION
a. qualitative research methods provide such evidence:
i. Theory-based methods (including theories or change,
programme theory evaluation, ‘realistic’ evaluation
methods)
ii. Goals-based evaluation methods
iii. Goals-free evaluation methods
b. In-depth interviews
c. Focus groups (including stakeholder analysis)
d. Consultative techniques (including user satisfaction
studies)
e. Ethnographies
f. Observational and participant-observational studies
g. Conversation and discourse analysis
TPL-5 MBM 34
Cont..
g. What did the policy seek to achieve? Create or build
capacity, transfer ideas, and, or have policy impact?
h. Who did it seek to influence directly or indirectly?
i. Did the policy rely on policy’s entrepreneurs or
consultant? Were they located inside or outside
government?
j. What barriers to success were anticipated at the
outset?
k. Was the policy attempting to take advantage of the
opening of certain “policy windows

TPL-5 MBM 35
Cont..
l. Did the designated policy maker or consultants meet or
exceed expectations?
m. Did new transport policy makers allies for the policies
emerge?
n. Were there any unanticipated events or opportunities
in transport?
o. Were the anticipated transport policy’s influences
achieved?
p. Did alternative ones emerge?
q. What could be done differently in the future?
(The policy Hub (2004)

TPL-5 MBM 36
Ethnographies Techniques
 Direct, first-hand observation of daily behavior.
This can include participant observation.
 Conversation with different levels of formality.
This can involve small talk to long interviews.
 The genealogical method. This is a set of
procedures by which ethnographers discover
and record connections of kinship, descent and
marriage using diagrams and symbols.
 Detailed work with key consultants about
particular areas of community life.
TPL-5 MBM 37
Cont..

 In-depth interviewing.
 Discovery of local beliefs and
perceptions.
 Problem-oriented research.
 Longitudinal research. This is continuous
long-term study of an area or site.
 Team research.
TPL-5 MBM 38
 TRANSPORT
POLICY

THE FUEL PRICING POLICY

PRODUCED BY
THE ECONOMIC PLANNING UNIT
PRIME MINISTER DEPARTMENT
MALAYSIA Mustakim Al Haj

TPL-5 MBM 39
 END

TPL-5 MBM 40