CHAPTER 9

Policy Evaluation

Meaning of Policy Evaluation

¨ “Involves the estimation, appraisal, or assessment of a policy, its content,
implementation, goal attainment, and other effects.” (Anderson, 2011, p.271)

Policy Evaluation is the systematic collection and analysis of information to make judgments
about contexts, activities, characteristics, or outcomes of one or more domain(s) of the
Policy Process. Evaluation may inform and improve policy development, adoption,
implementation, and effectiveness, and builds the evidence base for policy interventions.

Policy evaluation contributes fundamentally to sound public governance. It can help

governments improve the design and implementation of public policies that can, in turn,
lead to prosperity for their country and well-being for its citizens. Policy evaluation
contributes to promoting public accountability, learning and increased public sector
effectiveness through improved decision-making.

Policy evaluation is conducted for checking the effects of the policies of respective ministries
and for evaluating the policies in terms of necessity, efficiency, validity, etc. to improve the
planning and implementation process. For this purporse, the Government Policy Evaluations
Act has been enforced since April 2002.

Objectives of conducting policy evaluations are following:

1. Realization of efficient and high quality public administration focusing on the citizens
2. Shift to performance-based public administration from the viewpoint of the citizens;
and
3. Thorough accountability to the citizens

Policy evaluation applies evaluation principles and methods to examine the content,
implementation or impact of a policy. Evaluation is the activity through which we develop
an understanding of the merit, worth, and utility of a policy.

Why evaluate policy?

1. To access policy achievement
2. To access policy consequences
3. To suggest policy change
4. To suggest policy termination

Purpose of Policy Evaluation

1. To access policy achievement
¨ to check the policy effects in general or of the respective ministries/departments:
¤ whether a policy is successful/failed
¤ to what extend a policy is successful/failed
¤ what factors has contributed towards policy success/failure

2. To access policy consequences

 ¨ whether a policy has other effects/not – intended and unintended:
¨ who gets benefits / bear cost directly and indirectly

3. To suggest policy change

to suggest policy recommendation
¨ to find out whether the policy is necessary, valid, and efficient etc., hence to improve
the planning and implementation of policy process.

4. To suggest policy termination

to suggest to stop implementing policy
to find out whether the policy is still relevent

How to evaluate policy

1. Administrative technique
2. Cost-Benefit Analysis
3. Systematic analysis
4. Institutional techniques

¨ What technique to use?

¨ One or combination of techniques may be used to evaluate policy outcome, output,
impact, process, and success /failure

1. Administrative (operational) technique

¨ Refer Anderson p. 276-282
¨ Evaluating the interrelated process, procedures, rules, resources involved in policy
implementation. Purpose is to ensure effieciency, tranparency and accountability in
implementation.
¨ Example:
¤ overlapping roles/function in implementing policy A with other policy
program/activities;
¤ ethical issues arise in implementing policy, honesty, integrity, and corruption

2. Systematic technique
¨ Policy is evaluated through scientific research, by the relevant group / agencies that
involve directly and indirectly in the implementation process.
¨ Example of systematic techniques are: case study, survey design, experimental
design, before and after study.

3. Cost Benefit Analysis technique

¨ A formal, quantitative evaluation technique
¨ Identify and weigh costs against benefits of policy or program
¨ E.g. Cleaner environment program -- the costs involve (opportunity cost) and
benefits are generated and compared

Cost-benefit analysis (CBA) is a technique used to compare the total costs of a

programme/project with its benefits, using a common metric (most commonly monetary
units). This enables the calculation of the net cost or benefit associated with the
programme.
CBA adds up the total costs of a programme or activity and compares it against its total
benefits. The technique assumes that a monetary value can be placed on all the costs and
benefits of a programme, including tangible and intangible returns to other people and
organisations in addition to those immediately impacted. As such, a major advantage of
cost-benefit analysis lies in forcing people to explicitly and systematically consider the
various factors which should influence strategic choice.

Example
In 2005 the UK Government undertook a value for money analysis of Government
investment in different types of childcare. The choice was between higher cost "integrated"
childcare centres, providing a range of services to both children and parents, or lower cost
"non-integrated" centres that provided basic childcare facilities.

The analysis included both a 'hard exercise' and a 'soft exercise'. The hard exercise
identified, quantified and monetised direct costs and benefits. The soft exercise identified
and described qualitatively non-monetisable impacts, leading to option ranking.

CBA Process
¤ Step 1: analyse costs
¤ Step 2: analyse benefits
¤ Step 3: Assign monetary values (RM) to the various cost & benefits
¤ Step 4: Allocate discount rate to equate present value for future effects
¤ Step 5: Compare overall cost and benfits
¤ Step 6: Decide option that maximize benefits/minimize costs

4. Institutional technique
¨ Evaluation is done by the various governmental and non-governmental agencies
base on their given (accepted) roles / functions:
¨ Examples: Parliament, Ministries, and Special Committees.
¨ Purpose is to evaluate the behavior of the policy implementors, hence to achieved
policy efficiency and effectiveness

Institutional technique: Examples

Example of institutions appointed by government for the purpose are:
¨ Public Accounts Committee (evaluate the administration and management of public
money),
¨ Ombudsman (investigate allegations of maladministration to ensure public
accountability – example – work delay),
¨ Media and interest group – to give feedback and suggestion to further improve
policy achievement / overcome implementation problems/constraints

What aspects of policy to evaluate?

1. Evaluating policy output
2. Evaluating policy outcome
3. Evaluating policy impact
 4. Evaluating policy implementation process
5. Evaluating policy successful / failure factors

Types of policy evaluation

Evaluating Policy Output, Outcome & Impact

1. Evaluating policy Output

i.e. to assess:
(a) who (or what department/organzation) are involve in a particular policy implemention
(b) what are they doing (their roles and functions) pertaning to a particular policy
implementation
Example:
Policy Who/What organization What Roles & Functions
Education Act, 1996 Ministry of Education (MOE)

Privatization policy PLUS, TNB, TM

2. Evaluating policy outcome

to appraise the result, the government (or a particular government agencies)
intended/unintended to see/get from a particular policy implementation.

The following questions is asked:

1. what has hapend to the public after a policy has been implemented (direct/intended
result) whether or not the targeted (and untargeted) public problem is resolve,
2. who gets benefit directly / indirectly from implementing the policy
3. who bear costs/ expenses (direct and indirect) incurred to comply with policy
requirement

Example:
Fiscal policy: Tax on cigarates
Intended outcome:
Number of smokers / number of cigarates smoked reduced
Number of illness realed to smoking reduced
Unintended outcome:
Amount of household expenses decrease/increases.

What are costs and benefits towards government? (direct/indirect and intended and
unintended)

3. Evaluating policy impact

to evaluate the intended and unintended, direct and indirect, positive and negative effects
from policy implementation towards society/ state.
assuming implementors (agencies) have implemented a policy – its objectives is achieved
(output), its intended result on the target group is achieved (outcome), then, what hapend
to the society at large / to the nation / state (impact)

Impacts are the externalities from the succesfully / unsuccesfully policy implementation –
the positive/negative, direct and indirect

What are impact of Vision 2020

Example: Privatization policy

TNB +
Output electricity
supply

Efficient
Outcome electricity
supply

Public
satisfaction
increases
Impact Economy
developed

4. Evaluating policy implementation process

Evaluating the means of by which a policy is delivered to target group, or the way a program
is implemented (process of implementation)
-assess client satisfaction and program activities
-seek to identify management problems if any
-e.g. Could the program be carried out more efficiently?

5. Evaluating policy success / failure

 q assessing impact on the original problems addressed by the program/policy.
- -e.g. Has the problems (e.g. crime, poverty, tardiness) been reduced because
of the program/policy?
¨ Policy evaluation may also seek to identify critical factors of success or failure so
remedial actions can be made

Problems / constraints of policy evaluation

¨ Objectives
¨ Defining criteria for success
¨ Side effects
¨ Data problems
¨ Methodological problems
¨ Political problems
¨ Cost
¨ Distribution of impact
¨ How much is enough

1. Objectives
¨ Generally, policy objectives are not stated objectively/ not measurable/ not rank
¤ If objectives are unclear or are not specified in any measurable form , the
criteria for policy success are also unclear
¤ When a statement of objectives are clear, specific, and reasonable goals is
attained is well stated, there are still a number of problems arise such as the
relative importance of goals and objectives

2. Defining criteria for success

¨ Generally, criteria for policy success not clearly defined
¤ Even if objectives are clearly stated, the questions of how the success of the
objectives is to be judged or measured would arise
¤ How to judge – what measurement to use - If direct measurement is not
practical a more indirect indicators may be used

3. Side effects
¨ There is always a side effects from a policy that would effects the success / failure of
other policy
¤ A program/policy under evaluation may be impacted by other
programs/policies
¤ Thus it may be difficult to identify the side-effects of any one program or
policy

4. Data problems
¨ Accurate and reliable data is required to measure success / failure / consequences of
policy
¤ the necessary information / data to assess the impact may be unavailable or
available in unsuitable form
¤ E.g. not all states kept a particular data or kept them in a desired format
6. Methodological problems
¨ Many methods available to measure policy success / failure; each has its own
strengths and weaknesses
¨ A problem or target population which is the target of several programs with the
same or related objectives is difficult to be evaluated since evaluators may not know
which program produces an effect if any
¨ E.g. poor people in an area are affected by several programs – PPRT, Jabatan
Kebajikan Masyarakat (programs / schemes) – which method to use to evaluate, if
the target population is better off, which policy contributed toward it

7. Political problems
¨ Some people are threatened by evaluation
¨ Evaluation may be seen as a threat to:
¤ the continuation of a policy or program in which a number of people have
stake
¤ the personal reputations
¤ the careers of politicians or administrators

8. Cost
¨ Substantial amount of cost incurred to evaluate policy
¤ It may take as much as 1% of the total program cost especially if
sophisticated methods are used (e.g. experiment)
¤ Can be a diversion from the delivery of the policy or program

9. Distribution of impact
¨ How to effectively measure policy impact distribution:
¤ whether it is actually going to the groups intended to benefit most
¤ A policy may not have equal impact to people. E.g. better educated people
are likely to be more knowledgeable or having the resources to maximize
advantage of policy benefits

10. How much is enough

¨ Even if objectives have been specified and priorities among them established,
questions remain of what outcomes are seen as relevant to meeting those objectives
and what level of achievement in meeting those objectives would make up success