Impact evaluation ‘elective’ course

PADM7239A

Overview of the module

27 July 2023 | @The Wits School of Governance |

Johannesburg | Gauteng, South Africa

Kambidima Wotela, PhD (UCT)

▪ Presentation outline
▪ About the module
▪ Learning outcomes and content plan
▪ Teaching and learning plan
▪ Assessment plan
▪ Correspondence
▪ About the facilitators

Overview of the module

▪ Institution
▪ The University of the Witwatersrand (WITS)
▪ Faculty of Commerce, Law, and Management (CLM)
▪ WITS School of Governance (WSG)

▪ Programme
▪ Masters of Management in the field of Governance:
Monitoring and evaluation

▪ Module
▪ Impact evaluation elective (PADM7239A)
▪ This is a ten (10)-credit points course with six (06)
contact/virtual sessions of 3.5 hours each

About the module

▪ It is inevitable that we track and assess development
interventions
▪ We use evaluation to assess interventions, that is,
▪ Design or formative evaluation
▪ Implementation or process evaluation
▪ Summative evaluation (outcomes and impact)

▪ We use evaluation, including impact evaluation, to assess

interventions for …
▪ Effectiveness
▪ Sustainability
▪ Relevance
▪ Efficiency

About the module

▪ This course is a course on impact evaluation of interventions
▪ Being an elective and introductory course, we focus on
▪ Notable features of impact evaluation
▪ The design as well as procedure and methods of impact
evaluation

About the module

▪ At the end of this course, participants should understand
what impact evaluation is
▪ At the end of this course, participants should have an idea on
how they can do an impact evaluation of their own

▪ First, we contextualise monitoring and evaluation

▪ Second, we differentiate (i.) evaluation from monitoring

and, thereafter, (ii.) summative evaluation from formative
evaluation and process evaluation, and eventually (iii.)
impact evaluation from outcomes evaluation

▪ Third, we shall then discuss impact evaluation using a

systems thinking approach

▪ Lastly, we shall discuss the designs as well as procedure

and methods of impact evaluation
7

Learning outcomes and content plan

▪ Contextualising impact evaluation
▪ Generally, … as part of monitoring and evaluation
▪ Specifically, … as a summative evaluation

▪ An introduction to impact evaluation

▪ The six questions (systems thinking approach)
▪ What are the key issues in impact evaluation

▪ Research as an impact evaluation process

▪ Strategy and design
▪ Procedure and methods

▪ Notable impact evaluation design, procedure and methods

Learning outcomes and content plan

▪ Lesson plan
Lesson_01_Contextualising impact evaluation
Dr Kennedy Manduna

Lesson_02_An introduction to impact evaluation

Dr Thandiwe Letsie

Lesson_03_Research as an impact evaluation process

Dr Sindi Msimango

Lesson_04_Notable impact evaluation design, procedure and

methods: Part 1
Prof Kambidima Wotela

Lesson_05_Notable impact evaluation design, procedure and

methods: Part 2
Dr Kennedy Manduna

Lesson_06_Notable impact evaluation design, procedure and

methods: Part 3
Dr Sindi Msimango

Learning outcomes and content plan

▪ Prescribed reading
▪ Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., &
Vermeersch, C. M. (2016). Impact evaluation in practice,
the International Bank for Reconstruction and
Development/the World Bank.

▪ Khandker, S. R., Koolwal, G. B., & Samad, H. A. (2010).

Handbook on Impact Evaluation: Quantitative Methods and
Practices, the International Bank for Reconstruction and
Development/the World Bank.

10

Learning outcomes and content plan

▪ Recommended reading
▪ Markiewicz & Patrick (2015) Evaluation questions—
Determining what we want to know
▪ World Health Organisation (2013) Evaluation practice
handbook
▪ Barrett (2008) Principles and practice of evaluation

11

Learning outcomes and content plan

▪ Blended learning (Flipped classroom) so that most
routine learning (theoretical) will occur outside the
classroom
▪ Beforehand, we will post the relevant material on ULWAZI
accompanied by detailed instructions

▪ Second, as we prepare for class, we should collaborate

through the ULWAZI ‘discussion’ platform. This way you do
not have to wait until class to ask questions

▪ Third, the first part of class is reviewing the online

formative assessment using ‘quizzes’ platform

▪ Fourth, the second part of class in most cases is a group

presentation

13

Teaching and learning plan

▪ CONTINUED … Blended learning (Flipped classroom) so
that most routine learning (theoretical) will occur outside
the classroom
▪ Fifth, the third part of class is the facilitator driven
discussion meant to …
▪ Highlight the key points
▪ Address problem areas raised on the ‘discussion’
platform
▪ Fill in the gaps left by the reading material

▪ We should have already read the material in our own time

before coming to class

▪ Raised the problem areas on the discussion platform

▪ Done the pre-class preparatory test responses 14

Teaching and learning plan

▪ Tour of Ulwazi ...

15

Teaching and learning plan

▪ Ulwazi
▪ 24-7, 27th July to 28th September 2023

▪ Face-to-face
▪ Not applicable

▪ Virtual (Online)
▪ Thursdays, 27th July to 31st August 2023

16

Teaching and learning plan

Assessment Proportion Due date Effort
Assignment_1.1_Pre/post 15 per cent *Scheduled Read material
class online tests for each
Lesson
Assignment_2.1_Group 10 per cent *Scheduled Preparing the
presentations presentations
and questions
Assignment_3.1_Group 10 per cent 08 Sep 2023 Source, select,
Microsoft Excel and
spreadsheet summarise
literature
Assignment_3.2_Individual 20 per cent 18 Sep 2023 Select and
Microsoft Excel synthesis
spreadsheet literature
Assignment_3.3_Individual 45 per cent 28 Sep 2028 A discussion of
Microsoft Word document impact
write-up evaluation
using a
systems
thinking
approach and
then applying
our
understanding
of evaluation

18

Assessment plan
▪ Learning theme
▪ Using how others have done their impact evaluations to
think about our own impact evaluation
▪ The third set of hand-ins are also meant to provide us with a
structured approach to reading, reflecting, and writing as well
as correct referencing (in-text and reference list)
▪ We shall be discussing impact evaluation
▪ Interrogate an example of our own choice to spell out the:
▪ Aim and objectives
▪ Approaches employed
▪ Results yielded
▪ Conclusions drawn

19

Assessment plan
▪ In sum …
▪ Formative assessment
▪ Are we reading? … Pre-class preparation tests
▪ Are we understanding what we are reading? … Class
presentations
▪ Can we relate what we have learnt? The phased-out
assignment (Parts 1 and 2)
▪ Summative assessment
▪ Can we relate what we have learnt? The phased-out
assignment (Part 3)

20

Assessment plan
▪ Group presentations and submissions
▪ I have proposed three groups of about three (3) to six (6)
members each
▪ Each group has to prepare and present on a given topic
▪ Each group has to do a group assignment

22

Assessment plan
▪ Group(s) ALL —Session/Lecture 2 (2023-08-10)
▪ Discussing impact evaluation based on Chapters 1 and 2
of Gertler and colleagues. 2016. Impact evaluation in
practice
▪ Describing impact evaluation?
▪ What is the purpose of impact evaluation?
▪ What are the established facts in impact evaluation?
▪ What are the key issues and debates in impact
evaluation?
▪ What are the components (structure and function) of
impact evaluation?
▪ What are the processes in impact evaluation?

23

Assessment plan
▪ Group(s) 1 and 10—Session/Lecture 4 (2023-08-24)
▪ Discussing causal inference and counterfactuals (Chapter 3)
and randomisation assignment (Chapter 4) of Gertler and
colleagues. 2016. Impact evaluation in practice
▪ Description
▪ Purpose
▪ Steps in carrying out
▪ Strengths
▪ Limitations

24

Assessment plan
▪ Group(s) 2, 5, and 9—Session/Lecture 5 (2023-08-31)
▪ Discussing Instrument variables (Chapter 5) and
Regression discontinuity design (Chapter 6) of Gertler and
colleagues. 2016. Impact evaluation in practice
▪ Description
▪ Purpose
▪ Steps in carrying out
▪ Strengths
▪ Limitations

25

Assessment plan
▪ Group(s) 3 and 8—Session/Lecture 5 (2023-08-31)
▪ Discussing difference-in-difference (Chapter 7) and
matching (Chapter 8) of Gertler and colleagues. 2016.
Impact evaluation in practice
▪ Description
▪ Purpose
▪ Steps in carrying out
▪ Strengths
▪ Limitations

26

Assessment plan
▪ Group(s) 4, 6, and 7—Session/Lecture 6 (2023-09-07)
▪ Discussing how to implement an impact evaluation
(Chapters 11 through 14) of Gertler and colleagues. 2016.
Impact evaluation in practice
▪ Choosing an impact evaluation method
▪ Managing and impact evaluation
▪ The ethics and science of impact evaluation
▪ Dissemination results and achieving policy impact

27

Assessment plan
▪ Preparation for the series of assignments
▪ Think of any intervention whose impact you would want to
measure
▪ Describe this intervention
▪ Point out why you would like to assess its impact

28

Assessment plan
▪ All questions and comments on the teaching and
learning material should be on Ulwazi discussion
platforms

▪ Correspondence of a personal nature

▪ kambidima.wotela@wits.ac.za copying in
▪ kambidima.wotela.wsg@gmail.com and
▪ Lehlohonolo.Mmolotsane@wits.ac.za (the coordinator Mr
Lehlohonolo Mmolotsane )

▪ Please note that some issues are better handled by the

coordinator rather than me

30

Correspondence
▪ In any human interaction, there will be disagreements.
The most important thing is to resolve this disagreement
▪ First, engage me copy in the coordinator as well as the
Academic Delivery Unit Manager, Bindu Cherian
(Bindu.Cherian@wits.ac.za)

▪ Second, if not resolved, write to the Programme Leader

and Convenor copying in the class representative as well
as the Academic Delivery Unit staff

▪ Third, if not resolved, write to the Academic Director

copying in the Programme Leader and Convenor and the
Programme Coordinators as well as the Academic Delivery
Unit Manager

▪ Lastly, if not resolved, you should then write to the Head

of School, thereafter then the Dean, and then finally the
Vice Chancellor in this order
31

Correspondence
▪ In any human interaction, there will be disagreements.
The most important thing is to resolve this disagreement
▪ Try as much as possible not to skip any of these grievance
steps so that the issue is resolved amicably
▪ Please remember that any of these mentioned will be
consulted when you present your case to their superior
▪ If they are unaware of the problem their superior will
refer you back to him/her

32

Correspondence
▪ Dr Letise, Thandiwe
▪ Dr Manduna, Kennedy
▪ Dr Msimango, Sindi
▪ Prof Wotela, Kambidima

34

About the facilitators

Thank you very much for your audience
▪ Link to lecture recording
▪ [https://witscloud-
my.sharepoint.com/:v:/g/personal/a0006907_wits_ac
_za/EYnv69V1qrBAhWSAiUmGbk4B5D-
4SikgY5uoLp_X3ie5GA]

36

Lecture recording