Evaluation terms of reference

The evaluation terms of reference, or ToR, is a

planning tool that is developed directly before
the evaluation is conducted. It is much more
detailed than the summary evaluation table
and becomes a planning document for the
evaluation itself.

Evaluation ToR template

81
MEAL in project calendar

Gantt chart A bar chart

that illustrates a project
schedule, identifying the
start date, end date and
expected durations of all
activities.

82
83
Now that MEAL planning process is complete, the
next step is to get started with collecting data.
Timely, high-quality data are the foundation upon
which project teams can measure progress, make
decisions and learn. Data quality is an important
consideration for all MEAL practitioner.

84
 Data quality

You need to determine, with the help of your

stakeholders, what quality and quantity of data is
“good enough” for your decision-making, learning
and accountability needs. It is useful to consider
the following five data quality standards.

85
 Data quality
Validity Data are valid when they accurately represent what you intend to measure. In other
words, the data you collect helps you measure the indicators you have chosen.
Reliability Data are reliable when the collection methods used are stable and consistent. Reliable
data are collected by using tools such as questionnaires that can be implemented in the same way
multiple times. In practice, this means that if you use the same questionnaire to ask the same
person the same questions and nothing else has changed, you should get the same answer.
Precision Data are precise when they have a level of detail that gives you an accurate picture of
what is happening and enables you to make good decisions. For example, precise data allow you
to compare results between men and women, if this is important for your project. Precise data
are collected using appropriate sampling methods.
Integrity Data have integrity when they are accurate. Data should be free of the kinds of errors
that occur, consciously or unconsciously, when people collect and manage data. Following the
guidance on the design and implementation of collection tools and the management of the data
you collect will increase the integrity of your data.
Timeliness Timely data should be available when you need it for learning that informs decisions
and for communication purposes. Data are not useful to you when they arrive too late to inform
these processes.
86
Basic data collection techniques

87
Basic data collection techniques

An interview is a qualitative research method that relies on asking questions in

order to collect data. Interviews involve two or more people, one of whom is the
interviewer asking the questions.
Key informant interviews are qualitative in-depth interviews with people who know
what is going on in the community. The purpose of key informant interviews is to
collect information from a wide range of people—including community leaders,
professionals, or residents—who have first hand knowledge about the community
or subject.

88
Basic data collection techniques

Focus group discussion A guided discussion between respondents in a group. It is a

qualitative data collection tool designed to explore and understand the rich depth and
context of a group’s perspectives, opinions and ideas.

A focus group discussion provides an opportunity for a small group of participants to

interact with each other, build on ideas and present divergent perspectives. However,
while the exchange between group participants provides rich insights, a focus group
discussion requires more experience to facilitate. As well as an experienced facilitator,
they require a notetaker. Typically, a focus group includes 8 to 12 participants.

89
Basic data collection techniques

Observation is a technique where you observe participants and

phenomena in their most natural settings. This enables us to make
choices and react to situations in their natural setting, as opposed
to structured settings like research labs or focus groups.

90
 Quantitative data collection

Questionnaire A structured set of questions designed to elicit specific

information from respondents.

Closed-ended questions are questions that provide a predefined list of

answer options. This makes it easier for responses to be coded
numerically allowing for statistical analysis.

91
Types of close-ended questions (Questionnaire)

92
 Qualitative data collection:

Semi-structured interview A guided discussion between an

interviewer and a single respondent designed to explore and
understand the rich depth and context of the respondent’s
perspectives, opinions and ideas.

Focus group discussion A guided discussion between respondents in a

group. It is a qualitative data collection tool designed to explore and
understand the rich depth and context of a group’s perspectives,
opinions and ideas.

93
 Qualitative data collection:

Open-ended questions are those that allow someone to give a free-

form response in their own words.
There are two types of open-ended questions:
Content-mapping questions are also known as opening questions. These
are intended to initiate the exploration of a topic by raising and broadly
exploring an issue.
“What are the major barriers IDPs face when accessing potable water?”

Content-mining questions are also known as probing questions. These are

follow-up questions that elicit more detail or explanation about a
response to a content-mapping question.
“Can you tell more about why you said, ‘don’t feel safe?’” and “Can you
provide an example?”
94
Group exercise (15 min):

FGD guiding questions- Any 3

96
Example of focus group discussion guiding questions

97
 Creating samples
Sample A subset of the population or community that you choose to study that will help
you understand the population or community as a whole.
Population A set of similar people, items or Margin of error expresses the maximum expected
events that is of interest for some question or difference between the true population and the sample
experiment. estimate. To be meaningful, the margin of error should
be qualified by a probability statement (often
Sampling unit The individual person, category expressed in the form of a confidence level).
of people, or object from whom/which the
measurement (observation) is taken. Confidence level refers to the percentage of all possible
Examples of sampling units might include samples that can be expected to include the true
children under 5 years of age, adolescents, population parameter.
women, men, households, etc.
Sample size The sample size references the Sample frame A specific list of units (men, women,
total number of respondents included in a households, individuals, children, adolescents, etc.)
study, and the number is often broken down that you will use to generate your sample. Examples
into sub-groups by demographics such as age, could be a census list or a list of employed teachers, a
gender, and location so that the total sample registration log or a list of project participants.
achieves represents the entire population. 98
Example
Karnaphuli IDP project conducts a survey learns that “75% of respondents report
“washing their hands after critical times.” The confidence level for the survey is cited as
95 percent, with a margin of error of plus or minus 3 percent.

This information means that if the survey were conducted 100 times, the percentage that
reported “washing their hands after critical times” would range between 72 and 78
percent most (95 percent) of the time.

Sample size calculator

99
Sampling can be divided into two basic types: random sampling and purposive
sampling.
Random sampling is a probability sample that includes respondents selected from a list of the
entire population of interest so that each respondent has an equal chance of being selected.

Step 2:
Step 1: Defining Step 3:
Choosing a Step 4:
your population Determining
method to Selecting your
and the your sample
calculate your sample units
sampling unit size
random sample

100
101
Purposive (selective) sampling is a non-
probability sample where sampling units that
are investigated are based on the judgement of
the researcher/evaluator. Sampling units are
selected based on characteristics of a
population and the objective of the study.

103
104
Data entry

The term “data entry” means putting the data you have collected into a
form you can use by entering it into an electronic database.

Quantitative • MS Excel, MS Access

Data Entry • SPSS, Stata KoboToolbox is
a mobile data
collection tool
Qualitative • MS Excel, MS Word that uses to
Data Entry • Atlas.ti, NVivo, Dedoose, MAXQDA etc. capture data
in real time
basis and for
simple analysis
105
Data cleaning
Detecting and removing errors and inconsistencies from data to
improve its quality.

Conducting quality
Identifying outliers
checks

Data cleaning
methods

Removing duplicate Keep a record of data

entries errors

106
107
The four levels of measurement

108
Quantitative data analysis basics

At a basic level, there are two kinds of quantitative analysis: descriptive

and inferential

Descriptive data analysis

• is the analysis of a data set that helps you describe, show or summarize data in a
meaningful way so that patterns might emerge.

Inferential data analysis

• enables you to use data from samples to make statistical generalizations about the
populations from which the data were drawn.

109
Categories of calculations

Three categories of calculations used to

analyse data using descriptive statistics

Measures of frequency
• Display the number of occurrences of a particular value(s) in a data set
(frequency tables, cross-tabulation tables).
Measures of central tendency
• Calculate the center value of data sets (mean, median, mode).
Measures of variability
• Determine the extent to which data points in the data set diverge from
the average, and from each other (range).

110
Measures of frequency

Frequency table: Water access

Cross-tabulation table: Level of satisfaction

with household water access

111
Measures of central tendency
The mean

Raw data: Meters walked to collect water

The median

The mode

Ms excel helps to calculate central tendency automatically 112

Which measure of central tendency should you use?

113
Measures of variability

Range The difference between the lowest and highest values of a data set.

The longest distance walked is 2,000 meters, and the shortest distance is 100
meters. Thus, the range is 1,900 meters.
Standard deviation calculates how far responses differ (deviate) from the mean
(average).

A high standard deviation indicates that the data set’s values differ greatly from
the mean. A low standard deviation means that values are close to the mean. A
zero standard deviation means that the values are equal to the mean.
For example, if the average height of Bangladeshi men is 64 inches, with a
standard deviation of 3 inches, then most men have a height between 3 inches
taller and 3 inches shorter than the average (61"–67").

114
Inferential analysis

Inferential analysis helps:

1. Compare the significance of differences between groups:

Determining whether the differences that exist between subgroups are
large enough to matter.

2. Examine the significance of differences between variables to

determine correlation and, potentially, causation: Determining
whether your activities contributed to the changes you are seeing.

115
Exploring the significance of differences between subgroups

116
Examining differences between variables to determine correlation and causation
Regression analysis gives you an understanding of how changes to variable(s) affect other variable(s). “Regression
analysis is a way of mathematically sorting out which of those [independent] variables does indeed have an impact [on
your dependent variable]. It answers the questions: Which factors matter most? Which can we ignore? How do those
factors interact with each other? And, perhaps most importantly, how certain are we about all these factors?
Regression analysis gives you an understanding of correlation. In other words, this type of analysis will give you a sense of
how closely your variables are related.

Correlation A statistical measure (usually expressed as a number) that describes the size and direction of the relationship
between two or more variables
For example, regression analysis could possibly tell you the different correlations between the reduction in waterborne
disease rates (your independent variable) and the use of two prevention methods: provision of potable water and
handwashing campaigns (your dependent variables). The analysis will also give you an understanding of the strength of
this correlation. If it is strong, then you can be more confident that your intervention is related to the changes you are
seeing. It is important to note that correlation does not necessarily imply causation.

Causation for example, if your analysis shows a correlation between handwashing messaging, improved handwashing
practices, and the reduction of waterborne disease, you can’t necessarily say that your project caused these changes.
It is extremely difficult to prove causation—saying with 100 percent certainty that your project caused a particular change.
This is especially true when working in the “real world,” outside of a laboratory environment.

117
 Qualitative data analysis basics

Step 1: Code
Step 2: Index Step 3: Frame
data: Begin to
data data
identify themes

119
Example:
Qualitative analysis matrix: “Access to water meets my household consumption needs.”

120
Data Visualization

121
Data visualization helps share detailed insights into data in the quickest and
most efficient way. This helps with:
● Analysis
● Interpretation
● Communication

Step 2: Define
Step 1: Define Step 3: Design Step 4: Build
the data
the and test your your
visualization
stakeholder(s) visualization visualizations
content

122
 GOOD DONUT

Donut Chart A B

• The Donut Chart shows the

numerical proportion to a
whole
• Input:
A 27%
B 63%
Bar Chart
Bar Graph
100%
90%
• The Bar Chart shows 80%
categorical data grouped into 70%

discrete groups 60%

50%
• Input: 40%
30%
Regions % of Projects
20%
A 10%
10%
B 15%
C 18% 0%
A B C D E F G
D 50%
E 57%
F 59%
G 100%
 Stacked Bar 100%
Stacked Bar Chart

Chart 90% 4% 12% 20%

10% 30%
80%

70% 20%
• The Stacked Bar Chart shows 42% 15%
the data’s proportion of 60%
40%

% of Projects
67%
values they represent while 50%

comparing across grouped 40%

data 30% 60%

55%
• Input: 20% 42% 38%
Intensive Advocacy Moderate Advocacy Little or No Advocacy 10% 20%
Region A 20% 67% 4%
Region B 42% 42% 12% 0%
Region A Region B Region C Region D Region E
Region C 60% 20% 20%
Region D 38% 40% 10% Regions
Region E 55% 15% 30%
Intensive Advocacy Moderate Advocacy Little or No Advocacy
 Error Bar
Error Bar Chart 50%

45%

40.0%
40%

• Error Bars show the 35%

35.0%

variability of data. They 30%

30.0%

indicate the error or

25%
uncertainty in a reported
measurement 20%

• Input: 15%

10%
10%
Means CI Range 7%
5%
5%
Program A 35.0% 10%
Program B 30.0% 7% 0%
Program A Program B Program C
Program C 40.0% 5%
Means CI Range
Benchmark chart
Benchmark chart
70% 67%
• The Benchmark chart shows
comparison between 60% 58%
benchmarks and actual data 55%

• Input: 50%

40%
Performance Benchmark
Project A 55% 50%
30%
Project B 58% 50%
Project C 67% 50%
20%

10%

0%
Project A Project B Project C

Performance Benchmark
128
Using data is about
much more than
generating required
reports

129
Adaptive management?
is an intentional approach to making decisions and
adjustments to the project in response to new
information and changes in context.
In order to contribute to project improvements, MEAL information should
be used as part of ongoing project decision-making. Adaptive
management, encourages and supports this process.
Project managers need accurate, relevant and timely information to:
● Assess project progress.
● Inform ongoing problem-solving and good management decisions.
● Understand the perspectives of participants and their levels of
satisfaction with the project.
● Address feedback raised by community members, both participants
and non-participants. 130
A culture of adaptive management results from a series of intentional investments related to project
design, staffing, budgeting, decision-making and more.
Is your project designed to promote adaptive management?

131
Progress reporting
High-quality, transparent reports in line with your donor or other internal and external stakeholder
requirements are vital.

Good reporting captures and explains both the successes and the challenges facing the project, and offers
evidence of robust evaluative thinking in the search for solutions. Guidance for reporting;

Consult your project

Identify donor
communications Identify or develop
reporting
plan and data flow report templates
requirements
map

132
Project Monitoring Reporting guideline

133