THE SUPPORTING
ENVIRONMENT FOR M&E
A monitoring and evaluation (M&E) system is usually harder to manage and implement than a one-off
review or evaluation. This is because it relies on a system, instead of being a specific task carried out by
individuals over a defined period. For an M&E system to work well, it is important to create a supportive
environment, within which accurate and honest information can spread freely between interested parties.
There is a big difference between carrying out a discrete
task, such as a traditional evaluation, a review, or an impact
assessment, and implementing a monitoring and evaluation
(M&E) system. An M&E system is often considered more
difficult to do well because it relies on a system, instead of
being a one-off effort, carried out over a defined period by
individuals with a specific task (Gosling and Edwards 1995,
Simister 2000).
To serve a useful purpose, an M&E system needs to
generate information and analyses that are accurate and
relevant. Unfortunately, however, information produced by
CSOs throughout the world is often of poor quality. This is
sometimes because of technical issues, such as setting
inappropriate objectives or indicators, failing to conduct
baselines where needed, or using unsuitable tools and
methodologies.
But most M&E systems do not succeed or fail for technical
reasons. Instead, success or failure is more likely to be
conditional on whether or not a CSO has managed to create
an appropriate supporting environment, through which
accurate and honest information can be generated and
shared freely between interested parties.
Dealing with bias
They are many potential reasons why an M&E system
might generate incomplete or inaccurate information.
Sometimes, CSOs are unable to access the information they
need because it is too difficult or expensive to collect.
Sometimes they collect the wrong information. For
example, they may collect indicators that do not provide
the right kind of evidence to show whether desired or
anticipated changes have taken place. And sometimes they
are forced or requested to collect information that is of
little or no value to them to suit the needs of others, such
as donors or host governments.
Often, however, CSOs generate biased information that
leads to incorrect conclusions. There are many types of
bias, and they are called by different names. Some of the
more common ones are as follows (see Dozois et. al. 2010,
White and Phillips 2012).
• Framing bias occurs when people’s values, beliefs,
culture or attitudes affect how they interpret
events, issues or behaviours.
Case study: The truth game
During workshops, INTRAC sometimes conducts an exercise in
which people are asked in pairs or groups to think about their
daily lives, and discuss under what circumstances they would tell
lies – or at least fail to tell the whole truth.
There are two main purposes to this exercise. The first is to
emphasise the point that people do not always feel able or willing
to tell the complete truth. When carrying out M&E work it is often
assumed that people will generally tell the truth unless they have
a reason not to. But if we acknowledge there are many times
when we do not tell the full truth in our daily lives then we also
need to recognise that we cannot accept at face value every piece
of information arising from M&E work.
The second purpose of the exercise is to examine the conditions
under which people might not be wholly truthful. Findings reveal
that people are more likely to be dishonest when questioned:
• if they do not know (or trust) the person asking them
questions;
• if they think they (or friends or colleagues) will suffer if
they tell the truth;
• if they fear the truth will embarrass, upset, disappoint
or offend their audience;
• when they do not know how the information they
provide will be used;
• if they don’t think anyone is particularly interested in
that information, especially if people haven’t acted on
information they have provided in the past;
• if self-pride is at stake, and they don’t want to be
embarrassed;
• when issues are sensitive, and they don’t feel
comfortable discussing something;
• when they feel a certain answer is expected;
• when they feel threatened or bullied; or
• if giving a correct answer would involve too much time
and effort (in other words when they can’t be
bothered to find out or explain the truth).
This partly explains some of the biases described in this paper. It
also provides a basis for identifying processes that can help to
encourage more honest data collection and analysis.
• Confirmation bias happens when people only
notice or focus on evidence that supports what
they already believe, or want to be true, and
ignore contradictory evidence.
• Selection bias happens when people make
generalisations based on information acquired
© INTRAC 2019
 from individuals or groups that are not
representative of wider populations.
• Courtesy bias occurs when people being
interviewed provide information based on what
they think a questioner wants to hear, rather than
what they actually think.
• Social acceptability bias happens when people
provide responses based on what is considered
socially acceptable in a particular time and place.
Some of these biases are difficult to counter, and cannot be
eliminated completely, especially within an M&E system
where different people are routinely collecting and
analysing information on an ongoing basis. But CSOs can
develop processes to help minimise these and other biases.
Some of these processes are described below (based on
Simister 2000).

Processes for enhancing the quality of collection and analysis

(These guidelines can apply to staff working within an operational CSO, or staff working within partner organisations. The guidelines may also
apply to beneficiaries involved in participatory M&E systems. However, this is dealt with in a separate paper within the M&E Universe.)

Training, supervision At the most basic level, the collection and analysis of accurate information depends on the skills of
and motivation the staff responsible for carrying out the work, and on how well they are supervised and trained.
Well-trained and supervised staff are usually more capable of collecting high quality information, and
better able to understand and offset some of the biases described above, especially when assessing
social change, or identifying key lessons learned. There is much a CSO can do to support staff. Some
CSOs develop training materials to be used during induction or later on. Resources can range from
formal guides and manuals to interactive media and workshop materials. Training materials should
be designed to help staff recognise what is required of them under an M&E system, and provide
them with the knowledge and skills necessary to carry out required tasks. More informal training can
also be administered. This can range from peer-to-peer support and mentoring through to on-
demand assistance from outside experts.

Awareness of the It is important to ensure that staff know how the data they collect is used. Many studies (e.g. Connor
purpose of (1993)) have shown that staff are much more likely to spend time and effort collecting high quality
monitoring and data if they understand the value of the work they do, and are confident that those whose job it is to
evaluation analyse and use that data will take account of it, and act upon it if necessary. As well as
understanding the overall purpose of M&E within an organisation, programme or project, staff also
need to thoroughly understand their own roles and responsibilities. This means ensuring these are
clearly laid out in job descriptions and work plans.

Non-threatening To be effective, monitoring and evaluation need to be non-threatening exercises. If staff feel that
monitoring they are likely to be punished for making mistakes, or for failures in their projects or programmes,
they will try not to report them. At best, this leads to staff exaggerating successes and ignoring
failures; at worst, they may claim a project or programme is doing much better than it really is.
However, it is not enough simply not to penalise people. CSOs need to actively take steps to create an
environment in which different stakeholders feel they can be completely open and honest when
collecting, analysing, sharing and reporting information. This applies to staff within organisations, to
partners, and to beneficiaries.

Information flows It is also important to ensure that information flows both to and from the field. Nothing is worse than
a system where people collect information at field-level and pass it up to higher levels for analysis
and decision-making. Anyone participating within M&E work – whether staff, partners or
beneficiaries – should receive feedback on analyses or decisions made at higher levels. A monitoring
system should therefore encompass not only the collection of data but also the ‘communication
system in which information flows between all the people involved’ (Gosling and Edwards 1995, p86).
A two-way flow of information has two important benefits. Firstly, it ensures that data is actually
analysed and used, and so serves an important function. Secondly, the feedback of results and
analyses to the field allows staff to see how the data they provide is used.

Opportunities to If an M&E system is to progress beyond a rigid process in which data is collected at field level and
comment on data analysed at higher levels, mechanisms need to be established allowing field staff to participate in the
and analyses analysis process. If field staff are given the opportunity to comment on the data they produce, and
resulting analyses, through regular review mechanisms, their sense of ownership of their work is
likely to increase, and so therefore is the quality of that work. CSO staff at all levels should therefore
be involved in informal review processes. Reviews should be regularly organised at a variety of
different levels, and may need to be institutionalised throughout an organisation, programme or
project.

© INTRAC 2019
 Informal monitoring No formal M&E system can cover all the different eventualities that might occur during a project’s
channels lifetime. Instead, project managers and field staff often spend much of their time collecting
information on a range of matters not covered by pre-defined indicators. These could include
changes in the wider environment – such as important political events or changes in weather
conditions – unanticipated problems or opportunities affecting the implementation of projects, and
any unexpected impacts of the projects themselves, including effects on beneficiaries. However, in
order for this kind of information to be systematically identified, and any lessons learned fed back
into future activities in other areas, channels of communication need to be established through which
information can be passed to higher-level managers on an irregular basis. This requires the co-
operation and commitment of field staff, who will only volunteer information if they are confident
that it will be considered and used if necessary. It is therefore important to reassure staff at all levels
of a CSO that their experience, knowledge and opinions are valued.

Use and analysis of A golden rule is that the quality of information collection and analysis is highest when the people
data at field level carrying out the work actually need the information for their own purposes. Staff are therefore much
more likely to spend time and effort collecting high quality data if it is useful to them in their own
work. If staff are encouraged to make their own analyses of M&E data, and act upon those analyses
where necessary, they will no longer be collecting data entirely for the benefit of others, but to fulfil
their own needs and requirements as well. The more that M&E information is used by staff in the
field, the greater interest they will have in ensuring that it is accurate, and that analyses are valid.

Decentralisation of If partners, field-staff and/or beneficiaries are to be encouraged to use data for their own purposes,
decision-making there are two major implications. Firstly, they need to be allowed a say in the design of M&E systems.
This is to ensure that at least some of the data they collect is relevant to their own needs. Secondly,
decision-making needs to be decentralised to enable partners, staff or beneficiaries to act on the data
and analyses they generate. This may mean developing and encouraging participatory M&E systems
that facilitate the active involvement of different stakeholders.

Organisational Whilst formal processes and systems are important, the effectiveness of any M&E system will
culture ultimately depend on whether or not a CSO is able to develop an organisational culture that supports
effective M&E processes, and encourages learning and constructive criticism (Save the Children
2004). This means ensuring that a CSO is able to look at its own performance in a self-critical light,
and report honestly on errors or failures to other stakeholders, all the way from donors and senior
management through to service users. It also means cultivating a sense of curiosity in a CSO, so that
staff and partners are interested in the kind of results they are achieving, and what they are learning
along the way. This culture cannot be developed simply by imposing formal systems and procedures
on different stakeholders. An appropriate M&E culture can take many years to develop.

Effective leadership It is ultimately the responsibility of a CSO’s leadership to ensure that this kind of culture is developed.
on M&E Leadership is responsible for conveying the vision of an organisation, programme or project (the
direction in which it is going) to its staff, and organising their efforts to pursue that vision. It is also
responsible for creating the conditions under which stakeholders at all levels feel that M&E work is
valued, and feel that they can contribute fully within M&E processes. It is therefore particularly
important that leadership encourages open, honest information collection and analysis at the top
levels of an organisation, programme or project.

The processes listed above are all important. Some of them
can be accomplished relatively easily. For example, it is
fairly easy to put in place measures designed to ensure that
staff are properly trained and supervised. On the other
hand, an appropriate M&E culture can take years to
establish. But unless these issues are addressed within
CSOs, M&E systems can quickly degenerate into ‘zombie’
systems where information is collected mechanically, but
ends up unanalysed and unused.
At the same time, whilst CSOs need to ensure that the right
processes are in place, they also need to accept that it is
rarely possible to get perfect information, especially when
dealing with complex social change. There will always be
biases, and although sometimes these can be mitigated, at
other times they simply need to be recognised. This means
it is important to be critical, albeit constructively. This in
turn means refraining from always taking evidence at face-
value, and recognising the need to probe further where
necessary.
It is particularly important to create relationships based on
trust when CSOs work through partners or community-
based organisations, who may tend to regard them as a
donor, as much as a partner. The more different
organisations trust and respect each other, the more likely
they are to be open and honest when carrying out M&E
work.

© INTRAC 2019
Further reading and resources
Other papers in this section of the M&E Universe deal with two other subjects relating to the supporting environment for M&E,
namely ‘resources for M&E’ and ‘data and knowledge management’. Separate papers deal with project and complex M&E
systems.

Project M&E systems Overview of complex M&E systems

Data and knowledge management Resources for M&E

References
 Connor, A (1993). Monitoring and Evaluation Made Easy: A Handbook for Voluntary Organisations. Crown, Edinburgh.
 Dozois, E; Langlois, M and Blanchet-Cohen, N (2010). A Practitioner’s Guide to Developmental Evaluation. The J.W.
McConnell Family Foundation and the International Institute for Child Rights and Development. Montreal, Quebec, Canada.
 Gosling, L and Edwards, M (1995). Toolkits: A Practical Guide to Assessment, Monitoring, Review and Evaluation. Save the
Children Development Manual No.5, London.
 Simister, N (2000). Laying the Foundations: The role of data collection in the monitoring systems of development NGOs.
Occasional paper 01/00. Bath, Centre for Development Studies, University of Bath.
 Save the Children (2004). Global Impact Monitoring: Save the Children UK’s experience of impact assessment, by Simon
Starling, Marta Foresti and Helen Banos Smith. Save the Children UK, January 2004.
 White, H and Phillips, D (2012). Addressing Attribution of Cause and Effect in Small-n Impact Evaluations: Towards an
integrated framework. Working Paper 15, 3ie, June 2012.

Author(s): INTRAC is a specialist capacity building institution for organisations involved in international relief
Nigel Simister and development. Since 1992, INTRAC has contributed significantly to the body of knowledge on
monitoring and evaluation. Our approach to M&E is practical and founded on core principles. We
encourage appropriate M&E, based on understanding what works in different contexts, and we work
with people to develop their own M&E approaches and tools, based on their needs.

INTRAC Training M&E Universe

M&E Training & Consultancy M&E Universe
We support skills development and learning on a range of For more papers in
INTRAC’s team ofhigh
themes through M&E specialists
quality offer consultancy
and engaging and
face-to-face, For
themore
M&Epapers in
Universe
training in all aspects of M&E, from core
online and tailor-made training and coaching.skills development the series
M&E Universe
click the
through to the design of complex M&E systems. series
homeclick the
button
Email: training@intrac.org Tel: +44 (0)1865 201851 home button
Email: info@intrac.org Tel: +44 (0)1865 201851

© INTRAC 2019