Monitoring & Evaluation of

Social Marketing Programs

Report

Module 7
Presented by Poppet Celdran
Monitoring & Evaluating
Advocacy Communications & Social Mobilization (ACSM) Programs

WHO. 2007. Advocacy Communication and Social Mobilization (ACSM) for Tuberculosis Control. A Handbook for Country Programs.
Monitoring
• 1) conducting short-term and long-term monitoring and tracking;
• 2) recognizing problems via feedback from the field; and
• 3) making mid-course corrections based on feedback.

“Before Advocacy Communications & Social Mobilization

(ACSM) activities begin, create monitoring mechanisms to
receive feedback on the interventions and identify any
problems early.”
1. Conducting short-term and long-term monitoring and tracking

It is impossible to anticipate every problem, a monitoring system will help identify difficulties quickly so
that they can be addressed. Determine the roles of partners in solving problems. “Process evaluation”
is the day-to-day monitoring of ACSM activities and operations. Determine the objectives of a process
evaluation prior to establishing the monitoring mechanisms.

The objectives should focus on:

• whether ACSM activities are on track,

• how close they are to meeting the projected timeline and budget, and
• whether staff members understand and perform their roles correctly.
Once objectives have been determined, select monitoring and tracking mechanisms. Some suggested sources of
information for tracking the various components of ACSM activities include:

• inventory of materials;
• distribution list;
• activity reports;
• public diaries;
• television and radio logs;
• media-clipping services;
• staff surveys or focus groups;
• partner feedback;
• timeline and budget assessments;
• news and information searches;
• legislative tracking systems; and
• attitude or household surveys.
2. Recognizing problems via feedback from the field

Responding to relevant information in real time allows a programme to improve immediately, rather
than realizing in retrospect what it should have done. The following strategies for effectively using
feedback from the field can be implemented:

• Involve key decision-makers, stakeholders and TB advocates in helping to analyse and use feedback;
• Use process evaluation to uncover problems or opportunities for the ACSM intervention during
implementation;
• Conduct preliminary evaluations to identify potential improvements and highlight and share
successes before the completion of ACSM activities; and
• Use summative research to make future ACSM programme decisions part of a process rather than
just an end point.
3. Making mid-course corrections based on feedback

Consider the feedback questions below when deciding whether mid-course corrections are necessary.

• Have goals and objectives shifted as activities have been conducted? If so, revise the original goals
and objectives to meet the new situation.
• Are particular objectives not being met by programme or ACSM interventions? Why? What barriers
are being encountered? How can the barriers be overcome?
• Has a strategy or approach met all its objectives or does it seem not to be working at all? Consider
ending that tactic.
• Is there any new intervention or treatment information that should be incorporated into the
messages or design?
• Which strategies or activities have not succeeded?
• Review why they do not work and determine what can be done to correct any problems.
Evaluation
• 1) types of evaluations;
• 2) setting evaluation goals;
• 3) determining suggested indicators;
• 4) developing an evaluation plan;
• 5) selecting monitoring and evaluation methodologies;
• 6) developing and pre-testing data-collection instruments; and
• 7) writing an evaluation report.

At some point all programs need to ask the question, “How effective were the ACSM strategies?”
This is the time to reflect on what has been achieved, what has worked and what has not, and to
make recommendations to improve future efforts. A summative evaluation will not only help to
answer these questions, but also to demonstrate to funders and partners the effectiveness of
activities.
1. Types of evaluations
Outcome evaluation measures how well the
ACSM intervention has met its objectives and what should
be changed to improve future ACSM activities.
Follow the steps below to conduct an outcome evaluation:
• determine what information the evaluation must provide;
• define the data to collect;
• decide on data collection methods;
• develop and pre-test data collection instruments;
• collect data;
• process data;
• analyse data to answer the evaluation questions;
• write an evaluation report; and
• disseminate the evaluation report.
When conducting an outcome evaluation the limitations
and constraints can include:
• limited funds;
• limited staff time or expertise;
• length of time allotted to the ACSM intervention and
its evaluation;
• restrictions on hiring consultants or contractors to
conduct the evaluation;
• policies that limit the programme’s ability to collect
information from the public;
• difficulty in defining or establishing consensus on
ACSM objectives;
• difficulty in isolating the effects of ACSM intervention
from other influences; and
• a perception by funders or others involved in the
programme that evaluations are not valuable.
Impact evaluation connects behaviour change to health or social
outcomes. An impact evaluation answers the question: “Did the people
who adopted new actions or behaviour experience improved health and
well-being related to their TB status?”

Impact evaluations are not often used by planners of ACSM activities –

mostly because of the high costs usually involved in carrying them out
successfully. However, evaluators can select a few programmatic
objectives to measure, such as changes in the rates of treatment
completion or increases in case detection.
2. Setting evaluation goals

To set evaluation goals, first determine who wants to learn from the evaluation. This will be
primarily the stakeholders involved in ACSM activities or those with a direct interest in the
programme.

Other interested parties – such as the broader community, the various levels of the health
system, people from the municipal/district/regional/national governments and donors –
should also be invited to suggest evaluation goals or participate in other ways.

There may also be people and organizations that are interested in learning from the
evaluation although they may not have been directly involved in the process. Potential future
partners might also be invited to participate, to provide an external perspective and to
further their understanding of the approach used and its results.
What are the ACSM objectives?
• What should the members of the intended group
think, feel or do as a result of the ACSM activities, in
contrast to what they thought, felt or did before?
How is change expected to occur?
• Will it be slow or rapid? What measurable
intermediate outcomes – steps toward the desired
behaviour – are likely to take place before the
behaviour change can occur?
How long will the intervention last?
• What kind of changes in attitude, awareness, behaviour
or policy, for example, can be expected in the time
period? Sometimes, when the outcomes are measured,
activities or interventions will not have been in place
long enough for objectives to have been met.
Which outcome evaluation methods can capture the
scope of the change that is likely to have occurred?
• When sample sizes are small (usually due to funding
constraints), the evaluation is said to lack statistical
power and only fairly large changes will be statistically
significant. Programmes should consult a statistician to
make sure that the size of the sample is adequate to
measure the amount of change that they expect to see.
3. Determining suggested indicators

Changes in the evaluation indicators, or key outcomes to be measured, indicate whether

objectives have been achieved. Select the indicators that identify where ACSM initiatives have
been the most successful or where additional work is needed. Link indicators to the objectives
set during the planning process. All indicators must be measurable.

• Individual-level indicators measure:

• knowledge;
• attitudes;
• beliefs;
• behaviour;
• stage of willingness to change; and
• demographics.
4. Developing an evaluation plan
At a minimum, an evaluation plan should answer several
key questions:
a) What are the project’s objectives and expected
outcomes? What questions should be asked?
• What elements of the project worked?
• What elements did not work?
• • role plays to present important milestones or events;
What were some successes?
• What were some failures?
• What still remains to be done?
• What is the vision for the future?
• How has capacity been strengthened during the
intervention?
• Which results obtained during the intervention are likely
to be sustained or improved upon?
b) What information is needed to answer the above
questions and how will the information be collected?
• List the instruments and methods that will be used to
collect the information, such as:
• repeated surveys (baseline and follow-up);
• stories (peak moments, peak achievements);
• drawings (e.g., of project history);
• in-depth interviews with project participants and observers;
• group discussions;
• picture-card pile sorts (sort interventions that worked or
did not);
• rankings (rank initiatives or interventions, from those that
worked best to least); and
• review of the project proposal, reports and documents.
c) Who will collect the information?

• It is more cost-effective to use existing human resources and platforms, if possible, to collect information.
Ideally, staff should not collect information that directly measures the success or failure of their own efforts.

d) What resources/materials will be needed?

• List who needs support in collecting the information from different sources and what support they need.

e) When will this information be collected?

• Create a timeline for data collection.

5. Selecting the monitoring and evaluation methods

For the broadest view of areas where the ACSM programme has been most effective
and those where improvement may be needed, use quantitative and qualitative
research methods to collect evaluation data.

• Qualitative methods may include in-depth interviews, focus groups or anecdotal

feedback mechanisms such as diaries and observational studies (watching people in
a natural setting without their awareness and observing their behaviour).

• Quantitative methods may include sales data, service statistics or surveys.

6. Developing and pre-testing data collection instruments

Most outcome evaluation methods involve collecting data on participants through observation, questionnaires or participatory methods. To develop
data collection instruments– or to select and adapt existing ones – ask the questions:

Which data?
• The data collected should be directly related to the evaluation questions. For example, if members of the intended group need to know more about a
topic before behaviour change can take place, ask knowledge-related questions in the evaluation.

From whom?
• Evaluators should decide how many members of each group are required in the evaluation to measure change. There should be adequate resources
to collect information from the number of people required. Different data-collection instruments and methods for different groups may be necessary.

How?
• Assess available resources before deciding how to collect data. Are skilled interviewers accessible or can staff members be trained? Will self-reports
from participants be used? If so, do any confidentiality issues need to be addressed?

Consider also whether participants will be comfortable with the data collection-methods.
• Will they be willing and able to fill out forms?
• Will they share personal information with interviewers?
• Will the interviews and responses need to be translated?
7. Writing an evaluation report

To prepare an evaluation report, have staff with appropriate expertise analyse the outcome
evaluation data and work closely with the evaluators to interpret the data and develop
recommendations.

An evaluation report should:

• present the lessons learnt in a clear format that can be easily digested by others who may be
planning future ACSM activities;
• demonstrate accountability to employers, partners and funding agencies;
• provide evidence of the effectiveness of the ACSM interventions and activities; and
• create a formal record to serve as an institutional memory of what has been tried, which partners
had strong skills or experience in specific areas, what problems were encountered and what
successes were achieved.
Monitoring & Evaluating
for Social and Behavioral Change Communication

Weiner, Renay.
Monitoring and evaluation plan
A monitoring and evaluation (M&E) plan is a document that outlines how an
implementation research project is monitored and evaluated, and that links strategic
information obtained from various data collection systems to decisions about how to
improve the project on an ongoing basis.

The M&E plan serves several main purposes, including:

• stating how achievements of the programme/project will be measured;
• documenting consensus, thereby encouraging transparency, accountability and
responsibility;
• guiding implementation of M&E; and
• preserving institutional memory.
Standards for an M&E plan
An effective M&E plan should conform to the following standards:

• Utility: It must be useful and serve the practical and strategic information needs of the intended users for
decision-making purposes, these may range from assessing programme performance to allocating resources,
etc.

• Feasibility: Be realistic and practical. Given the scarcity of resources, the M&E plan should make the best use
of existing data collection systems. However, if new data collection systems are involved, resources (cost and
technical capacity) must carefully be considered.

• Ethically sound: Abide by ethical principles with regard to those involved in and affected by the M&E activities.

• Accuracy: Provide technically accurate and useful information for decision-making and programme
improvement.
Key components of an M&E plan
There are four key components that form the foundation upon which the
M&E plan should be built. Answering these four corresponding questions
is critical to M&E planning:

• What does the project want to change and how?

• What are the specific objectives that are designed to achieve this
change?
• What are the indicators and how will they will be measured, and,
• How the M&E data will be collected and analysed?
Key steps in developing an M&E plan
Stakeholder consultation and participation

Stakeholder consultations and participation should be regular occurrences throughout the entire
process of developing and implementing your M&E plan. These consultations ensure dialogue, a
clear understanding of the project goals and objectives, and how these will be assessed.

They also ensure that various perspectives are understood and integrated, and that authentic needs
are being met. Stakeholder participation in the design of the M&E plan facilitates the selection of
appropriate and useful M&E indicators. Furthermore, taking extra measures to promote stakeholder
participation creates a sense of ownership and responsibility among partners.

Stakeholder involvement increases the probability that the information and results guided by the
M&E plan will be consistent with their expectations.
Developing the M&E plan
Consensus should be reached on key questions in the following areas:

“What do we want to know at the end of the project?”

and
“What do we expect to change by the end of the project?”

Again, answering the question of what you expect your project to change will guide decisions
about what strategic information is needed for project management decisions as well as
what elements should be monitored and evaluated in order to assess progress.

The rigor and scope of your M&E plan will depend on what you commit to and what results
or outcomes your project is accountable for.
Developing the M&E plan provides your team with a clear picture of the following:

• How project activities are linked to expected outputs, outcomes and population-level impacts.

• How different types of information will be collected and used by different levels of the health system.

• What elements need to be measured (e.g. resources, service statistics, coverage and quality, costs,
and outcomes associated with the project).

• Appropriate indicators to be selected. To enable standardization and comparison with other similar
projects, indicators should be consistent with international/national standards. They should also be
feasible and realistic to collect. The data sources identified must provide the information needed to
measure the indicators.
Determining the M&E methodology
Once your team has developed the M&E plan, defined the indicators, and identified the data
sources necessary, the appropriate methods by which data can be collected and analysed
should be determined.

For example, your team should determine whether you will use existing data collection
systems or if new systems need to be developed.

Your team must also determine how information will be recorded, analysed and reported.

Furthermore your team should also carefully consider the resources available in terms of
technical competencies, costs, and time when determining the methods and tools to be
used.
Assign responsibilities for implementation
After developing the M&E plan, the roles and responsibilities of the different
stakeholders should be described clearly.

This step will determine how the M&E plan will be specifically implemented and
what reporting system will be adopted.

The implementation of the plan should include the data collection plan (i.e. who is
responsible for collection of specific data; ensuring quality control at each stage;
how often the data will be collected; format of the data (e.g. raw, summary); what
resources will be required at each stage; who will analyse the data) and the
dissemination plan.
Setting targets
Target should be set in consultation with all stakeholders so that everyone understands what the project has
committed to achieve.

By setting targets, you will have a concrete measure by which to judge whether the project is progressing as
expected.

The process of target setting, must focus on answering the question:

“What can realistically be achieved given the resources and the environment in which the project is operating?”

The factors to consider include: baseline levels; past trends; expert opinions; research findings; what has been
achieved elsewhere; client expectations; and the capacity and logistics to achieve targets.

When setting specific project targets, you must also decide the direction of any potential changes that may be
indicated over time.
Defining reporting system, dissemination and
utilization of results
Here are some practical considerations in planning information reporting and utilization planning:

• Design the M&E dissemination plan around the information needs of the users. It is important to be mindful that the content and
format of data reports will vary according to their intended use. For example, is the M&E required to monitor processes? To conduct
strategic planning?

• To comply with requirements? Help identify problems? Justify a funding request? Or to conduct an impact evaluation?

• Identify the frequency of data reporting needs. For example, project managers may want to review M&E data frequently to assess
project progress and make planning decisions, whereas donors may need data only once or twice a year to ensure accountability.

• Tailor the reporting formats to the intended audience. Since reporting may entail different levels of complexity and technical language,
the report format and media should be tailored to specific audiences and different methods used to solicit feedback.

• Identify appropriate outlets and media channels for communicating M&E data. This should consider both internal reporting, such as
regular project reports, to management and progress reports to donors, as well as external reporting, such as public forums, news
releases, briefings, and websites.
Implementing the M&E plan
Implementation of the M&E plan occurs in three stages, namely:

• checking and measuring progress;

• analysing the situation; and
• reacting to new events, opportunities and issues.
Checking and measuring progress
Ideally, monitoring focuses on the project’s three main characteristics of quality,
time and cost.

The project manager coordinates the project team and should always be aware of
the status of the project.

When checking and measuring progress, the project manager should communicate
with all team members to find out whether planned activities are implemented on
time and within the agreed quality standards and budget.

The achievement of milestones is measured and reflects the progress of the project.
Analysing the situation
The second stage of monitoring consists of analysing the situation. The
status of project development is compared to the original plan, and
causes and impact of potential deviation are identified.

Actions are identified to address these causes and the impacts of any
deviations.
Reacting to new events, opportunities and issues
It is important to anticipate and react quickly to new situations, events,
opportunities and issues, and to identify the possible actions to be taken.

If appropriate, various options are considered and discussed with the

project team and a decision is taken regarding the most appropriate path
to pursue.
Adjustments to/updating the M&E plan
The M&E plan should be seen as dynamic and should always reflect the reality of what is known and understood.
Each time a deviation from the original M&E plan is identified, whether or not it requires any further action, the
M&E plan should be revised and changes documented accordingly.

Here are examples of questions that can be considered to help your team assess how well the M&E plan is working.

• Are M&E activities progressing as planned?

• Are the evaluation questions being answered sufficiently?
• Have new evaluation questions been raised and, if so, should they be incorporated into the M&E plan?
• Are there any methodological or evaluation design issues that need to be addressed?
• Are there any outside factors (political, environment) that are affecting the M&E plan?
• Are appropriate staff and funding still available to implement the M&E plan?
• Are M&E findings being disseminated and used by stakeholders for decision-making and programme
improvement?
Monitoring & Evaluation of
Communication and Social Mobilization Guidelines

PMI.
Minimum Monitoring and Evaluation Standards for
Communication Activities
Following is a list of required minimum monitoring and evaluation standards for communication activities.
Describe the process of strategy, materials, and activity development and implementation, including methodology and when the
steps happened. The following steps should be described in detail, including how, when, how many, and target audience:
• Audience research including instruments, methodology, and results
• Strategy formulation
• Print materials developed
• Broadcast materials developed
• Local communication channels
• Pre-testing including instruments, methodology, and results
• Production
• Broadcast
• Dissemination/Orientation
• Training/orientation
• Supervision
• Monitoring including instruments, methodology, and results
• Evaluation including instruments, methodology, and results

Provide human and financial inputs for each step above, including development costs, if available.
The Rapid Monitoring Checklist
The Rapid Monitoring Checklist below gives examples of questions that can be used to collect relevant monitoring information, a list of
tools to collect the information, and suggested approaches for using the tools. As mentioned earlier, this checklist can be used to
receive feedback on the interventions and to identify any problems early so that they can be addressed quickly.
References:
• WHO. 2007. Advocacy Communication and Social Mobilization (ACSM) for Tuberculosis Control. A
Handbook for Country Programmes. Chapter 9

• Weiner, Renay. Monitoring and evaluation for social and behavioural change communication

• PMI Communication and Social Mobilization Guidelines

(http://www.pmi.gov/resources/publications/communication_social_mobilization_guidelines.pdf.)