Understanding the Assignment/Terms of Reference

The communities of Galadi districts of dollo Zone, Gode town, Gode Ellaleh, and Adadle
districts of Shabelle Zone, Marshin district of Korahey Zone, Barey, Chereti, and Hargele
districts of Afder Zone, Kersa, Meta, and Fadiss districts of East Harerghe Zone are long
decimated by the El Nino induced droughts. According to reports by different organizations,
Drought occurs within less than five years interval which continuously draws them back to
poverty and there is a high level of community engulfed within poverty trap. Rainfalls set in late
and terminate early which affects the crop production and livestock rearing particularly
facilitating reproduction of pests and animal diseases. The pastoralists are the most affected by
this situation. Livelihoods are emaciated with high prevalence of malnutrition manifested by
stunting and wasting of children. Children and pregnant women are affected by malnutrition.
There are also reported rapes in some areas.
The situation is exacerbated by the recent conflicts fermented along the Oromia-Somali corridors
resulting in to displacement of over one million peoples. This in turn put strain/overburden on
facilities like WASH for humans and animals and caused facilities fractures and dysfunctional.
This further created destabilization of social cohesions.
Cognizant of this, OFAM /OWDA together with AAH and IRE made intervention.
The overall objective of the project is to contribute to the reduction of mortality and morbidity
due to acute malnutrition, waterborne and outbreak related diseases among communities in
Somali, Harari regions and DireDawa City Administration affected by the drought and conflict.
The specific objective of this project is that targeted IDPs and host communities affected by the
drought and conflict in Somali and Oromia region have improved access to inclusive and
accessible WASH, food and livelihoods, protection emergency health care and nutrition
prevention and treatment services including persons with disabilities. To this end, the project has
five result areas in different target areas and these are:
Result 1: Targeted most vulnerable drought IDPs and host communities have increased food
security through improved access to food, income and livelihood opportunities.
Result 2: Enhanced access to quality emergency health response, treatment and prevention of
acute malnutrition for children [6-59 months], pregnant and lactating women.
Result 3: Increased access to safe drinking water and improved sanitation and hygiene facilities
and knowledge for targeted drought IDPs and host communities
Result 4: Increased access to safe drinking water and improved sanitation and hygiene facilities
and knowledge for targeted conflict affected IDPs and host communities
Result 5: Targeted most vulnerable conflict-affected IDPs have increased food security through
improved access to food and income.
Accordingly, this proposal is to conduct a final evaluation of this project.
The purpose of this evaluation is to assess OXFAM\OWDA, IRE, and AAH’s performance and
delivery of the ECHO funded project according to the OECD-DAC evaluation criteria. The
evaluation will help ECHO, OXFAM\OWDA, IRE, and AAH to improve their future
programming through lessons learned and best practices generated through this project.
The aim of this terminal evaluation is to appraise\assess the overall implementation and
achievement of the project against its planned outputs [and outcomes where possible] as
stipulated under the five result areas and activities under each result. Accordingly, it provides
information on what worked and what did not work and why and whether the underlying theories
and assumptions used in the program development were valid. This evaluation mainly intends to
provide information on the relevance of the projects, timeliness, effectiveness, efficiency,
impacts and sustainability of the project action, integration of accountability, gender and
protection, diversity, linkages and lessons for advocacy.
The specific objectives of this project are:
To assess the extent to which the ECHO consortium project has delivered against its set
objectives and expected results of the project based on the log frame.
To appraise the value addition, comparative advantages, strengths and weaknesses of working in
a consortium structure, and to provide recommendations for future direction and \or how to
improve this approach
To highlight key successes, lessons learned, good practices and area for improvement, and
provide key recommendations for future drought and conflict response projects in Dire Dawa,
Somali, Oromia and Harari regions in Ethiopia.
Geographically, the final evaluation covers the Warder and Galadi districts of dollo Zone, Gode
town, Gode Ellaleh, and Adadle districts of Shabelle Zone, Marshin district of Korahey Zone,
Barey, Chereti, and Hargele districts of Afder Zone, Kersa, Meta, and Fadiss districts of East
Harerghe Zone, Dire Teyare, Erer, Hakim, and Sofi kebeles of Harar City Administration,
Millennium Camp of Dire Dawa City Administration.
 Regarding the target Beneficiaries, the project has planned to reach 501,185 individuals [76,160
households of IDPs and host communities] in the most drought and conflict affected areas of
Korahe, Shebelle, Afder and Dollo Zones of Somali Region, and conflict affected IDPs in Harari
and Dire Dawa.
Evaluation Matrix
The consultant adopted a five column evaluation matrix to orchestrate the OECD-DAC criteria based evaluation
dimension as shown.

Evaluation Evaluation question Indicators or Sources of Information Methods for data

criteria measures collection
Relevance Did this programme Level of involvement Project document, mid-term Document analysis,
and effectively reach the and inclusiveness of evaluation, key project institutional and
Timeliness most vulnerable stakeholders and stakeholders and individual key
households beneficiaries in beneficiaries informant interviews,
project design and and focus group
implementation discussions
Did the project address Strength of link Project document, mid-term Document analysis,
the priority needs of the between project evaluation, key project institutional and
affected population results and the needs stakeholders and individual key
of relevant beneficiaries informant interviews,
stakeholders and and focus group
beneficiaries discussions
Criteria used for
target group and
beneficiary
identification
What factors affected Time of contractual Project Documents – Assessment of donor
positively or negatively agreement, and progress reports, monitoring agreement, progress
the timeliness of the official launching, reports and monitoring reports.
evaluation funders timely Key informant
resource release, interview of the project
delays in personnel
implementation and
backlogs of some
activities
Effectiveness Were the monitoring Existence of Monitoring and evaluation Document analysis,
mechanisms effective in monitoring system plan\strategy, tracer reports, interviews with the
providing timely data to monitoring reports project
inform programming personnel\consortium
decisions leader at CARE
Why did the project Changes to the log Project document, log Document analysis,
achieve certain planned frame and frame, project change order, interview with the
results or not achieved justification Addendum, and derogation project personnel
results issues
What were the main Knowledge Reports of best and worst Document analysis of
factors for success or management plan practices monitoring and regular
 failure of the different Human stories, learning reports, key informant
project components Risk management reports interviews and focus
matrix group discussions with
project beneficiaries,
interview with the
project personnel
To what extent did the Partnership Joint review meeting report Document analysis
consortium structure engagement and Project memorandum of Interview with partner
allow to reinforce the involvement understanding, memos and organizations
effectiveness of this plan\strategy, circulars within partnering
action evidences of organizations
partnerships
including financial
contributions
Did the consortium Partners joint action Partnership contribution Document analysis
structure [including review analysis Interview with partner
TWGs] enable to Counterfactual [if not organizations
improve synergies, partnered] analysis
harmonization of
intervention, to ensure
the quality of the work
of the different partners
Impact To what extent did this Outcome logs, impact Mid-term evaluation, Document analysis,
project achieve the logs terminal evaluation and focus group discussions
intended outcome and impact assessment and interviews with
impact individuals and
institutions
How is the ECHO Impact pathways Contribution and attribution Document analysis,
consortium project Outcome pathways analysis focus group discussions
bringing difference in Focused comparisons and key informant
the lives of conflict and [among project beneficiaries interviews with
drought affected people and non-project institutions, and project
beneficiaries] beneficiaries and non-
beneficiaries
What was the Indicators Tracking Baseline report, mid-term Document analysis,
performance against the system evaluation, terminal focus group discussions
stated indicators evaluation and interviews with
individuals and
institutions
Are there any ill effects Registry of ripple Causal links conceived Document analysis,
or unplanned impacts effects and negative within the log frame focus group discussions
because of this project externalities and interviews with
individuals and
institutions
Efficiency How is the project cost Level of discrepancy Budget utilization report Document analysis
efficient when it is between planned and Auditor report Interview with project
analyzed utilized financial Performance audit report financial accountants
expenditures. and program manager
 What evidence is Availability and Budget report Document analysis
available on the cost quality of financial Budget tracking report Interview with project
effectiveness of the and progress reports. Budget amendment financial accountants
project Quality of notification letters and program manager
management and
monitoring systems
in place and ability to
make necessary
changes if and when
needed.
How do intervention Operation and Administration budget Document analysis
costs compare with other administration budget report Interview with project
modalities utilization and burn Operation\programmatic finance and program
rates budget manager
Auditors rate on qualified or
unqualified status
Was technology Evidence of Package of technology Document analysis,
deployed to improve technology and acquisition and outsourcing institutional interview,
efficiency automation systems documents and beneficiaries
adopted interview, vendor
interview
To which extent did the Comparative analysis Knowledge documents and Document analysis,
used technology improve of resource use automation hardware interview with trained
efficiency efficiency before and List of enhanced Project project teams and
after the technology management team and beneficiaries
adoption beneficiaries
Could the project’s Comparative analysis Knowledge documents and Document analysis,
efficiency have been of efficiency before automation hardware interview with trained
increased using other and after technology List of enhanced Project project teams and
technology adoption management team and beneficiaries
beneficiaries
To what extent did the Evidences by minutes Partnership strategy, and Document analysis,
consortium structure or reports entailing partnership agreement interview with partner
allow to reinforce the the added value of organizations
efficiency of this action the synergy
Sustainability To what extent did the Evidence of Project specific Inter-sector Interview with project
programme utilize institutionalization by structures and mechanisms personnel and relevant
established relevant authorities sector authorities
institutions\mechanisms by program
to ensure sustainability component and
at the end of the project sectors
To what extent were Evidence of adopting Component\sector Document analysis,
relevant project strategies Sustainability strategies Interview with project
partnerships\capacity personnel and relevant
developed to ensure sector authorities
sustainability
Was an exit strategy Evidence of Institutional and financial Document analysis,
developed to ensure functional and active mechanisms installed Interview with project
 sustainability structures and personnel and relevant
mechanisms sector authorities
To what extent did the Policy and strategy Project document, Document analysis,
project consider Alignment factors government sector strategies interview with project
measures, where personnel and relevant
necessary, to improve government sector
resilience of drought and representatives\policy
conflict affected analysts
communities of the
action
Are the measures Assessment of the Contribution and attribution Document analysis,
implemented adequate to ‘SCALE’ targeted by analysis interview with project
guarantee sustainability each partner personnel
and improve resilience

Methodology and approaches

The consultant will adopt a mobile Android assisted data collection using Open Data Kit (ODK). ODK is
a set of open source applications which allow one to create a questionnaire form in the Xform format, fill
it out on a mobile phone or tablet running the Android operating system, store and view the aggregated
information on a central server, and retrieve the aggregated data to one’s computer for analysis. Data
capture includes GPS coordinates for real-time mapping of responses in Google Maps, or near-real time
once the surveyor has an Internet connection to send the collected forms back to the server. Data can be
collected and sent to a centralized server using internet connected android devices in real time. ODK
Collects data with application installed on the Android based mobile/ tablet having Camera, GPS and
GPRS connection.
The methodology of data collection work is as follows:
1. Decide the data to be collected and create a form by logging in at http://build.opendatakit.org. Final
form in the form of a XML file can be downloaded in the (/odk/forms folder) mobile device or uploaded
on the Aggregate server.
2. Download the ODK collect in your Android mobile device. In the settings option set the server address
to the aggregate application on the server. Also, download the form from the server if already not
downloaded. Now use the fill blank form option to fill all the details in the form and if finalized can send
the finalized form to the server or save it on the mobile itself. There are options to edit and delete the
saved forms.
The mobile user will send the filled finalized form through the mobile device(with GPRS connectivity)
which will be received in real time by the ODK aggregate server application running on the a server with
a fixed IP address.. The data will be stored in the MySQL or PostgreSQL database. If in case there is no
GPRS connectivity the forms can be filled and stored on the mobile can be sent when there is connectivity
or ODK Briefcase application can be used to transfer data from Collect to Aggregate.
Prior to deploying to field survey, the consulting firm will give a one-day intensive training to its survey
team on the topics related to: Introduction to mobile phone data collection, Setting up aggregate server
with Google App Engine, Designing & building a form using ODK build, Uploading the form to
aggregate & send to Android phone, Setting server preference on android phone, Data collection [Enter
primary field data, capture GPS coordinates, take photos, record audio sounds and videos, Submitting
completed data to aggregate server, Exporting data from ODK aggregate as CSV, Importing ODK - CSV
file into statistical applications, Exporting GPS data for Mapping/Visualizing (fusion map)
Evaluation TOOLS/Instrumentation
The consultant adopts both qualitative and quantitative data collection tools from primary and secondary
sources.
Desk Review: The consultant reviews using existing documents and data. The documents to be reviewed
includes project document, project baseline report, project mid-term evaluation report, project monitoring
reports, other emergency and humanitarian reports and researches of non-governmental organizations like
IOM DTRM, OCHA reports, government sector specific policies and strategies, partnership agreements
and MoUs, launching workshop reports, and so forth. The data gathered through the desk review will be
used to prepare and improve the evaluation plan. The evaluation questions will also be reviewed based on
specific information obtained through the desk review.
Focus group discussions [FGDs]: The consultant organizes highly interactive focus group discussions
by preparing FGD protocols to guide discussions. FGDs will be facilitated by moderators who explains
the purpose of discussions (to assess the achievements of the interventions independently, identify gaps
and correct same in future so as to better deliver services), how the discussions would be conducted, and
confirm that all information gathered during the discussions will be used confidentially. All the FGDs
will be conducted in the respective local language.
Thus, the consultant makes FGDs with women and farmer based organizations, traditional institutions and
structure members, village water committees, PSNP committees, WASH and water user associations,
IDPs camps households, etc.
Women only FGDs will be conducted for WASH and nutrition purposes.
Key Informant Interviews [KIIs]. Key informant interviews (KIIs) based on the semi-structured in-
depth interview method were conducted with key stakeholders on all main facets of the project
intervention, with a special focus on issues relevant to most marginalized communities. The
questionnaires will be developed covering all aspects including the main evaluation questions and logical
framework. Separate questionnaires will be prepared for each component namely, EFSVL, WASH,
Health and Nutrition interventions.
Accordingly, the consultant undertakes KIIS with Health Office, Pastoral development Office, Drought
risk Management Office, Women Affairs, Water, Mineral and Energy Office, private veterinary vendors
and pharmacies, and so forth.
Spontaneous site visits: The consultant makes spontaneous site visits to check infrastructure facilities.
The opportunity will be used to meet with community members living near these sites depending on
availability of time. Short notes of site visits and community visits will be made for use in analysis.
The consultant makes photo-elicitation during KIIS, transect physical observations and FGDs capturing
videos, audios and photography of the participants.
Household Survey
The consultant will undertake household survey both in IDPs camps and host communities using a
structured questionnaire addressing the thematic interventions. As stipulated in the terms of reference, the
number of households to be accessed as direct beneficiaries has been 76,160 households of IDPs and host
communities equivalent to 501, 185 individuals (total project target population). Accordingly, the sample
size to be contacted by the household questionnaire is estimated by the following sample size formula
determination, Yamane formula as

which gives 501, 185 /{1+501, 185 (0.05)2}=400 individuals and dividing
this by 6.58 of average family size by household we get 61households.

Thus the household questionnaire will make contact with 61 persons through a structured questionnaire.
This will be a total for IDps and host communities. The number per each of the districts will be decided in
consultation with the client organization, OXFAM and consortium members.
All the FGDs, KIIs, and household questionnaires will be managed through ODK application and are
fully loaded. For the analysis purpose, the consultant extracts the data from ODK to SPSS/STATA, cleans,
edits and runs the analysis.
The consultant will make the following tools for data quality control processes.
Accompany calls in which 30% of interviews will be observed by the survey team leaders to ensure that
the moderators and enumerators are conducting the interview well, asking the questions in the right
manner, and interpreting the answers correctly.
The consultant also conducts spot-checks to ensure that the questions are being asked properly and filled
in the right way, and selecting some of the addresses selected for interviewing to be sure that the
moderators and enumerators interviewed the right participant, and correctly identified the eligible
respondents and interviewed them. Thus survey team leaders regularly undertake observations of FGDs,
KIIs and facility assessments.
The consulting firm also conducts questionnaire completion reviews to ensure that they are complete and
internally consistent.
The firm also conducts 20% back-check interviews to be sure that all the answers given by the respondent
are recorded correctly for at least 5% of the data moderators and enumerators\ interviews.
The consultant also make multiple cross-validations among data obtained through different methods and
also makes abduction analysis with globally available evidences like the data dashboard of ECHO,
OCHA and IOM, regional, and national data bases.
The consultant is abiding by appropriate ethical considerations in which the consent of respondents is first
confirmed, and the privacy and anonymity of their data is kept confidential.
Data Analysis
The consultant analyzes quantitative and qualitative data according to the results in the program logic
framework. The evaluation planning matrix will be used to integrate different sets and types of data. The
relevance and timeliness, impact, effectiveness, efficiency and sustainability of the interventions will be
taken into consideration under this analysis.
Work plan
The consultant has proposed the following tentative evaluation work plan to be accomplished in 51 days with
indicative duration and milestones using Gantt chart.

S\N Activity Items W1 W2 W3 W4 W5 W6 W7 W Duration Milestones

8 in days
1 Evaluation Planning
1.1 Preliminary discussion with OXFAM 1 Minute
1.2 Grant declaration, Contractual Agreement 1 Contractual
agreement
1.3 Conduct desk review 3 Synthesis of
literature
1.4 Inception report [detailed sampling 7 Draft
strategy, data collection tools, timed-work inception
action plan] report
1.5 Commenting and refining inception 3 Final
report\plan meeting with the members of inception
evaluation team report
1.6 Hold meeting with members of evaluation 1 Agreed upon
team to brief on the tools and questions, tools
objectives and outcomes
1.7 Training of survey team on mobile data 1 Trained
collection survey team
1.8 Arrange logistics, assign responsible 2 Ready
leads, informing and appointing zonal and logistics and
district level respondents particularly of staff
sector offices
2 Field work
2.1 Deployment of survey team to project 10 Field work
sites for data collection [lead consultants
from the consultant office and locally
recruited professionals]
2.2 Debrief meeting with the evaluation team 1 Minute
3. Report synthesis
3.1 Data cleaning, editing, entering, inter- 5 Multi-
application transfer adjustment like from format
excel\SPSS\STATA to ODK and vice Clean data
versa, and between android mobile and
computers, curating to google and other
suitable databases like Oracle\SQL\MS-
ACCES.
3.2 Data analysis 3 Analyzed
and
synthesized
data
3.3 First draft report 7 First draft
report
3.4 Hold Validation workshop 1 Workshop
report
3.5 Incorporate comments and feedback 3 Comments
and
feedback
3.6 Submit final report in word, ppt and pdf 2 Final report
Total number of days 51

Survey Administration and Management

The consultant firm will deploy highly experienced multi-disciplinary professionals in nutrition,
livelihoods, public health, WASH, and disaster risk management. Accordingly, six researchers
are deployed to the field survey, and twelve local professionals who can command local
language will be oriented and deployed as moderators.
There is one reference team at consulting firm’s head office to continuously report the progress
of the work to the client organization.
The professional combination and experience of our team are more detailed in the CVs section.