KENYA MICRO, SMALL AND MEDIUM

ENTERPRISE COMPETITIVENESS PROJECT

TERMS OF REFERENCE FOR IMPACT ASSESSMENTS

The impact of the Kenya MSME Competitiveness Project on enterprises that participate
in three project activities will be assessed using impact assessment techniques. The three
groups of enterprises concerned are those that:
• Borrow money from financial institutions assisted by the Financial Sector
Deepening Trust or the SME Risk Capital Fund.
• Take part in the integrated strategies for developing selected value chains and
receive matching grants under this activity.
• Participate in the Business Plan/Innovation Competition.

The objective of the impact assessment is to determine whether some or all of these
project activities conferred benefits on participating MSMEs, particularly by helping
them to expand employment and raise value added per worker.
• Employment will be measured in two ways: (1) by the number of full-time
equivalent workers employed by the firm on average over the past year and (2) by
the total value of wages paid.
• Value added per worker will be measured as the gross sales revenue of the firm
over the past year less the cost of inputs (measured in U.S. dollars), divided by the
average number of full-time equivalent workers employed.

The impact assessment should have both quantitative and qualitative components. The
quantitative component will consist of a longitudinal survey of enterprises that participate
in the three activities listed above, along with a control group of comparable non-
participating firms. The qualitative component of the impact assessment will consist of
in-depth interviews and focus group discussions with participants in the activities
concerned. It is also important that the study take full account of the context in which the
activities are taking place. The updated survey of Kenyan MSEs and other relevant
sources should be consulted to establish the context of the project activities being
assessed.

The quantitative part of the impact assessment will measure changes in the two target
variables (employment and value added per worker) within a sample of participating
enterprises and compare these changes with an estimate of the counterfactual – what
would have happened to the same enterprises if the project had not been carried out.
Under the quasi-experimental methodology, which will be used in these studies, changes
in the same two variables experienced by a sample of comparable enterprises that do not
participate in the project activities (the “control group”) are taken to approximate the
counterfactual. The true counterfactual can never be known with certainty, since it
represents a hypothetical situation.

The quantitative component of the impact assessment will be based on a longitudinal

design that measures the same variables for the same panels of participants and controls
at two or more points in time. The classic design is a baseline survey around the start of
the program, followed by at least one follow-up taken at least two years later. In all three
of the Kenya MSME Competitiveness Project activities to be assessed, participants will
be joining the project continuously from Year 1 on. Accordingly, a baseline survey
should be conducted in Year 2 of the project, covering all or some of the participants
identified by that time, along with a similar number of non-participants. Insofar as
possible, information collected should relate to the year prior to their joining the project.
A follow-up survey should be taken late in Year 4 or early in Year 5. To reflect as much
project impact as possible, the follow-up survey should be conducted as late in the project
as possible, subject to the need to have the results available for the End of Project
Review. Another consideration, however, is seasonality; it may be important that the
follow-up survey be conducted at the same time of the year as the baseline survey.

At the time of the follow-up survey, consideration should be given to the possibility of
adding participants who joined the project after the baseline survey was conducted. This
would have to be done using retrospective (recalled) data for the late-joining enterprises
at the time they initially took part in project activities. This would be more feasible for
enterprises which keep formal books than for smaller, more informal, enterprises, so the
design of the sample would have to be carefully considered and the results of the survey
interpreted accordingly.

The survey will include three groups of project participants, namely firms that participate
in each of the three project activities included in the impact assessment. The control
group will consist of MSMEs that take part in none of these activities.

Data analysis will occur after each survey round. The main purpose of data analysis
following the baseline survey is to establish the baseline values of the target variables
(employment and value added per worker). At that time, tabulations and cross-tabulations
can also be prepared to describe and compare the participant and control samples. Data
analysis after the follow-up survey measures changes in the target variables within the
participant and control samples. Taken together, the two surveys permit changes in
employment and value added to be compared across the two samples. If larger changes
are found in the participant sample, an analysis should be carried out to determine
whether the differences are statistically significant. This analysis should use analysis of
co-variation (ANCOVA) or a comparable methodology and should take into account
significant influences on levels of employment other than program participation.

This quantitative research should be complemented with qualitative inquiries among

participants in the three program activities. Possible forms for these inquiries include
selected case histories of program participants, interviews with participants in various
aspects of the project activities (e.g., bankers as well as borrowers; producers, processors,
marketers, and customers of products in the selected value chains), and focus group
discussions in which key issues are reviewed with participants with varying points of
view. These qualitative methodologies have the potential of enriching the quantitative
results by confirming, explaining, and/or modifying the statistical findings.
While performance monitoring is best conducted by the bodies responsible for managing
project activities, impact assessment should be carried out independently – because of the
complexity of the task and also to ensure that the study is scientifically objective. The
project secretariat will be responsible for holding competitive tenders for the impact
assessment activity and hiring a suitable contractor to carry out the work. The impact
assessments contractor should meet several criteria:
• Experience in designing and implementing sample surveys.
• Experience collecting data from MSMEs; demonstrated rapport with business
sources.
• Proven ability to engage and train a sufficient number of experienced and capable
enumerators.
• Capacity to manage the logistics of fielding survey research teams.
• Demonstrated social science ability to carry out qualitative research.
• Demonstrated statistical ability to provide high-quality data analysis and reports.
• Willingness to work with the project secretariat and tailor the assessment to meet
project needs.
• Independence from the steering committee, project secretariat, project
management contractor, and contractors for individual project activities.
• Finally, prior experience in carrying out longitudinal impact assessments is
desirable.

The impact assessment contractor will be selected initially to conduct the baseline survey
during Year 2 of the project. If the contractor’s performance is judged to be adequate by
the steering committee, the contract can be renewed without competitive bidding to cover
the follow-up survey round and attendant qualitative work, data analysis, and reporting.

For the baseline survey, the duties of the impact assessment contractor are to:
• Design a research plan for study in consultation with the project secretariat and
obtain approval of that plan from the secretariat and steering committee. The
research plan should include the following elements:
o Analytical description of the activities to be studied and the setting in
which they operate.
o Statement of key questions to be examined and hypotheses to be tested.
o Delineation of a causal model for each activity.
o Sample design for survey, including over sampling of both participants
and controls to allow for attrition between survey rounds.
o Quantitative data collection strategy and survey questionnaire.
o Plan for collecting qualitative information.
o Data analysis and reporting plan.
• Carry out the baseline survey.
o Design participant and control samples and questionnaire.
o Engage and train enumerators.
o Pilot test survey.
o Conduct full survey with adequate supervision and quality control.
o Tabulate and verify data.
o Analyze data and present in tabular form.
 • Carry out qualitative research as appropriate.
o In-depth interviews with selected participating entrepreneurs and relevant
others (e.g., bankers, buyers, input suppliers).
o Selected life histories of entrepreneurs.
o Focus group discussions of key analytical issues among program
participants.
• Prepare draft report incorporating findings from qualitative research.
• Circulate draft report to project secretariat, project management contractor, and
activity contractors for review and comment.
• After considering comments and incorporating any change the contractor deems
necessary, submit the final report to the project secretariat for transmission to the
steering committee.

Terms of reference for the follow-up survey two years later would be similar to those for
the baseline survey. However, the several additional issues will have to be weighed:
• Enough respondents (participants and controls) from the baseline survey need to
be located and interviewed to provide a statistically valid panel.
• The possibility of incorporating late-joining participants into the survey by asking
them questions about the current and past state of their enterprises needs to be
considered.
• The data analysis needs are more demanding for the follow-up survey than for the
baseline survey, as discussed above.