Hebrew University of Jerusalem

Faculty of Social Sciences

Seminar: “Program Evaluation for Community Development and Social Change”
Teacher: Dr. Nancy Strichman
Participant: Oscar Santiago Vargas Guevara
25th October 2018

Critique of Evaluation Report

Final Evaluation of the Project

“Supporting a Conversation on Youth Leadership in Côte d’Ivoire”
Search for Common Ground (SFCG)

Starting in September 2008, Search for Common Ground (SFCG) in Côte d’Ivoire developed an
18-month project, supported by the US Department of State’s Bureau of Democracy, Human
Rights and Labor (DRL), with the goal of strengthening youth capacity to mitigate social conflict
and promote tolerance. The project, titled “Supporting a Conversation on Youth Leadership in
Côte d’Ivoire”, contemplated an external evaluation in its Terms of Reference (ToR) to be
carried out three months after the end of the project. To this end, SFCG contracted Clotilde
Gouley as external evaluator and team leader, as well as SFCG’s Quentin Kanyatsi as internal
evaluator, in July 2010.

The evaluation’s stated objectives were: (a) to evaluate the results of the project, (b) to capture
concrete examples of change owed to the project, and (c) to give recommendations to improve
future planning and replication. Therefore, the evaluation is summative in nature, looking at the
finished project retrospectively and raising issues to be taken into account for future similar
projects. For this reason, the intended audience of this evaluation is SFCG staff, as well as the
DRL. A possible set of recognised secondary users are the general public, partner organisations,
final beneficiaries and other organisations in the same field of work.

The evaluation espouses four main principles of observation. It adopts a multi-level approach,
which amounts to, first, separately analysing the effectivity of each individual activity
(workshops, follow-up sessions, media partnerships and interactive theatre workshops), and
second, observing changes at four different levels: individual (knowledge, skills, behaviour, self-
realization), community (shared experiences, resources for conflict transformation),
institutional (youth involvement in decision-making) and intergroup (youth networks across
different areas). Though this differentiation does come across in the text, it could have been
helpful for the reader to also structure the analysis chapters according to these criteria.
Additionally, the team adapted Kirkpatrick’s four-level framework for training assessment (no
citation is provided), thus separately addressing reactions (beneficiaries’ perceptions), learning
(knowledge/skills), behaviour (executed initiatives), and result (effects on the general
environment of the youth). Success was then measured in each of these four dimensions. The
evaluation also undertook a contribution analysis, thus recognising the challenges of fully
attributing causality for social transformations on specific projects and events, and rather
choosing to portray it as a complex and cumulative process. Finally, the evaluation engaged in
comparison, employing findings and conclusions from the evaluation of SFCG’s comparable
project “Youth and Non-Violence in Guinea”.

The evaluation details three steps of analysis: documentation review, fieldwork, and analysis.
Asides from a list and brief description of the materials employed for preparation in advance of
the field visits, few insights are given on this foundation. It would be pertinent to mention the
specific documents employed. This is done in the bibliography at the end of the evaluation.
However, the absence of in-text citations makes the attribution of specific concepts and ideas
unnecessarily problematic for the reader. This flaw is evidenced both in this and its
methodological section. These omissions make it hard to place the text within the established
body of literature on the regional context and on program evaluation processes.

Before initiating fieldwork, the evaluators conducted a theory of change workshop with SFCG
staff, based on their observation of a lack of a clear logic model. The evaluation details this
initial workshop as an attempt to get at the underlying assumptions of the SFCG’s original
project design. The results of the workshop are, however, not directly addressed on the spot.
The closest the evaluation gets to explaining the overarching logframe of the project is: “SFCG
uses an integrated approach that combines media work with community outreach to directly
address some of the key problems in Côte d’Ivoire, including the politicization of social
dialogue, marginalization of moderate voices, deepening ethnic, political, and regional
cleavages; and the preponderance of misinformation and rumours” (17). How the engagement
in the fields of media work and community work ultimately relate with the described social
problems on a theoretical level is left unaddressed. Admittedly, this might be owing to the
project’s own lack of clarity in this regard than to a direct flaw in the evaluation itself.

The fieldwork step of the evaluation followed a Mixed Methods approach, employing both
qualitative and quantitative methods of data generation, which were then triangulated and
contrasted in the analysis stage. On the qualitative side, the evaluators performed eight two-
hour focus group discussions (FGD) in Bas-Sassandra and Vallé du Bandama, two of the seven
communities that were targeted by the project. Focus groups were divided in two categories,
one involving only youth leaders that had been directly targeted by SFCG’s activities, and one
involving indirect beneficiaries, i.e. youth that had participated in events and initiatives
organised by the group of youth leaders. The evaluators also carried out several semi-
structured interviews with stakeholders, including youth leaders and young beneficiaries,
broadcast technicians, radio program directors, members of local and international
organisations and religious, political and administrative authorities. Regrettably, the evaluation
does little to present and discuss the structure of and questions asked during both focus group
discussions and individual interviews. On the quantitative side, the evaluators carried out two
separate surveys, also destined for the two groups outlined above. The surveys, which are
annexed at the end of the evaluation, are concise and easily understandable – they’re also
productively structured differently for both populations. From the evaluation, it seems like all
stages of the data generation process were carried out in a culturally sensitive way, with one
exception: there are no specific considerations on how to address gender aspects, although
women were markedly quieter and more reserved than their male counterparts. This is owed to
flawed indications by SFCG staff, which also points at the need to strengthen this aspect at the
organisational level and in the planning of future projects. The results of these processes are
presented in understandable graphics throughout the analysis chapter.

The evaluation concludes with a series of recommendations for future projects, including
improving on the selection criteria for participants, taking into account gendered identities, as
well as building on stronger partnerships and networks with local authorities and international
organisations. All recommendations have been addressed before, and they seem to follow from
observations carried out during the analytical phase of the evaluation. Its last recommendation
closes in a positive note, encouraging the continuation and expansion of the project to other
areas in Côte d’Ivoire.

Overall, the evaluation process evidenced a very strong structure, drawing from a detailed
methodology and a diverse set of both qualitative and quantitative methods that effectively
complement each other. While the main focus of the data generation process is laid on the
individual and community levels, the evaluation process goes to great lengths (such as specific
individual interviews with local authorities and international organisations) to also include
institutional and intergroup dynamics. This diversity achieves a very rounded perspective on the
real impact of SFCG’s project. Also, the inclusion of narratives on concrete examples in which
the youth leaders have made use of the skills learned through the project helps to ground an
otherwise abstract discussion. If anything, more such examples would have been desirable.
Similarly, the recommendations are successful in condensing the previous discussion into useful
and accessible advice for future projects.

On the downside, the evaluation suffers from its lack of in-text citations, which make it difficult
to locate the argument within the broader discussions in the academic and political literature.
Also, more space should have been dedicated to tracing the project’s theory of change. It was
only addressed briefly, and it did not look into the dynamics between the project design’s
underlying assumptions. The evaluation also suffers from its limited geographic scope, which
only looks at two, out of the seven communities, targeted by the project. The two observed
communities are also similar in terms of ethnic composition, which may detrimentally influence
the results. Finally, due to the timing of its execution (three months after the end of the
project), the evaluation is only in a position to analyse short-term and, to a lesser extent, mid-
term outcomes. Whether these effects will consolidate into a long-term transformation cannot
be inferred as of yet.