Oxfam GB

Evaluating Fair Trade as a Development Project: Methodological Considerations Author(s): Elisabeth Paul Reviewed work(s): Source: Development in Practice, Vol. 15, No. 2 (Apr., 2005), pp. 134-150 Published by: Taylor & Francis, Ltd. on behalf of Oxfam GB Stable URL: http://www.jstor.org/stable/4030075 . Accessed: 10/11/2012 08:46
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Taylor & Francis, Ltd. and Oxfam GB are collaborating with JSTOR to digitize, preserve and extend access to Development in Practice.

http://www.jstor.org

Development Practice,Volume Number April2005 in 15, 2,

Taylor6Francis Group

Fair as a development Trade Evaluating project: methodological considerations

Elisabeth Paul
Thisarticle identifiesthe needfor an appropriatemethodology evaluatingFair Trade,given for that most evaluationsto date have been in-house or commissionedreviews and hence have not followed a consistentapproach.Focusing on the developmentaspects of Fair Trade,the article reviews a range of impactevaluationmethodsand presents a detailed methodology analyfor sing Fair Trade. This methodology incorporatesstandardproject evaluation criteria and is based on a wide range of proven methods collecting and analysing data, principallyqualifor tative but also quantitative.Thisframework is a modularpackage from which practitioners may select according to their needs and means, while still retaining an overarching logic. The article illustrates its use by reference to evaluations undertakenin Costa Rica, Ghana, Nicaragua, and Tanzania. The approach allows for a comprehensiveunderstandingof Fair Tradeprogrammesand enables these to be comparedwith conventionaldevelopment projects.

Introduction
Developmentin the South is intimatelylinked to international trade.While Africa has incurred considerable losses through lost export earnings, participationin world trade has played an importantpart in most developmentalsuccess stories. Moreover,trade is perhapsbetter than international in helpingovercome poverty,thoughit can achieve its full developmentpotenaid tial only when opportunities distributed are fairly amongthe most disadvantaged (Oxfam2002). While the system of global tradeis controversial, forms of tradeattemptto exploit alternative commercialpotentialin the Northto improveconditionsfor producers the Southandthusconin tributeto theircountries'development.ThusFairTradeis at once a social movement,an alternative form of trade,and a developmentintervention. FairTradedeals mainly with food or craft productsand may involve direct marketingor indirectmarketingvia a label. The exact criteria anddefinitionsmay vary,but it is basicallya businessrelationship basedon the idea of a partnership between social players from the North and the South, intendedto bring about change in tradingpractices by laying down new rules (setting a fair price, paying up front, stability in tradingpartners, etc.) and raisingconsumerand public awarenessin the North,with the overall aim of improvingproducers'conditionson the economic (fight againstpoverty),social (gender and welfare issues), and environmental (sustainable development)fronts(ULg 2002-2003). For the producers,the 'added value' of Fair Trade is three-dimensional: above-market an price (the premium), price stability (because prices are fixed in advance), and sometimes pre-financing.In addition,Fair Trade is often linked to initiatives in areas such as production and organisationaltechniques, social and infrastructure projects, savings and loans, etc. Most
134
!SSN 0961-4524 Print/lSSN 1364-9213 Online 020134-17 tj 2005 OxfamGB DOI: 10.1080/09614520500040437 RoutledgePublishing

Evaluating Fair Tradeas a developmentproject

evaluationsof Fair Tradeconclude that its main benefit lies in the reinforcementof local producers' capacities ratherthan in monetarygain. Indeed,the profitsare often used to fund social and development activities and so do not directly reach the small producer infrastructures (Hopkins2000; IIED 1999; Jones and Bayley 2000; Lake and Howe 1999; ULg 2002-2003). While thereis a growingdemandfor an evaluationof FairTrade,most of the existing studies are very muchof an 'in-house'or 'consultancyreport'nature-what has been writtenaboutFair Tradehas largelybeen commissionedby those involvedin it, andtherehas been little independent scientificresearchon the subject.This researchis highly practicefocused and often of very high quality, but the methodologiesused have inevitablybeen somewhatad hoc, respondingto the concerns of the commissioningbody and so not suitablefor the purposesof comparisonwith This article seeks to addressthis shortcomingby applying a other developmentinterventions. numberof provenprojectevaluationmethodsto the subjectof FairTrade,with the goal of estabof and lishing a more 'universal'approach findingsthatwill both add to our basic understanding the role of FairTradeand also providea flexible set of evaluationtools for use in the field. We begin by definingthe natureof FairTradeas a developmentprojectbefore looking at the of appropriateness differentmethodsfor evaluatingit. We shall then set out an integrated,comevaluation methodology, and conclude with a discussion of prehensive, and interdisciplinary this approachbased on practicalfield testing.

FairTrade as a development project

Generally speaking, a project could be defined as a 'process of providing services limited in time: by using resourcesallocated for the purpose,activities are carriedout and services (outcomes) are providedwith the intentionof producinga pre-definedeffect (the objective of the project)' (GTZ 1996). The project is thus by naturelimited in time and space, integratedand interventionist-an attemptto effect change. Among the differentmodalities of development aid (programmes,technical assistance, emergency aid, budgetarysupport), internationalaid agencies typically use the projectapproach,formalisedthroughthe widespreadProjectCycle Management(PCM). While Fair TradeOrganisations(FTOs) tend to view it as an alternative in its own right, Fair Trade undeniablypossesses all the features of a development project: its goal is to support supposedly weak producersin the South. Fair Trade is the brainchild which seek to establish principallyof NGOs such as Oxfam, Max Havelaar,Twin, or Traidcraft, 'partnership'with their beneficiaries-a principle at the heart of the new development cooperationfashion adopted even by the World Bank. It manifests itself in defined and selfit containedinitiatives,and, althoughintendedto be based on 'partnership', is clearly intervenby tionist, little understood its directbeneficiariesandperceivedas an importednotion(Hopkins 2000; ULg 2002-2003). Its funds emanate from the North, as do its operatingcriteria. Of course, Fair Tradegoes beyond the conventionaldevelopmentproject,going in an ideological scale from the commercial-a business intendedto help disadvantagedproducersenter the world market-to the militant-a commitmenton the partof volunteersin the West andpart of an alternativesocial movement aimed at redressingthe balance of North-South relations (ULg 2002-2003), specifically through awareness raising and internationaladvocacy. Fair Trade also seeks to establish longer term relationshipsthan do most aid projects. type of develForthe purposesof this article,however,we focus on FairTradeas a particular intervention,leaving to one side the wider movement.This will allow us to examine the opment Fair Trade experiment using well-known and established evaluation methods in view of the growing demand for evaluation in a field still dominated by ad hoc consultancy methods. We have thereforedrawn up a logical frameworkfor Fair Trade based on the stated goals of several organisations in relation to development. Such a logical framework does not, of
Development in Practice, Volume 15, Number 2, April 2005

135

Table 1: Logical FrameworkAnalysis (LFA) for Fair Trade Logic of the intervention Global objectives * Improve traderelations in favour of producers from the South * Achieve sustainable productionand consumption Project-specific objective * Improveproducerliving conditions and reduce poverty
S

Means of verificatio * Increase of social capital * Growth in FT market share * Drop in number of rejects * Respect for and/or improvement of naturalcapital * Increase producer incomes ^ Increase savings * Increase physical, financial, and human capital
* Difference between market price

* Evaluation of net * Statistics from EF etc. * Standardsand lab * Evaluation of the environment
*

Survey/enquiry * SRL evaluation * Accounting recor of organisations

| Expected outcomes 1. Growth in producerincome

* Worldwide statis * Statistics from and price actually paid FTO/organisation * Amount of total premium paid Premium received by the producer * Producer survey * * Number of jobs created

m-

2. Stabilisationof prices

* Amount of advance payments and updating of premium ' Developmentofaverageprices * Amount of loans granted (if microcredit bank open) * Type of organisation * Size * Decision-making/participation
method

Statistics from FTO/organisation

13. Fair working conditions

* Diagnosis by org
*

Survey

* Label (criteria co

* Proportionof women Salary/wage differential * Respect for work standards * IEC activities/conscientisation

4. Reinforcementof capacities

* SRL evaluation * Diagnosis from o (infrastructures) * Increase in human capital (services) * Use of new production techniques * Improvements to management * Increase in physical capital methods (equipment, tools ...)

* Use of risk-coverage mechanisms * Number of full-time/qualified managers 5. Direct access to markets and marketing channels * Increase in social capital (diversification of networks/
customers)

* Structureof integrateddistribution * Diagnosis and sta * SRL evaluation

organisations/FT

N
d

_ 6. Viability of the activity o

* Existence of label/trademark/ quality standards | * New products

* Volume, value, and

* Statistics from or

development in sales Financial/economic productivity * Level of dependency in relation to FTO * Openings to higher value-added activities * Natural capital * Environment-friendlytechniques Resources and activities 1. Premium(payment of higher-than-market price)
_

FTO
* SRL evaluation

Means Quantity * Financial resources * Human resources

* Training sessions

Costs

Origin of * Profits * Higher

paid by

Table 1: Continued
+Logicof the intervention

Means of verificatio
* Operatingcapital * Contingency

| 2.1 Paymentsup front 2.2 Long-termpartnership

10%

|
*

Volunt

* Donati

L
3.1 Respect for rights, democracy, and transparency 3.2 Special measures
in favour of women

TOTAL

Tax cu

ti It

4.1 Materialinvestments 4.2 Technical assistance activity: * 4.2.1 Trainingon productiontechniques | 4.2.2 Management | advice and tools * 4.2.3 Legal assistance 4.3 Communityprojects: - 4.3.1 Health
* 4.3.2 Education

5.1
x

| 5.2 5.3 6.1 6.2 6.3

I 4.3.3 Water Direct access to markets (cut out the middleman) Improvementof quality New products Economic productivity Diversificationof activities Environmentalmeasures

Izz

EvaluatingFair Tradeas a development project

course, have immutableor universalvalue but offers an insight into the logic and philosophy underlyingthe average Fair Trade initiative (see Table 1).

Evaluationmethods and theirappropriatenessfor FairTrade

Broadeningthe scope and rigour of Fair Trade evaluation allows us to compare it with other developmentprojects.The five criteriaused by the OECDDevelopmentAssistance Committee (DAC) are: * Effectiveness-the attainmentof resultsby referenceto the logical framework(or a more or less formal equivalent). * Sustainability-estimates the probabilityof the project and/or its benefits continuing after external assistance stops, and relates to criteria such as 'ownership' by the beneficiaries, of respect for the environment,appropriateness the technologies, economic viability, etc. * Relevance-the appropriateness the projectin tackling the actualproblemsand priorities of of the beneficiaries,as well as the wider environmentin which the projecttakes place. * Efficiency-the ratiobetween the benefitsand the qualityof the resultsand theircost. This is accomplishedmainly througha cost-benefit analysis or a cost-effectiveness analysis if the benefits of the action are difficult to quantify (Baker 2000:1-16; EuropeanCommission 2001). * Impact-measures a project'seffect on its environmentand its contribution the wider polto itical and sector-basedobjectives. Evaluatingimpact is an attemptto isolate the results-or outcome-of an action from otherpossible factorsand aims to determinethe relationshipof causality. Quantitativemethods for evaluating impact are based on the estimation of a counter-factual which considers what might have happenedhad there been no intervention. Thus the studygroup(the beneficiaries)is comparedwith a controlgroup,selected at random from among the same population,or a comparativegroupthat is not involved in the project. the Qualitativemethodsestimatethe impactof a programme tryingto understand existing by processes, behaviour,and conditions. These methods better capturethe perceptionsof the stakeholders,but are more subjective and less 'robust' since they do not addresscausality. In practice, one often resortsto a sort of 'grey' evaluationin which the searchfor causality (quantitativeapproach)is abandonedin favour of plausibility (qualitativeapproach). Below, we set out in brief the differentmethods generally used for evaluating the impact of development programmes, and discuss their appropriatenessfor evaluating Fair Trade. Where possible, we illustratethese remarkswith concrete examples drawnfrom case studies conducted by the University of Liege (ULg 2002-2003) relating to coffee in Tanzania and Nicaragua,and bananasin Ghanaand Costa Rica. Quantitative methods Quantitative evaluations aim to establish what the outcome would have been without the intervention in question. The scientific proof of a link of cause and effect relies on a 'with-and-without'comparisonbetween two populationsets, or on a 'before-and-after' comparison, where the test group and the control group are the same and the impact is calculated by comparingthe situationbefore and afterthe intervention(Baker2000; World Bank 2002a). Experimental designs (randomisation) involve the randomdivision of the individualseligible for the programmeinto two groups:one which receives the intervention(the study group) and the other which does not (the control group). Both groups are statisticallyequivalent, which enables impact to be determined by the comparison-of-averagesmethod. Though sound,
Development in Practice, Volume 15, Number 2, April 2005

139

Paul Elisabeth however, these methods could not realisticallybe applied to the evaluationof Fair Trade:it is difficultto envisage a FairTradeorganisation being bent on evaluationto the pointof conducting an experimentwithin a cooperativewhich entailed the randomselection of beneficiaries! Quasi-experimental designs arenon-random methodswhich comparethe studygroupwith an equivalentgroupthat did not benefit from the intervention.The constructedcontrol (matching) method identifies non-participantswho are basically comparable to the participants, and matches the two groups on the basis of certain observed characteristicswhich are deemed to influence the results of the programme.The reflexive comparison constructs the counterfactualon the basis of the participants' circumstancesbefore the programme,so they constitute both the study and the comparativegroup. One might envisage these models being used for evaluatingthe impactof FairTradeprovidedone had quantitative dataon sufficientlyrepresentative samples. For instance, one could establish a control group with producersfrom outside the Fair Trade network to be compared with one within it. For example, in our study of coffee in Tanzania,we comparedthe situationof producersbelonging to cooperatives which were membersof unions who, in turn,were the FTO partners,with the situationof producers fromthe same villages but who were not partof any cooperative.Nevertheless,the lack of quantitative data limited generalisationof the outcomes observed. One alternativeis to comparethe groupbenefitingfromFairTradebefore andafterthe 'intervention'. Laterin this paper,we shall describethe applicationof such a model and attemptsto reconstitutedata (indicatorsof poverty, welfare, etc.) on the situationbefore FairTradebegan. This appearsvery relevant,but shouldbe approached with cautionbecause the evaluatorhas to attemptto reconstitutethe situationas it would have been now in the absence of the intervention-which is not the same thing as the situationex ante. For instance, the fall in the world coffee price at the end of the 1990s is an exogenous factor and would have to be discounted from any projections.However, this risks maskingthe real impactof FairTradewhich actually softened the blow for many producers,enabling them to escape the more brutaleffects of the coffee crisis. Finally,in theory,the participants a programme in may be comparedwith non-participants by non-experimentaldesigns, i.e. by using statistical methods such as instrumentalvariables or multivariateanalysis. However, in practice these methods are too complex to be of interest in evaluating Fair Trade, especially since the impact of Fair Trade is intangible ratherthan quantifiable.

Qualitative methods
Qualitativemethods are essential in evaluatingimpact to apprehendthe processes underlying the results observed, assess change in the perceptionsof the stakeholders,and help to evolve hypotheses. In particular,those known by the generic name of Rapid Appraisal Methods (RAMs), located somewhere between the formal and informal ways of collecting data, have gained legitimacy over the past 20 years. The most common methods employed are the key informant interviews, focus groups, community or group sessions, direct, structured observation, and the informal survey. RAMs obviously have their limitations (absence of random sampling, influence of the interviewer'sjudgement, low validity and reliability of the information, etc.), but these are counterbalancedby their relative strengths: they can provide relevant informationrapidly,contributingto in-depthunderstanding the processes, of their flexibility enables new lines of enquiry to be pursued,and they requirerelatively little investmentand few resources (Kumar1993:1-22). On the whole, those evaluatingFair Tradeare likely resortto rapid,qualitativemethods for two basic reasons:the limited means availableandthe natureof the subjectmatter,thatis to say 140 Developmentin Practice, Volume15, Number2, April 2005

EvaluatingFair Tradeas a development project

a projectwhich is containedand where the impact is principallynon-monetary. They will tend to take the form of monographs,from which more generalresultsmay be extrapolated,and are the most widely used qualitativemethods in impact evaluation (World Bank 2002a). Clearly,the in-depthinterviewis bettersuitedto collecting datarelevantto FairTradethanis the quantitative questionnaire. The field studies we carried out used different interview approaches,each with its strengthsand weaknesses, and were substantiated a guide to interby viewing or at least an analysis matrix, to enable the informationto be systemised and for comparative purposes. Thus, standardised,open interviews with the heads of organisations and the producers enabled the collection of systematic, quantifiable,and comparabledata, specifically on productionand living costs, as a basis for evaluating the economic outcomes of the intervention.Semi-structuredinterviews with key informants(heads of cooperatives, partnerinstitutions,research centres, etc.), enabled more intangible socio-economic impacts to be detected, while informal discussions assisted in identifying lines of enquiry and posing researchhypotheses.Finally,joint interviews and focus groups (with the additionof participatory methods, discussed below) provide a range of perspectiveson a given social dynamic. Direct observationis an unavoidableelement of any evaluation, if only for drawingup an inventoryof the projectcomponents(infrastructure service projectscarriedout as a conseor quence of Fair Trade, etc.). A well-prepared observation matrix allows the researcher cheaply and quickly to obtain useful details on the activities, behaviour, and equipment related to the project. In our analysis of projects on the ground, these observationsrelated not only to the materialoutcomes and the actual working conditions but also to the strategies being adoptedand the local socio-political realities. Finally, informationgatheringcannot get by withoutreferenceto existing documentationas much for preparingfield studies (statistics,reportsrelatingto the topic or dealing with comparable interventions,etc.) as for completing them (contracts setting out the obligations of all parties, the FTO's own criteria, quantitative data on prices and production costs, to be broughtinto sharpfocus, any other rules and regulations,etc.). In Tanzania,for instance, an analysis of sale prices obtainedat auctionrevealedthatalthoughour key informants maintained that the Fair Tradepremiumwas used to rewardproducersof high-qualitycoffee, in fact highquality coffee was sold at a markedlyhigher marketprice anyway. This finding enabled us to develop a new hypothesisand deepen our understanding how the premiumwas used. Crucial of data such as these can be collected at little cost and from varioussources,whetherinternational organisations,governmentministries,or producerorganisations. Participatory methods A set of methods known generically as Rapid Rural Appraisal(RRA) were developed in the early 1980s (Chambers1981). The RRA follows precise rules and uses a non-standard of set methods for collecting and analysing information,rangingfrom semi-structured interviews to analytical games. On this basis, miscellaneous methods of participativeenquiry have been designed with a view to bridgingthe social and cognitive gap between a project'sbeneficiaries and its evaluators(Neubert 2000:78-80). RRA methods are often used to complement other more formal methods. Participatory techniques seem wholly appropriate the 'philosophy' of Fair Trade, given to that it is founded on the principles of partnershipand strengtheningof local capacities; they could be used to evaluate existing projects,give them a new direction,or to identify new modalities. Severalmethods(for example, those presentedat WorldBank 2002a) could very well be used in the context Fair Tradeprojects.For instance,BeneficiaryAssessment is a consultative methodology,based on obtainingsystematicqualitativeinformation,which aims to enable the
Development in Practice, Volume 15, Number 2, April 2005

141

Elisabeth Paul

voices of the beneficiariesand otherstakeholders a projectto be heardby those monitoringit. in methodsbuild upon the capacitiesof the most disadvantaged Education/training populationsby engaging their creative capacities in project planning,problem solving, and evaluation.Participatory Rural Appraisal(PRA) is based on the principles of joint participationand learning, flexibility, and triangulationof information,while its systematic side ensures that outcomes are valid and reliable. Interviewsand focus groupsthat use such techniquesmake a majorconthe tributionto understanding social dynamics, commitments,and impact of Fair Trade. For instance,our case studiesused variousvisual techniques,includingchartsillustratingthe history of the communityand highlightingchanges which had occurredin orderto show how living conditions had developed and to reconstitute a before-and-aftercomparison group. Institutionalchartsrepresentingthe various differentgroups and bodies presentenabled us to understandtheir relationships and their respective importance in decision making. In the same way, exercises in classification (of problems, of preferences regarding different options,etc.) were also used, togetherwith analysisof trends-specifically of the seasonaltimetable-and diagramsof the daily activities of the differentsubgroups. Finally, while the producerswe met seemed relativelyautonomousto us, differenteducation/ trainingtechniques can be used to evaluate Fair Trade. Creative techniques employing open visual tools (non-sequential posters),andresearchtechniques(pocketcharts)or analyticaltechniques (sorting into three piles), stimulate producers'creativity and allows them to prioritise their problems and examine certain questions in detail. Planningtechniques(a story of shortcomings .. .) can also assist in decision making by getting the less powerful to take part. It is mainly thanks to these participatorysessions that we were able to understandproducersand to tackle power and gender issues so as to see who most benefits from Fair Trade(employees or smallholders,genderandfamily issues, etc.) andbe in a position to suggest ways of improvement, both as regardsthe terms and conditions of Fair Trade-for example when to pay the premium-and in relationto subsequentprojects (technical assistance, social projects,etc.).

Otherspecific tools
Our methodology for evaluatingFair Tradeis based on a structureclose to the World Bank's Poverty and Social Impact Analysis (PSIA) model (World Bank 2002b), which aims to analyse the distributionalimpact of reforms on the welfare of different groups in terms of income and other dimensions of poverty. A PSIA comprises: 1. 2. 3. 4. 5. 6. analysis of the protagonists,necessary for understanding interestsat stake; the identificationof the channels throughwhich the reformis likely to have an impact; analysis of the implementinginstitutions; gatheringand processing of information; social impact analysis; risk analysis, togetherwith monitoringand social accounting.

We adoptedthis approach,supplementedwith otherprovenmethods,because it enables evaluation to be undertaken a rigorousand comprehensivemanner.It seems obvious to startany in evaluationwith an analysis of the protagonistsand institutionsin place. The analysis, comprising tools such as static or process mappingwhich identify the organisationsin chargeof implementingthe projecttogetherwith their currentrelationships,practices,and standards, work the requiredto improveperformance,etc., or additionalevaluationtools to assess the effectiveness of the institutions,or allow for the identificationof the rules and incentives affectingbehaviour and possible project constraints. 142
Development in Practice, Volume 15, Number 2, April 2005

EvaluatingFair Tradeas a developmentproject

the Impactanalysisaims to understand effect of feedbackfrom the reformsbeing undertaken. The PSIAs proposedirectimpactanalysis,behaviouralanalysisor othermethodssuch as partial or general equilibriumanalyses, all clearly too sophisticatedby far for the needs of evaluating Fair Trade. On the other hand, the following tools are well suited and relevant: * Social Impact Assessment (SIA) analyses how the costs and benefits of a given reform are distributedamong the stakeholdersinvolved over time. It uses a set of instrumentsfor collecting qualitative data (targeted surveys, key informantinterviews, beneficiary workshops, etc.) to evaluate the assets and capacities which people need in order to benefit from a programme, and helps with understandinghow social assets and institutional mechanisms affect the outcomes. When an SIA was carriedout on the bananacase study in Ghana, it showed that the weak structureof the cooperativelabourforce combined with the absence of relateddevelopmentprojectslimited the impact of Fair Trade. In Tanzania, the same analysis enabled us to monitor the evolution of the Fair Trade premium:first it was used for operatingpurposes(recoveringcosts of the cooperativeunions) and strengthening capacities (training,etc.), but recently,a sharehas been paid back directlyto some of the producers. * A poverty chart showing its geographicaldistributioncan be useful at the nationallevel to identify producergroups for futureprojects,and to evaluate an existing projectat the local level. * Througha series of analyticalexercises, the Participatory PovertyAssessment(PPA) enables the views of the poor to be takeninto accountin assessing the impactof any action, and is of interestfor understanding non-incomedimensionsof povertyand the processes the particular at work in any intervention. * Evaluationof social capitalby surveyinginstitutions,networks,relationships,standards, and values which govern interaction between people and which contributeto social andeconomic development at different levels (households,communities,organisations).It identifies how corporateassets determineaccess to resources(via transactioncosts), and affect productive behaviourand the responseto the intervention. This appearsvery appropriate FairTradein to view of the fact that social capital conditions survival strategies,access to resources(including the Fair Trade premium),and the reductionof risk for the producers. Ourcase studies confirmedthe importanceof social capitalboth as a preconditionfor Fair Trade (the producersconcemed being those already enrolled in cooperatives whereas the least favoured often remain outside the Fair Trade network, something which could be viewed as a deficiency of this particulardevelopment project);and as an impact (contact with the FTOs generally leads to an increase in the social capital of producerorganisations which then, thanks to the networking and technical assistance from which they benefit, enables them to find a marketniche, win new markets,and even become the benchmark for other developmentprojects. Our evaluation methodology also drew inspirationfrom the MAPP method (Neubert 2000), designed to meet the needs of the Germandevelopmentcooperationagencies. It constitutesa simple, transparent, satisfyingly valid method of impact evaluation,since it is based on a and reasonablysound method of assessing the changes which occur in the beneficiarygroups. It aims to determine whether poverty has been reduced and how the benefits of the project have been distributedamong the different social groups. It is based on the concept of key social processes, which assumes thatone can limit the numberof relevantfactorsand operationalise the community'ssituationthroughfour criteria:living conditions,resources,knowledge, and rights. It is generally based on DFID's concept of SustainableRuralLivelihoods (SRLs),
Development in Practice, Volume 15, Number 2, April 2005

143

Paul Elisabeth consisting of definingthe position and developmentof differentgroupsaccordingto the fivefold yardstickof their natural,human,physical, social, and financial assets. In concreteterms,the MAPP methodconsists of applyingthe PRA surveytechniques,modified and combined to form a logical system. It comprises five complementaryparticipatory instrumentswhich permit the development profile of the community and the impact of the comprojectto be drawnup. The evaluationmodel is based on hypothetical'before-and-after' parisons of living conditions, based on the 'systematised' memory of the target groups. It is implementedthroughopen discussion groups for collecting data from people's recollections, from the observationand systematicmoniin the same way as for triangulating information, the toring of the project. We have retained the first two stages: a 'life-line', which enables the minimumfactors for life in the targetgroups, showing the yearly changes in their perceptions of their quality of life; and a 'trendanalysis', which gives a differentiated pictureof the community's social developmentand profile, and majorchanges and the reasons for them.

An integratedapproachto evaluatingFairTrade
Whateverthe respective qualities and shortcomingsof the various methodsfor evaluatingFair Trade,it is useful to integratethese in orderto facilitate monitoringof consistency, to analyse the situationat differentlevels, or even to respondretroactivelyin orderto betterinterpretthe outcomes. Quantitativemethods may be the subject of statistical analyses and give generalisable results, but they are costly to administerand their scale far exceeds the needs of a Fair Tradeproject. By contrast,the qualitativeapproachand, specifically, the rapid appraisal method, corresponds better to the objectives of and means available for such an evaluelements and ation, particularlysince the impact of Fair Trade arises from non-quantifiable processes. We thereforeput forwardan integrated,sound, and flexible methodologyfor evaluatingFair Trade,' arising out of a multidisciplinaryapproachcombining economic and social sciences, and mainly using the rapidappraisalmethods.It is essentially qualitativein naturebut includes some quantitativeaspects, specifically for the economic analysis contentand throughmatching with quasi-experimentalmodels (comparison with a group of non-beneficiariesor reconstitution of a reflexive comparisonof the producers'before-and-after situation).It incorporates differentmethodsfor collecting and analysingdatain orderto obtaindifferentiated perspectives and to enable consistency of monitoring.The methodologicalarchitecture similarto that of is the PSIAs but adaptedto the scale of the study. It comprises three stages: 1. An analysis of the protagonistsand the institutionsbroughtinto play; 2. The evaluationproperof FairTradeaccordingto classic projectevaluationcriteria,insisting on the analysis of the stakeholders'strategicchoices; 3. A risk assessment. In addition to document review, data are gatheredthroughinterviews (individualand group, informal and semi-structured),and direct observation, and some participatorymethods. Within the structure,evaluatorsmay select those tools they deem appropriate, accordance in with theirown termsof referenceandthe means available.2This packageof methodsmay therefore be combinedby tryingto split the effects of the three 'instruments' FairTrade(premium, of price stabilisation,and relatedprojects),and to capturethe variousoutcomes dependingon the methods of interventionadoptedby the FTO. 144 Developmentin Practice, Volume15, Number2, April 2005

EvaluatingFair Tradeas a development project

Analysis of the protagonistsand of the institutions

This study serves to identify the interestswhose supportor opposition influence the impact of Fair Trade, and to analyse the relationships,capacities, and internalprocesses of the organisations involved. It comprises several stages and tools available: * The identificationof the playersandmarketstakesat the nationallevel, on the basis of a trend analysis.3 * The studyof the local arena(Olivierde Sardan1995), of the local players(socio-professional analysis), relationshipsof power, and existing or potential strategies and conflicts. This is carried out through document analysis, interviews with key informants,and direct observation, in particularusing participatorytechniques such as social charts, wealth ranking, seasonal schedules, and daily activity diagrams. * Marketanalysis based on statisticaldata. * Organisational mappingof the FTOs, supportbodies, and producerassociations, i.e. inventhe means, players, and their relationshipsin orderto understand tory of behaviour,internal This stage uses documentanalysis (articlesof processes, and the writtenand unwrittenrules. association or statutes,contracts,FTO strategicdevelopmentplan, etc.), direct observation, and interviews with key informantswhere participatory techniquescan be introduced.

Evaluationof FairTrade
Effectiveness and strategicanalysis
an This stage shows the concreteresultsachieved by FairTradeand undertakes in-depthanalysis of the strategiesimplementedby the variousplayers. First, effectiveness must be measured (i.e. were the FairTradeobjectives attained?)by comparingthe resultsachievedwith a 'virtual' logical framework.Wherepossible, the outcomes can be refinedby comparingthe study group with a counter-factual, with producerswho arecomparable not engagedin the FairTrade i.e. but network. Strategic analysis aims to determinethe players' logic and strategies,and the groundsfor theirchoices concerningdifferentproductsand commercialsectors.This socio-anthropological their behaviour. analysis is crucial to understanding Our study in Nicaragua,for example, broughtto light one strategyof the producerswho, in the event of competition on the marketor the need for immediate cash, sell their coffee to private buyers and not to their Fair Trade cooperative partner.In our Costa Rican study, the bananas are sold mainly through the same importerwho in tum decides what share will be allocated to the Fair Trade market.This means that strategicdecisions about how production is allocated is not within the gift of the producercooperative. This analysis is also of interest from the point of view of the new institutionaleconomy (marketimperfections,asymmetryof data, the theoryof games, etc.) for detectingthe possible existence of coordinationpolicies which might improve the situation. While this part of the analysis needs to be placed within a consistenttheoreticalframework,the empiricalevaluation may be based on direct observationand the semi-structured interview with, if possible, validation of the hypotheses derived from the statistics.

and cost-benefit analysis Efficiency

This entails splitting the costs and the increase in value between the different levels of the sector (Fair Trade and classic) over time, just as for the allocation and use of the Fair Trade premiumbetween the levels. These data may be identifiedfrom documents(the organisations'
Development in Practice, Volume 15, Number 2, April 2005 145

Elisabeth Paul accountingrecords,statistics,etc.), but also from interviews.In addition,it is worthundertaking a cost-benefit analysis (or at least a cost-effectiveness analysis) of the Fair Trade project, including its dimensions in the North, in order to compare it with conventionaldevelopment projectsintendedfor similar target groups.

Impact
The most significantelement in any evaluationof Fair Tradeis to find out what impact it has had-whether intendedor not. If at all possible, it is interestingto separateout the impacts of different Fair Trade instruments,and to differentiate them according to their nature-for instance in terms of incomes and living and working conditions, access to markets,capacity strengthening (technical, knowledge, organisation, governance, political influence, etc.), gender, environment,and more intangibleimpacts (social capital, self-esteem, etc.). This analysis is based on interviewswith key informantsand in groupsessions incorporating some participatory the techniques,and triangulating informationobtainedthroughobservation. It can be combined with an SIA in orderto find out how the project's costs and benefits have been distributed,how the projectis received, the assets (physical and financial)and capacities (human and organisational) required in order to derive any benefits, strategies adopted, and to detect any possible negative effects of the project. In the case of the NicaraguanFair Trade project, the beneficiaries genuinely appeared to have taken up ownership of the scheme. Although it was initially an imported idea, the project is responding to producer demand, and producersfeel involved in its achievement-which is not the case, for example, in Tanzania. An SRL analysis could be used to assess the development of different aspects of the 'capital'-natural, human, physical, social, and financial-of the beneficiaries (producers and their associations). In particular,evaluation of the social capital of producersand their organisationmakes it possible to assess how social assets (relationships, institutions,networks, etc.) determineaccess to resourcesand affect productivebehaviour.Ourresearchers' reportsare structured aroundthe impact of Fair Tradein relationto these five 'capital' aspects. The identificationof the key social processes in the context of a PPA reveals producers'own view of poverty, the strategies which could reduce it, and developments in their living conditions (improvement or worsening), of their resources (access or exclusion), knowledge (expansion or reduction), and rights (participationor deprivation).This part of the analysis offers a particularly good opportunity suggesting new ways to use the Fair Tradepremium. for Visual andparticipatory techniquesarewell worthemploying duringgroupsessions. The use of life-lines, trend analyses (seasonal schedules, activity diagrams),and chartingor mapping techniques-in particularhistorical and social charts-gives a better perception of developments in producers' living conditions and commitments, together with distributionof the impactof FairTradeover time, in space, andby social subgroup.Otherparticipatory techniques can also be used as the evaluation team sees fit, particularlyto highlight gender-baseddifferences in perceptionand to suggest ways of improvingthe project.It is importantto note that analyses based on producer perceptions and a 'before-and-after'comparison need to take careful account of natural 'off-project' evolution, in order to discern the impact proper of Fair Trade.

Viability
This entails preparinga diagnosis of the long-term viability of Fair Trade and of the current productionmode. The viabilityof the productionconditionsshouldbe consideredin accordance 146 Developmentin Practice, Volume15, Number2, April 2005

EvaluatingFair Tradeas a development project

with the threedimensionsof sustainabledevelopment,namely economic (dependencyrate,cost recovery, etc.), social (social and political dependency,institutionalcapacities, etc.), and ecological (naturalresources,special measuresfor the environment,etc.). The viability of the Fair Tradeapproach, as particularly regardsits 'ownership'by the beneficiariesandits capacitiesfor growth and self-financing,can be based on a SWOT (Strengths,Weaknesses, Opportunities, Threats) analysis. For example, the organisation studied in Nicaragua uses the Fair Trade premiumto finance a capitalisationprogrammein orderto become autonomous.In the Costa Rican cooperative, however, the producersare only able to survive thanks to the premium, which shows the financialimpact but perhapsalso the project's lack of viability. Relevance The final criterionis that of the relevance of Fair Trade, to assess whether it is an adequate means of supportingsustainabledevelopmentin the light of the other four criteria,compared with a traditionaldevelopmentproject.Ideally, one would comparethe Fair Tradeexperience with projectscarriedout with a similar targetgroup.

Risk assessment
This step emphasises the limits both of the evaluationand of the interventionunderreview in orderto assess the potentialfor scaling up the FairTradeprogramme. This entails firstsystematically spelling out the assumptionsunderlyingthe evaluationand identifyingthe risks associated with it in terms of the likelihood and consequences of its assumptionsproving valid or invalid. It is importanthere to flag up complementaryhypotheses or assumptionsas well as unresolvedquestions which might be worth studyingin greaterdepth.Then the risks and constraintsinherentto Fair Trade which might constrainits potential for being scaled up should also be highlighted.

The methodologyput to the test: principal outcomes

This methodologywas field testedin 2002 and2003 on FairTradecoffee in TanzaniaandNicaragua, andbananasin Ghanaand CostaRica. With the exceptionof the Ghanacase study,which was contractedout, the team from the Universityof Liege was composed of one researcherin each settingassistedby a local consultantplus a multidisciplinary supervisoryteam.These practical applicationsallow us to draw some initial conclusions on the methodology. The methodology's main advantage lies in its broad, multidisciplinaryapproachto Fair Trade,and in the flexibility of the tools at its disposal, which can readilybe adaptedto different subjects,conditions, and fields. It allows for comparisonon the basis of the five classic evaluation criteriabetween differentinterventions,whetheror not they arerelatedto FairTrade.Each of the methods proposedhas proved useful for collecting particularinformation,in particular the participatory techniques which producedfascinatingresults, especially in contexts where the impact of Fair Trade on the producersis palpable. It proved necessary to compare information gatheredin the field with documents or statistics in order to check assumptionsand generatefurtherresearchquestions. The logical frameworkwe put forwardappearsto fit our purposereasonablywell, althoughit is worth fine tuning to each context since FTOs sometimes have medium-termobjectives and differentways of working.The flexibility of the methodologyallows for these adaptations. The logical frameworkhad to be adaptedin the case of Tanzania,for instance,where the FairTrade are partners cooperativeunions,in orderto capturethe impactof FairTradeon the unionsrather
Development in Practice, Volume 15, Number 2, April 2005

147

Elisabeth Paul

than on the producersthemselves, whereasthe frameworkin the case of Costa Rica was more focused on the grassrootscooperatives. severalteams in very differentfields showed thatusing the formatof a common Coordinating matrix (providedit is applied flexibly) enables comparableresults to be obtained.Apartfrom the inherentlimitations to some of the specific techniques employed, however, one problem peculiar to this methodology lies in the membershipof the researchteam: the team needs to be equippedwith both economic and socio-anthropological skills to analyse such diverse information. In addition,while this methodologyprovideda good overview of the evolution of producers' living conditions, the difficulties in establishing an acceptable counter-factualmeant thatit was not always easy to distinguishthe independentimpactof FairTrade-so the evaluators have to make a special effortto do this. Thus, this comprehensivebut flexible methodology allows the evaluatorsplenty of freedom,but it is also highly dependenton theirhaving a critical mindset, as the following example shows. While most of the producerswe met appearedto be unawareof the benefits of Fair Trade, they were surprisinglyknowledgeable in one Nicaraguancooperative. This was, however, almost certainly due to the number of Fair Trade field monitors and evaluators passing through. In fact, our researcher (the seventh foreigner to question some of the peasant farmerson the subject) dubbedthis particular cooperativethe 'touristtrail for sociologists'! While this article does not seek to produce generalisablefindings on Fair Trade, our field studies did confirm the distinction between the two trends within the Fair Trade movement: 'business' at one end of the spectrumand 'militancy' at the other. In practice,these leanings are manifestin the differentinterventionmethodswhich are influencedboth by the philosophy of the FTO in question (workingwithin a closed circuit with stable partnersor launchingproducers on the world market), and by the historical backgroundand context of each specific setting. Thus, Fair Trade coffee in Tanzaniainitially had an ideological goal, while the Fair Trade involvement in bananasin Ghanahas more of a commercialorientation. Ourstudiesalso confirmedthatthe impactof FairTradedependson the contextsandmethods of intervention,but relates primarilyto the strengthening local capacities,both of producers of (e.g. throughtrainingin production In techniques)andespecially of organisations. Tanzania,for instance, because the cooperative unions are the only Fair Trade partners,Fair Trade has a strongimpact on strengthening unions' capacities.The unions have benefitedfrom liberalthe isation (thanksto supportfrom the FTOs) and have become significantplayers in the export market, going on to finance training for their members and now working towards a quality approach-which nicely illustrates the dynamic behind this type of intervention. This is also the case in Nicaragua,where the FTOs played a significantpart in setting up the coffeemarketingorganisationwhich we studied, and in the training and commercial security they have providedthe grassrootscooperatives.However, the most noticeable impact has been on the producersthemselves as they have benefited both from improvedearnings and from and price stabilisation. Overall, we found that one of the principalareas in which Fair Trademakes an impact is in enlargingmarketaccess for the producersby providingopeningsfor new products,in particular by providing additionalincentives for productionquality and for convertingto organic crops which are growth marketsand thus offer a greaterguaranteeof sustainability.In addition,in most of the cases we studied,FairTradehas indeed strengthened partners'capacities(technical and managerial),promotedthe cooperative principle and attentionto gender issues-though there is still room for improvementin that regard. Anothermajor advantageof this methodology is to enable comparisonbetween Fair Trade and the conventional development project, in particular in terms of efficiency-though again there is still more to be done in this area. Nevertheless, it does appearthat Fair Trade 148
Development in Practice, Volume 15, Number 2, April 2005

EvaluatingFair Tradeas a developmentproject

complements ratherthan competes with other development projects.Thus, Fair Trade seems more appropriate to bringgreaterbenefitswhere the beneficiarycooperativesare also supand ported by NGOs, as was the case in our Nicaraguanstudy. On the other hand, in Ghana,the impact of Fair Trade is limited by the absence of related development projects and by the poor level of organisationamong membersof the cooperative we studied. Finally, while a more detailed efficiency study should corroboratethe economic viability diagnosis, the Fair Trade approachappearsto offer certain assurancesfor medium- to longterm viability, particularly by promoting organic crops and emphasising high product quality, an opening to growing marketsand, in some cases, ownershipby the beneficiaries.

Conclusions
This articlehas addressedthe need for a soundand appropriate methodologyfor evaluatingFair and the growingdemandfor evaluation, Trade.Recognisingboth its potentialto reducepoverty we first defined FairTradeas a developmentproject,using a logical framework.We then preto sented a range of impactevaluationmethodsand discussed their appropriateness FairTrade the before going on to presenta detailed methodologyfor analysingFair Trade, incorporating classic projectevaluationcriteriaand based on a wide range of proven methods for collecting and analysing qualitative and quantitativedata. This is a modular evaluation framework,a package from which practitionersmay pick and choose according to their needs and means, while still retainingan overarchinglogic. The advantageof this approachis to be found less in terms of the methodology as such than in the application of sound project evaluation methods. This allows for a comprehensive understandingof Fair Trade, provides for basic researchsuch as applied evaluations,and permitscomparisonof Fair Trade with other development projects. This method-which is interestingin itself in termsof its multidisciplinary to global approach evaluationand its pooling of perspectivesand otherrecognisedmethods-has providedconvincing initialresultsin the field. In particular, is readilyadaptedto differentsubjectsand realities it becauseof its participatory, qualitative dimensions,while at the same time retaininga soundconThe differenttechniquessit well withinthis integrated ceptualand analyticalframework. framein work, particularly terms of the critical mindsetand the insistence on reviewing the working hypotheses. The flexibility of the methodology offers interestingpossibilities for Fair Trade and practitioners researchers because,unlikeall the one-off studiestailoredto suit the evaluators' aims, it makes possible the empiricalvalidationof theoreticalassumptions aboutFairTradeand opens new lines for research.Finally,the originalityof the methodologicalapproach outlinedin this article, which calls on evaluatorsto maintaina critical mindset, is to enable comparisons between very diverse FairTradeexperiencesand developmentprojects.

Acknowledgements
This article is based on a researchpaperentitled 'Un commerceequitableet durable,entre marcheet solidarite:diagnostic et perspectives' ['SustainableFair Trade, between Business and Solidarity:Diagnosis and Prognosis'] resulting from research carried out by the University of Liege (ULg 2002-2003) on behalf of the Belgian Federal Science Policy Office. Thanksto my colleagues for their contributionsto this article, in particularF. Navez, G. Pirotte,G. Pleyers, M. Poncelet, and S. Vancutsem.

Notes
1. The list of main indicatorsto be identified, interview guides, and analysis matrices relating to the variousdifferentmethods is available from the author. Development in Practice, Volume 15, Number 2, April 2005

149

Elisabeth Paul 2. For example, a small-budgetevaluationcould be confinedto interviews,whereasa largerundertaking date. in could use systematicconsultationsor even a formalquestionnaire orderto gatherquantitative be studied if this has not alreadybeen identified. 3. Choice of arenato

References
Baker, Judy L. (2000) Evaluating the Impact of DevelopmentProjects on Poverty-A Handbookfor Practitioners,Washington,DC: World Bank. and Chambers, Robert (1981) 'RapidRuralAppraisal:rationaleand repertoire',Public Administration Development 1(2):95-106. Deutsche Gesellschaft fur Technische Zusammenarbeit (GTZ) (1996) Objectives Oriented Project Planning (ZOPP): A Planning Guidefor New and Ongoing Projects and Programmes,Eschborn:GTZ. European Commission (2001) Manual-Project Cycle Managementet Project Cycle ManagementTrainingCourses Handbook,Version 1.1, Brussels: EUROPEAIDCo-operationOffice. Hopkins, Raul (2000) 'Impact Assessment Study of Oxfam Fair Trade-Final Report', Unpublished Report,Oxford:Oxfam GB. International Institute for Environment and Development (IIED) (1999) Fair Trade: WhoBenefits?, London:IIED. Jones, S. and B. Bayley (2000) Fair Trade: Overview, Impact, Challenges: Study to InformDFID's Supportto Fair Trade,Oxford:Oxford Policy Management. Kumar, Krishna (ed.) (1993) RapidAppraisalMethods,Washington,DC: World Bank. Lake, R. and C. Howe (1999) TheDevelopmentImpactof Fair Trade:Evidencefrom the Workof Traidcraft and Challengesfor the Future, London:TraidcraftExchange and ChristianAid. Neubert, Susanne (2000) Social Impact Analysis of Poverty Alleviation Programmes and Projects, London:FrankCass. Olivier de Sardan, Jean-Pierre (1995) Anthropologieet Developpement.Essai en socio-anthropologie du developpement[Anthropology and Development:Essay on the Socio-anthropologyof Development], Paris:L'Harmattan. Oxfam (2002) Rigged Rules and Double Standards-Trade, Globalisation,and the Fight against Poverty, Oxford:Oxfam International. University of Liege (ULg) (2002-2003) 'Un commerce equitableet durable,entre marcheet solidarite: diagnostic et perspectives' ['Sustainable Fair Trade between Business and Solidarity: Diagnosis and Prognosis'], researchconducted for the Belgian FederalScience Policy Office. World Bank (2002a) 'Impact Evaluation' website, available at http://www.worldbank.org/poverty/ impact/index.htm World Bank (2002b) 'A User's Guide to Poverty and Social Impact Analysis', available at 1 http://lnweb 8.worldbank.org/ESSD/sdvext.nsf/8 IByDocName/PovertySociallmpactAnalysis

The author
ElisabethPaul is a ResearchFellow at the Fonds national de la recherchescientifique(FNRS) [NationalFoundationfor Scientific Research]at the Universityof Liege. Her doctoralresearch is on the managementof developmentassistance.Contactdetails: FNRS, PoLE-SuD,Universite de Liege, Blvd du Rectorat7, B31/8, 4000 Liege, Belgium. < e.paul@ulg.ac.be> Editor's note: Thanksare due to Caroline Beaumontfor translating this from the French original.

150

Developmentin Practice, Volume15, Number2, April 2005