Gender - Mainstreaming Bustelo PDF

Evaluation
http://evi.sagepub.com
Evaluation of Gender Mainstreaming: Ideas from a Meta-Evaluation Study

María Bustelo
Evaluation 2003; 9; 383
DOI: 10.1177/1356389003094002
The online version of this article can be found at:

http://evi.sagepub.com/cgi/content/abstract/9/4/383
Published by:
http://www.sagepublications.com
On behalf of:
The Tavistock Institute
Additional services and information for Evaluation can be found at:
Email Alerts: http://evi.sagepub.com/cgi/alerts
Subscriptions: http://evi.sagepub.com/subscriptions
Reprints: http://www.sagepub.com/journalsReprints.nav
Permissions: http://www.sagepub.com/journalsPermissions.nav
Citations (this article cites 3 articles hosted on the

SAGE Journals Online and HighWire Press platforms):
http://evi.sagepub.com/cgi/content/refs/9/4/383
Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

© 2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
02 Bustelo (ds) 19/11/03 11:34 am Page 383
Evaluation
Copyright © 2003
SAGE Publications (London,
Thousand Oaks and New Delhi)
[1356–3890 (200310)9:4; 383–403; 040429]
Vol 9(4): 383–403
Evaluation of Gender Mainstreaming

Ideas from a Meta-evaluation Study1
MARÍA BUSTELO
Complutense University at Madrid, Spain
The new policy strategy of ‘gender mainstreaming’ poses particular

challenges for the evaluation of public gender-equality policies. To elaborate
on this issue, a first step is to analyse the evaluation of those public gender-
equality policies that have formally adopted the gender mainstreaming
strategy. In this article, results are summarized from a meta-evaluation of 11
evaluation processes of gender-equality plans implemented between 1995
and 1999, both at regional and national levels in Spain. This meta-evaluation
focused on analysing the evaluation processes rather than the outcomes of
those evaluations. First, the different types of evaluation carried out in the
11 studies are discussed. Second, some contextual factors are identified as
influencing elements in the evaluations. Third, some conclusions and lessons
learnt are presented, using the framework provided by the meta-evaluation
criteria previously established. Finally, taking into account those lessons, a
discussion of the evaluation of gender mainstreaming is presented,
elaborating on the ways in which gender mainstreaming strategies and
gender perspective can be evaluated and which should be used to conduct
useful evaluations.
K E Y WO R D S : gender-equality policies; gender mainstreaming; meta-evaluation;

programme and public-policy evaluation; public administration
Introduction
Public policies aiming to promote gender equality have evolved substantially in
the last decades. Ever since 1975, when the United Nations established Women’s
International Year, and most of the western nations started to acknowledge
gender inequality – then known as ‘women’s discrimination’ – as a public issue
that deserved public intervention, the strategies and political instruments of
those policies have been changing. Focus on sex discrimination (discrimination
based on biological differences) and especially women’s discrimination has
evolved to focus on gender (based on the cultural and social consequences of
those biological differences). In addition to specific actions for women – positive
actions – ‘gender mainstreaming’ emerged as a new and necessary strategy for
combating gender inequality in the long term. Gender mainstreaming means
383

Evaluation 9(4)
that, in addition to specific policies addressing gender discrimination – which are
still necessary to deal with actual gender discrimination – there is a need to look
for a gender perspective in all public policies.2 This strategy implies a much more
complex and widespread political action, and a revolutionary change in the
process of policy making. Thus, evaluation of gender-equality policies must take
into account this new strategy and the challenges it poses about the way in which
those policies should be evaluated. How should gender-equality public policies
be evaluated if gender mainstreaming is the assumed strategy? How can gender
mainstreaming itself be evaluated?
The first step for elaborating on that issue is to analyse the way public gender-
equality policies are currently evaluated. If these policies formally assume
‘gender mainstreaming’ as a strategy in their formulation, and, supposedly, in
their implementation, how do they deal with it in the evaluation phase? A good
way to analyse and assess evaluations is by means of meta-evaluations. Meta-
evaluation refers to the evaluation of evaluations and, in addition to assessing
evaluation quality, it may contribute to the description, analysis and assessment
of evaluation studies or processes.
Alternatively, the evaluation phase has been the least studied in the area of
policy analysis and public policy studies. However, governments and policy
makers must evaluate in order to obtain explicit feedback to adjust their actions.
There are other information sources, such as polls, statistics, observatories,
investigations and so on, but evaluation is an especially useful tool for facilitating
decision making – for the improvement, accountability and enlightenment of
public policies – and for organizational learning. Although the practice of public
policy and programme evaluation has developed rapidly in recent years, this is
not the case for the study of their evaluation.3 Thus, there is a strong need for
studying and analysing public policies with specific regard to their evaluation.
This is not only a way to better understand those policies, but also to improve
evaluation processes.
This article proceeds from a meta-evaluation in which 11 evaluation experi-
ences of public gender-equality policies in Spain, at the national and regional
levels, were analysed, compared and evaluated. In that work, we did not conceive
meta-evaluation as a procedure for evaluation quality control, but as a means to
study and analyse public gender-equality policies, in the specific aspect of their
evaluation. Therefore, instead of evaluation quality, we focused on its suitability
to the evaluative and political context and its contribution to the performance of
the evaluation. In addition, we proposed to analyse whether or not those evalu-
ations took into account the new strategy of gender mainstreaming, and shed
some light on how gender-equality evaluation may be sensitive to that strategy.
The Case Study: The Evaluation of Public Gender-equality

Policies in Spain
The starting point in Spain of the so-called ‘state feminism’ or ‘Femocracies’ (e.g.
Meehan and Sevenhuijsen, 1991; Stetson and Mazur, 1995; Gardiner, 1997) can
be set at the end of 1983, when the Instituto de la Mujer (Women’s Institute) was
384

Bustelo: Evaluation of Gender Mainstreaming

created at the national level (Valiente, 1995, 1996, 1997). From that time on, what
has been called ‘state feminism’ extended to the regional governments (Comu-
nidades Autónomas), which created their own ‘women’s agencies’ between 1988
and 1994, (Bustelo, 1999a; Granados, 1999).4 ‘State feminism’ has been the
driving force behind gender-equality policies in Spain.
Similar to the Instituto de la Mujer at the national level, regional gender-
equality agencies present themselves not as the direct executors of gender-
equality policies, but as the promoter and co-ordinator of such policies. Their
aim is to persuade other government units to assume and develop objectives and
activities to promote gender equality in their respective areas. For this purpose,
the main policy instrument is the elaboration and approval of gender-equality
plans, which refer to gender mainstreaming as an important strategy. These plans
consist of a structured set of initiatives in the various areas affecting women and
gender inequality, and almost all the government departments are involved in
their implementation. These plans are approved by the national or autonomous
(regional) cabinets for a certain period of time (usually from three to five years).
The elaboration and implementation of these gender-equality plans have
entailed, in some cases, parallel evaluation processes. That is, evaluation of some
of the plans has begun, although in a heterogeneous and irregular way. Similarly,
there have been several generations of gender-equality plans in Spain (at the
national and the regional levels), and generations of evaluations of those plans
can be distinguished. The first generation consisted of four very limited evalu-
ation experiences that took place in the early 1990s. The second-generation
evaluations are the object of this investigation and consisted of 11 evaluation
experiences at the national and regional levels. These evaluation processes took
place from 1995 to 1999.
As mentioned, the first-generation evaluation attempts were very feeble and
limited to checking the implementation of the actions proposed in the plans. This
weakness was to be expected, not only because of the lack of previous experi-
ences and of an evaluation culture and competence, but also because the tech-
nical quality of those first plans was poorer than that of the subsequent ones. The
second-generation evaluation experiences were also limited. However, all these
experiences are of value by themselves, just because they exist and, in some way,
were milestones for the evaluation of gender-equality policies in Spain. Also, due
to these experiences, an evaluation culture has been created and the evaluation
of gender-equality plans has become widespread. In almost all the third-gener-
ation gender-equality plans, some kind of evaluation is included within their
design, which was not the case with the first two generations of plans.
Methodology
Eleven evaluation processes, among the regional and national gender-equality
plans, were selected for the meta-evaluation study (one evaluation of the national
plan and 10 of regional plans). All second-generation evaluations, which actually
took place between 1995 and 1999, were selected. We decided to select all the
existing evaluation processes of the overall plans, excluding evaluations of
385

Evaluation 9(4)
specific programmes within the plans. Although, in some cases, it was not clear
whether these experiences were real evaluations, they were included in the study,
providing that the women’s agency regarded them as such and referred to them
as evaluations of the respective plans.
As explained above, the evaluation processes themselves, and not their results,
were the focus of the study. From the meta-evaluation perspective, the purpose
of the study was to evaluate those evaluation processes. In order to define the
assessment criteria of the evaluation processes, an evaluation framework was
established, based especially on the works of:
• Patton (1980, 1997): importance of useful evaluations and utilization, stake-
holder participation, flexibility, context and communication;
• Weiss (1987, 1998): political nature of evaluation, importance of context and
actors;
• Scriven (1995): concept and centrality of systematic judgment in evaluation;
and
• Chelimsky (1998): implications of the political nature of the evaluation,
including the need for credibility and timeliness.
This evaluation framework was also reinforced by the author’s previous experi-
ence as an evaluation consultant.
Thus, the (meta) evaluation criteria were established around three main
aspects:
1. The overall design of the evaluation processes, including their responsive-
ness to context, clarity of purposes and objectives, existence of institutional
structures and resources for the evaluation, and the utilization of different
types of evaluations.
2. Key evaluation elements, such as the stakeholders involved in the evaluation
processes, the time when these processes took place, the evaluation criteria
established and the methodology and techniques used for gathering and
analysing information.
3. The evaluation utilization, taking into account the adequacy and usefulness
of the information produced, its communication and dissemination
processes, and its utilization and impact both on gender-equality policies
and on women’s agencies.
The techniques for gathering and analysing information employed in the meta-
evaluation included document collection and analysis, on-site visits to women’s
agencies, 17 in-depth interviews with the heads of women’s agencies, and tele-
phone and on-site interviews with the technicians in charge of the evaluation in
the women’s agencies.
Types of Evaluation
In order to present a summary of the meta-evaluation results, first, the types of
evaluations are described. Four basic classifications are used to describe the types
of evaluation performed: formative versus summative evaluation; design, process
386


or results evaluation; ex ante versus ex post facto evaluation; and internal versus
external evaluation (see Table 1 for comparisons).
Evaluation role All 11 experiences analysed are summative. A summative

evaluation occurs when the role is one of recapitulation of the plan or programme
evaluated. A formative evaluation is used as an instrument for changing the plan
during the implementation process. There are three regional evaluations that could
be considered partially formative. In the case of the evaluation of the first plan in
Aragón, to some extent, it was formative in nature because it was implemented
after a political change in the regional government, and its purpose was mainly to
establish priorities for the last year of the evaluated plan. In the other two cases
(La Rioja and País Vasco), annual reports were made, so that the evaluation could
be used for correcting certain aspects during plan implementation. However,
Table 1. Types of Evaluation in the Second-generation Evaluations of Gender-equality

Plans in Spain
Region Role Content Timing Agent
Instituto Mujer Summative Process and Ex post facto Mixed:
II (1993–95) results internal design,
(National) part of external
implementation
Andalucía Summative Design, process Ex post facto External
II (1995–97) and results
Aragón Formative/ Process and Ex post facto Internal
I (1994–96) Summative results
Canarias Summative Process and Ex post facto External
I (1995–96) results
Castilla-La Mancha Summative Design, process Ex post facto External
Castilla y León Summative Results Ex post facto External
I (1994–96)
Cataluña Summative Design, process Ex post facto Mixed:
II (1994–96) and results external technical
assistance in design,
internal
implementation
Galicia Summative Design, process Ex post facto External
Madrid Summative Process and Ex post facto External
II (1993–95) results
La Rioja Formative/ Process and Ex post facto Internal
II (1996–99) Summative results
País Vasco Formative/ Design, process Ex post facto Mixed:
II (1996–2000) Summative and results external technical
assistance in design,
internal
implementation
387

Evaluation 9(4)
collecting information at the beginning and continuously does not necessarily imply
a formative nature. In fact, the annual reports of the Basque Women’s Institute pre-
sented in Parliament are more summative in nature because they emphasize the
accountability function and the recapitulation of what had been accomplished each
year. In La Rioja, the collection of information from the beginning was more due
to the need to keep records during the process in order to evaluate at the end; so
it appears that the intention was more summative than formative.
Evaluation content Three types of evaluation are distinguished: design evalu-

ation, process evaluation and results evaluation. A design evaluation refers to the
assessment of the plan as a whole, including evaluation of the plan’s rationale,
objectives, internal coherence, strategies for implementation, etc. Only in two of
the evaluation experiences (Andalucía and Castilla-La Mancha) is a design evalu-
ation explicitly considered. From these two experiences, the Andalusian is the
more complete, considering more design aspects, including analysis of the
(in)equality concept for which the plan was designed. Another three experiences
(Cataluña, Galicia and País Vasco) contain some design evaluation elements.
These experiences analysed the types of activities or actions implemented, target
population and budget, and some interesting classifications were made. This kind
of analysis allows assessing the plan as a whole and provides interesting infor-
mation about the relative importance of each kind of action or activities. This is
also, to some extent, a way to evaluate the design of the plans.
Regarding process evaluation, all the experiences, excepting one, formally
included this, but this kind of evaluation was often wrongly conceived. In most
cases, the so-called ‘process or implementation’ evaluation consisted exclusively
of assessing the ‘accomplishment level’, a term referring to a general inspection
of the actual implementation of the planned actions. The author suggests that
these kinds of evaluations could be considered more as a product or output evalu-
ation than a process evaluation. Indeed, a process evaluation should assess the
implementation process, timetable adjustment, whether the target population is
being reached, difficulties that arise during that implementation process (infor-
mation, time or location problems) and co-ordination processes with other
departments and agencies required for that implementation. This last aspect is
especially important for the evaluation of gender-equality plans and the gender
mainstreaming strategy, because the plans involve many stakeholders and need
a lot of co-ordination for their implementation. However, the analysis of the
experiences found that these aspects were not all evaluated. The more complete
process evaluations were carried out in Andalucía, Castilla-La Mancha, La Rioja
and País Vasco, although they still have some limitations.
The results evaluation can be divided into ‘product’ or output evaluation and
‘effect’ or outcome evaluation. Some product evaluation was carried out in
almost all the experiences analysed by means of the aforementioned ‘accomplish-
ment level’: by the study of what had been generated by the plan, its products
or outputs. However, in some experiences, this was limited to simply verifying
which planned actions had actually been implemented. Rarely was the quality of
the products assessed.
388


In three of the experiences analysed (the second national plan and those of
Madrid and Castilla y León), an effects assessment was aimed for, although they
were not really achieved. The experiences were very good studies of women’s
social status but they did not assess plan effects. It is well known how difficult it is
to assess the so-called ‘net effects’ of a programme or plan (i.e. the effects attribu-
table to the plan and not to other factors: effects that would have never occurred
in the absence of the programme or plan). In the three above-mentioned cases,
several variables related somehow to the plan objectives (e.g. increases of women
at university) were measured at the beginning and at the end of the plans. These
measures were compared, indicating their evolution during that period of time. But
there is no variable or factor connecting that evolution to the existence of the plan;
that is, if there were any changes, they might have been caused by many other
factors and not only by the plan implementation. The only experience that had
planned an actual effect evaluation was the evaluation of the second Andalusian
plan (it had an evaluation plan which included four different surveys, combining
different variables, moments and populations, and a women’s social-status study
carried out in 1988). However, due to various factors, that evaluation was not
completed in the way it was planned at the beginning of the evaluation period.
Evaluation timing All the second-generation evaluation experiences were ex

post facto, launched when the plan had already begun. There were no ex ante
evaluations, including a previous evaluation design or pilot study of some actions
in order to subsequently extend them. Moreover, one of the main difficulties of
this generation of evaluations is not their ex post facto nature itself, but the fact
that the evaluation design and implementation usually started very late (when a
large part of the plan had already been implemented). This circumstance weakens
the evaluation tool, firstly, because it is not designed parallel to the plan, and sec-
ondly, because the opportunity of collecting valuable information from the begin-
ning of the implementation process is lost. The only two evaluation experiences
that were planned from the beginning were those in the País Vasco and La Rioja.
The others started shortly before the last year of their plan periods.
Evaluation agent Evaluations may be internal or external, depending on

whether or not they are conducted by someone who is related to the financing,
planning or execution of the evaluation plan. There were three mixed experi-
ences, although each somewhat different. In the case of the second national plan,
the evaluation design was internal, and an external team participated in the infor-
mation collection and analysis of only one part of the evaluation study. However,
in the Basque and the Catalonian cases, an external team was employed for the
evaluation design, although part of the evaluation implementation was internal.
Contextual Factors that Influence Gender-equality Policy

Evaluation
Some contextual factors were identified as elements that influence the evaluation
of gender-equality policies, either positively or negatively. We called them
389

Evaluation 9(4)
contextual because they do not directly involve the evaluation processes, but are
institutional factors concerning gender-equality policies or specific characteristics
of those policies. By means of the meta-evaluation, 10 contextual factors were
identified, five with positive influence on the evaluation, and another five with
negative influence.
Factors that Influence Gender-equality Policy Evaluation Positively

The existence of structured plans The fact that plans are the main policy instru-
ment facilitates evaluation. All public actions are included in a structured plan,
including objectives and actions, a time dimension and resources assigned (at
least, formally). Moreover, the existence of a plan increases government aware-
ness of what is being done and what needs to be done. Evaluation is a good way
to keep governments and policy makers informed.
The increase of formal commitments concerning gender-equality plans and their

evaluation Gender-equality plans in Spain are approved by the respective
Cabinet (either at the national or regional level). This may include some sort of
commitment to plan evaluation.
International support for gender-equality policies The 4th World Conference

on Women in Beijing in 1995 and the 4th Action Programme on Equal
Opportunities between Men and Women (1996–2000) of the European Union
were especially important. Moreover, these international forces have led to some
formal agreement on the need for gender-equality policies and the strategies for
implementing them – specific policies and gender mainstreaming. Theoretically,
gender-equality policies should be revised, monitored and evaluated.
The existence of consolidated gender-equality organisms Currently in Spain,

although heterogeneously, there is a relatively powerful ‘women’s machinery’ or
‘state feminism’ at the national and regional levels.
Acknowledgement of programme and policy evaluation in recent years

Although this acknowledgement is mainly symbolic – it is declared that public
programmes and policies must be evaluated, although usually they are not –
evaluation is currently fashionable. This is a big step and could be a facilitating
factor for real evaluation.
Factors which Influence Gender-equality Policy Evaluation Negatively

The generality of gender-equality plans Although, across generations of plans,
they have become more specific, they are still quite general because of their
nature – multi-dimensional and with many stakeholders involved. Evaluating
general and ambiguous plans is much more difficult than evaluating specific ones.
In the author’s opinion, this generality is partly an indicator of a lack of clarity
about what should be done and how; i.e. the ‘programme theory’ and the
‘implementation theory’ (Weiss, 1998) of the gender-equality plans are not
390


explicitly clear.5 We still need not only to keep on investigating gender relations
but also to ‘translate’ theory and evidence into practice, implementing sound
policies and plans.
The integral and horizontal nature of gender-equality plans Gender-equality

plans are integral in nature: they are intended to solve a problem (gender inequal-
ity) from all the possible points of view, containing actions involving education,
health, employment, culture, social and political participation, and so on.
Therefore, they are complex and multi-dimensional in nature, making their
evaluation more difficult. In addition, discrimination at the base of gender
relationships is, itself, very complex and profoundly rooted in social structures.
Combating that discrimination is also complex, because it implies a different
outlook and a different way of analysing and dealing with reality. Moreover, plans
are conceived as ‘horizontal’, proposing actions through ‘regular’ public policies
and with gender mainstreaming as the general strategy to follow. For that
purpose, the dynamic and co-ordinated engagement of multiple actors and stake-
holders is needed.
The policy of persuasion Because of gender mainstreaming and the horizontal

nature of plans, ‘femocrats’ must persuade their colleagues in the government to
implement and become responsible for gender-equality actions, which are a part
of those plans. It is very interesting to analyse the strategies that femocrats
employ for this purpose but, in general, we think that the personal factor is a key.
This is a weakness, and quite difficult to evaluate. Femocrats have no power to
sanction their colleagues if they decide to pay no real attention to gender issues,
so the strategies they use for persuading them tend to be informal and personal,
for example, ‘appealing to statistics’, ‘being patient, repeating the same things
over and over’, ‘appealing to justice’, ‘keeping a low profile or no direct con-
frontation’, ‘allowing colleagues to be political protagonists of gender actions’,
‘appealing to colleagues’ personal experience’, ‘using a sense of humour’ and so
on (Bustelo, 2001: 412–15).
Lack of experience, culture of evaluation and consolidation of the evaluation

function within Spanish public administrations6 For these second-generation
experiences, there were practically no previous references to gender-equality
plan evaluation. Thus, evaluation was something new and, in a sense, nobody
knew exactly how to do it. Also, because evaluation is not yet a part of the regular
and daily tasks in public administrations, it was rarely conceived at the appropri-
ate time, leaving it until almost the end of the plan.
Lack of competent and prepared teams for the evaluation of gender-equality

policies Related to the former factor is the fact that, at that time in Spain, there
was little evaluation specialization at the academic or the professional levels.
Moreover, it is quite difficult to find evaluators who, in addition, are knowledge-
able and sensitive about gender issues.
391

Evaluation 9(4)
Results and Lessons Learnt
Some broad results are now summarized, following the meta-evaluation
criteria established and provided earlier in this article. Firstly, for each of the
three main issues (overall evaluation design, key evaluation elements and
evaluation utilization), some general conclusions are drawn from the evidence
found about the state of evaluation practice in this area. Secondly, and follow-
ing the format of the conclusions, some lessons learnt are commented upon.
For each lesson, some evidence is also presented. These lessons learnt from
the analysed evaluation experiences may also contribute to: the general
discussion about which aspects of gender mainstreaming can be evaluated and
how; and guide the improvement and optimization of the evaluation function
of gender policies.
Some Results and Conclusions about the State of the Evaluation

Practice in the Field
For the overall design of the evaluation processes, we analysed evaluation
responsiveness to context, clarity of evaluation purposes and objectives, exist-
ence of institutional structures and resources for the evaluation, and utilization
of different types of evaluations (discussed above).
Regarding the first two issues (responsiveness to context and clarity of evalu-
ation purposes), we found evidence that may be summarized in the following
conclusions.
1. There was a lack of clarity in the evaluation purposes and this led to con-
fusion of gender-equality policies and plan evaluation, and the investi-
gation of the status of women.
2. There was also a lack of a global outlook of public action taken to promote
gender equality. That is, we found a generalized trend to evaluate specific
actions and plans instead of policies.
3. We also noted the lack of acknowledgement of the political nature of
evaluation, considering plan evaluation to be essentially technical.
Regarding the institutional structures and the resources invested in evaluation,
we found the following.
4. It was very important to have adequate institutional structures for the
evaluation processes. Given the ‘horizontal’ nature of the evaluated plans,
these structures have positive influence on the complex but necessary
gathering of information.
5. Generally speaking, the resources invested in these second-generation
evaluations were very poor.
The key evaluation elements that we took into account were the stakeholders
involved, timing, the evaluation criteria established and the methodology and
techniques used. The following was found regarding these key elements.
6. Evaluation processes were exclusively run by technicians, either internal
or external (evaluation was generally conceived as an essentially
392


technical endeavour). This led to the result that the involvement of other
stakeholders seldom occurred.
7. Most of the experiences analysed were planned almost at the end of the
implementation period of the plan. This led to serious difficulties in infor-
mation collection.
8. There was no rigorous and previous establishment of evaluation criteria.
As a result, the evaluation experiences were organized around specific
information-gathering techniques instead of around evaluation criteria.
Finally, regarding evaluation utilization, we took into account the adequacy
and usefulness of the produced information, communication and dissemination
processes, and the evaluation impact both on gender-equality policies and on
women’s agencies.
9. We found an important deficit in almost all the experiences in what we call
‘practical elaboration’: the establishment of judgements and recommen-
dations.
10. Generally speaking, communication and dissemination of the evaluation
processes and results were also very poor.
11. Except for the case of the Basque Women’s Institute, evaluation was
perceived as a secondary function in the national and regional women’s
agencies.
Lessons Learnt
1. There is a need to distinguish between gender-equality policy evaluation and
investigation of gender inequality or the status of women.
Analysing the evaluation experiences, the conclusion was reached that it is neces-
sary to distinguish between research on the status of women and evaluation of
gender-equality plans and policies. We have already observed that these two
concepts were confused in some experiences, such as those carried out in Castilla
y León, Madrid and at the national level. Essentially, these investigations
produced indexes for measuring gender inequality. But, clearly, they are not
evaluations of gender-equality plans because there was no direct or formal
linkage between the variables studied and the plan being evaluated.
There are many reasons for distinguishing research from evaluation (Bustelo,
1999b), but here two are highlighted. The first is related to the object. In this
case, the object of research is gender inequality (how the status of women differs
from that of men) and its evolution over time. The object of evaluation, however,
is the way in which the authorities deal with the gender inequality problem and
how public actions influence the public issue of gender inequality: the plans and
policies themselves, not the problem they are meant to address. Secondly, there
are differences in the purposes, with improvement and accountability being the
main purposes of evaluation, and knowledge construction the purpose of
research.
Obviously, research on gender inequality and the status of women contributes
to better policy formulation. But this does not preclude the need for evaluating
393

Evaluation 9(4)
those policies. Both activities – investigation and evaluation – are crucial for
women’s agencies. Moreover, the so-called ‘equality observatories’ (whose aim
is to collect information periodically about gender inequality) are very useful for
evaluating gender-equality policies, in addition to their utility for policy formu-
lation. But the very fact of their existence does not imply evaluation.
2. Evaluations should enhance a global view of the public action taken to

promote gender equality. In addition to programme evaluation, a broader policy
evaluation perspective is required.
None of the experiences analysed is an evaluation of the global gender-equality

policy, but rather, they evaluate specific plans approved by their respective
governments. The aim of the plans is to promote, co-ordinate and gather all
the government actions regarding gender equality so that, in a sense, the best
way to evaluate gender-equality policies is to evaluate gender-equality plans.
Plan evaluation facilitates observation of the development and results of that
plan but does not question the role of the plan or the plans themselves. Issues
concerning the adequacy of the plan as the main policy tool, the actual effec-
tiveness of the gender mainstreaming strategy and its pertinence to the role
and functions of women’s agencies were not addressed in the evaluations
studied.
3. There is a need to acknowledge the political nature of evaluation.
In most cases, the evaluation function was considered an exclusively ‘technical’

matter, leaving key elements in the evaluation processes unattended. This
circumstance has, in part, led to useless or residual evaluation experiences.
This acknowledgement of the political nature of evaluation has been increas-
ing in the last few years. Weiss was one of the first authors who pointed out the
relationship between politics and evaluation:
1. policies and programmes are products of political decisions;
2. evaluation is intended to feed the decision-making process; and
3. by its own nature, evaluation implicitly establishes programme problems,
needs, etc., and this implies a political stance (Weiss, 1987).
But lately, this political nature has become a key issue not only for evaluation
practice but also for evaluation theory. Chelimsky (1998) considered it necessary
to have a view of politics as central to evaluation. Chelimsky (1998: 39) states
‘still, even today, we see politics as merely the “context” of an evaluation, as
something that “intrudes” upon good practice, rather than the engine driving it’.
This acknowledgement entails the need to extend the way we think about politics
and policy making, the need for credibility, timeliness and flexibility of the evalu-
ation processes.
394


4. The importance of adequate institutional and co-ordination structures for
evaluation.
One of the main problems, especially mentioned by the femocrats in charge of

the evaluation processes, was the difficulty of gathering information from other
government units that participate actively in the implementation of the evalu-
ated plan. As a result of the horizontal nature of gender-equality plans, much of
the information required was not held in the women’s agency concerned (which
was the responsible unit for the evaluation) but in various other government
units. This task of collecting information was not an easy one.
After the analysis, we concluded that it is extremely important to design and
have adequate institutional and co-ordination structures for the evaluation,
which allow the collection of valid and reliable information. We think this is even
more important than the specific techniques used for data collection and analysis.
Having a valid and reliable instrument for gathering information is a necessary
but not a sufficient condition to obtain the relevant and reliable information for
evaluating the plan.
In addition to having appropriate information systems, there is a need for
adequate co-ordination structures. This is especially important, for at least the
following two reasons.
• Due to the horizontal nature of gender-equality plans, their actions are imple-
mented by different agencies, so that the information is going to be ‘spread
out’; thus, it is necessary to design systems that allow the centralizing of that
information in a continuous, stable and homogeneous manner.
• For an evaluation, not only is the information that is gathered ad hoc very
important, but also the information produced by and for plan implemen-
tation and management. Availability of valid information of that kind
implies less effort in the evaluation and a better opportunity for under-
standing the evaluated programme.
We analysed the different co-ordination structures of the 11 evaluation experi-
ences and we concluded that the involvement of the following three types of
structures might drive evaluation.
1. Structures linked to the plans and to women’s agencies (it is beneficial for
these general structures to have responsibilities, at least formally, in the
evaluation).
2. Structures linked to the legislative power (the legislative control over the
executive power is a very good stimulus for evaluation).
3. Formal structures for women’s participation (we found almost no partici-
pation of this type in the processes studied).
Thus, the existence of an explicit evaluation commitment within the plan or the
commitment to report to the parliament, the existence of ‘ad hoc’ government
committees for monitoring and evaluating the plan, the existence of a partici-
pation mechanism in the monitoring and evaluating processes, were all key and
facilitating elements for evaluation.
395

Evaluation 9(4)
5. There is a need for greater resource investment in evaluation.
It was almost impossible to measure the real investment of Spanish women’s

agencies in the evaluation of second-generation gender-equality plans. It was
easy to calculate the direct evaluation expenses in the external experiences, but
it was rather difficult to objectively estimate the indirect expenses, both in
external and internal evaluations (cost of time spent by internal professionals on
the evaluation processes).
Nevertheless, we estimated investment in evaluation by means of the direct
expenses employed. For this purpose, we compared the direct evaluation
expenses of the external experiences using two different figures: plan expenses
and women’s agency expenses during the plan period. However, in some cases,
we could not even do this, because there was no specific estimation of the general
plan expenses. Logically, when those estimates were available, the general plan
expenses were much higher than the women’s agencies expenses, because other
government units, and not only the respective women’s agency, contributed to
the plan. Although with limitations, after the analysis, we were able to conclude
that investment in evaluation was extremely low. For example, the most expen-
sive external experience (which was three times more expensive than the next
one, in absolute terms), the evaluation of the second Andalusian plan, represents
0.97 percent of the agency’s expenses (Instituto Andaluz de la Mujer) and only
0.07 percent of the general plan expenses.7 Obviously, there is a need for
increased resources.
6. There is a need for promoting greater stakeholder participation in the evalu-

ation processes.
Active stakeholder participation is considered to be a key element for obtaining

context-responsive and useful evaluations (Patton, 1997; Monnier, 1995; Weiss,
1998). The use of stakeholders’ opinions as a source of information does not
imply that those stakeholders participate in the evaluation process. That is why
we mention ‘active’ participation, that is, in one or more parts of the evaluation
process: study design, definition of and decisions about evaluation criteria, tech-
nique selection, information gathering, analysis and judgement, establishment of
recommendations, report writing, and communication of results. Curiously
enough, because gender-equality plans are conceived as ‘horizontal’, proposing
actions by means of ‘regular’ public policies, dynamic and co-ordinated engage-
ment of multiple actors and stakeholders is needed, and this should have been
taken into account in the evaluation.
However, although participation of the women’s movement and other actors
in public gender-equality policies is considered an important strong point for
those policies, generally speaking, stakeholder participation in the evaluations
analysed was almost nonexistent. This was partially due to the technical charac-
ter of the evaluation processes, with evaluation considered exclusively an affair
of professionals or specialists. In some cases, there was symbolic participation by
the women’s movement, because they were invited to the presentation of the
396


evaluation results. Formal representative structures (e.g. Consejos de la Mujer)
had practically no opportunity to participate in the evaluation processes.
However, a lack of participation in the evaluation phase was to be expected;
Spanish public gender-equality policies have always been half-hearted about the
participation of the Spanish women’s movement in the other phases of policy-
making processes (formulation and implementation).
7. The importance of timeliness.
There are two time-related issues, which are key elements for evaluation
processes, and gender-equality policies are no exception. The first issue is the
moment in which the evaluation is planned and started. Evaluation is a rather
more powerful tool if it is planned from the start and in a continuous way. As
already pointed out, most of the evaluations analysed were planned almost at
the end of the implementation period of the plan, and this led to serious diffi-
culties in data collection.
Secondly, there is a need for timeliness in the evaluation processes. Evaluation
criteria arise in a specific context, situation and moment. If much time is required
to answer evaluation questions, they might become irrelevant or the beneficiaries
might not be the same. Evaluation responsiveness is clearly linked to the evalu-
ation timetable. The time factor has deeply determined the experiences analysed,
and delays in some evaluation processes have decreased the chances of it being
useful.
8. The importance of evaluation criteria as evaluation organizers.
One of the first and crucial steps in the evaluation process is the determination
of what is going to be evaluated. As already mentioned, the evaluation criteria
determine and guide the evaluation study, being the main orientation for the
gathering and analysis of the necessary information. That is, evaluation criteria
determine which variables to study and, also, the logic base for systematically
judging the information previously gathered and analysed.
In the experiences analysed, there seems to have been no previous and
profound reflection about what to evaluate. Mostly, the evaluation criteria were
established and defined by means of the specific techniques used, giving priority
to this term (what we have called the ‘dictatorship’ of methodology and tech-
niques). A good example is the evaluation of the second national plan, a mixture
of three different studies organized by the techniques applied.8
9. The importance of judgement and recommendations as essential parts of

practical evaluation.
In addition to information collection and interpretation, evaluation involves two

more levels of analysis: systematic judging (guided by the evaluation criteria
previously set) and establishing recommendations (Patton, 1980). Judging and
establishing recommendations is what we call ‘practical elaboration’ because, in
397

Evaluation 9(4)
order to be useful and used, information must be elaborated or ‘translated’ into
specific and relevant guides of action. It is clear that evaluation should be useful,
practical and action-oriented for the improvement, accountability and enlighten-
ment functions. Through the meta-evaluation study, the author concluded that
these last analysis phases – judging and establishing recommendations – were
almost nonexistent in the experiences analysed.
In some cases, there was even a lack of a global analysis. If there were a
‘process’ evaluation and a ‘results’ evaluation, there was no overall or final
analysis for ‘putting everything together’. In a few cases, there were some final
conclusions – although when they existed, they were very short and not global –
but none of the 11 reports contained any recommendations. In sum, it seemed
as though the evaluation processes were not really finished, missing opportunities
to be really useful.
10. A need to consolidate communication and dissemination throughout the

evaluation process.
Communication processes are extremely important to aid the use of evaluation.

A first step is to have a good ‘product’ that reflects the evaluation process well:
complete and appropriate evaluation reports. Following that, there is a need for
dissemination of those reports, i.e. of the evaluation results.
The communication processes that were part of the experiences studied were
quite deficient, and evaluation results could have been disseminated much more
effectively. A related issue is that of the accessibility of the evaluation reports.
When gathering information for the meta-evaluation, in some regions, we had
considerable problems obtaining the evaluation reports and were unable to get
the whole evaluation report. In general, the accessibility and the public nature
of evaluation reports are still not clear and there are no accessibility criteria. This
is an issue requiring earnest discussion.
11. The important role which women’s agencies should play in policy evaluation.
As mentioned above, in Spain, women’s agencies have been the main leaders
and promoters of gender-equality policies. Although the key component of these
policies is gender mainstreaming (i.e. they are intended to involve other govern-
ment actors), the agencies are still the main actors formulating, promoting, co-
ordinating and monitoring the policies. Taking this into account, policy
evaluation should be a crucial issue among agency functions. The evaluation
function is precisely the one which might help to define the specific role of
women’s agencies in gender-equality policies that are conceived with a main-
streaming strategy. Moreover, the fact that the final aim of these policies is to
achieve a ‘gender perspective’ in other public policies could place at risk the
existence of specific women’s agencies. Therefore, playing a leading evaluation
role could aid their definition. In addition, the evaluation function is an especially
important one for these policies because of their innovative nature.
However, and with the exception of the Basque woman’s institute, the
398


evaluation function is still a residual one. In fact, the Spanish agency at the
national level (Instituto de la Mujer) has missed a good opportunity to play a
leading evaluation role among the regional agencies. That could have been a
sound chance to recover a leading position, which was lost with the second- and
third-generation plans. In the third generation of regional plans, influence is
much more international (the 4th World Conference on Women in Beijing in
1995 and the 4th Action Programme on Equal Opportunities between Men and
Women [1996–2000] of the European Union) than national, which was obvious
in the first regional plans (Bustelo, 1999a: 377).
Conclusions: Ideas about how Gender Mainstreaming Can

Be Evaluated
Following the discussion of the lessons learnt by the meta-evaluation, the criteria
that should be used for evaluating gender-equality policies will now be described
in more detail. Generally speaking, and in the long term, one should try to assess
whether there is an evolution of women’s social status and gender relationships.
However, as already seen, it is very difficult to assess the ‘net effects’ of such a
broad and general policy, i.e. the quantity of that change which is due to the
policy and not to other factors. In the assessment of public actions, it is neces-
sary not only to evaluate the effects of the action, but also the way in which that
public action is conceived and implemented to deal with a particular problem.
Specifically, ‘intermediate’ aspects (design, implementation, quality of outputs)
should be evaluated before attempting to assess the final effects. Currently, the
evaluation of those ‘intermediate’ aspects is sometimes more useful than that of
the final effects for policy, especially for improving purposes.
Moreover, when the aim is so abstract (such as to modify gender relationships
and the status of women and, hence, of men), it is necessary to establish inter-
mediate aims and objectives that normally can be defined by means of the
strategies designed to deal with the problem. And here, one should take into
account the strategy of gender mainstreaming. If the main strategy of gender-
equality policies were gender mainstreaming, one would probably have to seek
gender perspective as the searched effect in other public policies (that is, whether
public policies – not the gender-equality policy – are formulated, executed and
evaluated with gender perspective), in addition to evaluating the gender policy
itself. In this sense, there have been some interesting experiences, such as the
Gender Impact Assessment tool, developed in the Netherlands, Belgium and
New Zealand (Verloo and Roggeband, 1996; Meier, 1998; Verloo, 1998).
Although this instrument was conceived more as a tool for an ex ante rather than
a ex post facto evaluation of the effects of a specific gender-equality policy, in
our opinion, it might be a useful source to evaluate gender mainstreaming efforts.
However, this is still not sufficient. It is crucial to recognize the importance of
evaluating the strategies employed to fulfil the proposed aims, and not only of
measuring the achievement of those aims. In this sense, the chief aim to achieve
(and, thus, the effects to pursue) is gender perspective in all public policies, with
gender mainstreaming being the strategy to deliver this. Therefore, in addition
399

Evaluation 9(4)
to their results, it is necessary to evaluate the design of those gender main-
streaming strategies and the process of their implementation.
However, by means of the meta-evaluation, the author concludes that one of
the factors identified as crucial by the femocrats was what can be called the ‘co-
ordination factor’. This factor was considered essential for:
1. persuading others to plan and implement gender actions and to assume a
gender perspective; and
2. obtaining valid and reliable information about those actions and the plan
management or evaluation.
None of these aspects was addressed by any of the evaluations studied. In the
author’s opinion, this co-ordination factor is directly related to the mainstream-
ing strategy. How do femocrats persuade other government units to assume a
truly active role in gender-equality plans? Which strategies and instruments work
better, and in what circumstances? Which are the best co-ordination structures
for the plan and its evaluation and at what level – political or technical?9 Why
are some government units more sensitive about gender issues than others? What
is needed to awaken that sensitivity, or which factors influence it? To what extent
is the ‘personal factor’ counted on? If this dimension is not evaluated, we will
probably fail to understand much about that ‘horizontal’ action and, hence, about
the gender mainstreaming strategy. We must be aware of the fact that evaluation
of gender mainstreaming strategies is complex, because the first step is to define
and typify the gender mainstreaming strategy in different contexts, moments and
policies. In relation to this, the Group of Specialists on Mainstreaming in the
Council of Europe elaborated an excellent document defining the term, promot-
ing ideas and demonstrating good practices to deal with it. If they are imple-
mented, evaluating those practices might be especially important due to their
innovative nature. It is not sufficient to implement them; they must be evaluated
in order to determine their effectiveness.
Moreover, any form of gender mainstreaming poses particular challenges for
evaluation practice; and as we have seen, current evaluation practice does not
address those challenges completely. In order to facilitate progress in this area,
there is a strong need for new tools and methodological approaches, which also
take seriously into account the lessons learnt from the analysis of the practice of
evaluation.
A new evaluation perspective implies that both the gender mainstreaming
strategy itself should be evaluated, in addition to the evaluation of its effects.
Alternatively, rather than exclusively emphasizing the evaluation of the
implementation of proposed actions, evaluation of gender-equality policies should
stress the assessment of their global design as policy instruments, their implemen-
tation processes, government co-ordination and organizational performance.
We have presented above a case for meta-evaluation in gender-equality policy
evaluation. However, meta-evaluation has potential value in other policy areas.
Important conclusions and guidance to improve the evaluation function in a
specific field can be drawn from the systematic analysis and assessment of evalu-
ation practices, their strengths and weaknesses and their final utility.
400


Notes
1. Earlier versions of this article were presented at the 4th Conference of the European
Evaluation Society in Lausanne (October 2000) and at the European Consortium for
Political Research (ECPR) 2001 General Conference, University of Kent in the UK
(September 2001). This final version of the article was made possible in part by a ‘Del
Amo-Complutense’ Fellowship in the University of California at San Diego during the
summer 2002.
2. ‘Gender mainstreaming’ is defined as ‘the (re)organization, improvement, develop-
ment and evaluation of policy processes, so that a gender-equality perspective is incor-
porated in all policies at all levels and at all stages, by the actors normally involved in
policymaking’ (Council of Europe, 1998). This concept is based on the idea that no
public policy is gender neutral, so that the different effects any public action may have
on women and men must be taken into account.
3. The author is not able to find any published reference of a similar study. The term
meta-evaluation has been sometimes used to name synthesis of evaluations, that is,
studies that focus on the evaluation results, instead of on the evaluation processes.
Some other references have appeared after this meta-evaluation was conducted.
4. The Spanish Constitution is from 1978 and the 17 regional governments and parlia-
ments were created between that year and 1983. Thus, the creation of these women’s
agencies was, in a sense, a parallel process to the creation of the political and admin-
istrative regional organization.
5. ‘Programme theory’ refers to the mechanisms that mediate between the distribution
and delivery of a programme and its results (the set of beliefs underlying action or the
set of hypotheses on which the programme is built), and ‘implementation theory’ deals
with what is required for translating the objectives into programme operations.
Whereas the focus of ‘programme theory’ is on people’s responses to programme activi-
ties (the operational mechanism is not the programme activities per se but the
responses those activities generate), ‘implementation theory’ emphasizes the services
or activities that are implemented and the way they are offered (Weiss, 1998: 57).
6. There is still a low evaluation culture in Spain. Possible causes for this are, at least, the
following (Ballart, 1993: 221–2): evaluation emerged in Spain in a time of economic
crisis and resource scarcity; the parliamentary system – at the national and the regional
level – is based on strong party discipline which tends to limit its controlling function;
the National Audit Office, and the similar offices at the regional level, limit their role
to controlling exclusively public finances; the number of professionals specialized in
evaluation is very scarce; and, finally, information systems within the public adminis-
tration, although they have improved a lot in recent years, are still inadequate for evalu-
ation purposes.
7. In Andalucía the cost of the external evaluation study was 55,980,000 pesetas, the
general plan expenses were 80,401,000,000 pesetas, and the expenses of the Instituto
Andaluz de la Mujer (IAM) during the three-year plan period (1995, 1996 and 1997)
were 5,792,819,000 pesetas. In addition to that, we estimate that the indirect expenses
of this evaluation experience were not very high because the external team was quite
autonomous.
8. The evaluation of the II National Plan consisted of three separate studies: ‘quantita-
tive evaluation’, ‘qualitative evaluation’ and ‘implementation evaluation’.
9. There is no agreement among femocrats about whether the co-ordination structures
should be political or technical or a mixture of both. Being political – for example, as
in Andalucía, the committee for monitoring the plan must be made up of viceconsejeros
401

Evaluation 9(4)
(the equivalent to a vice minister) – is the only way to have real ‘horizontal’ authority,
but being technical is the only way to be operative. Some of the femocrats think that it
is very difficult to be effective if there are no motivated technicians who are sensitive to
gender issues.
References
Ballart, X. (1993) ‘Evaluación de políticas. Marco conceptual y organización institucional’,
Revista de Estudios Políticos 80(3): 199–224.
Bustelo, M. (1999a) ‘Políticas Públicas de Igualdad de Género en España: evolución y
evaluación’ in M. Ortega, C. Sánchez and C. Valiente (eds) Género y ciudadanía. Revi-
siones desde el ámbito privado, pp. 367–89. Madrid: Ediciones de la Universidad
Autónoma de Madrid.
Bustelo, M. (1999b) ‘Diferencias entre evaluación e investigación: una distinción nece-
saria para la identidad de la evaluación de programas’, Revista Española de Desarrollo
y Cooperación 4(Spring-Summer): 15–29.
Bustelo, M. (2001) La evaluación de las políticas públicas de igualdad de género de los
gobiernos central y autonómicos en España: 1995–1999, Doctoral dissertation. Madrid:
Universidad Complutense de Madrid.
Chelimsky, E. (1998) ‘The Role of Experience in Formulating Theories of Evaluation
Practice’, American Journal of Evaluation 19(1): 35–55.
Council of Europe (1998) Gender Mainstreaming. Conceptual Framework, Methodology
and Presentation of Good Practice. Final Report of Activities of the Group of Specialists
on Mainstreaming (EG-S-MS). Strasbourg: Council of Europe Publishing.
Gardiner, F., ed. (1997) Sex Equality Policy in Western Europe. London: Routledge.
Granados, E. (1999) ‘El Instituto Andaluz de la Mujer: La creación de un órgano
autonómico para la igualdad entre los géneros’, in M. Ortega, C. Sánchez and C.
Valiente (eds) Género y ciudadanía. Revisiones desde el ámbito privado, pp. 391–405.
Madrid: Ediciones de la Universidad Autónoma de Madrid.
Meehan, E. and S. Sevenhuijsen, eds (1991) Equality Politics and Gender. Newbury Park,
CA: Sage.
Meier, P. (1998) ‘Policy Innovation in Terms of Mainstreaming? The Development of
Gender Impact Assessment in Flanders, Belgium’, paper presented in the ECPR 26th
Joint Sessions of Workshops, Warwick, UK.
Monnier, E. (1995) Evaluación de la acción de los poderes públicos. Madrid: Instituto de
Estudios Fiscales, Ministerio de Economía y Hacienda.
Patton, M. Q. (1980) Practical Evaluation. Newbury Park, CA: Sage.
Patton, M. Q. (1997) Utilization-Focused Evaluation, 3rd edn. Thousand Oaks, CA: Sage.
Scriven, M. (1995) ‘The Logic of Evaluation and Evaluation Practice’, in D. M. Fournier
(ed.) Reasoning in Evaluation: Inferential Links and Leaps. San Francisco, CA: Jossey-
Bass.
Stetson, D. M. and A. Mazur, eds (1995) Comparative State Feminism. Thousand Oaks,
CA: Sage.
Valiente, C. (1995) ‘The Power of Persuasion. The Instituto de la Mujer in Spain’, in D.
M. Stetson and A. Mazur (eds) Comparative State Feminism, pp. 221–36. Thousand
Oaks, CA: Sage.
Valiente, C. (1996) ‘El feminismo institucional en España: El Instituto de la Mujer,
1983–1994’, Revista Internacional de Sociología 13(1): 163–204.
402


Valiente, C. (1997) ‘State Feminism and Gender Equality Policies. The Case of Spain
(1983–95)’, in F. Gardiner (ed.) Sex Equality Policy in Western Europe, pp.127–41.
London: Routledge.
Verloo, M. (1998) ‘Policy Design and Frame Alignment: The Case of Mainstreaming’,
Paper presented in the ECPR 26th Joint Sessions of Workshops, Warwick, UK.
Verloo, M. and C. Roggeband (1996) ‘Gender Impact Assessment: The Development of
a New Instrument in the Netherlands’, Impact Assessment 14(1): 3–20.
Weiss, C. H. (1987) ‘Where Politics and Evaluation Research Meet’, in D. J. Palumbo (ed.)
The Politics of Program Evaluation, pp. 47–69. Newbury Park, CA: Sage.
Weiss, C. H. (1998) Evaluation. Methods for Studying Programs and Policies, 2nd edn.
Upper Saddle River, NJ: Prentice-Hall.
MARÍA BUSTELO is associate professor of political science and public

administration and Head of the Evaluation Unit at the Centro de Gestión
(Complutense University). She has worked for 15 years as an evaluation
consultant and her main interests are public policies evaluation, evaluation
training, and the evaluation of gender equality and drug prevention policies.
Please address correspondence to: Departamento de Ciencia Política y de la
Administración II, Facultad de Ciencias Políticas y Sociología, Universidad
Complutense, Campus de Somosaguas s/n. 28023 Madrid, Spain.
[email: mbustelo@cps.ucm.es]
403


Gender - Mainstreaming Bustelo PDF

Uploaded by

Copyright:

Available Formats

You might also like

Gender - Mainstreaming Bustelo PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Gender - Mainstreaming Bustelo PDF

Uploaded by

Copyright:

Available Formats

Evaluation

Evaluation of Gender Mainstreaming: Ideas from a Meta-Evaluation Study

The online version of this article can be found at:

The Tavistock Institute

Additional services and information for Evaluation can be found at:

Email Alerts: http://evi.sagepub.com/cgi/alerts

Citations (this article cites 3 articles hosted on the

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Evaluation of Gender Mainstreaming

The new policy strategy of ‘gender mainstreaming’ poses particular

K E Y WO R D S : gender-equality policies; gender mainstreaming; meta-evaluation;

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

The Case Study: The Evaluation of Public Gender-equality

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Bustelo: Evaluation of Gender Mainstreaming

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Bustelo: Evaluation of Gender Mainstreaming

Evaluation role All 11 experiences analysed are summative. A summative

Table 1. Types of Evaluation in the Second-generation Evaluations of Gender-equality

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Evaluation content Three types of evaluation are distinguished: design evalu-

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Bustelo: Evaluation of Gender Mainstreaming

Evaluation timing All the second-generation evaluation experiences were ex

Evaluation agent Evaluations may be internal or external, depending on

Contextual Factors that Influence Gender-equality Policy

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Factors that Influence Gender-equality Policy Evaluation Positively

The increase of formal commitments concerning gender-equality plans and their

International support for gender-equality policies The 4th World Conference

The existence of consolidated gender-equality organisms Currently in Spain,

Acknowledgement of programme and policy evaluation in recent years

Factors which Influence Gender-equality Policy Evaluation Negatively

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Bustelo: Evaluation of Gender Mainstreaming

The integral and horizontal nature of gender-equality plans Gender-equality

The policy of persuasion Because of gender mainstreaming and the horizontal

Lack of experience, culture of evaluation and consolidation of the evaluation

Lack of competent and prepared teams for the evaluation of gender-equality

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Some Results and Conclusions about the State of the Evaluation

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Bustelo: Evaluation of Gender Mainstreaming

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

2. Evaluations should enhance a global view of the public action taken to

None of the experiences analysed is an evaluation of the global gender-equality

3. There is a need to acknowledge the political nature of evaluation.

In most cases, the evaluation function was considered an exclusively ‘technical’

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Bustelo: Evaluation of Gender Mainstreaming

One of the main problems, especially mentioned by the femocrats in charge of

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

It was almost impossible to measure the real investment of Spanish women’s

6. There is a need for promoting greater stakeholder participation in the evalu-

Active stakeholder participation is considered to be a key element for obtaining

Downloaded from http://evi.sagepub.com at INDIANA UNIV on July 9, 2008

Bustelo: Evaluation of Gender Mainstreaming