American Journal of Evaluation

http://aje.sagepub.com/

Integration of Social Justice Into the Teaching of Evaluation

Veronica G. Thomas and Anna Madison
American Journal of Evaluation 2010 31: 570
DOI: 10.1177/1098214010368426

The online version of this article can be found at:

http://aje.sagepub.com/content/31/4/570

Published by:

http://www.sagepublications.com

On behalf of:

American Evaluation Association

Additional services and information for American Journal of Evaluation can be found at:

Email Alerts: http://aje.sagepub.com/cgi/alerts

Subscriptions: http://aje.sagepub.com/subscriptions

Reprints: http://www.sagepub.com/journalsReprints.nav

Permissions: http://www.sagepub.com/journalsPermissions.nav

Citations: http://aje.sagepub.com/content/31/4/570.refs.html

>> Version of Record - Oct 26, 2010

What is This?

Downloaded from aje.sagepub.com by guest on November 20, 2012

Teaching Evaluation
American Journal of Evaluation
31(4) 570-583
ª The Author(s) 2010
Integration of Social Reprints and permission:
sagepub.com/journalsPermissions.nav
Justice Into the Teaching DOI: 10.1177/1098214010368426
http://aje.sagepub.com
of Evaluation

Veronica G. Thomas1 and Anna Madison2

Abstract
This article argues that social justice should be included in evaluation education as a fundamental
value in evaluation practice. A social justice orientation will provide students with a perspective
that will enable them to challenge existing evaluation hegemonic ontological, epistemological,
theoretical, and methodological practices that diminish groups at the margins of society and
normalize injustice. We suggest four major areas where educators can intersect social justice and
evaluation in classroom and field experiences. These include intersecting social justice with
evaluation (a) theoretical knowledge, (b) methodological knowledge, (c) interpersonal knowledge,
and (d) professionalism. Further, we maintain that a social justice orientation can be evident in
pedagogical approaches and the professors’ articulation of students’ expected learning outcomes.
Selected readings and various activities are provided that educators can utilize to integrate social
justice, evaluation theory, and methodology in graduate training in an effort to produce a more
critical evaluator.

Keywords
social justice, teaching evaluation, evaluation theory, methods, professionalism

Over the past several decades, there has been a growing body of literature linking social justice to
evaluation practice. Barry MacDonald (1976) and Ernest House (1980, 1990) were among the first
scholars to explicitly couple social justice interests and evaluation. In particular, MacDonald advo-
cated the use of democratic evaluation as a mechanism to depict the multiple realities of a program
with ‘‘justice and truth.’’ He argued for democratizing evaluation knowledge and broadening the
interests served beyond decision makers and experts to also encompass the interests of citizens,
at large. House (1990) viewed social justice as one of the most important values that the field should
aspire to fulfill with evaluation being an institution for democratizing public decision making by
making programs, policies, and decisions open to public scrutiny and deliberation. From this

1
Department of Human Development and Psychoeducational Studies, Howard University, Washington, DC, USA
2
College of Public and Community Service, University of Massachusetts at Boston, USA

Corresponding Author:
Veronica G. Thomas, Department of Human Development and Psychoeducational Studies, School of Education, Howard
University, 2441 4th Street, NW, Washington, DC 20050, USA
Email: vthomas@howard.edu

570 Downloaded from aje.sagepub.com by guest on November 20, 2012

Thomas and Madison 571

perspective, evaluation can contribute to enhancing fair and just distribution of benefits and respon-
sibilities, or, to continuing inequality and distorting such distributions.
In a volume of New Direction for Evaluation, Sirotnik (1990) provided a framework for examining
evaluation and social justice in both philosophical and methodological terms. He called for making
explicit the intimate connection between an ethical stance rooted in social justice and a methodological
stance rooted in the sociopolitical context of evaluation. One of the many profound questions raised in
this volume was regarding the place of social justice in the profession. Specifically, he asked ‘‘if eva-
luation of social programs is to serve the interests of society, then should not techniques and proce-
dures derive from, rather than frame, the moral and ethical dimensions of the evaluative problem’’
(Sirotnik, 1990, p. 1). More recently, evaluators are increasingly anchoring their work in an intentional
commitment to democratic social justice, equality, and emancipation (Greene, 2006). In doing so, as
Greene points out, evaluators are rejecting the possibility of value neutrality in evaluation and, instead,
fully embracing the intertwinement of values with evaluation practice.
Evaluators committed to social justice have pushed for using inclusive types of evaluation
approaches such as transformative (Mertens, 2003), democratic deliberative (House & Howe,
2000), critical evaluation (Fay, 1987), and cultural/contextually responsive (Frierson, Hood, &
Hughes, 2002; Hopson, 2000; Madison, 1992; Thomas & Stevens, 2004). Mertens’ (2003, 2009)
transformative paradigm, for example, argues that evaluators need to be cognizant of issues of social
justice that are operating in society and affect the definition of social problems. As such, evaluators
must be wary of deficit models that, essentially, blame individuals for social problems, rather than
consider how institutional practices or societal responses to the certain individuals or cultural groups
place them at increased risk for negative outcomes. Evaluators must work to facilitate an open dis-
course on social justice issues and how these issues affect the program being evaluated. The
approach of House and Howe (2000) calls for deliberation in service of social justice and argues that
evaluation must attend to the interest of all stakeholders, particularly the poor and powerless.
Deliberation in service of social justice, according to these authors, involves an evaluator not simply
ignoring imbalances of power in the evaluation process. Instead, the evaluator acts as a conscien-
tious professional who adheres to a set of defensible, carefully considered principles for engaging
inclusion, dialogue, and deliberation among relevant stakeholders. Critical evaluation, an approach
informed by critical social science epistemology, advocates for evaluations that perform social anal-
yses that illuminate structural injustices and promote action to redress these inequities. Cultural
competence in evaluation is grounded in work emphasizing evaluators’ explicit attention to issues
of diversity, cultural responsiveness, and multicultural validity (Botcheva, Shih, & Huffman,
2009; Frierson et al., 2002; Kirkhart 1995; Madison, 1992; Thompson-Robinson, Hopson, &
SenGupta, 2004). All of these issues have implications for the promotion of social justice and equity.
To better ensure that evaluation promotes equity and democracy, it is important that attention to
social justice be more explicit in training the next generation of evaluators. In this respect, a critical
question becomes ‘‘How do we shape graduate training in such a manner that both faculty and stu-
dents focus not only on evaluation theory, methods, and/or practice, but also on the social inequities
that shape problem identification and ultimately efforts to resolve such problems?’’ Historically,
evaluation training, as Patton and Patrizi (2005) pointed out, relied mainly on traditional didactic
teaching in the classroom to ground students in the scientific approaches that are the cornerstone
of the field. We argue that evaluation students also must be inspired to challenge the status quo,
to care about the interests of the disadvantaged, and to uncover weaknesses within the system that
contribute to inequities within society. Accomplishing this goal will require open and purposeful
foci on infusing social justice issues in classroom, clinical, and field experiences. We are not arguing
that social justice should be the sole focus of teaching evaluation; instead, our position is that expli-
citly integrating social justice issues in evaluation training can help students envision evaluation as a
practice aimed at contributing to social betterment, social action, and social change.

Downloaded from aje.sagepub.com by guest on November 20, 2012 571

572 American Journal of Evaluation 31(4)

The current article offers a perspective for infusing social justice into the teaching of graduate-
level courses in evaluation in an effort to produce a more critical evaluator. This must be explicit in
classroom content and field experiences, the expected student learning outcomes articulated in
course syllabi, as well as the instructor’s pedagogical approach. Without a concerted effort to pur-
posefully integrate social justice issues in evaluation courses, professors may overlook this content
or they may (erroneously) assume it to be understood by students.

Rationale for a Social Justice Orientation

A social justice-oriented evaluation examines the holistic nature of social problems. It seeks to
increase understanding of the interdependency among individual, community, and society using a
more judicious democratic process in generating knowledge about social problems and social inter-
ventions and using this knowledge to advance social progress. This orientation is not embraced by
all evaluators; yet, it has a nascent presence in evaluation theory and practice. Advocates of social
justice-centered evaluation begin with the moral perspective that respect for the rights of others are
important to conducting fair and valid evaluations and in engaging in meaningful public discourse.
This includes respect for the diverse world views, indigenous lived experiences, and the right to dis-
agree and to learn from disagreement. As Greene (2005) points out, evaluation provides a mechan-
ism to engage with our diversity, to learn how to live with, appreciate, and accept differences.
Evaluation should provide reasoned knowledge to stimulate public discourse about the outcomes
of social programming and the development of social policy.
We assert that social justice should be included in evaluation education as a fundamental value in
evaluation practice. Not all evaluation educators will value social justice as a laudable societal goal.
Nor will all evaluators accept our position that inclusion of diverse voices is the only way to achieve
universal validity in knowledge to fuel authentic discourse in a civil society. However, we are pre-
senting a framework for those educators who share our professional values and believe that social
justice not only should guide our work but should be the ultimate goal of social policy and programs.
Mertens’ (2009) presentation of the transformative paradigm, for example, brings clarity to how eva-
luators and evaluation practice can advance a diverse civil society toward social justice and human
rights goals.
Kushner (2009) summarized five outcomes as the argument for a social justice orientation in eva-
luation. These include (a) providing an open exchange of ideas across stakeholder groups, (b) mak-
ing the evaluation process a space where power inequalities can be (procedurally) neutralized (i.e.,
equal treatment for all including mother, manager, and minister), (c) promoting independence and
impartiality of the evaluator whose obligation is to address everyone’s dilemma, (d) promoting free
and open publication of evaluation reports, and (e) increasing recognition of collective responsibility
for enhancing public information. Consideration of these outcomes provides a useful starting point
for evaluation educators to begin their own reflection about the place of social justice in their own
teaching and learning.
We, as some other evaluators, acknowledge that there are competing or conflicting views about
the purpose of evaluation and its judicious use. These tensions include disagreement about the roles,
obligations, responsibilities and accountability of evaluators. Tension exists between the experimen-
talists, who view evaluation as a scientific objective enterprise alien from the individuals and com-
munities affected by the evaluator’s claims. For example, the whole idea of experimentation, as
conceptualized by Donald Campbell (1969) in this seminal article, ‘‘Reforms as Experiments,’’ is
to provide an approach to meaningfully learn about persistent social problems and to use knowledge
for social progress and reform. How then, can learning be meaningful if a large component of the
knowledge base is omitted? Kirkhart (2005) notes, and we agree, that if there is to be meaningful

572 Downloaded from aje.sagepub.com by guest on November 20, 2012

Thomas and Madison 573

knowledge, the methods and practice of knowledge construction must be culturally valid and must
be embedded in evaluation education. We are not challenging the merits of experimentation as a
research method. Rather, we are suggesting, consistent with Kirkhart, that the evidence generated
using the experimental design must be culturally valid to justify the evidence-based claims.
What then is value added when a social justice orientation is integrated into teaching evaluation.
First, social justice provides a professional value base for the practice of evaluation that allows the
evaluator to separate personal interests from the collective public good. Students’ professional orien-
tation to evaluation will prepare them to be critical, self-reflective practitioners. However, a social
justice orientation will also provide students with a perspective that will enable them to challenge
existing evaluation hegemonic ontological, epistemological, theoretical, and methodological prac-
tices that diminish groups at the margins of society and normalize injustice. They will be better able
to externalize and investigate power relationships and to examine themselves in relations to others,
thus transforming their own thinking about the communities in which they conduct evaluations. This
skill will not only improve evaluators’ relationship to and position in communities but will allow
them to generate a more valid evidence-based knowledge.
Another value-added feature of social justice integration in evaluation education is that it will
facilitate students’ ability to recognize social policy and programs that fail to address systemic,
structural injustices, and aid in their skills to pose evaluative questions to illuminate these deficits.
Too often, social programs address the symptoms of social problems rather than the causal condi-
tions. Emphasis is placed on interventions that address disparities in outcomes rather than causal
conditions linked to the structural dysfunction of society and the absence of procedural mechanisms
to correct inequity in opportunity. Thus, social injustices continue to be normalized and social policy
continues to be ineffective in addressing the root causes of persistent social problems. A social jus-
tice orientation prepares the student to challenge the notion that one group or groups should be per-
petually dependent on the charity of others, when the reason for the dependency is inequity in
opportunity and oppression that prohibits access to resources. These students will understand their
role as ‘‘stewards of the public good’’ (Greene, 2005) and that their evaluation work is inextricably
linked to good policy development and social programming that leads to social change.

Social Justice, Teaching, and Evaluation Course Content: Getting

Started
A social justice approach to teaching rejects the position that knowledge (and the knower) is neutral
and untainted by human interests but instead acknowledges that knowledge reflects power and social
relationships within a given society. Educators must be aware that adopting this position within the
classroom can sometimes lead to being criticized for being biased, political and value-laden
(Greene, Lincoln, Mathison, Mertens, & Ryan, 1998), and/or being marginalized from those espous-
ing more traditional approaches. Focusing on social justice requires evaluation educators to not
allow criticism to deter them from seeking connections that relate evaluation course topics to issues
of social importance and inequalities. Professors must purposefully select course readings, class-
room activities, and clinical/field experiences that help students identify or recognize persistent
social inequities and social privilege that might have relevance on the design, implementation, and
evaluation of social programs, as well as the outcomes of persons in such programs. Class coverage
and out-of-class experiences could be designed in such a manner to draw attention to applications of
evaluation processes and findings that illuminate inequities and promote social change.
Educators must expose students to a literature base and set of experiences that support the notion
of ‘‘evaluator as agent of change.’’ A focus on social justice changes the role of the evaluator from
the traditional judge of merit or worth to a social change agent (Patton, 1997), and therefore, changes

Downloaded from aje.sagepub.com by guest on November 20, 2012 573

574 American Journal of Evaluation 31(4)

students’ understanding of this role. Here, professors can facilitate discourse on values about indi-
viduals and systems and assumptions about the nature and causes of behavior. Furthermore, they can
provide opportunities to help students learn how to challenge oppression and societal injustice in
evaluation practice by illuminating inequities as they become apparent. Some may argue that this
moves the evaluator toward advocacy. Although it has been noted that advocacy is irreconcilable
with the notion of a respectable evaluation in most evaluation communities (Greene, 2006), there
is clearly a point of intersection among social justice, evaluation, and advocacy that can be addressed
in graduate training in evaluation. We value advocacy in evaluation to the extent that it entails an
explicit commitment to providing information that can be used to promote equity and fairness in
rewards, resources, and opportunities through the use of evaluative approaches that engages key sta-
keholders, provides voice to the less powerless, and illuminates structural and other inequities in
programs and policies.
Infusing social justice in evaluation course content can be done through efforts of professors to
expose students to theoretical frameworks that provide them with an understanding of social jus-
tice’s role in evaluation practice and engaging students in guided discussions and practice (via case
studies, simulations, and field experiences) of evaluation designs, methods, and procedures appro-
priate for addressing social justice questions. Professors can engage students in critical discussion of
ways that an evaluation might be done to enhance its utility for yielding valid data on project imple-
mentation, outcomes, and efficiency, as well as providing useful information for enhancing equity
and opportunity.
Within evaluation courses, readings, assignments, case studies, and field experiences can be tai-
lored with the goals of having students use them to make decisions about certain courses of action,
reanalyze decisions, work through, and draw the intersection among theory, methods, ethics, and
social justice. As an example, focus on work by Kirkhart (1995, 2005) can scaffold students’ under-
standing of how validity and justice intersect to the extent evaluating with validity must address
issues of power and privilege and consideration of whose interests are served and not served and
how those interests are registered and understood. A selected listing of recommended readings for
educators interested in integrating a social justice orientation in evaluation teaching is found in
Appendix A. This listing is not intended to be comprehensive but rather an identification of seminal
literature for individuals seeking to learn more about the intersection of evaluation with social justice
agendas.
Building upon work on the essential evaluator competencies (Stevahn, King, Ghere, & Minnema,
2005) and types of critical knowledge (e.g., Levin-Razalis & Rosenstein, 2003), we suggest four
major areas where educators can intersect social justice and evaluation in classroom and out-of-
class experiences. These include intersecting social justice with evaluation (a) theoretical
knowledge, (b) methodological knowledge, (c) interpersonal knowledge, and (d) professionalism.
Furthermore, we maintain that a social justice orientation can be evident in pedagogical approaches
and the professors’ identification of students’ expected learning outcomes that are usually explicitly
articulated in course syllabi.

Theoretical Knowledge and Social Justice-Oriented Evaluation

Evaluation theory, although disparate and in most cases unproven, provides a wide range of
theoretical perspectives concerning the construction of knowledge, use of knowledge, and valuing
knowledge. Social justice evaluation encompasses strains of methods, use and valuing theories, or
approaches that provide a more equitable distribution of power in judging and valuing. Existing eva-
luation theoretical constructs and paradigms that advance social justice-oriented evaluation include
transformative evaluation (Mertens, 1999, 2009; Mertens & Hopson, 2006), deliberative democratic

574 Downloaded from aje.sagepub.com by guest on November 20, 2012

Thomas and Madison 575

evaluation (Greene 2005; Greene, Millett, & Hopson, 2004; Henry, 2000; Hopson, 2009; House,
2000), contextually and culturally responsive evaluation, (Hopson, 2009; Hood, Hopson, &
Frierson, 2005; Thomas & Stevens, 2004), and multicultural validity (Kirkhart, 1995, 2005). Propo-
nents of these approaches address fundamental deficits in current evaluation practice including the
epistemological hegemony in social knowledge construction, conflicting interests in the use of
knowledge, the marginalization of some groups in society to the benefit of others, and lack of trans-
parency in public policy use or lack of use. In response to such deficits, these and other scholars
advance normative theories, paradigms, approaches, or models to improve evaluation practice. The
central theme in this body of work is advocacy for the engagement of diverse voices in the knowl-
edge construction process and in public discourse to affect social change in the public interest.
In addition to theoretical works, social justice-centered evaluation education can include empiri-
cal observations of the application of social justice principles in evaluation practice. There is a grow-
ing body of social justice-centered evaluations conducted in the United Kingdom, Australia, and the
United States (Asada, 2005; Cooper & Christie, 2005; Courtwright, 2007; Mackenbach & Gunning-
Schepers, 1997; Saunders, 2006; Weiner, 1990) that can be critically examined. Two themes predo-
minate in this work: (a) evaluation as a means to give voice to those who have suffered generations
of inequities in the social and political power structures maintained by the status quo and (b) the role
of evaluation, in a democratic society, to ensure that evaluation results find their way into the public
domain to engage public discourse. This emerging body of work provides opportunities for profes-
sors to assign critical readings on the application of social justice principles in evaluation. Addition-
ally, using this work as a foundation, assignments can be constructed that provide students an
opportunity to form their own perspectives regarding the relevance, feasibility, and potential bene-
fits of social justice-centered evaluations. Knowledge accrued from this work can provide graduate
students an opportunity to begin to pose questions to advance theoretical frameworks to guide their
own evaluation practice. Additionally, capstone experiences, such as the doctoral dissertation or
clinical internships, offer students intensive opportunities to formulate innovative questions leading
to the exploration of new theoretical constructs to guide social justice-oriented evaluation.

Methodological and Procedural Knowledge

Generally speaking, evaluation education focuses heavily on the methodological and procedural
aspects of evaluation, with little to no attention to issues of social justice and equity. However, meth-
ods are only the beginning of what a good evaluator needs to understand to succeed (Patton &
Patrizi, 2005). We argue that educators have a unique opportunity to use a social justice lens in
teaching evaluation methodology. Kushner (2000) maintained that methodology should be thought
of as a means of reconciling the twin demands of the evaluator’s own values in respect for social
justice and the characteristics of the field under study. Although he clearly did not propose that eva-
luation move into advocacy, per se, Kushner recognized that the evaluator’s values play a role in
shaping the inquiry.
Evaluation methodology itself has sometimes led, in complex and subtle ways, to systematic
injustices with the main source of error lying in our standard conception of causation, which is inad-
equate and incorrect (House, 1990). To redress this omission, methods courses must move away
from the singular positivist perspective stressing objectivity and professional distance. Such a
stance, according to House (1991), only serves to minimize, if not totally ignore, stakeholders needs
and goals. Furthermore, he argues that in an attempt to remain objective, contrary to embracing con-
stituent agendas, evaluators often fall prey to only hearing and valuing the program’s or funder’s
agenda to the exclusion of competing or complimentary views. As a result, House recommends a

Downloaded from aje.sagepub.com by guest on November 20, 2012 575

576 American Journal of Evaluation 31(4)

more open process where evaluation methodologies are determined by program realities including
stakeholder perspectives and goals, planned as well as consequential, or unanticipated outcomes.
In evaluation methods courses, it is imperative that professors convey to students that although
methods are critical, first and foremost, they must understand that evaluation is also very much a social
enterprise that is best understood by taking into consideration the social, cultural, economic, and polit-
ical contexts surrounding the program under consideration. As Waters (1998) so aptly noted, evalua-
tors cannot ignore the fact that they become a part of the never-ending struggle to make judgment calls
about a social activity that creates the conditions or obstacles for social mobility. Through discussion,
case studies, and simulations, evaluation students can be provided multiple opportunities to reflect on
how their own experiences of privilege and discrimination may either help or hinder social justice
interests though influencing their understanding of the lived experiences of persons in programs.
Explicit discussion regarding specific ways an evaluator’s values, beliefs, and prejudices can and
do influence a number of critical aspects of the research process is important. Thomas and McKie
(2006) delineated several ways in which this can occur, all having fairness and social justice
implications. These ways include influencing (a) what questions an evaluator asks and ultimately
not ask, (b) what an evaluator illuminates and ultimately minimizes, (c) what evaluation approach
is used and ultimately not used, (d) what data are collected and ultimately overlooked, (e) how inter-
pretations are made and whose interpretations are held in high or low esteem, (f) what conclusions
are drawn and what conclusions are not considered, and (g) how results are presented and to whom
such results are disseminated. For example, while evaluators generally ask questions, based on the
demands of the funder or project administrators, a social justice approach broadens the lens through
which issues are raised and questions are formulated. This calls for professors to facilitate students’
appreciation for and understanding of the necessity of engaging in continued discourse with stake-
holders, especially those individuals whose voices have been historically excluded from shaping
research, to ensure their concerns are addressed. It also calls for shaping questions to include atten-
tion to contextual issues (e.g., geographical location, social climate, political climate, economic con-
ditions, etc.) that might influence project implementation and participant outcomes.
Furthermore, it is imperative that educators prepare students to consider application of both quan-
titative and qualitative techniques that can offer a useful depiction of the program’s context, imple-
mentation, and outcomes. Mertens (2003) discussed the potential strength in combining qualitative
and quantitative methods and how the intersection of mixed methods and social justice has impli-
cations for the role of the researcher and the choice of specific paradigmatic perspectives. Interest
in social justice may result, for example, in an evaluator’s decision to hold focus groups or inter-
views as a way of exposing the plurality of diverse voices in a program under study, in addition
to collecting quantitative data. Mixed methodologies can provide a mechanism for addressing the
complexities of evaluation research in culturally complex and diverse settings which, in turn, can
facilitate empowerment, social change, and social justice.

Interpersonal Knowledge and Skills

Justice, as House and Howe (1999) maintain, requires giving stakeholders an effective voice in
negotiating goods and defining their own needs. Few graduate courses in evaluation provide formal
training in the interpersonal skills generally required for successful completion of an evaluation and
giving authentic voice to stakeholders, especially those who are members of marginalized groups.
Such skills include verbal and listening communication skills, negotiation and conflict resolution
skills, constructive interpersonal interaction (e.g., teamwork, group facilitation, and processing), and
cultural competence (Stevahn et al., 2005). Social science research is often firmly grounded in social
relations and trust. As such, it is important for evaluators to mindfully attend to trust building as a

576 Downloaded from aje.sagepub.com by guest on November 20, 2012

Thomas and Madison 577

foundation for ensuring high-quality evaluations that are both valid and just. Without such attention,
a researcher’s roles and responsibilities often automatically engender fear and mistrust (Symonette,
2009). Through discussions, case studies, and simulations, professors can provide students with myr-
iad opportunities to practice multiple dialogues (e.g., oral and written) with diverse stakeholders in
contextually diverse settings. Students can also be introduced to the concept of ‘‘interpersonal valid-
ity,’’ as discussed by Kirkhart (1995), which refers to the soundness and trustworthiness of a kind of
understanding emanating from personal interactions. This dimension relates to the skills and
sensitivities of the evaluator and how one uses oneself as knower and as inquirer. Educators can
emphasize critical interpersonal skills such as active listening that demonstrates authentic interest
in another’s point of view which, in turn, places the evaluator in a better position of trust building.
Cultural competence, which we contend is both an interpersonal and technical skill, is a critical
dimension for enhancing students’ effectiveness as an evaluator working across diverse settings.
Some even argue that cultural competency, per se, is a major social justice intervention (Vera &
Speight, 2003). Cultural issues should be explicitly covered in evaluation courses seeking to infuse
social justice issues. Through critical reviews of actual evaluation reports (in whole or part), students
can be more effectively trained to recognize when evaluation results may be culturally inappropriate,
which can ultimately put the evaluation at risk of either creating or perpetuating stereotypes of
underrepresented and socially oppressed groups.

Evaluation Professionalism
It is essential that graduate training emphasizes evaluation professionalism, which includes
understanding of professional ethical standards. Ethics is concerned with formulating principles and
codes of moral behavior. Professors seeking to infuse a social justice emphasis related to ethics and
professionalism must aim to heighten students’ awareness of various ethical responsibilities that take
center stage when conducting evaluations particularly with diverse populations and historically
disenfranchised groups. Teaching ethical responsibility from a social justice perspective requires
professors to obviously include focus on the American Evaluation Association’s (AEA) Guiding
Principles for Evaluators but also to reach beyond discussion of the long-standing and obvious ethi-
cal concerns (e.g., doing no physical harm to participants) to consider other ethical issues that may
be more subtle. These include consideration of issues such as the right of evaluators to impose their
ideology on the people being studied, unequal power relations between the researcher and the
researched, the right of oppressed individuals to help shape evaluation questions and interpretations,
and the lack of input from participants about how knowledge generated from studies of their lives
should be used (Thomas, 2009).
Supervised clinical/field experiences provide a great opportunity to scaffold students’ under-
standing for intersecting issues of professionalism, social justice, and evaluation. Greene’s
(2006) borrowed notions of ‘‘visiting’’ and ‘‘listening well,’’ in particular, are important skills for
students to hone in during clinical and field experience. Visiting and listening well are described
as ‘‘respecting diverse standpoints through dialogue with other people, listening to their stories,
and relating to their uniqueness without collapsing these divergent views into a generalized amal-
gam . . . . [without essentializing them, and without losing your own unique standpoint] (Coulter &
Wiens, 2002, p. 18). Creating class and field experiences that nurture ‘‘visiting’’ and ‘‘listening’’
skills increases students’ capacity to obtain information that can uncover injustices and generate
information that can be used to reform such inequities. Ultimately, students must be challenged to
think about ways in which the evaluation process can be democratized through more respectful
and participatory approaches.

Downloaded from aje.sagepub.com by guest on November 20, 2012 577

578 American Journal of Evaluation 31(4)

Table 1. Sample Expected Student Learning Outcomes for Infusing Social Justice Issues and Evaluation
Content

By the end of the course, students should be able to:

(a) explain how evaluations are inextricably tied to particular social and institutional structures and practices
and how this influences what is studied
(b) critique and apply social justice approaches to evaluation theory and practice;
(c) demonstrate effective skills for engaging stakeholders in a just and equitable manner, including listening,
oral and written communication skills, problem solving, critical thinking, and diversity awareness;
(d) assess their commitment to social justice equity and demonstrate how this might be realized in evaluation
planning, implementation, dissemination and use of findings

Pedagogical Considerations and Expected Student

Learning Outcomes
Social justice can be integrated not only by the content that is taught but the manner in which that
content is taught. Regardless of the subject manner, professors who are concerned with social justice
often question conventional notions of authority in college classrooms because it may parallel or
reinforce how power is exercised more broadly in society (Meyers, 2007). Paulo Freire (1971),
Brazilian educator, who is most closely associated with teaching for social justice, maintains that
teaching, itself, is a political act that is not neutral. Professors must critically examine not only what
they teach in evaluation courses but the ways in which they teach and how the teaching and learning
process serves to either enhance or restrict social justice agendas. There are numerous examples of
pedagogical practices drawn from the multicultural and feminist literature that can assist evaluation
educators in creating classroom environments and activities to help students appreciate and value a
social justice orientation. Pedagogical frameworks for instructors to implement a social justice
agenda in their classroom often emphasize three major elements: (a) reliance on egalitarian methods
such as participatory learning, dialogue, and self-reflection; (b) holistic learning based in coming to
consciousness about oppression and using this consciousness to teach and empower others; and (c)
efforts to equalize power inequalities in the teacher–student relationship and classroom dynamics.
A critical aspect influencing an instructors’ pedagogy entails their articulation of what successful
students should know, value, and be able to do on successful completion of the course. These
expected student learning outcomes generally appear in the course syllabus. When infusing social
justice in the teaching of evaluation, it is important that professors explicitly articulate a set of
expected learning outcomes aimed at fostering student development in this area. These outcomes
must be clear, widely shared, and operational. In Table 1, we offer some illustrative examples.
As an example from Table 1, the first learning objective states that by the end of the course,
students should be able to ‘‘explain how evaluations are inextricably tied to particular social and
institutional structures and practices and how this influences what is studied.’’ Students could ben-
efit from interactive activities designed to provide a range of reflective questions that encourage
them to think critically about institutional structures, practices, and opportunities. Table 2 depicts
an example of group activity that might be useful in this respect. In class, the professor could chal-
lenge students to expand their understanding of ‘‘access’’ and ‘‘opportunity’’ as issues directly
related to social justice and equity, not only for persons from underrepresented groups but for soci-
ety as a whole. Additionally, professors can become actively engaged with students in a meaning-
ful analysis about the neutrality and objectivity as values that are held in high esteem in many
sectors of the evaluation profession. However, this dialogue can be coupled with the reality that
power, as Prilleltensky and Nelson (2002) assert, can quite often operate under the guise of
neutrality.

578 Downloaded from aje.sagepub.com by guest on November 20, 2012

Thomas and Madison 579

Table 2. Example of Classroom-Based Activity to Foster Discussion of Influence of Social and Institutional
Structures and Practices on Evaluation Planning and Implementation

Small Group Exercise: ‘‘Evaluating Mathematics and Science Projects’’

Instructions: You have been given the opportunity to evaluate a mathematics project serving students from
underrepresented groups. The goal of the project is to increase mathematical knowledge and skills, as well as
enhance students’ understanding of the usefulness of mathematics in everyday life. Organize into small groups
(2–4 students) and discuss the following:
(1) What kinds of questions would you ask about access and opportunity?
(2) What data would be essential to collect to examine contextual clues (e.g., social and institutional practices)
while simultaneously paying attention to issues of access and opportunity?
(3) How would you consult with various stakeholder groups, throughout the entire evaluation process, with
sufficient deliberation to enable them to help define their own needs and shape and bound the evaluation
project?
(4) How would you utilize methodologies that encourage, rather than deter, participation to authentically
capture the lived experiences of stakeholders?
(5) Why is it important to design, implement, and evaluate this project with the use of ‘‘contextual lenses’’ (e.g.,
historical and political perspectives)?
After discussion, the groups report back to the class for feedback and reflection.

Professors are encouraged to incorporate multiple pedagogical approaches in teaching evaluation

and scaffolding students toward the expected learning outcomes. In the example of the first learning
outcome in Table 1 and the accompanying sample activity (Table 2), cooperative learning is pro-
moted through organizing students to work together to maximize their own as well as other group
members’ learning experiences. Here, students are engaging in exploration and critical thinking
among themselves as well as interacting with the professor who sets the boundaries for the discus-
sion and reflection on the intersection of evaluation and social justice in the particular case study
provided. Active learning can be embedded in class-based (e.g., readings, case studies, role play,
and course projects) and out-of-class activities (e.g., practical, field experiences) by affording stu-
dents opportunities for learning through doing and reflecting on how social justice issues have rele-
vance in this work. For example, in clinical or field experiences, self-directed learning is another
pedagogical technique that can be adopted by the professor to promote students’ responsibility in
examining their own assumptions and values and their understanding about how power dynamics
will affect their (as evaluators) interactions with stakeholders.
It must be pointed out that in graduate programs, a faculty member is not only a teacher but also a
mentor, advisor, and sometimes, learner. Graduate professors who conduct evaluations can serve as
invaluable role models by practicing their craft in ways that attend to social justice issues and bring-
ing those experiences back to the classroom as vivid examples for students. Furthermore, professors
can use these opportunities to highlight the real-world opportunities and challenges in practicing
evaluation from a social justice orientation.

Conclusions
This article calls for infusing social justice issues into the teaching of evaluation. Over 15 years ago,
House (1991) argued that evaluations should be socially just and that they should attend to the inter-
ests of everyone in society. However, issues of social justice are oftentimes absent or minimally
addressed in evaluation education. It is imperative that the next generation of evaluators is armed
with the theoretical knowledge, methodological skills, interpersonal skills, and evaluation profes-
sionalism to help move us closer toward a profession where social justice and fairness are at the cen-
ter, rather than the margins.

Downloaded from aje.sagepub.com by guest on November 20, 2012 579

580 American Journal of Evaluation 31(4)

On the most basic level, educators can provide a space for students to critically explore social
justice issues in evaluation training. In doing so, professors must take responsibility in selecting,
developing, and implementing course content (i.e., materials and experiences) in a manner that
explicitly address social justice in ways that students can link it to good evaluation practice. How-
ever, it must also be recognized that widespread infusion of a social justice orientation into evalua-
tion curricula has some inherent challenges. First, it is unlikely that all professors of graduate courses
in evaluation will embrace this orientation inasmuch as this approach, as pointed out earlier, changes
the focus of the evaluator’s role (from judge of merit and worth to change agent) and it sets aside the
values of objectivity and neutrality. Second, because there are few graduate degree granting pro-
grams in evaluation, most evaluation courses are taught within various graduate degree programs
such as education, sociology, psychology, social work, or public health. In these instances, the pro-
gram typically offers only 1–2 courses in evaluation where the professors are generally oriented
toward providing students, over a 15-week period, with practical, methodological skills thus leaving
little time to attend to important subject matter relevant to a social justice agenda such as engaging
stakeholders, cultural competence, multicultural validity, interpersonal skills, and the role of social
inequities in shaping how problems are defined and solutions (i.e., social programs) are shaped.
Undoubtedly, there are myriad evaluation approaches and methodologies that can be incorpo-
rated into the teaching of evaluation. Infusing a social justice orientation into evaluation curricula
is one approach that we believe is both valuable and essential. Although doing so will, at times,
involves teaching at the borders where issues of privilege, democracy, human rights, and methodol-
ogy intersect, it is indeed a position that many evaluators already fully embrace. It is our hope that
the issues highlighted in this article and the suggestions offered will provide teachers of evaluation
with ideas about how cross-fertilization of social justice, evaluation theory, and methodology might
take place in graduate training in an effort to develop a more critical evaluator.

Declaration of Conflicting Interests

The author(s) declared no conflicts of interest with respect to the authorship and/or publication of
this article.

Funding
The author(s) received no financial support for the research and/or authorship of this article.

Appendix A
Preliminary List of Recommended Readings for Infusing Social Justice in
Evaluation Course Content
Greene, J. C. (2005). Evaluators as stewards of the public good. In S. Hood, R. Hopson, & H.
Frierson (Eds.). The role of cultural and cultural context: A mandate for inclusion, the
discovery of truth, and understanding in evaluative theory and practice (pp. 7–20). Greenwich,
CT: Information Age Publishing.
Greene, J. C., Millett, R. A., & Hopson, R. H. (2004). Evaluation as a democratizing practice. In
M. T. Braverman, N. A. Constantine, & J. K. Slater (Eds.), Foundations and evaluation: Con-
texts and practices for effective philanthropy (pp. 96–118). San Francisco, CA: Jossey-Bass.
Hood, S., Hopson, R., & Frierson, H. (Eds.). (2005). The role of culture and cultural context:
A mandate for inclusion, the discovery of truth and understanding in evaluation theory and
practice. Greenwich, CT: Information Age Publishing.

580 Downloaded from aje.sagepub.com by guest on November 20, 2012

Thomas and Madison 581

Hopson, R. (2009). Reclaiming knowledge at the margins: Culturally responsive evaluation in the
current evaluation moment. In K. E. Ryan & J. B. Cousins (Eds.). The Sage international hand-
book of educational evaluation (pp. 429–446). Thousand Oaks, CA: SAGE.
House, E. (1990). Methodology and justice. In K. Sirotnik (Ed.), Evaluation and Social Justice:
Issues in education (New Directions for Program Evaluation, 45, pp. 23–36). San Francisco,
CA: Jossey-Bass.
House, E. (1980). Evaluation with validity. Thousand Oaks, CA: SAGE.
House, E. R., & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks,
CA: SAGE.
Kirkhart, K. E. (2005). Through a cultural lens: Reflections on validity and theory in evaluation.
In S. Hood, R. Hopson, & H. Frierson (Eds.). The role of cultural and cultural context: A man-
date for inclusion, the discovery of truth, and understanding in evaluative theory and practice
(pp. 21–39). Greenwich, CT: Information Age Publishing.
Kirkhart, K. E. (1995). Seeking multicultural validity: A postcard from the road. Evaluation
Practice, 16, 1–2.
Kushner, S. (2009). Own goals: Democracy, evaluation, and rights in millennium projects. In K.
E. Ryan & J. B. Cousins (Eds.). The Sage international handbook of educational evaluation
(pp. 413–428). Thousand Oaks, CA: SAGE.
Kushner, S. (2000). Personalizing evaluation. London, England: SAGE.
Madison, A. (Eds.). (1992). Minority issues in program evaluation (New Directions for Program
Evaluation, 53). San Francisco, CA: Jossey-Bass.
Mertens, D. M. (2009.). Transformative research and evaluation. New York, NY: Guilford.
Sirotnik, E. A. (Ed.). (1990). Evaluation and social justice: Issues in public education (New
Directions for Program Evaluation, 45). San Francisco, CA: Jossey-Bass.
Thompson-Robinson, M., Hopson, R., & SenGupta, S. (Eds.). (2004). In search of cultural
competence in evaluation: Toward principles and practices. (New Directions for Evaluation,
102). San Francisco, CA: Jossey-Bass.

References
Asada, Y. (2005). A framework for measuring health inequality. Journal of Epidemiology and Community
Health, 59, 700-705.
Botcheva, L., Shih, J., & Huffman, L. C. (2009). Emphasizing cultural competence in evaluation: A process-
oriented approach. American Journal of Evaluation, 30, 176-188.
Campbell, D. T. (1969). Reform as experiments. American Psychologist, 24, 409-429.
Cooper, C. W., & Christie, C. A. (2005). Evaluating parent empowerment: A look at the potential of social
justice evaluation in education. Teacher College, 107, 2248-2274.
Coulter, D., & Wiens, J. R. (2002). Educational judgment: linking the actor and the spectator. Educational
Researcher, 31, 15-25.
Courtwright, A. M. (2007). Justice, health, and status. Theoria: A Journal of Social and Political Theory, 112,
1-24.
Fay, B. (1987). Critical social science: Liberation and its limits. Ithaca, NY: Cornell.
Freire, P. (1971). Pedagogy of the oppressed. New York, NY: Harper & Row.
Frierson, H. T., Hood, S., & Hughes, G. (2002). Strategies that address culturally responsive evaluation. In The
2002 user-friendly handbook for project evaluation (pp. 63-73). Arlington, VA: National Science
Foundation.
Greene, J. C. (2005). Evaluators as stewards of the public good. In S. Hood, R. Hopson, & H. Frierson (Eds.),
The role of culture and cultural context: A mandate for inclusion, the discovery of truth and understanding
in evaluation theory and practice (pp. 7-20). Greenwich, CT: Information Age Publishing.

Downloaded from aje.sagepub.com by guest on November 20, 2012 581

582 American Journal of Evaluation 31(4)

Greene, J. C. (2006). Evaluation, democracy, and social change. In I. F. Shaw, J. C. Greene, & M. M. Mark
(Eds.), The Sage handbook of evaluation (pp. 118-140). Thousand Oaks, CA: SAGE.
Greene, J. C., Lincoln, Y. S., Mathison, S., Mertens, D., & Ryan, K. E. (1998). Advantages and challenges of
using inclusive evaluation approaches in evaluation practice. Evaluation Practice, 19, 101-122.
Greene, J. C., Millett, R. A., & Hopson, R. H. (2004). Evaluation as a democratizing practice. In M.
T. Braverman, N. A. Constantine, & J. K. Slater (Eds.), Foundations and evaluation: Contexts and practices
for effective philanthropy (pp. 96-118). San Francisco, CA: Jossey-Bass.
Henry, G. (2000). Benefits and limitations of deliberation. In K. Ryan & L. Destefano (Eds.), Evaluation as a
democratic process (New Directions for Evaluation, 85, pp. 91-96). San Francisco, CA: Jossey-Bass.
Hood, S., Hopson, R., & Frierson, H. (Eds.) (2005). The role of culture and cultural context: A mandate for
inclusion, the discovery of truth, and understanding in evaluative theory and practice. Greenwich, CT:
Information Age Publishing.
Hopson, R. (Ed.). (2000). How and why language matter in evaluation (New Directions for Evaluation, 86). San
Francisco, CA: Jossey-Bass.
Hopson, R. (2009). Reclaiming knowledge at the margins: Culturally responsive evaluation in the current
evaluation moment. In K. E. Ryan & J. R. Cousins (Eds.), Sage international handbook of evaluation
(pp. 429-446). Thousand Oaks, CA: SAGE.
House, E. (1980). Evaluation with validity. Thousand Oaks, CA: SAGE.
House, E. R. (1990). Methodology and justice. In K. Sirotnik (Ed.), Evaluation and social justice: Issues in
education (New Directions for Program Evaluation, 45, pp. 23-36). San Francisco, CA: Jossey-Bass.
House, E. R. (1991). Evaluation and social justice: Where are we? In M. McLaughlin & D. Phillips (Eds.), Eva-
luation and education at quarter century, National Society for the Study of Education Yearbook
(pp. 233-246). IL: University of Chicago Press.
House, E. R., & Howe, R. H. (1999). Values in evaluation and social research. Thousand Oaks, CA: SAGE.
House, E. R., & Howe, R. H. (2000). Deliberative democratic evaluation. In K. E. Ryan & L. DeStefano (Eds.),
Evaluation as a democratic process: Promoting inclusion, dialogue, and deliberation (New Directions for
Evaluation, 85, pp. 3-11). San Francisco, CA: Jossey-Bass.
Kirkhart, K. E. (1995). Multiculturalism, validity, and evaluation theory. Evaluation Theories, 2, 1-3, 5–6.
Kirkhart, K. E. (2005). Through a cultural lens: Reflections on validity and theory in evaluation. In S. Hood,
R. Hopson, & H. Frierson (Eds.), The role of cultural and cultural context: A mandate for inclusion, the
discovery of truth, and understanding in evaluative theory and practice (pp. 21-39). Greenwich, CT: Infor-
mation Age Publishing.
Kushner, S. (2000). Personalizing evaluation. London, England: SAGE.
Kushner, S. (2009). Own goals: Democracy, evaluation, and rights in millennium projects. In K. E. Ryan & J.
B. Cousins (Eds.), The Sage international handbook of educational evaluation (pp. 413-428). Thousand
Oaks, CA: SAGE.
Levin-Razalis, M., & Rosenstein, B. (2003). A mentoring approach to the one year course in evaluation.
American Journal of Evaluation, 24, 245-259.
MacDonald, B. (1976). Evaluation and the control of education. In D. Tawney (Ed.), Curriculum evaluation
today: Trends and implications (pp. 125-136). London, England: Macmillan.
Mackenbach, J. P., & Gunning-Schepers, L. J. (1997). How should interventions to reduce inequalities in health
be evaluated? Journal of Epidemiology and Community Health, 51, 359-364.
Madison, A. (Ed.). (1992). Minority issues in program evaluation (New Directions For Program Evaluation,
53). San Francisco, CA: Jossey-Bass.
Mertens, D. M. (1999). Inclusive evaluation: Implications of transformative theory for evaluation. American
Journal of Evaluation, 20, 1-14.
Mertens, D. M. (2003). Mixed methods and the politics of human research: The transformative-emancipatory
perspective. In A. Tashakkori & C. Teddle (Eds.), Handbook of mixed methods in social & behavioral
research (pp. 135-164). Thousand Oak, CA: SAGE.

582 Downloaded from aje.sagepub.com by guest on November 20, 2012

Thomas and Madison 583

Mertens, D. M. (2009). Transformative research and evaluation. New York, NY: Guilford.
Mertens, D.M. & Hopson, R.K. (2006). Advancing evaluation of STEM efforts through attention to diversity
and culture. In D. Huffman & F. Lawrenz (Eds.), Critical issues in STEM evaluation. (New Directions for
Program Evaluation, 109, pp. 35-51). San Francisco, CA: Jossey-Bass.
Meyers, S. A. (2007). Putting social justice into practice in psychology courses. Observer, 20, 45-48.
Patton, M. Q. (1997). Utilization focused evaluation (3rd ed). Thousand Oaks, CA: SAGE.
Patton, M. Q. & Patrizi, P. (Eds.). (2005). Using the case method to teach evaluation (New Directions for Eva-
luation (p. 104)). San Francisco, CA: Jossey-Bass.
Saunders, M. (2006). The ‘presence’ of evaluation theory and practice in educational and social development:
Toward an inclusive approach. London Review of Education, 4, 197-215.
Sirotnik, E. A. (Ed.). (1990). Evaluation and social justice: Issues in public education (New Directions for
Program Evaluation, 45). San Francisco, CA: Jossey-Bass.
Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program
evaluators. American Journal of Evaluation, 26, 43-50.
Symonette, H. (2009). Cultivating self as responsive instrument: Working the boundaries and borderlands for
ethical border crossing. In D. M. Mertens & P. E. Ginsberg (Eds.), The handbook of social research ethics
(pp. 279-294). Thousand Oak, CA: SAGE.
Thomas, V. G. (2009). Critical race theory: Ethics and dimensions of diversity in research. In D. M. Mertens &
P. E. Ginsberg (Eds.), The handbook of social research ethics (pp. 54-68). Thousand Oak, CA: SAGE.
Thomas, V. G., & McKie, B. (2006). Collecting and utilizing evaluation research for public good and on behalf
of African American children. Journal of Negro Education, 75, 341-352.
Thomas, V. G., & Stevens, F. I. (Eds.). (2004). Co-constructing a contextually responsive evaluation frame-
work: The Talent Development Model of School Reform (New Directions for Evaluation, 101). San
Francisco, CA: Jossey-Bass.
Thompson-Robinson, M., Hopson, R., & SenGupta, S. (Eds.). (2004). In search of cultural competence in eva-
luation (New Directions for Evaluation, 102). San Francisco, CA: Jossey-Bass.
Vera, E. M., & Speight, S. L. (2003). Multicultural competence, social justice, and counseling psychology:
Expanding our roles. The Counseling Psychologist, 31, 253-272.
Waters, G. A. (1998). Critical evaluation for education reform. Education Policy Analysis Archives, 6, 1-39.
Weiner, G. (1990). Ethical practice in an unjust world: Educational evaluation and social justice, Gender and
Education, 2, 231-238.

Downloaded from aje.sagepub.com by guest on November 20, 2012 583