Evaluation and Program Planning 79 (2020) 101787

Contents lists available at ScienceDirect

Evaluation and Program Planning

journal homepage: www.elsevier.com/locate/evalprogplan

The whole elephant: Defining evaluation T

Amy M. Gullickson
Centre for Program Evaluation, University of Melbourne, Melbourne, Australia

ARTICLE INFO ABSTRACT

Keywords: Definitions help us understand the characteristics of an object or phenomenon and are a necessary precursor to
Evaluation understanding what a good version of it looks like. Evaluation as a field has resisted a common definition (Crane,
Evaluation education 1988; Morell & Flaherty, 1978; M. F. Smith, 1999), which has implications for marketing, training, practice, and
Evaluation theory quality assurance. In this position paper, I describe the benefits and challenges of not having a clear, agreed-upon
Evaluation practice
definition, then propose and explore the implications of two definitions for the evaluation profession based on
values and valuation as the core of evaluation practice. The purpose is to describe a possible way forward
through definition that would increase our professional profile, power, and contribution to social justice. The
paper concludes with implications for evaluator competencies and evaluation education and questions for fur-
ther research.

Good definitions are at the heart of understanding what something
is (and isn’t); how it should look, feel, taste, smell, and operate (or not);
what, if anything, it is supposed to produce; and what characteristics
make it a good (or not) version of that kind of something (Scriven,
1967). Different perspectives on a given something will often lead to
different definitions, as illustrated in the tale of the blind men of In-
dostan, who disputed furiously about whether the same elephant was a
wall, snake, rope, tree, spear or fan because they were each touching a
different part (Saxe, 2020). I posit that evaluation, as an emerging
profession, is the elephant in a similar situation, with sectors, practi-
tioners, and academics all defining it differently based on their per-
spectives. For example, economists may see evaluation only as cost-
benefit analysis (CBA), health and community sectors as social return
on investment (SROI), government policy makers as randomised con-
trolled trials (RCTs) (Gluckman, 2013), Indigenous people as an ex-
tension of colonial values, and social scientists as a type of applied
research. As with many aspects of evaluation, the answer to “What is
the definition of evaluation?” is “It depends.” If evaluation is to emerge
as a distinct profession, a more robust and agreed-upon definition is
needed; herein, I make a case for why a clear, unique definition is
important, offer possible definitions, and discuss their implications.
1. The case for a definition
Scholars researching and reporting on evaluation for more than
forty years have agreed that for evaluation to emerge as a profession, it
must have a unique body of knowledge and a logical boundary of the
problem area it is addressing (Crane, 1988; Morell & Flaherty, 1978;
Worthen, 1994). The way the profession defines its work provides the
boundaries for it, determining what areas of expertise are necessary for
its practitioners, and enabling descriptions of quality performance.
Without a definition, or with multiple definitions, those areas are un-
clear, and the criteria for quality will shift accordingly. As Fournier
(1995) expressed it: “[H]ow the phenomenon is defined (that is, so-
cially constructed)… influences the source or locus of the values from
which criteria are selected…[which] affects the validity of the con-
clusions” (p. 22). Take, for example, the different definitions of eva-
luation and their implications for expertise presented in Table 1.
Since evaluation is a market-based activity (Lemire, Nielsen, &
Christie, 2018; Smith, 1998), the profession needs clarity about ex-
pertise and quality not only for ourselves, but for our clients and other
stakeholders. All parties need a shared understanding of what con-
stitutes evaluation and evaluation quality to know whether they are
delivering it/getting it. Understanding quality in evaluation begins at
the same place as it does for any other evaluand – with the definition.
To date, evaluation has lacked a common definition that claims a
unique domain of expertise, although it has been called for (Smith,
1999). Discussions and publications on the topic provide evidence that
the field has resisted efforts in this direction, particularly if summative
judgement is included (Migotsky et al., 1997; Morell & Flaherty, 1978;
Poth, Lamarche, Yapp, Sulla, & Chisamore, 2014; Smith, 1999; Stake
et al., 1997). Unique domain is also a problem, as demonstrated by the

E-mail address: amy.gullickson@unimelb.edu.au.

https://doi.org/10.1016/j.evalprogplan.2020.101787
Received 7 August 2019; Accepted 20 January 2020
Available online 27 January 2020
0149-7189/ © 2020 Elsevier Ltd. All rights reserved.
A.M. Gullickson Evaluation and Program Planning 79 (2020) 101787

Table 1
Example definitions and their implications.
Source Definition Primary Expertise/Quality

Rossi, Lipsey, & Freeman,
2004, p. 2)
Scriven (1991, p. 1)
Mayne and Rist (2006, p. 97)
American Evaluation
Association (2004)
“a social science activity directed at collecting, analyzing, interpreting, Research methodology and methods.
and communicating information about the workings and effectiveness
of social programs”
“Evaluation is the process of determining the merit, worth, and value of Values and valuation
things, and evaluations are the products of that process”
In our view, evaluation and evaluators should be playing key roles in all Information and knowledge systems management; synthesis of
aspects of evaluative information in an organization: in building RBM empirical evaluation, research, monitoring and performance studies.
[results based management] capacity, in managing evaluative
knowledge systems, and in creating evaluative information and
knowledge, including through the conduct of evaluation studies…. We
use the term “evaluative information” and its counterparts to include
empirical information on and about the results of a service, program, or
policy…information from evaluation studies, results monitoring
systems, other one-off results-focused research, and performance
studies (Mayne & Rist, 2006), as well as evaluative information derived
from these various information sources through analysis, aggregation,
and synthesis (Rist & Stame, 2006.)”
“bettering products, personnel, programs, organizations, governments, Unclear. Could be change management, consulting skills, research,
consumers and the public interest; contributing to informed decision social justice, public administration, organisational theory and
making and more enlightened change; precipitating needed change; development, advocacy, human resources, decision science, etc.
empowering all stakeholders by collecting data from them and engaging
them in the evaluation process; and experiencing the excitement of new
insights”

perpetual EvalTalk question, “What is the difference between research
and evaluation?” The question has appeared every year since the list-
serv began more than 10 years ago – and answers vary by respondent.
In an example response, Rogers (RealEvaluation accessed 2019 02 12),1
posted four different ways to think about the answer: i) as a dichotomy,
ii) as mutually independent, iii) evaluation as a subset of research, or
iv) research as a subset of evaluation. While the global evaluation
community does seem to have agreement, for the most part, on five to
seven common competency domains (AES Professional Learning
Committee, 2013; American Evaluation Association, 2018; Aoteroa
New Zealand Evaluation Association, 2011; Canadian Evaluation
Society, 2010), these also do not clarify what makes evaluation unique
in the marketplace.
The benefit of the current “it depends” definitions is their inclusivity
Broad definitions like that of the 2004 AEA Guiding Principles and the
competency sets referenced above demonstrate a desire to include the
various activities in which evaluators engage and the various contexts
in which they work. Inclusiveness is even stated as an intent for the
2018 AEA competencies (American Evaluation Association, 2018).
However, ensuring that all are included tends to conflate the activities
of people who do evaluation and the contexts in which they do it with
evaluation itself. Usually evaluators are multi-purpose kinds of people,
accustomed to adapting what they do to their clients’ needs using a
variety of skills. Rogers’ four relationships between evaluation and re-
search provide further evidence of this inclusiveness issue, adapting
evaluation to suit the diversity of contexts and roles in which evaluators
work. By ensuring that evaluation encompasses all the activities eva-
luators do, we include everyone. By ensuring that evaluation fits in all
contexts and roles, we make evaluation a more marketable activity and
make evaluators more marketable for diverse roles. However, when
evaluation includes everything that evaluators do, it makes our defi-
nitional boundaries fuzzy and, thus, leaves evaluation vulnerable to co-
opting by other disciplines, who may define it in specific ways to meet
their needs. It does not make our profession distinct and recognisable as
one that provides a unique contribution in the marketplace.
1 of “fully describe, fully judge,” the task of evaluation is to provide a full
http://betterevaluation.org/blog/framing_the_difference_between_research_
and_evaluation
2. How might we define evaluation?
Along with a variety of other authors (Crane, 1988; Schwandt,
2008; Scriven, 1993), I propose that to make evaluation distinct, we
need to claim the professional space of judgement based on values by
applying the logic of evaluation – to set criteria and standards, measure
performance, and arrive at a judgement about that performance
(Fournier, 1995; Scriven, 1991). Values and criteria are an essential
conversation in today’s global political milieu, where political actors
have often overlooked sustainability and equity in favour of economic
or other short-term gains with increasingly visible negative con-
sequences (House, 2014; Mertens, 1994; Symonette, Mertens, &
Hopson, 2014). Claiming the process of value judgements differentiates
us from researchers, auditors, and organisational consultants and pro-
vides a distinct set of knowledge and skills to pursue for those seeking
to become professionals.
Historically, however, the field has resisted the explicit requirement
for a summative judgement – or really any judgement at all that ac-
companies this type of definition (Chelimsky, 2013; Weiss, 1988). An-
other challenge is the technical language, which has been difficult for
non-philosophers to grasp and use. As a colleague recently expressed it,
“I can’t talk with a client about the logic of evaluation – it’s too aca-
demic. Our clients won’t understand it, and they will not want to en-
gage with it.” While we need to have technical language as part of our
profession (Morell & Flaherty, 1978; Worthen, 1994), we do not need it
to be a barrier to people understanding what evaluation is and what
evaluators do. What is needed, therefore, is a definition of evaluation
that would (a) still make us distinct from other similar activities, like
research and consultancy; (b) include many of the activities evaluators
do so as to maintain the desired inclusiveness; (c) provide the defining
characteristics necessary for us to take the next step toward evaluating
ourselves; and (d) do it in plain language.
A combination of Stake’s (1977) definition (“fully describe, fully
judge”) and an expanded version of Scriven’s logic of evaluation
(Fournier, 1995; Scriven, 1991) would meet these needs. Stake (1977)
articulated the task of evaluation as follows: “Both description and
judgement are essential – in fact, they are the two basic acts of eva-
luation… To be fully understood, the educational program must be fully
described and fully judged” (Stake, 1977, p. 374). Under this definition
description of the evaluand, a justified set of criteria and standards by

2
A.M. Gullickson
Table 2
The Expanded Logic of Evaluation.
Step
0. Clarify evaluation purpose
and assess evaluability
1. Define the evaluand
2. Define the group to which
the evaluand belongs
3. Identify criteria/ delineate
evaluation questions
4. Identify performance
standards
5. Justify the criteria and
standards
6. Measure; observe evaluand's
performance
7. Justify the measures
8. Synthesise an evaluative
judgement
9. Justify the synthesis method
10. Report judgement
which to judge it, and a process that enables generation of a judgement.
Or, in simpler terms: deciding what makes something a something,
deciding how to know that something is good, and then deciding how
good a specific something is. To flesh it out, I have expanded Scriven’s
original logic (Gullickson, 2018) based on Nunns (2016) to make the
defining, judging, and the inherent reasoning aspects more visible
(Table 2).
Mapping Purpose
Elucidates the reasons for conducting an evaluation and
determine whether it is appropriate to do so.
Fully describe Describes the phenomenon to be evaluated (Fournier, 1995)
and sets the boundaries for the evaluation.
Fully describe Enables search for existing theories, norms, research,
criteria, standards, frameworks, taxonomies, and evaluations
to inform the evaluation process.
Fully judge Sets the parameters for how we can know if the something
defined in Steps 1 and 2 is a good something.
Fully judge Defines the threshold of performance that equates to “good”
for this evaluand.
Fully judge Sets up support for the definitions of “good” in the
evaluation process; addressed potential claims of evaluative
as subjective.
Fully describe Provides the evidence about what the evaluand does.
and fully judge Primarily descriptive, but informed by values, e.g., what
counts as evidence, what data sources and indicators have
been determined as important.
Fully describe Addresses threats to quality in Step 6.
and fully judge
Fully judge Combines the descriptions and aspects of judgement into a
verdict about the goodness of the evaluand.
Fully judge Provides support for the choices made in Step 8.
Communicates judgement to audiences. Considers all the
possible ways in which the process and findings of an
evaluation will be reported and the audiences to whom they
are reported.
In the expanded logic, the steps are listed in a linear order for the
sake of simplicity and referencing; however, working through the logic
may be a more iterative process that begins in a variety of places. For
instance, I often discuss reporting (Step 10) when talking with clients
about the evaluation purpose (Step 0) and criteria (Step 3). Discovering
which steps, if any, are inappropriate starting points, would need fur-
ther research and discussion. In the table I have used the term evaluand,
3
Evaluation and Program Planning 79 (2020) 101787
Potential Activities
Documenting the needs for the evaluation, its stakes and
stakeholders, the level of certainty required in the
conclusions, and the general state of the evaluand and its
fitness for the process.
Documenting (as appropriate) the needs or problem the
evaluand is addressing, the evaluand’s program logic and/or
theory and costs, acquiring specific context and evaluand
information, such as stakeholders and their power
relationships, purpose, goals, intended audience/consumers,
commissioners, required level of certainty for findings, stakes
for the evaluation and/or the program, etc.
Defining the evaluand’s field (e.g., program, policy, etc.);
discipline (e.g., education, political science, social work);
sector (government, NGO, private business, education); and
context (power dynamics, political pressures, relevant policy),
etc.
Identifying potential criteria and/or evaluation questions
(which have criteria embedded in them); negotiating
agreement on them and weighting of them (if needed), and
delineation of indicators (qualitative or quantitative) through
literature review, collaboration, stakeholder consultation,
negotiation, needs assessment and engagement with
normative ethical theories, theories of justice, values and
valuing.
Setting thresholds on indicators, determining what kind of
evaluative operations are best suited to each (grading,
ranking, scoring, apportioning, attributing), assessing context
(e.g., literature, culture, etc.), and determining required level
of sophistication, from complicated (multiple gradations of
good/not) to simple (e.g., acceptable or not, does no harm,
pass/fail).
Reasoning, argumentation, warranting, literature searches,
consultation, facilitation, negotiation, discussions on
contextual values and valuing, including understanding what
counts as evidence in this context.
Evaluation and research design, including economic analyses,
selection of data sources, selection and implementation of
evaluation methods (e.g., grading, ranking, scoring,
apportioning) and research methods, data collection and
analysis, measurement, adherence to relational and
procedural ethics, documentation and/or attribution of
impact and effectiveness of causal chains, etc.
Providing a rationale for the types of evidence (indicators)
selected and a warrant for their connection to the criteria and
standards, description of the accuracy and level of certainty of
the evidence, documenting trade-offs, discussions and analysis
related to ethics, validity, reliability, trustworthiness,
generalisability/transferability, and other relevant principles.
Rendering a judgement (formative or summative), at one or
several levels (individual components, criteria, or overall),
providing different weighting of data sources or criteria (Steps
4–7), using evaluation specific methods such as qualitative or
numerical weight and sum, rubrics, algorithms, decision trees,
etc.
Describing the rationale behind the judgements were made, at
what level and how, including the choice of synthesis
methods, providing reasons based on evidence from the
evaluand’s description and the evaluation requirements. A
clear and transparent process in Step 8 will streamline this
step.
Data visualisation, writing and verbal communication,
technology (e.g., web based, software tools), qualitative and
quantitative reporting, storyboarding, storytelling, video
production, etc., which address needs for use.
A.M. Gullickson
assuming a technical audience, but that could be replaced with “the
object” or “the something” to make it more accessible. Steps 0 and 10
acknowledge the nature of evaluation as a professional activity done for
an audience—and that the act of evaluation is nested in a negotiation of
the task and reporting of the findings. These two steps are not part of
the logical process of arriving at a judgment about the goodness of the
evaluand, although they can profoundly affect what happens in the
evaluation.
As demonstrated in Table 2, using “fully describe, fully judge” as the
general definition of evaluation means that activities that have been
categorised as evaluative, but are, in fact, descriptive, can be delineated
in the process. This provides several advantages, including: (a) clar-
ifying the difference between evaluators and evaluation, (b) resolving
some long-running debates in evaluation, (c) providing an orienting
frame to place cost-benefit analysis, social return on investment, ran-
domised controlled trials, and other activities within an evaluative
process; and iv) enabling us to be clearer with our clients and stake-
holders.
Consideration of a few key questions from our history will flesh out
these advantages. First, is judgement necessarily the job of the eva-
luator? Under this definition, no, but it is necessarily the task of an
evaluation, which fully judges. With that distinction clarified, we can
explain to clients how what we have done, if it is more on the “fully
describe” side, fits into the general process of evaluation. If getting to
judgement is not within the means or expectations of our work with a
client, the specialised knowledge evaluators would have under this
definition allows us to explicitly prepare them to do it on their own.
For example, in a recent evaluation, I used a much more basic
version of this framework with a client. Because their data had not been
set up for evaluation, we spent much of our time and budget cleaning
up and analysing their existing data and providing specific feedback on
what they could do to improve their data collection tools. We spent very
little time on establishing criteria and no time on standards or detailed
evaluative synthesis. Therefore, when reporting, we simply placed the
project in this process, explaining that we spent most of our time in the
fully describe space (program logic, observing performance), and,
based on what we reported, they could engage in ongoing conversations
about whether the performance was good enough – a process with
which we would be happy to assist. In this case, we did not “do” eva-
luation, but we prepared the clients to do it by explaining where what
we had done fit into the overall process of conducting an evaluation,
and flagged them on the importance of explicitly defining what good
looked like.
Second, are RCTs evaluation? Under this definition, RCTs alone are
not because they report on the size and significance of an impact, but
not its value. Thus, RCTs would fall on the “fully describe” side of the
definition, contributing information towards fully judging the goodness
of the evaluand. RCTs could inform answers to evaluative questions,
such as: were the impacts big enough to make a difference in people’s
lives? Did they benefit the most vulnerable in the population?
Third, are economic analyses like CBA evaluation? Again, under this
definition, no, because they are not a comprehensive description and
judgement of all the aspects of the program that relate to its value –
instead, they focus on one criterion to describe and judge. In both the
case of RCT and CBA, the values focus on just one aspect of the eva-
luand’s goodness (i.e., its causality or its economic performance). Either
may be critically important, but alone they lack the comprehensive
status of a good criteria of merit list, which is required to fully judge
(Scriven, 1994). The “fully describe, fully judge” definition allows us to
clearly explain how these important descriptive activities fit into the
overall task of evaluation or, in and of themselves, judge one aspect of
an evaluand, enabling us to operate on a more level playing field with
colleagues from business and research by clarifying that our distinctive
contribution is about value in the broader sense of merit, worth, and
significance (Scriven, 1991).
Finally, what is the relationship between research and evaluation?
4
Evaluation and Program Planning 79 (2020) 101787
In this case, the answer is clear. Under this definition, it is Rogers’
fourth relationship: research in the service of evaluation. Research
questions and findings are necessary to fully describe the evaluand, but
to fully judge requires additional tasks and skills.
3. Getting technical
Within the profession, a more technical, single sentence definition
might be valuable to clarify the knowledge and skills embedded in
“fully describe, fully judge,” which make evaluation distinctive and
highlight areas for both education and evaluation of quality practice. To
that end, consider a hybrid definition of evaluation based on Scriven
(1991) and Fitzpatrick, Sanders and Worthen (2011):
Evaluation is the generation of a credible and systematic determi-
nation of merit, worth, and/or significance of an object through the
application of defensible criteria and standards to demonstrably
relevant empirical facts
Scriven’s logic of evaluation provides the foundation of the defini-
tion because it is simultaneously differentiating and encompassing. It is
differentiating because it subsumes Scriven’s (1991) claim, “What dis-
tinguishes evaluation from other applied research is at most that it leads
to evaluative conclusions, and to get to them requires identifying
standards and performance data, and the integration of the two” (pp.
143–144). It is encompassing because it implicitly underpins all eva-
luation activity (Shadish, Cook, & Leviton, 1991) no matter the eva-
luand or its context. This definition states that what makes something
an evaluation is the synthesis of a judgement about an evaluand's value
based on criteria, performance standards, and factual evidence. The
addition of the adjectives “credible,” “systematic,” “defensible,” and
“relevant” from Fitzpatrick, Sanders, and Worthen provides criteria by
which we can judge the quality of evaluative activity and differentiates
personal preference evaluation from professional evaluation. What
makes a professional evaluation good is that those criteria apply to the
evidence, reasoning, and judgements involved.
4. Implications for educating evaluators
The combination of “fully describe, fully judge” and the more
technical definition has implications for competencies and evaluator
education, i.e., what makes an evaluator an evaluator instead of
something else. Based on the definitions above, an evaluator is someone
with the skills and knowledge to systematically and credibly determine
the merit, worth, and significance of an evaluand based on evidence. No
matter the context of the evaluation or title of the person conducting it,
the task and logic of evaluation remain the same. This has implications
for what professional evaluators need to know and be able to do. I
present an analysis of potential gaps below, informed by evaluator
competency sets and literature on evaluation education, begining with
some overall observations, and then progressing to specific areas that
require attention.
Overall, in the competency sets and evaluation education literature,
most attention has been given to the skills and knowledge needed for
the “fully describe” aspect of evaluation. As a result, there are fewer
gaps here. However, the “fully describe” activities for Steps 1 and 2
demonstrate that the profession may need a more in-depth investigation
and use of existing theories and models from disciplines and contexts in
which our evaluands reside. Evaluation education would need to attend
to these theories along with evaluation approaches.
Interpersonal skills are present in all the competency sets, but not
present in evaluation texts or often taught in formal curriculum; there is
some assumption that evaluators already have them (Dewey,
Montrosse, Schröter, Sullins, & Mattox, 2008). Analysis of the activities
column in Table 2 demonstrates that evaluators will frequently need
negotiation, facilitation, and communication skills. A more sophisti-
cated version of the public speaking idea of reading the audience could
A.M. Gullickson
subsume these into a higher order competency in the interpersonal
domain. Reading the audience in evaluation could mean understanding
the listeners’ or readers’ paradigms about evidence, the view they have
of evaluation, the political climate, the social norms, the cultural in-
fluences, etc. that will influence the way the evaluator needs to engage
with them about evaluation. It would combine situational analysis and
awareness, negotiation, facilitation, and communication, summarising
the interaction of these key competencies for evaluation practice.
As stated above, the competencies related to the “fully judge” side of
these evaluation definitions have generally had the least attention in
publication, competency sets, and curriculum. Based on the technical
requirements of “fully judge,” what knowledge, skills, abilities and
other characteristics align specifically with making systematic, credible,
relevant and defensible determinations of merit, worth, and sig-
nificance based on defensible criteria? An analysis of the existing AEA
competencies (2018) shows that many of these key competencies are
already covered, for example, good evaluation questions (2.2), ethical
practice (1.1), and using systematic evidence to make evaluative jud-
gements (1.4). Yet, based on the above definitions, we would need to be
a bit more specific and detailed about the skills and knowledge essential
to define those who would engage in professional evaluation, as fol-
lows.
4.1. The logic of evaluation and evaluation-specific methods
At the heart of practice based on these definitions is the logic of
evaluation: setting criteria of merit, determining performance stan-
dards, measuring performance, and conducting evaluative synthesis to
generate a value judgement. Therefore, an evaluator must know and
demonstrate the skills to answer questions such as: What are criteria of
merit? What are standards? What are criteria and standards for this
evaluand? Are any more important than the others? What methods and
resources can we use to generate them? What are the methods we can
use to synthesise evaluative conclusions from values and data? Which
methods are the best in what circumstances? How can we engage in
these methods as transparently as possible? What makes a synthesis
method a good choice in an evaluation? If we are about producing
defensible judgements of merit, worth and significance, then this
knowledge must be our stock in trade. Several scholars have noted the
importance of these questions to the discipline (e.g., Greene, 2005;
Mark, Henry, & Julnes, 2000; Schwandt, 2002), yet we have dedicated
very little time to these questions, and the requisite knowledge and
skills to address them, in the literature and in evaluation textbooks. A
recent review of research on evaluation published between 2005 and
2014 identified that few empirical investigations address questions re-
lated to values and valuing (Coryn et al., 2017). A recent study of
American Evaluation Association members showed that nearly three-
fourths were not at all familiar or only a little familiar with the logic of
evaluation (Ozeki, Coryn, & Schröter, 2019). To date, Davidson’s
(2005) Evaluation Methodology Basics is the only English text that dis-
cusses criteria, standards, and evaluative synthesis explicitly, and it is
14 years old.
4.2. Critical thinking and argumentation
Under these definitions, the task of evaluation involves reasoning
and critical thinking to shape arguments about the goodness of some-
thing (Patton, 2008; Shadish et al., 1991). Yet how many of our formal
training programs and professional development opportunities involve
in-depth work in the discipline of critical thinking and decision theory,
on understanding and using their tools and ideas? Based on the analyses
of LaVelle (2014) and Dewey et al. (2008), not many. Buckley and
colleagues (Buckley, Archibald, Hargraves, & Trochim, 2015) explored
how understanding and implementing the principles of critical thinking
could improve evaluation capacity building efforts, but valuing is
conspicuously absent from their definition and discussion on evaluative
5
Evaluation and Program Planning 79 (2020) 101787
thinking. This leaves work to be done. How can we deliberately use the
tools of critical thinking to build defensible, credible, and justified ar-
guments about the value of something? If evaluation is higher order
thinking (Bloom & Krathwohl, 1956), what are the lower orders that
need to be grasped first by the learner? How much practice and of what
type is needed?
4.3. Worldviews and values
The culture(s) of an evaluator’s upbringing and disciplinary edu-
cation will affect the way she sees the world, just as those same influ-
ences affect the clients and stakeholders involved in the evaluation.
These worldviews determine a person’s perspective on what constitutes
knowledge or evidence, appropriate ways to develop and discover it,
whether values are acceptable in seeking knowledge, the prioritisation
of individuals or community in that process, and what good looks like.
Philosophical traditions from Western ethics, for example, provide
normative descriptions of what is good, based aspects such as con-
sequences, duty, or an ethic of care (Newman & Brown, 1996). Some
scientific traditions require that for research to be good, it must be
objective (Creswell, 2003). Indigenous communities, in contrast,
prioritise relationship with people, community, and environment and
co-creation of knowledge, rather than objective investigation (Cram,
2018). Worldviews and values underlie what counts as credible and
relevant data, what constitutes culturally appropriate approaches or
procedures, and may constitute an area for an evaluator’s professional
development; however, recognizing and navigating worldviews is not
explicitly named in the AEA competencies.
If we accept the above definitions of evaluation as a profession, then
we claim that knowledge about values is central to the task. Thus, it
would not be acceptable for an evaluator to operate exclusively and
unknowingly within her own worldview and accompanying values, or
that of her client, or to assume that all parties involved share the same
values. Evaluation education, therefore, must provide not only an in-
troduction to the basic research paradigms, but also introduce Western
normative ethical philosophies and other worldviews that underpin
perspectives on what makes something good. Evaluators would need to
learn to recognise their own worldviews and the worldviews of others
and understand the implications of those for what good looks like for
the evaluation process and the evaluand. This is likely a gap in current
evaluation practice; according to Newman (1999), most evaluators do
not know that the Western normative ethical principles are sources of
values, nor do they understand the depth of tradition behind them or
how they are likely influencing value choices in evaluation. The recent
New Directions for Evaluation issue on Indigenous Evaluation (Cram et al.,
2018) demonstrates the consequences of not attending to the world-
view issue for Indigenous communities, and provides key insights on
what constitutes good in those communities. These resources provide a
starting point, but further work needs to be done to determine what
level of understanding evaluators need to have in this area to engage in
quality practice.
4.4. Deep understanding about subjective/objective claims
Related to worldviews, and particularly important for evaluation, is
that at the heart of a credible argument (above) about merit, worth, and
significance is assembling evidence in support of the evaluative claim.
Yet a major challenge in evaluation is that stakeholders will have dif-
fering epistemologies or may, in fact, make claims that the use of values
renders judgements invalid. Scriven’s arguments about the fallacy of
the value-free doctrine are well documented (Scriven, 1991, 2003,
Scriven, 2007, 2016); however, as evaluators, we need to be prepared
to counter the challenge of “that’s just subjective.” Petrie (1995) has
given us a potential start with his discussion of purpose, context, and
synthesis through the lens of perceptual control theory, as does
Scriven’s (2005) delineation of different types of value claims (personal,
A.M. Gullickson
market, contextual, essential). But we have several questions yet to
answer. Are value sets always subjective? Or contextual? Is a research
lens of external validity and/or transferability sufficient when we need
to consider whether an evaluation’s findings will generalise to other
contexts? How can we, in clear and compelling ways, articulate a de-
fence against claims of subjectivity and help clients see their own as-
sumptions about truth and value as part of the picture, rather than the
whole picture (Kegan & Lahey, 2001, Kegan & Lahey, 2009)?
4.5. Ethical conduct related to values
While the competency sets include ethical practice, the ethical im-
plications of evaluation go beyond observing research ethics protocols
and compliance with the Program Evaluation Standards (Yarbrough,
Caruthers, Shulha, & Hopson, 2010) or other guidelines. Current re-
search on ethics in evaluation has delineated a variety of issues that
evaluators face (Azzam, 2016; Morris, 2015). Several of these, like
subjectivity, normative ethical traditions, and types of value claims, can
be addressed through research and exploration of the suggested topics
above. However, the definition of evaluation as a determination of
merit, worth, or significance adds ethical challenges. For instance, what
are the ethical challenges inherent in working in communities who
have competing value sets related to the evaluand? Or where stake-
holders have opposing epistemologies and different types of evidence
offer conflicting perspectives? Much of the attention to culture and
advocacy has begun to address this, but very little has made it into the
field’s conversation around ethics. For instance, is it ethical for an
evaluation to be conducted by a team of people external to the com-
munity, using the values of an external funder or values implicit in the
literature, rather than on what the community holds to be of value?
I propose that these five areas of competency (the logic of evalua-
tion and evaluation-specific methods, critical thinking and argu-
mentation, deep understanding about subjective/objective claims,
worldviews and values, and ethical conduct related to values) in com-
bination are what make evaluation distinct as a profession. Certainly,
other skills and knowledge are important – even necessary – to con-
ducting evaluation, as Table 2, the existing competency sets, and the
discussion above delineate. However, research methods, project man-
agement, and interpersonal and consultancy skills are not what differ-
entiate evaluation from other fields. Placing evaluation activities within
the frame of making value judgements means that shapes the activities
and expertise of evaluators. Under this definition, it would be our aim
to make these judgements, or to support others in the making of them. If
we are going to be able to judge what makes someone an evaluator and
what makes something an evaluation, we must have a clear picture of
what differentiates us. The definitions proposed above and this list of
core competencies can provide that differentiation.
5. Visualising evaluation competence
With that in mind, I propose a visualisation of the set of compe-
tencies recently approved by AEA (Fig. 1), which orients the compe-
tencies in order of importance. The core evaluation specific compe-
tencies are in the centre because they delineate evaluation as a distinct
transdiscipline (Scriven, 2003) and provide the foundation for all the
other competencies. Their marked absence from texts, literature, and
most competency sets indicates the emphasis is necessary. In addition,
evaluators need these skills so they can make judgements about their
choices (or others’ recommendations) in the rest of the domains. Pro-
fessional practice becomes a second core set of competencies containing
standards, guidelines, approaches, and the work of evaluation theorists
– the methods for engaging with the logic of evaluation in practice.
Meta-evaluation could be placed within this domain or the core set as
an evaluation-specific competency. It seems likely that there are es-
sential aspects of the other domains, such as basic inquiry design, cost
analysis, reading the audience, understanding contextual values and
6
Evaluation and Program Planning 79 (2020) 101787
Interpersonal
Context
Evalua on specific
logic, methods &
ethics; worldview,
values, &
subjec vity;
accompanying
skills
Methodology Project
(incl. cost Planning and
analysis) Management
© 2019 A.M.Gullickson University of Melbourne
Fig. 1. Competency wheel, based on the 2018 AEA competencies with defini-
tionally differentiating and professional practice domains at the centre
(Adapted from Stevahn, King, Ghere & Minnema, 2004).
theories, and contracting, that would form a third layer, indicated by
the dotted circle around professional practice. The outer layers of the
diagram are the aspects of evaluation that an evaluator could contract
out, like project management, or that might be discipline specific, like
SROI.
6. Lessons learned
Definitions create boundaries and priorities by determining what
something is and what aspects of it are related to whether it is good. For
instance, if this paper had built instead on the (Rossi, Lipsey, &
Freeman, 2004) definition of evaluation as applied social science shown
in Table 1, the visualisation would have put research methods, rather
than evaluative reasoning and methods, in the middle. Building on
Mayne and Rist (2006) might have put knowledge management at the
centre. Research on evaluation degree programs indicates that a dia-
gram depicting their implicit definition would include whatever re-
search methods are appropriate to the discipline in which the course is
being taught, plus some evaluation approaches and professional prac-
tice items like ethics and the Program Evaluation Standards (Davies &
Mackay, 2014; Dewey et al., 2008; LaVelle, 2014). However, those
depictions and accompanying definitions would be less likely to clearly
differentiate evaluation as its own field, separate from research and/or
the discipline in which evaluation is being practiced.
A definition that clearly describes evaluation as including judge-
ments of merit, worth or significance is not necessarily as limiting as
previous discussions in the literature have indicated. Using Stake’s
“fully describe, fully judge” as the broad frame and elaborating care-
fully the steps within it related to description, judgement, and the
reasoning that connects them may resolve ongoing debates, provide
clarity for ourselves, our clients, and our competitors, and enable us to
unpack and prioritise the knowledge and skills evaluators need to do
the work of evaluation. Such a definition provides a way for evaluation
to position itself more clearly as a profession by defining its contribu-
tion.
The “fully judge” aspect of the definition places our work squarely
in the realm of value judgements. Organisations and governments need
a way to make stakeholders’ values visible, negotiate when they are
competing, and gather evidence of impact created by programs in re-
lation to those values. Claiming a definition that includes “fully judge”
A.M. Gullickson
gives us access to this space, to follow the research on it, and to bring
our various skills and tools to bear on the criteria that are used for
making decisions not only about what makes an existing policy or
program good, but what criteria should be used for designing programs.
This emphasis can increase the significance of our contribution. For
while improving evaluation designs (Bamberger, Rugh, & Mabry, 2012;
Campbell & Stanley, 1963; Campbell, 1969) and increasing evaluation
use and influence have had much attention historically (Kirkhart, 2000;
Patton, 2008; Weiss, 1988), history and a systems perspective show that
they are low leverage points that lack the capacity to change the system
(Meadows & Wright, 2008; Pawson & Tilley, 1997) because they are too
far away from key decision-making points.
Criteria are part of the blueprint of an evaluand and, thus, have
greater potential for impact, i.e., better leveragge. Changing the criteria
changes the whole understanding of the evaluand’s value. When values
and criteria are discussed at the beginning of program design, it is much
easier to make changes, just as it is much easier to change a blueprint
than a building (Carlis, 2013). Developmental evaluation (Patton,
2011) and evaluators’ recent engagement with design thinking (Chen,
Gargani, Stead, & Norman, 2015; Dart, Webb, & Tolmer, 2017) in-
dicates many evaluators are already onto this idea. Claiming the “fully
judge” definition provides us a clear pathway to become experts in
valuation and criteria, thereby laying the foundation to potentially in-
crease our impact on social justice. Consider the difference, for in-
stance, in a school building project where environmental sustainability
is the primary criterion, rather than ease for dropping off children by
car (Rowe, 2017).
There are risks inherent in adopting the “fully describe, fully judge”
framework. First, it may neutralise long-standing debates, which have
been personally important and professionally defining for evaluation
theorists. Second, placing evaluative reasoning and judgement at the
heart of the evaluation reduces the importance of evaluation ap-
proaches. Approaches have been personally important and profession-
ally defining to many, providing a pathway for individual evaluators to
gain recognition and reputation. Under these definitions, while ap-
proaches are still relevant, they are not central to understanding what
evaluation is and what it does. Instead they are strategies for engaging
with the “fully describe, fully judge” process (Fournier, 1995; Smith,
1995). Third, if this definition is applied retrospectively, it will reveal
that many activities that have been called evaluation were simply “fully
describe” activities – critically important and valuable, but incomplete
as evaluations because they lacked the judgement aspect. Other activ-
ities may be shown to have provided a judgment with scant attention to
the “fully judge” steps and thus fall short of what is required to deliver a
defensible evaluation (Nunns, 2016). Fourth, it would reveal that many
of us currently practicing evaluation have more to learn, particularly in
evaluation-specific reasoning and methods. The gap may feel threa-
tening and contrary to the inclusive and consensus-based nature of
competency efforts globally. Fifth, it would follow that those teaching
evaluation courses would have significant work to do revising their
curricula, which have focused on approaches and research methods.
This has time and cost implications, which are particularly significant
for higher education as it loses public funding internationally. In-
dividuals and professional development organisations may be more
agile and able to access expertise needed to respond quickly. Finally, it
points us to a significant gap in our research regarding values and va-
luing, particularly evaluation-specific reasoning and methods (Coryn
et al., 2017; Szanyi, Azzam, & Galen, 2012).
Agreement on a definition of evaluation has previously seemed
unlikely at best. Agreeing on this definition for our profession would
have significant implications for teaching and practice. However, it
meets the needs of (a) being specific enough to differentiate and
7
Evaluation and Program Planning 79 (2020) 101787
position the task of evaluation in comparison to other similar activities,
which is essential to claiming our space in the market; (b) being broad
enough to encompass the variety of relevant activities, which resolves a
variety of ongoing conflicts in the field; and (c) bounding, clarifying
and prioritising the key knowledge and skills required for those who
would engage in it, making it a specialist activity (Morell & Flaherty,
1978). If we choose to pursue professionalisation, these steps will pave
the way to certification, licensure and exclusion of unqualified persons
– an essential aspect of that pursuit (Worthen, 1994). It would enable us
to access the benefits that come with a clearly bounded profession:
power to control quality, leverage to define and keep a place in the
market, and leverage to create the kind of influence on public good that
evaluation has been claiming for years, but not achieving (Campbell,
1969; Cronbach, 1982; Pawson & Tilley, 2014).
Like the blind men of Indostan describing the elephant, academics
and practitioners have disputed about evaluation, arguing over
methods, values, use, and positivist, constructivist, and other view-
points. Ultimately, the result is a diffuse definition of evaluation that
consumes resources on internal debates that are not particularly of in-
terest outside our field, and that reduce our ability to create influence
on society because we lack a cohesive definition and framework for our
transdiscipline and, thus, cannot emerge fully as a profession. Without a
clear definition, we will continue to dispute loud and long while other
professions and evaluation commissioners actively determine, based on
their own perspectives, what evaluation is and the kind of information
it should provide for decision making (Crane, 1988; Lemire et al.,
2018). Evaluation will continue to sacrifice its seat at the table and its
potential to create the social change it purports to seek in favour of
internal debate. Instead, we could adopt the “fully describe, fully judge”
definition proposed herein, which covers the whole elephant. It clarifies
what makes evaluation a something, it provides a map for the various
activities we engage in related to the task of evaluation, points us in the
direction of what makes evaluation and evaluators good, and provides a
pathway for the research, education, and practices we need to get us to
good.
Funding
Funding for this article was provided by the Melbourne Graduate
School of Education under the conference travel grant scheme and the
Research Development Award [grant number 01-4600-06-xxxx-
000319-RDQ-12-01].
Acknowledgments
The author acknowledges Kate McKegg and Donna Podems for the
invitation to participate in an AEA Conference panel in 2014, which
was the impetus for early thinking on this article; Jean King who did the
presentation when I was unable to attend, and then mentored me in
moving it to publication; Ghislain Arbour for discussions that clarified
my thinking; Lori Wingate for the elephant metaphor associated with
evaluation; Mathea Roorda, Kelly Hannum and the reviewers for their
critical review and comments; and Arlen Gullickson, for recommending
Stake’s 1977 article to me at the start of my PhD studies.
Appendix A. Supplementary data
Supplementary material related to this article can be found, in the
online version, at doi:https://doi.org/10.1016/j.evalprogplan.2020.
101787.
A.M. Gullickson
References
AES Professional Learning Committee (2013). Evaluators’ professional learning competency
framework. Melbourne, Australia: Australasian Evaluation Society.
American Evaluation Association (2004). American evaluation association guiding principles
for evaluators.
American Evaluation Association (2018). AEA evaluator competencies.
Aoteroa New Zealand Evaluation Association (2011). Evaluator competencies. Retrieved
fromhttp://www.anzea.org.nz/wp-content/uploads/2013/05/110801_anzea_
evaluator_competencies_final.pdf.
Azzam, T. (2016). Ethical issues in evaluation practice: An empirical study. European
Evaluation Society Conference.
Bamberger, M., Rugh, J., & Mabry, L. (2012). RealWorld evaluation: Working under budget,
time, data, and political constraints (2nd). Thousand Oaks, Calif: SAGE.
Bloom, B. S., & Krathwohl, D. R. (1956). Taxonomy of educational objectives, the classifi-
cation of educational goals by a committee of college and university examiners. Handbook
I: Cognitive domain. New York: Longmans.
Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and teaching
evaluative thinking: Insights from research on critical thinking. American Journal of
Evaluation, 36(3), 375–388.
Campbell, D. T. (1969). Reforms as experiments. The American Psychologist, (4), 409.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for
research. Chicago, IL: Rand McNally College Publishing Company.
Canadian Evaluation Society (2010). Competencies for Canadian evaluation practice.
Retrieved fromhttps://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_
practice.pdf.
Carlis, J. V. (2013). Design: The key to writing a one-draft thesis. Self-publishedSelf-pub-
lished.
Chelimsky, E. (2013). Balancing evaluation theory and practice in the real world.
American Journal of Evaluation, 34(1), 91–98. https://doi.org/10.1177/
1098214012461559.
Chen, H. T., Gargani, J., Stead, B., & Norman, C. (2015). Program design business
meeting: Program design – Evaluation’s new frontier? American Evaluation Association
Annual Conference.
Coryn, C. L. S., Wilson, L. N., Westine, C. D., Hobson Kristin, A., Ozeki, S., Fiekowsky, E.
L., ... Schröter, D. C. (2017). A decade of research on evaluation: A systematic review
of research on evaluation published between 2005 and 2014. American Journal of
Evaluation, 38(3), 329–347.
Cram, F. (2018). Conclusion: Lessons about Indigenous evaluation. In F. Cram, K.A.
Tibbetts, & J. LaFrance (Eds.), Indigenous Evaluation. New Directions for Evaluation,
159, 121-133.
Cram, F., Tibbets, K.A., & LaFrance, J. (Eds.). (2018). Indigenous Evaluation. New Directions
for Evaluation, 159.
Crane, J. A. (1988). Evaluation as scientific research. Evaluation Review, 12(5), 467–482.
https://doi.org/10.1177/0193841X8801200501.
Creswell, J. W. (2003). Research design, qualitative, quantitative, and mixed methods ap-
proaches (2nd ed.). Thousand Oaks, CA: SAGE.
Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San
Francisco, CA: Jossey-Bass, Inc.
Dart, J., Webb, S., & Tolmer, Z. (2017). Stepping out: Evaluators working as designers.
Australasian Evaluation Society Annual Conference.
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evalua-
tion. Thousand Oaks, CA: Sage Publications.
Davies, R., & Mackay, K. (2014). Evaluator training: Content and topic valuation in
university evaluation courses. American Journal of Evaluation, 35(3), 419. https://doi.
org/10.1177/1098214013520066.
Dewey, J. D., Montrosse, B. E., Schröter, D. C., Sullins, C. D., & Mattox, J. R. (2008).
Evaluator competencies: What’s taught versus what’s sought. American Journal of
Evaluation, 29(3), 268–287. https://doi.org/10.1177/1098214008321152.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative
approaches and practical guidelines (4th). Upper Saddle River, N.J: Pearson Education.
Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between gen-
eral and working logic. New Directions for Program Evaluation, 1995(68), 15–32.
https://doi.org/10.1002/ev.1017.
Gluckman, P. (2013). The role of evidence in policy formation and implementation.
Greene, J. C. (2005). A Value-Engaged Approach for Evaluating the Bunche-Da Vinci
Learning Academy. In M. C. Alkin, & C. A. Christie (Vol. Eds.), Theorists Models in
Action New Directions for Evaluation: 2005, (pp. 27–45). Jossey-Bass.
Gullickson, A. M. (2018). Doing evaluation: Task analysis as a pathway to progress
evaluation education. Australasian Evaluation Society Conference.
House, E. R. (2014). Evaluation: Values, biases, and practical wisdom. Charlotte, NC:
Information Age Publishing.
Kegan, R., & Lahey, L. L. (2001). How the way we talk can change the way we work: Seven
languages for transformation. San Francisco: Jossey-Bass.
Kegan, R., & Lahey, L. L. (2009). Immunity to change: How to overcome it and unlock po-
tential in yourself and your organization (1st). Boston, MA: Harvard Business School.
Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of influ-
ence. New Directions for Evaluation, 2000(88), 5–23. https://doi.org/10.1002/ev.
1188.
LaVelle, J. M. (2014). In S. I. Donaldson (Ed.). An examination of evaluation education
programs and evaluator skills across the world. United States – California: The
Claremont Graduate University.
Lemire, S., Nielsen, S. B., & Christie, C. A. (2018). Toward understanding the evaluation
8
Evaluation and Program Planning 79 (2020) 101787
market and its industry—Advancing a research agenda. In S. B. Nielsen, S. Lemire, &
C. A. Christie (Vol. Eds.), The evaluation marketplace: Exploring the evaluation industry.
New directions for evaluation: Vol. 2018, (pp. 145–163). . https://doi.org/10.1002/ev.
20339.
Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation : An integrated framework for
understanding, guiding, and improving policies and programs. The Jossey-Bass nonprofit
and public management series. San Francisco: Jossey-Bass.
Mayne, J., & Rist, R. (2006). Studies are not enough: The necessary transformation of
evaluation. The Canadian Journal of Program Evaluation, 21(3), 93.
Meadows, D. H., & Wright, D. (2008). In D. Wright, & Sustainability Institute (Eds.).
Thinking in systems: A primer. White River Junction, VT: Chelsea Green Publishing.
Mertens, D. M. (1994). Training evaluators: Unique skills and knowledge. New Directions
for Program Evaluation, 62(Summer), 17–27.
Migotsky, C., Stake, R. E., Davis, R., Williams, B., DePaul, G., Cisneros, E. J., ... Feltovich,
J. (1997). Probative, dialectic, and moral reasoning in program evaluation.
Qualitative Inquiry, 3(4), 453–467. https://doi.org/10.1177/1098214006296430.
Morell, J. A., & Flaherty, E. W. (1978). The development of evaluation as a profession:
Current status and some predictions. Evaluation and Program Planning, 1(1), 11–17.
https://doi.org/10.1016/0149-7189(78)90003-4.
Morris, M. (2015). Research on evaluation ethics: Reflections and an agenda. In P. R.
Brandon (Vol. Ed.), Research on evaluation. New directions for evaluation, 148, 31–42:
Vol. 2015, (pp. 31–42). . https://doi.org/10.1002/ev.20155.
Newman, D. L. (1999). Education and training in evaluation ethics. New Directions for
Evaluation, 82, 67–76. https://doi.org/10.1002/ev.1138.
Newman, D. L., & Brown, R. D. (1996). Applied ethics for program evaluation. Thousand
Oaks: Sage Publications.
Nunns, H. (2016). The practice of evaluative reasoning in the Aotearoa New Zealand public
sector. Massey University.
Ozeki, S., Coryn, C. L. S., & Schröter, D. C. (2019). Evaluation logic in practice: Findings
from two empirical investigations of American Evaluation Association members.
Evaluation and Program Planning, 76(March), 101681. https://doi.org/10.1016/j.
evalprogplan.2019.101681.
Patton, M. Q. (2008). Utilization-focused evaluation. Thousand Oaks: Sage Publications.
Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance
innovation and use. New York: Guilford Press.
Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage.
Pawson, R., & Tilley, N. (2014). Realistic evaluation (2nd). London: Sage Publications, Ltd.
Petrie, H. G. (1995). Purpose, context, and synthesis: Can we avoid relativism? New
Directions for Evaluation, 1995(68), 81–91. https://doi.org/10.1002/ev.1021.
Poth, C., Lamarche, M. K., Yapp, A., Sulla, E., & Chisamore, C. (2014). Toward a defi-
nition of evaluation within the Canadian context: Who knew this would be so diffi-
cult? Canadian Journal of Program Evaluation, 29(1), 87–103. https://doi.org/10.
3138/cjpe.29.1.87.
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach
(7th). Thousand Oaks, CA: Sage.
Rowe, A. (2017). Evaluation for the Anthropocene. AES International Evaluation
Conference.
Saxe, J. G. (n.d.). The Blind Men and The Elephant. Retrieved from https://www.
allaboutphilosophy.org/blind-men-and-the-elephant.htm.
Schwandt, T. A. (2002). Evaluation practice reconsidered. New York: Peter Lang.
Schwandt, T. A. (2008). Educating for intelligent belief in evaluation. American Journal of
Evaluation, 29(2), 139–150. https://doi.org/10.1177/1098214006296430.
Scriven, M. (1967). The methodology of evaluation. In R.W., Tyler, R.M., Gagné & M.,
Scriven (Eds.) Perspectives of Curriculum Evaluation. AERA Monograph Series on
Evaluation. No. 1, pp. 39-83. Chicago: Rand McNally.
Scriven, M. (1991). Evaluation thesaurus (4th). Newbury Park, CA: Sage.
Scriven, M. (1993). The nature of evaluation. New Directions for Program Evaluation,
1993(58), 5–48. https://doi.org/10.1002/ev.1640.
Scriven, M. (1994). The final synthesis. American Journal of Evaluation, 15(3), 367.
https://doi.org/10.1177/109821409401500317.
Scriven, M. (2003). Evaluation in the new millennium: The transdisciplinary vision. In S.
I. Donaldson, & M. Scriven (Eds.). Evaluating social programs and problems: A vision for
the new millenium (pp. 19–41). Mahwah, N.J: Lawrence Erlbaum.
Scriven, M. (2005). Logic of evaluation. In S. Mathison (Ed.). Encyclopedia of evaluation
[electronic resource] (pp. 235–238). Thousand Oaks, CA; London: SAGE.
Scriven, M. (2007). The logic of evaluation. In H. V. Hansen (Vol. Ed.), Dissensus & the
Search for Common Ground: 138, (pp. 1–16). Windsor, ON: Ontario Society for the
Study of Argumentation.
Scriven, M. (2016). Roadblocks to recognition and revolution. American Journal of
Evaluation, 37(1), 27–44.
Shadish, W. R. J., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation:
Theories of practice. Newbury Park, CA: Sage Publications.
Smith, M. F. (1999). Should AEA begin a process for restricting membership in the pro-
fession of evaluation? American Journal of Evaluation, 20(3), 521. https://doi.org/10.
1177/109821409902000311.
Smith, N. L. (1995). The influence of societal games on the methodology of evaluative
inquiry. New Directions for Program Evaluation, 1995(68), 5–14. https://doi.org/10.
1002/ev.1016.
Smith, N. L. (1998). Professional reasons for declining an evaluation contract. American
Journal of Evaluation, 19(2), 177.
Stake, R. E. (1977). The countenance of educational evaluation. In A. A. Bellack, & H. M.
Kliebard (Eds.). Curriculum and evaluation (pp. 372–390). Berkeley, CA: McCutchan.
Stake, R. E., Migotsky, C., Davis, R., Cisneros, E., Depaul, G., Dunbar, C., ... Williams, B.
A.M. Gullickson
(1997). The evolving syntheses of program value. Evaluation Practice, 18(2), 89–103.
Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2004). Essential Competencies for
Program Evaluators Self-Assessment. Retrieved 31 January 2020, from https://www.
youtube.com/watch?v=8E0aZ387M_I.
Symonette, H., Mertens, D. M., & Hopson, R. (2014). The development of a diversity
initiative: Framework for the graduate education diversity internship (GEDI) pro-
gram. In P. M. Collins, & R. K. Hopson (Eds.). New directions for evaluation (pp. 9–
22). .
Szanyi, M., Azzam, T., & Galen, M. (2012). Research on evaluation: A needs assessment.
Canadian Journal of Program Evaluation, 27(1), 39–64.
Weiss, C. H. (1988). Reports on topic areas: Evaluation for decisions: Is anybody there?
Does anybody care? American Journal of Evaluation, 9(1), 5–19. https://doi.org/10.
1177/109821408800900101.
Worthen, B. R. (1994). Is evaluation a mature profession that warrants the preparation of
evaluation professionals? New Directions for Program Evaluation, 62(Summer), 3–15.
Yarbrough, D. B., Caruthers, F. A., Shulha, L. M., & Hopson, R. K. (2010). The program
evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand
Oaks, CA: Sage.
Evaluation and Program Planning 79 (2020) 101787
Amy is a Senior Lecturer at the University of Melbourne
Centre for Program Evaluation. Her evaluation and research
experience spans academia, health, education, business, not
for profit, international development, and faith commu-
nities. She has taught and developed more than 10 different
evaluation courses, she served as academic lead for the
University of Melbourne's first online graduate programs,
and helped to lead formation of the International Society
for Evaluation Education, which includes more than 90
people across 12 countries. Her current research projects
include evaluator education, evaluative synthesis and eva-
luation mainstreaming.