THEORY OF EVALUATION (ToE): A practitioner's guide to constructing a 'theory of evaluation' and developing evaluative knowledge products

— Developed and designed by Khalil Bitar (Willy Brandt School of Public Policy, University of Erfurt)
Evaluative Activity per ToE Phase
Domain Scope
1. Conceptualization 2. Proposal 3. Inception 4. Implementation
1 Context and background Evaluand Explore evaluation ToR and Reflect on understanding and ask Examine and assess available Analyze and relate to evaluand
evaluand documentation (if clarifying questions if any information and data processes and results
available) to understand
2 Scope, rationales, objectives, Evaluation Analyze to understand Reflect on understanding and ask Examine and discuss with Maintain and keep in mind
and potential use and users clarifying questions if any evaluation manager/ client and during all evaluation activities
other potential users
3 Logic model/ Program theory/ Evaluand Interpret (if available) / Reflect on understanding (and Examine and assess (and Examine in detail and relate to
Theory of change (ToC) Conceptualize (if unavailable) interrogate if needed) reconstruct, if needed) evaluation questions and
findings
4 Criteria Evaluation Examine in detail Reflect on understanding, ask Interpret, explore strategies to Maintain and collect relevant
questions, and suggest changes (if tackle, and connect across data and evidence per criterion
any) criteria and with evaluation
questions
5 Questions Evaluation Examine in detail Reflect on understanding, ask Interpret, explore strategies to Answer supported by data and
questions, and suggest changes (if answer, and connect across evaluative evidence
any) questions and with evaluation
criteria
6 Evaluability Evaluand Check in relation to evaluation Ask clarifying questions if any, Investigate and determine areas Validate and only evaluate
criteria and questions and including about data availability of weak/ uncertain evaluability evaluable aspects
before proceeding to the next
steps
7 Stakeholders Evaluand/ Analyze and map (if possible, at Connect to evaluand processes Analyze and map systematically Analyze and reflect on power
Evaluation this stage) to understand and results (and evaluation and strategize engagement dynamics (if possible),
objectives and potential use/ connections, and influence of
users) the various stakeholders
8 Approach Evaluation Take stock of and assess Determine an approach for the Examine if the approach is Adopt and apply, and modify if
potential, appropriate evaluation, explain, justify, and appropriate and useful needed
approaches identify limitations
9 Matrix Evaluation Conceptualize Point out the conceptualized Complete matrix and probe its Employ with questions, sub-
matrix, esp. in relation to appropriateness questions, data sources, and
methodology tools
10 Methodology Evaluation Conceptualize Suggest, explain, justify, identify Test adequacy and Employ and adjust if needed,
limitations, and offer alternatives appropriateness with justifications

Page | 1 | v. 1.22 | DOI: 10.13140/RG.2.2.14274.66240/6

 Evaluative Activity per ToE Phase
Domain Scope
1. Conceptualization 2. Proposal 3. Inception 4. Implementation
11 Data collection tools Evaluation Deliberate options for tools Suggest, explain, justify, identify Draft tools and test them to Employ and adjust if needed,
considering evaluation limitations, and offer alternatives check their appropriateness and with justifications
approach, methodology, adequacy
stakeholders, data gaps, and
ethical considerations
12 Stakeholder engagement, Evaluation Consider in relation to Propose a plan for engagement/ Engage different stakeholders at Ensure meaningful involvement
participation, and role evaluation approach and participation, and ask questions if this inception stage (not only of all stakeholders and engage
methodology (and objectives needed the client) them in all evaluation
and potential use/ users) processes where possible
13 Methodological challenges Evaluation Consider in relation to Acknowledge and explain, and Check, control, and improve Address and mitigate following
and limitations evaluation approach, include potential mitigation mitigation strategies where specified strategies, and
methodology, and questions strategies possible declare where not possible
14 Risks (incl. security risks) Evaluation Consider in relation to evaluand Identify and offer ways to Specify and test, and explain in Be aware of and mitigate
and evaluation contexts mitigate them detail how risks will be following the specified
mitigated strategies
15 Competency Evaluator Assess in relation to evaluand Explain in relation to evaluand Check if appropriate and Be transparent about areas of
and evaluation nature and and evaluation nature and needs adequate for all aspects and lack of competency at any
needs processes of the evaluation stage of the evaluation
16 Background/ background and Evaluator Consider in relation to evaluand Explain and relate to evaluand Check if appropriate for all Consider in all aspects and
team composition and evaluation nature and and evaluation nature and needs aspects and processes of the processes of the evaluation
needs evaluation
17 Quality assessment and Evaluation Consider your/your team's Clarify quality assessment and Reflect on quality assessment Employ throughout the
assurance quality assessment and assurance strategies and and assurance strategies and evaluation process and for final
assurance capability and measures measures, and recognize any products
resources gaps/ shortcomings
18 Ethical considerations Evaluation Determine, for all aspects and Introduce and explain how ethical Expand and enhance ethical Address at all stages and
processes of the evaluation considerations will be addressed consideration strategies and observe as planned. Document
throughout the evaluation planned actions challenges/ limitations and
disclose to evaluation
manager/ client and in report
19 Equity, social justice, and Evaluation Consider for all aspects and Reflect on initial understanding of Analyze power dynamics and Interrogate power dynamics
power dynamics processes of the evaluation power dynamics and explain the plan to tackle equity and social and address equity and justice
considerations plan to address equity and social justice issues and aspects issues and aspects
justice issues and aspects systematically systematically in the evaluation
20 Cultural considerations Evaluation Consider for all aspects and Reflect on initial understanding of Analyze cultural aspects and Be aware of cultural contexts
processes of the evaluation cultural aspects and issues and issues systematically and address cultural
your plan to deal with them considerations systematically
during the evaluation and sensitively

Page | 2 | v. 1.22 | DOI: 10.13140/RG.2.2.14274.66240/6

 Evaluative Activity per ToE Phase
Domain Scope
1. Conceptualization 2. Proposal 3. Inception 4. Implementation
21 Roles and responsibilities Evaluator Think about the roles and Reflect on understanding and ask Adjust if needed, in agreement Document and maintain, and
(Evaluator, Evaluation responsibilities of each party, questions if any with the evaluation manager/ adjust (with evaluation
Manager, and other considering what is specified in client and other stakeholders manager/ client agreement) if
stakeholders) evaluation ToR if any needed
22 Communication arrangements Evaluation Think about for each stage of Reflect on understanding and ask Adjust if needed, in agreement Document and maintain, and
the evaluation, considering as questions if any with the evaluation manager/ adjust (with evaluation
defined in evaluation ToR if any client and other stakeholders manager/ client agreement) if
needed
23 Required support/ information Evaluator Analyze initial information gaps Identify and describe potential Detail support and information Document, observe, and follow
from evaluation manager/ (if possible at this stage) and support and information needed needed per evaluation stage progress per evaluation stage
client and other stakeholders potential support needed and stakeholder
24 Questions to the evaluation Evaluator Start compiling questions Ask questions if any/ where Make sure all questions are Document answers and ask
manager and other throughout the possible before submitting a addressed by the evaluation further questions at any
stakeholders conceptualization phase proposal manager/ client and other evaluation stage if needed
relevant stakeholders
25 Suggestions to evaluation Evaluator Start compiling potential Include primary suggestions, e.g., Communicate all suggestions in Document evaluation manager/
manager/ client suggestions if any about evaluation criteria, detail, document responses, client responses, and make
questions, approach, and and observe agreements with further suggestions if needed/
methodology the evaluation manager/ client helpful
26 Availability Evaluator Consider for all stages of the State and declare if unavailable at Ensure availability after any Commit to declared availability
evaluation, including field data any stage(s) of the evaluation or changes during the inception unless due to forcible reasons,
collection if part of the could be unavailable due to any phase, e.g., due to changes in and—if so—inform evaluation
evaluation probable reason data collection schedule manager/ client ASAP
27 Phases, activities, and timeline Evaluation Plan considering methodology, Propose, and provide more than Finalize timeline with specific Commit to agreed plan unless
deliverables deadlines, and roles one scenario if possible activities and outputs, and due to forcible reasons, and—if
and responsibilities of determine the agreed plan so—inform evaluation
evaluator(s) and client manager/ client ASAP
28 Data collection Evaluation Consider options of sound data Specify and explain potential data Collect initial data (if possible) Triangulate, collect with ethical
collection strategies, and collection strategies, and relate to and finalize data collection considerations in mind, cross-
consider relevant risks ethical considerations strategies and overall plan examine, clean, and verify
29 Data synthesis Evaluation Start thinking of options for Elaborate on data synthesis Start documenting any patterns Bring together collected data
bringing data together, strategies and plan identified across results from through the various tools and
considering data collection/ initial data collected start extracting findings and
analysis instruments and plan drawing conclusions
30 Data analysis Evaluation Start thinking of options for Elaborate on data analysis plan Start analyzing patterns Extract findings and draw
logical/ statistical techniques for and techniques, and include identified across results from conclusions per evaluation
analysis options if needed, considering initial data collected criteria and question
data availability

Page | 3 | v. 1.22 | DOI: 10.13140/RG.2.2.14274.66240/6


Domain Scope
1. Conceptualization
31 Evidence Evaluation Deliberate the type of evidence
needed considering evaluation
criteria, questions, approach,
and methodology
32 Findings, lessons, and Evaluation Contemplate in relation to
conclusions evaluation rationales,
objectives, and potential use, as
well as evaluation matrix
33 Recommendations Evaluation Contemplate in relation to
evaluation rationales,
objectives, and potential use, as
well as evaluation matrix
34 Deliverables Evaluation Plan based on evaluation ToR
but considering evaluation
approach, methodology, and
data collection and analysis
35 Privacy and confidentiality Evaluator/ Think of potential strategies and
Evaluation measures to protect the privacy
of all evaluation stakeholders
and confidentiality of data
36 Impartiality and independence Evaluator/ Think of ways to ensure
Evaluation impartiality and independence
of evaluator(s) and evaluation
processes
37 Conflict of interest Evaluator/ Examine if there is any
Evaluation (potential) conflict of interest
before embarking on any step of
the evaluative processes
38 Budget Evaluator/ Contemplate, considering
Evaluation planned evaluation activities
and consider logistical costs if
any (e.g., travel,
accommodation, data collection
costs)
Page | 4 | v. 1.22 | DOI: 10.13140/RG.2.2.14274.66240/6
Evaluative Activity per ToE Phase
2. Proposal
Clarify the type of evidence
needed, how it will be derived,
and how to ensure credibility and
validity
Explain how findings, lessons, and
conclusions will be derived and
how will they relate to evaluation
objectives and potential use
Ask clarifying questions about
areas of most useful
recommendations
Reflect on understanding of
deliverables and their relevant
submission deadlines, and ask
questions if needed
Explain strategies to protect all
evaluation participant’s privacy
and strategies to maintain
confidentiality in data storage,
handling, and sharing
Clarify evaluator(s’) impartiality
and independence and how they
will be maintained throughout the
evaluation. State any threat(s)
State lack of conflict of interest
clearly and mention potential
threat(s) if any
Detail per evaluation activity and
team member, and include other
costs (travel, accommodation,
data collection, etc.). Propose
separately in a financial proposal
if required in the ToR
3. Inception
Check if potential evidence is
adequate, credible, valid, and
help form valid conclusions and
value judgments
Start analyzing, and document
any patterns identified across
results from initial data
collected
Start collating initial
recommendations
Check if it is possible to deliver
on time, adjust deadlines if
needed (with evaluation
manager/ client agreement)
Put a clear and detailed plan
concerning privacy and
confidentiality, including data
collection, storage, handling,
and sharing
Assure evaluator(s’) impartiality
and independence and if
anything changed about them
since submitting the proposal
Assure if needed and maintain
throughout the evaluation, and
clarify if anything changed since
submitting the proposal
Detail and adjust if needed, and
justify changes if any
4. Implementation
Derive systematically from
collected data and validate
using different information and
data collection sources
Articulate and present based
on collected data and evidence,
corresponding to each
evaluation criteria/ questions
Articulate and present based
on collected data and evidence,
and suggest timing, priority,
justification, and responsibility
Develop and submit per plan.
Use client’s preferred format/
template if any
Protect stakeholders’ private
data throughout the evaluation
and maintain confidentiality as
agreed with each stakeholder
Maintain impartiality and
independence throughout the
evaluation and notify
evaluation manager/ client in
case of changes
Maintain throughout the
evaluation and notify the
evaluation manager/ client if
anything changes
Consider throughout the
evaluation and per agreement
with evaluation manager/ client