PARTICIPANT-ORIENTED EVALUATION APPROACHES EDF5461 TT ae. Ea Jenna Oster Oma ad CeDEFINITION =Participant-Oriented Evaluation Approaches "currently include many different models, but their commonality is that they all use stakeholders- people with an interest or "stake" in the program- to assist in conducting the evaluation." (Fitzpatrick 189)KEY CONCEPTS Involves participants in the program, managers, staff, and other key stakeholders (anyone involved with planning and implementation of the evaluation) as a key principle to good evaluation Involve stakeholders to teach evaluators about program Founded in Constructivism (humans gain knowledge and meaning from interaction and experience)DATA COLLECTION METHODS =In Situ Observation =Observing stakeholders in their interaction with the program =Interviews =Talking to staff, managers and stakeholders to learn more about the program =Documents =Using program documents to help with the evaluationeS Pre-1967 1967 Mid-1970s Present No desire to A push to bring Observations Jean King: include the human and identifying “overarching human elements concerns and term that invetves, 5 Tike program staff elements and/or issues of sctival in and/or qualities into stakeholders decision making” qualities within the evaluation >) were examined >| the evaluation process: make and became Cousins and Earl: process people/ enwound with cee ene stakeholders a the P-PE poet partnership part of the evaluation Detoen Sisiuater evaluation and stakeholders” process 1975; Responsive New methods of evaluation participatory evaluation continue to emergeMAJOR DEVELOPERS = Robert Stake: One of the first advocates for participant- orientated evaluation « The Countenance of Educational Evaluation (1967) * Allows for a wide range of data to be collected * A congruence between the intention and what is observed = This model looks for continuity between what happens and what is intended « Countenance Framework (1967) Intended Observed = Countenance Matrix. ————> eongrnce i ‘ Trtended Antecedents, <—> Observed Antecedents » Responsive Evaluation Provides qualitative evidence of a eognence programs effectiveness Intended Transactions. <——> Observed Transactions = Includes stakeholders in the implementation of policies and goals 4 | fegcalcnnngeney ———_anpircatontingeney A Leica cantngerey enpivcolcontingsrey ¥ eorgrancs Intended Outcomes, ¢——> Observed OutcomesMAJOR DEVELOPERS (ore aL TD) = Egon Gruba and Yvonna Lincoln * Naturalistic Inquiry (4985) ® Collects qualitative data to interpret and understand unique situations where there is a level of uncertainty ® Reliability and Vall ty: “Trustwort Credibility Transferable Dependable Conformability * Naturalistic Evaluation = Responsive Evaluation + Naturalistic Methodologies = Naturalistic Evaluation * Evaluators participate in the evaluation and try to uncover the needs of the all members » Three phases 1. Familiarizing Phase 2. Action Phase 3. Synthesis Phase * Fourth-Generation Evaluation (1989) = Uses constructionist paradigm to gather, analyze, and organize evaluative material based on the needs of stakeholders = Never-ending processAPPROACHES Participatory evaluation: “An overarching term for any evaluation approach that involves program staff or participants actively in decision making and other activities related to planning and implementation of evaluation studies.” (p. 199) Two main types of participatory evaluation: i a he, ee Practical Participant Evaluation Transformative Participant is used for practical reasons, Evaluation means just that, to limiting the evaluation to a transform and empower the particular organization. stakeholders (social change).CATEGORIES OF PARTICIPATORY APPROACHES =There are 3 dimensions in which participatory approaches differ, according to Cousins, Donohue, and Bloom (Fitzpatrick 200): =1) Control over the evaluation or technical decision- making process =2) Stakeholder selection =3) Depth of ParticipationAPPROACHES (CONTINUED) Di ee Geieae DD Practical Participatory Evaluation (P-PE) Transformative Participatory Evaluation (T-PE) Empowerment Evaluation Democratically Oriented Evaluation + Explicit focus on evaluation decisions + Evaluators work closely with primary stakeholders + Stakeholder control + Many stakeholders + Emphasis on program participants + Much involvement * Stakeholder control + Much involvement * Evaluator control + Many stakeholders Limited involvement Formative decisions Participants are oppressed Social justice is a concern Participant empowerment is the goal Need for empowerment to build staff Need for building internal Self-monitoring and program improvement Need for a dialogue among stakeholdersSTRENGTHS OF PARTICIPANT-ORIENTED EVALUATION APPROACHES = When stakeholders are involved in evaluations, they... =... provide data sources, distribute surveys, and set up focus groups *... are familiar with program context and environment ... establish validity of the study because they possess/incorporate knowledge to inform decisions "... Can communicate program situation/details to facilitate evaluator’s understanding = Including stakeholders in evaluations... =... gains their trust in the evaluator and the evaluation process leads to better chances of stakeholder use/adoption =... improves stakeholders’ understanding of program/organizationLIMITATIONS OF PARTICIPANT-ORIENTED EVALUATION APPROACHES = Competence/feasibility « Is the evaluator truly capable of acting as facilitator, coordinator, and consultant during the evaluation process? « Are stakeholders able to perform tasks required of them? = Credibility » Because stakeholders are deeply invested in the program, will they act to change the program based on the evaluation results? = Political = Due to involvement of people (primary stakeholders, managers, staff) = Time-consuming and expensiveAAA = Evaluators include stakeholders in evaluation planning and implementation = Stake and Gruba and Lincoln responded in the 1970s and 1980s to established methods of evaluation Wanted approaches offering consideration of program stakeholders = Two main branches of participant-oriented approaches Practical participatory evaluation (P-PE): encourage involvement/ownership, thus increasing use of results by stakeholders Transformative participatory evaluation (T-PE): empower stakeholders by direct involvement in evaluation process = Three primary dimensions of participatory approaches Evaluator or stakeholders possess control Individual stakeholder selection Extent of stakeholder involvementaaa (ore Ba TD) = Practical participatory (P-PE) approaches often rely on = Stakeholder involvement * Practical participatory evaluation = Developmental evaluation = Transformative approaches often rely on «= Empowerment evaluation = Deliberative democratic evaluation = Many definitions for participative evaluation exist = All involve stakeholders, but no additional details are standard