Planning An Evaluation

2012/9/20
Planning an Evaluation
PSY 811 Program Evaluation
Evaluation Theory
Evaluation can mean many different things to different people Determine what model/theory defines the stakeholders view Determine what model/theory defines your stance
Evaluation Theory
Evaluation theory is NOT
Program theory Theory-based evaluation
Evaluation theory outlines specific models or approaches to evaluation and outlines why certain evaluation practices lead to particular kinds of results across situations
1
Print to PDF without this message by purchasing novaPDF (http://www.novapdf.com/)
2012/9/20
Evaluation theory is who we are

All evaluators should know evaluation theory because it is central to our professional identity. It is what we talk about more than anything else, it seems to give rise to our most trenchant debates, it gives us the language we use for talking to ourselves and others, and perhaps most important, it is what makes us different from other professions (W.R. Shadish)
Evaluation Theory Should

1.
2.
3.
Clarify the activities, processes, and goals of evaluation Explicate the relations among evaluative activities and the processes and goals they facilitate Empirically test propositions to identify and address those that conflict with research about evaluation
Evaluation Theory
Functional Perspective 5 components

Social Programming Knowledge Construction Valuing
Use
Practice
2
2012/9/20
Evaluation Theories/Models
Traditional Model
Impressionistic Implicit or explicit bias may have influence Often serve to maintain the status quo
Social Science Research Model
Examine program success via controlled

experimentation
Introduced rigor and objectivity Inherent limitations for applied settings
Industrial Inspection Model
Inspecting a product at the end of the production line Very inefficient and costly Does not provide timely information for correcting problems Example - one exam at the end of 4 years in university
3
2012/9/20
Black Box Evaluation Model
Examines the output of a program without examining its internal operations or activities Good for consumer evaluations of products Does not provide information on processes of change Does not provide information relevant to program improvement
Objectives Based Evaluation Model
Most prevalent model Focus on measuring concrete goals and objectives A rigid focus neglects:
Why the program is succeeding/failing Unanticipated effects Whether the stated goals are appropriate
Goal-Free Evaluation Model
Evaluators work most effectively if they do not

know the stated goals
Allows them to examine and present data on

all observable outcomes without bias
Very time consuming, very expensive and

often quite threatening to stakeholders
4
2012/9/20
Fiscal Evaluation Model
Calculations of the financial investment (input)

and the return on that investment
Cost-Benefit Analysis Hard to place a price on many psycho-social

variables
Accountability Model
Program audit Compliance with mandates and regulations Are very individualized
Expert Opinion Model
Experts evaluate the program Often used when the program is large,
complex and unique - qualitative impressions will enhance the results
Accreditation
5
2012/9/20
Qualitative Model
Naturalistic evaluation Deep and thorough understanding of:

the program the clients the social environment
Success Case Method
Seeks detailed information from those who

have benefitted most from the program
Provides clues as to what works best Caution can result in targeting program to
those most likely to succeed
Empowerment Evaluation
Stakeholders actively participate in the

evaluation and gain skills from the process
Enables them to become:

More competent in dealing with social problems More competent in dealing with social agencies More informed citizens More self-confident and self-directed
6
2012/9/20
Theory Driven Evaluation
Focuses on examining the relations between:
The services and the participants The services and the immediate changes The immediate changes and the outcome variables
Improvement Focused Approach
Focuses on discovering discrepancies:
The program objectives and the target population need The plans and the implementation Target population expectations and services delivered Outcomes expected and outcomes achieved
Utilization-Focused Evaluation Emphasis on utility and actual use Focus is on intended use by intended users
7
2012/9/20
Developmental Evaluation
A form of evaluation designed to support program adaptation to emergent and dynamic realities from a complex systems perspective
Stakeholder Relations
Many preconceived notions regarding evaluations Important to dispel myths and address attitudes likely to impede evaluation efforts
Dysfunctional Attitudes
My program is perfect!
People think they are better than average,

even though on average that cannot be!
Dramatic program effects rarely occur Assist stakeholders in estimating a realistic or

practical amount of improvement
8
2012/9/20
Fear that the evaluation will offend staff
Examining services creates an atmosphere of

distrust
Asking for client feedback threatening to

professionals
Evaluation inhibits innovation
Evaluation does not limit day to day flexibility

in social programs
Major restructuring should be curtailed Need to strike a balance between research

and service needs
Fear the program will be terminated
Unfavorable results most often result in

refinement rather than termination
Involve stakeholders from beginning Circulate draft to managers for their input
9
2012/9/20
Fear that the information will be misused
Confusion between program evaluation and

personal performance appraisal
Evaluator must make distinction clear (to

everyone!!)
Qualitative understanding will be supplanted
Service providers trust their own conclusions

based on observation and experience
Include both qualitative and quantitative to get full

picture
Evaluations drain program resources
Takes money away from service provision Usually a required budget line Educate how it can improve service delivery
10
2012/9/20
Fear of losing control of the program
Staff and managers fear losing decision-making

power
Work closely with stakeholders Address their concerns and questions within the
evaluation
Evaluation has no impact
Often true that evaluation results are filed Can raise the odds of utilization by:
Designing well constructed evaluation plan Address issues of importance to stakeholders Present findings in a manner conducive to use
Vital for Planning Evaluations
Consulting Collaborating Coordinating Creativity Contracting
11

Planning An Evaluation

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Planning An Evaluation

Uploaded by

Copyright:

Available Formats

2012/9/20

PSY 811 Program Evaluation

Evaluation theory is NOT

Program theory Theory-based evaluation

Evaluation theory is who we are

Evaluation Theory Should

Functional Perspective 5 components

Social Science Research Model

Examine program success via controlled

Introduced rigor and objectivity Inherent limitations for applied settings

Industrial Inspection Model

Black Box Evaluation Model

Objectives Based Evaluation Model

Goal-Free Evaluation Model

Evaluators work most effectively if they do not

Allows them to examine and present data on

Very time consuming, very expensive and

Fiscal Evaluation Model

Calculations of the financial investment (input)

Cost-Benefit Analysis Hard to place a price on many psycho-social

Expert Opinion Model

Naturalistic evaluation Deep and thorough understanding of:

Success Case Method

Seeks detailed information from those who

Stakeholders actively participate in the

Enables them to become:

Theory Driven Evaluation

Focuses on examining the relations between:

Improvement Focused Approach

Focuses on discovering discrepancies:

People think they are better than average,

Dramatic program effects rarely occur Assist stakeholders in estimating a realistic or

Fear that the evaluation will offend staff

Examining services creates an atmosphere of

Asking for client feedback threatening to

Evaluation inhibits innovation

Evaluation does not limit day to day flexibility

Major restructuring should be curtailed Need to strike a balance between research

Fear the program will be terminated

Unfavorable results most often result in

Fear that the information will be misused

Confusion between program evaluation and

Evaluator must make distinction clear (to

Qualitative understanding will be supplanted

Service providers trust their own conclusions

Include both qualitative and quantitative to get full

Evaluations drain program resources

Fear of losing control of the program

Staff and managers fear losing decision-making

Evaluation has no impact

Vital for Planning Evaluations

Consulting Collaborating Coordinating Creativity Contracting

You might also like