Chapter 6:

Program-Oriented
Approaches

Presentation by Jay Kerstetter, Amanda Brown,

Jody Yoos, & Jane Lightner

J.K
Orienting Questions:
#1 What are the key concepts of the objectives-oriented evaluation
approach?
#2 How has this approach influenced evaluation?

What is it?
 Objectives-oriented evaluation approach helps determine
whether some or all of the program objectives are
achieved and, if so, how well they are achieved.
Evaluators may work with stakeholders to establish if
program objectives are met.
Information from this approach can assist with deciding to
maintain, terminate, or change approaches within the
program.

J.K.
1. Ralph Tyler is credited with initiating this approach in the 1930’s

2. Tyler began to formulate his views on education and evaluation.

3. His approach included the following steps:

 Establish goals or objectives
Classify the goals or objectives
Define objectives in behavioral terms
Find situations in which achievement of objectives can be shown
Develop or select measurement techniques
Collect performance data
Compare performance data with behaviorally stated objectives

4. This approach was readily adoptable by evaluators and had great influence
on evaluation theorists.

Tylerian Evaluation Approach

J.K.
Provus’s
Discrepancy Evaluation Model:
1. Developed by Malcolm Provus; viewed evaluation as a continuous
information management process.
2. Provus stemmed key characteristics of his proposal from Tyler.
3. Provus viewed evaluation as a process.
4. This process was called DEM; Discrepancy Evaluation Model, which are
broken into four developmental stages.
◦ Definition
◦ Installation
◦ Process
◦ Product
◦ Cost-benefit analysis (optional)

 The DEM was designed to facilitate the development of programs in large

public school systems and later applied to state evaluations by federal
bureau.
The DEM was one of the earliest approaches to evaluation and elements of it
can still be found today in many evaluation.
J.K
Orienting Question #3:
How is the objective-oriented
evaluation approach used today?

Standards-based testing
Accountability in education
Performance monitoring systems used in many
government programs

Many refinements to the system since the

1930’s
J.Y.
 Ralph W. Tyler:
Tylerian Evaluation Approach

Influenced the Elementary and Secondary

Education Act (ESEA) of 1965, first act to
require evaluation of educational
programs
Started the National Assessment of
Educational Progress (NAEP), only way to
examine educational achievement of al 50
states.

J.Y.
Approach evaluated in Pittsburgh public
schools
Viewed evaluation as a continuous
information- management process
designed to serve as “the watch-dog of
program-management” and the
“handmaiden of administration in the
management of program development
through sound decision making”

Malcolm Provus: Provus’s Discrepancy

Evaluation Model
J.Y.
 Orienting Question #4a: How are
Logic Models used in evaluation?

Developed as an Program planners/evaluator

extension of Identify program
objectives- o Inputs
oriented o Activities
evaluation o Outputs (immediate
Designed to fill in
program impacts)
those steps o Outcomes (long-term
between the objectives/goals)
program and its
objectives
J.Y.
Today Logic Models are used in program
planning/evaluation
Help program staff articulate /discuss how program
might achieve goals
What elements are important to evaluate at any
given time
Build internal evaluation capacity or think in an
evaluative way

Example Companies:
United way of America
WK Kellogg Foundation
Annie E Casey Foundation

J.Y.
Theory-Based evaluation
Is used by evaluators to gain a better understanding of
the program
Better define the evaluation questions the study should
address
To aid their choices of what concepts to measure and
when to measure them
To improve their interpretation of results and their
feedback to stakeholders to enhance use.

Orienting Question #4b:

How are Program Theories used in
evaluation?
J.Y.
Steps in Theory Based Evaluation
1) Engage Relevant stakeholder
2) Develop a first draft of program theory
evaluator or evaluation team
3) Present draft to stakeholders for further
discussion, reaction, and input
4) Conduct a plausibility check
5) Communicate findings to key
stakeholders
6) Probe arrows for model specificity
7) Finalize program impact theory

J.Y.
Theory Driven Evaluation

Work with stakeholders to identify key questions to be answered

in the evaluation and the appropriate designs and methods for
answering those questions
Emphasis is on testing the program model
Provide guidance as to what to measure and when to measure it
The selection of the evaluation questions to be addressed depend
on the stage of the program and what stakeholders hope to learn
Provides evaluator with critical information that can be used
throughout the evaluation

J.Y.
Orienting Question #5: How do theory-based
evaluation and objectives approaches differ?
Objectives Approach: Theory-Based

1. The objectives are 1. The evaluator discusses

goals, basics, and objectives
identified by group
of program w/the
looking for the stakeholders.
evaluation. 2. The evaluator decides how
2. Purpose of activity the program should work
and then sees if it works
specified and then
that way.
those purposes are 3. Huey Chen and Leonard
being acheived. Bickman helped to develop
3. Individual credited- approaches to theory based.
Ralph W. Tyler 4. Science based and
quantitative.
A.B.
Question #6: What are the strengths and
limitations of program oriented
evaluation approaches?

Strengths:
Objective oriented – the simplicity of the
concept and program makes it easy to
understand, follow and possibly
implement.
◦ Face validity – the evaluator is being held
accountable for what is being asked to be
evaluated. They want to see if what is currently
happening or working based on their own goals
and objectives for the program.
A.B.
Theory oriented – the chance for the
evaluator to partake in dialogue with the
stakeholders is a strength of the program
because it helps them to expand their
knowledge on the topic.
◦ This then leads to the evaluator to have a clear
understanding of the program so they know
how to properly evaluate the program.

Strengths (con’t)
A.B.
Weaknesses:

Objective oriented – the evaluator can have a

single minded focus on the objectives which will
cause them to overlook the complications,
elements and factors contributing to that
programs success or failure.
a. This approach does not ask for the evaluator to gain
an understanding of the context in which the
program operates this could then be what is affecting
the program’s success or failure.
b. Evaluator could ignore the actual value of the
objective. Since the evaluator is told what the
objectives are they are not asked to evaluate
whether that objective even fits the program.

A.B.
Theory oriented – this like the objective
oriented can cause the evaluator to ignore
certain aspects of the program that are
important because the evaluator is so focused
on the theory of how it runs and not how it is
actually running.
a. Evaluators may ignore the needs or values of
stakeholders involved with the program.
b. This approach may also oversimplify the complexity
of the program making it feel easier to evaluate
than it really is because not all of the surrounding
factors are accounted for in the process.

Weaknesses (con’t)
A.B.
Orienting Question # 7: What is
goal-free evaluation?

Rationale: “Goals should not be taken as

givens.”
Developed by Scriven (1972)---believes
the most important function of a goal-free
evaluation is to reduce the bias that occurs
from knowing program goals
Thus…increase objectivity in judging the
program as a whole.

J.L.
Orienting Question #8: What does it
teach us about conducting an evaluation?

Goals can act as “blinders” causing us to

possibly miss the most important outcomes
not related to the goals.
Goal-free evaluation was proposed to
primarily indentify the unanticipated side
effects that an objectives-oriented
evaluation might miss

J.L.
 Major Characteristics of a Goal-
Free Evaluation

2. Predetermined 3. Goal-free
1. Evaluator
goals are not evaluation focuses
purposely avoids
permitted o narrow on actual outcomes
becoming aware of
the focus of the rather than intended
program goals.
evaluation study. program outcomes.

4. The goal-free 5. Goal-free

evaluator has evaluation increases
minimal contact with the likelihood that
the program unanticipated side
manager and staff. effects will be noted.

J.L.
 External Goal-Free
Internal Goal Evaluator vs.
Evaluator
 Goal-directed  What does the
 How well is program program do?
meeting goals  Looking at ALL the
 Provide information programs
to administrator outcomes,
intended or not

Goal Directed + Goal-Free=

can work together
J.L.
 Information taken from
Program Evaluation:
Alternate Approaches and
Practical Guidelines
p. 153-171