History of Evaluation

Emmanuel Tembo (University of Zambia)

Introduction

Decision-makers and stakeholders in various institutions; Non-governmental organisations, private

and public institutions, want to ensure that programs that are put in place, largely to improve the
wellbeing of society are accomplishing their intended purpose. That is to say, they are interested in
assessing the effects of programs especially considering the fact that large sums of resources are
put it. To make this assessment, program evaluation is employed. Against this background, the
purpose of this paper is; to discuss the history of the development of theory-based evaluation.

To aid the discussion, the paper discusses a history of the development of theory-based evaluation
and conclusions are drawn at the end.

1. History of the Development of Theory Based Evaluation and its influence on Evaluation
today

This section of the assignment focuses on the history of the development of theory based evaluation
and its influence on evaluation today. It needs to be noted on the onset that the history of theory-
based evaluation is discussed in line with the general history of program evaluation within which
it lies.

1.1 Evolution of program evaluation as evaluation research

Evaluation is by most often considered a recent occurrence. However, literature shows that
evaluation is a very old practice. Madaus et al., (2000) “have traced the development of program
evaluation to the period 1792-1900’s when it is argued that the first formal use of evaluation took
place”. During this period, evaluation of programs traces its roots in the field education where
efforts to address the problems of illiteracy, poor student performance and determining teachers’
salaries. The term ‘Evaluation’ was actually coined in 1930 as seen in the writings of Gargani
(2003) who says, “Tyler coined the term evaluation around 1930 in order to differentiate his work

1
at Ohio State University from what he considered to be the standard practice of testing” (Gargani,
2003: 19).

Following the end of World War II onwards, program evaluation began to gain prominence as the
assessment of programs went beyond education; programs were launched to rebuild, as such
evaluation included programs in education, public health, housing, etc. All in all, evaluation also
began to include other sectors of society and human development. The significant phase in the
development of program evaluation however was in the 1960’s, in the United States of America.
This was during the presidencies of John F. Kennedy and Lyndon B. Johnson. During this period,
the government of the United States initiated various social programs that received substantial
funds. Due to this development, it became apparent that the programs had to be assessed so that
the resources allocated could be justified.

Rossi, Freeman and Lipsey (2004) have defined program evaluation as “the use of social research
procedures to systematically investigate the effectiveness of social intervention program in ways
that are adapted to their political and organizational environment and are designed to inform social
action to improve social conditions” (Rossi et al, 2004:16). Implicit in this definition is the fact that
program evaluation applies social research methods and in there lies its methodological history.

During the 1960’s, not only did programs receive a lot of funds, and had to be evaluated, a number
of books and articles on evaluation were also written during the period. It was during this period
that the first book on evaluation was written by Edward Suchman in 1967 (titled Evaluation
Research). It should be noted that the title of Suchman’s book demonstrates the view of evaluation
as a form of research. Prior to the book by Suchman, another book was written by Donald Campbell
and Stanley in 1963 (titled Experimental and Quasi-experimental designs for research). In their
book, Campbell and Stanley expounded on the shortcoming of using experimental designs in social
research and also put forward the quasi-experimental design. In their writings, what is emphasised
is the importance of rigorous scientific methods in the conduct of good quality studies. It is on the
works of Campbell and Stanley that Suchman built his arguments. In his writings, Suchman
strongly observes the importance of conducting evaluative research in scientific ways in addition,
Suchman also implores that an evaluative research must also recognize administrative criteria for
determining the value for money of doing the study.

1.2 Theory-based evaluation and its influence on evaluation today

2
Because of his work, theory-based evaluation is in most cases traced back to Edward Suchman and
the Experimental and Quasi-experimental designs for research. The development of theory-based
evaluation however is mostly argued to have begun in 1972 when some researchers met in
Cambridge to explore how evaluation practice founded on the experimental model could be
revised. It was observed in the conference that the experimental design-oriented evaluation practice
did not pay attention to program processes and did not provide useful information regarding the
problems inherent in the programs under investigation thus resulting in minimal information for
decision making processes on the program. Following the 1972 meeting for researchers, Carol
Weiss published an article in which she proposed the use of the program model in undertaking an
evaluation study. It is argued that hers is the earliest reference to what is now termed theory-based
evaluation. According to Hudson, the prepositions in the article by Weiss takes care of the concerns
raised at the Cambridge conference as can be seen below:
“The program model presented in the article incorporates program processes, as
intended, constructed through the identification of the steps and mechanisms by which
a program is proposed to operate. This evaluation approach was similar to that
advocated by the Cambridge Conference manifesto in that it supported the
consideration of complex program processes in an evaluative undertaking, yet it was
dissimilar to a majority of the program evaluations conducted” (Hudson, 2008: 17)

During the same period, in contributing to the debate, Fitz-Gibbon and Morris (1975) also
advocated the use of theory-based evaluation, coined the term, and put forward a description of
theory based evaluation as stated: “A theory-based evaluation of a program is one in which the
selection of program features to evaluate is determined by an explicit conceptualization of the
program in terms of a theory which attempts to explain how the program produces the desired
effects” (Fitz-Gibbon et al, 1975). Theory in this sense does not really mean a ‘theory’ as
understood in social sciences but the logic of the program; consisting statements that describe the
program, under what conditions the program effects occur, forecast the outcomes of the program,
and stipulate the requirements necessary to generate the anticipated program effects. All in all
therefore, the writings highlighted, especially by Weiss urge evaluators to use the program’s
assumptions or logic as the framework for evaluation.

Although most scholars trace theory-based evaluation to Suchman in 1967 and Weiss and Fitz-
Gibbon in the 1970s as highlighted above, Gargani (2003) argues that the thoughts that underlie

3
theory-based evaluation go back further than 1967. In his writing titled, ‘A Historical Review of
Theory Based Evaluation’, Gargani actually traces its roots to the 1890s and writes:

“The term theory-based evaluation appears to have been coined in 1975, so there are
no references to TBE per se prior to that date. Under other names, however, the
evaluation and education literature has for almost 100 years featured the
recommendation that evaluators and researchers articulate and test the causal
assumptions underlying programs”.

From the above, it can be construed that it is yet to be established and agreed exactly when theory-
based evaluation was born, but with the available literature, we would safely credit Weiss and Fitz-
Gibbon and Morris for the development of the approach.

The discussion on the history of the development of theory-based evaluation however is incomplete
without a discussion on the period 1980 to 1990s, a period during which theory-based evaluation
gained momentum. In the early 1980s, many evaluation scholars became dissatisfied with
evaluation studies where the outputs of programs were investigated without regard to their
underlying causal mechanisms (Black Box evaluation). Notable among the proponents were Chen
and Rossi, who, in 1981 began to formulate their own method for conducting evaluation studies.
Like other scholars of the time, Chen and Rossi argued that black-box evaluation studies were of
little value to program planners and managers whose intention was to maximize the effectiveness
of programs.

In 1990, Chen wrote a book, ‘Theory-Driven Evaluations’, in which he is said to have “introduced
a comprehensive framework for evaluation that used program theory as its primary organizing
principle” (Gargani, 2003: 35). It is worth noting that program theory has been defined earlier in
the discussion; however, it is from the writings of Chen that it is clearly shown how this approach
works. It works in such a way that an evaluator starts an evaluation by enunciating a program theory
that links program activities to outcomes. This program theory is constructed using two approaches,
either a stakeholder or a social science approach. In a stakeholder approach, the evaluator works in
collaboration with stakeholders to bring to fore their beliefs and articulate those beliefs as a
comprehensible theory. In the social science approach, evaluators use their understanding of how
the program is operating and their knowledge of existing social science theory to build the program
theory.

4
Following Chen’s book in 1990, theory driven evaluation boomed and we are seeing its influence
in evaluation today. Currently theory-driven evaluation which “is sometimes referred to as
program-theory evaluation, theory-based evaluation, theory-guided evaluation, theory-of-action,
theory-of-change, program logic, logical frameworks, outcomes hierarchies, realist or realistic
evaluation and, more recently, program theory-driven evaluation science, among many others”
(Coryn et al, 2011: 200) has gained eminence and a lot of studies by individuals and organisations
have been conducted and commissioned using the approach.

Some of the organisations that have conducted and commissioned evaluation studies using such
approaches include: “W.K. Kellogg Foundation; the United Way of America; centres for Diseases
Control and Prevention (CDC); Overseas Development Institute (ODI); the International Initiative
for Impact Evaluation Group (3ie); the United Nations Evaluation Group (UNEG) and the World
Bank” (Coryn et al, 2011). Some of the prominent individual evaluators who “acknowledge the
usefulness of program theory in the design of program evaluations include Bickman, Chen,
Funnell, Pawson and Tilley, Rodgers, Weiss, and Worthen” (Hudson, 2008: 22). In the year 2000,
Birkmayer and Weiss (2000) published an article in which they examined six published evaluation
studies all of which take a theory based approach. The key reason why theory-based evaluations
have gained prominence is because theory-based evaluation helps the evaluator tell if the program
is working; how it is working; and why or why not it is working; by following each stage advanced
by the theory.

1.3 Conclusion
In conclusion, it suffices to say that despite its growing popularity, theory-based evaluation has
shortcomings that have been acknowledged even by the proponents of the approach. Some of the
notable shortcomings include: the programme theory is unclear; the articulated theory is not the
only possible theory; they are expensive looking at the time needed to construct the programme
theory; and lack of technical knowhow by evaluators to develop theories. A notable critic has been
Michael Scriven who has maintained that an evaluation whose aim is to determine the value of a
program does not need to address a theory linking the program components or the internal
operations of the program.

5
REFERENCES

Johanna D. Birckmayer and Carol Hirschon Weiss (2000). Theory-Based Evaluation in Practice :
What Do We Learn? Evaluation Review, 24: 407

Chris L. S. Coryn, Lindsay A. Noakes, Carl D. Westine, and Daniela C. Schroter (2011). A
Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009. American Journal
of Evaluation 32(2) 199-226

Carol Taylor Fitz-Gibbon and Lynn Lyons Morris (1975). Theory based Evaluation. The Journal
of Educational Evaluation, vol 5, Number 1.

John Gargani (2003). RUNNING HEAD: A History of Theory-Based Evaluation. University of

California, Berkeley. 2625 Alcatraz Avenue Number 508 Berkeley, CA 94705

Sheri L. Hudson (2008). The influence of program theory-based evaluation on the use f
evaluation information: A longitudinal study. Murdoch University,

Rossi, P.H., Lipsey, M.W., & Freeman, H.E., (2004). EVALUATION: A systematic Approach (7th
ed). California: Sage Publications.

George F. Madaus, Daniel L. Stufflebeam, and Thomas Kellaghan (2000). Evaluation Models:
Viewpoints on Educational and Human Services Evaluation.Kluwer Academic publishers,
Boston.