See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/11573504

The Nature and Methods of Learning by Doing

Article  in  American Psychologist · December 2001

DOI: 10.1037/0003-066X.56.11.964 · Source: PubMed

CITATIONS READS
29 688

1 author:

Alan M. Lesgold
University of Pittsburgh
86 PUBLICATIONS   2,836 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Intelligent Tutor Development View project

All content following this page was uploaded by Alan M. Lesgold on 02 February 2016.

The user has requested enhancement of the downloaded file.

 Nature and Methods of Learning by Doing 1

THE NATURE AND METHODS

OF LEARNING BY DOING

Alan M. Lesgold

University of Pittsburgh

Lesgold, A. (2001). The nature and methods of learning by doing. American Psychologist, 56(11), 964-973.

CONTACT:

Alan M. Lesgold
Dean, School of Education
University of Pittsburgh
5T01 WWPH
230 S Bouquet St
Pittsburgh, PA 15260
Phone: 412-648-1773
FAX: 412-648-1825
AL@pitt.edu
Nature and Methods of Learning by Doing 2

Abstract
This presentation reviews some of the psychological research on learning by
doing and discusses the role that learning-by-doing approaches can play in
education and training. It includes a discussion of the author’s implementations
of this approach and the lessons learned from these implementations. Improved
forms of learning by doing now can be supported by information technologies,
and there are prospects for extensions to group learning by doing and group
learning from examples in the near future.

Table of Contents
Abstract................................................................................................................................. 2
Experiences Building Systems to Promote Learning by Doing ................................. 5
The Sherlock Years...................................................................................................... 6
Sherlock 2: Transfer...................................................................................................11
The Intel Experience.................................................................................................13
Learning from Conversations .........................................................................................18
Final Comment..................................................................................................................19
References...........................................................................................................................20
Figures .................................................................................................................................24
Brief Biography of Alan Lesgold....................................................................................28
Selected Bibliography of Alan Lesgold, Chronologically Ordered ..........................31
Nature and Methods of Learning by Doing 3

In the history of education, the most striking phenomenon is that schools of

learning, which at one epoch are alive with a ferment of genius, in a succeeding
generation exhibit merely pedantry and routine. The reason is, that they are
overladen with inert ideas. Education with inert ideas is not only useless: it is,
above all things, harmful…. Let the main ideas which are introduced into a
child's education be few and important, and let them be thrown into every
combination possible. The child should make them his own, and should
understand their application here and now in the circumstances of his actual
life. (Whitehead, 1929, P. 14)

Whitehead observed that we learn from experience, abstract or codify that

experience, and then deal only in the codifications when we teach the next
generation. His views on the problems with such an approach have driven much
of my work for the last couple decades. On the one hand, we see signs in the
academy that starting from abstractions won’t work. For example, it is not
unusual for a doctoral program in mathematics to culminate in a few courses
labeled with titles like “Introduction to Algebra.” Mathematicians know that the
abstractions they use to organize their knowledge cannot be learned except after
substantial mathematics experience.

On the other hand, we also see remarkable examples of the problems Whitehead
raised in his essay. Schooling in the United States is all too often a “mile wide
and an inch deep.” Fragmentary topics are taught, quickly memorized, tested, and
then forgotten. Also, training courses all too often consist of “theory” first and
brief practical experience afterwards. Further, any university course that provides
practical experience solving complex problems is pejoratively labeled as “training”
and not “education.” My own psychology department subscribes to the view that
the right preparation of an applied cognitive psychologist is training in basic
Nature and Methods of Learning by Doing 4

research, which provides a “foundation” upon which later practical efforts can be
grounded. This is defensible if the foundation is grounded in experience actually
doing basic research, though experience doing applied research would broaden
the experiential base for conceptual understanding.

While I believe that Whitehead’s view is well supported by psychological research

findings, the issue is not completely settled. In particular, it is not yet clear
whether or not certain kinds of learning goals are best achieved with an earlier
emphasis on abstracted and formally codified knowledge. However, it is quite
clear that there is a role for learning by doing and that it is sometimes much more
effective than traditional schooling approaches, especially when the goal is for a
person to be able to use the acquired knowledge to attack complex, incompletely
structured, and novel problems. Such problems are a major component of
technical and professional work in our information economy.

My work in the past two decades has been driven increasingly by the belief that
more learning by doing is needed in most education and training situations.
Psychologists’ aversion to learning by doing as the fundamental form of learning
is driven in part by an incomplete understanding of some basic psychological
principles, especially those relating to transfer. The dominant theory of transfer
for the past century has been the “identical elements” view put forward by
Thorndike (Thorndike & Woodworth, 1901). Further, extensions to the
Thorndike view to account for transfer have tended to see abstraction as the key
to transfer (e.g., Laird, Newell, & Rosenbloom, 1987).

An alternative view has arisen in the computational study of case-based reasoning

(see Carbonell, 1986; Kolodner, 1993; Leake, 1996). While there is a lot of
complexity in the research on case-based reasoning, the fundamental method has
three basic steps, situation assessment, retrieval of a relevant case deemed most
Nature and Methods of Learning by Doing 5

likely to be useful, and adaptation of the solution in that case to fit the new
circumstances. From the point of view of case-based reasoning researchers, then,
transfer comes not from identical “elements” but rather from experience with
cases that, as a group, cover the range of situations likely to arise in a given
domain. While case-based reasoning approaches have proven very useful in the
world of machine cognition, there is less explicit testing of such approaches in
humans (but see Kolodner, 1994; and Schank, 1995). Nonetheless, the success of
case-based reasoning in artificial intelligence suggests that approaches based upon
case-based reasoning principles should be effective, something colleagues and I
set out to test beginning in about 1984.

The fundamental thought we had in mind was that if tough tasks in the real world
are handled best via case-based reasoning, then we ought to consider what
knowledge is needed to become a good case-based reasoner. Certainly, one needs
experience carrying out the three steps of situation assessment, retrieval of
relevant cases, and adaptation of past solutions. In addition, though, one needs a
mental library of cases, a collection rich enough to come close to most situations
in which we expect the student to be able to perform later. With these two kinds
of training, high levels of domain expertise should become possible. Much of my
work for about ten years was directed at practical implementations of this idea.

Experiences Building Systems to Promote Learning by Doing

When it really matters, our culture follows the learning by doing scheme pretty
well. Consider the coaching of football. People learn to play football by playing
football. There are drills on fragments of football play, but they represent
focused learning by doing on components that are themselves complex problems.
Further, detailed coaching consists in large part of helping players notice the
situations they are in and map appropriate cases onto them. So, for example,
modern football coaches videotape all practices and games and index the
Nature and Methods of Learning by Doing 6

individual plays so that they can present a player with a collection of recent game
experiences and help the player sort out the features that determine which of
several cases – or abstractions of cases – is the best guide to a given situation. Put
another way, football is learned by experiencing a large number of cases and
receiving detailed advice on how to do the situation assessment and adaptation
needed to apply old cases to new situations.

Certainly, there is abstraction. That is what we hear when we listen to a radio

broadcast of a football game. The terms used are terms relevant to situation
assessment and adaptation. But, they get their meaning from our experience
either playing football or watching it. Few people learn how to comprehend
football games by reading a book on the theory of football, and fewer still actually
learn to play the game that way.

We have a view of this kind of learning that becomes less charitable when we step
back and abstract it. We then call it “the school of hard knocks” or “sink and
swim.” In reality, all of the application of psychology to the design of learning by
doing is in assuring that no one sinks and in developing coaching sufficient to
assure that the learner not only doesn’t sink but also refines situation awareness
and adaptation capabilities as much as possible given the set of case experiences
he or she has.

The Sherlock Years

As these ideas about how learning by doing should work began to take shape in
my thinking, the United States Air Force, in the form of Dr. Sherrie Gott,
provided me the opportunity to learn by doing myself. Sherrie energetically
pushed the command structure to support development of an intelligent tutor to
help technicians learn a very difficult job, the repair of the elaborate test stations
that are used to maintain aviation electronic equipment in the field. Over a period
Nature and Methods of Learning by Doing 7

of more than ten years, Sherrie Gott and I, and teams of great colleagues in both
the University of Pittsburgh and the Air Force, worked at developing two
iterations of a system we called Sherlock (Gott & Lesgold, 2000) the name was
meant to characterize the complex problem solving we were teaching, though
during a few rough periods we wondered whether the 7% solution might not
have been a better path to follow).

It was not uncommon during the initial years of our work for people to insist that
intelligent support of learning by doing was beyond the potential of artificial
intelligence. Indeed, I used to joke that my colleagues in computer science wrote
dissertations proving that what we were doing with Sherlock was theoretically
impossible. Actually, if we had followed the standard paradigm for intelligent
tutoring systems, it would have been!

The standard paradigm for an intelligent tutoring system is to develop a scheme

for modeling the student’s knowledge based upon his or her performance and
then providing specific advice that fills in knowledge gaps. The most striking
example of successfully using such a strategy is the family of mathematics tutors
developed by John Anderson, Ken Koedinger, and Al Corbett at Carnegie
Mellon University (Anderson et al., 1995; Corbett, Koedinger, & Anderson,
1997). Their tutors work by attempting to match the performance of the student
using a set of rules that represent both successful and unsuccessful strategies for
solving problems in mathematics. When the system knows which rules are being
exhibited by the student and which are not, it is in a position to provide
information about additional rules that the student needs to learn.

When one shifts from the relatively simple and constrained world of academic
course problems to real world problems, the student modeling approach cannot
always work. As the number of possible ways of addressing a problems and the
Nature and Methods of Learning by Doing 8

number of collections of knowledge from which a solution can be forged both

rise, there is a combinatorial explosion in the amount of rule matching that is
needed and in the additional work needed to figure out which rules should be
taught or coached next. Not only must a system model the student’s
performance, it also must figure out which categories of domain knowledge and
strategies the student is trying to apply, lest it suddenly offer advice totally
unrelated to the student’s current thinking.

We solved this problem in Sherlock by abandoning the basic approach of student

modeling. Rather, we invested the machine intelligence in understanding the
current problem solution situation rather than the student’s mind. Instead of
modeling which bits of knowledge the student had and had not acquired, we
modeled what was known from the actions the student had performed so far on
a given problem and what kind of strategy an expert might use to proceed from
that point.

Because we used the expert model of the domain – in the case of Sherlock this
was a particular piece of complex equipment – to organize the representation of
the device, many aspects of Sherlock worked together to help teach situation
assessment and the adaptation of solutions learned from earlier problems.
Students gathered information to solve equipment maintenance problems via an
interface that was itself organized the way experts organize the system.
Specifically, a division was made between switching circuitry and the effects
produced with the circuitry switched to a particular configuration, and a further
division was made among major categories of switched configuration. Every
time the student needed to ask for information, he or she was exposed to a
typography of system component types that was helpful to the situation
assessment process.
Nature and Methods of Learning by Doing 9

Further, whenever a student asked for advice that was best conveyed via a circuit
diagram, that diagram was configured to expose how an expert would be
conceiving the apparatus given the actions the student had already taken.
Whatever the expert would be thinking about took up more space on the screen
and was presented in more detail – exactly the detail that would figure in the
relevant expert situation assessment at that instant in the course of problem
solution. The need for a complex interface to permit access to many different
kinds of information was turned into an opportunity to superimpose lessons in
problem solving strategy on top of specific case experiences. And, of course, this
was “sink-or-swim” without the possibility of sinking. The student could always
keep asking for information that was progressively more explicit in telling what to
do next toward a problem solution.

Overall, we had built a system that embodied the idea of learning from cases with
coaching about situation assessment and case adaptation. It provided partial
simulations of the domain in which knowledge was to be used, structuring that
partial simulation to reflect expert knowledge that helped in situation assessment.
It had an expert model that drove its coaching capabilities, but it did not do
student modeling. Rather, it modeled the current problem situation as it was
transformed by actions the student took. This permitted more effective situation-
specific coaching and was computationally tractable.

At another level, though, we had to do a lot of analysis and design work that was
more fundamentally psychological rather than computational in character.
Specifically, we needed to build a family of progressively more expert models of
expertise in the domain, which in turn required a substantial amount of cognitive
task analysis. This aspect of the work, perhaps more than any other, benefited
greatly from the insights and activity of my colleague Sherrie Gott. Sherrie
pioneered a structured interview strategy that was extremely helpful in doing the
Nature and Methods of Learning by Doing 10

needed cognitive task analysis (Gott, 1987; Gott, Bennett, & Gillet, 1986; Means
& Gott, 1988). This scheme made it easy to reconstruct the entire process of
expert solution to a complex problem, including the way in which the expert
represented the problem situation and assessed it, the kinds of cases it reminded
him or her of, and the ways in which schemes anchored in other cases were
adapted to produce a solution plan.

So far, I have not addressed two basic issues in applying learning by doing
strategies, namely the selection and sequencing of problems and the potential for
transfer of learning beyond the immediate scope of the problems set on which
learning occurs. The problem sequencing issue was addressed in our first
Sherlock effort and was based upon Gott’s task analysis scheme. Specifically, it
was possible to derive, from the cognitive task analysis, a progression of cases
that required progressively more of the solution strategies experts bring to their
work and that built from problems requiring simple application of a strategy to
those requiring iterative or recursive application of multiple solution strategies.

The results of the first Sherlock venture were quite heartening. It was clear that
airmen receiving the learning-by-doing training from Sherlock improved their
problem solving capabilities dramatically (see Lesgold et al., 1992). Indeed, a
crude and at least partly reasonable account equates the amount of improvement
in complex problem solving ability produced by about 20 hours of Sherlock
training as equivalent to that produced by about four years of on-the-job
experience.

This figure, while only partly defensible, raises an important point. If simple on-
the-job experience were all that was needed to become a knowledgeable expert,
then on-the-job learning should be relatively efficient as a source of expertise. If,
in fact, it takes years of on-the-job experience to produce results as good as those
Nature and Methods of Learning by Doing 11

produced by a modest set of simulated and coached experiences, then we need to

understand what the special stuff of the coached learning might be. There are
multiple candidates. First, as we have noted, it isn’t just experience with cases but
rather experience with a sequenced set of cases for which coaching is available
and for which every aspect of the interface supports learning situation assessment
and representation rules that can leverage real work experience. Second, we note
that Sherlock presented a concentration of useful cases in a brief period of time.
The real world mostly provides opportunities to do the routine. Expertise
involving the non-routine is harder to get from everyday work experience,
because the right situations occur rarely, and often they are handled by
established experts when they do occur, not by students. In medical education,
we concentrate hard cases into teaching hospitals in order to provide a good case
mix, and we use grand rounds to assure that situation assessment and solution
adaptation are made explicit and salient to students. Different domains will
differentially benefit from real-world versus simulation approaches to this.

Sherlock 2: Transfer
There were a number of technical problems surrounding the first Sherlock. It ran
on very exotic equipment, and it was very difficult to figure out what it really cost,
since it evolved out of a research project on cognitive task analysis. More
important, though, we did not have a good model for transfer nor did we have
any tests to see if Sherlock produced transfer. So, like many a film maker faced
with a successful product, we convinced our sponsors to let us produce a sequel,
in which we focused our evaluation on transfer issues.

The results have been reported elsewhere in some detail (Gott & Lesgold, 2000).
We added some tools that permitted students to compare their solutions with
those of experts and to get more explanation of differences between their
approach and a more expert approach. Figure 1 shows one simple example, a
Nature and Methods of Learning by Doing 12

check list by which the student could evaluate his or her performance, after which
the system would display an expert evaluation and explain the reasons for
suggesting room for improvement on particular issues.

Figure 1 about here

Basically, this kind of approach amounts to telling about the principles for
situation assessment and adaptation of strategies but anchoring all of this telling
explicitly to the case just presented. The learning by doing is supplemented by
explication, one way or another, of the experience one has just had. It turns out
that this sort of approach works remarkably well, when measured in terms of
subsequent capability to actually do complex problem solving. Gott and Lesgold
(2000) tested the adequacy of both Sherlock and Sherlock 2 using what Gott
called a verbal troubleshooting test.

In such a test, a problem is posed, and the student is asked to state each
successive action he or she would take. The experimenter gives the result of the
action,1 and the student then goes on to the next action, eventually stating his
diagnosis for the machine problem that was posed. To test for transfer, a new
machine was designed that could be diagnosed using the same principles but that
had different kinds of functions as part of its design, along with some differences
in electronics and basic operations. We called this mythical system the
Frankenstation. It had the same family resemblance to the domain of original
training as several actual machines, but by using an imaginary machine, we could
control for any prior experience that a few students might have had with other
real machines.

The basic results were that students trained on Sherlock 2 diagnosed both the
domain machine and the Frankenstation much better than controls and almost as
Nature and Methods of Learning by Doing 13

well as senior experts. So, we conclude that the approaches used in Sherlock and
Sherlock 2 are effective and do produce transfer (see Gott & Lesgold, 2000, for
details).

The Intel Experience

After we finished the Air Force work, an opportunity arose to work jointly with
Intel Corporation to extend the learning by doing approach, which we then called
intelligent coached apprenticeship, to use in training technicians who maintain
equipment used to make computer chips. This allowed us to learn how to make
learning by doing as economically efficient as possible and also to extend some of
the transfer-related affordances we experimented with in the Sherlock 2 effort. I
learned a lot from this experience, in many different ways.

By being part of a joint working team, partly at the University of Pittsburgh and
partly at Intel, I got to experience some of the best of modern business
organization and practice. The team included people ranging from technicians
with work experience but no more than two years of formal post-secondary
education to people with doctorates. On any given day, the real planning and
management of the project was led by whoever had the most useful information
for the next steps. Technicians sometimes modified the project Gantt charts and
working plan just like managers did, if they happened to have relevant
information. The project was also remarkably free of individual ego, with
everyone working toward shared goals. If a team member had a personal or
family emergency, the work went on without interruption, with others filling in
the missing contributions of whoever was temporarily hors de combat. My sense
today of the new demands of the work world on our education system is driven
substantially by the experiences I had with Intel and the universality of certain
skill requirements in the work team, regardless of level of professional or
technical education.
Nature and Methods of Learning by Doing 14

The Air Force differs from the business world in some important ways. Most
important, they make decisions to be cost-effective, but without the added
discipline of profit structures for their work. Some defense investments are so
important, that we have to spend the money, regardless of how much it is. To be
useful for the rest of the economy, an approach needs to be cost-effective within
the scope of normal business operations, and it has to have a cost that is
consistent with the decision making structure of everyday businesses.

The first issue, general cost-effectiveness, is quite straightforward. Sherlock cost

about $2.5 million, and Sherlock 2 cost about $1 million. We did two more
iterations of learning by doing technology with Intel, and they did a fifth iteration
on their own. While the exact cost figures are proprietary, I can report that the
pattern of each generation costing less than 50% of its predecessor held up, more
or less, across the entire series of implementations. The current cost for a learning
by doing system produced by a practiced team is in five figures, and the return on
investment for this type of training is huge, because the domains involved include
problems that sometimes take hours or days to be solved and that hold up
significant chunks of product manufacturing lines.

However, there are two types of problems we have not yet overcome. First,
businesses do not have reliable, replicated models of the learning curve for
entering the learning by doing business. They can see the six or seven figure
initial cost for the first effort, but are not confident that the second or third effort
will be clearly practical. Until we in the psychology world do enough work of this
kind to clarify the learning curve, companies will see a move into learning by
doing as too risky.

A second problem has to do with the structure of organizations and who within
the organization has to take the risk. It is quite feasible to think of building
Nature and Methods of Learning by Doing 15

intelligent systems for learning by doing for under $100,000 and for those
systems to produce returns on investment in the millions of dollars. However,
training departments often operate with average budgets for a new “course” that
are in the low five figure range. It is an act of courage for a training manager to
spend two to three times as much money on an approach he or she is just starting
to understand and with which there has been little departmental experience than
is spent on projects that the department knows it can do well (by current
standards). Either, we need to introduce significant training on developing
learning by doing systems into the programs that produce training managers, or
we will need to get the costs of such systems down even lower.

The work with Intel also provided additional opportunities to refine our
approach to training for transfer, largely because of one of those serendipitous
experiences Skinner used to talk about. When Intel first raised the prospect of
building an intelligent coached apprenticeship system, they considered several
possible jobs for which they needed more training in complex problem solving.
Eventually, they ended up developing the training for an ion deposition system,
one that puts layers on chips. In our second effort, we ended up using an ion
beam implant machine, one that writes circuit components on chips. Amazingly,
it turned out that a physicist interested in science education who had been
working with me on some youth apprenticeship approaches, happened to hold
patents from other domains on both ion deposition and ion beam implant
processes. This person, Martin Nahemow, was the main source of a transfer-
producing approach we called the Process Explorer (see Lesgold and Nahemow,
2001 for details).

The core of Nahemow’s thinking was that there are both categories of problem
manifestations and categories of system failures that index much of the
knowledge an expert accumulates about complex problem solving in a domain,
Nature and Methods of Learning by Doing 16

and that when a particular manifestation occurs, this is a perfect opportunity to

learn more about the ways in which a certain kind of system failure leads to a
certain kind of symptom. Computationally, this basic idea can readily be turned
into a learning affordance for an intelligent learning by doing system.

The first task is to produce the equivalent of a matrix in which the rows are
system failure types and the columns are types of manifested failures. This would
be a very big matrix, so big in fact that we would never store the whole thing,
since most of the cells would be empty. Displaying such a matrix to a student
would be worse than useless – it would be unnecessarily confusing. However,
only a small piece of the matrix is relevant to any given problem, and that is
exactly the piece that can be a useful source of transfer-related knowledge that is
anchored in the problem itself. Here’s how it works.

Suppose that a particular problem that is presented to a student involves a

manifestation like “sandy grit is being deposited on chip wafers during the
manufacturing process.” One could then select all of the system failures that can
produce grit. This defines the rows of a submatrix that could be useful. We then
select all the manifestations that (a) can be produced by the subset of system
failures, and (b) can be explained with a simple functional relationship. These
define the columns of the submatrix. With appropriate criteria for relevance, one
generally ends up with a matrix of perhaps half a dozen rows and roughly as
many columns, which is quite tractable. Since most manifestations in a
manufacturing process are a deviation, upward or downward, from a quality
standard (statistical process control drives most productive manufacturing
processes, and the manifestations of system failure are usually deviations of
quality parameters), the cells in such a matrix can have markers like ↓↓ or ↑ to
indicate the direction and magnitude of deviation. Figure 2 shows an example of
the Process Explorer as presented to a student, and Figure 3 shows the
Nature and Methods of Learning by Doing 17

explanation of the relationship selected by the student in Figure 2 (the cell with
black and white reversed).

Figures 2 and 3 about here

The Process Explorer also solves a major problem in technical education. A

machine like those used at Intel requires many areas of knowledge to be
understood. As educators, we often approach such situations by listing all the
domains of knowledge that are needed and then just telling the student to take
courses in all that stuff. It does not make sense to tell a student with at most two
years of post high school education that he needs courses in physics, physical
chemistry, chemical engineering, electrical engineering, and quantum mechanics
before he has met the prerequisite for job-specific training. Nor is it possible to
replace all this disciplinary knowledge with a set of rote-learned rules. The
dilemma we face is that technical people need real scientific knowledge to
underpin their work skills but they cannot spend years acquiring it. The learning
by doing approach offers a way to select bite-sized pieces of this knowledge and
make them available in the specific contexts for which they are needed (see also
Bonar, 1986).

Through vehicles like the Process Explorer, it is quite possible to insert relevant
conceptual presentations and explanations into learning by doing, and it can be
done dynamically and under control of the learner. This is one way – and I am
sure there are others – of realizing the idea of learning by doing and the idea of
learner-controlled learning in a disciplined and powerful way. We know that this
approach works. It has been evaluated (see Lesgold & Nahemow, 2001). What
remains to be established is the specific range of learning contexts for which it
works – we only looked at complex problem solving in technical domains.
Nature and Methods of Learning by Doing 18

Learning from Conversations

In the past few years, my work has moved in a somewhat different direction,
about which I comment only briefly, because the work is not yet finished, and it
is not certain exactly how it will play out. While many opportunities for learning
are grounded in one’s personal and immediate work experiences, others can be
grounded in situations that one might study. Just as the good student studies
worked out examples in a physics book (Chi et al., 1989), it is possible for
students to have conversations about what they observe in a situation. An
example of this colleagues and I currently are exploring is discussion among
teachers about an example of teaching they have just observed.

While our theoretical work is independent, Dan Suthers and I have been working
with a team including Daniel Jones, Amy Soller, Megan Hall and Lauren Resnick
to develop web-based conversational environments in which teachers can learn
by discussing specific video examples of teaching. The system being produced
will be the primary learning environment for the Institute for Learning, a
University of Pittsburgh effort in professional development of teachers and
school leaders. The trick, when learning by doing is not guided by a problem that
the learner is trying to solve, is to guide the observations of cases and their
discussion among learners. This will be done in two ways. First, there are rubrics
that shape the discussions, and often these rubrics are first used in an in-person
discussion facilitated by instructional experts. Then, after a structured
introduction, students can discuss the cases they have observed. This discussion,
when it takes place on a web site, can, in principle, be coached by an intelligent
system, though much work is still needed to determine how such systems might
work.

My colleague Dan Suthers (2001a, 2001b), now at the University of Hawaii, has
been addressing a number of important questions about collaborative learning
Nature and Methods of Learning by Doing 19

over web sites, including how to diagrammatically represent an ongoing

discussion in ways that assure shared meaning among the discussion’s
participants, how to help discussions “point” to multimedia sources, and how to
help groups of learners keep track of all that is being discussed so they can
integrate and structure what they have learned.

Taking a somewhat different approach, my doctoral student Amy Soller is trying

to determine how an intelligent system could recognize, from the sequence of
speech acts in a discussion, whether learning is proceeding effectively and
whether any one of the participants seems less likely to really be learning (cf.
Soller, in press; Soller & Lesgold, 2000). So far, she has had some success
applying a hidden Markov modeling scheme to the sequence of speech acts in an
online discussion. When one takes a set of network conversations and classifies
them according to whether the discussants successfully learned to carry out a task
together, it turns out that one can train a hidden Markov model2 that is quite
successful at distinguishing the success of additional conversations not used to
train the model. The next step is to see whether collections of conversations
manifesting particular coachable problems can be distinguished using this
approach.

Final Comment
I have had unbelievably good fortune in my career – good mentors, good
students, good collaborators – and all have contributed to whatever I may have
achieved. I hope that is clear from this brief presentation. There is another
message that I hope I have conveyed. This is the potential for a field of cognitive
engineering that can be helpful to our society and to the world. However, we will
not have such a field unless we encourage our students to acquire enough
mathematics, science, and formal skills to be able to work with and seek out the
complementary expertises they will need to be true cognitive engineers. Not all
Nature and Methods of Learning by Doing 20

psychology students will aspire to an engineering career, but given the range of
knowledge needed in many areas of psychological research, none will be hurt by
being encouraged to seek strong mathematical, formal-symbolic, and scientific
background as part of their initial liberal educations. Had I not had teachers,
family, and friends who embodied this advice, I would not have been able to do
the work that APA decided to honor.

References
Anderson, J. R., Corbett, A. T., Koedinger, K., & Pelletier, R. (1995). Cognitive
tutors: Lessons learned. The Journal of Learning Sciences, 4 (2), 167-207.

Bonar, J., R. Cunningham, and J. Schultz, (1986). An Object-Oriented

Architecture for Intelligent Tutoring. Proceedings of the ACM Conference
on Object-Oriented Programming Systems, Language and Applications. New
York: ACM.

Carbonell, J. (1986) Derivational analogy: a theory of reconstructive problem

solving and expertise acquisition. In R.S. Michalski, J.G. Carbonell,
T.M. Mitchell (eds.): Machine Learning - An Artificial Intelligence
Approach, Vol.II, Morgan Kaufmann, pp. 371-392.

Chi M.T.H., Bassok, M., Lewis, M.W., Reimann, P. & Glaser, R. (1989) Self-
Explanations: How Students Study and Use Examples in Learning to
Solve Problems. Cognitive Science, 13, 145-182.

Corbett, A. T., Koedinger, K. R., & Anderson, J. R. (1997). Intelligent tutoring

systems. In Helander, M. G., Landauer, T. K., & Prabhu, P. V.
(Eds.), Handbook of Human-Computer Interaction, (pp. 849-874).
Amsterdam, The Netherlands: Elsevier Science B. V.
Nature and Methods of Learning by Doing 21

Gott, S. P. (1987). Assessing technical expertise in today’s work environments.

Proceedings of the 1987 ETS Invitational Conference (pp. 89-101).
Princeton, NJ: Educational Testing Service.

Gott, S. P., Bennett, W., & Gillet, A. (1986). Models of technical competence for
intelligent tutoring systems. Journal of Computer-Based Instruction. 13, 43-
46.

Gott, S. P., & Lesgold, A. M. (2000). Competence in the Workplace: How

Cognitive Performance Models and Situated Instruction Can
Accelerate Skill Acquisition. In R. Glaser (Ed.), Advances in
instructional psychology. Hillsdale, NJ: Erlbaum.

Kolodner, J. (1993). Case-Based Reasoning. San Mateo, CA: Morgan Kaufmann.

Kolodner, J. (1994). From Natural Language Understanding to Case-Based

Reasoning and Beyond: A Perspective on the Cognitive Model That
Ties It All Together. In R. C. Schank (Ed.), Beliefs, Reasoning and
Decision Making (pp. 55-110). Hillsdale, NJ: Lawrence Erlbaum
Associates.

Laird, J.E., Newell, A., & P.S. Rosenbloom. (1987). Soar: An architecture for
general intelligence. Artificial Intelligence, 33, 1-64.

Leake, D. (1996). Case-Based Reasoning: Experiences, Lessons, and Future Directions.

Menlo Park, CA: AAAI Press/MIT Press.

Lesgold, A. M., Lajoie, S. P., Bunzo, M., & Eggan, G. (1992). SHERLOCK: A
coached practice environment for an electronics troubleshooting job.
In J. Larkin & R. Chabay (Eds.), Computer assisted instruction and
Nature and Methods of Learning by Doing 22

intelligent tutoring systems: Shared issues and complementary approaches (pp.

201-238). Hillsdale, NJ: Lawrence Erlbaum Associates.

Lesgold, A., & Nahemow, M. (2001). Tools to assist learning by doing: Achieving
and assessing efficient technology for learning. In D. Klahr & S.
Carver (Eds.), Cognition and instruction: Twenty-five years of progress.
Mahwah, NJ: Erlbaum.

Means, B., & Gott, S. P. (1988). Cognitive task analysis as a basis for tutor
development: Articulating abstract knowledge representations. In J.
Psotka, D. Massey, and S. Mutter (Eds.). Intelligent tutoring systems:
Lessons learned (pp. 35-58). Hillsdale, NJ: Erlbaum.

Schank, R. C., & Cleary, C. (1995). Engines for Education. Mahwah, NJ: Erlbaum.

Soller, A.L. (in press). Supporting Social Interaction in an Intelligent

Collaborative Learning System. International Journal of Artificial
Intelligence in Education, 12.

Soller, A., & Lesgold, A. (2000, November). Modeling the Process of

Collaborative Learning. Paper presented at the International Workshop
on New Technologies in Collaborative Learning, Awaji-Yumebutai, Japan.

Suthers, D. (2001a). Architectures for Computer Supported Collaborative

Learning. To appear in Proceedings of the IEEE International Conference on
Advanced Learning Technologies (ICALT 2001), 6-8 August 2001,
Madison, Wisconsin.

Suthers, D. D. (2001b). Towards a Systematic Study of Representational

Guidance for Collaborative Learning Discourse. Journal of Universal
Nature and Methods of Learning by Doing 23

Computer Science, 7 (3). Electronic publication:

http://www.jucs.org/jucs_7_3/towards_a_systematic_study

Thorndike, E. L. & Woodworth, R. S. (1901). The influence of improvement in

one mental function upon the efficiency of other functions (I).
Psychological Review, 8, 247-261.

Viterbi, A. J. (1967). Error bounds for convolutional codes and an asymptotically

optimal decoding algorithm. IEEE Transactions in Information Theory,
IT-13, 260-269.

Whitehead, A. N. (1929). The Aims of Education and Other Essays. New York:
Macmillan.
Nature and Methods of Learning by Doing 24

Figures

Figure 1. Example of Transfer-Related Coaching.

Nature and Methods of Learning by Doing 25

Figure 2. Process Explorer Example.

Nature and Methods of Learning by Doing 26

Figure 3. Example explanation for a cell of the process explorer.

Nature and Methods of Learning by Doing 27

Footnotes

1
We went to this approach initially because our control groups did not have
experience using the tutor. Consequently, if we had used the tutor to present
test problems, we would not have known whether differences were due to
unfamiliarity with the interface or to actual differences in troubleshooting
capability.
2
A Markov model is one that is completely specified by listing its states, an
identifiable observable outcome corresponding to each state, and probability of
getting from each state to any of the others that can be reached directly. A
hidden Markov model is one whose states have specific probabilities of
generating any given observable outcome, rather than always having 100%
probability of generating a specific outcome. An algorithm is available that
determines the best fitting hidden Markov model for a process given a set of
sequences generated by that process (Viterbi, 1967).
Nature and Methods of Learning by Doing 28

Brief Biography of Alan Lesgold

Alan Lesgold received his Ph.D. in psychology from Stanford University in 1971,
where he was a student of Gordon Bower (the other members of his doctoral
committee were Richard Atkinson and Herbert Clark). He joined the Learning
Research and Development Center and Department of Psychology at the
University of Pittsburgh that same year. He worked with Charles Wrigley on a
variety of large-scale multivariate data analysis projects as an undergraduate at
Michigan State. This allowed him to develop his psychological knowledge at the
same time as he was learning to use information technologies, and it also
provided his first teaching opportunity, helping with a class on computer
programming for high school students in 1964, one of the first such classes to be
offered. Lesgold credits Robert Glaser, Lauren Resnick, and Arthur Melmed as
important mentors in his post-doctoral career. Currently at Pitt, he is Professor
and Dean of the School of Education, Professor of Psychology and Intelligent
Systems, and Senior Scientist in the Learning Research and Development Center.
His current research interests are uses of digital video and web-based discussions
in teaching and learning. Until July 2000, he served as executive associate director
of the Learning Research and Development Center at the University of
Pittsburgh. Lesgold founded and initially directed Pitt’s interdisciplinary doctoral
program in cognitive science and artificial intelligence. He holds an honorary
doctorate from the Open University of the Netherlands and is a fellow of the
Divisions of Experimental, Applied, and Educational Psychology of the
American Psychological Association, a fellow of the American Psychological
Society, and a past president of the Society for Computers in Psychology. He
was Secretary/Treasurer of the Cognitive Science Society from 1988 to 1997 and
continues to serve on its Board of Governors. In 1995, he was awarded the
Educom Medal by Educom and the American Psychological Association for
contributions to educational technology.
Nature and Methods of Learning by Doing 29

Lesgold served on the National Research Council Board on Testing and

Assessment from 1993 through 1998 and chaired the Board’s Roundtable on
Schooling, Work, and Assessment. Previously, he served on the personnel
performance panel of the National Research Council Committee on Strategic
Technologies for the Army and the NRC review committee for the Army
Research Laboratory. He served on two Congressional Office of Technology
Assessment advisory panels and was the chair of the Visiting Panel on Research
of Educational Testing Service.

In the 1970’s, Lesgold slowly moved his work from laboratory studies of memory
to experimental work on reading comprehension and its acquisition. In writings
with Charles Perfetti, he addressed reading as a flexible form of expertise and
discussed trade-offs among word recognition skills, prior domain knowledge, and
general comprehension skills. For the remainder of his career, he has dealt, one
way or another, with learning by doing and how it can be facilitated.

In the early 1980’s, Lesgold published articles on the acquisition of expertise and
complex skills in medicine and technical domains. Then, he undertook to
translate his and other findings into a useful technology of intelligently coached
instruction. One result was Sherlock – an electronic learning environment for
learning to troubleshoot complex electronic equipment. Sherlock provided
learners with real-life cases, and guided them through their learning process by
“coached apprenticeship”.

In the late 1980’s and early 1990’s, Lesgold continued his work on electronic
apprenticeship environments. But now, the focus was more on the assessment of
complex performances, on the authentic measurement of job performance, and
on the design of intelligent systems for testing. An early advocate – following
Robert Glaser – of full integration of teaching and testing, he stressed the
Nature and Methods of Learning by Doing 30

importance of context-specific feedback for the development of competencies.

Lesgold and colleagues developed the technology of intelligently coached learning
by doing over the period from 1986 to the 1999, with sponsorship by the U. S.
Air Force, U S WEST, and Intel Corporation. It is this work that is the basis for
the present article.

More recently, he and colleagues also developed a technology for supporting rich
collaborative engagement of students and professionals with complex issues and
complex bodies of knowledge. This work, currently focused on teacher
professional education, previously was sponsored by the National Science
Foundation, the Defense Advanced Research Projects Agency, the President’s
Technology Initiative, and currently is funded by the U. S. Department of
Education. The World Bank sponsored related work.

Over the course of his career, Lesgold has collaborated on written work with
Bruce Bender, B. Berardi, A. Block, Jeff Bonar, Gordon Bower, John Seely
Brown, Marilyn Bunzo, Violetta Cavalli-Sforza, Susan Chipman, Kwang-Su Cho,
Bill Clancey, Michal Clark, John Connelly, Mary Beth Curtis, Hildrene De Good,
Sharon Derry, Gary Eggan, Paul Feltovich, Mike Feuer, Robert Fitzhugh, Sipke
Fokkema, Jim Fox, Claude Frasson, Gareth Gabrys, Gilles Gauthier, Dedre
Gentner, Morton Anne Gernsbacher, Robert Glaser, Susan Goldman, Roberta
Golinkoff, Maria Gordin, Sherrie Gott, Linda Greenberg, J. Guttman, Kathleen
Hammond, Nira Hativa, Ted Hughes, Joyce Ivill, Rob Kane, Sandra Katz,
William Keith, Dale Klopfer, Susanne Lajoie, Clayton Lewis, Joel Levin, Debra
Logan, Heinz Mandl, Sr. Claire McCormick, Martin Nahemow, Luis Osin,
Massimo Paolucci, Charles Perfetti, Dan Peters, Mitch Rabinowitz, Govinda Rao,
Fred Reif, Lauren Resnick, Steve Roth, Harriet Rubinson, Osnat Sarig, Mark
Seidenberg, Colleen Seifert, Mike Shafto, Joseph Shimron, Elliott Soloway, Dan
Nature and Methods of Learning by Doing 31

Suthers, David Tieman, Eva Toth, Joe Toth, Gregg Vesonder, Yen Wang, Arlene
Weiner, David Winzenz, and Dick Wolf.

Selected Bibliography of Alan Lesgold, Chronologically Ordered

Bower, G.H., & Lesgold, A.M. (1969). Organization as a determinant of
part-to-whole transfer in free recall. Journal of Verbal Learning and
Verbal Behavior, 8, 501-506.

Lesgold, A.M. (1972). Pronominalization: A device for unifying sentences in

memory. Journal of Verbal Learning and Verbal Behavior, 11, 316-323.

Lesgold, A.M., & Goldman, S.R. (1973). Encoding uniqueness and the imagery
mnemonic in associative learning. Journal of Verbal Learning and Verbal
Behavior, 12, 193-202.

Lesgold, A.M., & Perfetti, C.A. (1978). Interactive processes in reading

comprehension. Discourse Processes, 1, 323-336.

Lesgold, A.M., Roth, S.F., & Curtis, M.E. (1979). Foregrounding effects in
discourse comprehension. Journal of Verbal Learning and Verbal
Behavior, 18, 291-308.

Lesgold, A.M. (1984). Acquiring expertise. In J. R. Anderson & S. M. Kosslyn

(Eds.), Tutorials in learning and memory: Essays in honor of Gordon Bower.
San Francisco, W. H. Freeman.

Lesgold, A.M., Resnick, L.B., & Hammond, K. (1985). Learning to read: A

longitudinal study of word skill development in two curricula. In G.
Waller and E. MacKinnon (Eds.), Reading Research: Advances in Theory
and Practice. New York: Academic Press.
Nature and Methods of Learning by Doing 32

Lesgold, A., Rubinson, H., Feltovich, P., Glaser, R., Klopfer, D., & Wang, Y.
(1988). Expertise in a Complex Skill: diagnosing X-ray Pictures. In
Chi, M. T. H., Glaser, R., and Farr, M. (Eds.), The nature of expertise.
Hillsdale, NJ: Erlbaum.

Glaser, R., Lesgold, A., & Lajoie, S. (1987). Toward a Cognitive Theory for the
Measurement of Achievement. In R. R. Ronning, J. Glover, J. C.
Conoley, & J. C. Witt. The influence of cognitive psychology on testing.
Hillsdale, NJ: Erlbaum.

Lesgold, A. M., Lajoie, S. P., Bunzo, M., & Eggan, G. (1992). SHERLOCK: A
coached practice environment for an electronics troubleshooting job.
In J. Larkin & R. Chabay (Eds.), Computer assisted instruction and
intelligent tutoring systems: Shared issues and complementary approaches (pp.
201-238). Hillsdale, NJ: Lawrence Erlbaum Associates.

Lesgold, A. (1994). Assessment of intelligent training technology. In E. Baker

and H. O'Neil, Jr. (Eds.), Technology assessment: In Education and Training
(vol. 1) pp. 97-116. Hillsdale, NJ: Lawrence Erlbaum Associates.

Lajoie, S. P., & Lesgold, A. M. (1992). Dynamic assessment of proficiency for

solving procedural knowledge tasks. Educational Psychologist, 27, 365-
384.

Katz, S., & Lesgold, A. (1993). The role of the tutor in computer-based
collaborative learning situations. In S. Lajoie & S. Derry (Eds.),
Computers as cognitive tools (pp.289-317). Hillsdale, NJ: Lawrence
Erlbaum Associates.
Nature and Methods of Learning by Doing 33

Lesgold, A. (1993). Information technology and the future of education. In S.

Lajoie & S. Derry (Eds.), Computers as cognitive tools (pp. 369-383).
Hillsdale, NJ: Lawrence Erlbaum Associates.

Lesgold, A., Katz, S., Greenberg, L., Hughes, E., & Eggan, G. (1992).
Extensions of intelligent tutoring paradigms to support collaborative
learning. In S. Dijkstra, H. P. M. Krammer, & J. J. G. van
Merrienboer (Eds.), Instructional models in computer-based learning
environments, pp. 291-311. Berlin: Springer-Verlag.

Lesgold, A. (1996). Quality control for educating a smart work force. In L. B.

Resnick, J. Wirt, (Eds.), Linking school and work: Roles for standards and
assessment, (pp. 147-191). San Francisco: Jossey-Bass.

Derry, S. P., & Lesgold, A. M. (1996). Toward a situated social practice model
for instructional design. In R. Calfee & D. Berliner (Eds.), Handbook
of Educational Psychology. New York: Macmillan.

Osin, L., & Lesgold, A. (1996). A proposal for the reengineering of the
educational system. Review of Educational Research, 66(4), 621-656.

Gott, S. P., & Lesgold, A. M. (2000). Competence in the Workplace: How

Cognitive Performance Models and Situated Instruction Can
Accelerate Skill Acquisition. In R. Glaser (Ed.), Advances in instructional
psychology. Hillsdale, NJ: Erlbaum.

Lesgold, A., & Nahemow, M. (2001). Tools to assist learning by doing: Achieving
and assessing efficient technology for learning. In D. Klahr & S.
Carver (Eds.), Cognition and instruction: Twenty-five years of progress.
Mahwah, NJ: Erlbaum.
 Nature and Methods of Learning by Doing 34

View publication stats