Value of Training Evaluation

The Value of Evaluation:
Making Training Evaluations

More Effective
AN ASTD Research Study

The Value of Evaluation:
Making Training Evaluations
More Effective
AN ASTD Research Study

© 2009 by the American Society for Training &
Development. All rights reserved.
No part of this publication may be reproduced, distributed,

or transmitted in any form or by any means, including
photocopying, recording, or other electronic or mechanical
methods, without the prior written permission of the
publisher, except in the case of brief quotations embodied
in critical reviews and certain other noncommercial uses
permitted by copyright law. For permission requests,
write to ASTD Research, 1640 King Street, Box 1443,
Alexandria, VA 22313-1443.
Ordering Information
Research reports published by ASTD can be purchased

by visiting our website at store.astd.org or by calling
800.628.2783 or 703.683.8100.
ASTD Product Code: 790907

ISBN-10: 1-56286-563-3
ISBN-13: 978-1-56286-563-4
ASTD Editorial Staff
Director of Content: Dean Smith

Manager, ASTD Research: Andrew Paradise
Manager, ASTD Press: Jacqueline Edlund-Braun
Senior Associate Editor: Justin Brusino
Interior Design and Production: Kristi King
Cover Design: Rose Mcleod and Kristi King
| contents |
3 | ForeWorD
4 | Executive Summary
7 | Introduction
11 | Section I—What evaluation

techniques are companies using?
19 | Section II—How and when Do

organizations conduct evaluations?
25 | Section III—What are the Barriers

to Training Evaluation?
29 | Section IV—What are Companies

Spending on Training Evaluation?
33 | Section V—How Can Organizations

Improve Training Evaluation?
36 | Conclusion and Policy

Recommendations
39 | References
41 | Appendix—The Value of Evaluation

Survey Overview
61 | About the Authors and Contributors
63 | About the Contributing Organizations
| 1
| foreword |
M easuring the impact of learning continues to

be one of the most challenging aspects of the
learning function. The tough economy demands
that business leaders scrutinize costs to find even greater ef-
ficiencies, but it can be difficult to show hard dollar savings
This study takes a close look at the value of learning.
The Value of Evaluation: Making Training Evaluations
More Effective explores the complex issue of learning
evaluation, the techniques being used, barriers to
effective implementation, and strategic use of learning
for areas like training and development that have so many metrics. With only about one quarter of the survey
intangible processes. Companies employ myriad strategies respondents telling us they get a solid “bang for their
to identify and quantify the results of training, but most are buck” from their training evaluation efforts, I believe
not satisfied with the evaluation efforts. the best practices in this report can help many organi-
zations become much more proficient and strategic in
A 2006 ASTD/IBM report titled C-Level Perceptions of the evaluating learning.
Strategic Value of Learning noted that while chief learning
officers and chief executive officers have different needs I hope you will use the information in this report to take
with regard to learning evaluation, at a high level they action as you make learning evaluation a process that
agree learning is strategically valuable. Both groups also strengthens your organization.
agreed that isolating and measuring learning’s financial
contribution to business are difficult, and often perceptions
of stakeholders (employees, business unit leaders, and
executives) are a key indicator of learning’s success. Both
CLOs and CEOs recognized the need for strong governance
processes for planning, allocating, and managing learning Tony Bingham
investments, and that efficiency of the learning function can President and CEO
be increased by streamlining and standardizing processes, ASTD
leveraging technology, and right-sourcing the function.
Agreement on these broad topics is important because it
means common ground exists on which to build a better,
more effective learning evaluation process, which will in
turn positively affect business goals.
Foreword | 3
| executive summary |
T he pursuit of excellent learning evaluations continues,

but so far few organizations think they’ve mastered
them, according to our study. In fact, only about
one-quarter of respondents to our Value of Evaluation
Survey said they agreed that their organization got a solid
be used. Many experts recommend that organizations
evaluate at all five levels but trim the number of programs
that get evaluated as they move up the levels. Instead,
it appears that organizations are evaluating at the first
few levels and then dropping off completely. Companies
“bang for the buck” from its training evaluation efforts. also employ the Brinkerhoff Success Case Method, which
Reactions were only slightly better when respondents were can be described as “evaluation studies with successful
asked if their learning evaluation techniques helped meet trainees.” About half of respondents use some version of
learning and business goals. this method, which highlights individual training success
stories to communicate the value of learning.
These findings are in line with some other recent studies,
including a 2009 survey that Chief Learning Officer mag-
azine conducted among its Business Intelligence Board. What are the Barriers to
In that survey, 41 percent of respondents said they were Effective Evaluation?
dissatisfied with their organization’s learning measurement
Respondents cite numerous barriers to the evaluation of
versus 35 percent who said they were satisfied (Anderson,
learning. For example, metrics such as business results
2009). In a 2007 survey conducted by Expertus and
and return-on-investment (ROI) are sometimes seen as
Training Industry Inc., 77 percent of respondents said
too complex and time-consuming to calculate. Difficulty
their evaluation efforts fall short in providing meaningful
in isolating training as a factor that affects behaviors and
information for business planning (Radhakrishnan, 2008).
results also impedes learning evaluation.
But organizations are not giving up on successful
Perhaps most disturbing is that many training professionals
measurement of the learning function. They continue to
claim that leadership isn’t actually interested in training
explore ways to communicate and document the value of
evaluation information. This is worrisome in an age when
the training and development they provide to employees.
employee skills are more critical than ever, being one of the
Our study finds that most companies do at least some
few differentiators among businesses in a global economy.
form of evaluation, although they may be unsure how
Learning professionals need to gain a better understanding
to go about it and not sure what to do with the results.
of the roots of such apathy toward evaluation.
We believe the data in this report can help many firms
become much more proficient in these areas.
What are Companies Spending
What Evaluation Techniques on Training Evaluation?
are Companies Using? Depending on their size, companies can spend anywhere
from a few thousand dollars to tens of millions of dollars on
The five-level Kirkpatrick/Phillips model of learning
training annually. About 5.5 percent of that gets spent
evaluation is the most common practice. The five levels
on evaluating the training—a little more among small
include participant reaction (Level 1), level of learning
companies and a little less among large companies. Most
achieved (Level 2), changes in learner behavior (Level
of the money is spent on internal resources, but the
3), business results derived from training (Level 4), and
smaller the company, the higher the external expenditures.
the ROI from training (Level 5). Ninety-two percent of
Organizations tend to spend the largest share of their evalu-
respondents said they measure at least Level 1 of the
ation budget on Level 1 (reaction) evaluations and then
model. But the use of the model drops off dramatically
less on subsequent levels. On average, companies spend
with each subsequent level. This tendency suggests that
half of their evaluation budget measuring the reactions
managers may not fully grasp how the model should
of participants. The question for learning professionals is
4 | Value of Evaluation: Making Training Evaluations more Effective

whether this is the best use of their limited resources. Our This study also looked at specific practices. It found, for
study suggests otherwise. example, that firms using evaluation to gauge whether
employees are learning what is required or to calculate
learning’s effects on business outcomes are far more
How Can Organizations Improve likely to view their evaluation efforts as successful and
Training Evaluation? to have better market performance. Other important
actions include making sure learning positively influ-
This study uncovers how companies use information
ences employee behavior and demonstrating the value of
gathered from learning evaluations. It also identifies
learning to others in the organization. The most valuable
which techniques are most strongly linked to suc-
use of evaluation, however, is to improve overall business
cessful evaluation and, more generally, better market
results. Companies that say they do this to a high extent
performance.
tend to see results, as this action was the most positively
correlated to market performance.
Our study found that, in most cases, using Kirkpatrick/
Phillips (K/P) levels is associated with greater success in
The key findings suggest that today’s organizations do
the area of learning metrics. That is, if respondents said
not need to settle for learning evaluations that are done
their organizations used a given K/P level of evaluation,
as more of a ritual than as a business-improvement strat-
then they were also more likely to give their organizations
egy. Good evaluation techniques are available, and they
a higher score in terms of learning evaluation success.
can be used to make the whole organization stronger.
The exception to this rule is when organizations only
measure Level 1.
executive summary | 5
| introduction |
I n a difficult economy, business leaders begin to

scrutinize every aspect of their organization with
an eye toward costs. From the supply chain to real
estate to office supplies, companies want to make sure
When only about a quarter
of participants say their
they are getting their money’s worth—and training is no
exception. A large company can spend tens of millions of
companies get a good
dollars every year on training and developing employees,
yet very rarely do leaders get a clear picture of what they
return on their learning
are getting in return. As with most “soft” issues, it can
be difficult to show tangible results from such intangible
metrics investments, then
processes. Companies employ various strategies to iden-
tify and quantify the results of training, and this study
something is deeply wrong.
examines some of the more common ones, how they are
used, what prevents evaluation success, and what could
These findings illustrate that there’s much more progress
lead to future successes.
to be made in the area of learning evaluation in most
organizations. When only about a quarter of participants
say their companies get a good return on their learning
Satisfaction with Learning metrics investments, then something is deeply wrong.
Metrics and the Evaluation One of the most important purposes of evaluations
Success Index should be to determine what is most and least effective
and then to invest more resources in what’s most effec-
When asked if their company got a solid “bang for the
tive. Evaluation is a feedback tool intended not only to
buck” from learning metrics, more than one-third of
help employers meet their learning goals but also to make
the respondents remained neutral, neither agreeing nor
learning itself more efficient. With these findings in hand,
disagreeing. Slightly less said they disagreed, while only
learning professionals in many companies clearly need
about one-quarter said they agreed that their organiza-
to take a hard look at their evaluation programs. If they
tion did indeed get a good return on its investment.
don’t think they’re getting a solid return on investment,
Reactions were slightly more positive when respondents
then they should conduct a rigorous analysis of how, in
were asked if their learning evaluation techniques helped
the future, this can be achieved.
meet learning and business goals. Specifically, more than
one-third of respondents (36.5 percent) reported that
their learning evaluation techniques help them meet
their organization’s business goals to a high or very high
extent. Slightly more (41.3 percent) reported that their
evaluation techniques help them meet their organization’s
learning goals to a high or very high extent. These results
are in line with a 2009 survey Chief Learning Officer
magazine conducted among its Business Intelligence
Board. In that survey, 41 percent of respondents said
they were dissatisfied with their organization’s learning
measurement, and 35 percent said they were satisfied
(Anderson, 2009).
introduction | 7
Figure 1 | Please state the degree to which you agree with the
following statements on evaluation practices
(Percentage that agree or strongly agree)
Our learning evaluation techniques help us meet 41.3%
our organization’s learning goals
Our learning evaluation techniques help us meet 36.5%

our business goals
We get a solid “bang for our buck” when it 25.6%

comes to using learning metrics
Responses to these three items were combined to create Another evaluation technique that has gained headway
an Evaluation Success Index (ESI), which was used to in recent years is the Brinkerhoff Success Case Method.
correlate against the specific evaluation practices. Readers That method, developed by Robert O. Brinkerhoff,
will see this index frequently cited in this study, along professor of education at Western Michigan University
with the Market Performance Index (MPI). The MPI was and author of several human resource development
determined by asking a series of questions about respon- books including The Success Case Method, entails
dent organizations’ overall market performance over the identifying successful learners and interviewing them
past five years. The results of these indices were mapped to find out what made the learning experience work
to other survey responses. for them and what they achieved as a result. It can
also be applied to unsuccessful candidates to determine
what went wrong. These case studies can then be used
Describing Learning to communicate the value of learning throughout the
Evaluation Methods organization, identify what works and what doesn’t,
and help improve learning programs.
While there are various ways to evaluate the effectiveness
of learning within an organization, most methods tend
When study respondents were asked about other types of
to fall somewhere into the range of evaluation techniques
learning evaluation techniques, their answers included a
making up the Kirkpatrick/Phillips model. That model
variety of responses, including
consists of five levels of evaluation and was developed by
Donald L. Kirkpatrick, professor emeritus at the University • Action learning projects
of Wisconsin and author of Evaluating Training Programs: • ADDIE (assess, design, develop, implement, evaluate)
The Four Levels, who developed Levels 1 through 4, and model
Jack Phillips, author of The Human Resource Scorecard, • Alignment measures
who developed Level 5. The first four levels measure
(1) the reaction of participants, (2) the level of learning • Balanced scorecards
achieved, (3) changes in learner behavior, and (4) business • Benchmarking, often against local metrics, ASTD
results derived from training. Level 5 measures the return metrics, or other industry standards
on investment (ROI) that learning programs deliver. • Bloom’s taxonomy
• Cost avoidance
• Customer/client assessments
• Employee engagement
8 | value of evaluation: making training evaluations more effective

• Evaluation capacity building
Another evaluation technique
• Evidenced-based analysis
• Executive satisfaction with learning being delivered that has gained headway in
to business units
• Impact maps recent years is the Brinkerhoff
• Impressions of trainers Success Case Method.
• Interviews and focus groups
• Leading and lagging indicators of business trends
• Linkage model
• Relevance index
• Return on expectations
• Seat count and number of hours delivered
A Note About Correlation Data
• Stakeholder satisfaction Many of the graphs and tables in this report
• Technical certification training. contain correlation information. The correlation
coefficient is used to measure the strength and the
It was beyond the scope of this study to investigate every direction of the relationship between two variables. For
learning evaluation technique in a detailed manner, and example, the closer a correlation is to +1, the stronger
there can be debate about how many of these other the positive relationship between the two variables
methods could legitimately fall under the umbrella of the such that an increase in one variable is associated
Kirkpatrick/Phillips model. Still, we found it enlighten- with an increase in the other. But, just because two
ing to see how many other techniques participants view variables are found to be correlated does not mean
as available to them, and there is clearly room for future that a cause-and-effect relationship exists. When a
studies that go into greater detail on these topics. correlation between the two variables is significant, for
example at p<.05, there is only a 5 percent chance that
these results would have occurred by chance. Stated
differently, you can be 95 percent confident that these
results are not in error and that you would get these
same results if you conducted this research again. With
a correlation of p<.01, which represents the majority
of correlations reported in this study, you can be 99
percent confident that these results are not in error.
introduction | 9
| section I |
What Evaluation Techniques Are Companies Using?
T he majority of the respondents (91.6 percent) said

their companies evaluate at least some of their
learning programs at Level 1, the reaction of the
participants. This makes sense as it is typically the easiest
measurement to take. Whether it is an online survey given
Level 3 brings more complexity and another drop in usage.
This level evaluates the change in employee behavior that
the training achieved. Only 54.6 percent of respondents
said their companies evaluate at this level, signifying its
greater level of difficulty. According to Kirkpatrick, this
after completing a web-based course or “smile sheets” difficulty stems from the unpredictability of behavior
handed out following classroom instruction, it is common changes. First, employees must be presented with an op-
practice to assess the experiences of learners following portunity to use the new behaviors. Second, it is impossible
training. Companies can easily gauge reactions and to know when the change will occur, if at all. Even if the
determine which courses employees enjoy and which change does occur, says Kirkpatrick, an employee may or
ones they don’t. may not continue the new behavior. All of this conspires
to make behavior changes somewhat difficult to measure.
The usage of other types of evaluation drops off as we
move up the K/P model levels. A little more than four Kirkpatrick’s final level, Level 4, measures what is arguably
out of five survey respondents said their organizations the most important metric: results. Yet only 36.9 percent
evaluate at Level 2, where companies try to find out what of respondent companies evaluate at this level to any
knowledge was actually gained, which skills were developed or degree. Again, the difficulty of such measures may be one
improved, and what employee attitudes were changed by of the reasons, although we could argue that business results
the training. The difficulty in obtaining this information are easier to measure than employee behaviors. Another
may explain the drop in usage, but the value that participants possible reason is that, according to some experts,
attach to this type of evaluation helps explain why the
drop isn’t larger.
Figure 2 | Percentage who use these Kirkpatrick/Phillips levels in

their organizations and percentage who attribute a high
or very high level of value to those levels
Percentage who use the Percentage who say this level

corresponding level to any extent has high or very high value*
Reactions of participants 91.6% 35.9%
Evaluation of learning 80.8% 54.9%
Evaluation of behavior 54.5% 75.0%
Evaluation of results 36.9% 75.0%
Return on Investment 17.9% 59.4%
None of the above 4.1%
* Only among those who say they use this level to some extent
section i: What Evaluation Techniques Are Companies Using? | 11

Our study shows that Level 5 ROI = (Benefits/Costs) x (100).
gets the least participation, A modified alternative is
with only 17.9 percent of ROI = ((Benefits - Costs)/Costs) x (100).
respondent companies There’s also a forecasted calculation:
using it to any extent. ROI = ((Benefits - Costs)/Costs) x (100) x (%A) x (%P).

Where %A is the percentage of learners who are
expected to use the training and %P is the percentage
of employees who received the training (Murray and
companies should measure learning via Levels 1 through
Efendioglu, 2007, pp. 374–375; Phillips, Pulliam,
3 before moving on to Level 4. By then, however, many
& Wurtz, 1998).
companies will have stopped bothering to evaluate.
Another reason is that it’s simply hard to prove the
These and other, more complicated equations such as inter-
relationship between business results and training.
nal rate of return and net present value can be intimidating,
Businesses are complex, multidimensional entities, and
especially when it can be so difficult to identify what exactly
it’s often difficult to tease out cause and effect. It becomes
to put in the “Benefits” spot. But these equations can be
virtually impossible to prove “beyond a shadow of a
used to determine the benefits of training over time, allow-
doubt” that a learning program has been responsible
ing companies to calculate training ROI similarly to the
for better business results. The best many businesses can
way in which they calculate ROI for more tangible assets
hope for is that “a preponderance of evidence” points to
(“Is Training Worth It?,” 2008). According to Jack Phillips,
the idea that training produces better business results.
data collected in Level 4 through follow-up questionnaires,
program assignments, action plans, and performance moni-
While some would argue that Kirkpatrick’s Level 4
toring can all be converted into monetary data (Phillips,
includes the idea of return on investment, or ROI, Jack
Pulliam, & Wurtz, 1998). Designing training programs
Phillips proposed that the word “results” does not go far
with traceable outcomes is the key to being able to translate
enough. He put forth Level 5, dedicated solely to ROI.
training benefits into dollar amounts that can be used in any
Our study shows that Level 5 gets the least participation,
given equation (Platt, 2008).
with only 17.9 percent of respondent companies using it
to any extent.
It was somewhat surprising to find that the use of the
Kirkpatrick Levels 1–4 is more prevalent at larger or-
With ROI for all business lines being so important to
ganizations but that the use of Level 5 is more likely at
senior leadership, why is it not tracked more for learning
smaller companies. It’s possible that the data is easier to
programs? First, it is probably the most difficult aspect of
measure or less time-consuming to gather for a smaller
learning to measure. It can be a difficult process to trans-
company and is therefore a more manageable process
late the value of a training program into dollars and cents.
than for a larger firm.
There are, of course, numerous calculations that can be
used to determine ROI, including a cost-benefit analysis:

Companies Value Some levels We need to ask why ROI
More than Others
As was shown in figure 2, the degree of usage of a K/P isn’t seen as having as much
evaluation level does not tell us much about its perceived
value. Although Level 1 is the most commonly used type value as Levels 3 and 4.
of evaluation, it is not the most likely to be seen as hav-
ing high or very high value. In fact, only 35.9 percent of
respondents whose companies use Level 1 evaluation said We need to ask why ROI isn’t seen as having as much
it had high or very high value. value as Levels 3 and 4. One theory is that it simply
isn’t used widely enough for it to be seen as having high
By comparison, 54.9 percent said Level 2 had high or value. That is, as the next section will show, even in
very high value, and three-quarters said the same about organizations where it is used, it is used for only a small
Level 3 and Level 4. This puts Level 1 at the bottom of fraction of learning programs (though not out of line
the value chain for the four Kirkpatrick levels. It should with Kirkpatrick’s recommendations). Given this, it may
be noted that this data pertains only to those respondents become difficult for firms to assign it greater value. It’s
whose organizations actually use these levels of also possible that executives, who are accustomed to see-
learning evaluation. ing ROI attached to more measurable business practices,
are sometimes skeptical of seeing it assigned to learning-
Although these findings might strike some as puzzling, related practices.
it makes a certain sense. There is obviously more value
in knowledge gained, behaviors changed, and results
achieved than in participants’ reactions. What isn’t as The Type of Evaluation and
clear is why more companies aren’t evaluating more pro- Perceived Success
grams at these higher levels. Our study found that, with the exception of Level 1,
using Kirkpatrick/Phillips levels of evaluation is always
Phillips’ ROI-based level was seen as having high or very associated with higher Evaluation Success Index (ESI)
high value by 59.4 percent of respondents, counting only scores. That is, if respondents said their organizations
those companies that measure it. By contrast, only 8.6 used a given K/P level of evaluation, then they were also
percent went so far as to say ROI carried no value at all, more likely to report learning evaluation success.
the highest for any level.
Figure 3 | In instances where you use it, how much value

do you think each of the following types or levels
of evaluation has for your organization?
A little Very high

Responses No value value Some value High value value
Reactions of participants 1.2% 14.2% 48.6% 22.8% 13.1%
Evaluation of learning 0.5% 4.2% 40.4% 38.1% 16.8%
Evaluation of behavior 1.3% 2.8% 21.0% 46.4% 28.6%
Evaluation of results 2.8% 3.4% 18.8% 35.7% 39.3%
Return on Investment 8.6% 6.0% 25.9% 26.7% 32.7%

The method in which training Evaluating Different Types
of Training
is delivered can be a strong As noted above, those who responded that their organiza-
tions use the Kirkpatrick/Phillips levels to evaluate train-
determinant for whether ing do not typically use them for all of their programs. In
fact, respondents in organizations that use Level 1 evalua-
and how that training will tions say that, on average, only 78 percent of all learning
programs use such evaluations. What’s more, from there
be evaluated. the percentage of programs evaluated drops off steeply as
the level increases. Respondents in organizations that use
Level 2 evaluation said such evaluation is used for about
The exception to this rule is at Level 1. More than nine half of all learning programs. The percentage for Level 3
out of 10 respondents say their organizations gather programs is 25.2 percent.
information on the reaction of participants to learning
programs, but this type of evaluation is not significantly Such results are somewhat out of line with Kirkpatrick’s
related to success. Levels 2 through 5 are, however, own recommendations, which target measuring 100 percent
strongly correlated with success. What’s more, three of of programs at Level 1, 60 percent at Level 2, 30 percent
the levels are statistically linked with market performance at Level 3, 10 percent at Level 4 and 5 percent at Level 5
scores as well (see figure 4). (ASTD, 2005). Our respondents were nowhere near 100
percent for Level 1, and they were well short of expectations
Correlation is not the same as causation, of course, but for Level 2 as well. Program evaluation for Levels 3, 4, and
the bottom line is that using most K/P levels and types of 5 were more in line, for companies that used these levels.
learning metrics is linked with self-reported evaluation success.
Figure 4 | Percentage of organizations using

Kirkpatrick/Phillips levels and significant correlations between
the usage of these levels and scores on ESI and MPI
Correlations Correlations
Average with Evaluation with Market
Responses percentage Success Index Performance Index
Reactions of participants 91.6%
Evaluation of learning 80.8% ** **
Evaluation of behavior 54.6% ** *
Evaluation of results 36.9% ** *
Return on investment 17.9% **
*ANOVA analysis results show that the mean responses are significantly different at the p<.05 level.
**ANOVA analysis results show that the mean responses are significantly different at the p<.01 level. In each case,
the mean for those who answered “yes, we use this level of evaluation” was higher than for those who answered
“no, we don’t.”

Results for individual programs such as leadership devel-
opment and technical learning were similar to the overall
Learning professionals
results, but companies seem to evaluate sales programs
differently. Organizations are far less likely to evaluate
should check to see if there
employee reaction to sales training when compared with
other types of learning programs. But they are slightly
are evaluation opportunities
more likely to evaluate the ROI. This may be because
ROI for sales training is easier to determine. If sales go
being lost in regard to
up, it is likely that the training was effective.
technology-based learning.
The method in which training is delivered can be a strong
determinant for whether and how that training will be far more classroom programs, with an average of
evaluated. While more than 80 percent of live classroom 70 percent of learning being delivered this way,
training programs are evaluated by employee reaction, while an average of 30 percent is delivered electronically.
only 52 percent of e-learning programs are measured at
this level. In fact, classroom programs were more likely Still, we remain somewhat puzzled by the relative lack
to be evaluated at every level than e-learning programs. of evaluation among technology-based methods of learning.
It could be that the time, effort, and money needed to It would seem a relatively simple matter to deliver a
produce classroom programs warrants greater scrutiny, Level 1 evaluation to those who use such systems. And,
while e-learning is sometimes cheaper to produce and by comparing evaluation data from technology-based
requires less of the learners’ time. Companies also use learning systems with data from other systems—such as
Figure 5 | What percentage* of your learning programs is evaluated at

each level of the Kirkpatrick/Phillips five-level evaluation model?
Level 1 Level 2 Level 3 Level 4 Level 5
78.4%
75.3%
69.7%
54.0%
48.9%
49.2%
43.0%
36.7%
28.1%
27.1%
25.2%
23.8%
17.7%
14.4%
16.3%
15.4%
9.9%
7.1%
7.1%
7.0%
All Learning Leadership Sales Programs Technical Programs

Programs Development
Programs
* Percentages are based on the group of respondents that use these Kirkpatrick/Phillips levels to any extent

performance management, customer relationships, or These stories can then be disseminated throughout the
sales—companies might be able to more easily find cor- company, highlighting the positive effect of the learn-
relations between training program successes and other ing experience. The method can also be used to identify
types of business or behavioral results. In short, learning non-success stories, allowing learning professionals to
professionals should check to see if there are opportunities discover what is not working. We specifically asked
lost in regard to evaluation of technology-based learning. respondents if their organizations conducted evaluation
studies or interviews with successful trainees, though we
did not name the Brinkerhoff Success Case Method. The
The Brinkerhoff Success study found that almost half of respondents said they did,
Case Method with several more using an “other” selection as a chance
to write in “Brinkerhoff” or “Success Case Method” in a
Another popular technique used to evaluate training is
previous question.
the Brinkerhoff Success Case Method. This method in-
volves identifying likely success cases and then interview-
ing those individuals to verify that they are a true case
of learning success. If they are, companies learn how the
person used the training and what results were achieved.
Figure 6 | What percentage* of your delivery methods is evaluated at

each level of the Kirkpatrick/Phillips five-level evaluation model?
Level 1 Level 2 Level 3 Level 4 Level 5
80.9%
52.0%
49.5%
43.4%
24.7%
14.6%
13.6%
9.6%
6.1%
5.8%
Live Classroom Technology-based

* Percentages are based on the group of respondents that use these Kirkpatrick/Phillips levels to any extent

According to Brinkerhoff, this method can be used to find This study found that, among those who said their
out how many employees are using the training success- organization has conducted an evaluation interview with
fully, what the business value of those successes are, and successful trainees, just three-quarters said the inter-
how much value from the training has been unrealized views have helped them develop more effective learning
(Brinkerhoff, 2006). Among the companies that use this services, only 59.7 percent said they disseminate the
method, however, we found that many do not use the positive stories throughout the company, and only a little
evaluation studies the way Brinkerhoff intended. more than a third use the studies to identify factors that
enhance or impede business impact.
Successful case studies should be disseminated through-
out the company. Also, the results of both successful and
unsuccessful case studies should be used to determine
what factors impede or facilitate learning. And finally, the
results of the Brinkerhoff Success Case Method should be
used to improve learning programs.
Figure 7 | Has your organization ever conducted an evaluation

study (e.g., interviews) with successful trainees?
No Yes
52.3% 47.7%

Therefore, although the Brinkerhoff approach does, We also looked at the potential effectiveness of the
in fact, seem to be influential as a learning evaluation Brinkerhoff Success Case Method. Using an ANOVA
technique, it apparently is not used as systematically as its analysis, we compared the mean scores of respondents
creator would advocate. This finding bears a resemblance who answered that they had conducted an evaluation
to the data about the Kirkpatrick/Phillips model in that study with successful trainees against respondents who did
many organizations apparently use components of the not. The group that answered they had done so had a mean
model without using it in the kind of integrated manner score of 3.2 on the Evaluation Success Index, compared to
that learning experts would recommend. 2.8 for the “no” group. This difference was significant,
which means that those respondents who have conducted
evaluation studies with successful trainees report higher
ESI scores compared with those who did not use the suc-
cess case method.
Figure 8 | In regard to those evaluation studies of

successful trainees, please state whether your organization
has taken the following actions
(Percentage that agree or strongly agree)
We have developed more effective learning

products and services 76.5%
We have taken stories of the positive business

impacts of learning programs and disseminated 59.7%
them throughout the company
We have tracked the factors that enhance or 36.5%

impede business impact

| Section II |
How and When Do Organizations Conduct Evaluations?
T here are several points at which a company can

measure changes in job behavior (that is, Level 3
of the Kirkpatrick/Phillips model), and organizations
often take measurements at more than one time. A major-
ity (55.8 percent) of respondents whose organizations use
More than a quarter of responding organizations use
action planning to a high or very high extent, and an
additional one-quarter use on-the-job observation and
performance record monitoring to the same extent. The
least likely technique used is follow-up focus groups (9.1
Level 3 evaluations say they take measurements between percent), which, ironically, is one of the strategies most
two weeks and two months after the training is administered, strongly correlated with the ESI.
and 51.7 percent say measurements occur more than two
months after the learning event. Such focus groups may be more time-consuming and re-
quire more managerial sophistication than various other
More than one-third take this measurement within the follow-up practices, but they would likely have a greater
first two weeks of training, which is somewhat surprising impact. That so few organizations are currently using
because Kirkpatrick recommends that ample time be this strategy, despite its effectiveness, suggests it may
given for employees to have an opportunity to use the be a cutting-edge practice that delivers a true
behavior (Kirkpatrick, 2007). It is also surprising that competitive advantage.
only about 40 percent of the companies that evaluate at
Level 3 take a measurement prior to training. According Another notable strategy is monitoring performance
to Kirkpatrick, measuring the behaviors of a group records. Although there is sometimes skepticism about
of employees prior to training creates a control group the accuracy of performance appraisals in today’s orga-
against which further measurements can be compared nizations, linking performance recordings with learning
(Kirkpatrick, 2007). experiences does, in fact, appear to be a useful practice.
That is, there’s a significant correlation, even after doing
The most common way to measure changes in job behavior a multiple regression analysis, between performance
is with follow-up surveys of participants, used to a high record monitoring and the effectiveness of learning evalu-
or very high extent by 31.0 percent of respondent companies. ations in organizations. Of course, the success of this
These surveys are also significantly correlated with the practice hinges on other factors, such as the excellence of
Evaluation Success Index (ESI). Supplemental statistical performance feedback systems, but if those systems are
analyses also identified follow-up surveys as a good strategy in order, then using them for Level 3 evaluations should
to drive evaluation success. reap dividends.
Figure 9 | If your organization uses Level 3 learning metrics (that is, the
evaluation of behaviors), when does it regularly take measurements?
Prior to learning intervention 39.3%
Immediately post-learning intervention (within 37.9%

first two weeks after intervention/learning)
Short-term post-learning intervention (two weeks 55.8%

to two months after intervention/learning)
Long-term post-learning intervention (more than 51.7%
two months after intervention/learning)
section iI: how and when do organizations conduct evaluations? | 19

Measuring Program Impact
If an Organization Must Choose:
How do organizations most commonly measure the
Which Approaches Work Best for
results of learning, that is, Level 4 evaluation? The most
Level 3 Measurement?
frequently used metric is customer service, used to a high
Nine different factors were significantly correlated
or very high extent by 38.9 percent of respondents’
with the Evaluation Success Index. We recognize, however, that
organizations. Measuring learners’ perceptions of
most organizations can utilize only a few measurement
techniques at any one time. So, which approaches, if an
program impact is also common, as is measuring
organization must choose, lead to the best evaluation proficiency and competency levels.
results? Based on a further analysis of the data using
multiple regressions, the ASTD/i4cp team suggests that The least likely metric is actual business outcomes, which
organizations focus on the following three factors: nearly a quarter of respondents said they do not measure
• Follow-up focus groups at all in terms of learning evaluation. Only 22.4 percent
• Performance records monitoring said they use this approach to a high or very high extent.
• Follow-up surveys of participants About 18 percent also said they do not use productivity
Taken as a group, these three measurement metrics at all to evaluate the impact of learning programs,
techniques explain nine percent of the differences
and just 26.1 percent use such metrics to a high or very
in successful evaluation. That is, the more frequently
high extent.
respondents used these measurement approaches, the
more likely they were to report having overall success
with their evaluation process. Such findings strike us as alarming. After all, one of the
primary—if not the primary—goals of learning experienc-
es is to positively influence business outcomes. Everything
Figure 10 | To what degree does your organization use the following

approaches for measuring the behavior and the application or
transfer of information (that is, Level 3 of Kirkpatrick/Phillips model)?
Correlation
Percentage who use these approaches with Evaluation
Responses to a high or very high extent Success Index
Follow-up surveys of participants 31.0% 0.21**
Action planning 26.9% 0.21**
Performance records monitoring 24.3% 0.19**
Observation on the job 23.9% 0.17**
Program follow-up session 18.5% 0.25**
Follow-up surveys of participants’ supervisors 16.5% 0.17**
Interviews with participants 15.8% 0.14*
Interviews with participants’ supervisors 15.2% 0.17**
Follow-up focus groups 9.1% 0.23**
* Correlation is significant at the 0.05 level (2-tailed)
** Correlation is significant at the 0.01 level (2-tailed)

else—from boosting retention and engagement to ensur-
ing workers have solid skill sets—is generally intended to If an Organization Must Choose:
increase business results. Indeed, we were not surprised Which Approaches Work Best for
to find that a multiple regression analysis showed that Level 4 Measurement?
“actual business outcomes” was one of three Level 4 Eight different factors were significantly correlated
outcomes to explain a sizable portion of success when with the Evaluation Success Index. Given the similarity of
it comes to learning evaluations. If companies have to the level of correlations, it can be difficult to choose only
choose three approaches to focus on in terms of Level a few to implement into a learning evaluation program.
4, we recommend actual business outcomes, learning/ So, which approaches, if an organization must choose,
lead to the best evaluation results? Based on a further
employee perceptions of impact, and the proficiency/com-
analysis of the data using multiple regressions, the ASTD/
petency levels of learning recipients.
i4cp team suggests that organizations focus on
the following three factors:
Having said this, we should note that each of the Level 4 • Learner/employee perceptions of impact
approaches we listed in the survey is significantly correlated • Actual business outcomes (e.g., revenue, sales)
to the ESI as well as the Market Performance Index (MPI), • Proficiency/competency levels
with the exception of supervisor perceptions for the MPI. Used together, these three measurement
Employee satisfaction was the metric most highly correlated techniques explain 21 percent of the difference in
with market performance. None of them is likely to be a successful evaluation. That is, the more frequently
waste of time. And a survey of ASTD’s BEST Award-win- respondents used these measurement approaches, the
ning organizations found that those firms use employee and more likely they were to report having overall success
with their evaluation process.
customer satisfaction, quality of products and services, cycle
time, retention revenue, and overall productivity as mea-
sures for effective learning (Murray & Efendioglu, 2007).
Figure 11 | To what degree does your organization use the

following approaches to measure program impact or results
(that is, Level 4 of the Kirkpatrick/Phillips model)?
Percentage who use these ap- Correlation

proaches to a high or very high with Evaluation
Responses extent Success Index
Customer satisfaction 38.9% .33**
Employee satisfaction 37.0% .25**
Learner/employee perceptions of impact 36.3% .33**
Proficiency/competency levels 33.0% .35**
Business leader/supervisor perceptions of impact 31.4% .29**
Productivity indicators (e.g., time, output per employee) 26.1% .31**
Turnover/promotion 24.8% .27**
Actual business outcomes (e.g. revenue, sales) 22.4% .31**
** Correlation is significant to the 0.01 level (2-tailed)

The least likely metric is Holding Managers Accountable
Increases Evaluation Success
actual business outcomes, Supervisors play a key role in training evaluation, but not
many companies use them to their full potential. In fact, a
which nearly a quarter of minuscule 11.4 percent of respondents say their organiza-
tions hold managers accountable for tracking pre- and
respondents said they do not post-training performance to a high or very high extent.
But organizations that do this are much more likely than
measure at all….Such findings others to say their learning evaluation systems are suc-
cessful. Therefore, this uncommon practice may well help
strike us as alarming. organizations boost the performance of their learning
evaluation systems.

It is also important for supervisors to set goals with em-
ployees prior to training, yet more than a quarter of par-
Are training activities simply
ticipants said that supervisors are not responsible for this,
and just 17.4 percent said their companies hold managers
squandered in many of
responsible for this to a high or very high extent.
today’s organizations?
Companies were more likely to make supervisors respon-
sible for giving employees the opportunity to use the
knowledge gained from training, but even here the low
scores are surprising. Only 22.9 percent did this to a high
or very high extent, raising the question of whether train-
ing activities are simply squandered in many of today’s
organizations. Firms in which managers are, in fact, held
accountable for this are more likely than others to enjoy
higher market performance.
Figure 12 | To what extent are supervisors

held accountable for the following:
Correlation
Correlation with Market
Not Small Moderate High Very high with Evaluation Performance
Responses at all extent extent extent extent Success Index Index
For setting goals
with employees prior 26.7% 33.7% 22.3% 13.3% 4.1% .32**
to training
For giving employees

opportunities to 15.1% 29.1% 33.0% 19.0% 3.9% .37** .14**
use new knowledge
after training
For tracking pre-

and post-training 36.5% 32.4% 19.7% 8.8% 2.6% .37**
performance

Note: Only statistically significant correlations are presented.

| section III |
What Are the Barriers to Training Evaluation?
G iven the fact that learning evaluations can have

sizable benefits for organizations, why don’t
companies use them to a greater extent? To
get at this question, our research team asked about the
barriers to the proper measurement of learning.
a significant negative correlation between the extent to
which respondents see this as a barrier and their own
learning evaluation effectiveness (see figure 13).
One possible takeaway is that companies should not try to

prove that a learning experience affects results but rather
We found that the barrier that looms largest is the show that, given the preponderance of evidence, it very
difficulty companies have isolating learning as a factor likely does. Managers and executives should understand this
that has an impact on results. Of course, this tends argument if it’s made carefully, especially if they recognize,
to come into play mostly in Level 4 and Level 5 of as this study indicates, that viewing this as an insurmount-
the Kirkpatrick/Phillips model. It is not a barrier for able barrier does the company more harm than good, in
Level 1, since there are no other factors involved in the terms of both learning evaluation and market performance.
participants’ reactions. Phillips suggests that the effects
of training can be isolated through the use of control The next-most commonly noted barrier, cited by 40.8
groups, trend-line analysis, forecasting models, and percent of respondents, is the lack of a useful evaluation
impact estimates (Phillips, Pulliam, & Wurtz, 1998). function as part of the company’s learning management
system (LMS). This may signal that, as companies rely
We should note, however, that there might be a self- more heavily on software solutions to create and deliver
fulfilling prophecy problem here. Companies that simply learning, they become dependent on those systems to
give up on evaluation because they can’t “prove” it has provide metrics. If their system does have the proper re-
results might well be sabotaging their own learning effec- porting functions, it may not be robust enough to provide
tiveness. After all, data from this study shows that there’s the data they need for effective evaluation.
Figure 13 | To what extent are the following seen as barriers

to the evaluation of learning in your organization?
Percentage who see these as barriers Correlation with Evalu-

to a high or very high extent ation Success Index
Too difficult to isolate training’s impact 51.7% -.18**

on results versus other factors’ influence
Our LMS does not have a useful 40.8% -.21**

evaluation function
Evaluation data is not standardized 38.0% -.23**

enough to compare well across functions
It costs too much to conduct higher-level 32.2%

evaluations
Leaders generally don’t care about 24.1% -.12**

evaluation data
Evaluation data is too difficult to interpret 18.9%

for most people
Evaluations are not seen as credible 14.5% -.20**

section III: what are the barriers to training evaluation? | 25

There might be a
self-fulfilling prophecy
problem here.
Another and sometimes related problem that organizations

encounter is that the learning data they collect is not
standardized across functions and is, therefore, difficult
to compare. This can be both a process and a technology
problem—one that can be difficult to address without the
willing participation of a multifunctional team.
High costs also keep some companies from evaluating

learning effectively, especially when it comes to the higher
levels of the K/P model. This raises the interesting question,
“What is the ROI of calculating ROI?” If companies simply
don’t think the ROI is high enough, then they’ll be more
likely to cut such calculations from the learning budgets.
Apathy toward the data can also be a barrier. Almost a

quarter of respondents said that business leaders who
don’t actually care about the evaluation data represent
a barrier to a high or very high extent. It’s possible that
leaders don’t value the data because there is not much
they can do with participation level and employees’ reac-
tion data, which is what gets measured most. If leaders
knew they were going to get information on how training
improves results and boosts the bottom line, they might
be more interested. On the other hand, some studies
show that leaders can also be skeptical of certain business
results data, especially if the assertions about training’s
impact seem uncertain or slippery.
Other research produces these same recurring themes

when it comes to the difficulties in measuring learning.
The Chief Learning Officer survey cited earlier found
that the most common barriers to effective learning
evaluation included a lack of resources, interest, and
leadership support (Anderson, 2009). Companies often
find it so challenging to isolate training as a factor affecting
bottom-line results that they abandon trying to calculate
ROI altogether, saying that knowing the information is
not worth the time and effort it takes to measure

An LMS system should be The strongest correlation with the ESI occurs when
evaluation data is not standardized enough to compare
purchased or modified with well across functions (r=-.23**). In other words, the
more a respondent says that his or her evaluation data
evaluation usage in mind. can’t be compared easily across functions, the less likely
that respondent is to report organizational success with
overall evaluation efforts.
(Naughton, 2008). Others say that senior management

Another barrier with a strong correlation to the ESI is
isn’t actually interested in it or that they do not accept
“Our LMS does not have a useful evaluation function”
the numbers as valid (Wilhelm, 2007).
(r=-.21**). This negative association suggests that the
more respondents believe that their LMS does not have
A 2004 study by IBM and ASTD asked business executives
an evaluation function that meets their needs, the less
which factors should be used to hold learning departments
likely they are to give their evaluation efforts high marks.
accountable for adding value to the business, and ROI
ranked as one of the least important factors (Redford,
These findings provide insights into specific steps organizations
2007). Another report, this one from the Corporate
can take to avoid learning evaluation pitfalls. First,
University Xchange, says that management often does
learning evaluation data loses some of its value when it
not accept the data that is presented and, in many cases,
cannot be compared across functions. The investment
simply believes that training adds value without asking
of time to coordinate across functions may be seen as
for evaluation data at all (Todd, 2008).
excessive, yet it appears that when organizations fail to
standardize key metrics initially, the overall effectiveness
Another interesting point of view comes from a 2008
of the evaluation process suffers. Additionally, taking steps
WorldatWork survey of European HR leaders. It found
to ensure the usefulness of the evaluation function of an
that ROI for learning was high on the scale for importance
organization’s LMS can be worth the effort.
when it comes to the ROI metrics that HR can report,
ranking them higher than data related to demographics
Generally speaking, an LMS system should be purchased
and diversity, compensation and benefits, productivity,
or modified with evaluation usage in mind. A system
and workplace health and safety. But when it comes to
that does not help learning professionals gain clear
HR’s ability to report those metrics, ROI for training is
insight into the success of their learning programs is not
dead last (Collins & Hill, 2008).
doing its job well enough. If learning professionals are
essentially “stuck” with such a system, then at least they
should make it clear to the vendor or creator that a better
Addressing Evaluation Barriers evaluation function will be expected in the next version
Each of the barriers documented in this study was correlated of the application.
with the Evaluation Success Index (ESI), a measure of the
extent to which respondents believe their learning metrics
are a worthwhile investment of time and resources. The
purpose of these correlations is to understand which
barriers are most strongly associated with successful (or
unsuccessful) evaluation. In short, the more that barriers
are seen as hindering learning evaluations, the less likely a
respondent is to say that his or her organization has a suc-
cessful learning evaluation program.
section III: what are the barriers to training evaluation? | 27

The more respondents say
their leadership doesn’t
hold evaluation metrics in
high regard, the less likely
they are to report strong
market performance across
leading indicators.
In addition to correlations with the ESI, each barrier was

correlated with the Market Performance Index (MPI).
Compared with the ESI, there were fewer significant
correlations, and those that were significant were not as
strong. However, three of the six barriers were found to
be significantly and negatively correlated with the MPI.
The strongest correlation with the MPI (-.12**) was
with the idea that “leaders don’t generally care about
evaluation data.” In this case, the more respondents say
their leadership doesn’t hold evaluation metrics in high
regard, the less likely they are to report strong market
performance across leading indicators. In general, if
specific leaders or the prevailing organizational culture
downplay the value of evaluation data, it is possible that
the quality of decisions and confidence in those decisions
suffer and lead to compromised market performance.

| section IV |
What Are Companies Spending on Training Evaluation?
L earning budgets vary widely across respondent

companies, with some reaching tens of millions
of dollars per year. The proportion of the budget
spent on evaluating learning is relatively consistent regardless
There are instances when
participants will love the
of company size, although small companies, those with
fewer than 100 employees, spend an average of 6.5 percent
training sessions because
of their learning budget on evaluation, whereas large
companies, those with more than 10,000 employees,
they are fun and exciting,
allocate 4.4 percent of the learning budget to evaluation.
but there is no change in
About two-thirds of that evaluation budget is then allocated
mostly to internal resources, while the rest is spent
behavior at all.
externally. Large companies spend more internally (72.4
percent) than small companies (58.2 percent), most likely
because they have the resources to do so. Small companies
may need to rely on more outside assistance to conduct
any evaluation.
Figure 14 | Please indicate what percentage of your organization’s

total expenditure for employee learning is allocated to evaluation
Small (fewer than 100 employees) 6.5%
Mid-size (from 100 to 10,000 employees) 5.7%
Large (10,000 or more employees) 4.4%
All organizations 5.5%
section iV: what are companies spending on training evaluation? | 29

The clear front-runner, Measuring the reaction of learners is probably the least
expensive endeavor of all five levels, yet that is where the
based on the percentage most money is being spent simply because it is done the
most. Of course, as we’ve noted elsewhere in this study,
of evaluation spending, is Level 4 and Level 5 evaluations are done much less fre-
quently and for a smaller range of programs than lower
Level 1 evaluation. levels of evaluation, so it makes sense that it is seen as

being a much smaller portion of total expenditures.
However, this raises the issue of whether such upper-

Where Companies Are Spending level evaluations are really as expensive as is commonly
Their Budgets thought. Jointly, Level 4 and Level 5 are only 10.6 percent
When companies use the Kirkpatrick/Phillips model, they of all evaluation spending.
tend to spend their budgets in parallel to how they use
the levels. Nearly half of the budget is spent on Level 1,
the reaction of the participants, where the majority of
programs are evaluated. From there, the spending drops
by about half for each subsequent level until it reaches
three percent for Level 5.
Figure 15 | Of the total expenditure for learning evaluation,

what percentage is spent on:
Outside products/services, consultants, workshops, and
external training programs
Internal resources such as learning staff salaries, administrative
costs, development costs, and internal IT costs
41.8%
Small organizations
58.2%
30.9%
Mid-size organizations
69.1%
27.6%
Large organizations
72.4%
31.8
All organizations
68.2%

Improper Spending evaluating something is better than doing nothing, result-
ing in a significant positive correlation. In this case, how-
Although the budget allocations for Kirkpatrick/Phillips
ever, a strong inverse relationship exists between Level 1
make sense based on how companies use the model, they
spending and both indices.
don’t seem to make as much sense for executing effective
evaluations or even for having a successful company.
This is not suggesting, however, that Level 1 measurement is
an inherently flawed measure or investment of time. More
As mentioned above, the clear front-runner based on
likely, these results suggest that spending on Level 1
the percentage of evaluation spending, is Level 1 evalua-
evaluation is not effective unless it is combined with
tion. Almost 50 percent of evaluation budgets are spent
other levels.
on measuring the reactions of participants to learning
programs. Despite the fact that this method is used most
Clearly, asking how participants feel about the training
frequently, the correlations point to problems. Specifi-
alone may not gather any good, substantive informa-
cally, the larger the percentage of the evaluation budget
tion. Ian Cunningham, visiting fellow at the University
spent on participant reactions, the less likely our survey
of Sussex in the United Kingdom, says he has seen many
respondents were to score well on the Evaluation Success
examples where participants gave bad reviews to the
Index (ESI) (r=-.38). They were also more likely to score
training in Level 1 evaluations but their behavior had
lower in terms of their market performance (r=-.12).
been positively changed when measured at Level 3
(Cunningham, 2007).
Finding no relationship at all between an investment
in evaluation and the ESI would be surprising—surely
Figure 16 | Of the total expenditure for learning evaluation,

what percentage is spent on the following:
Correlation
Percentage who see these as barriers to with Evaluation
Responses a high or very high extent Success Index
Reaction of participants 49.8% -.38**
Evaluation of learning 22.8% .13**
Evaluation of behavior 14.3% .25**
Evaluation of results 7.6% .32**
Return on investment 3.0% .26**

Other forms of individual 2.5%
learning program evaluation
section iV: what are companies spending on training evaluation? | 31

Level 4 appears to deliver
the greatest value.
The opposite can be true as well. There are instances when

participants will love the training sessions because they
are fun and exciting, but there is no change in behavior at
all. Our study results suggest that many companies would
never know if their training was a failure because they
never went any further than Level 1 and were satisfied
that the employees gave the program high marks. It
seems clear that companies using predominantly Level 1
measurements are less successful in both their evaluation
efforts and their business results.
Interestingly, the correlations between the ESI and every

other level of measurement are significant and positive.
This data suggests that the larger the percentage of the
evaluation budget that is spent on any level except
Level 1, the more likely respondents are to report
overall evaluation success.
This seems to be especially true for Level 4, the evaluation

of results. Not only does evaluation of results have the
highest correlation with the ESI (r=.32**), but it also has the
only significant positive correlation with the MPI (r=.11**).
According to these results, regardless of the other levels an
organization chooses to measure, the most value appears to
be gained by evaluation of training results.

| section V |
How Can Organizations Improve Training Evaluation?
I dentifying which programs to evaluate, determining

how extensively to measure them and conducting the
actual evaluations are really only half of the equation.
Once the metrics have been acquired, what can a company
It is only through proper
application of the data
do with the information that has been gathered?
that evaluation can be
It is only through proper application of the data that
evaluation can be considered a success. To that end, the
considered a success.
survey asked respondents what actions are taken based
on the information derived from evaluation. Out of a
It is interesting that, in every case, companies clearly
list of 14 possible uses for evaluation results, no action
believe they should be taking these actions to a higher
is taken to a high or very high extent by a large majority
extent than they actually are. The biggest gap between
of respondents.
what companies are doing and what they should be doing
comes with calculating the effect that learning has on
The action taken to the highest extent involves helping
business results, where only 17.5 percent said they are
improve learning programs, although it is a bit puzzling
doing this to a high or very high extent and 85.8 percent
why this number isn’t even higher. It is hard to imagine
said they should be doing it.
the reason an organization wouldn’t use such evaluation
results to improve its learning programs.
Figure 17 | To what extent does your organization use learning

evaluation results to take each of the following actions?*
To help improve learning programs 52.9%
To gather performance data about instructors 47.8%
To make sure employees like the programs 47.4%
To satisfy legal requirements in a regulated industry 46.9%

To gauge whether or not employees are learning 39.1%
what’s required of them
To help meet the performance goals of employees 36.2%
To ensure that learning programs positively 33.1%
influence the behaviors of employees
To satisfy management that the training function 32.6%
is doing its job
To ultimately improve overall business results 31.4%
To demonstrate to others in the organization the 27.8%
value of the learning function
* Percentage who answered they do this to a high or very high extent
section v: how can organizations improve training evaluation? | 33

This correlates to the fact that so few companies evaluate
Which Uses of Evaluation Results at Level 4 (results) to any extent. It appears they know
Are Most Important? they should be evaluating at Level 4 more often in order
Fourteen different common actions were correlated to measure learning’s effect on business results.
with the Evaluation Success Index. Most companies can’t
organize the staff or the resources necessary to work on In contrast, it seems companies are satisfied with their
14 different objectives at once, even if each individually use of participant reaction measurement (Level 1). There
is a worthwhile goal. Therefore, the goal of the ASTD/ was only an 8.6 percentage-point difference between
i4cp team was to narrow the list to only those factors respondents who said they use evaluations to make sure
that had the strongest independent impact on successful
employees like the programs to a high or very high extent
evaluation. By further analyzing the data using multiple
and those that said they should do so.
regressions, the results suggest that organizations focus
on the following three factors:
• To ultimately improve overall business results Making sure employees like programs was one of the most
• To demonstrate to others in the organization common evaluation actions taken—most of the average
the value of the learning function company’s evaluation time, energy, and money are spent on
• To ensure that learning programs positively it. Ironically, Level 1 is actually seen as the least important.
influence the behaviors of employees.
Taken as a group, these three factors explain 31 Respondents also said their organizations were not calcu-
percent of the differences in successful evaluation. That lating learning’s ROI enough, with only 10 percent saying
is, the more strongly respondents agreed with these they calculated ROI to a high or very high extent and
statements, the more likely they were to report that
70.6 percent saying they should. Again, this is borne out
their overall evaluation programs were a success in
by the small percentage of programs evaluated at Level 5
their organizations.
and the small portion of budgets allocated to that area.
Figure 18 | To what extent does your organization use

learning evaluation results to take each of the following actions,
and to what extent should it?*
To calculate the effect that learning has on
important business results 68.3%
To ultimately improve overall business results 62.8%
To gauge the ROI of learning programs 60.5%

To ensure that learning programs positively
influence the behaviors of employees 58.7%
To demonstrate to others in the organization
the value of the learning function 55.5%
To determine whether training is cost-effective 49.6%

To gauge whether or not employees are
learning what’s required of them 48.6%
To help meet the performance goals
of employees 47.6%
To help improve learning programs 40.2%

To take advantage of the built-in functionality
of a learning management system 40.2%
* Percentage point difference between the degree to which companies actually take these actions
to a high or very high extent and the degree to which they say they should do so

Study participants also wish their companies were using
learning evaluation more to improve business results,
Until these types of issues
ensure learning programs positively influence employee
behavior, and demonstrate the value of learning to others
are addressed, behaviors
in the organization. These are actions that are easier to
undertake when the proper evaluation is done, whether
will not change and there’s
it is Level 3 for behaviors, Level 4 for results, or the
Brinkerhoff Success Case Method for demonstrating the
little chance that a learning
value of learning.
experience can have a
Priorities for Improvement positive impact on
We believe that the practices with the five largest “should/ business results.
do” gaps—all with percentage-point differences exceed-
ing 50—provide major clues about what learning profes-
sionals can and should be focusing on in the near future. That is, behaviors aren’t really changed by training alone.
Here are the top five: In fact, success is often related to how well managers fol-
• Calculate the effect that learning has on important low up on the training with their employees. Do they allow
business results. the training to be put to use? Are workers encouraged to
use it? Until these types of issues are addressed, behaviors
• Ultimately improve overall business results.
will not change, and there’s little chance that a learning
• Gauge the ROI of learning programs. experience can have a positive impact on business results.
• Ensure that learning programs positively influence
the behaviors of employees. The fifth action that has a very large should/do gap is
• Demonstrate to others in the organization the value “demonstrate to others in the organization the value
of the learning function. of the learning function.” That sounds self-serving for
learning professionals, and perhaps it is to some extent.
But there’s also a legitimate need to demonstrate and
Note that the first three of these are related to business
reinforce the most effective learning experiences because
results and one to ROI in particular. It’s clear that, in
those will be one of the best means of ratcheting up the
general, respondents believe their firms should be doing
skill levels and effectiveness of the organization over time.
a better job of using learning evaluations to improve busi-
ness results. There needs to be a plan to accomplish this
Our gap analysis of these five areas is buttressed by a
in many of today’s organizations.
multiple regression analysis showing that three out of
these five practices (overall business results improvement,
But learning and learning evaluations can’t do much to
positive influences on employee behaviors, and demon-
improve business results until they change and track the
strating the value of the learning function) do much to
behaviors of employees. “Training that is well learned,
help explain the success of learning evaluation. If compa-
but never used, or poorly used, produces no value for
nies want to make their learning evaluation systems bet-
the business that invested in the training,” Brinkerhoff
ter, these items, in particular, may be great places to start.
has noted. He cites research estimates claiming that only
about 15 out of 100 people ever use new training to pro-
duce valuable performance results (Brinkerhoff, 2006).
section v: how can organizations improve training evaluation? | 35

| conclusion and policy recommendations |
I t’s clear that learning professionals want to demon-

strate the value of the programs they deliver and at
the same time improve those programs. They want to
deliver programs that employees enjoy, impart knowledge
and skills, change behaviors for the better, deliver busi-
completed. These are the basics of learning evaluation, of
course, but this study clearly shows that the basics aren’t
all that common in today’s organizations.
ness results, and provide the best bang for the buck. Recommended Actions
This study has resulted in excellent insights into the
But organizations are having a hard time determining if current state of learning evaluation. But, having gone
their programs do these things, and that makes it hard to through the findings, what are the main takeaways that
improve the learning function. There are strategies avail- learning professionals can quickly act on? Every orga-
able to achieve this goal, but the information has to be nization’s situation will be different, of course, and not
collected properly and used effectively. every practice will work equally well in all of them. Given
those caveats, below are our suggestions for those intent
The type of evaluation that is done is greatly influenced by on improving learning evaluations:
the audience for the data. Instructors are mostly focused
on employees’ reactions and the amount of learning that • Collect data that is meaningful to leaders. Otherwise,
takes place, which allow them to critique and improve they will never see the value of evaluations.
their programs. Yet business leaders are not as interested in • Where possible, standardize evaluation data across
that information. They want to know how performance, different functions within the organization to make
productivity, and other business metrics have improved as it easier to use the data effectively.
a result of the training (Berge, 2008). With so much focus • Spend more time and money evaluating behaviors
on Levels 1 and 2, it’s not surprising that many leaders do and results and less on participant reactions. At the
not buy into conventional learning evaluation. very least, don’t rely on reactions so completely.
• Give supervisors more responsibility when it comes
To obtain valuable learning metrics, one main question
to learning evaluation training. They should give
has to be answered: What is the purpose of the train-
employees opportunities to use their training, and
ing? The answer to that question may not be as simple
they should help track performance both prior to
as “to improve performance.” Training can be used for
and after the training.
any number of additional reasons, whether to change be-
haviors, to increase retention, or for motivation (Rands, • When evaluating changes in behavior, use strate-
2007). Whatever the reason, it becomes the guide by gies such as follow-up sessions, focus groups, and
which the learning is measured. participant surveys. Used with action planning and
performance monitoring, these strategies are the
There should be clear objectives and goals to be measured most highly correlated with evaluation success.
from the outset of the training program. Trying to identify
things to measure after the fact can be difficult and not
entirely effective. If evaluating at Level 3, for example, the
behaviors that are to change need to be identified and mea-
sured before and after the training. The business results
identified in Level 4 need to be measured prior to learning
to provide a baseline for comparison after the program is

• When evaluating the impact that training has had on • When choosing a learning management system,
results, find some ways to help filter out factors other investigate the evaluation tools available. A robust
than training that may have an influence. But also solution can make evaluation much easier and the
keep in mind that organizations are not laboratories results more reliable. In addition, it becomes a way
where all the factors can be controlled. The learning to audit the effectiveness of the LMS itself.
function should not be held to a higher standard • Most important, don’t simply give up on evaluation.
than others. What learning professionals should be While learning metrics must be used wisely to be ef-
trying to show is that a preponderance of evidence fective, simply abandoning them is no solution. After
indicates that learning has an impact. Leaders should all, using metrics well is associated not only with
not try to hold the learning function to a “beyond evaluation success but also with overall organiza-
a shadow of a doubt” level of evidence. If they do, tional success.
then evaluations are likely to fail, and this is associ-
ated with weaker organizations.
• When evaluating results, focus on metrics such as
proficiency and competency levels, customer satisfac-
tion, and employee perceptions of training impact.
Actual business outcomes and productivity measures
such as time and employee output are also metrics
worth capturing. In any case, prior to training, iden-
tify the key performance indicators to be measured.
conclusion and policy recommendations | 37

| references |
Anderson, C. (2009, May). “Overcoming Analysis Paralysis.” Naughton, J. (2008, September). “IOL: Determining the
Chief Learning Officer, 54-56. Impact of Learning.” Chief Learning Officer, 32-36.
ASTD (American Society for Training & Development). Platt, G. (2008, August). “The Hard Facts About Soft
(2005, November). ASTD Benchmarking Forum 2005 Skills Measurement.” Training Journal, 53-56.
Learning Evaluation Practices Report. Alexandria, VA.
Phillips, J., P. Pulliam Phillips, and R. Stone. (2001).
Berge, Z. (2008). “Why It Is so Hard to Evaluate Train- The Human Resources Scorecard. Boston:
ing in the Workplace.” Industrial and Commercial Butterworth-Heinemann.
Training, 390-395.
Phillips, J., P. Pulliam, and W. Wurtz. (1998, May).
Brinkerhoff, R. (2006). “Increasing Impact of Training “Level 5 Evaluation: Mastering ROI.” Infoline.
Investments: An Evaluation Strategy for Building
Organizational Learning Capability.” Industrial and Radhakrishnan, M. (2008, October). “Learning
Commercial Training, 302-307. Measurements: It’s Time to Align.” Chief Learning
Officer, 36-39.
———. (2003). The Success Case Method. San Francisco:
Berrett-Koehler. Rands, A. (2007, February). “Extending the Half-Life of
Training.” Training Journal, 40-43.
———. (2005, February). “The Success Case Method:
A Strategic Evaluation Approach to Increasing the Redford, K. (2007, June). “What’s the Point of ROI?”
Value and Effect of Training.” Advances in Developing Training & Coaching Today, 12-13.
Human Resources, 86-101.
Todd, S. (2008, September). “Mission Accomplished?
Collins, M., and Hill, S. (2008, May). “Trends in Measuring Success of Corporate Universities.” Chief
European Human Capital Measurement.” workspan, Learning Officer, 38-40.
64-70.
Wilhelm, W. (2007, December). “Does Senior Leadership
Cunningham, I. (2007). “Sorting Out Evaluation of Learning Buy ROI for Learning?” Chief Learning Officer,
and Development: Making It Easier for Ourselves.” 86-88.
Development and Learning in Organizations, 4-6.
“Is Training Worth It?” (2008). Development and

Learning in Organizations, 34-36.
Kirkpatrick, D. (1994). Evaluating Training Programs:

The Four Levels. San Francisco: Berrett-Koehler.
———. (2007, January). “The Four Levels of Evalua-

tion.” Infoline.
Murray, L., and Efendioglu, A. (2007). “Valuing the

Investment in Organizational Training.” Industrial
and Commercial Training, 372-379.
References | 39
| Appendix |
The Value of Evaluation Survey Overview
Survey Process
Target Survey Population
The target survey population of the ASTD/i4cp Value of Evaluation Survey consisted of an email list of primarily high-
level business, HR, and learning professional contacts from ASTD and i4cp. In total, 704 people responded to the survey.
Respondents represented a variety of organizational sizes and industries.
Survey Instrument
In this survey, multiple questions used the well-accepted 1–5 Likert-type scale, with a 1 rating generally designated as “not
at all” and a 5 rating as “a very high extent.” Additionally, several questions asked respondents to assign percentages (for
example, percentage of learning programs are evaluated at each level) in their organizations. There were 38 questions in
all, including those geared toward the demographics of respondents.
Some questions were open-text types, and that data does not appear below. Moreover, survey data from multiple questions
was combined in certain tables. In a few cases, low-priority data is not shown at all in this appendix.
Procedure
A link to an online survey was emailed to the target population during May 2009.
Demographic/Company Profile Questions and Results

Q1: What is your current title?
Respondents represented a broad array of mid- to high-level management positions, with more than half of those surveyed
at the director level or above. Directors make up the largest proportion of those respondents (43.2 percent).
What is your current title?

Responses Percent
CEO/President/Chairman 4.8%
Executive VP/Senior VP 2.8%
Vice President 9.8%
Director 43.2%
Chief Learning Officer 3.3%
Manager 20.9%
Supervisor 0.9%
Other 14.4%
value of evaluation survey overview | 41

Q2: Describe your organization’s type of operation.
Forty percent of survey respondents represented firms that were multinational (defined as operations that act independent-
ly of one another) or global (defined as having a high level of global integration) in nature. Fifty-nine percent of respon-
dents were from national organizations that were currently operating in one country only.
Describe your organization’s type of operation.

Responses Percent
National 59.4%
Multinational 17.6%
Global 22.9%
Q3: In what region is your organization headquartered?

A large majority of the respondents were from organizations headquartered in North America (82.7 percent). Participants
also included firms headquartered in Europe (7 percent) and Asia (6 percent), with smaller representations among firms
headquartered in Oceania/Australia, Latin America, the Mideast, and Africa.
In what region is your organization headquartered?

Responses Percent
North America 82.7%
Latin America 1.1%
Europe 6.6%
Mideast 2.4%
Africa 0.7%
Asia 5.7%
Oceania/Australia 0.9%

Q4: In what sector does your organization operate?
Respondents represented a wide variety of industries. The largest specified proportion was employed in
health care (12.2 percent).
Within which sector does your organization primarily work?

Responses Percent
Aerospace & Defense 1.3%
Agriculture 0.4%
Automotive & Transport 1.1%
Banking 3.3%
Beverages 0.1%
Business Services 5.6%
Charitable Organizations 0.1%
Chemicals 1.1%
Computer Hardware 0.4%
Computer Services 1.4%
Computer Software 1.7%
Construction 2.5%
Consumer Products Manufacturers 2.5%
Consumer Services 0.5%
Cultural Institutions 0.0%
Education 8.9%
Electronics 0.6%
Energy & Utilities 2.7%
Financial Services 8.1%
Food 3.0%
Foundations 0.0%
Government 4.5%
Health Care 12.2%
Industrial Manufacturing 4.0%
Insurance 6.3%
Leisure 1.1%
Media 1.0%
Membership Organizations 0.6%
Metals & Mining 0.4%
Not-for-profit 6.0%
Pharmaceuticals 2.5%
Real Estate 0.6%
Retail 2.5%
Security Products & Services 0.2%
Telecommunications Equipment 0.4%
Telecommunications Services 1.6%
Transportation Services 1.9%
Other 8.9%

Q5: What is the size of your organization’s workforce?
Organizations of all sizes responded to the survey. More than half of respondents are employed by organizations
with workforces of 1,000 or more, while about a quarter work in organizations with more than 10,000 employees.
What is the size of your organization’s workforce?

Responses Percent
Fewer than 100 employees 16.2%
100-499 17.6%
500-999 11.9%
1,000-3,499 17.7%
3,500-4,999 4.5%
5,000-9,999 6.7%
10,000-24,999 9.6%
25,000-49,000 6.7%
50,000-99,999 4.8%
100,000 or more 4.2%
Q6: What is your organization’s total revenue?

Participating organizations reported a wide range of revenue. The largest single revenue category noted revenues less than
$10 million. Nonetheless, respondents were well distributed among total revenue categories, with just more than three out
of 10 representing firms with revenues of $1 billion or more.
What is your organization’s total revenue?

Responses Percent
Less than $10 million 20.5%
$10 to $24 million 7.9%
$25 to $49 million 9.1%
$50 to $99.9 million 8.6%
$100 to $249 million 9.1%
$250 to $499 million 6.2%
$500 to $999 million 6.9%
$1 billion to $2.99 billion 11.1%
$3 billion to $9.99 billion 9.8%
$10 billion or more 10.7%

Performance Question and Results
Q7: Compared with the past five years, how would you rate your company’s performance now?
Most participating organizations characterized their performance in the areas of revenues, market share, profitability, and
customer satisfaction as holding steady or improving somewhat over the past five years. These four factors comprise the
Market Performance Index, which is the basis for grouping companies into high, medium, or low performers. Cronbach’s
alpha, a statistical measure of internal reliability for a scale, is .81 for the Market Performance Index, indicating good
consistency for the scale.
When compared to the past five years, how would you rate your company’s performance now?
Don’t know/
Responses Not applicable All-time low Worse Same Better All-time high
Revenue growth 10.3% 4.2% 28.6% 20.5% 30.8% 5.6%
Market share 13.4% 0.4% 11.6% 34.1% 36.5% 4.0%
Profitability 13.5% 2.9% 26.5% 25.0% 28.7% 3.4%
Customer satisfaction 7.3% 0.4% 4.5% 38.8% 41.5% 7.5%
Metrics Questions and Results

Q8: Please state the degree to which you agree with the following statements.
Only a minority of survey participants were in agreement with statements indicating that their learning metrics techniques
are successful. The rest were neutral or in disagreement. These three questions were designed to form an index of the effec-
tiveness of an organization’s evaluation techniques, which we refer to as the Evaluation Success Index (ESI). Overall, the
scores to these three items are normally distributed, and the Cronbach’s alpha for the index is .88, indicating a high degree
of internal reliability.
Please state the degree to which you agree with the following statements.
Not Strongly Strongly
Responses applicable disagree Disagree Neutral Agree agree
We get a solid “bang for our
buck” when it comes to using 8.4% 6.8% 25.6% 33.7% 22.0% 3.6%
learning metrics.
Our learning evaluation techniques
help us meet our organization’s 5.0% 5.4% 24.4% 23.9% 34.9% 6.4%
learning goals.
Our learning evaluation techniques 5.4% 7.4% 24.0% 26.7% 30.8% 5.7%
help us meet our business goals.

Q9: Please select all of the Kirkpatrick/Phillips levels that you use to any extent in your organization.
Of study participants who use the Kirkpatrick/Phillips model, the most common level of measurement is the reactions of
participants (Level 1), with 91.6 percent of companies evaluating learning with this technique.
Please select all of the Kirkpatrick/Phillips levels that you use to any extent in your organization.
Percentage who use the corresponding level to any extent
Small Midsize Large All
Responses organizations organizations organizations organizations
Reactions of participants (Level 1) 88.9% 89.7% 97.8% 91.6%
Evaluation of learning (Level 2) 72.6% 79.0% 90.4% 80.8%
Evaluation of behavior (Level 3) 54.7% 50.6% 63.5% 54.6%
Evaluation of results (Level 4) 40.2% 31.8% 46.6% 36.9%
Return on investment (Level 5) 26.5% 13.4% 22.5% 17.9%
None of the above 5.1% 5.1% 1.1% 4.1%
Q10: Do you use other forms of learning program evaluation aside from the five levels described in
the Kirkpatrick/Phillips model?
The Kirkpatrick/Phillips model is clearly the preferred model for learning evaluation, with 82.7 percent of respondents
reporting that they use this model exclusively. However, this data should be interpreted cautiously because responses to
subsequent questions indicate there may be greater diversity of evaluation techniques than is apparent from this question.
In particular, the results from Q12 indicate nearly half of the respondents conduct evaluation studies, so there appears to
be some confusion about what is technically included in the K/P model.
Do you use other forms of learning program evaluation aside from the five levels
described in the Kirkpatrick/Phillips model?
Responses Percentage
Yes 17.3%
No 82.7%

Q11: In instances where you use it, how much value do you think each of the following types or levels
of evaluation has for your organization?
For those organizations using the Kirkpatrick/Phillips model at these levels, evaluation of results (Level 4) and evaluation
of behavior (Level 3) have the highest value to organizations.
In instances where you use it, how much value do you think each of the following types or levels of
evaluation has for your organization?
A little Some High Very high

Responses No value value value value value
Reactions of participants (Level 1) 1.2% 14.2% 48.6% 22.8% 13.1%
Evaluation of learning (Level 2) 0.5% 4.2% 40.4% 38.1% 16.8%
Evaluation of behavior (Level 3) 1.3% 2.8% 21.0% 46.4% 28.6%
Evaluation of results (Level 4) 2.8% 3.4% 18.8% 35.7% 39.3%
Return on investment (Level 5) 8.6% 6.0% 25.9% 26.7% 32.7%
Q12: Has your organization ever conducted an evaluation study (for example, interviews)
with successful trainees?
Responses are divided almost equally between organizations that have and have not conducted an evaluation study with
successful trainees. Almost 48 percent of survey participants say they have used this as a learning evaluation method.
Although this does not necessarily contradict responses to Q10, it does show that many companies are doing more than
just checking learning assessments. And it may reveal that there is not a clear understanding of all the techniques in the
K/P model. In a sense, the companies that are conducting evaluation interviews with successful trainees are using aversion
of the Brinkerhoff Success Case Method.
Has your organization ever conducted an evaluation study (e.g., interviews) with successful trainees?
Percentage of total
Responses respondents
Yes 47.7%
No 52.3%
While this data tells us that respondents are almost equally divided in their use of the Brinkerhoff method, it does not tell
us about the potential effectiveness of this method. To answer that question, a one-way analysis of variance (ANOVA)
was conducted. The mean scores of respondents on the Evaluation Success Index were compared for those who answered,
“Yes, we have conducted an evaluation study” to those who answered that they had not. The group that answered “yes”
had an average of 3.2 compared to the “no” group, whose average score was 2.8. This difference was significant (F=34.9,
p<.01), which indicates that those respondents who have conducted evaluation studies report greater evaluation success.
No significant difference was found between the two groups for the Market Performance Index.

Q13: In regard to those evaluation studies of successful trainees, please state whether your
organization has taken the following actions (select all that apply):
Of those organizations that have conducted evaluation studies, the most popular action they have taken is to develop
more effective learning products and services (76.5 percent). Those companies that take all three of these actions seem to
be using the Brinkerhoff Success Case Method as it was intended to be used. Among those who answered yes to Q12, 23.9
percent take all three of these actions. In addition, the more of these actions they take, the more likely they are to report
learning evaluation success.
In regard to those evaluation studies of successful trainees, please state whether your organization
has taken the following actions (select all that apply):
Percentage of companies that have

Based on those evaluation studies: taken the following actions
We have taken stories of the positive business impacts of learning 59.7%

programs and disseminated them throughout the company.
We have tracked the factors that enhance or impede business impact. 36.5%
We have developed more effective learning products and services. 76.5%
In regard to those evaluation studies of successful trainees, please state whether your organization
has taken the following actions (select all that apply):
Based on your evaluation studies,

how many of the previously Percentage of companies Correlation with Correlation with
mentioned actions has your that have taken the Evaluation Success Market Performance
company taken? following actions Index (ESI) Index (MPI)
All of the three 23.9%
Two of the three 30.3%
r=.29** r=.14**
One of the three 40.3%
None of the three 5.5%
**Correlation is significant at the 0.01 level (2-tailed)

Q14: If your organization uses Level 3 learning metrics (that is, the evaluation of behaviors),
when does it regularly take measurements? (select all that apply)
For survey participants using Level 3 metrics, evaluations are most likely to occur within two weeks to two months after
learning has taken place (55.8 percent). Thirty-nine percent of respondents typically assess baseline learning before the
learning solution has occurred.
If your organization uses Level 3 learning metrics (that is, the evaluation of behaviors),
when does it regularly take measurements? (select all that apply)
Percentage of companies that have

Responses taken measurements at each time
Short-term post-learning intervention (two weeks 55.8%

to two months after intervention/learning)
Long-term post-learning intervention (more than 51.7%

two months after intervention/learning)
Prior to learning intervention 39.3%
Immediately post-learning intervention (within 37.9%

first two weeks after intervention/learning)

Q15: To what extent does your organization use learning evaluation results to take each of the
following actions, and to what extent should it?
Participants reported that their organizations are most likely to use learning evaluation results to improve learning
programs (52.9 percent do this to a high or very high extent). The action that respondents most frequently believe their
organization should take in response to learning evaluation results is to improve overall business results (94.2 percent).
The largest gap between what is done and what participants believe should be done is calculating the effect that learning
has on important business results (68.3 percent).
To what extent does your organization use learning evaluation results to take each
of the following actions, and to what extent should it?
Percentage Percentage who

who answered answered they Percentage-
they DO this to SHOULD do point difference
a high or very this to a high or between DO and
Responses high extent very high extent SHOULD do
To help improve learning programs 52.9% 93.1% 40.2%
To gather performance data about instructors 47.8% 80.0% 32.2%
To make sure employees like the programs 47.4% 56.0% 8.6%
To satisfy legal requirements in a regulated industry 46.9% 60.3% 13.4%
To gauge whether or not employees are learning 39.1% 87.7% 48.6%

To help meet the performance goals of employees 36.2% 83.8% 47.6%
To ensure that learning programs positively 33.1% 91.8% 58.7%

To satisfy management that the training function 32.6% 59.6% 27.0%
is doing its job
To ultimately improve overall business results 31.4% 94.2% 62.8%
To demonstrate to others in the organization 27.8% 83.3% 55.5%

the value of the learning function
To determine whether training is cost-effective 19.7% 69.3% 49.6%
To calculate the effect that learning has on 17.5% 85.8% 68.3%

important business results
To take advantage of the built-in functionality 17.5% 57.7% 40.2%
of a learning management system
To gauge the ROI of learning programs 10.1% 70.6% 60.5%

Q16: To what extent does your organization use learning evaluation results to take each of the
following actions?
Of the various actions that can be taken in response to learning evaluation results, those organizations that use measure-
ment outcomes to improve overall business results had the strongest correlation with the Evaluation Success Index (r=.52)
and with the Market Performance Index (r=.15).
To what extent does your organization use learning evaluation results to take
each of the following actions?
Percentage who Correlation Correlation

answered they DO with Evaluation with Market
this to a high or Success Index Performance
Responses very high extent (ESI) Index (MPI)
To ultimately improve overall business results 31.4% .52** .15**
To ensure that learning programs positively 33.1% .46** .12**

To calculate the effect that learning has on 17.5% .45** .11*
important business results
To gauge whether or not employees are learning 39.1% .43** .11*
To demonstrate to others in the organization the 27.8% .41** .10*
value of the learning function
To help improve learning programs 52.9% .37** .09*
To determine whether training is cost-effective 19.7% .37**
To help meet the performance goals of employees 36.2% .35** .10*
To gauge the ROI of learning programs 10.1% .32**
To satisfy management that the training function 32.6% .31**

is doing its job
To take advantage of the built-in functionality 17.5% .29**
of a learning management system
To gather performance data about instructors 47.8% .25**
To make sure employees like the programs 47.4% .12**
To satisfy legal requirements in a regulated industry 46.9%
*Correlation is significant at the 0.05 level (2-tailed)


Q17: To what degree does your organization use the following approaches for measuring the behavior
and the application or transfer of information (that is, Level 3 of the Kirkpatrick/Phillips model)?
The most popular method of conducting Level 3 evaluations is using follow-up surveys of participants (31.0 percent). Of
the various approaches for measuring behavior changes, those most strongly correlated to the Evaluation Success Index
are using a program follow-up session (r=.25) and using follow-up focus groups (r=.23). None of the approaches were
significantly related to the Market Performance Index.
To what degree does your organization use the following approaches for measuring the behavior and
the application or transfer of information (that is, Level 3 of the Kirkpatrick/Phillips model)?
Percentage who use Correlation with

these approaches to a Evaluation Success
Responses high or very high extent Index (ESI)
Follow-up surveys of participants 31.0% .21**
Action planning 26.9% .21**
Performance records monitoring 24.3% .19**
Observation on the job 23.9% .17**
Program follow-up session 18.5% .25**
Follow-up surveys of participants’ supervisors 16.5% .17**
Interviews with participants 15.8% .14*
Interviews with participants’ supervisors 15.2% .17**
Follow-up focus groups 9.1% .23**
To what degree does your organization use the following approaches for measuring the behavior and
the application or transfer of information (that is, Level 3 of the Kirkpatrick/Phillips model)?
Small Some High Very high

Responses Not at all extent extent extent extent
Follow-up surveys of participants 6.4% 20.5% 42.1% 20.5% 10.4%
Follow-up surveys of participants’ supervisors 15.8% 25.9% 41.8% 13.5% 3.0%
Observation on the job 18.2% 21.2% 36.7% 18.2% 5.7%
Interviews with participants 16.5% 29.6% 38.1% 11.8% 4.0%
Interviews with participants’ supervisors 21.2% 30.3% 33.3% 11.8% 3.4%
Follow-up focus groups 35.0% 33.3% 22.6% 7.4% 1.7%
Action planning 19.2% 22.6% 31.3% 20.5% 6.4%
Program follow-up session 19.9% 26.6% 35.0% 15.8% 2.7%
Performance records monitoring 27.6% 21.9% 26.3% 18.9% 5.4%

Q18: To what degree does your organization use the following approaches to measure program
impact or results (that is, Level 4 of the Kirkpatrick/Phillips model)?
The Level 4 measurement techniques that are used to the greatest extent by survey participants are assessments of customer
satisfaction (38.9 percent) and employee satisfaction (37.0 percent). The method of measuring program impact/results
most strongly correlated to the Evaluation Success Index was assessment of proficiency or competency levels (r=.35).
Measuring employee satisfaction was most strongly associated with the Market Performance Index (r=.23).
To what degree does your organization use the following approaches to measure program impact
or results (that is, Level 4 of the Kirkpatrick/Phillips model)?
Percentage who use Correlation with Correlation with

these approaches to a Evaluation Success Market Performance
Responses high or very high extent Index (ESI) Index (MPI)
Customer satisfaction 38.9% .33** .16**
Employee satisfaction 37.0% .25** .23**
Learner/employee percep- 36.3% .33** .12*

tions of impact
Proficiency/competency levels 33.0% .35** .16**
Business leader/supervisor 31.4% .29**

perceptions of impact
Productivity indicators (e.g., 26.1% .31** .12*
time, output per employee)
Turnover/Promotion 24.8% .27** .14*
Actual business outcomes 22.4% .31** .12*

(e.g., revenue, sales)

Q18: Continued
To what degree does your organization use the following approaches to measure program impact
or results (that is, Level 4 of the Kirkpatrick/Phillips model)?
Small Some High Very high

Responses Not at all extent extent extent extent
Actual business outcomes 23.4% 26.1% 28.1% 18.8% 3.6%
(e.g., revenue, sales)
Productivity indicators (e.g., 18.5% 27.7% 27.7% 20.8% 5.3%
time, output per employee)
Customer satisfaction 11.9% 19.1% 30.0% 27.1% 11.9%
Employee satisfaction 11.9% 17.8% 33.3% 28.7% 8.3%
Turnover/Promotion 17.2% 26.1% 32.0% 18.8% 5.9%
Proficiency/competency levels 7.9% 19.5% 39.6% 25.7% 7.3%
Business leader/supervisor 9.2% 26.1% 33.3% 23.8% 7.6%

perceptions of impact
Learner/employee perceptions 7.6% 20.1% 36.0% 27.7% 8.6%
of impact

Q19: To what extent are the following seen as barriers to the evaluation of learning in
your organization?
The most commonly cited barrier to the evaluation of learning was that it is too difficult to isolate training’s impact on
results versus the impact of other factors (51.7 percent). The barrier most strongly associated with lowered evaluation
success occurs when evaluation data is not standardized enough to compare well across functions (r=-.23). Similarly,
the more participants report that evaluations are not seen as credible, the lower the Market Performance Index for
that organization (r=-.13).
To what extent are the following seen as barriers to the evaluation of learning in your organization?
Percentage who Correlation Correlation
see these as bar- with Evaluation with Market
riers to a high or Success Index Performance
Responses very high extent (ESI) Index (MPI)
Too difficult to isolate training’s impact 51.7% -.18** -.09*
on results versus other factors’ influence
Our LMS does not have a useful 40.8% -.21**
evaluation function
Evaluation data is not standardized 38.0% -.23**
enough to compare well across functions
It costs too much to conduct higher- 32.2% -.10**
level evaluations
Leaders generally don’t care about 24.1% -.12** -.12**
evaluation data
Evaluation data is too difficult to 18.9%
interpret for most people
Evaluations are not seen as credible 14.5% -.20** -.13**


Q20: What percentage of the following learning programs are evaluated at each level of the
Kirkpatrick/Phillips five-level evaluation model?
The most commonly reported level at which organizations evaluate all learning programs is Level 1 (78.4 percent).
This result was consistent for leadership development (75.3 percent), sales (54.0 percent) and technical programs (69.7
percent). Similarly, as shown in Q20, Level 1 is the most popular method for evaluation regardless of whether learning
program delivery is classroom-based (80.9 percent) or electronic (52.0 percent).
What percentage of the following learning programs are evaluated at each level
of the Kirkpatrick/Phillips five-level evaluation model?
Responses Average of percentages for:

Leadership
development Technical
Percent evaluated at: All programs program Sales program
Level 1 (Reaction) 78.4% 75.3% 54.0% 69.7%
Level 2 (Learning) 48.9% 43.0% 36.7% 49.2%
Level 3 (Behavior) 25.2% 28.1% 23.8% 27.1%
Level 4 (Results) 15.4% 16.3% 17.7% 17.4%
Level 5 (ROI) 7.1% 7.1% 9.9% 7.0%
Q21: What percentage of all learning programs that are delivered using the following two learning
delivery methods are evaluated at each level of the K/P evaluation model?
What percentage of all learning programs that are delivered using the following two learning
Kirkpatrick/Phillips five-level evaluation model?
delivery methods are evaluated at each level of the
Responses Average of percentages for:

Live classroom-based Electronically based
Percent evaluated at: delivery method delivery method
Level 1 (Reaction) 80.9% 52.0%
Level 2 (Learning) 49.5% 43.4%
Level 3 (Behavior) 24.7% 14.6%
Level 4 (Results) 13.6% 9.6%
Level 5 (ROI) 6.1% 5.8%

Q22: To what extent are supervisors held accountable for the following?
Overall, it is less likely for a supervisor to be held accountable for goal setting with employees prior to training (17.4
percent) or for tracking pre- and post-training performance (11.4 percent) than to be held responsible for giving employees
opportunities to use new knowledge after training (22.9 percent). Holding supervisors accountable for tracking training
performance and giving employees opportunities to use new training knowledge were the most highly correlated with the
ESI (r=.37). Only the latter of those two was significantly correlated with MPI (r=.14).
To what extent are supervisors held accountable for the following:

Correlation Correlation
with Evalua- with Market
Not Small Moderate High Very high tion Success Performance
Responses at all extent extent extent extent Index Index
For setting goals with 26.7% 33.7% 22.3% 13.3% 4.1% .32**
employees prior to training
For giving employees

opportunities to use new 15.1% 29.1% 33.0% 19.0% 3.9% .37** .14**
knowledge after training
For tracking pre- and 36.5% 32.4% 19.7% 8.8% 2.6% .37**
post-training performance

Q23: Please indicate your organization’s total expenditures for employee learning in 2008.
The average expenditure across all organizations for employee learning in 2008 was $5,570,000. However, the average
amount spent differs depending on the size of the organization. In general, smaller and midsize organizations spent
significantly less than large organizations in total for employee learning in 2008.
Please indicate your organization’s total expenditures for employee learning in 2008.
Organizational Size Average expenditure
Small (fewer than 100 employees) $68,000
Midsize (from 100 to 10,000 employees) $1,942,000
Large (10,000 or more employees) $19,195,000
All organizations $5,570,000

Q24: Please indicate what percentage of your organization’s total expenditure for employee learning
is allocated to evaluation.
The percentage of an organization’s total expenditure for employee learning evaluation is greatest for small organizations
(6.5 percent) and least for large organizations (4.4 percent). Economies of scale seem to play a role for larger organizations.
Please indicate what percentage of your organization’s total expenditure for employee learning is
allocated to evaluation.
Average expenditure as a percentage of

Organizational size total expenditure for employee learning
Small (fewer than 100 employees) 6.5%
Midsize (from 100 to 10,000 employees) 5.7%
Large (10,000 or more employees) 4.4%
Q25: Of the total expenditure for learning evaluation, what percentage is spent on:
The largest percentage of the total expenses for learning evaluation is spent on internal resources (68.2 percent)
as compared with outside products/services (31.8 percent).
Of the total expenditure for learning evaluation, what percentage is spent on:
Responses Average percentage
Outside products/services, consultants, workshops,
and external training programs
Small organizations 41.8%
Midsize organizations 30.9%
Large organizations 27.6%
Internal resources such as learning staff salaries, administrative
costs, development costs, and internal IT costs
Small organizations 58.2%
Midsize organizations 69.1%
Large organizations 72.4%

Q26: Of the total expenditure for learning evaluation, what percentage is spent on the following:
Assessing the reactions of participants (Level 1) assumes the greatest percentage of the total expenditure for learning
evaluation (49.8 percent). However, the more participants report that Level 1 measurements are a greater percentage of
their total expenditure for learning evaluation, the less likely it is that they report evaluation success (r=-.38). Similarly,
the more organizations spend on Level 1 measurement, the lower their MPI (r=-.12).
Of the total expenditure for learning evaluation, what percentage is spent on the following:
Correlation with Correlation with
Average Evaluation Success Market Performance
Responses percentage Index (ESI) Index (MPI)
Reactions of participants (Level 1) 49.8% -.38** -.12*
Evaluation of learning (Level 2) 22.8% .13**
Evaluation of behavior (Level 3) 14.3% .25**
Evaluation of results (Level 4) 7.6% .32** .11*
Return on investment (Level 5) 3.0% .26**
Other forms of individual learning 2.5%

program evaluation

| authors and contributors |
David Wentworth has been a senior research Mike CzarnowskY is former director of re-
analyst for the Institute for Corporate Productivity since search with the American Society for Training & Devel-
2005. David has previously worked with digital media opment (ASTD). In that capacity, Czarnowsky led the ASTD
development and delivery, and he currently researches a Research team in the strategy, planning, and execution of
variety of topics for i4cp, including performance manage- all research studies, the Workplace Learning and Perfor-
ment, workforce technology, compensation trends and mance Scorecard, and the ASTD Benchmarking Forum.
the outsourcing of human resources. Contact informa-
tion: 727.345.2226 or david.wentworth@i4cp.com.
OTHER CONTRIBUTORS
Various staff members of the Institute for Corporate
Holly B. Tompson, PhD, is a senior research Productivity provided background research, writing
analyst at i4cp. Holly has taught in the management depart-
and other support for this report. Special thanks to Greg
ments of several universities, including the University of
Pernula, who worked on the survey implementation, to
Waikato in Hamilton, New Zealand, and, most recently,
Joe Jamrog, who created the graphs, to Mindy Meisterlin,
the University of Tampa. Her research has focused on
who created tables and worked on data analysis, to Judy
work-life balance and leadership development, with an
Wall, who proofed the report. Also thanks to Mitch
emphasis on training high-potential employees to sustain
LaQuier and Ellen Serrano.
maximum success without burnout. Holly is also active
in the University of Tampa’s Executive Education program,
where she is currently a leadership and development coach.
ASTD Research jointly determined the initial
Contact information: 813.601.5638 or holly.tompson@
concept and design of the project, jointly developed the
i4cp.com.
survey instrumentation, conducted supplemental statistical
analyses of the survey data, and jointly interpreted the
Mark Vickers is the vice president of research at findings. Contact information: 703.683.8100 or
i4cp and served as editor of this report. He has authored ASTDResearch@astd.org
many reports and white papers for the institute and
worked as managing editor for the Human Resource
Institute. He is also the editor of i4cp’s TrendWatcher
and has authored and coauthored numerous periodical
articles. Contact information: 727.345.2226 or mark.
vickers@i4cp.com.
Andrew Paradise, PhD, is a research manager

with the American Society for Training & Development
(ASTD). He is responsible for a variety of projects such
as the annual ASTD State of the Industry Report and
ad-hoc surveys. He has authored many reports and
research articles for the association on topics ranging
from sales training to global leadership. Paradise joined
ASTD in September 2006. Prior to that, Paradise worked
in the Research Department at the Automotive Aftermarket
Industry Association in Bethesda, Maryland, for five
years. Contact information: 703.683.7267 or aparadise@
astd.org.
authors and contributors | 61

| about the contributing organizations |
The American Society for Training & Development The Institute for Corporate Productivity
The American Society for Training & Development The Institute for Corporate Productivity (i4cp) improves
(ASTD) is the world’s largest association dedicated to corporate productivity through a combination of
workplace learning and performance professionals. research, community, tools, and technology focused on
ASTD’s members come from more than 100 countries the management of human capital. With more than 100
and connect locally in 136 U.S. chapters and 25 global leading organizations as members, including many of the
networks. Members work in thousands of organizations best known companies in the world, i4cp draws upon one
of all sizes, in government, as independent consultants, of the industry’s largest and most experienced research
and suppliers. ASTD started in 1944 when the organization teams and executives-in-residence to produce more than
held its first annual conference. ASTD has widened the 10,000 pages of rapid, reliable, and respected research
profession’s focus to link learning and performance to annually, surrounding all facets of the management of
individual and organizational results and is a sought-after people in organizations. Additionally, i4cp identifies and
voice on critical public policy issues. For more information, analyzes the upcoming major issues and future trends
visit www.astd.org. that are expected to influences workforce productivity
and provides member clients with tools and technology
to execute leading-edge strategies and “next practices”
on these issues and trends. For more information, visit
www.i4cp.com.
about the contributing organizations | 63

100937.54710
ISBN 978-1-60728-365-2
69500
2 | TRANSFORMING LEARNING WITH WEB 2.0 TECHNOLOGIES

9 781607 283652
790907 $695.00

Value of Training Evaluation

Uploaded by

Copyright:

Available Formats

You might also like

Value of Training Evaluation

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Value of Training Evaluation

Uploaded by

Copyright:

Available Formats

The Value of Evaluation:

Making Training Evaluations

AN ASTD Research Study

AN ASTD Research Study

No part of this publication may be reproduced, distributed,

Research reports published by ASTD can be purchased

ASTD Product Code: 790907

ASTD Editorial Staff

Director of Content: Dean Smith

11 | Section I—What evaluation

19 | Section II—How and when Do

25 | Section III—What are the Barriers

29 | Section IV—What are Companies

33 | Section V—How Can Organizations

36 | Conclusion and Policy

41 | Appendix­—The Value of Evaluation

61 | About the Authors and Contributors

63 | About the Contributing Organizations

M easuring the impact of learning continues to

T he pursuit of excellent learning evaluations continues,

4 | Value of Evaluation: Making Training Evaluations more Effective

I n a difficult economy, business leaders begin to

Our learning evaluation techniques help us meet 36.5%

We get a solid “bang for our buck” when it 25.6%

8 | value of evaluation: making training evaluations more effective

T he majority of the respondents (91.6 percent) said

Figure 2 | Percentage who use these Kirkpatrick/Phillips levels in

Percentage who use the Percentage who say this level

section i: What Evaluation Techniques Are Companies Using? | 11

gets the least participation, A modified alternative is

with only 17.9 percent of ROI = ((Benefits - Costs)/Costs) x (100).

respondent companies There’s also a forecasted calculation:

using it to any extent. ROI = ((Benefits - Costs)/Costs) x (100) x (%A) x (%P).

12 | value of evaluation: making training evaluations more effective

Figure 3 | In instances where you use it, how much value

A little Very high

section i: What Evaluation Techniques Are Companies Using? | 13

Figure 4 | Percentage of organizations using

14 | value of evaluation: making training evaluations more effective

Figure 5 | What percentage* of your learning programs is evaluated at

All Learning Leadership Sales Programs Technical Programs

section i: What Evaluation Techniques Are Companies Using? | 15

Figure 6 | What percentage* of your delivery methods is evaluated at

Live Classroom Technology-based

16 | value of evaluation: making training evaluations more effective

Figure 7 | Has your organization ever conducted an evaluation

section i: What Evaluation Techniques Are Companies Using? | 17

Figure 8 | In regard to those evaluation studies of

We have developed more effective learning

We have taken stories of the positive business

We have tracked the factors that enhance or 36.5%

18 | value of evaluation: making training evaluations more effective

T here are several points at which a company can

Prior to learning intervention 39.3%

Immediately post-learning intervention (within 37.9%

Short-term post-learning intervention (two weeks 55.8%

section iI: how and when do organizations conduct evaluations? | 19

Figure 10 | To what degree does your organization use the following

11 | Section I—What evaluation

19 | Section II—How and when Do

25 | Section III—What are the Barriers

29 | Section IV—What are Companies

33 | Section V—How Can Organizations

36 | Conclusion and Policy

41 | Appendix—The Value of Evaluation