Professional Documents
Culture Documents
Chapter 006
Chapter 006
Training Evaluation
Objectives
O Explain why evaluation is important
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Objectives
O Choose the appropriate evaluation design
based on the characteristics of the company
and the importance and purpose of the
training
O Conduct a cost-benefit analysis for a
training program
O Explain the role of workforce analytics and
dashboards in determining the value of
training practices
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Key Terms
O Training effectiveness: benefits derived
training
O Training evaluation: process of determining
training effectiveness
O Training outcomes: measures to evaluate
training effectiveness
O Evaluation design: data will be collected
and which data will be collected for
training evaluation
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Formative Evaluation
O Refers to evaluation of training that takes
place during program design
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Summative Evaluation
O Refers to evaluation conducted to
determine if training has lead to desirable
outcomes
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Importance of Evaluation
O There are multiple reasons to evaluate:
o identify a program’s strengths
and weaknesses
o assess what features of training
content
and context matter
o identify which trainees benefited
o gather information for
marketing training
o determine financial benefits and
costs
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
The Evaluation Process
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Training Outcomes
Affective Return on
Results
Outcomes Investment
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Reactions
O Trainees’ perceptions of the training
experience relating to the content,
facilities, trainer, and methods
O Key questions to consider include:
o Did the trainees like the program?
o Did the environment help
learning?
o Was the material meaningful?
O Typically measured at the end of
training
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Learning & Cognitive
Outcomes
O Relate to familiarity with information,
including principles, facts, techniques,
procedures, and processes
O Typically measured via paper-and-pencil
tests and self-assessments
O Tests often preferred over self-
assessments
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Behavior & Skill-Based
Outcomes
O Relate to proficiency with technical skills,
motor skills, and behavior
O Include learning and transfer
O Learning often assessed via work
samples
O Transfer often assessed via observation
or managerial/peer ratings
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Affective Outcomes
O Include attitudes and motivation
o e.g., self-efficacy, employee
engagement, motivation to learn,
tolerance for diversity, attitudes toward
safety
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Return on Investment
O Involves comparing the training program's
benefits to its costs in monetary terms
o Benefits: value the company gains
Relevance Reliability
Discrimination Practicality
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Relevanc
O The extent toe
which training outcomes are
related to the learned capabilities
emphasized in the program
O Contamination refers to the inclusion of
inappropriate or irrelevant outcomes
O Deficiency refers to the omission of
important information
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Deficiency, Relevance, &
Contamination
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Reliability
O The extent to which outcomes can be
measured consistently over time
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Discrimination
O The extent to which measured
performance reflects a true difference
O We want tests that discriminate between
high and low performance
o A test that is too easy may not
discriminate
o In this instance, both high and low
performers would do well and appear
“good” even though they are not
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Practicality
O The extent to which outcomes can be
easily measured and collected
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Training Evaluation Practices
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Which Outcomes?
O While collecting data on all outcomes is
ideal, it may not always be necessary
O Consider the scope of the training and
practical considerations
O Do not assume that positive reactions lead
to transfer
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Threats to Validity
O Threats to validity refer to factors that will
lead an evaluator to question the results
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Controlling for Threats
O There are three ways to control for threats:
o Use pre-tests and post-tests
o Use a control group
o Random assignment of employees
to control and training groups
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Post-test Only
O Involves collecting only post-training
outcomes
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Pre-test / Post-test
O Involves collecting both pre-training and
post-training outcomes to determine if
a change has occurred
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Pre-test / Post-test with
Comparison Group
O Includes pre-training and post-training
outcomes and use of a control group
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Time Series
O Involves collecting measures at periodic
intervals pre- and post-training
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Solomon Four-Group
O Combines the pre-test/post-test
comparison group design and the
posttest-only control group design
o Pre-test, treatment, post-test
o Pre-test, no treatment, post-test
o No pre-test, treatment, post-test
o No pre-test, no treatment, post-
test
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
When Evaluation May Not
Be Necessary
O Time constraints
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
When Evaluation is a Must
O The training is ongoing and has the
potential to affect many
employees
O The training program involves multiple
classes and a large number of
trainees
O The expertise exists to evaluate
O The cost of training is significant
O There is sufficient time and interest
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Cost-Benefit Analysis
O Companies may desire to quantify whether
the benefits of training outweigh the
costs
O Cost-benefit analysis
o Process of determining the economic
impact of training using accounting
methods that look at training costs and
benefits
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Cost Categories
1. program development or purchase
2. instructional materials
3. equipment and hardware
4. facilities
5. travel and lodging
6. salary of the trainer and support staff
7. cost of lost productivity or replacement
workers while trainees are away
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Determining Benefits
O A number of methods can be used to
identify benefits
o literature that summarizes benefits
o pilot training programs
o observing successful job
performers asking trainees and
managers for estimates
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Calculating Return on
Investment (ROI)
O Identify annual change in outcomes
O Place a monetary value on the outcomes
O Determine the annual change in value
O Determine training costs
O Calculate net benefit
O Divide net benefits by costs
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Utility Analysis
O Assessing the dollar value of training
based on:
o estimates of the difference in
performance between trained and
untrained employees
o number of individuals trained
o length of time training is expected
to influence performance
o variability in performance in
the untrained group of
employees
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Success Cases & Return on
Expectations
O Success cases refer to concrete examples
showing how learning has led to results
the company finds worthwhile and
credible
O Return on expectations (ROE)
demonstrates to key stakeholders that
their expectations about training
have been satisfied
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Big Data
O Big data refer to complex datasets
compiled across different systems,
including marketing, sales, HR, finance,
accounting, customer service, and
operations
O Three dimensions characterize big data
o volume
o variety
o velocity
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Big Data & Training
O Big data help make decisions about human
capital based on data, rather than intuition
and conventional wisdom
O Big data can be used to:
o evaluate the effectiveness of programs
o determine their impact on
business results
o develop predictive models for
forecasting training needs, course
enrollments, and outcomes
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.