ROI On a Shoestring

Jack J. Phillips, Ph.D.

Biography: Dr. Jack J. Phillips is Chairman of the ROI Institute, the leading provider of services for
measurement, evaluation, metrics, and analytics. Former Learning and Development Director, HR
Executive, and Bank President, Phillips provides consulting services for over half of the Fortune 100
companies and workshops for major conference providers worldwide. Author of the first book on
training evaluation in the USA, Phillips has authored or edited more than 75 books. He is the
developer of the ROI Methodology, the most used evaluation system in the world. His work has been
featured in the Wall Street Journal, Bloomberg BusinessWeek, Fortune, and on CNN.

Objectives
After completing this session, participants should be able to:

1. Select a learning or development program to evaluate at the ROI level.

2. Plan the evaluation study using job aids and simple tools.
3. Execute the study with shortcuts and cost savings tips.

Copyright © 2015 ROI Institute, Inc.

No part of this may be reproduced, stored in a retrieval system, or transmitted in any form or by a means without written permission

Phone: 205-678-8101 • Fax: 205-678-8102

Email: info@roiinstitute.net
Web site: www.roiinstitute.net

Follow, Like or Visit us:

 The Executive View*
We Currently We Should My Ranking of the
Measure Measure This Measure This Importance of This
in the Future Measure

1. Inputs: “Last year, 78,000 employees 94% 85% 6

received formal learning.
2. Efficiency: “Formal learning costs $2.15 78% 82% 7
per hour of learning consumed.”

3. Reaction: “Employees rated our training 53% 22% 8

very high, averaging 4.2 out of 5.”

4. Learning: “92% of participants increased 32% 28% 5

knowledge and skills”

5. Application: “At least 78% of employees 11% 61% 4

are using the skills on the job”

6. Impact: “Our programs are driving our 8% 96% 1

top 5 business measures in the
organization.”

7. ROI: “Five ROI studies were conducted 4% 74% 2

on major programs yielding an average
of 68% ROI.”

8. Awards: “Our learning and development 40% 44% 3

program won an award from the
American Society of Training and
Development”

Fortune 500 and Top Private Organizations

 Surveys administered by top executives
 Completed directly by CEO
 10 actions taken to increase response rate
 96 returned
 21% Response rates

Exercise:
1. What conclusions do you make from this data?

2. What are the barriers to doing more “business” evaluation?

3. Would these results be different for other functions? Other organizations? Other countries?

3
 Status of Measurement

Level Measurement Category Current Goal in Comments About

Status* 5 Years* Status
0 Inputs/Indicators 100% 100% This is being
accomplished now
Measures the number of programs, participants,
audience, costs, and efficiencies
100% †
1 Reaction and Planned Action 100% 100% Need more focus on
content and perceived
Measures reaction to, and satisfaction with, the value
experience, contents, and value of program
79% †
2 Learning 30 – 40% 80 – 90% Must use simple
learning measures
Measures what participants learned in the
program – information, knowledge, skills, and
contacts (take-aways from the program)
54% †
3 Application 10% 30% Need more follow-up
Measures progress after the program – the use 31% †
of information, knowledge, skills, and contacts
4 Business Impact 5% 10% This is the connection to
business impact
Measures changes in business impact variables
such as output, quality, time, and cost-linked to 14.4% †
the program
5 ROI 1% 5% The ultimate level of
evaluation
Compares the monetary benefits of the business
impact measures to the costs of the program. 4.3% †

* Percent of Programs Evaluated At This Level

† Best Practice Benchmarking (user for 5 plus years)
» Add your numbers in each box

Exercise:
What concerns do you have about these percentages?

4
 The ROI Methodology
Calculating the Return on Investment
of Performance Solutions
 HR Solutions
 Training Solutions/E-Learning
 Major Projects 8
 Technology Solutions
 Organization Development Solutions CAPTURE
 Change Initiatives COSTS OF
SOLUTION

EVALUATION PLANNING DATA COLLLECTION DATA ANALYSIS REPORTING

1 2 3 4 5 6 9 10
DEVELOP COLLECT DATA CONVERT
DEVELOP/REVIEW COLLECT DATA ISOLATE CALCULATE GENERATE
EVALUATION DURING DATA TO
OBJECTIVES AFTER SOLUTION EFFECTS THE RETURN ON IMPACT STUDY
PLANS AND SOLUTION MONETARY
OF SOLUTION IMPLEMENTATION OF SOLUTION INVESTMENT REPORT
BASELINE DATA IMPLEMENTATION VALUE

1. REACTION and 3. APPLICATION and 5. ROI

PLANNED ACTION IMPLEMENTATION
2. LEARNING and 4. BUSINESS IMPACT 7
CONFIDENCE
IDENTIFY
Guiding Principles INTANGIBLE
ROI Calculation - Example EVALUATION FRAMEWORK MEASURES
1. When conducting a higher-level evaluation, collect data at lower levels.
2. When planning Net a evaluation, theLev
higher level Benefits
Program/Project
el
previous Measurement Focus
level of evaluation is not required to be
INTANGIBLE
comprehensive.
ROI = 1. Reaction Measures participant satisfaction with the
BENEFITS
& Planned Action program/project and captures planned action.
3. When collectingProgram/Project
and analyzing Costs
data, use only the most credible sources.
4. When analyzing data, select the most conservative 2. Learningalternatives
& Confidence for
Measures changes in knowledge, skills,
calculations.
and attitudes.
5. Use at least one method to isolate the effects of the program or project.
6. If no improvement
Costs data
of program/project are available for a population or from a specific source, assume that no
$230,000
improvement has occurred.
Benefits of program/project (1st year) $430,000 3. Application & Measures implementation and changes in
7. Adjust $430,000
estimates of improvements for the potential Implementation behavior in the performance setting.
error of the estimates.
- $230,000
ROI =use of extreme data
8. Avoid = 0.87items
x 100 =and
87%unsupported claims
4. Business Impact when calculating ROI calculations.
Measures changes in business impact
$230,000 variables. solutions.
9. Use only the first year of annual benefits in the ROI analysis of short-term
10. Fully load all costs of the solution, project, or program
5. Return when analyzing
on Investment Compares ROI.
benefits to the costs.
11. Intangible measures are defined as measures that are purposely not converted to monetary values.
12. Communicate the results of the ROI Methodology to all key stakeholders.
5
 Job Aid
Exercise: Matching Evaluation Levels with Objectives
Instructions: For each objective listed below, indicate the level of evaluation at which the
objective is aimed.

1. Reaction and Planned Action 4. Business Impact

2. Learning 5. Return on Investment
3. Application
Evaluation
Objective Level
After completing this program or project, participants should: _________
1. Improve work group productivity by 20% in six months. _________
2. Initiate at least three cost reduction projects in 15 days. _________
3. Achieve an average cost reduction of $20,000 per project. _________
4. Use counseling discussion skills in 90% of situations where
work habits are unacceptable. _________
5. Achieve a 2:1 benefit to cost ratio one year after the new
performance- management system is implemented. _________
6. Be able to describe the four elements of the employee assistance program. _________
7. Increase the annual employee satisfaction index by 25% in 3 months. _________
8. Integrate employee records into new database following the 5-step process. _________
9. Increase research grant funds by 20%. _________
10. Achieve a leadership simulation score average of 75 out of a possible 100. _________
11. Conduct a performance review meeting with direct reports in
three weeks to establish performance improvement goals. _________
12. Provide a 4 out of 5 rating on appropriateness of new ethics policy. _________
13. Decrease the time to recruit new professional staff from 35 days to 20 days. _________
14. Complete all action items on the action plan in 60 days. _________
15. Perceive the flextime work schedule system as important to their success. _________
16. Enroll in the career enhancement program with a goal of a participation rate
of 15% _________
17. Decrease the number of security breaches of supplier records. _________
18. Achieve a post-test score increase of 30% over pre-test. _________
19. Use the new software as reflected by an 80% score on an
unscheduled audit in 3 weeks. _________
20. Submit suggestions for improvement in the first year (with a10% target). _________

6
 Job Aid
Program Alignment
V Model

Start Here End Here

Payoff Needs 5 ROI Objectives 5 ROI

Business Needs 4 Impact Objectives 4 Impact

Performance Needs 3 Application Objectives 3 Application

Initial
Analysis Measurement and
Evaluation
Learning Needs 2 Learning Objectives 2 Learning

Preference Needs 1 Reaction Objectives 1 Reaction

Business Alignment and Forecasting Project The ROI Process Model

7
 Job Aid
Program Alignment
The Alignment Process
Start Here V Model Example End Here

Payoff Needs 5 ROI Objectives 5 ROI

Absenteeism is ROI of 25% Calculate ROI;
costing $10,000
monthly

Business Needs 4 Impact Objectives 4 Impact

Monitor absenteeism
Unexpected absenteeism is Reduce absenteeism to records for six months
9 % and growing; 5% six months after
benchmark data is at 5% program

Performance Needs 3 Application Objectives 3 Application

Discussions between team Follow-up questionnaire
member and supervisor are Counseling discussions to participants to check
not occurring when there is conducted in 95% of frequency of discussions
an unplanned absence situations when an – three months
unexpected absence occurs

Initial Learning Needs 2 Learning Objectives 2 Learning Measurement

Analysis Deficiency in counseling/ Be able to demonstrate Skills practice session and Evaluation
discussion skills the use of counseling during program
skills

Preference Needs 1 Reaction Objectives 1 Reaction

One-day counseling skills
workshop must provide usable,
necessary and relevant skills;
facilitator-led; participants are
supervisors
Program receives favorable Reaction questionnaire at the end of
rating of 4 out of 5 on need for program
program, relevance of the
program and the practicality of
program

Business Alignment and Forecasting Project The ROI Process Model

8
 Step 1: Develop Project Objectives

1. Reaction objectives describe desired 4. Impact objectives describe

immediate reaction to the program, consequence of applying skills or
highlighting issues that are important to implementing the project and are expressed
the success of the program and as specific measures of output, quality,
emphasizing planned action, if feasible. cost, time, and satisfaction.

2. Learning objectives communicate 5. ROI objectives set the acceptable level

expectations for obtaining new of monetary benefits versus costs of the
information, skills, and knowledge. program and may be expressed as an
ROI percentage, a benefit-to-cost ratio, or
3. Application objectives describe a time for payback.
intermediate outcomes, including use of
skills and knowledge, on-the-job
performance changes, and program
implementation.

TIP: 1) Teach Others to Write Objectives

Step 2: Plan for Evaluation

Data Collection Plan

Data collection planning answers fundamental questions about data collection: What, How, Who,
When, Where, and How Much?

ROI Analysis Plan

The ROI Analysis Plan details how improvement in business measures will be isolated to the
project and converted to monetary value. Cost categories, intangible benefits, and target
audiences for communication are also identified.

Evaluation Project Plan

The project plan details each step of the evaluation.

TIPS: 1) Use Templates 2) Share the Joy

9
 Step 3: Collect Data During Project Implementation
Reaction Data
The typical methods for capturing reaction data are:
• Feedback questionnaires—simple, • Interviews—sometimes used when the project is
straight-forward and easy to tabulate in its first offering, to help probe for and capture
• Action plans—measures intent to use detail
with planned actions • Focus groups—used when it is important for
participants to hear the input of others
Learning Data
Learning data are captured through a variety of measurement processes, ranging from formal testing to
informal self-assessments. Several methods are used, including:
• Surveys and questionnaires—determine the • Performance demonstrations—provide direct
extent to which participants have acquired evaluation of the ability to apply knowledge and
skills, knowledge, and information skills
• Facilitation assessments—ratings from • Simulations—enable assessment of skills and
facilitators or project leaders based on knowledge acquisition
observations during the project • Team assessments—assess the extent of skills
• Written tests and exercises—measure and knowledge acquisition
changes in knowledge and skills • Skill/confidence building exercises—an
• Skill practices—help assess the degree of interactive approach to capturing skill and
applied learning and acquisition of problem- knowledge levels
solving skills
TIPS: 1) Use Self Assessment 2) Automate

Step 4: Collect Data After Project Implementation

Level 3 Level 4 • Interviews—conducted to determine how
Survey  extensively the project is used
Questionnaires  
Observation  • Focus groups—conducted to determine the
Interviews  extent to which the project is used
Focus Groups  • Action plans—developed by participants
Action Planning   during the project and are implemented after
Performance Contracting  

the project is completed. Follow-up on action
Performance Monitoring
plans provides evidence of application and
• Surveys—taken to determine how business impact success
extensively the participants have used
various aspects of the project • Performance contracts—developed by the
participant, the participant's supervisor, and
• Questionnaires—usually more detailed than
the facilitator who all agree on performance
surveys and can be used to uncover a wide
outcomes
variety of data in open-ended and forced-
• Performance monitoring—useful where
response options
various performance records and
• Observation—captures actual skill
operational data are monitored for changes
application and use. Observations are
particularly useful in customer service
projects and are effective when the observer
is either invisible or transparent.

TIPS: 1) Use action plans and questionnaires 2) Use technology to automate process
3) Push responsibility to participants 4) Build in evaluation tools 5) Use sampling

10
 Step 5: Isolate the Effects of the Project
While isolating the effects of the project with other influences is sometimes difficult, it is necessary for
credibility of the study. Without this step, there is no proof that the project is connected to a business
measure.
Benchmarking Data

Frequency
Method
60%
1. Control Groups
2. Trend Line Analysis 50%
3. Forecasting Methods
4. Participant Estimates 40%
5. Manager Estimates
6. Sr. Management Estimates 30% 55%

7. Expert Input
20% 37%
8. Customer Input 32%
29%

Survey of Users, N = 235 10% 13%

11% 11%
5%
0%
1 2 3 4 5 6 7 8

TIPS: 1) Use estimates with an error adjustment 2) Build capability for analysis

Step 6: Convert Data to Monetary Values

To calculate the ROI, improvement in business measures must be converted to money.

Standard values are available for most Internal or external experts are used to
output and quality measures. Output data are estimate a value for a unit of measure,
converted to profit contribution or cost based on their
savings, based on their unit contribution to
profit or the unit contribution to cost savings. • Position • Knowledge
Quality improvements are directly converted • Experience • Credentials
to cost savings. • Neutrality • Publications

Participants’ wages plus employee External databases provide the value or

benefits are used to develop the monetary
value for time where employee time is saved.
Historical costs, developed from cost
statements and reports, are used to calculate
the value for a specific measure.
Soft measures are sometimes linked
mathematically to other measures that are
easier to convert to money.
cost of data items. Research, government,
and industry databases– usually available
through the Internet–can provide important
information for these values.
Participants estimate the value of the unit
of data.
Supervisors and managers provide
estimates when they are capable of
assigning values to the data item.

TIPS: 1) Find the value internally, quickly 2) If it matters, the money is there

11
 Step 7: Identify Intangible Benefits

Intangible benefits are project benefits that we choose not to convert to money. They are measures
that cannot be converted to money credibly with minimal resources. Intangible data should be
collected in some way, even if participants must indicate the degree to which the project has
influenced intangible measures. Intangible benefits are usually reported in a table in the complete
report.

Typical intangibles

• Image • Customer Service • Networking

• Job Satisfaction • Complaints • Partnering
• Organizational Commitment • Conflicts • Brand Awareness
• Employee Engagement • Stress • Creativity
• Teamwork • Communications • Social Responsibility

To decide whether or not to convert a measure to monetary value, use this four-part test.
1. Does an acceptable, standard monetary 3. Can the conversion be accomplished with
value exist for the measure? If yes, use it minimum resources? If not, list it as an
in the ROI calculation; if not, go to the intangible; if yes, go to the next step.
next step.
4. Can the conversion process be described to
2. Can a method be used to convert the an executive audience and secure buy-in in
measure to money? If not, list it as an two minutes? If yes, use it in the ROI
intangible; if yes, go to the next step. calculation; if not, list it as an intangible.

TIP: 1) Connect intangibles to a program with a single question

Step 8: Tabulate the Costs of the Project

Cost Item Prorated Expensed

Needs Assessment 
Design and Development 
Acquisition/Purchase (if applicable) 
Delivery/Implementation 
• Salaries/Benefits-Facilitator Time 
• Salaries/Benefits-Coordination Time 
• Program Materials and Fees 
• Travel/Lodging/Meals 
• Facilities 
• Salaries/Benefits-Participant Time 
• Operating Expenses 
Ongoing Operations Cost (if applicable) 
Evaluation 
Overhead 

TIPS: 1) Track cost during program 2) Estimate indirect costs

12
 Step 9: Calculating the ROI
Return on Investment (ROI) is a financial metric, representing the ultimate measure of project success.
ROI is calculated using the project benefits and costs.
The benefit-cost ratio is the project Example Calculation
benefits divided by cost. In formula form, The BCR is calculated as:
it is:
$436,300
Project Benefits BCR = = 1.96:1
BCR = $222,550
Project Costs
Interpretation: For every dollar invested, there is
The return on investment calculation
$1.96 in benefits
considers the net benefits divided by
project costs. The net benefits are the The ROI is calculated as:
project benefits minus the costs. In formula
form, the ROI becomes: $436,300 - $222,550
ROI (%) = X 100 = 96%
$222,550
Net Project Benefits
ROI (%) = X 100
Project Costs Interpretation: For every dollar invested, 96¢ is
returned after the investment is
This is the same basic formula used in
recovered
evaluating capital investments where the
ROI is traditionally reported as earnings
divided by investment.

TIP: 1) Calculate BCR and ROI using shortcut

Step 10: Report Data to Key Stakeholders

Reporting the results of the study is an important final step in the ROI Methodology. Properly
identifying the audience and providing appropriate information is essential.
A variety of media can be used to communicate the project success:
Impact Study (50-100 pages) One Page Summary
• Detailed analysis • Charts, tables
• Historical document • Follows ROI data categories
• Learning tool
Newsletter Article
Executive Summary (4-8 pages) • General interest
• Summarizes impact • Brief story
• Follows the ROI Methodology
Web Site (brief summary of impact study)
Face-to-Face Meeting (1/2-2 hours) • Charts, tables
• Usually the first opportunity to see ROI data • Link to methodology
• Builds support for the ROI Methodology • Downloadable
• Presents results and improvements
Case Study (15-25 pages)
Brochure • Published internally or externally
• Project description • Tells a story
• Emphasis on results

TIPS: 1) Plan early 2) Use templates 3) Streamline to one page

13
 PWhen Properly Implemented, High ROI Values Can Be
Achieved With Programs On:
rograms On:
• Leadership Development
• Performance Management
• Management Development
• Supervisor Training 100% to 700% ROI is not
• Talent Retention Uncommon.
• Flexible Work Systems
• Reward Systems
• Skill Based Pay
• Business Coaching

ROI Methodology: The Payoff

• Align projects to business needs

• Show contributions of selected projects
• Earn respect of senior management/administrators
• Build staff morale
• Justify/defend budgets
• Improve support for projects
• Enhance design and implementation processes
• Identify inefficient projects that need to be redesigned or eliminated
• Identify successful projects that can be implemented in other areas
• Earn a “seat at the table”
Exercise:
What are your payoff opportunities?
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________

ROI is Feasible

• Not very expensive

• Many shortcut methods
• Time requirement can be managed
• Fits all types of programs
• Technology helps with costs/time
• Implementation is planned/systematic

14
 Simple ROI Studies

• In Store Selling Skills

o Retail store
• Leadership Development for First Level Managers
o Car rental company
• Absenteeism Reduction
o Large city
• Management Development
o Restaurant chain
• Safety Training
o Construction company
• Mobile Learning for Selling Upgrade
o Software company

These are available for download at http://www.roiinstitute.net/atd-handouts/.

15