EM2 Oral Presentation

For the oral presentation (30 minutes) to the assessor, you will be required to focus
on the following areas during the presentation:

▪ How the evaluation plan was developed

Purposes of the Evaluation

Given that it is the first time that Pytheas is spending in training its staff members to
adopt a customer-centric mindset, the programme evaluation is important as it will
help to provide feedback to ensure that the learning outcomes are linked to the
organization’s needs and expectations of improving net promotor score (NPS),
meeting its desired Return on Expectations, and review if learning is indeed
transferred to the workplace. Pytheas intends to implement the training programme
at the Group level. As such, this evaluation is important to provide quality control to
ensure good implementation, and guide decision making in strategic training
investments.

Stakeholders and Audience of the Evaluation

The evaluation is develop in consultation with the following internal and external
stakeholders of Pytheas Infosys:

General • To assess attitudinal and behavioural change of the UX

Manager of designers to improve in customer service
Pytheas • To enhance customer satisfaction, reduce customer complaints
Infosys and promote a companywide culture of service excellence
• To determine whether the training budget has been well spent
and justify the investment in training and decide whether or not
a full roll-out of the programme should proceed
Head of • To assess if the training delivery and assessment provided by
Training and a mixture of internal and external trainers and assessors is an
Facilitators ideal training strategy
• To review the sustainability and impact of the learning process
through post-training follow-up
Staff • To obtain feedback on the scheduling, facilitation, assessment,
members and coaching sessions
undergoing • To get feedback on the impact of the training programme to the
training workplace and ascertain any transfer of learning from the
training to the workplace
Customers • To obtain feedback from customers to ascertain whether
customer service standards have improved that differentiates
Pytheas Infosys from its competitors.
Evaluation Approaches
The evaluation will be guided by the objectives-oriented and management-
oriented approach. The evaluation seeks to review the extent to which the training
programme has been able to achieve the intended learning and behavioural goals
and objectives desired of Pytheas Infosys service excellence and customer
satisfaction.

The evaluation seeks to determine whether the programme has achieved the aim of
developing the graduate profile of an Service Professional possessing and promoting
the spirit of service excellence for the organization. Management would also need to
make a decision on the continuation of the programme.

The boundaries of the evaluation will involve Team Leaders, The General Manager,
and customers. The evaluation will include the Training Programme and a Reflective
Journal. The company will provide the necessary staff to support the evaluation,
however, the availability of staff members in Myanmar for data collection might be a
possible limitation. Legal consent for collection and use of the data from the staff
members there will be sought. For internal company staff, consent has already been
sought and approved at the Management level.

Evaluation Model
Given the context for evaluation, it is decided that the Kirkpatrick and Guskey
Model (4 Levels of Evaluation) will be most suitable for the evaluation of Pytheas
Infosys’s Service Professional Programme.

The 4 Levels of Evaluation caters most suitably to the key stakeholder of the
management of the company, clearly addressing the low Net Promoter Score (NPS)
of 5.5 (ROE) and increase sales revenue (ROI) at different stages and offering the
management clarity to make informed decisions regarding the programme,
especially in funding future training runs for its staff and their partners in Myanmar
who have yet to experience the training.

The 4 Levels of Evaluation process is also a more systematic and meaningful model
that ponders on the overall strategic training strategy. In the end, this evaluation
model should offer useful insights in a concise manner particular to the
management, especially in relation to business needs versus the buy-in required for
further investment and rollout of the training programme at the Group level.
▪ How the evaluation was implemented according to plan

Steps Information Evaluation Evaluation Evaluation Data Collection Data Types Analysis
(Kirkpatrick) Needed Questions Criteria Standards Method and Source Required

Level 1 Were Participants Facilitator Statistical

Need to verify End-of-course Survey Quantitative:
Learner learners of the evaluation calculation
training rated at least
satisfied programme Training effectiveness rating,
effectiveness 4.5 out of 5
with the are satisfied Facilitators rating
and Learners’
training with the Course Qualitative: Graphical
satisfaction on
programme facilitator. evaluation representation
the training Learners’ comments on of data (charts
? Training rated at least Interview of Learners
programme 4.3 out of 5 Facilitator’s ability to engage etc)
content was
learners, mastery of the
useful Overall knowledge and skills, delivery
Training effectiveness
of the content
Programme rated at least
was effective 4.2 out of 5

Level 2 Need to What did Learners are 100% of Graphical

Assessment Records Quantitative:
Learning determine the able to learners are representation
whether learners demonstrate certified Summary of Assessment of data (charts
learn from competency competent for Records etc)
learners Reflective Journal of
the in the skills assessment
acquired the programme and and complete Learners Qualitative: Thematic
attitude, skills ? knowledge their Reflective analysis
Assessment comments and
and obtained Journal within
evidence gathered
knowledge during the 3 months from
needed during programme completing the
training programme
Steps Information Evaluation Evaluation Evaluation Data Collection Data Types Analysis
(Kirkpatrick) Needed Questions Criteria Standards Method and Source Required

Level 3 Have Participants Each UX Workplace Performance

Need to know Quantitative:
Performance learners are able to designer who observation rating of
whether apply what attended the learners at the
applied the Supervisor rating of staff
learners have they have programme workplace
attitude,
applied what learned showed
skills and during the improvement
they learnt
knowledge course to in their ability
during training
acquired their day to to answer Qualitative:
to the
during day work customers’ Supervisor’s Report
workplace Supervisor report on staff
training to queries.
performance
the They are able
workplace? Management appraisal of staff
to make
actual performance of the job
appropriate
recommendati
ons to
customers
when their
request is not
in the contract
Level 4 Need to know Has the Pytheas has There is a 5% Statistical
Net Promotor Score Quantitative
Business whether the revenue seen an increase of calculation
training spent increased? increment in revenue. Survey Tool
Net Promotor Score
increased the revenue. ROI calculation
revenue of the Sales figures
Extant data review of Net Promoter
company (ROI)
Score
and improved company financial
Calculation
Net Promoter Has statements
Score (ROE) Pytheas Pytheas has Graphical
which is Net seen an Net Promoter representation
currently at Promoter increase in Score (NPS) of data (charts
5.5. Score their Met increase to 9 etc)
improved? Promoter
Score
▪ The findings of the evaluation study

The evidence of the evaluation suggested high degrees of effectiveness experienced by the
UX designers from the programme as well as improved customer satisfaction through the
Net Promoter Score, loyalty, and sales results. On the other hand, there were significant
expressions of increased stress levels among the staff, to meet the higher sales targets. The
detailed findings are as follows:

Effectiveness of Training
The training delivery and assessment provided by a trainers and assessors has been every
effective. The learners found the concepts and skills presented useful and relevant to their
work (rating 4.21). It is encouraging to know that the rating for intending to apply the
knowledge and skills learned to the workplace is higher than the former (4.42). The rating for
the course achieving its stated objectives is also the same (4.42). The chart below shows the
average rating for the training programme.

Training Programme Rating

4.45
4.43 4.43

4.4

4.35

4.3

4.25

4.21
4.2

4.15

4.1
The course achieved its stated The concepts and skills presented will I intend to apply the knowledge and
objectives be useful and relevant to my work skills learnt to my workplace

Ratings for the trainer were higher. The rating for trainer having good knowledge and
practical understanding of the subject and effective in communicating ideas and concepts
were 4.64. However, the ability of the trainer to stimulate and maintain interest of the
learners at all times was lower at 4.57. Overall, the learner’s reaction to the programme was
positive as the rating ranged from 4.21 to 4.64. The chart below summarizes the average
rating for the trainers.
 Trainer Rating
4.66
4.64 4.64
4.64

4.62

4.6

4.58 4.57

4.56

4.54

4.52
The trainer had good The trainer was effective in The trainer was able to
knowledge and practical communicating ideas and stimulate and maintain interest
understanding of the subject concepts of the learners at all times
matter knowledge

As for the feedback from the learners on the training duration and time demand of the entire
programme, 100% of the learners were able to complete their Reflective journal comfortably
within 3 months from completing the modules. However, over 50% commented that the
training did demand a lot of their time and attention. Overall, most intend to apply what was
covered in the training programme.

Better Customer first Mindset and Service Culture

The training programme has changed the behaviour of UX designers to serve customers
better, to enhance customer satisfaction, reduce customer complaints and promote
company-wide culture of service excellence.

Through the interview with learners, one said that “The programme help me see customer
service in a new angle”. Another said, “I am now more confident to provide better service to
customers”.

Supervisors also commented positively of the UX designers on how they dealt with
customers, especially those that give rise to service challenges after undergoing the training
programme.

One supervisor said “he (the UX designer) was able to effectively defuse the angry customer
over the phone”. Another said “with the effective proposal of alternatives that meet the
customer’s needs, the customer readily accepted it without any questions. I was amazed
since this customer was usually very difficult.”

There more many other positive comments which showed the customer-centric mindset in
the UX designers.
Improved Net Promoter Score (NPS)
Better customer service standards creates a sustainable competitive advantage for Pytheas
Infosys in the travel industry. Findings from the NPS survey yielded very favourable
feedback that the programme has been highly effective in providing GEMS to customers,
responding positively to services challenges and working effectively in a diverse work
environment.

Net Promoter Score

10
9.1
9
8
7
6 5.3 5.5
5
4
3
2
1
0
H1 2020 H2 2020 H1 2021

From the bar chart above, the NPS increased from 5.5 to 9.1 after training which showed the
effectiveness of the training programme.
▪ Evaluate the cost-benefit or return of investment of the programme

Based on extant data provided through the finance team, there was an increase in revenue
since after the training programme. Overall, for the three months following the completion of
the programme, the company experienced a 6% growth compared to the same period last
year. This amounted to an ROI of 106% on the training investment. The ROI calculation is
shown below:
21200
𝑅𝑂𝐼 = × 100 = 106%
20000

The data gathered also show more favourable customer responses following better service
standards that the team has consciously made effort to extend to customers. Based on the
interviews with the UX designers who have undergone the training, they felt more confident
in handling customers and providing batter alternatives that customers will accept compared
to those who have not attended the training. The training has proven to be effective with the
first run. Hence, to ensure further growth of the business, more runs should be planned for
the other staff, particularly those in Myanmar.

The effectiveness of the training results may appear to positive at the first evaluation abut it
needs to be maintained on an ongoing basis. All supervisors will continue to observe their
staff members on an ongoing basis and commend them for their efforts where they are due.
Based on the above findings, it is concluded that the training budget has been well spent to
justify the investment in training and subsequent roll-out of the full programme.

▪ Effectiveness of the Evaluation Process

In summary, the evidence of the evaluation suggested high degrees of effectiveness

experienced from programme. The first run of the programme have clearly demonstrated
results in pushing Pytheas Infosys to its next frontier, bring service excellence to new
heights. It is therefore meaningful to continue to roll-out the programme to all in the
organisation to foster a culture of service excellence.

The training and assessment delivery can continue to be performed by the same team of
facilitators, since the learners have rated their effectiveness as high. The effectiveness and
sustainability of the training will be monitored on an ongoing basis, to provide follow-up
interventions, if needed.

Limitations of the Evaluation

One major limitation of the evaluation is the small size of the number of respondents, even
though the sample is representative of the company. The programme has only run once, as
such there has not been an opportunity to introduce a full pre- and post-training evaluation to
accurately reflect the actual results of the programme. Furthermore, there is no control group
to provide a basis for comparison at this stage.

▪ Any other relevant issues