Surg Endosc (2018) 32:62–72
https://doi.org/10.1007/s00464-017-5634-6
Can a virtual reality surgical simulation training provide a
self-driven and mentor-free skills learning? Investigation
of the practical influence of the performance metrics
from the virtual reality robotic surgery simulator on the skill
learning and associated cognitive workloads
Gyusung I. Lee1 • Mija R. Lee1
Received: 6 January 2017 / Accepted: 1 June 2017 / Published online: 20 June 2017
Ó Springer Science+Business Media, LLC 2017
Abstract
Background While it is often claimed that virtual reality
(VR) training system can offer self-directed and mentor-
free skill learning using the system’s performance metrics
(PM), no studies have yet provided evidence-based con-
firmation. This experimental study investigated what extent
to which trainees achieved their self-learning with a current
VR simulator and whether additional mentoring improved
skill learning, skill transfer and cognitive workloads in
robotic surgery simulation training.
Methods Thirty-two surgical trainees were randomly
assigned to either the Control-Group (CG) or Experiment-
Group (EG). While the CG participants reviewed the PM at
their discretion, the EG participants had explanations about
PM and instructions on how to improve scores. Each
subject completed a 5-week training using four simulation
tasks. Pre- and post-training data were collected using both
a simulator and robot. Peri-training data were collected
after each session. Skill learning, time spent on PM (TPM),
and cognitive workloads were compared between groups.
Results After the simulation training, CG showed sub-
stantially lower simulation task scores (82.9 ± 6.0) com-
pared with EG (93.2 ± 4.8). Both groups demonstrated
improved physical model tasks performance with the actual
robot, but the EG had a greater improvement in two tasks.
The EG exhibited lower global mental workload/distress,
Presented at the 2015 Clinical Congress, Chicago, Illinois, October
7th, 2015.
& Gyusung I. Lee
gyusunglee@gmail.com
1
Department of Surgery, Johns Hopkins University School of
Medicine, 600 North Wolfe Street Blalock 1204, Baltimore,
MD 21287, USA
123
and Other Interventional Techniques
higher engagement, and a better understanding regarding
using PM to improve performance. The EG’s TPM was
initially long but substantially shortened as the group
became familiar with PM.
Conclusion Our study demonstrated that the current VR
simulator offered limited self-skill learning and additional
mentoring still played an important role in improving the
robotic surgery simulation training.
Keywords Virtual reality
Simulation
Robotic surgery

Training
Performance metrics
Mentoring
The use of robotic surgery systems is rapidly and contin-
uously increasing as more surgical specialties have started
using this new technology in their patient care [1–7].
Approximately, 570,000 robotic procedures were per-
formed in 2014, representing an increase of about 9% since
2013. This substantial rise is often attributed to several
unique features possessed by the robotic surgical systems
but not present in traditional laparoscopy—notably, 3D
stereo visualization, increased degrees of freedom, motion
scaling, and tremor reduction [8, 9]. While robotic surgery
provides several benefits in patient care, the performance of
robotic surgery, especially by novice surgeons, may be
challenging. For instance, surgeons remotely manipulate
very powerful robotic arms with the lack of tactile feed-
back due to the nature of robotic surgery as a remote sur-
gery. This challenge may cause a higher risk for the
patient; therefore, robotic surgeons are required to learn
how to operate the robotic surgery system accurately and
carefully to maximize the benefits of robotic surgery while
maintaining patient safety [10, 11].
Surgical education plays a very important role in safe
patient care. There has been a growing need for more
Surg Endosc (2018) 32:62–72
effective and efficient surgical training due to increasingly
restrictive limits on resident duty hours (i.e., 80-h maxi-
mum, no more than 24 h’ continuous duty) resulting in
increased workload and fewer hours for education [12–14].
As a potential solution, Virtual Reality (VR) technology
has become a popular tool in minimally invasive surgery
(MIS) education during the past decade, largely due to the
increase in commercially available surgical trainers/simu-
lators [15–17]. As a result, VR surgical education has been
integrated into various MIS training at numerous residency
and fellowship programs [18–20]. These VR simulators
provide trainees not only with easy access to various
training modules that often provide very realistic recre-
ations of both the human anatomy and relevant surgical
tasks but also with performance evaluation metrics
enabling immediate feedback to trainees following each
task performance, something not always readily available
or provided in a lab training setup [21–23].
Several research studies have demonstrated that the VR
training system is a useful training tool in robotic surgery
(content validity) regarding providing sufficient surgical
skill learning [24, 25]. It has also been shown that the VR
training environment and the console experience was very
realistic (face validity), the training models were effective
in distinguishing between novice and expert surgeons
(construct validity), and the performance scores from the
VR system correlated well with the experts’ assessments
(concurrent validity) [26–29].
The da Vinci robotic surgical system from Intuitive
Surgical, Inc., offers a VR training option which provides a
comprehensive set of performance metric (PM) for trainee
practice evaluation. These PMs are designed to assess
trainees’ task performance levels quantitatively and to
show the performance components requiring skill
improvement. Several research studies revealed that the use
of these metrics was very beneficial especially for crite-
rion-based robotic basic skill training [24, 30]. Considering
trainees’ potential benefit from the performance metrics
(PM) provided by VR surgical simulators, it is often
claimed that VR simulators could provide trainees with a
self-driven and mentor-free surgical skill learning envi-
ronment; however, no evidence-based data yet support
such a claim. Additionally, the system’s provision of
feedback on how specifically trainees can improve the
scores of the not-well-performed performance metrics may
still be limited.
As demand exists for well-structured robotic training
curricula providing more effective and efficient robotic
surgery training and VR training can be a good option
[31–36], the true value of VR simulator in robotic surgery
education should be carefully examined. No research
studies have investigated how carefully surgical trainees
interact with the PM available from the VR surgical
63
simulator and how actively the trainees use the information
available from the PM for their skill improvement. It has
not been thoroughly examined whether surgical trainees
understand the meaning of each PM provided by a VR
robotic surgical skill trainer after each task performance
and/or more importantly, whether surgical trainees would
know how they can improve particular PM scores to
improve their overall skill learning.
While mentoring or coaching of providing feedback to
surgical trainees is considered very crucial component in
surgical education, a recent publication by the Association
for Surgical Education addressed that the best methods of
feedback or debriefing had not been established yet and
continuous research investigations on different feedback
deliveries should be conducted in the future [37]. While
several research studies demonstrated that mentoring or
coaching could be valuable in surgical training including
VR simulation [38–41], no studies have yet investigated
the effectiveness of mentoring on the skill learning during
VR robotic surgery training.
The goal of this study was to examine whether the current
VR simulation training with the PMs from the simulator
could provide sufficient self-learning environment for sur-
gical trainees and investigate whether additional mentoring
are still required for improving the trainees’ skill learning
and transfer and their cognitive workloads in robotic surgery
education. We hypothesized that the surgical trainees who
used the PMs at their discretion during their robotic VR
simulation training could not achieve high skill levels and
the surgeons who were trained using the same system but
received additional mentoring explanations and instructions
would demonstrate faster skill acquisition, higher perfor-
mance scores, better learning transfer to the performance
with an actual robotic system, and exhibit less mental
workload substantially.
Methods
Participants
This IRB-approved study was performed in the Robotic
Surgery Training Laboratory of the Minimally Invasive
Surgical Training and Innovation Center (MISTIC) at
Johns Hopkins University School of Medicine (JHSOM).
Thirty-two (n = 32) surgical residents from different spe-
cialties at the JHSOM were recruited for this study.
Informed consent was obtained from all participants. All
participants ranged from 30 to 42 years of age and were
residents or fellows residents who did not have any robotic
training experience previously and were randomly assigned
to one of two groups: the Control Group (CG, n = 16) or
Experiment Group (EG, n = 16).
123
64 Surg Endosc (2018) 32:62–72

Training protocol camera pedals, and ergonomic adjustment settings. Once

Participants were engaged in a training session lasting 5
weeks. A detailed training and evaluation plan is summa-
rized in Table 1. Once potential study participants were
identified, an email was sent to them to provide a link to a
server where the example performance videos were avail-
able. These video files included example performances of
the four VR simulation task and the four physical model
tasks. Subjects were instructed to watch these videos before
coming to their orientation sessions to learn about the tasks
which they will perform later for skill learning and eval-
uations. The purpose of these videos was to help partici-
pants’ understanding of the individual tasks and their goals,
but they did not necessarily demonstrate the expert ways to
perform each task. During their orientation sessions, these
videos were reviewed again with the research team mem-
ber again to make sure the participants were familiar with
the VR simulation and physical model tasks before the
actual involvements in the study. The research team
member provided a didactic orientation to address the brief
history of robotic surgery, its nature as a remote surgery,
main system components, and how to use the surgeon
console including the master controller, clutch buttons,
confirmed that each participant had no questions regarding
the tasks and goals, subjects tried each of the tasks once
with no scores recorded.
When a VR simulation task is about to be launched, the
simulator displayed the list of purpose and objectives of
each task, and this information page was reviewed with the
research team member so that all participants, regardless of
their group assignments, could have a clear understanding
of the tasks. After the completion of each simulation task,
the simulator displayed the performance evaluation pages
including PMs. During this orientation, no explanations
were provided by the research team member because only
the EG participants should receive additional mentoring
later during their study participations. All of these initial
instructions and explanations were given to the participants
in both CG and EG and were not a part of the mentoring
which this study investigated.
After the orientation, the pre-training evaluation was
performed for each participant, and the details of this
evaluation are described in Pre-, Peri-, and Post-Training
Evaluations. Each participant attended 5-week training
sessions as shown in Table 1. During each session, par-
ticipants repeated each VR simulation training task three

Table 1 Training/evaluation protocol and tasks

Weeks Description

Week 0 Watching the example videos of the four simulation training and four physical model tasks.
Week 1 1. Introduction and system orientation: Basic learning about system master manipulation, control pedals, clutches, instrument, and
camera navigation
2. Review of the simulation and physical model tasks and single trial with no score recording
3. Pre-training data collection (Simulator and da Vinci system)
Four simulation training task performance at da Vinci Skills Simulator
Four physical model task performance with da Vinci robot system
Base-line performances and mental workloads were measured
4. 32 subjects were randomly assigned to the EG or CG group.
Week 2 Skill training at the da Vinci Skills Simulator
EG group (16 subjects) CG group (16 subjects)
1. First performance of four simulation tasks 1. First performance of four simulation tasks
2. Mentoring provision 2. Metric review at their own discretion
3. Second performance of four simulation tasks 3. Second performance of four simulation tasks
4. Mentoring provision 4. Metric review at their own discretion
5. Third performance of four simulation tasks 5. Third performance of four simulation tasks
6. Peri-training evaluation 6. Peri-training evaluation
7. Survey 7. Survey
Week 3–4 Skill training and peri-training evaluation (same as week 2)
Week 5 1. Same as week 2
2. The peri-training evaluation data collected in week 5 was used as post-training evaluation data for the simulator
3. Post-training evaluation data collection with da Vinci robot system

123
Surg Endosc (2018) 32:62–72 65

times. After completing the training, participants were
asked to repeat the four training tasks for evaluation (de-
scribed below). Surgeons were trained using a standardized
number of repetitions (not training time) to accommodate
better anticipated time differences between training meth-
ods and whether or not participants use provided feedback.
Training and evaluation tasks
Training sessions included the four selected VR training
tasks such as Camera Targeting 2, Ring and Rail 2, Thread
the Ring, and Continuous Suturing at the da Vinci Skills
Simulator (Table 2). These four training tasks from the
simulator were used both for skills training and evaluation.
These tasks were selected based on published data docu-
menting their construct and content validities [26, 35, 36].
This selection was also influenced by the four categories of
essential robotic skills (camera navigation and clutch con-
trol, wrist manipulation, needle driving, and suturing)
identified at the Fundamentals of Robotic Surgery (FRS)
consortium meetings. Additionally, four physical model
tasks such as Lazy Susan tower, Loopy Rollercoaster, cus-
tom-developed needle passing and driving, and custom-de-
veloped tension running suturing considered compatible to
the selected simulation tasks in terms of the skills required to
perform tasks were identified (Table 2). These physical
model tasks were used for the pre- and post-training evalu-
ations to examine how much the skill learning using the VR
simulation training models will be transferred to the actual
robot performance using these physical model tasks. These
physical model tasks were not used as a part of the training
because the comparison between the VR simulation training
and physical model training was not a part of this study’s
scope. Only the VR training simulation tasks were used for
participants’ skill learning.
Pre-, peri-, and post-training evaluations
To investigate the learning curves and the changes in mental
workload experienced by each participant throughout the
training program, pre-, peri-, and post-training evaluations
were conducted. Two pre-training evaluations were per-
formed: one with the VR simulator and another with the da
Vinci robot. Pre-training evaluations at the VR simulator
consisted of all subjects performing each of the four VR
simulated tasks once before beginning the training program.
The simulator’s PM scores were recorded, and the mental
workloads were assessed using several cognitive tools
described in detail in the Cognitive Workload Assessment
section. These data comprised participants’ base-line data-
set. Additional pre-training evaluations at the da Vinci robot
consisted of all subjects performing four physical model
tasks that were different from but relatable to the four VR
simulation training tasks. Due to the lack of automated PM
with the da Vinci robot, we recorded the scope view during
the performance evaluation and quantified the task perfor-
mance based on certain metrics including time, accuracy, the
number of errors, and how often an instrument was moved
out of the camera view. Mental workload was assessed as
well.
Peri-training evaluations were only done with the VR
simulator. During each session, participants repeated each
training task two times. After the repetition, participants
were asked to repeat the four training tasks one more time
for evaluation. The participants’ PM scores were recorded,
and their mental workloads were assessed using the same
evaluation methods used to assess peri-training
performance.
Finally, post-training evaluations took place immedi-
ately following the last training session of the 4-week
training program. Using the same evaluation methods used
in pre-training evaluation, each participant’s performance
and mental workload were measured at both the VR sim-
ulator and the da Vinci system for post-training evaluation.
Feedback provision
To maintain consistency in providing guidance, a set of the
explanations and instructions associated with any PM were
documented prior to the subject recruitment. The expla-
nations and instructions were then uniformly provided to
all EG group trainees. When any individual metrics did not
receive green check-mark symbols, but yellow triangle or
red X mark, feedback for improving the metric scores were
provided for trainees.
Study procedures
As described earlier, participants had an orientation session
to establish a basic level of competency using the system’s

Table 2 VR training tasks and

Training category
physical model tasks
Camera and clutching
Endo-wrist manipulation
Needle driving
Suturing
VR simulator task
Camera Targeting 2
Ring and Rail 2
Thread the Ring
Continuous Suturing
Physical model task
Lazy Susan tower
Loopy Rollercoaster
Custom needle passing/driving
Custom tension running suturing

123
66
master manipulators and control pedals to accomplish basic
instrument movement and camera navigation at the
beginning of their participations and to understand the
skills, goals, and tasks which they will perform. After this
training, the pre-training evaluation was performed. Par-
ticipating surgeons were then randomized to the CG or the
EG.
After each task performance, there was a break for
reviewing the PM which was provided by the VR simu-
lator. The CG participants were allowed to spend time on
the metric review alone and at their discretion. However,
the EG participants worked with a research team member
to review the metrics together. This intervention provided
an additional opportunity for the research team member to
provide instructions and explanations regarding the prac-
tical meanings of individual PM, instructions on how to
improve specific metric scores, and specific information
regarding which metrics should be improved in order to
earn better global performance scores as needed or
requested by the participant. This research team member
who provided this orientation session and mentoring during
their learning was the Director of Robotic Surgery Edu-
cation. This person developed a comprehensive robotic
surgery training curriculum and offered the basic and
advanced level training to Johns Hopkins surgeons from
multiple-specialties using this curriculum.
During the reviews, the actual time spent on PM (TPM)
for each participant was recorded to assess indirect atten-
tion levels for data analysis. At the end of the each day’s
training session, participants were asked to complete a brief
survey documenting their perception of the usefulness of
the PM on improving their robotic skill learning.
Cognitive workload assessments
Mental workload assessments comprised a vital portion of
the evaluations included in this study since lower mental
workload is associated with ease of task performance and
ability to assimilate information efficiently. The assessment
was performed by using several psychologic survey tools
including National Aeronautics and Space Administration-
Task Load Index (NASA-TLX), Multiple Resources
Questionnaire (MRQ) and Short Stress State Questionnaire
(SSSQ). NASA-TLX allowed participants to rate their
perceived levels of mental demand, physical demand,
temporal demand, effort, performance, and frustration
during task execution. MRQ is a subjective mental work-
load assessment characterizing the nature of the mental
process used in task performance. Some examples are the
auditory process, manual process, and spatial attentive
process. SSSQ is a subject mental stress assessment tool.
The questions can be categorized to investigate three
mental stresses. Engagement demonstrates subjects’
123
Surg Endosc (2018) 32:62–72
interest, motivation, and concentration, distress represents
unpleasant mood and tension with the lack of confidence
and perceived control, and worry represents self-esteem
and cognitive interference.
Results
Learning progress with the simulation training
Figure 1 shows the learning progress with the simulation
training over the training weeks for each task. The pre-
training scores of the CG and EG were statistically com-
pared, and no significant differences were found between
the two groups. This result demonstrated the comparability
on the pre-training evaluations of the CG and EG perfor-
mance scores. The CG participants’ post-training simula-
tion performance scores (82.9 ± 6.0) for all the tasks were
substantially lower than the EG subjects’ performance
scores (93.2 ± 4.8) (p \ 0.05). It was demonstrated that
the performance scores of the CG participants plateau early
at the lower performance score for several tasks. When we
examined the learning slopes across the training weeks, it
was found that the EG participants demonstrated signifi-
cantly greater improvements between the pre-training and
the first training session when compared with the CG data
(p \ 0.05). It seemed that the EG participants learned more
during the first session and so could perform better than the
CG. For the following weeks, both displayed similar
learning effects over training weeks. With this learning
trend, the EG demonstrated greater performance improve-
ment with all tasks except for the Camera Targeting 2 task.
Simulator performance score improvements
after training
Figure 2A shows the amount of performance improve-
ments—the difference between pre- and post-training per-
formance scores—for each simulation training task for
each group. After the simulation training, both groups
showed performance improvement; however, the EG
exhibited significantly greater improvement than the CG
participants with three simulation tasks including the Ring
and Rail 2, Thread the Ring, and Continuous Suturing.
However, the improvements exhibited by the two groups
were not statistically different with the Camera Targeting 2
task.
Actual time spent on performance metrics (TPM)
The TPM was used as an indirect measure of trainees’
attention levels with PM, and the results from both groups
are shown in Fig. 3. The EG participants spent 76.6 s on
Surg Endosc (2018) 32:62–72 67

Fig. 1 The learning progress

with the simulation training
over the training weeks for each
task. A Camera Targeting 2,
B Ring and Rail 2, C Thread the
Ring, and D Continuous
Suturing

Fig. 2 The amount of performance improvements which was calculated as the difference between pre- and post-training performance scores for
A simulation training tasks and B bench-top da Vinci training tasks for each group

average for reviewing the PM and receiving mentoring
feedback from a research team member. The TPMs for the
EG trainees were initially very long due to the provision
of explanations about performance evaluation page and
the details regarding individual PM. However, it
substantially shortened as the EG trainees became more
familiar with the PM and instructions. The CG partici-
pants spent only 11.6 s on average for the review, and
their TPM did not significantly change across the training
weeks.

123
68
Fig. 3 Actual time spent on performance metrics (TMP)
Robot performance improvements after training
Figure 2B shows the amount of performance improve-
ments as skill learning transfer which was calculated as the
difference between pre- and post-training performance
scores for each bench-top training task for each group at
the da Vinci robot. After simulation training, both CG and
EG showed improvements. Similar to the results of the
simulation performance improvement, the EG showed
greater improvements with training. The EG showed sig-
nificantly greater performance improvement after simula-
tion training when performing the Lazy Susan tower and
tension running suturing tasks (p \ 0.05). These were no
significant differences between groups with the Loopy
Rollercoaster and needle driving tasks.
NASA-TLX analysis with the simulation training
The NASA-TLX global scores from the CG and EG par-
ticipants are shown in Fig. 4A. The result presented that
the both groups had decreased global workload as they
participated in the simulation training. A significant
Training 9 Group interaction showed that the EG had a
greater decrease in the global workload than the CG
(p \ 0.05). When we looked into the individual scales, it
was found that the Training 9 Group interaction was
mainly contributed by the ‘performance’ scale. The EG
participants exhibited higher performance satisfaction level
as they received more training than the CG subjects. This
result agreed with the performance evaluation analysis
which showed that the EG’s performance improvement
was greater with training.
123
Surg Endosc (2018) 32:62–72
NASA-TLX analysis with the robot performances
The NASA-TLX workload analysis with the da Vinci
performance is shown in Fig. 4B. Similar to what we
observed in the NASA-TLX results with the simulation
training, both the groups had decreased global workload
after training. However, a significant Training 9 Group
interaction showed that the EG had a greater decrease in
the global workload after training (p \ 0.05). When the
individual scales were examined, the EG participants
exhibited higher performance satisfaction levels with the
lower level of effort involved after training.
Other cognitive workload analysis
SSSQ analysis data are displayed in Fig. 5A, B. We found
that the EG participants exhibited substantially decreased
distress levels after training (p \ 0.05), while the CG
participants showed no significant difference between the
pre- and post-training data. It was also observed that the
EG showed higher improvement in engagement levels after
training when compared with the CG (p \ 0.05). SSSQ
analysis on worry level did not show any differences
between the two groups.
Figure 5C shows the global rating of the MRQ analysis
which compared how differently each group used resources
for their task performance. Our results demonstrated that
the CG used less amount of cognitive resources while the
EG involved more cognitive resources after training.
Performance metric survey analysis
After each training session, all participants completed a PM
survey with the multiple statements. The rating results from
the both groups are summarized in Table 3 with p values
displaying statistically significant differences between the
two groups. Each participant rated their impressions on
each statement from 1 to 5 with 1 for strongly disagree, 3
for neutral, and 5 for strongly agree. For creating Table 3,
their ratings were averaged over training sessions because
the preliminary statistical analysis had shown that their
ratings did not significantly change over training sessions.
Overall, the EG participants rated significantly higher than
the CG for each statement. For the EG participants, there
were two extra questions. These questions asked how
helpful the extra instructional feedback provided during
their training sessions were for better understanding the
meaning of each PM and the strategies on how to improve
their performance. The EG participants rated the extra
feedback very helpful in both aspects (4.85 ± 0.36 for
both).
Surg Endosc (2018) 32:62–72
Fig. 4 The NASA-TLX global
scores with the A VR simulator
and B da Vinci robotic system
Fig. 5 Additional cognitive
workload assessments:
A SSSQ—distress, B SSSQ—
engagement, C MRQ—global
rate
Discussion
This study investigated how practically and sufficiently the
VR simulator for robotic surgery training could offer self-
learning and mentor-free learning environment for trainees
and whether additional mentoring could still enhance the
trainees’ robotic skill learning using the VR simulator.
Because both the EG and CG participants exhibited skill
learning, it was exhibited that the current VR simulator
provided trainees with a certain degree of self-learning.
However, the lower post-training performance scores from
the CG participants demonstrated that the VR simulation
training was not as effective when trainees used the PMs at
their own discretion without having a full understanding
about the PMs and they need to figure out how to improve
69
their simulation task performance by themselves. The EG
participants who received the additional explanations and
instructions showed better skill learning at the VR simu-
lator and improved skill transfer at the da Vinci system
with higher satisfaction level regarding their performance
and learning. These data confirmed that proper mentoring
still played a key role in VR simulation education for
robotic surgery and suggested that extra mentoring should
be included in the VR simulation training curriculum to
improve the training efficiency and effectiveness. Although
the EG participants initially spent longer time on PM as
they learned about the PM and received additional men-
toring, the TPM became shorter as the trainees got more
familiar with the PM and instructions. When we examined
the learning progress, the largest differences in learning
123
70
Table 3 Performance metric survey
Statements
1. The overall performance scores were very helpful for my training today.
2. I knew how to obtain higher overall performance scores.
3. I confidently understood the meaning of each performance metric.
4. The individual performance metrics were very helpful for my training today.
5. I knew specific strategies for higher individual metric scores.
6. I understood well how overall performance score was connected with individual metrics.
7. I feel my skill is improving.
8. Extra instructions help me to better understand the meaning of each performance metric.
9. Extra instructions help me to better understand how I can improve my performance.
between two groups were observed during the first training
session as the EG participants demonstrated greater skill
improvement during the session, whereas both groups
displayed similar slopes in their skill improvements over
the following training weeks as shown in Fig. 1. Most EG
participants reached over 90% in their overall performance
scores for each task. However, most CG participants had
their scores lower than 90%. After the completion of CG
subjects’ participation, the research team provided the CG
participants with the same explanations and instructions
which were used for the EG participants and asked them to
perform each of the four simulation training tasks once
more. It was observed that most CG participants showed
immediate score improvement to the similar score levels of
the EG participants. These extra task performance scores
were not officially collected for all CG participants, so no
statistical analysis on these could be performed. However,
this was another confirmation that self-learning can be
improved with proper mentoring. Reflecting what we
learned from this study, our current VR simulator training
begins with the comprehensive explanations regarding the
performance evaluation pages and PMs. After performing a
few VR training tasks and reviewing the performance
evaluation pages and PMs, trainees continue their skill
training by themselves and our robotic surgery skill edu-
cator provides skill improving instructions when needed or
requested by trainees.
The cognitive workload analysis presented that the EG
group exhibited substantially decreased cognitive work-
loads after their training in terms of decreased NASA-
TLX global workload, higher engagement with lower
distress levels. The MRQ analysis showed that the EG
participants’ better performances involved more cognitive
resources during their task performances. Also, the survey
data showed the EG subjects better understood about the
PM, highly acknowledged the value of extra explanations
and instructions, and better knew the strategies that were
123
Surg Endosc (2018) 32:62–72
CG rating EG rating p
4.25 ± 0.59 4.83 ± 0.38 \0.05
3.80 ± 0.91 4.83 ± 0.38 \0.05
3.28 ± 1.26 4.70 ± 0.46 \0.05
3.98 ± 0.83 4.75 ± 0.49 \0.05
3.33 ± 1.07 4.73 ± 0.45 \0.05
3.49 ± 1.10 4.58 ± 0.68 \0.05
4.18 ± 0.81 4.73 ± 0.45 \0.05
N/A 4.85 ± 0.36 N/A
N/A 4.85 ± 0.36 N/A
required for performance improvement. These outcomes
well-reflected how practically the learners perceived the
importance of proper mentoring provision during their
simulation training. It is often assumed that surgical
trainees can learn fundamental surgical skills using VR
surgical simulators without having experienced mentors
or educators with them because the simulators provide
trainees with performance feedback and the trainees can
learn from the feedback. Our study demonstrated that the
current voluntary use of the VR system is not as effective
as it potentially could be and VR simulators could not
provide trainees with a self-driven and mentor-free
learning environment as many assumed or expected. The
provision of extra explanations and instructions will still
play a very important role in complementing the benefits
in VR simulation training with PM and in making the
current surgical VR training more effective, efficient, and
accurate.
During this study, it was noticed that many participants
did not have a clear understanding about two PMs, the
motion economy, and master workspace range. The par-
ticipants were not sure if these metrics were associated
with the instruments, camera, master controllers, or
something else. The motion economy measures the total
distance traveled by all instrument during an exercise.
However, the camera motion was not included. The master
workspace range reports the larger of the two radii of
motion of the user’s working volume on the master con-
trollers. Because the definitions of these metrics were not
intuitively understood by study participants, most of CG
participants did not know how they could improve the
scores associated with these metrics. When VR surgical
simulators provide with the more user-friendly interface
with their PM, the surgeons will have a clear understanding
regarding the metrics so they can repeat each training tasks
applying score improving strategies accordingly.
Surg Endosc (2018) 32:62–72
Conclusions
Our study demonstrated that the robotic surgery trainees
had a limited self-learning and mentor-free skill learning
when trained using the current VR simulator and this
limitation could be reduced by providing trainees with
additional mentoring for their better understanding of the
PMs and skill improving techniques for more effective and
efficient VR robotic simulation education. As such men-
toring explanations and instructions are traditionally pro-
vided by experienced educators, this approach still poses a
challenge in surgical education due to the restriction caused
by the limited availability of the educators. Therefore, our
research endeavor in improving traditional surgical edu-
cation will continue by developing a surgeon-friendly
mentoring interface which can provide trainees with the
useful mentoring instructions in an electronic delivery.
Such virtual mentoring will help trainees to avoid repeating
the same mistakes that cause lower level task performance
and guide them to understand the goal of each skill training
better so that they can achieve more proficient skill
learning.
Acknowledgements Dr. Gyusung Lee received the Association for
Surgical Education Center for Excellence in Surgical Education,
Research and Training (CESERT) Grant and Intuitive Surgical
Clinical Robotic Research Grant as the Principle Investigator of this
study. The authors acknowledge the generous support of Dr. Michael
Marohn, the Director of MISTIC, and the thoughtful and careful
assistance of Karyn Rhyder in editing this manuscript.
Funding This project was supported by the Association for Surgical
Education Center for Excellence in Surgical Education, Research and
Training (CESERT) Grant and Intuitive Surgical Clinical Robotic
Research Grant.
Compliance with ethical standards
Disclosures Dr. Gyusung Lee received the Association for Surgical
Education Center for Excellence in Surgical Education, Research and
Training (CESERT) grant and Intuitive Surgical Clinical Robotic
Research Grant for this study. Dr. Mija Lee was the co-investigator in
this study and is Dr. Gyusung Lee’s spouse.
References
1. Ahmed K, Khan MS, Vats A, Nagpal K, Priest O, Patel V, Vecht
JA, Ashrafian H, Yang GZ, Athanasiou T, Darzi A (2009) Cur-
rent status of robotic assisted pelvic surgery and future devel-
opments. Int J Surg 7(5):431–440. doi:10.1016/j.ijsu.2009.08.008
2. Patel VR, Tully AS, Holmes R, Lindsay J (2005) Robotic radical
prostatectomy in the community setting–the learning curve and
beyond: initial 200 cases. J Urol 174(1):269–272. doi:10.1097/01.
ju.0000162082.12962.40
3. Patel MN, Bhandari M, Menon M, Rogers CG (2009) Robotic-
assisted partial nephrectomy. BJU Int 103(9):1296–1311. doi:10.
1111/j.1464-410X.2009.08584.x
4. Anger JT, Mueller ER, Tarnay C, Smith B, Stroupe K, Rosenman
A, Brubaker L, Bresee C, Kenton K (2014) Robotic compared
71
with laparoscopic sacrocolpopexy: a randomized controlled trial.
Obstet Gynecol 123(1):5–12. doi:10.1097/AOG.0000000000000
006
5. Martino MA, Berger EA, McFetridge JT, Shubella J, Gosciniak
G, Wejkszner T, Kainz GF, Patriarco J, Thomas MB, Boulay R
(2014) A comparison of quality outcome measures in patients
having a hysterectomy for benign disease: robotic vs. non-robotic
approaches. J Minim Invasive Gynecol 21(3):389–393. doi:10.
1016/j.jmig.2013.10.008
6. Ma J, Shukla PJ, Milsom JW (2011) The evolving role of robotic
colorectal surgery. Dis Colon Rectum 54(3):376–377. doi:10.
1007/DCR.0b013e318204a8d5
7. Wilson EB (2009) The evolution of robotic general surgery.
Scand J Surg 98(2):125–129
8. You JY, Lee HY, Son GS, Lee JB, Bae JW, Kim HY (2013)
Comparison of robotic adrenalectomy with traditional laparo-
scopic adrenalectomy with a lateral transperitoneal approach: a
single-surgeon experience. Int J Med Robot Comput Assist Surg
MRCAS 9(3):345–350. doi:10.1002/rcs.1497
9. Orady M, Hrynewych A, Nawfal AK, Wegienka G (2012)
Comparison of robotic-assisted hysterectomy to other minimally
invasive approaches. J Soc Laparoendosc Surg 16(4):542–548.
doi:10.4293/108680812X13462882736899
10. Dulan G, Rege RV, Hogg DC, Gilberg-Fisher KM, Arain NA,
Tesfay ST, Scott DJ (2012) Developing a comprehensive, profi-
ciency-based training program for robotic surgery. Surgery
152(3):477–488. doi:10.1016/j.surg.2012.07.028
11. Rashid HH, Leung YY, Rashid MJ, Oleyourryk G, Valvo JR,
Eichel L (2006) Robotic surgical education: a systematic
approach to training urology residents to perform robotic-assisted
laparoscopic radical prostatectomy. Urology 68(1):75–79. doi:10.
1016/j.urology.2006.01.057
12. Antiel RM, Van Arendonk KJ, Reed DA, Terhune KP, Tarpley
JL, Porterfield JR, Hall DE, Joyce DL, Wightman SC, Horvath
KD, Heller SF, Farley DR (2012) Surgical training, duty-hour
restrictions, and implications for meeting the Accreditation
Council for Graduate Medical Education core competencies:
views of surgical interns compared with program directors. Arch
Surg 147(6):536–541. doi:10.1001/archsurg.2012.89
13. Ahmed N, Devitt KS, Keshet I, Spicer J, Imrie K, Feldman L,
Cools-Lartigue J, Kayssi A, Lipsman N, Elmi M, Kulkarni AV,
Parshuram C, Mainprize T, Warren RJ, Fata P, Gorman MS,
Feinberg S, Rutka J (2014) A systematic review of the effects of
resident duty hour restrictions in surgery: impact on resident
wellness, training, and patient outcomes. Ann Surg
259(6):1041–1053. doi:10.1097/SLA.0000000000000595
14. Lee JY, Mucksavage P, Sundaram CP, McDougall EM (2011)
Best practices for robotic surgery training and credentialing.
J Urol 185(4):1191–1197. doi:10.1016/j.juro.2010.11.067
15. Liu JJ, Gonzalgo ML (2011) Minimally invasive training in
urologic oncology. Arch Esp Urol 64(9):865–868
16. Schreuder HW, Oei G, Maas M, Borleffs JC, Schijven MP (2011)
Implementation of simulation in surgical practice: minimally
invasive surgery has taken the lead: the Dutch experience. Med
Teach 33(2):105–115. doi:10.3109/0142159X.2011.550967
17. Johnson E (2007) Surgical simulators and simulated surgeons:
reconstituting medical practice and practitioners in simulations.
Soc Stud Sci 37(4):585–608
18. Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP,
Moses G, Smith CD, Satava RM (2005) Virtual reality simulation
for the operating room: proficiency-based training as a paradigm
shift in surgical skills training. Ann Surg 241(2):364–372
19. van Empel PJ, Verdam MG, Strypet M, van Rijssen LB, Huirne
JA, Scheele F, Bonjer HJ, Meijerink WJ (2012) Voluntary
autonomous simulator based training in minimally invasive
123
72
surgery, residents’ compliance and reflection. J Surg Educ
69(4):564–570. doi:10.1016/j.jsurg.2012.04.011
20. Chan B, Martel G, Poulin EC, Mamazza J, Boushey RP (2010)
Resident training in minimally invasive surgery: a survey of
Canadian department and division chairs. Surg Endosc
24(3):499–503. doi:10.1007/s00464-009-0611-3
21. Rosenthal R, Schafer J, Hoffmann H, Vitz M, Oertli D, Hahnloser
D (2013) Personality traits and virtual reality performance. Surg
Endosc 27(1):222–230. doi:10.1007/s00464-012-2424-z
22. Pitzul KB, Grantcharov TP, Okrainec A (2012) Validation of
three virtual reality Fundamentals of Laparoscopic Surgery (FLS)
modules. Stud Health Technol Inform 173:349–355
23. Brinkman WM, Buzink SN, Alevizos L, de Hingh IH, Jaki-
mowicz JJ (2012) Criterion-based laparoscopic training reduces
total training time. Surg Endosc 26(4):1095–1101. doi:10.1007/
s00464-011-2005-6
24. Kang SG, Yang KS, Ko YH, Kang SH, Park HS, Lee JG, Kim JJ,
Cheon J (2012) A study on the learning curve of the robotic
virtual reality simulator. J Laparoendosc Adv Surg Tech Part A
22(5):438–442. doi:10.1089/lap.2011.0452
25. Lerner MA, Ayalew M, Peine WJ, Sundaram CP (2010) Does
training on a virtual reality robotic simulator improve perfor-
mance on the da Vinci surgical system? J Endourol
24(3):467–472. doi:10.1089/end.2009.0190
26. Hung AJ, Zehnder P, Patil MB, Cai J, Ng CK, Aron M, Gill IS,
Desai MM (2011) Face, content and construct validity of a novel
robotic surgery simulator. J Urol 186(3):1019–1024. doi:10.1016/
j.juro.2011.04.064
27. Kenney PA, Wszolek MF, Gould JJ, Libertino JA, Moinzadeh A
(2009) Face, content, and construct validity of dV-trainer, a novel
virtual reality simulator for robotic surgery. Urology
73(6):1288–1292. doi:10.1016/j.urology.2008.12.044
28. Perrenot C, Perez M, Tran N, Jehl JP, Felblinger J, Bresler L,
Hubert J (2012) The virtual reality simulator dV-Trainer((R)) is a
valid assessment tool for robotic surgical skills. Surg Endosc
26(9):2587–2593. doi:10.1007/s00464-012-2237-0
29. Lee JY, Mucksavage P, Kerbl DC, Huynh VB, Etafy M,
McDougall EM (2012) Validation study of a virtual reality
robotic simulator–role as an assessment tool? J Urol
187(3):998–1002. doi:10.1016/j.juro.2011.10.160
30. Brinkman WM, Luursema JM, Kengen B, Schout BM, Witjes JA,
Bekkers RL (2013) da Vinci skills simulator for assessing
learning curve and criterion-based training of robotic basic skills.
Urology 81(3):562–566. doi:10.1016/j.urology.2012.10.020
31. Stefanidis D, Wang F, Korndorffer JR Jr, Dunne JB, Scott DJ
(2010) Robotic assistance improves intracorporeal suturing
123
Surg Endosc (2018) 32:62–72
performance and safety in the operating room while decreasing
operator workload. Surg Endosc 24(2):377–382. doi:10.1007/
s00464-009-0578-0
32. Stefanidis D, Hope WW, Scott DJ (2011) Robotic suturing on the
FLS model possesses construct validity, is less physically
demanding, and is favored by more surgeons compared with
laparoscopy. Surg Endosc 25(7):2141–2146. doi:10.1007/s00464-
010-1512-1
33. Patel YR, Donias HW, Boyd DW, Pande RU, Amodeo JL,
Karamanoukian RL, D’Ancona G, Karamanoukian HL (2003)
Are you ready to become a robo-surgeon? Am Surg
69(7):599–603
34. Shaligram A, Meyer A, Simorov A, Pallati P, Oleynikov D
(2013) Survey of minimally invasive general surgery fellows
training in robotic surgery. J Robot Surg 7(2):131–136. doi:10.
1007/s11701-012-0355-2
35. Hung AJ, Patil MB, Zehnder P, Cai J, Ng CK, Aron M, Gill IS,
Desai MM (2012) Concurrent and predictive validation of a novel
robotic surgery simulator: a prospective, randomized study.
J Urol 187(2):630–637. doi:10.1016/j.juro.2011.09.154
36. Finnegan KT, Meraney AM, Staff I, Shichman SJ (2012) da Vinci
Skills Simulator construct validation study: correlation of prior
robotic experience with overall score and time score simulator
performance. Urology 80(2):330–335. doi:10.1016/j.urology.
2012.02.059
37. Johnston MJ, Paige JT, Aggarwal R, Stefanidis D, Tsuda S,
Khajuria A, Arora S (2016) An overview of research priorities in
surgical simulation: what the literature shows has been achieved
during the 21st century and what remains. Am J Surg
211(1):214–225. doi:10.1016/j.amjsurg.2015.06.014
38. Panait L, Rafiq A, Tomulescu V, Boanca C, Popescu I, Carbonell
A, Merrell RC (2006) Telementoring versus on-site mentoring in
virtual reality-based surgical training. Surg Endosc
20(1):113–118. doi:10.1007/s00464-005-0113-x
39. Alaker M, Wynn GR, Arulampalam T (2016) Virtual reality
training in laparoscopic surgery: a systematic review & meta-
analysis. Int J Surg 29:85–94. doi:10.1016/j.ijsu.2016.03.034
40. Ahlborg L, Weurlander M, Hedman L, Nisel H, Lindqvist PG,
Fellander-Tsai L, Enochsson L (2015) Individualized feedback
during simulated laparoscopic training:a mixed methods study.
Int J Med Educ 6:93–100. doi:10.5116/ijme.55a2.218b
41. Paschold M, Huber T, Zeissig SR, Lang H, Kneist W (2014)
Tailored instructor feedback leads to more effective virtual-re-
ality laparoscopic training. Surg Endosc 28(3):967–973. doi:10.
1007/s00464-013-3258-z