Virtual Reality (2020) 24:527–539

https://doi.org/10.1007/s10055-019-00415-8

ORIGINAL ARTICLE

An augmented reality‑based training system with a natural user

interface for manual milling operations
Chih‑Kai Yang1 · Yu‑Hsi Chen1 · Tung‑Jui Chuang1 · Kalpana Shankhwar1 · Shana Smith1

Received: 27 July 2019 / Accepted: 27 November 2019 / Published online: 3 December 2019
© Springer-Verlag London Ltd., part of Springer Nature 2019

Abstract
This study developed an augmented reality (AR)-based training system for conventional manual milling operations. An Intel
RealSense R200 depth camera and a Leap Motion controller were mounted on an HTC Vive head-mounted display to allow
users freely walk around in a room-size AR environment to operate a full-size virtual milling machine with their barehands,
using their natural operation behaviors, as if they were operating a real milling machine in the real world, without additional
worn or handheld devices. GPU parallel computing was used to handle dynamic occlusions and accelerate the machining
simulation to achieve a real-time simulation. Using the developed AR-based training system, novices can receive a hands-on
training in a safe environment, without any injury or damage. User test results showed that using the developed AR-based
training resulted in lower failure rates and inquiry times than using video training. Users also commented that the AR-based
training was interesting and helpful for novices to learn the basic manual milling operation techniques.

Keywords Augmented reality · Natural operation behavior · Manual milling operation · Occlusion

1 Introduction avoid collisions. Chardonnet et al. (2017) used a handheld

Machining operations are often complicated and dangerous.
Because of safety issues and the lack of one-to-one train-
ing resources, many machine tool trainings are not effec-
tive. Some researchers started to apply virtual reality (VR)
or augmented reality (AR) to assist machining trainings
to avoid potential injury or damage. For example, Zhang
et al. (2008) developed an AR-assisted computer numerical
control (CNC) machining simulation system. The system
allowed operators to analyze machining processes on real
machines. Neugebauer et al. (2010) created a VR-based
numerical control (NC) milling machine system by com-
bining a virtual environment with an actual control panel.
The system enabled novices to quickly comprehend complex
operation processes and recognize potential errors without
any extra cost or danger. Kiswanto and Ariansyah (2013)
developed an AR-based 3-axis CNC machining simulation
system to machine a virtual workpiece on a real milling
machine. Users can validate their machining processes to
* Shana Smith
ssmith@ntu.edu.tw
1
Department of Mechanical Engineering, National Taiwan
University, Taipei, Taiwan
device to display AR images to assist CNC machining. Real-
time operation conditions and cutting tool motions were dis-
played on the handheld device.
Although NC machines have many advantages over con-
ventional machine tools, most small-quantity and low-cost
projects are still carried out by conventional machine tools.
Conventional machining training is an important job train-
ing in vocational education. Although many prior studies
applied AR or VR technologies to machining simulations,
most of them were applied to NC machines, which do not
require heavy manual skills. On the contrary, operating a
conventional manual machine tool requires a highly skilled
operator. However, there was very little research on applying
AR or VR in conventional manual machine tool training.
The purpose of this study was to create a realistic AR-
based training system for novices to learn how to operate
a manual milling machine at beginner’s level. The system
provided users with a natural user interface that allowed
users to interact with virtual milling machines with their
natural operation behaviors as if they were operating a real
machine in the real world, without additional worn or hand-
held devices. Furthermore, the system also used GPU par-
allel computing to provide dynamic occlusion effects and

13
Vol.:(0123456789)
528
real-time machining simulation to increase the realism and
immersiveness of the simulation.
1.1 Interaction interfaces
In an AR environment, an effective and real-time interac-
tion interface not only enhances the accuracy of simulation
but also increases the sense of presence. Interaction inter-
faces can be natural or non-natural. Non-natural interactions
often use an intermediary stylus or wand, e.g., PHANToM,
Razer Hydra, or HTC Vive controller, to interact with vir-
tual objects. Users need to take more time to learn how to
use these intermediaries. In addition, the usages of inter-
mediaries are often different from users’ natural operation
behaviors in operating a real machine tool in the real world.
Thus, the AR training might not be effective. However, an
intermediary device often provides a higher controllability
and stability.
Some other non-natural interactions use gestures to inter-
act with virtual objects. For example, Shim et al. (2016)
developed a gesture-based AR system to interact with vir-
tual objects without extra intermediary devices. However,
gesture-based interaction is often not intuitive, neither in
line with human natural operation behaviors. In addition,
users often need to learn and remember complex gestures.
Unlike non-natural interaction interfaces, natural interac-
tions are more intuitive. Regazzoni et al. (2018) stated that
natural user interface design was one of the most important
issues in AR/VR applications, and a good natural user inter-
face allowed users to be concentrated on the achievement of
the final goals instead of monitoring the correct execution
of the gestures. Users often do not need to learn how to use
complex intermediary devices or gestures. They can move
their bodies and interact with objects in a very natural way.
For example, Qiu et al. (2013) proposed a VR assembly and
maintenance simulation system. The system integrated a
data helmet and a data glove into a virtual human control
system. Users can execute tasks by controlling an avatar in a
VR environment to grasp, move, and release virtual objects
naturally in an interactive way. Sportillo et al. (2015) placed
color markers on the thumb and the index fingers to track
hand positions. Through color filtering with a RGB depth
camera, users could grab and drag virtual objects naturally
in a virtual environment.
Recently, with the advances of the sensor technology,
natural interactions without any worn devices become pos-
sible. For example, Kinect is a body motion sensing device
released by Microsoft. It consists of an RGB camera, an
infrared emitter, an infrared camera, and a multi-array
microphone which provide full-body 3D motion track-
ing, facial recognition, and voice recognition capabilities.
Corbett-Davies et al. (2012) used Kinect to capture the
depth information of a fixed scene and analyzed the relative
13
Virtual Reality (2020) 24:527–539
locations of the foreground and the virtual objects to achieve
intuitive interaction and masking. Contrary to Kinect’s full-
body tracking, Leap Motion is a hand tracking device which
can recognize and track hands and fingers. Weichert et al.
(2013) showed that the accuracy of the Leap Motion device
was about 0.2 mm for static setup and 1.2 mm for dynamic
setup.
Penelle and Debeir (2014) promoted the fusion of the
3D data acquired by Leap Motion and Kinect to improve
hand tracking performances. By registering the position and
orientation of the Leap Motion in the reference frame of
Kinect, the system could accurately detect the interactions
between real hands and virtual objects.
Natural user interfaces without worn devices provide
users with more intuitive and immersive sensations. How-
ever, in the prior research, most of the Leap Motion or
Kinect devices were non-movable. It limited the portability
and mobility of the interactions in the AR environments. In
this research, a portable natural user interface integrating
a Leap Motion controller, an Intel RealSense R200 RGB
depth (RGB-D) camera, and an HTC Vive HMD was devel-
oped. The system allowed users to freely walk around in a
room-size AR environment to interact with a full-size con-
ventional manual milling machine using their natural opera-
tion behaviors.
1.2 Occlusion
Occlusion is another significant issue which determines the
degree of presence and the realism of an AR environment.
Without occlusion handling, virtual objects are always ren-
dered on the top layer of the color image captured by an
RGB color camera. Therefore, it is difficult for users to cor-
rectly identify the relative positions of the virtual objects.
Lu and Smith (2009) accessed the images before and after
virtual objects appeared and compared the two images to
compute the potential occlusion area for stereo matching.
The research divided the image into multiple layers and
applied GPU to reduce the matching cost. Gheorghe et al.
(2015) built a virtual CNC machining simulation using AR.
They managed the occlusion problem based on the prior
knowledge of the position and the shape of the real objects
in two particular scenarios. However, their method was not
applicable to the objects which were not in their database.
Leal-Meléndrez et al. (2013) presented a pixel-wise
occlusion handling method using Kinect to achieve real-time
occlusion. Khattak et al. (2014) fixed a Leap Motion con-
troller on a table to track hand movements and attached an
RGB-D (RGB depth) camera on an Oculus HMD to create
an AR environment. Their method first captured the static
environment and reconstructed the environment using the
RGB-D camera. Then, they compared the depth map of the
Virtual Reality (2020) 24:527–539
dynamic scene and the static scene to handle occlusions
using a fragment shader.
Although most of the prior occlusion methods can suc-
cessfully handle occlusion problems, the requirement
of expensive computational resources often resulted in
rendering delay, which not only reduced the realism and
Fig. 1  A conventional LC-1 1/2 series milling machine
Fig. 2  System architecture
529
immersiveness of the AR simulation but also caused dis-
comfort to users. A more efficient and intuitive occlusion
method is needed to handle real-time dynamic occlusions
in AR applications.
This paper is organized as follows. Section 2 gives an
overview of the hardware and software employed in this
study. Section 3 describes the unification of different coor-
dinate systems. Section 4 describes the dynamic occlusion
method developed in this study. Section 5 presents the
machining simulation and user interface designs. Section 6
gives a user test. Finally, Sect. 7 offers conclusions.
2 System architecture
This study developed an AR-based training system for an
LC-1 1/2 series conventional manual milling machine,
as shown in Fig. 1. The system architecture is shown in
Fig. 2. An HTC Vive HMD was used to track head posi-
tions and display AR images. In this study, in order to
achieve natural operation interactions, the HTC Vive hand-
held controllers were not used, so that users could operate
the virtual milling machine with their barehands. The Intel
RealSense R200 RGB-D camera was used to capture the
color images and the depth information of the real scenes.
The R200 camera contained three components: an RGB
color camera, an infrared laser projector, and two infrared
cameras which could get the depth information of the real
13
530
scenes. Leap Motion was used to track hand positions.
Leap motion was a tiny hand tracking device containing
three infrared emitters and two infrared cameras.
To increase users’ mobility in the AR environment, the
Real Sense R200 camera and the Leap Motion controller
were fixed on the HTC Vive HMD (as shown in Fig. 3),
so that users’ hand motions could be captured and viewed
according to users’ head position and orientation. It also
allowed users to walk freely in a room-size environment
to interact with a full-size virtual milling machine using
their natural operation behaviors.
Unity3D was a cross-platform game engine developed
by Unity Technologies. Unity3D could use compute shader
to accelerate computing processes on GPU. In this study,
Unity3D was used to build the AR environment. Four
modules were constructed: rendering module, interaction
module, user interface module, and milling machine simu-
lation module.
The rendering module integrated the real scene image
and the virtual world scene and ensures that the results
can be rendered using the HTV Vive HMD. The interac-
tion module integrated user motion information tracked by
Leap Motion and maintained the logics and the rules of the
interaction behaviors between users and virtual objects.
The user interface module provided text or image instruc-
tions to users during the AR-based training. The milling
machine simulation module handled machining logics and
provided dynamic machining behaviors.
Two HTC Vive lighthouses were placed at the two cor-
ners of a 3.0 m × 2.25 m room. When users put on the HTC
Vive HMD, the two lighthouses in the room would track
users’ head movements. HTC Vive handheld controllers
were not used in this study, so that users could operate the
virtual milling machine with their barehands.
Fig. 3  Leap motion and RealSense R200 were fixed on HTC Vive
13
Virtual Reality (2020) 24:527–539
3 Coordinate unification
In the AR system, each device had its own coordinate sys-
tem. These coordinate systems needed to be integrated
together in order to have correct data communication. In
this study, the following coordinates were conformed to
each other:
• Uinty3D (virtual world)
• HTC Vive HMD
• Intel RealSense R200 color camera
• Intel RealSense R200 depth camera
• Leap Motion
There were four unification steps: from HTC Vive to
Unity3D (MVive∕Unity ) , from R200 color camera to Uni-
ty3D (MR200Color∕Unity ) , from Leap Motion to Unity3D
(MLeap∕Unity ), and from R200 depth camera to color camera
(Mdepth∕color ).
3.1 Step 1: from HTC Vive to Unity3D
“SteamVR” from Unity Asset Store was used to manage
Vive in Unity3D. “CameraRig” in “StemVR” was a Unity
gameobject that controlled stereo rendering and head track-
ing. “CameraRig” contained one Unity camera which
anchored at the center between the poses of the left eye and
the right eye. By using “CameraRig”, users could also access
the script “SteamVR_TrackedObject” to get the position and
the orientation of the HMD between the two lighthouses in
each frame. Therefore, the transformation matrix between
Vive and the Unity virtual world could be calculated by:
MVive/Unity = Mt × Mr × Ms (1)
where MVive/Unity = Transformation matrix from Vive to
Unity3D, Mt = Translation matrix, Mr = Rotation matrix,
Ms = Scale matrix, scaling from meter to millimeter.
3.2 Step 2: from Intel RealSense R200 color camera
to Unity3D
In this study, the Intel RealSense R200 color camera was
mounted in the front center of the HMD. Figure 4 shows
the relative position of the color camera to the Vive HMD.
The transformation matrix between the R200 color camera
and Vive was:
⎡ 1 0 0 0 ⎤
⎢ 0 1 0 0.015 ⎥
MR200Color/Vive =⎢ ⎥
⎢ 0 0 1 0.07 ⎥
⎣ 0 0 0 1 ⎦
Virtual Reality (2020) 24:527–539
Fig. 4  Relative position of the R200 color camera to Vive
Fig. 5  Unity3D game view in Vive
Therefore, the transformation matrix between the R200
color camera and the Unity3D virtual world was:
MR200Color/Unity = MVive/Unity MR200Color/Vive
⎡ 1 0 0 0 ⎤
⎢ 0 1 0 0.015 ⎥
= Mt × Mr × Ms ⎢
⎢ 0 0 1 0.07 ⎥⎥
⎣ 0 0 0 1 ⎦
Now, the virtual objects were integrated with the real
world, and users could see the integrated scenes using Vive in
Unity3D, as shown in Fig. 5. However, at this stage, there was
still no depth information concerning the real-world images.
3.3 Step 3: from Leap Motion to Unity3D
The relative position of the Leap Motion controller and the
R200 color camera was fixed on the Vive HMD. Therefore,
the transformation matrix from Leap Motion to Unity3D
531
could be obtained by two steps: the transformation from
the Leap Motion controller to the R200 color camera and
from the R200 virtual color camera to Unity3D. In this
study, Zhang (2000)’s camera calibration method was used
to obtain the exact transformation matrix between the Leap
Motion controller and the R200 color camera. Twenty-two
pairs of images were sampled by the R200 color camera
and the Leap motion controller at the same time to obtain
the transformation matrix, as shown in Fig. 6. In Fig. 6,
the right camera was the Leap Motion controller, and the
left camera was the R200 color camera.
The result is shown as below:
⎡ 0.9996 −0.0182 −0.0227 22.3281 ⎤
⎢ 0.0158 0.9944 −0.1041 22.0347 ⎥
=⎢
0.1037 0.9943 −40.0447 ⎥⎥
MLeap/R200Color
⎢ 0.0245
⎣ 0 0 0 1 ⎦
Then, the transformation between Leap Motion and
Unity3D could be obtained as follows.
MLeap/Unity = MR200Color/Unity × MLeap/R200Color (3)
Now, the Leap Motion controller could be used to track
users’ hands in the AR environment.
3.4 Step 4: from R200 depth camera to R200 color
camera
It was necessary to correctly project the depth informa-
tion onto the corresponding color image to obtain cor-
rect occlusion effects. However, there existed a deviation
between the depth information and the corresponding
color image. RealSense SDK was used to query the intrin-
sic and extrinsic parameters of the color streams and depth
streams of the RealSense R200 RBG-D camera as follows.
⎡ 2.0604 −0.0305 −1.7019 ⎤
Mdepth/color = ⎢ −0.0023 2.0683 −0.6136 ⎥
⎢ ⎥
(2) ⎣ 0 −0.0001 1.0136 ⎦
Af ter MVive/Unity , MR200Color/Unity , MLeap/Unity , and
Mdepth/color were obtained, all the coordinate systems were
unified in Unity3D.
4 Occlusion handling
The real scene captured by the color camera was repre-
sented as a 2D image plane in Unity3D. The image plane
has no relative depth information. Without occlusion
handling, virtual objects were always rendered in front
of the real scene, as shown in Fig. 7a. Typical occlusion
approaches cut occluded images and paste the forefront
13
532
Fig. 6  Twenty-two pairs of images for obtaining the transformation matrix
image back to the scene, which were time-consuming and
difficult to achieve real-time dynamic occlusions. In this
study, a more efficient and intuitive method was proposed
by overwriting the Z-buffer in the rendering engine of
Unity3D.
Figure 8 shows a standard Unity3D rendering pipeline.
Before the frame buffer step, per-sample operations would
make a series of tests to ensure that the rendering was cor-
rect. Among these tests, the depth test was applied to check
Fig. 7  Occlusion handling
(a) Without occlusion handling
13
Virtual Reality (2020) 24:527–539
the depth order of the objects and make sure that only the
forefront objects were drawn in the scene. In this study,
before the per-sample operations step, a custom fragment
shader was used to overwrite the Z-buffer of the image plane
in the pixel processor, so that each pixel would have correct
depth information.
Whenever the RealSense R200 RGB-D camera
refreshed the accepted images, the system mapped the
depth data to the color images. Then, both the color
(b) With occlusion handling
Virtual Reality (2020) 24:527–539
Fig. 8  Unity3D rendering pipeline
information and depth information was written into the
texture of the image plane. After the fragment shader
computed the pixel values from the texture, the depths of
the real-world objects would be reproduced on the image
plane correctly. After comparing the depth information of
the virtual objects and the real objects in the per-sample
operations step, the system would render the forefront
objects and reject the farther objects.
Most of the Z-buffer resolutions are 8, 16, 24, or 32 bits.
The resolution determines how many bits of the depth value
of a pixel can be stored. The higher the resolution is, the
higher precision it represents, but more memory is needed.
In this research, a full 32-bit Z-buffer was used on the Uni-
ty3D PC platform. Figure 7b shows the occlusion result of
using the custom fragment shader. The virtual object was
correctly occluded by the real hand.
5 AR‑based training system
5.1 Milling simulation
In this study, a GPU implementation of the marching cubes
algorithm was used to accelerate the machining simulation.
The marching cubes algorithm was a voxel reconstruction
method proposed by Lorensen and Cline (1987). By search-
ing a lookup table, the method established an iso-surface
mesh for each voxel in the scalar field. The mesh of each
voxel was then connected together to reconstruct a 3D
object.
In this study, the size of the virtual workpiece was an
80 mm × 80 mm × 80 mm cube, which was represented by
80 voxels × 80 voxels × 80 voxels. Thus, the resolution of the
milling simulation is 1 mm. Since the system was for novices
533
to learn the basic manual milling machining operations at
beginner’s level, resolution of 1 mm was enough to achieve
the training purpose. The scalar value of each vertex (or each
grid) of the voxel was set to be 0 or 1, representing whether
the vertex was removed by the cutter or not. The isolevel
was set to be 0.5.
Figure 9 shows a flowchart of the milling simulation.
The process contained four steps: (1) establishing a scalar
field for the workpiece, (2) detecting the relative location of
the cutter and the workpiece, (3) using the marching cubes
algorithm to update the geometry of the workpiece, and (4)
rendering the meshes of the workpiece.
This study first assigned the scalar value of each grid
to the “R” channel of the RGBA color information using
the “Texture3D” class in Unity3D. The parallel computing
capability of the GPU was applied to calculate the relative
positions of the cutter and all grids. If a grid was within the
cutting area, the “R” value of the grid would be changed to
0. Then, the mesh of each voxel was reconstructed. Finally,
using the function “Graphics.DrawProceduralIndirect()” in
Unity3D, the system could redraw the scene directly in GPU,
without sending data back to CPU.
Figure 10 shows a snapshot of the milling simulation.
The blue lines in the figure were used to guide the machin-
ing process during training. Using the parallel processing
capability of GPU, the machining simulation could reach a
real time of 30 frames per second.
5.2 User interface
In order to assist learning, augmented instructions and ani-
mations were provided in the system. Figure 11a is a users’
first-person view from within the HTC Vive HMD. The aug-
mented translucent texts would guide the users to operate
the machine step by step. Figure 11b is a third-person view
of the system. Users could freely walk around in the room-
size AR environment to operate the full-size virtual milling
machine with their barehands using their natural operation
behaviors. Figure 11c shows an augmented instruction and
an auxiliary view to help users to understand the environ-
ment which were not in users’ view. Figure 11d shows an
interaction between users and the virtual objects. Users
could grasp a virtual object and place it to the proper loca-
tion. Objects which needed to be activated would also be
highlighted to remind the users, as shown in Fig. 11e. For
complex operations or important milling techniques, the
system also provided illustrated 3D animations to show
the operation procedures during the training, as shown in
Fig. 11f. Users could easily realize the operation procedures
and follow the instruction to operate the virtual milling
machine step by step.
13
534 Virtual Reality (2020) 24:527–539

Fig. 9  Flowchart of the milling simulation

6 User test

6.1 Training session

To verify the capability of the developed AR-based train-

Fig. 10  Real-time machining simulation
ing system, a user test was conducted to find the subjective
and objective performance. Twenty subjects, who did not
have any prior milling machine operation experience, were
randomly divided into an experiment group and a control
group. Subjects in the experiment group were trained using
the AR-based training system. They had to follow the AR
instructions and complete the specific operations in the sys-
tem. Subjects in the control group were trained using an
education video.
Both AR-based training and education video training
included a complete milling machining operation, which

13
Virtual Reality (2020) 24:527–539
Fig. 11  User interface of the AR-based training system
composed of four main steps: (1) introduction to the machine
tools, (2) introduction to the workpiece setup, (3) introduc-
tion to the cutter setup, and (4) introduction to the milling
task. Both training times were about 30 min.
Twenty-four hours after completing the training ses-
sions, the participants were asked to complete a practical
milling task. They had to complete a real milling task with
correct operation steps. After that, subjects in the experi-
ment group were asked to watch the same education video.
535
Finally, subjects in the experiment group were asked to fill
out a subjective questionnaire and a System Usability Scale
(SUS) analysis (Brooke 1996). The training flowchart is
shown in Fig. 12.
6.2 Practical milling operation session
Before operating the real milling machine, subjects needed
to wear safety equipment and to understand the safety rules.
13
536
Fig. 12  Training flowchart
During the test, whenever the subjects encountered any
problem or forgot any operation step, they could ask for help.
Each subject was given a 50 mm × 50 mm × 25 mm work-
piece and a two flute end mill cutter with 16 mm diameter. A
finished sample product and its dimensions were given to the
subjects for their reference. Subjects were asked to machine
the workpiece to the same geometry as the given sample
product. The finished product should include two adjacent
open slots. One was 16 mm wide, 2 mm deep, and 10 mm
away from the reference plane. The other one was 8 mm
wide, 1 mm deep, and 26 mm away from the reference plane,
as shown in Fig. 13. The evaluation consisted three parts:
In Part 1 (Workpiece), the participants had to correctly
fasten the workpiece on the vise of the milling machine. This
part included four steps: (1) put the baseplate on the vise,
(2) put the workpiece on the baseplate, (3) fasten the vise
to hold the workpiece, and (4) knock the workpiece using
a rubber hammer to remove the gap between the plate and
the workpiece.
In Part 2 (Cutter), the participants had to correctly install
the cutter assembly. This part included five steps: (1) assem-
ble the collet nut and the collet, (2) put the milling cutter in
the collet, (3) move the lever knob to the “IN” position to
lock the rotation of the chuck, (4) fasten the cutter assembly
on the chuck with a C spanner, and (5) move the lever knob
to the “OUT” position.
In Part 3 (Milling), the subjects had to complete the
milling task with the correct milling operations. This part
included five steps: (1) turn on the milling machine to the
13
Virtual Reality (2020) 24:527–539
Fig. 13  A sample product
clockwise rotation mode, (2) rotate the x–y–z handle to
adjust the cutter to the proper location, (3) machine the
workpiece in the clockwise direction against the feeding
direction, (4) machine the workpiece to the correct width,
and (5) machine the workpiece to the correct depth.
6.3 Milling performance
After the 20 participants completed the user test, their
performance was evaluated based on three items: failure
rate, the number of inquiries, and the time taken. The time
recorded did not include the time for inquiry. Since all par-
ticipants did not have any prior milling machining experi-
ence and this was their first time of operating a real milling
machine, the dimension accuracy was not rigorously eval-
uated. The allowances of the locations and widths of the
two slots were ± 4 mm, and the allowances of the depths of
the two slots were ± 0.5 mm. Table 1 shows the evaluation
results. The independent sample t test was used to check
whether there were significant differences in failure rate,
inquiry number, and the time taken between the control
group and the experiment group.
In Part 1 (Workpiece), there were no significant differ-
ences between the control group and the experiment group
in all three evaluation items. It might be because the steps
in Part 1 were easy and fast. In Part 2 (Cutter), there was a
significant difference in the number of inquiries. Subjects
in the video training group had more questions concerning
the setup sequence of the cutter, collet, and collet nut. In
Part 3 (Milling), the failure rate in the video training group
was significant higher. Subjects in the video training group
were likely to use the center of the cutter as the reference,
instead of the edge of the cutter. It results in incorrect mill-
ing dimensions. The number of inquiries in the video train-
ing group was also significant higher than the AR-based
training group.
Virtual Reality (2020) 24:527–539 537

Table 1  Results of the practical milling task

Task operation Failure rate (%) Inquiry number (avg.) Time taken (s)
Video AR Video AR Video AR

Part 1
Workpiece
Put the baseplate on the vise 0 0 1.1 0.7 107 85.78
Put the workpiece on the baseplate 0 0
Fasten the vise 0 10
Knock the workpiece using a rubber hammer 0 0
p value 0.339 0.213 0.163
Part 2
Cutter
Assemble the collet nut and the collet 10 10 2.4 1.1 148.2 145.1
Put the milling cutter in the collet 10 10
Move the lever knob to the “IN” position 10 10
Fasten the nut on the collet chuck with a C spanner 10 10
Move the lever knob to the “OUT” position 0 10
p value 0.646 0.023 0.563
Part 3
Milling
Switch on the milling machine 0 0 1.6 0.7 1018.8 913.3
Rotate the x–y–z handle to adjust the cutter to the proper position 0 0
Machine the workpiece in the clockwise direction against the feeding 50 20
direction
Complete the milling assignment with the correct width 40 10
Complete the milling assignment with the correct depth 50 0
p value 0.003 0.001 0.28

The results showed that AR-based training with natural
user interfaces helped users to have a deeper impression
concerning the practical operation processes. In contrast,
the video training was not interactive, so the subjects might
not have sufficient subjective experience concerning how to
operate a real milling machine.
6.4 Subjective questionnaire analysis
After completing the practical milling task, subjects in the
experiment group were asked to watch the same educa-
tion video. Then, they were asked to fill out a subjective
questionnaire and an SUS questionnaire. For the subjective
questionnaire, a 7-point Likert scale was used, 1 represent-
ing strongly disagree and 7 representing strongly agree.
The questionnaire contained 18 questions. Questions 1–8
were for the interaction mode of the AR system; questions
9–12 were for the occlusion effects; questions 13–16 were
for the instruction mode in the AR system; questions 17
and 18 were the subjective comparison between the video
training and the AR-based training. The results of the sub-
jective questionnaire are given in Table 2.
Regarding the interaction mode, most scores are higher
than 6, except questions 4, 5, and 7. These three questions
were related to the interaction between real hands and vir-
tual objects. One problem might be because although the
depth information of the real scenes could be obtained
using the RGB-D camera, the real-world images rendered
to the users were still in 2D images, not in stereoscopic
format. Therefore, it was difficult for the users to judge the
distance between the virtual objects and the real hands.
Another problem might be because the tracking error in
Leap Motion. If users moved their hands too fast or rotated
their hands to certain angles, Leap Motion failed to track
hands correctly.
Subjects generally believed that the color changing
features and the augmented instructions were helpful in
understanding the milling task. Finally, most subjects
agreed that AR-based training was more helpful and inter-
esting than video training.
6.5 System usability scale analysis results
The SUS analysis was based on the metric developed by
John Brook (Brooke 1996). The SUS metric provides a reli-
able usability evaluation for a product. The metric consists
of 10 questions. Tullis and Stetson (Tullis and Stetson 2004)
showed that using the SUS metric, a sample size of around

13
538 Virtual Reality (2020) 24:527–539

Table 2  Subjective questionnaire results

Mean SD

1 The AR system was easy to operate 6 1.20

2 The milling operations in the AR system were realistic 6.3 0.65
3 The milling operations in the AR system were accurate 6.6 0.51
4 The grabbing and moving of virtual objects in the AR system were stable and robust 5.1 1.38
5 The interactions in the AR system were real time 5.7 1.23
6 The interactions in the AR system were intuitive 6.2 0.72
7 It was easy to use the AR system to complete the milling operations 5.7 1.44
8 I would like to use this AR system again to complete similar tasks 6.5 0.67
9 The occlusion feature in the AR system was helpful 6.7 0.65
10 The occlusion feature made the AR system more realistic 6.7 0.65
11 The occlusion feature made the AR system more accurate 6.7 0.65
12 The occlusion feature made the AR system more intuitive 6.7 0.65
13 The color changing feature in the AR system was helpful 6.6 0.67
14 The augmented instructions in the AR system were helpful 6.8 0.45
15 It was helpful to learn the milling steps by using the AR-based training system 6.6 0.51
16 It was helpful to operate a real milling machine after using the AR-based training system 6.6 0.51
17 AR-based milling machining training was more helpful than the video training 6.6 0.67
18 AR-based milling machining training was more interesting than the video training 6.6 0.67

8–12 participants was enough to give a reasonably accu-
rate measure of the usability of a system. Table 3 shows the
results of the SUS analysis. The developed AR-based train-
ing system received an overall mean score of 79.6, which
was above the average mean score of 68 (Tullis and Stetson
2004). It indicated that the developed AR-based training sys-
tem was easy to use.
7 Conclusion
Manual milling machine operation training is an impor-
tant job training in vocational education. However, most
prior AR-based milling applications were for NC machine
operations, which did not need heavy manual skills.
Conventional manual milling operations require skillful
operators to produce a quality product. In this study, an
AR-based training system for conventional manual mill-
ing machine operations was developed to provide novices
a hands-on, but safe, training environment. Users can
operate a full-size virtual milling machine using their
natural operation behaviors, without any worn or hand-
held intermediaries.
The system was portable and allowed users to walk
around in a room-size environment. Dynamic occlusion was
realized using Unity3D fragment shader on GPU to increase
the realism and immersiveness of the simulation. Parallel
processing of GPU was used to implement the marching
cubes algorithm to have 30 frames per second of real-time
simulation. With the augmented instructions and illustrated

Table 3  Questionnaire results Mean SD

for SUS
I think that I would like to use this system frequently 6.0 1.54
I found the system unnecessarily complex 2.3 1.50
I thought the system was easy to use 5.7 1.56
I think that I would need the support of a technical person to be able to use this system 4.2 1.64
I found the various functions in this system were well integrated 6 0.85
I thought there was too much inconsistency in this system 2.4 1.24
I would imagine that most people would learn to use this system very quickly 5.6 1.38
I found the system very cumbersome to use 2.2 1.03
I felt very confident using the system 5.6 1.51
I needed to learn a lot of things before I could get going with this system 2.8 1.53
Total score 79.6

13
Virtual Reality (2020) 24:527–539
3D animation, users were able to learn and practice the mill-
ing operations step by step, without any fear of injury or
damage. In addition, by presenting users’ physical hands in
the vision, the sense of detachment from the AR environ-
ment was eliminated.
The user test results showed that users tended to have a
deeper impression after taking the AR-based training. The
failure rate in milling operations, the inquiry times in cutter
setup, and the inquiry times in milling operations of the AR-
based training were significant lower than the video training.
Although the occlusion handling greatly improved the
interaction effects, users still had trouble in grasping virtual
objects with their barehands. It was because the color images
of the real scenes were rendered in 2D images. Users could
not use their binocular vision to determine the relative posi-
tions of the virtual objects and the real objects. In the future,
stereo color cameras will be used to provide stereoscopic
color images of the real scenes in the AR environment. In
addition, the current system was designed to train novices to
learn the basic manual milling machine operations at begin-
ner’s level. In the future, training at professional level will
be added. The resolution of the marching cubes will also be
increased to improve the accuracy of the simulation.
Acknowledgements The authors would like to thank the Ministry of
Science and Technology, Taiwan, Republic of China for financially sup-
porting this research under Contract MOST 108-2221-E-002-161-MY2.
References
Brooke J (1996) SUS-A quick and dirty usability scale. Usability Eval
Ind 189(194):4–7
Chardonnet J-R, Fromentin G, Outeiro J (2017) Augmented reality
as an aid for the use of machine tools. In: The 15th management
and innovative technologies (MIT) conference, Sinaia, Romania
pp 1–4
Corbett-Davies S, Green R, Clark A (2012) Physically interactive tab-
letop augmented reality using the Kinect. In: Proceedings of the
27th conference on image and vision computing Dunedin, New
Zealand, pp 210–215
Gheorghe C, Rizzotti D, Tièche F, Carrino F, Khaled OA, Mugellini E
(2015) Occlusion management in augmented reality systems for
machine-tools. In: International conference on virtual, augmented
and mixed reality, Los Angeles, CA, USA, pp 438–446
Khattak S, Cowan B, Chepurna I, Hogue A (2014) A real-time recon-
structed 3D environment augmented with virtual objects rendered
with correct occlusion. In: 2014 IEEE games media entertain-
ment, Toronto, ON, Canada, pp 1–8
539
Kiswanto G, Ariansyah D (2013) Development of augmented reality
(AR) for machining simulation of 3-axis CNC milling. In: Interna-
tional conference on advanced computer science and information
systems (ICACSIS), Bali, Indonesia, pp 143–148
Leal-Meléndrez JA, Altamirano-Robles L, Gonzalez JA (2013) Occlu-
sion handling in video-based augmented reality using the Kinect
sensor for indoor registration. In: Proceedings of 18th Iberoameri-
can congress CIARP 2013 on progress in pattern recognition,
image analysis, computer vision, and applications, Havana, Cuba,
pp 447–454
Lorensen WE, Cline HE (1987) Marching cubes: a high resolution
3D surface construction algorithm. ACM SIGGRAPH Comput
Graphics Anaheim 21(4):163–169
Lu Y, Smith S (2009) GPU-based real-time occlusion in an immer-
sive augmented reality environment. J Comput Inf Sci Eng
9(2):024501
Neugebauer R, Klimant P, Wittstock V (2010) Virtual-reality-based
simulation of NC programs for milling machines. In: Proceedings
of the 20th CIRP design conference on global product develop-
ment, Ecole Centrale de Nantes, Nantes, France, pp 697–703
Penelle B, Debeir O (2014) Multi-sensor data fusion for hand tracking
using Kinect and Leap Motion. In: Proceedings of the 2014 virtual
reality international conference, Laval, France, pp 1–7
Qiu S, Fan X, Wu D, He Q, Zhou D (2013) Virtual human modeling for
interactive assembly and disassembly operation in virtual reality
environment. Int J Adv Manuf Technol 69:9–12
Regazzoni D, Rizzi C, Vitali A (2018) Virtual reality applications:
guidelines to design natural user interface. In: Proceedings of the
ASME 2018 international design engineering technical confer-
ences and computers and information in engineering conference,
Quebec, Canada, pp DETC2018-85867
Shim J, Yang Y, Kang N, Seo J, Han T-D (2016) Gesture-based inter-
active augmented reality content authoring system using HMD.
Virtual Real 20(1):57–69
Sportillo D, Avveduto G, Tecchia F, Carrozzino M (2015) Training
in VR: a preliminary study on learning assembly/disassembly
sequences. In: International conference on augmented and virtual
reality, Lecce, Italy, pp 332–343
Tullis TS, Stetson JN (2004) A comparison of questionnaires for
assessing website usability. In: Usability professional association
conference, Minneapolis, Minnesota, USA, pp 1–12
Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the
accuracy and robustness of the leap motion controller. Sensors
13(5):6380–6393
Zhang Z (2000) A flexible new technique for camera calibration. IEEE
Trans Pattern Anal Mach Intell 22(11):1330–1334
Zhang J, Ong S-K, Nee AY (2008) AR-assisted in situ machining simu-
lation: architecture and implementation. In: Proceedings of the
7th ACM SIGGRAPH international conference on virtual-reality
continuum and its applications in industry, p 26
Publisher’s Note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.
13