See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/224142319

Human-Computer Interaction Experiments Immersive Virtual Reality

Applications for the Mining Industry

Conference Paper · May 2010

DOI: 10.1109/AINA.2010.180 · Source: IEEE Xplore

CITATIONS READS

12 1,685

5 authors, including:

Tomasz Bednarz Con Caris

CSIRO Data61 / EPICentre UNSW Art & Design The Commonwealth Scientific and Industrial Research Organisation
121 PUBLICATIONS   687 CITATIONS    14 PUBLICATIONS   97 CITATIONS   

SEE PROFILE SEE PROFILE

Mark Dunn
The Commonwealth Scientific and Industrial Research Organisation
42 PUBLICATIONS   322 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Virtual Reef Diver View project

TeleRobotic Rock Breaker View project

All content following this page was uploaded by Tomasz Bednarz on 08 July 2014.

The user has requested enhancement of the downloaded file.

 Human-Computer Interaction Experiments
- Immersive Virtual Reality Applications for the Mining Industry
Tomasz P Bednarz, Con Caris, Jeremy Thompson, Chris Wesner, Mark Dunn
Exploration and Mining
CSIRO Queensland Centre for Advanced Technologies
Brisbane, Australia
tomasz.bednarz@csiro.au
Abstract—The present work concerns immersive virtual reality
(IVR) experiments carried out for prospective applications in the
mining industry. Demonstrated technologies are also suitable for
use in generic training and e-learning fields. Visualisation is
handled by the Unity3D 2.5 multiplatform game development
tool, which communicates over .Net socket servers with data
feeds from a 5DT Data Glove Ultra that measures finger flexures,
and an iPhone based touchpad device. The iPhone also provides
orientation data pertaining to acceleration and rotational
attributes, such as, pitch, roll and yaw. The user is placed at the
focal point of a 4-m dome and experiences the immersive virtual
reality environment display. Navigation and object manipulation
by the user is made possible through the combination of hard-
wired buttons, in-world selection techniques and gesture
recognition. The techniques are directly applicable to the
monitoring of mining environments, in which mining equipment
is surrounded by various sensor networks.
Immersive Virtual Reality; Teleoperation; Dome; Data Glove;
iPhone; Sensor Networks; Data Communication
I. INTRODUCTION
The potential of immersive virtual reality (IVR) technology
and human-computer interaction (HCI) techniques as a major
technical advance for supporting teleoperation scenarios has
been already well recognized by multiple industries [1-3]. One
of its unique advantages is the capability to allow operators to
visualize abstract concepts, to visit environments and to
interact with objects or events that distance, time, scale or
safety factors make difficult or impossible to access. In
addition, teleoperation scenarios offer safety advantages
compared to physical presence of people in harmful
environments. The same techniques can also be used in e-
education and for fully interactive training purposes, where all
possible scenarios can be presented through a virtual reality
world. Therefore, early investigations of possible technologies
that can be used in such scenarios are beneficial.
It is also important to realise that the same IVR can be used
for training applications. Regian, Shebilske and Monk [4]
reported on empirical research that explored the instructional
potential of immersive VR as an interface for simulation-based
training. According to these researchers, VR may hold promise
for simulation-based training as the interface preserves both the
visual-spatial characteristics of the simulated world and the
linkage between motor actions of the student and resulting
simulated effects. VR offers the possibility of presenting both
small-scale (can be viewed from a single vantage point at a
single instant) and large-scale (experienced across time) [5]
spatial information in a three-dimensional format, eliminating
the need for users to translate between different visual
modalities. Van Dam, Laidlaw and Simpson [6] have provided
in their article a snapshot of immersive virtual reality use for
scientific visualization. They have emphasized especially a
great potential of IVR for dealing with the serious problem of
exponentially growing scientific datasets. Their examples have
shown IVR to be a potent tool for humans to “see” the patterns,
trends and anomalies in data well beyond what can be done
today using conventional 3D desktop displays.
The Cell Biology Project [7] serves as an example of a
study focusing on the impact of VR defined in terms of
immersion, natural interaction (via hand trackers) and
interactivity on the effectiveness of informal education (i.e.,
self-directed, unstructured learning experiences). The
application was developed in three formats (immersive VR,
desktop VR and video-tape viewing) to compare the various
visual modalities. Statistics of the modalities highlighted that
immersive users scored higher on post-testing of symbolic and
graphic retention and were much stronger in terms of level of
engagement. However, the non-immersive groups were able to
retain more cognitive information (at least for the short term).
Pouliquen, Bernard, Marsot and Chodorge [8] have addressed
the problem of VR applications for a higher level of integration
of safety and health requirements designs. They have reviewed
industrial applications using VR and presented virtual hands,
which are coupled with a virtual press-brake by using a system
of motion capture and a force feedback device. As a result, they
were able to estimate the risk level of machine tools.
Adamo-Villiani, Carpenter and Arns [9] have presented
their development of immersive 3D learning environments
designed to increase core mathematical skills in deaf children.
Their method taught mathematical concepts and sign language
through the user's interaction with fantasy 3D signers and with
the virtual environments. Bell and Fogler [10] suggest that VR
is a good medium for presenting 3D objects and relationships
as well as for illustrating concepts that have been covered
elsewhere. However, they did not find it to be an appropriate
medium for getting across factual information nor to substitute
for traditional educational methods. Yet, when used properly, it
can augment traditional methods and lead to improved long-
term retention of material. These benefits are also directly
applicable to visually oriented students, helping them to more
fully engage with the learning process.
 II. CONFIGURATION AND OPERATION
A. Apparatus
Fig. 1 shows the logical flow chart of the existing system
for the human-computer interaction (HCI) experiments in a
virtual reality environment. The user is the first element in the
diagram interacting with the system via two input devices: a
5DT Data Glove 5 Ultra (Fifth Dimension Technologies) and
an iPhone 3G (Apple Inc), which are attached to one hand. The
data glove is connected wirelessly to a PC using the 5DT Ultra
Wireless Kit.
The data glove measures the flexure/bending of all five
digits, reporting them as integer numbers as a function of the
angle of intermediate and proximal phalanges. Prior to running
the experiments, the glove is calibrated using an algorithm that
estimates the minimum and maximum flexion and extension
for each digit sensor and then calculates the normalized
floating point numbers from 0 to 1.0 i.e., from maximum
extension to maximum flexion. The glove calibration is
subsequently used in the experiments to characterize the
finger/hand movements.
Figure 1. Logical flow chart of the system.
The iPhone’s touch screen is utilised to display dynamic
information and to handle button presses. The iPhone's built in
multi-axis accelerometer is used to measure orientation
information. Unity3D 2.5 multiplatform game development
tool is used to display the user interfaces and 3D objects and to
process the interactions transmitted from the input devices. The
final output is displayed in an immersive 3D environment on a
four metre hemispherical dome screen using two Christie HD3
professional projectors. The projectors are mounted on a
ceiling 1.5 metres apart and are located three metres from the
focal point of the screen.
B. Data and User Interface Interactions
Using the 5DT SDK (Software Development Kit) supplied
with the Data Glove, sixteen different hand gestures are easily
implemented (excluding thumb readings). Gesture number 0 is
defined as all the fingers being closed and gesture number 15
as all fingers open. The index finger indicates the least
significant bit. For example, the index finger point gesture will
indicate number 1 and the little finger point gesture will
indicate number 8. If the gesture cannot be recognized, -1
integer value is returned.
All gestures were tested in the following way: a picture of a
specific gesture is held on the screen for 1500 ms with 500 ms
transitional time between successive gestures. These tests
provide 90% of the required responses. Although better results
can be achieved when the user has more time to prepare
specific gestures, different dynamic imitations are achievable
as described in detail in [11].
For prototype testing, a simple Paper-Stone-Scissors game
was included within the virtual world. In this mode, the user
interacts with the game by hand gestures only. The user has
three seconds to prepare one of three game gestures: stone
(gesture 0), paper (gesture 15) and scissors (gesture 3). The
computer randomly picks one figure and compares it with the
user's input. The users win when they pick stone against
scissors, paper against stone or scissors against paper. If the
user and the computer picked the same figure, it is a draw, and
in the remaining cases the user loses. The game was tested with
ten participants providing 100% correct system's recognition of
the input gestures 0, 3 and 15.
The orientation data is captured using three accelerometers
from the iPhone 3G: one acceleration sensor mounted along
each of the primary axes. This combination enables the
detection of device movements in any direction and tracks the
device's orientation relative to gravity. Access to raw data takes
place through the iPhone's SDK UI Accelerometer object. For
the experiments, the reporting occurs at 80 Hz as recommended
in the iPhone Application Programming Guide for "suitability
in applications needed detection of high-frequency motion such
as hitting the device or shaking it very quickly" [12]. The raw
data needs to be filtered in order to get the current orientation
of device and detecting instantaneous motion. Therefore,
simplified low-pass and high-pass filters were also
implemented.
The user interface and user interactions are implemented
via a schema of interaction modes for our testing system as
shown in Fig. 2. The system implements two modes of
operation: (a) a view mode and (b) an interactive mode. In view
mode, the user is not interacting with VR objects and only
watches the scenes presented by the graphics engine. The view
mode switches to the interactive by one of the following
actions: if the user at anytime shakes the iPhone, touches the
mode button on the iPhone's screen, or makes the hand gesture
number 8 using the data glove. In interactive mode, the main
menu is displayed on both the dome screen and the iPhone's
touch screen.
Input selection by a variety of hardwired buttons on the
iPhone’s touch screen enables the user to have access to pre-
determined controls. These were limited to behaviours such as
camera movement and selection, object interaction and menu
based functionality. The inbuilt accelerometer allows the user
to change the orientation of selected virtual objects by physical
translation or rotation of the iPhone.
 For gesture supported commands, a small icon is displayed
next to the dynamic element. The icon demonstrates the
specific gesture associated with the command, enabling
ongoing integrated user-interface tuition.
Figure 2. User interface interactions modes.
C. Multi-sensor Communication Protocol
Fig. 3 depicts the communication flow within the system
and outlines the major components. The central element in the
system is the Data Server, which acts as an information proxy
routing data between sensors and clients. For simplicity and
universality, the data server has been implemented in Ruby
1.8.6 using GServer Class [13]. Sensors connected to the data
server are either hard-wired or using wireless connection and,
after registration, are able to transmit information by a custom
XML protocol. This was developed specifically for the
experiments and is known as the Human Systems Integration
Protocol (HSIP).
Figure 3. Multi-sensor communication protocol.
The HSIP protocol has been designed to accommodate a
base set of sensors whilst allowing for the simple addition of
new devices. Data is transmitted using a set of common XML
command strings containing a command type identifier. The
general form is <command type=”type”> data </command>.
Table I provides examples of commonly used command
strings.
TABLE I. EXAMPLE OF HSIP PROTOCOL
Example command string Sensor Description
<vector type=”ac”> Android /
Acceleration vector
x, y, z</vector> iPhone / HTC
Touch at coordinates (x,y)
<vector type=”to”> Android /
with a state s (1=single,
x,y,s</vector> iPhone / HTC
2=double etc.)
<binary type=”img”>
Webcam Binary image data
aabbcc…</binary>
Control command to
<control type=”nv”> Android /
navigate the display to
p</control> iPhone / HTC
page
The combination of command and the type identifier
defines how the data should be interpreted and processed. This
arrangement allows for the addition of new sensors by simply
defining new command type identifiers.
III. IMMERSIVE VIRTUAL REALITY SCENARIOS
The main goal of the experiment presented in this paper is
to employ the apparatus and methods described above to create
a realistic and intuitive experience within immersive virtual
reality scenarios. The system is immersive in that the 3D
scenes fill the user's horizontal and vertical peripheral vision.
This enhances situational awareness and reduces possible
distractions when compared to viewing of the same virtual
world on the monitor screen. When the field of view is only
partially covered by the content, the user's eyes can easily catch
motion changes in the background not belonging to the visual
system (such as a person passing the room). The user attaches
the data glove and iPhone to the same hand/wrist for interaction
inputs (gestures and orientation). Two different scenarios are
described: a three-dimensional (3D) mine engineering e-book
and a teleoperated surface mine.
A. Virtual Mine E-book
The first example of IVR arrangement presents a 3D mine
engineering e-book (Fig. 4) for an underground mining
equipment system called "Longwall Shearer Guidance" as
developed by the CSIRO Mining Automation team [14]. The
3D virtual book responds to the inputs provided by the user: the
user can flip pages, zoom in and out of the pictures in the e-
book and access/navigate the virtual worlds linked to the text.
This functionality makes this system unique when compared to
standard books or book viewers simply because the user can
experience the content, potentiating longer-term recall.
 content including web-pages, multimedia and other 3D VR
(a) scenes. The user can change linked content to full-screen
display by making a specific hand gesture or by touching
corresponding menu item visible on the iPhone's screen.
As mentioned previously, hyperlinks are able to load
additional 3D VR scenes, potentially expanding the users
experience by layering additional features. These include
different camera modes for 3D navigation, the ability to record
experiences (so they can be re-played later) and also ability to
pick and change object features or orientations. Fig. 4c shows a
snapshot of a linked 3D world that allows dynamic interaction
with a previously static scene. The user can return to the source
e-book by making the "stone" gesture or by touching the "exit"
button on the iPhone.
(b)
B. Surface mine teleoperated scenario
The immersive virtual reality of the surface mine is another
working example of teleoperated scenario. In this case, the
landscape of the real surface mine is reconstructed based on
Geographic Information System (GIS) data and converted to a
format suitable for Unity 3D. The mine surface equipment is
then geo-referenced using GPS and passed to a virtual reality
data server to construct a calculated virtual mine
representation. Real-time tracking of equipment can be
improved by using coupled IMU/GPU devices as presented in
[15]. Input devices, such as, the data glove and iPhone, are
used to interact with the camera viewports and objects and,
eventually, to send specific safety commands to running
equipment, where required.
The cameras’ paths and views are dynamically optimised to
(c) include the location and orientation of equipment so that picked
objects display consistently and correctly. The geo-referenced
locations of the objects are stored in a database for replaying
scenarios or for referencing purposes.
Fig. 5 shows example of reconstructed surface mine
scenario. A user is located at the front of the 4m spherical
dome with the 5DT data glove and the iPhone attached to his
right hand. Using touch screen or hand gestures, the user is able
to pick up different options and send commands to displayed
objects. In this example, the user is observing a truck moving
along the virtual mine landscape and is able to check intrinsic
information, such as, carried tonnage on the iPhone screen.

Figure 4. Virtual Mine E-book: (a) inside Virtual Mine Centre (VMC) at
CSIRO QCAT, (b) virtual e-book, demonstrating gesture icons, (c) inside the
virtual underground mine.

As showed in Fig. 2, the user can be in two modes: view

mode and interactive mode. The view mode is a safe/lock mode
i.e., the user can only read the book pages and cannot do any
unexpected actions or send random inputs to the system, for
instance, when changing pose or talking to someone using
hands to gesture an alternative concept. In the interactive
mode, the users interact directly with the e-book; they can flip
the pages using gestures displayed in the lower right-hand
corner of the screen or touch equivalent icons on the iPhone's
touch screen. Discrete elements of the e-book's pages can be
configured as live hyperlinks, enabling the user to access other Figure 5. Surface Mine Scenario.
 IV. SUMMARY
In this paper, we have demonstrated the use of IVR
applications for delivering an interactive environment for
possible training purposes in underground coal mining and for
teleoperation in surface mining environment. The first example
has facilitated an e-learning experience by providing a familiar
book object as the target for user’s interaction. Importantly, this
IVR system has the potential for providing an "enhanced"
teaching medium as humans tend to remember
events/situations better when they experience those events in
person. The IVR environment enables a user to physically
interact with various 3D objects and behaviours, which, in turn,
reinforces the idea that doing leads to efficient learning. Our
prototype system can be implemented as an efficient and cost
effective e-learning and training platform for different
engineering domains. In addition, the same technology could
be used in the teleoperation of mining equipment as presented
in the second example. CSIRO has already developed VR
systems to remotely monitor mining equipment (LASC
Longwall Automation [14], and the Nexsys risk management
system). Collaborative industry projects are currently underway
that will see CSIRO leverage IVR technology to demonstrate
that teleoperation can be a reality for everyday use. It is
important that next generation e-learning systems are
developed to provide new training media for the effective use
of advanced control systems in the future.
REFERENCES
[1] D. Weidlich, L. Cser, T. Polzin, D. Cristiano, and H. Zickner, “Virtual
reality approaches for immersive design”, CIRP Annuals –
Manufacturing Technology, vol. 56, pp. 139-142, 2007.
[2] J. Nomura, and K. Sawada, “Virtual reality technology and its industrial
applications”, Annual Reviews in Control, vol. 25, pp. 99-109, 2001.
View publication stats
[3] M. W. Bowyer, K. A. Streete, G. M. Muniz, and A. V. Liu, “Immersive
virtual environments for medical training”, Seminars in Colon and
Rectal Surgery, vol. 19, pp. 90-97, 2008.
[4] J. W. Regian, W. L. Schebilske, and J. M. Monk, "Virtual reality: An
instructional medium for visual-spatial tasks". Journal of
Communication, vol. 42, pp. 136-149, 1992.
[5] A. W. Siegal, "The externalization of cognitive maps by children and
adults: In search of way to ask better questions". In I. Liben, A.
Patterson, & N. Newcombe (Eds.), Spatial representation and behaviour
across the life span , New York: Academic Press, pp.163-189, 1981.
[6] A. van Dam, D. H. Laidlaw, and R. M. Simpson, “Experiments in
immersive virtual reality for scientific visualization”, Computers &
Graphics, vol. 26, pp. 535-555, 2002.
[7] E. Gay, “Is virtual reality a good teaching tool?”, Virtual Reality Special
Report, Winter, pp. 51-59, 1994.
[8] M. Pouliquen, A. Bernard, J. Marsot, and L. Chodorge, “Virutal hands
and virtual reality multimodal platform to design safer industrial
systems”, Computers in Industry, vol. 58, pp. 46-56, 2007.
[9] N. Adamo-Villani, E. Carpenter, and L. Arns, "3D sign language
mathematics in immersive environment", Proc. of the 15th IASTED
International Conference, Rhodes, Greece, June 2006.
[10] J. T. Bell, J. S. Fogler, "Preliminary testing of a virtual reality based
educational module for safety and hazard evaluation training", Proc.
American Society for Engineering Education Annual Conference,
Indiana Sectional Meeting, Peoria, IL, 1996.
[11] B. J. Gold, M. Pomplun, and N. J. Rice, “A new way to quantify the
fidelity of imitation: preliminary results with gesture sequences”,
Experimental Brain Research, vol. 187, pp. 139-152, 2008.
[12] Apple Inc., “iPhone Application Programming Guide”, 2009.
[13] www.ruby-doc.org/stdlib/libdoc/gserver/rdoc/classes/GServer.html
[14] LASC Longwall Automation http://www.lascautomation.com/
[15] T. P. Bednarz, C. Caris, C. Wesner, P. B. Reid, G. Falco, G. Einicke, and
J. Malos, “Experiments utilizing data glove and high-performance INS
devices in an immersive virtual mining environment”, International
Global Navigation Satellite Systems Symposium Symposium 2009,
Gold Coast, 1-3 December 2009.