A MONITOR-BASED AR SYSTEM AS A SUPPORT TOOL

FOR INDUSTRIAL MAINTENANCE

Vesna Nikolic, Peter F. Elzer, Christian Vetter

Institute for Process and Production Control Technology

Clausthal University of Technology
Julius-Albert-Str. 6
D-38678 Clausthal-Zellerfeld
Federal Republic of Germany

Abstract: A prototype of a monitor-based augmented reality (AR) system is presented in

this paper. The system is completely mobile, consisting of a motorized controllable cam-
era and a laptop with a head set. The human-system interaction is realized by user-
oriented interfaces and is speech-based. Advantages of the system are its simplicity and
low cost combined with proven usability. Compared to other present tools for presenta-
tion of instructions for maintenance and operation, the use of this system results in time
savings and lower error rates, as shown by experiments undertaken at the IPP. Copyright
© 2006 IFAC

Keywords: Augmented Reality, Monitor-based, Maintenance, Maintenance and Operating

Instructions, Usability, Cognitive Time.

1. BACKGROUND years (Feiner, et al., 1993; Mizell, 2001), but com-

In order to make work more efficient, the source of
essential information for technicians in industrial
maintenance must fulfil a large number of specific
requirements. It has to be mobile and must neither
influence safety and health of the personnel, nor the
work flow and the work environment. It has to have
an ergonomic design and be reliable. It should help
to quickly find the information needed. Economical
engineering as well as easy update of the tool are
necessary. Furthermore, it should support fast input
and/or reading of important contextual multimedia
notes of the users. Such a system has to record auto-
matically all maintenance activities. Its user-centered
design is also very important: the information pre-
sented has to be adjusted to the user’s knowledge
and/or experience. And, additionally, industry de-
mands low cost and widely applicable solutions.
The idea of using AR technology for supporting
maintenance or assembly has existed for at least 15
mercial applications are still rare. Four major prob-
lems contribute to this situation: poor ergonomics of
head mounted displays, unsatisfactory technical per-
formance of tracking systems, high hardware and
engineering costs.
Therefore, designing an AR system as a maintenance
support tool is a challenging task. The AR technol-
ogy itself is very promising (Elzer, et al., 2001), es-
pecially taking into account the results of several
studies about its efficiency (Tang, et al., 2003; Alt,
2003; Wiedenmaier, et al., 2003). Although those
studies have quite varying experimental set-ups and
address serious problems in the implementation of
head mounted displays, they show that using AR in
maintenance or assembly leads either to time savings
or reduction of error rates (thus improving accuracy),
or both, along with the reduction of the cognitive
load of its user.

902

Fig. 1. Maintenance technician using the monitor-
based AR system as a supporting tool for repair
work at a pump.
In this paper a simple and low cost monitor-based
AR system is presented. It has been developed in the
framework of a doctoral thesis at the IPP (Nikolic,
2006). Its functions and main features are described.
The experiments conducted in order to assess its ef-
fectiveness are presented. In the conclusion some
future work and other possible implementations of
this system, e.g. in education, are addressed.
2. THE MONITOR-BASED AR SYSTEM
2.1 General
Because of the abovementioned ergonomic and tech-
nical problems with such components the develop-
ment efforts at the IPP were focused on an AR sys-
tem that did not need a head mounted display or a
tracking system. The work resulted in a different AR
system, consisting of a laptop with a head set, one or
more motorized controllable cameras (depending on
the task and the environment) and the AR software
(Fig. 1).
The camera is to be mounted on a prearranged quick
release fastener in the maintenance environment. The
camera records the environment and the video pic-
ture is presented in real time on the screen of the
laptop. Corresponding to pan, tilt and zoom chosen
(and of course dependent on user and task), the AR
software augments this picture with proper textual or
graphical objects. The result is that the user can see
his augmented environment on the laptop. He can
change the camera view or zoom according to his
needs. For the system realization we used a camera
with the following parameters: pan -170o to +170o,
tilt -90o to +10o, and 16x optical zoom.
The system was named CARIHBA, as an abbrevia-
tion for the German title: Computer Augmentierte
Realität für InstandHaltungs- und BedienungsAnlei-
tungen (= “Computer Augmented Reality for Main-
tenance and Operating Instructions”).
The user-system interaction is speech-based, and is
accomplished with commercial software. CARIHBA
is completely mobile; the display is an integral part
of the laptop while the camera is small and light
enough and therefore easy to pack in the technician’s
bag together with other maintenance tools. Mounting
and starting of the system takes just a few minutes,
without any need for calibration, independently of
the user. The camera itself has to be calibrated only
once in its operational life, at the very beginning of
its use.
Even with this system the user is partly immersed
into the computer generated environment presented
on the screen (Fig. 2).
The user can observe his hands and his actions on the
screen, together with the augmentation. As tests have
shown, this contributes to user’s self-assurance,
which is very important for trainees and novices, or
even for experienced technicians in the case of com-
pletely new maintenance environments.
Fig. 2. The immersion effect caused by using moni-
tor-based AR system.
The most important advantage of the system is the
low cost of its hardware. It comprises only commer-
cially available components so that user acceptance
is given and maintenance costs of the system are very
low.
903
Finally, during the use of CARIHBA the well known The basic system function of environment augmenta-
problems of using head mounted displays in common tion contributes to easy understanding of the ele-
AR systems, like interposition (Drascic and Milgram, ments of the facility or the equipment with their spa-
1996), eyes accommodation (Rottenkolber, et al., tial relations.
2004) and attention tunnelling (Yeh and Wickens,
2000) do not occur. This also holds for the imple-
mentation problems of tracking systems, as line of
sight, accuracy or environmental depending perform-
ance (Behnke, 2005).

One problem, however, remains: a certain amount of

attention switching between the source of the infor-
mation and the equipment to be maintained - similar
to work with paper documentation.

2.2 System functionality

Fixation of the camera on the fastener allows a de-

fined mathematical description of the camera envi-
ronment. The focal point of the camera (and its car-
danic suspension) is actually the centre of a spherical
coordinate system. Every visible point (i.e. object) in
the space around the camera, which can be seen on
its video picture, has a well-defined position de-
scribed by the two angles: pan and tilt. To each ob-
ject of the reality, various virtual objects can be allo-
cated.
The third dimension of the spherical coordinate sys-
tem is cameras focal length, which corresponds to
the chosen zoom. It is defined as the distance be-
tween the video picture and the centre of the sphere.
Both the centre of the video picture and the centre of
the coordinate system describe a half-line that inter-
sects the sphere just at the actual coordinates pan and
tilt. Together with the centre of the coordinate sys-
tem all pixels of the video picture also describe half-
lines. The transformation between spherical coordi-
nates and screen coordinates (and vice versa) of vir-
tual objects can be accomplished by applying two
rotations.
Fig. 3. The overview mode interface of CARIHBA.
All available information about one particular facility
is ordered in a hierarchical structure, which is visual-
ized as a pull down menu with seven levels. The user
can select a specific piece of equipment and call for
documents and actions about it, which are stored in
the system. He can also retrieve the information
needed from the plant legacy system - if available.
Below the real-time video picture, the interface has a
special pan-tilt field, where pan and tilt camera pa-
rameters can be changed simultaneously. This field
shows the complete camera view and indicates all
existing augmentations as white dots, so that the
camera can be quickly moved to the points of inter-
est. The chosen camera position can also be changed
by using functions of “step back” or “default posi-
tion”.
The overview mode is completed with a system set-
tings panel with a few functions for customizing the
visualization of the information.

2.3 Modes of operation The scenario mode leads the user through a series of
steps, helping him to accomplish the chosen mainte-
During the system development process, special at- nance task (Fig. 4).
tention was paid to the software features concerning
its use in industrial maintenance. Three modes of
operation with proper interfaces were created, one
for each of the three main situations in maintenance:
the overview, the scenario and the engineering mode.

The overview mode helps the user to explore the en-

vironment (Fig. 3).

After starting the system and logging in, the user

selects his environment and defines the camera posi-
tion. As a result, he can see his augmented environ-
ment on the screen. He can move the camera by
means of voice control (i.e. change pan, tilt and
zoom parameters) according to his needs and inten-
tions, thus exploring his surroundings. He also gets
additional textual information about the actual view, Fig. 4. The scenario mode interface of CARIHBA.
which is placed in a separate interface field.

904
Each step of one specific scenario includes up to four
pieces of information: the augmented video picture, a
textual task description, and eventually, a warning
and/or a hint. The warnings are visualized in textual
form in the middle of the video picture, framed by a
red line. Hints are placed in a separate field of the
interface, to the right of the task text field.
Overall orientation in the scenario is supported by
means of a status line, which shows both the total
number of steps within the task as well as the number
of already executed steps. The navigation is realized
by means of the words “next” and “back”, which
cause the loading of the next or the previous step in
the task. A short flashing of the status panel, and a
short sound, serve as system feedback.
On a separate panel to the right of the video picture,
three other useful functions are placed. The first one
allows fast browsing through a task. The second one
helps the user to save a certain multimedia note for
his colleagues or to read their notes, respectively.
The third function creates a log file with date and
time stamps about the maintenance activity.
The software allows for different presentations of
augmentations and task texts for different user
groups (novices, advanced users and experts).
In the scenario mode the camera moves automati-
cally, according to predefined position data for the
steps in the scenario. This turned out to be a great
help for the user in finding the specific location
within the equipment or the facility, because he can
simply follow the movement of the camera and re-
sulting changes in the video picture, which give him
additional information for orientation.
The scenario mode has not been completed yet. Its
design is currently under work, basically following
the principle of "WYSIWYG". E.g. it shall be possi-
ble to edit augmentation texts or action diagrams via
drag and drop. The engineering costs depend directly
on the quality of such engineering tools.
Nevertheless, the most important further develop-
ment task appears to be technical support for engi-
neering of the system without physical presence on
the maintenance site, as usual by preparation of
common technical documentation.
3. EXPERIMENTS
3.1 Aim
A series of experiments was conducted in order to
answer some questions.
The first one concerned the smoothness of the main-
tenance work flow with CARIHBA as a support tool.
It was important to observe how body and head
movements change during the work with CARIHBA,
compared with situations where somebody holds
paper documentation in his hand or looks at elec-
tronic documents on a laptop.
Secondly, the effectiveness of CARIHBA was a part
of the investigation. It was measured in two ways:
firstly, as the time needed to execute several mainte-
nance tasks, and secondly as the error rate during this
execution.
Finally, the personal opinion of the participants about
the work with this particular instruction medium was
an interesting topic. Therefore, every participant was
interviewed after completion of each task and was
asked to complete three questionnaires after the en-
tire set of experiments.
3.2 Procedure
As a medium for the provision of information during
maintenance, CARIHBA differs from standard paper
documentation in a number of ways: information is
presented on a display; human-system communica-
tion is based on voice control; the system has a user-
centered interface design with feedback possibilities
and at last, the user gets additional self-assuredness
by seeing his own hands on the screen, when touch-
ing augmented components in the real environment.
In order to compare CARIHBA and paper documen-
tation, it was necessary to express and distinguish
influences of those characteristics on the efficiency
of the maintenance work.
Therefore, for the purpose of the experiments, four
variants of instructions were designed, all providing
the same quantity of presented information, but dif-
fering in the manner of its presentation and in inter-
action possibilities. The first one was called “paper”.
Each step within a maintenance task was shown on a
separate piece of paper (all numbered and bound)
while the instruction itself was provided by means of
both augmented (annotated) photo and related text.
The second was called “monitor”, differing from the
first one in the medium (a TFT monitor placed at the
side of the maintenance workplace) and in the voice-
controlled navigation through the task. The third one
additionally provided an interactive user-oriented
interface, the same one as by already described moni-
tor-based AR system. This third instruction was
called “interface”. The fourth was CARIHBA (“cari-
hba”), where instead of augmented photos (as by
other three instructions), augmented real-time video
pictures were shown.
Tasks. Two pilot plants in the laboratory for process
automation at IPP were chosen as the experimental
environment. The first pilot plant is a model of a de-
salination water treatment plant. Amongst other
equipment, it has five metering pumps, four tanks,
vaporization and heating unit and complete process
control equipment. The other one is a model of a
production line for stamping of work pieces, also full
automated.
905
A total of four maintenance tasks on those two pilot
plants were formulated. They were designed to rep-
resent the majority of common maintenance activi-
ties. Therefore the tasks of demounting, inspection,
mounting and start-up were defined. Each task in-
cluded between 17 and 22 procedural steps (a total of
77 steps in all four tasks) and took not more then 8
minutes (in average) to be accomplished.
completion between the support tools (“paper” – P,
“monitor” – M, “interface” – I, “carihba” – C). An
alpha level of 0.05 (1-tailed) was used for all statisti-
cal tests (Table 1).
Table 1. Statistical significance of cognitive time
differences for various combinations of support tools
and all maintenance tasks

Participants. Sixteen male undergraduate students of Compared support tools

technical curricula at the Clausthal University of P-M M-I I-C P-C
Technology participated in the experiments for a Task p-values
money award. No one had previous experience with Demounting n.s.1 n.s. n.s. 0.010
AR systems or knowledge or experience concerning Inspection n.s. n.s. n.s. 0.010
Mounting 0.022 n.s. n.s. 0.022
the two pilot plants, which had to be maintained.
Start-Up n.s. n.s. n.s. 0.042
Measurement of effectiveness. All experiments were
Statistically significant differences for all tasks could
recorded on video. For each step within the each
only be found for “paper” and “carihba”. Cognitive
task, the completion time was measured, but for the
times were lower with AR support up to 40%, de-
evaluation only the cognitive time portion (Towne,
pending on the task. For other combinations of sup-
1985) was taken into account. These portions were
port tools, significance in cognitive time differences
quantified after the experiments on the basis of the
for all four tasks could not be clearly identified, al-
video recordings. The term “cognitive time” was de-
though the measurements showed a certain tendency.
fined as the portion of time for the execution of one
step that one person needs to direct her attention to
It can be expected that with a larger number of sub-
the medium, read and interpret the instruction, to
jects and more tightly controlled psychological char-
form a hypothesis about what has to be done, to di-
acteristics of the participants (such as the tendency to
rect her attention to the equipment, to inspect it, to
follow own rules and not the instructions or a certain
discriminate und to select its parts, to transpose in-
superficiality), with longer training or work with ex-
formation from the medium to the equipment (Neu-
perts as participants, such trends could be proved.
mann and Majoros, 1998), to recheck the hypothesis,
to make the decision to manipulate and for all atten-
Error rates were lowest with AR support i.e. with
tion switching underway. Cognitive time does not
“carihba”. As the total error rate of all steps in all
include time for the manipulation of an object or
tasks was very low (less then 5%) no statistical
time to verify the correctness of the manipulation. It
evaluation was undertaken for error rates. From a
was postulated that a medium for provision of infor-
total of 61 errors, only 40 were taken for further
mation in maintenance can affect only the cognitive
analysis, because the rest was not related to the de-
time portion.
sign of the support tool (Table 2).
The error rate was observed during the experiments
Table 2. Number of errors in all maintenance tasks as
and checked afterwards by analysing the video re-
a function of the support tool
cords.
Support tool
Performance. All participants had a short training of P M I C
10 minutes before the experiments, in order to be- Total errors 15 11 8 6
come familiar with all four instruction forms as well
as to practice voice control. During the experiments These results correspond to the personal opinion of
voice control was simulated through a human agent. subjects about the usability of “carihba” as an in-
struction form. All participants could grade each tool
All subjects completed all four tasks with one of the after the experiments. The tools “interface” and
four support tools. One subject never worked twice “carihba” got considerably better grades than “paper”
on the same task or with the same medium. Thus the and “monitor”. On one hand, these results confirmed
learning effect was excluded. that CARIHBA is a well designed maintenance sup-
port tool. On the other hand, it has to be investigated
for which tasks, environments and personal expertise
3.3 Results the use of augmented (annotated) photos or the use of
augmented real-time video is better suited.
General. The common maintenance work can be
executed with CARIHBA as support tool. All par-
ticipants succeed in doing all assigned tasks. How-
ever, the position of the monitor and the position of
the camera have to be carefully chosen for each task.

Effectiveness. The Mann-U-Whitney test was used to

1
estimate differences in cognitive time needed for task n.s. = not significant

906
 4. FURTHER WORK
Future development of the described monitor-based
AR system will be focused on system engineering
questions. Solutions that imply high automation of
the engineering processes will be preferred.
Some investigations will also be undertaken in order
to determine a minimum quantity and quality of
augmentation information needed for different user
groups in maintenance.
At last, the IPP has a special interest to apply this AR
solution to teaching courses. A very detailed visual-
ized real-time exploration of the available pilot
plants should be of great interest for students.
and P. Korhonen (Eds)), pp. 73-80. ACM Press,
New York.
Towne, D.M. (1985). Cognitive Workload in Fault
Diagnosis. Report No. ONR-107; Contract with
Engineering Psychology Group, Office of Naval
Reasearch, Los Angeles. Behavioral Technology
Laboratories, University of Southern California.
Wiedenmaier S., O. Oehme, L. Schmidt, and H. Luc-
zak, (2003). Augmented Reality (AR) for As-
sembly Processes Design and Experimental
Evaluation. In: International Journal of Human
Computer Interaction, Vol. 16 (3), pp. 497-514.
Yeh, M. and C.D. Wickens (2000). Attention and
Trust Biases in the Design of Augmented Reality
Displays. Technical Report: Aviation Research
Lab. University of Illinois, Urbana-Champaign,
Savoy, IL.

REFERENCES

Alt, T. (2003). Augmented Reality in der Produktion.

Herbert Utz Verlag GmbH, München.
Behnke, R. (2005). Verbesserung der Positionser-
kennung bei CAR Systemen durch Kopplung von
bild- und trägheitsbasierten Verfahren. Papier-
flieger Verlag, Clausthal-Zellerfeld.
Drascic, D. and P. Milgram (1996). Perceptual Issues
in Augmented Reality. SPIE: Stereoscopic Dis-
plays and Virtual Reality Systems III, Vol. 2653,
pp. 123-134.
Elzer, P.F., R. Behnke, and A. Simon (2001). New
Techniques for the Support of Maintenance and
Training in Technical Systems. In: Proceedings
of 8th IFAC/IFIP/IFORS/IEA Symposium on
Analysis, Design, and Evaluation of Human-
Machine-Systems, Kassel, Germany (G. Johann-
sen (Ed)), pp. 517-521. VDI Verlag GmbH,
Düsseldorf.
Feiner, S., B. Macintyre, and D. Seligmann (1993).
Knowledge-based Augmented Reality. Commu-
nications of the ACM, Vol. 36 (7), pp. 53-62.
Mizell, D. (2001). Boeing’s Wire Bundle Assembly
Project. In: Fundamentals of Wearable Com-
puters and Augmented Reality (W. Barfield and
T. Caudell (Eds)), pp. 447-467. Lawrence Erl-
baum Associates Publishers, London.
Neumann, U. and A. Majoros (1998). Cognitive,
Performance, and System Issues for Augmented
Reality Applications in Manufacturing and
Maintenance. In: Proceedings of IEEE VRAIS
’98, Atlanta, USA. IEEE Computer Society.
Nikolic, V. (2006). Einsatz der Computer Augmen-
ted Reality in der Instandhaltung: eine alternati-
ve gebrauchstaugliche und kostengünstige Sys-
temlösung. (in press).
Rottenkolber, B., M. Edelmann and F. Höller (2004).
Gestaltungsempfehlungen für Head Mounted
Displays. In: Benutzerzentrierte Gestaltung von
Augmented-Reality-Systemen (H. Luczak, L.
Schmidt and F. Koller (Eds)). VDI Verlag
GmbH, Düsseldorf.
Tang, A., C. Owen, F. Biocca and W. Mou (2003).
Comparative Effectiveness of Augmented Real-
ity in Object Assembly. In: Proceedings of ACM
CHI 2003 (V. Belloti, T. Erickson, G. Cockton,

907