JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO.
4, DECEMBER 2008
Advanced Medical Displays: A Literature Review of
Augmented Reality
Tobias Sielhorst, Marco Feuerstein, and Nassir Navab
Abstract—The impressive development of medical imaging
technology during the last decades provided physicians with an
increasing amount of patient specific anatomical and functional
data. In addition, the increasing use of non-ionizing real-time
imaging, in particular ultrasound and optical imaging, during
surgical procedures created the need for design and development
of new visualization and display technology allowing physicians to
take full advantage of rich sources of heterogeneous preoperative
and intraoperative data. During 90’s, medical augmented reality
was proposed as a paradigm bringing new visualization and
interaction solutions into perspective. This paper not only reviews
the related literature but also establishes the relationship between
subsets of this body of work in medical augmented reality. It
finally discusses the remaining challenges for this young and active
multidisciplinary research community.
I. INTRODUCTION
EDICAL augmented reality takes its main motivation
M from the need of visualizing medical data and the pa-
tient within the same physical space. It goes back to the vision of
having x-ray vision, seeing through objects. This would require
real-time in-situ visualization of co-registered heterogeneous
data, and was probably the goal of many medical augmented
reality solutions proposed in literature. As early as 1938, Stein-
haus [1] suggested a method for visualizing a piece of metal
inside tissue registered to its real view even before the inven-
tion of computers. The method was based on the geometry of
the setup and the registration and augmentation was guaranteed
by construction. In 1968, Sutherland [2] suggested a tracked
head-mounted display as a novel human-computer interface en-
abling viewpoint-dependent visualization of virtual objects. His
visionary idea and first prototype were conceived at a time when
computers were commonly controlled in batch mode rather than
interactively. It was only two decades later that the advances in
computer technology allowed scientists to consider such tech-
nological ideas within a real-world application. It is interesting
to note that this also corresponds to the first implementation of a
medical augmented reality system proposed by Roberts et al. [3]
in 1986. They developed a system integrating segmented com-
puted tomography (CT) images into the optics of an operating
microscope. After an initial interactive CT-to-patient-registra-
tion, movements of the operating microscope were measured
Manuscript received January 31, 2008; revised April 11, 2008. Current ver-
sion published November 19, 2008.
T. Sielhorst and N. Navab are with Institut für Informatik I16, Technische
Universität München, München 85748, Germany (email: navab@cs.tum.edu).
M. Feuerstein is with the Department of media Science, Graduate School of
Information Science, Nagoya University, Nagoya 464–8603, Japan.
Color versions of one or more figures are available online at http://ieeexplore.
ieee.org.
Digital Object Identifier 10.1109/JDT.2008.2001575
1551-319X/$25.00 © 2008 IEEE
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
451
using an ultrasonic tracking system. Early 1990s augmented re-
ality was also considered for other applications including in-
dustrial assembly [4], paperless office [5], and machine mainte-
nance [6].
While virtual reality (VR) aimed at immersing the user en-
tirely into a computer-generated virtual world, augmented re-
ality (AR) took the opposite approach, in which virtual com-
puter generated objects were added to the real physical world
[7]. Within their so-called virtuality continuum [8], Milgram
and Kishino described AR as a mixture of virtual reality (VR)
and the real world in which the real part is more dominant
than the virtual one. Azuma described AR by its properties of
aligning virtual and real objects, and running interactively and
in real-time [9], [10].
In augmented reality inheres the philosophy that intelligence
amplification (IA) of a user has more potential than artificial
intelligence (AI) [11], because human experience and intuition
can be coupled by the computational power of computers.
II. OVERVIEW OF MEDICAL AR SYSTEMS AND TECHNOLOGIES
The first setup augmenting imaging data registered to an ob-
ject was described in 1938 by the Austrian mathematician Stein-
haus [1]. He described the geometric layout to reveal a bullet
inside a patient with a pointer that is visually overlaid on the
invisible bullet. This overlay was aligned by construction from
any point of view and its registration works without any com-
putation. However, the registration procedure is cumbersome
and it has to be repeated for each patient. The setup involves
two cathodes that emit X-rays projecting the bullet on a fluoro-
scopic screen (see Fig. 2). On the other side of the X-ray screen,
two spheres are placed symmetrically to the X-ray cathodes. A
third sphere is fixed on the crossing of the lines between the
two spheres and the two projections of the bullet on the screen.
The third sphere represents the bullet. Replacing the screen with
a semi-transparent mirror and watching the object through the
mirror, the third sphere is overlaid exactly on top of the bullet
from any point of view. This is possible because the third sphere
is at the location to which the bullet is mirrored. Therefore, the
setup yields stereoscopic depth impression. The overlay is re-
stricted to a single point and the system has to be manually cali-
brated for each augmentation with the support of an X-ray image
with two X-ray sources.
In the next decades, different technologies followed that
allow for medical augmentation of images. This section will
introduce them as seven fundamental classes, including their
specific limitations and advantages. Each subsection begins
with the definition of the respective category. Fig. 15 gives a
short overview on these technologies.
452
Fig. 1. Inventions timeline of selected imaging and AR technology [2] © 1968 IEEE.
Fig. 2. Early suggestion for overlay of imaging data by Steinhaus [1] in 1938. Computation-free calibration (left and middle) and visualization (right) of the
proposed setup.
We start with devices that allow for in-situ visualization. This
means that the view is registered to the physical space.
A. HMD Based AR System
The first head-mounted display (HMD)-based AR system
was described by Sutherland [2] in 1968 (see Fig. 3). A stereo-
scopic monochrome HMD combined real and virtual images
by means of a semi-transparent mirror. This is also referred to
as optical see-through HMD. The tracking was performed me-
chanically. Research on this display was not application driven,
but aimed at the “ultimate display” as Sutherland referred to it.
Bajura et al. [12] reported in 1992 on their video see-through
system for the augmentation of ultrasound images (see Fig. 4).
The system used a magnetic tracking system to determine the
pose of the ultrasound probe and HMD. The idea of augmenting
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO. 4, DECEMBER 2008
live video instead of optical image fusion appears counterpro-
ductive at first sight since it reduces image quality and intro-
duces latency for the real view. However, by this means the real
view can be controlled electronically resulting in the following
advantages:
1) No eye-to-display calibration is needed, only the
camera-to-tracker transformation has to be calculated,
which may remain fixed.
2) Arbitrary merging functions between virtual and real ob-
jects are possible as opposed to brightening up the real view
by virtual objects in optical overlays. Only video overlay
allows for opaque virtual objects, dark virtual objects, and
correct color representation of virtual objects.
3) By delaying the real view until the data from the tracking
system is available, the relative lag between real and virtual
SIELHORST et al.: ADVANCED MEDICAL DISPLAYS: A LITERATURE REVIEW OF AUGMENTED REALITY 453

Fig. 3. The first (optical see-through) HMD by Sutherland [2].

objects can be eliminated as described by Bajura et al. [13]

and Jacobs et al. [14].
4) For the real view, the image quality is limited by the dis-
play specifications in a similar way as it is for the rendered
objects. Since the color spectrum, brightness, resolution,
accommodation, field of view, etc. are the same for real
and virtual objects, they can be merged in a smoother way
than for optical overlays.
5) The overlay is not user dependent, since the generation of
the augmentation is already performed in the computer, as
opposed to the physical overlay of light in the eye. The re-
sulting image of an optical see-through system is in general Fig. 4. First video see-through HMD: Augmentation of ultrasound slices [12].
not known. A validation without interaction is hardly pos- © 1992 ACM.
sible with optical overlays.
In 1996, in a continuation of the work of Bajura et al. [12],
[13], State et al. [15] reported on a system with 10 frames per point of time the tracking camera and the video cameras are
second (fps) creating VGA output. This system facilitates hy- genlocked, i.e., the tracking system shutter triggers the cameras.
brid magnetic and optical tracking and offers higher accuracy Their visualization software waits for the calculated tracking
and faster performance than the previous prototypes. The speed data before an image is augmented. This way, the relative lag is
was mainly limited by the optical tracking hardware. Nowa- reduced to zero without interpolating tracking data. The system
days, optical tracking is fast enough to be used exclusively. The uses inside-out optical tracking, which means that the tracking
continued system has been evaluated in randomized phantom camera is placed on the HMD to track a reference frame rather
studies in a needle biopsy experiment [16]. Users hit the targets than the other way around (see Fig. 5). This way of tracking
significantly more accurately using AR guidance compared to allows for very low reprojection errors since the orientation of
standard guidance. the head can be computed in a numerically more stable way than
In 2000, Sauer and colleagues [17] presented a video see- by outside-in tracking using the same technology [18].
through system that allowed for a synchronized view of real Wright et al. [19] reported in 1995 on optical see-through
and virtual images in real-time, i.e., 30 fps. In order to ensure visualization for medical education. The continuation of the
that camera images and tracking data are from exactly the same system [20] augments anatomical data on a flexible knee joint

Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
454
Fig. 5. Video see-through HMD without relative lag [17] © 2000 IEEE.
phantom in order to teach dynamic spatial behavior of anatomy.
Our group [21] suggested augmentation of recorded expert mo-
tions in regard to a simulator phantom in order to teach medical
actions. The system allows for comparative visualization and
automatic quantitative comparison of two actions.
Luo and Peli [22] use head mounted display visualization
as an aid for visually impaired rather than supporting physi-
cians. They use an optical see-through system to superimpose
contour images from an attached camera. The system is meant
to help patients with tunnel vision to improve visual search
performance.
Rolland and Fuchs [23] discuss in detail advantages and
shortcomings of optical and video see-through technology.
Cakmaci and Rolland [24] provide a recent and comprehensive
overview of HMD designs.
B. Augmented Optics
Operating microscopes and operating binoculars can be aug-
mented by inserting a semi-transparent mirror into the optics.
The mirror reflects the virtual image into the optical path of the
real image. This allows for high optical quality of real images
without further eye-to-display calibration, which is one of the
major issues of optical see-through augmentation. Research on
augmented optics evolved from stereotaxy in brain surgery in
the early 1980s that brought the enabling technology together
as for instance described by Kelly [25].
The first augmented microscope was proposed by Roberts et
al. [3], [26] showing a segmented tumor slice of a computed to-
mography data set in a monocular operating microscope. This
system can be said to be the first operational medical AR system.
Its application area was interventional navigation. The accuracy
requirement for the system was defined to be 1 mm [27] in order
to be in the same range as the CT slice thickness. An average
error of 3 mm [27] was measured for reprojection of contours,
which is a remarkable result for the first system. However, the
ultrasonic tracking did not allow for real-time data acquisition.
A change in position of the operating microscope required ap-
proximately 20 s for acquiring the new position.
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO. 4, DECEMBER 2008
Fig. 6. Augmented microscope: Ordinary and augmented view [29]© 2000
IEEE.
Fig. 7. Augmented binoculars [31] ©2002 IEEE.
In 1995, Edwards et al. [28] presented their augmented
stereoscopic operating microscope for neurosurgical interven-
tions. It allowed for multicolor representation of segmented 3D
imaging data as wireframe surface models or labeled 3D points
(see Fig. 6). The interactive update rate of 1–2 Hz was limited
by the infrared tracking system. The accuracy of 2–5 mm is in
the same range as the system introduced by Friets et al. [27]. In
2000, the group reported on an enhanced version [29] with sub-
millimeter accuracy, which was evaluated in phantom studies,
as well as clinical studies, for maxillofacial surgery. The new
version also allows for calibration of different focal lengths to
support variable zoom level settings during the augmentation.
For ophthalmology, Berger et al. [30] suggest augmenting
angiographic images into a biomicroscope. The system uses no
external tracking but image-based tracking, which is possible
because the retina offers a relatively flat surface that is textured
with visible blood vessel structures. According to the authors,
the system offers an update rate of 1–5 Hz and an accuracy of 5
pixels in the digital version of the microscope image.
Birkfellner and colleagues have developed an augmented op-
erating binocular for maxillofacial surgery in 2000 [31], [32]
(see Fig. 7). It enables augmentation employing variable zoom
and focus as well as customizable eye distances [33]. As op-
posed to the operating microscopes that are mounted on a swivel
arm, an operating binocular is worn by the user.
A drawback of augmented optics in comparison with other
augmentation technology is the process of merging real and
SIELHORST et al.: ADVANCED MEDICAL DISPLAYS: A LITERATURE REVIEW OF AUGMENTED REALITY
Fig. 8. Concept of integral videography based augmentation and examples [37]
©2004 IEEE.
computed images. As virtual images can only be added and may
not entirely cover real ones, certain graphical effects cannot be
realized. The impact of possible misperception is discussed in
paragraph IV.D. Additionally, the relative lag between the vi-
sualization of real and virtual images cannot be neglected for
head-worn systems (cf. Holloway [34]).
In addition to the superior imaging quality of the real view,
a noteworthy advantage of augmented optics is a seamless in-
tegration of its technology into the surgical workflow. The aug-
mented optics can be used as usual if the augmentation is not
desired. Furthermore, the calibration or registration routine in
the operating room need not be more complicated than for a
navigation system.
C. AR Windows
The third type of devices that allows for in situ visualization is
an AR window. In 1995, Masutani et al. [35] presented a system
with a semi-transparent mirror that is placed between the user
and the object to be augmented. The virtual images are created
by an autostereoscopic screen with integral photography tech-
nology (see Fig. 8). With microlenses in front of an ordinary
screen, different images can be created for different viewing an-
gles. This reduces either the resolution or limits the effective
viewing range of the user. However, no tracking system is nec-
essary in this setup to maintain the registration after it has been
established once. The correct alignment is independent of the
point of view. Therefore, these autostereoscopic AR windows
involve no lag when the viewer is moving. The first system could
not compute the integral photography dynamically. It had to be
precomputed for a certain data set.
In 2002 Liao et al. [36], [37] proposed a medical AR window
based on integral videography that could handle dynamic
scenes. The authors realized the system for a navigation sce-
nario, in which the position of an instrument was supposed
to be visualized in the scene. Their algorithm performed the
recalculation of a changed image in less than a second.
Blackwell et al. [38] presented in 1998 an AR window using
a semi-transparent mirror for merging the real view with vir-
tual images from an ordinary monitor. This technology requires
tracked shutter glasses for the correct alignment of augmented
objects and stereo vision, but it can handle dynamic images for
navigation purposes at a high resolution and update rate.
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
455
Fig. 9. AR window that needs polarization glasses [39] ©2003 IEEE.
For in situ visualization, AR windows seem to be a perfect
match to the operating room at first sight. For ergonomic and
sterility reasons it is a good idea not to make surgeons wear
a display. There are different ways of realizing AR windows.
In detail, each one introduces a trade-off: Autostereoscopic
displays suffer from poorer image quality in comparison with
other display technologies. In principle, they offer a visualiza-
tion for multiple users. However, this feature introduces another
trade-off regarding image quality.
Display technology using shutter glasses needs cables for
trigger signals and power supply. Polarization glasses, as for in-
stance used in the system introduced by Goebbels et al. [39],
do not need cables and weigh less than an HMD, but limit the
viewing angle of the surgeon to match the polarization. Non-
autostereoscopic AR windows need to track the position of the
user’s eye in addition to the position of the patient and the AR
window. This introduces another source of error.
Wesarg et al. [40] suggest a monoscopic AR window based on
a transparent display. The design offers a compact setup, since
no mirror is used, and no special glasses are required. However,
it cannot display stereoscopic images and only one eye can see
a correct image overlay. Since no mirror is used, the foci of the
virtual and real image are at completely different distances.
All AR window designs have to take care of distracting reflec-
tions from different light sources. Last but not least, the display
must be placed between the patient and the viewer. This may
obstruct the surgeons’ working area.
We believe that an optimal in-situ visualization device could
consist of combination of an AR window and an HMD; an ex-
ample may be an HMD attached to a boom.
D. Augmented Monitors
In this section, we cluster all systems that augment video im-
ages on ordinary monitors. The point of view is defined by an
456
additional tracked video camera. In 1993, Lorensen and Kikinis
[41] published their live video augmentation of segmented MRI
data on a monitor. This initial system did not include tracking
of the video camera yet. The camera-to-image registration had
to be performed manually. The successor of this setup included
a vision-based tracking system with fiducial markers [42].
Sato et al. [43] visualize segmented 3D ultrasound images
registered to video camera images on a monitor for image guid-
ance of breast cancer surgery. Nicolau et al. [44] describe a
camera-based AR system using markers that are detected in
the camera image. The system aims at minimally invasive liver
ablation.
As an advantage of augmented monitors, users need not wear
an HMD or glasses. By definition, augmented monitors do not
however, offer in situ visualization nor stereoscopic vision.
Using them adds a tracked camera to the clinical setup.
E. Augmented Endoscopes
A separate paragraph is dedicated to endoscope augmenta-
tion although it might be considered as a special case of mon-
itor-based augmented reality or augmented imaging devices (see
Section II-F). In contrast to augmented imaging devices en-
doscopic images need a tracking system for augmentation. As
opposed to monitor-based AR the endoscopic setup already con-
tains a camera. Hence, the integration of augmented reality tech-
niques does not necessarily introduce additional hardware into
the workflow of navigated interventions. A lot of investigative
work has been carried out that dealt specifically with endoscopic
augmentation.
The first usage of endoscopes as telescopic instruments uti-
lizing a light source dates back to the 19th century. Endoscopy
was mainly dedicated to diagnosis until the invention of video-
based systems in the 1980s. Video endoscopy permits different
team members to see the endoscopic view simultaneously. With
this approach, it is possible for an assistant to position the en-
doscope while the operating surgeon can use both hands for the
procedure. This feature opened the field of endoscopic surgeries.
The removal of the gallbladder was one of the first laparoscopic
surgeries. This operation also became a standard minimally in-
vasive procedure. Since then, endoscopy has been successfully
introduced into other surgical disciplines. Comprehensive liter-
ature reviews on the history of endoscopy can be found, for in-
stance, in [45], [46], and [47].
Although endoscopic augmentation seems to be a straight-
forward step it has been realized as recently as the end of the
1990s by Freysinger et al. [48] for ear, nose and throat (ENT)
surgery and Shahidi and colleagues [49] for brain surgery. Fig.
10 shows a visualization of the latter system including a tar-
geting help in the endoscope image. Scholz et al. presented
a navigation system for neurosurgery based on processed im-
ages [50]. Shahidi and Scholz use infrared tracking technology
and a rigid endoscope while Freysinger’s system uses magnetic
tracking.
Mourgues et al. [51] describe endoscope augmentation in a
robotic surgery system. The tracking is done implicitly by the
robot since the endoscope is moved by the robot’s arm. There-
fore no additional tracking system is necessary.
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO. 4, DECEMBER 2008
Fig. 10. Augmentation in an endoscope [55] ©2002 IEEE.
For endoscopic augmentation, the issues of calibration,
tracking, and visualization are partly different than for other
types of AR devices:
1) Calibration and Undistortion of Wide Angle Optics:
Because of their wide-angle optics, endoscopes suffer from a
noticeable image distortion. If a perfect, distortion-free pinhole
camera model is assumed for superimposition, a particular
source of error in the augmented image will be introduced [52].
This issue can be neglected in other AR systems with telephoto
optics. Common types of distortion are radial distortion (also
referred to as barrel distortion) and tangential distortion. Either
the endoscope image has to be undistorted or the rendered
overlay has to be distorted to achieve a perfect superimposi-
tion. While first approaches [53] required several minutes to
undistort a single endoscope image, this process can now be
completed in real-time: De Buck et al. [54] undistort sample
points in the image and map a texture of the endoscope image
on the resulting tiles; Shahidi et al. [55] precompute a look-up
table (LUT) for each pixel for real-time undistortion.
In order to model the geometry of an endoscope camera,
the intrinsic camera parameters focal length and principal point
need to be determined. This can be achieved using well-estab-
lished camera calibration techniques [56]–[58]. Most systems
assume the focal length of an endoscope camera to be kept
constant, although many endoscopes incorporate zoom lenses
to change it intraoperatively, invalidating a certain calibration.
Stoyanov et al. suggest to automatically adjust the calibration
for intraoperative changes of the focal length of a stereoscopic
camera [59]. Even though models for the calibration of mono-
scopic cameras with zoom lenses exist [60], they are not easily
applicable to endoscopes. These models require the (preferably
automatic) determination of the physical ranges for the lens set-
tings e.g., in terms of motor units. However, the zoom settings
of endoscopes are usually adjusted manually, rather than by a
precise motor.
To obtain a rigid transformation from the camera coordinate
frame to the coordinate frame of an attached tracking body or
sensor, most authors employ hand-eye calibration techniques
[51], [61]–[64]. An alternative approach makes use of a tracked
SIELHORST et al.: ADVANCED MEDICAL DISPLAYS: A LITERATURE REVIEW OF AUGMENTED REALITY 457

Fig. 11. Context sensing by texturing segmented model. [75] ©2002 IEEE.

calibration pattern, whose physical coordinates are known with Fig. 12. Camera-augmented c-arm (CamC) [78]. © 1999 IEEE. (a) Principle
respect to the tracker [54], [55], [65]. of CamC (CamC), (b) Camera image, (c) fused image, (d) Fluoroscopic X-ray
image.
In certain applications oblique-viewing endoscopes are used,
for which the viewing directions are changeable by rotating the
scope cylinder. Yamaguchi et al. [66] and De Buck et al. [67] [75] project endoscope images on segmented surfaces for pro-
developed calibration procedures for such endoscopes. viding context and creating endoscopic panorama images (see
2) Tracking of Flexible Endoscopes: Non-rigid endoscopes Fig. 11). Kawamata et al. [76] visualize the anatomical con-
cannot be tracked by optical tracking systems. Bricault et al. text by drawing virtual objects in a larger area of the screen
[68] describe the registration of bronchoscopy and virtual bron- than endoscope images are available. Ellsmere and colleagues
choscopy images using only geometric knowledge and image [77] suggest augmenting laparoscopic ultrasound images into
processing. The algorithms employed did not have real-time CT slices and using segmented CT data for improved context
capability, however, they proved to be stable when used on sensing.
recorded videos. In contrast to Bricault’s shape from shading
approach, Mori et al. [69] use epipolar geometry for image F. Augmented Medical Imaging Devices
processing. In order to improve the performance of their regis-
Augmented imaging devices can be defined as imaging de-
tration algorithm they suggest the addition of electromagnetic
vices that allow for an augmentation of their images without a
tracking of the bronchoscope [70]. To achieve a fusion of the
tracking system. The alignment is guaranteed by their geometry.
bronchoscopic video with a target path, Wegner et al. restrict
A construction for the overlay of fluoroscopic images on the
electromagnetic tracking data onto positions inside a previously
scene has been proposed by Navab et al. [78] in 1999 (see
segmented bronchial tree [71]. Some groups, for instance Klein
Fig. 12). An ordinary mirror is inserted into the X-ray path
et al. [72], use electromagnetic tracking exclusively.
of a mobile C-arm1. By this means it is possible to place a
3) Endoscopy Related Visualization Issues: The augmenta-
video camera that records light following the same path as the
tion of endoscopic data does not only entail fusion with other
X-rays. Thus it is possible to register both images by estimating
imaging data. Konen et al. [50] suggest several image-based
the homography between them without spatial knowledge of
methods with a tracked endoscope to overcome typical lim-
the objects in the image. The correct camera position is deter-
itations, such as replay of former images in case of loss of
mined once during the construction of the system. For image
sight, image mosaicing, landmark tracking, and recalibration
fusion, one image can be transformed electronically to match
with anatomical landmarks. Krueger et al. [73] evaluate endo-
the other using the estimated homography. The system provides
scopic distortion correction, color normalization, and temporal
augmented images without continuous X-ray exposure for both
filtering for clinical use.
patient and physician. The overlay is correct until the patient
One of the reasons for augmenting endoscope images is to
moves relative to the fluoroscope. A new X-ray image has to be
provide the anatomical context since the point of view and the
taken in such a case.
horizon are changing. Recovering each of these issues requires
Tomographic reflection is a subgroup of augmented imaging
a heightened level of concentration from surgeons since their
devices. In 2000, Masamune and colleagues [79], [80] proposed
field of view is very limited and the operating surgeon gen-
an image overlay system that displays CT slices in-situ. A semi-
erally does not move the endoscope personally. Fuchs et al.
transparent mirror allows for a direct view on the patient as well
[74] suggest provision of anatomical context by visualizing la-
as a view on the aligned CT slice (see Fig. 13). The viewer may
paroscopic images in situ with a head-mounted display. The
move freely while the CT slice remains registered without any
necessary three-dimensional model of the surface as seen by
the laparoscope is created with a pattern projector. Dey et al. 1C-arm: Medically widespread X-ray imaging device with a C-shaped gantry

Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
458
Fig. 13. CT reflection [79]: Concept and prototypical setup. © 2005 IEEE.
tracking. The overlaid image is generated by a screen that is
placed on top of the imaging plane of the scanner. The semi-
transparent mirror is placed in the plane that halves the angle
between the slice and the screen. The resulting overlay is correct
from any point of view up to a similarity transform that has to
be calibrated during the construction of the system. The system
is restricted to a single slice per position of the patient. For any
different slice, the patient has to be moved on the bed. Fischer
et al. [81] have extended this principle to magnetic resonance
imaging.
A similar principle has been applied to create an augmented
ultrasound echography device. Stetten et al. [82], [83] proposed
in 2000 the overlay of ultrasound images on the patient with a
semi-transparent mirror and a little screen that is attached to the
ultrasound probe (see Fig. 14). The mirror is placed on the plane
that halves the angle between the screen and the B-scan plane
of ultrasonic measurements. Similarly to the reflection of CT or
MRI slices, it allows for in situ visualization without tracking. In
addition to real-time images, it allows for arbitrary slice views,
as the ultrasound probe can be freely moved.
G. Projections on the Patient
Lastly, we present systems augmenting data directly onto the
patient. The advantage of these systems is that the images are
generally visible in situ without looking through an additional
device such as glasses, HMD, microscope, loupes, etc. As an-
other beneficial feature, the user need not be tracked if visual-
ization is meant to be on the skin rather than beneath. This also
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO. 4, DECEMBER 2008
Fig. 14. Ultrasound augmentation by tomographic reflection: Sonic Flashlight
[82], [83] ©2000 IEEE.
means that such a visualization can be used for multiple users.
The simplicity of the system introduces certain limitations as a
compromise, though.
Glossop et al. [84] suggested in 2003 a laser projector that
moves a laser beam into arbitrary directions by means of con-
trolled mirrors. Trajectories of the laser appear as lines due to
the persistence of vision effect. The images are limited to a cer-
tain number of bright monochrome lines or dots and non-raster
images. The system also includes an infrared laser for interac-
tive patient digitization.
Sasama et al. [85] use two lasers for mere guidance. Each of
these lasers creates a plane by means of a moving mirror system.
The intersection of both planes is used to guide laparoscopic in-
struments in two ways. The intersecting lines or the laser on the
patient mark the spot of interest, for instance an incision point.
The laser planes can also be used for determining an orientation
in space. The system manipulates the two laser planes in such a
way that their intersecting line defines the desired orientation. If
both lasers are projected in parallel to the instrument, the latter
has the correct orientation. The system can only guide instru-
ments to points and lines in space but it cannot show contours
or more complex structures.
SIELHORST et al.: ADVANCED MEDICAL DISPLAYS: A LITERATURE REVIEW OF AUGMENTED REALITY 459

Fig. 15. Simplified relationship between technology and potential benefits. Grey color indicates in situ visualization.

III. POTENTIAL BENEFITS OF AR VISUALIZATION

The crucial question regarding new visualization paradigms
is “What can it do for us that established technology cannot?”.
AR provides an intuitive human computer interface. Since in-
tuition is difficult to measure for an evaluation we subdivide
the differences between AR and ordinary display technology
in this section into four phenomena: Image fusion, 3D interac-
tion, 3D visualization, and hand-eye coordination. Fig. 15 de-
picts a simplified relationship between these phenomena and
AR technology.

A. Extra Value From Image Fusion

Fusing registered images into the same display offers the best
of two modalities in the same view.
An extra value provided by this approach may be a better
understanding of the image by visualizing anatomical context
that has not been obvious before. This is the case for endoscopic
camera and ultrasound images, where each image corresponds
only to a small area. (See paragraph II.E.3)
Another example for additional value is displaying two phys-
ical properties in the same image that can only be seen in either
of the modalities. An example is the overlay of beta probe ac-
tivity. Wendler et al. [86] support doctors by augmenting pre-
viously measured activity emitted by radioactive tracers on the
live video of a laparoscopic camera. By this means, physicians
can directly relate the functional tissue information to the real
view showing the anatomy and instrument position.
A further advantage concerns the surgical workflow. Cur-
rently, each imaging device introduces another display into the
operating room [see Fig. 16(b)] thus the staff spends valuable
time on finding a useful arrangement of the displays. A single Fig. 16. Current minimally invasive spine surgery setup at “‘Chirurgische
Klinik’”, hospital of Ludwigs-Maximilian Universität München. (a) Action
display integrating all data could solve this issue. Each imaging takes place on a very different position than the endoscope display. (b) Each
device also introduces its own interaction hardware and graph- imaging device introduces another display.
ical user interface. A unified system could replace the inefficient
multitude of interaction systems.
the viewport on virtual objects. Changing the eye position rela-
tive to an object is a natural approach for 3D inspection.
B. Implicit 3D Interaction
3D user interfaces reveal their power only in tasks that cannot
Interaction with 3D data is a cumbersome task with 2D dis- be easily reduced to two dimensions, because 2D user interfaces
plays and 2D interfaces (cf. Bowman [87]). Currently, there is no benefit from simplification by dimension reduction and the fact
best practice for three-dimensional user interfaces as opposed that they are widespread. Recent work by Traub et al. [88] sug-
to 2D interfaces using the WIMP paradigm (windows, icons, gests that navigated implant screw placement is a task that can
menus, and pointing). benefit from 3D user interaction, as surgeons were able to per-
AR technology facilitates implicit viewpoint generation by form drilling experiments faster with in situ visualization com-
matching the viewport of the eye/endoscope on real objects to pared to a navigation system with a classic display. Although

Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
460
the performance speed is probably not a valid metric for a med-
ical drilling task, the experiments indicate that the surgeons had
a faster mental access to the spatial situation.
C. 3D Visualization
Many augmented reality systems allow for stereoscopic data
representation. Stereo disparity and motion parallax due to
viewpoint changes (see Section III-B) can give a strong spatial
impression of structures.
In digital subtraction angiography, stereoscopic displays can
help doctors to analyze the complex vessel structures [89]. Cal-
vano et al. [90] report on positive effects of the stereoscopic
view provided by a stereo endoscope for in-utero surgery. The
enhanced spatial perception may be also useful in other fields.
D. Improved Hand-Eye Coordination
A differing position and orientation between image acquisi-
tion and visualization may interfere with the hand-eye coordina-
tion of the operating person. This is a typical situation in mini-
mally invasive surgery [see Fig. 16(a)]. Hanna et al. [91] showed
that the position of an endoscope display has a significant im-
pact on the performance of a surgeon during a knotting task.
Their experiments suggest the best positions of the display to
be in front of the operator at the level of his or her hands.
Using in situ visualization, there is no offset between working
space and visualization. No mental transformation is necessary
to convert the viewed objects to the hand coordinates.
IV. CURRENT ISSUES
We have presented different systems in this paper with their
history and implications. In this section we present current lim-
iting factors for most of the presented types and approaches to
solve them.
A. Registration, Tracking, and Calibration
The process of registration is the process of relating two or
more data sets to each other in order to match their content. For
augmented reality the registration of real and virtual objects is
a central piece of the technology. Maintz and Viergever [92]
give a general review about medical image registration and its
subclassification.
In the AR community the term tracking refers to the pose esti-
mation of objects in real time. The registration can be computed
using tracking data after an initial calibration step that provides
the registration for a certain pose. This is only true if the ob-
ject moves but does not change. Calibration of a system can be
performed by computing the registration using known data sets,
e.g., measurements of a calibration object. Tuceryan et al. [93]
describe different calibration procedures that are necessary for
video augmentation of tracked objects. These include image dis-
tortion determination, camera calibration, and object-to-fiducial
calibration.
Tracking technology is one of the bottlenecks for augmented
reality in general [10]. As an exception, this is quite different
for medical augmented reality. In medical AR, the working
volume and hence the augmented space is indoors, predefined,
and small. Therefore, the environment, i.e., the operating room,
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO. 4, DECEMBER 2008
can be prepared for the augmented reality system. Optical (in-
frared) tracking systems are already in use in modern operating
rooms for intraoperative navigation. In orthopedics, trauma
surgery, and neurosurgery, which only require a rigid body reg-
istration, available navigation systems proved to be sufficiently
accurate. King et al. [29] proved in clinical studies to have
overall errors in the submillimeter range for their microscope
based augmented reality system for neurosurgery. For the pose
determination of the real view infrared tracking is currently the
best choice. Only augmented flexible endoscopes have to use
different ways of tracking [see Section II-E2)].
As the last piece in the alignment chain of real and virtual
there is the patient registration. The transformation between
image data and patient data in the tracking coordinate system
has to be computed. Two possibilities may apply:
1) Rigid Patient Registration: Registration algorithms are
well discussed in the community. Their integration into the sur-
gical workflow requires mostly a trade off between simplicity,
accuracy, and invasiveness.
Registration of patient data with the AR system can be
performed with fiducials that are fixed on the skin or implanted
[94]. These fiducials must be touched with a tracked pointer
for the registration process. Alternatively, the fiducials can be
segmented in the images of a tracked endoscope rather than
touching them with a tracked pointer for usability reasons.
Whereas Stefansic et al. propose the direct linear transform
(DLT) to map the 3D locations of fiducials into their corre-
sponding 2D endoscope images [95], Feuerstein et al. suggest a
triangulation of automatically segmented fiducials from several
views [96], [97]. Baumhauer et al. study different methods for
endoscope pose estimation based on navigation aids stuck onto
the prostate and propose to augment 3D transrectal ultrasound
data on the camera images [98]. Using this method, no external
tracking system is needed.
Especially for maxillofacial surgery, fiducials can be inte-
grated in a reproducibly fixed geometry [29]. For spine surgery,
Thoranaghatte et al. try to attach an optical fiducial to the ver-
tebrae and use the endoscope to track it in situ [99].
Point-based registration is known to be a reliable solution in
principle. However, the accuracy of a fiducial-based registration
varies on the number of fiducials and quality of measurement of
each fiducial, but also on the spatial arrangement of the fidu-
cials [100].
Another approach is to track the imaging device and register
the data to it. This procedure has the advantage that no fiducials
have to be added to the patient while preserving high accuracy.
Grzeszczuk et al. [101] and Murphy [102] use a fluoroscope to
acquire intraoperative X-ray images and register them to dig-
itally reconstructed radiographs (DRR) created from preopera-
tive CT. This 2D–3D image registration procedure, which could
also be used in principle for an AR system, has the advantage
that no fiducials have to be added to the patient while keeping
high accuracy. By also tracking the C-arm, its subsequent mo-
tions can be updated in the registered CT data set.
Feuerstein et al. [97], [103] augment 3D images of an intra-
operative flat panel C-arm into a laparoscope. This approach is
sometimes also called registration-free [104], because doctors
need not perform a registration procedure. As a drawback such
SIELHORST et al.: ADVANCED MEDICAL DISPLAYS: A LITERATURE REVIEW OF AUGMENTED REALITY
an intrinsic registration is only valid as long as the patient does
not move after imaging.
Grimson et al. [42] follow a completely different approach by
matching surface data of a laser range scanner to CT data of the
head. For sinus surgery, Burschka et al. propose to reconstruct
3D structures using a non-tracked monocular endoscopic camera
and register them to a preoperative CT data set [105]. For spine
surgery, Wengert et al. describe a system that uses a tracked en-
doscope to achieve the photogrammetric reconstruction of the
surgical scene and its registration to preoperative data [106].
2) Deformable Tissue: The approaches mentioned above
model the registration of a rigid transformation. This is useful
for the visualization before an intervention and for a visualiza-
tion of not deformed objects. The implicit assumption of a rigid
structure is correct for bones and tissue exposed to the same
forces during registration and imaging, but not for soft tissue
deformed by, e.g., respiration or heart beat.
A well known example breaking this assumption is the brain
shift in open brain surgery. Maurer et al. [107] show clearly that
the deformation of the brain after opening the skull may result
in misalignment of several millimeters.
There are three possible directions to handle deformable
anatomy.
1) Use very recent imaging data for a visualization that in-
cludes the deformation. Several groups use ultrasound im-
ages that are directly overlayed onto the endoscopic view
[108]–[110].
2) Use very recent data to update a deformable model of the
preoperative data. For instance Azar et al. [111] model and
predict mechanical deformations of the breast.
3) Make sure that the same forces are exposed to the tissue.
Active breathing control is an example for compensating
deformations due to respiration [112], [113].
Baumhauer et al. [114] give a recent review on the perspec-
tives and limitations in soft tissue surgery, particularly focusing
on navigation and AR in endoscopy.
B. Time Synchronization
Time synchronization of tracking data and video images is an
important issue for an augmented endoscope system. In the un-
synchronized case, data from different points of time would be
visualized. Holloway et al. [34] investigated the source of errors
for augmented reality systems. The errors of time mismatch can
raise to be the highest error sources when the camera is moving.
To overcome this problem, Jacobs et al. [14] suggest methods to
visualize data from multiple input streams with different laten-
cies from only the same point of time. Sauer et al. [17] describe an
augmented reality system that synchronizes tracking and video
data by hardware triggering. Their software waits for the slowest
component before the visualization is updated. For endoscopic
surgery, Vogt [115] also uses hardware triggering to synchronize
tracking and video data by connecting the S-Video signal (PAL,
50 Hz) of the endoscope system to the synchronization card of
the tracking system, which can also be run at 50 Hz.
If virtual and real images do not show a relative lag it means
that the images are consistent and there is no error due to a time
shift. However there is still the visual offset to haptic senses. Ware
et al. [116] investigated the effect of latency in a virtual environ-
ment with a grasping experiment. They conclude that depending
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
461
on the size of the object the latency should be ideally as little as 50
ms. Little additions in latencies may cause big decreases of per-
formance. According to their experiment and theory a latency of
175 ms can result in 1200 ms slower grasping than for immediate
feedback. This depends on the difficulty of the task. Their exper-
iments showed also a significantly larger percentage of error in
their performance with 180 ms latency than with a system that had
only 80 ms latency. The experiments of our group [117] confirm
these findings for a medical AR system. Therefore an optimal
system should feature data synchronization and short latency. We
proposed recently an easy and accurate way of measuring the la-
tency in a video see-through system [118].
C. Error Estimation
Tracking in medical AR is mostly fiducial-based because it
can guarantee a predictable quality of tracking, which is neces-
sary for the approval of a navigation system.
For an estimation of the overall error calibration, registration
and tracking errors have to be computed, propagated, and ac-
cumulated. Nicolau and colleagues [44] propose a registration
with error prediction for endoscopic augmentation. Fitzpatrick
et al. [100] compute tracking based errors based on the spatial
distribution of marker sets. Hoff et al. [18] predict the error for
an HMD based navigation system.
An online error estimation is a desirable feature, since physi-
cians have to rely on the visualized data. In current clinical prac-
tice, navigation systems stop their visualization in case a factor
that decreases the accuracy is known to the system. Instead of
stopping the whole system it would be useful to estimate the re-
maining accuracy and visualize it, so that a surgeon can decide
in critical moments, whether to carefully use the data or not.
MacIntyre et al. [119] suggest in a non-medical setup to pre-
dict the error empirically by an adaptive estimator. Our group
[120] suggests a way of dynamically estimating the accuracy
for optical tracking modeling the physical situation. By inte-
grating the visibility of markers for each camera into the model,
a multiple camera setup for reducing the line of sight problem
is possible, which ensures a desired level of accuracy. Nafis et
al. [121] investigate the dynamic accuracy of electromagnetic
(EM) tracking. The magnetic field in the tracking volume can be
influenced by metallic objects, which can change the measure-
ments significantly. Also the distance of the probe to the field
generator and its speed have a strong influence on the accuracy.
Finally it is not enough to estimate the error, but the whole
system has to be validated (cf. Jannin et al. [122]). Standard-
ized validation procedures have not been used to validate the
described systems in order to make the results comparable. The
validation of the overall accuracy of an AR system must include
the perception of the visualization. In the next section we dis-
cuss the effect of misperception in spite of mathematically cor-
rect positions in visualizations.
D. Visualization and Depth Perception
The issue of wrong depth perception has been discussed as
early as 1992 when Bajura and colleagues [12] described their
system. When merging real and virtual images the relative posi-
tion in depth may not be perceived correctly although all posi-
tions are computed correctly. When creating their first setup also
462
Edwards et al. [28] realized that “Experimentation with intra-op-
erative graphics will be a major part of the continuation of the
project”. Drascic and Milgram [123] provide an overview of per-
ceptual issues in augmented reality system. While many prob-
lems of early systems have already been addressed, the issue of
a correct depth visualization remains unsolved. Depth cues are
physical facts that the human visual system can use in order to re-
fine the spatial model of the environment. These include visual
stimuli such as shading but also muscular stimuli such as accom-
modation and convergence. Psychologists distinguish between a
number of different depth cues. Cutting and Vishton review and
summarize psychologists’ research on nine of the most relevant
depth cues [124] revealing the relevance of different depth cues
in comparison to each other. They identify interposition as the
most important depth cue even though it is only an ordinary qual-
ifier. This means that it can only reveal the order but not a rel-
ative or absolute distance. Stereo disparity and motion parallax
are the next strongest depth cues in the personal space of up to
two meters distance in the named order. The visual system calcu-
lates the spatial information together with the depth cues of rela-
tive size/density, accommodation, conversion, and areal perspec-
tive. Especially the latest one is hardly taken into account for the
space under 2 meters unless the subject is in fog or under water.
It is the very nature of AR to provide a view that does not rep-
resent the present physical conditions while the visual system
expects natural behavior of its environment for correct depth
perception. What happens if conflicting depth cues are present?
The visual system weights the estimates according to its impor-
tance and personal experience [124].
Conflicting cues could result into misperception, adaption,
and motion sickness.
1) Misperception: If there are conflicting depth cues it means
that at least one depth cue is wrong. Since the visual system
is weighting the depth cues together the overall estimate will
generally not be correct even though the computer generates
geometrically correct images.
Especially optical augmentation provides different parame-
ters for real and virtual images resulting in possibly incompat-
ible depth cues. The effect is described as ghost-like visualiza-
tion resembling to its unreal and confusing spatial relationship
to the real world. The visual system is quite sensitive to relative
differences.
Current AR systems handle depth cues well that are based
on geometry, as for instance relative size, motion parallax, and
stereo disparity. Incorrect visualization of interposition between
real and virtual objects has already been identified to be a serious
issue by Bajura et al. [12]. It has been discussed in more detail
by Johnson [125] for augmentation in operating microscopes,
Furmanski et al. [126] and Livingston et al. [127] for an op-
tical see-through, and by our group [117] for video see-through
HMDs. The type of AR display makes a difference since rel-
ative brightness plays a role in depth perception and optical
see-through technology can only overlay brighter virtual images
on the real background. Opaque superimposition of virtual ob-
jects that are inside a real one is not recommended. Alternatives
can be a transparent overlay, wireframes, and a virtual window.
Each possibility imposes a trade off: Transparent overlay re-
duces the contrast of the virtual image, the wireframe is not
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO. 4, DECEMBER 2008
suitable for complex spatial geometry, and the virtual window
locally covers the real view. Lerotic et al. [128] suggest to super-
impose contours of the real view on the virtual image for better
depth perception.
2) Adaption: A wrong visual depth perception can be cor-
rected by learning if another sense can disambiguate the spatial
constellation. The sense of proprioception provides exact infor-
mation about the position of the human body. Biocca and Rolland
[129] set up an experiment where the point of view of each subject
was repositioned with a video see-through HMD. The adaption
time for hand-eye coordination is relatively short and the adap-
tion is successful. Unfortunately, another adaption process is
started when the subject is exposed to the normal view again.
3) Motion Sickness: In the worst case conflicting visual cues
can cause reduced concentration, headache, nausea etc. These
effects have been discussed in the virtual reality and psychology
community [130].
Modern theories state that the sickness is not caused by the
conflict of cues, but the absence of better information to keep
the body upright [131]. Therefore engineers should concentrate
on providing more information to the sense of balance (e.g., by
making the user sit, unobstructed peripheral view) rather than
reducing conflicting visual cues in order to avoid motion sick-
ness. However, motion sickness does not seem to play a big role
in AR: In a video see-through HMD based experiment with 20
surgeons we [117] found no indication of the above symptoms
even after an average performance time of 16 minutes. In the
experiment the subjects were asked to perform a pointing task
while standing. The overall lag was reported to be 100 ms for
fast rendering visualizations. The remote field of view was not
covered. For less immersive AR systems than this one based on
an HMD and for systems with similar properties motion sick-
ness is therefore expected to be unlikely.
E. Visualization and Data Representation
3D voxel data cannot be displayed directly with an opaque
value for each voxel as for 2D bitmaps. There are three major
ways of 3D data representation.
1) Slice Rendering: Slice rendering is the simplest way of
rendering. Only a slice of the whole volume is taken for visual-
ization. Radiologists commonly examine CT or MRI data rep-
resented by three orthogonal slices intersecting a certain point.
The main advantage of this visualization technique is the preva-
lence of this method in medicine and its simplicity. Another ad-
vantage that should not be underestimated is the fact that the
visualization defines a plane. Since one degree of freedom is
fixed, distances in this plane can be perceived easily. Traub et
al. [88] showed that slice representations as used in first gen-
eration navigation systems have superior capabilities in repre-
senting the precise position of a target point. They also found
that for finding a target point it can be more efficient to take a
different representation of data.
When two or three points of interest and their spatial relation-
ship are supposed to be displayed an oblique slice can be useful.
Without a tracked instrument however it is cumbersome to po-
sition such a plane.
The major drawback of slice rendering is that this visualiza-
tion does not show any data off the plane. This is not a constraint
SIELHORST et al.: ADVANCED MEDICAL DISPLAYS: A LITERATURE REVIEW OF AUGMENTED REALITY
for measuring visualizations in plane à la How far can I go with
a drill? but optimizing questions like In which direction would
a drill be furthest from critical tissue?
2) Surface Rendering: Surface rendering shows transitions
between structures.
Often these transitions are segmented and converted to
polygons. The desired tissue is segmented either manually,
semi-automatically, or automatically depending on the image
source and the desired tissue. The surface polygons of a
segmented volume can be calculated by the marching cubes
algorithm [132]. Graphic cards offer hardware support for this
vertex based 3D data. This includes light effects based on the
normals of the surfaces with only little extra computation time.
Recently ray casting techniques became fast enough on
graphic cards equipped with a programmable graphics pro-
cessing unit (GPU) [133]. As the surfaces need not be trans-
formed to polygons the images are smoother. They do not
suffer from holes due to discontinuities in the image and the
sampling is optimal for a specific viewing direction. Integration
of physical phenomena like refraction, reflexion, and shadows
are possible with this rendering technique.
As a welcome side effect of surface rendering distances and
cutting points can be calculated when visualizing surfaces.
The segmentation step is a major obstacle for this kind of
visualization. Segmentation of image data is still considered a
hard problem with brisk research going on. Available solutions
offer automatic segmentation only for limited number of struc-
tures. Manual and semiautomatic solutions can be time-con-
suming or at least time-consuming to learn. The benefit from
such visualization using an interactive segmentation has to jus-
tify the extra work load on the team.
3) Volume Rendering: Direct volume rendering [134] cre-
ates the visualization by following rays from a certain viewpoint
through 3D data. Depending on the source of data and the in-
tended visualization different functions are available for gener-
ating a pixel from the ray. The most prominent function is the
weighted sum of voxels. A transfer function assigns a color and
transparency to each voxel intensity. It may be further refined
with the image gradient. A special kind of volume rendering is
the digitally reconstructed radiograph (DRR) that provides pro-
jections of a CT data set that are similar to X-ray images.
The advantage of direct volume rendering is a visualization
that has the capability of emphasizing certain tissues without an
explicit segmentation thanks to a certain transfer function. Clear
transitions between structures are not necessary. Also cloudy
structures and their density can be visualized.
The major disadvantage used to be too slow rendering in par-
ticular for AR, but hardware supported rendering algorithms on
current graphic cards can provide sufficient frame rates on real
3D data. Currently, 3D texture based [135] and GPU acceler-
ated raycast [133] renderers are the state of the art in terms
of speed and image quality, where the latter offer better image
quality. Also the ray casting technique needs clear structures in
the image for acceptable results, which can be realized with con-
trast agents or segmentation.
F. User Interaction in Medical AR Environments
Classic 2D computer interaction paradigms such as windows,
mouse pointer, menus, and keyboards do not translate well for
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
463
3D displays in common. Bowman [87] gives a comprehensive
introduction into 3D user interfaces and detailed information
why 3D interaction is difficult. The book gives general advice
for creating new user interfaces. Reitinger et al. [136] suggest a
3D user interface for liver planning. Even though the suggested
planning is done in pure virtual space the ideas apply to AR as
well. They use tracked instruments and a tracked glass plane
for defining points and planes in a tangible way. They combine
tangible 3D interaction and classic 2D user interfaces in an ef-
fective way.
Navab et al. [137] suggest a new paradigm for interaction
with 3D data. A virtual mirror is augmented into the scene. The
physician has the possibility to explore the data from any side
using the mirror without giving up the registered view. Since the
interaction uses a metaphor that has a very similar real counter-
part, no extra learning is expected for a user.
Apart from 2D/3D issues, standard 2D computer interfaces
such as mice are not suited for the OR because of sterility and er-
gonomic reasons. Fortunately, medical systems are highly spe-
cialized on the therapy. Since a specialized application has a
limited number of meaningful visualization modes the user in-
terface can be highly specialized as well. Context aware systems
can further reduce the degree of interaction. Automatic work-
flow recovery as suggested by Ahmadi et al. [138] could detect
phases of the surgery and with this information the computer
system could offer suitable information for each phase.
V. PERSPECTIVE
After two decades of research on medical AR the basic con-
cepts seem to be well understood and the enabling technologies
are now enough advanced to meet the basic requirements for a
number of medical applications. We are encouraged by our clin-
ical partners to believe that medical AR systems and solutions
could be accepted by physicians, if they are integrated seam-
lessly into the clinical workflow and if they provide a signifi-
cant benefit at least for one particular phase of this workflow. A
perfect medical AR user interface would be integrated in such a
way that the user would not feel its existence, while taking full
advantage of additional in situ information it provides.
Generally, augmented optics and augmented endoscopes
do not dramatically change the OR environment, apart from
adding a tracking system imposing free line of sight constraints,
and change the current workflow minimally and smoothly. The
major issue they are facing is appropriate depth perception
within a mixed environment, which is subject of active re-
search within the AR community [128], [137], [139], [140].
Augmented medical imaging devices provide aligned views by
construction and do not need additional tracking systems. If
they do not restrict the working space of the physicians these
systems have the advantage of a smooth integration into the
medical workflow. In particular, the CAMC system is currently
getting deployed within three German hospitals and will be
soon tested on 40 patients in each of these medical centers.
The AR window and video-see-through HMD systems still
need hardware and software improvement in order to satisfy the
requirements of operating physicians. In both cases, the commu-
nity also needs new concepts and paradigms allowing the physi-
cians to take full advantage of the augmented virtual data, to
464
easily and intuitively interact with it, and to experience this dy-
namic mixed environment as one unique and correctly perceived
3D world. We share the expectations of business analysts [141]
that the hype level of augmented reality will reach its maximum
in a few years and that medical AR will be one of its first killer
applications, saving lives of many future patients.
REFERENCES
[1] H. Steinhaus, “Sur la localisation au moyen des rayons x,” Comptes
Rendus de L’Acad. des Sci., vol. 206, pp. 1473–1475, 1938.
[2] I. Sutherland, “A head-mounted three dimensional display,” in Proc.
Fall Joint Computer Conf., 1968, pp. 757–764.
[3] D. Roberts, J. Strohbehn, J. Hatch, W. Murray, and H. Kettenberger,
“A frameless stereotaxic integration of computerized tomographic
imaging and the operating microscope,” J. Neurosurg., vol. 65, no. 4,
pp. 545–549, 1986.
[4] T. Caudell and D. Mizell, “Augmented reality: An application of
heads-up display technology to manual manufacturing technology
augmented reality,” in Proc. Hawaii Int. Conf. on Syst. Sci., 1992.
[5] W. Mackay, G. Velay, K. Carter, C. Ma, and D. Pagani, “Augmenting
reality: Adding computational dimensions to paper,” Commun. ACM,
vol. 36, no. 7, pp. 96–97, 1993.
[6] S. Feiner, B. Macintyre, and D. Seligmann, “Knowledge-based aug-
mented reality,” Commun. ACM, vol. 36, no. 7, pp. 53–62, 1993.
[7] P. Wellner, W. Mackay, and R. Gold, “Computer augmented environ-
ments: Back to the real world,” Commun. ACM, vol. 36, no. 7, pp.
24–26, 1993.
[8] P. Milgram and F. Kishino, “A taxonomy of mixed reality visual dis-
plays,” IEICE Trans. Inf. Syst., pp. 1321–1329, 1994.
[9] R. T. Azuma, “A survey of augmented reality,” Presence: Teleopera-
tors and Virtual Environments, vol. 6, no. 4, pp. 355–385, 1997.
[10] R. T. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B.
MacIntyre, “Recent advances in augmented reality,” IEEE Comput.
Graphics Appl., vol. 21, pp. 34–47, 2001.
[11] F. P. Brooks, “The computer scientist as toolsmith ii,” Commun. ACM,
vol. 39, no. 3, pp. 61–68, 1996.
[12] M. Bajura, H. Fuchs, and R. Ohbuchi, “Merging virtual objects with
the real world: Seeing ultrasound imagery within the patient,” in Proc.
19th Annu. Conf. on Computer Graphics and Interactive Techniques,
1992, pp. 203–210.
[13] M. Bajura and U. Neumann, “Dynamic registration correction in video-
based augmented reality systems,” IEEE Comput. Graph. Appl., vol.
15, no. 5, pp. 52–60, 1995.
[14] M. C. Jacobs, M. A. Livingston, and A. State, “Managing latencyin
complex augmented reality systems,” in Proc. ACM 1997 Symp. on
Interactive 3D Graphics, 1997, pp. 49–54.
[15] A. State, M. A. Livingston, W. F. Garrett, G. Hirota, M. C. Whitton,
E. D. Pisano, and H. Fuchs, “Technologies for augmented reality
systems: Realizing ultrasound-guided needle biopsies,” in SIG-
GRAPH’96: Proc. 23rd Annu. Conf. on Computer Graphics and
Interactive Techniques, New York, NY, USA, 1996, pp. 439–446.
[16] M. Rosenthal, A. State, J. Lee, G. Hirota, J. Ackerman, E. D. P. K.
Keller, M. Jiroutek, K. Muller, and H. Fuchs, “Augmented reality
guidance for needle biopsies: An initial randomized, controlled trial in
phantoms,” Medical Image Anal., vol. 6, no. 3, pp. 313–320, 2002.
[17] F. Sauer, F. Wenzel, S. Vogt, Y. Tao, Y. Genc, and A. Bani-Hashemi,
“Augmented workspace: Designing an AR testbed,” in Proc. IEEE and
ACM Int. Symp. on Augmented Reality, 2000, pp. 47–53.
[18] W. A. Hoff and T. L. Vincent, “Analysis of head pose accuracy in
augmented reality,” IEEE Trans. Visualiz. Computer Graphics, vol. 6,
2000.
[19] D. L. Wright, J. P. Rolland, and A. R. Kancherla, “Using virtual reality
to teach radiographic positioning,” Radiologic Int. Technol., vol. 66,
no. 4, pp. 233–238, 1995.
[20] Y. Argotti, L. Davis, V. Outters, and J. Rolland, “Dynamic superimpo-
sition of synthetic objects on rigid and simple-deformable real objects,”
Computers & Graphics, vol. 26, no. 6, pp. 919–930, 2002.
[21] T. Sielhorst, T. Blum, and N. Navab, “Synchronizing 3d movements
for quantitative comparison and simultaneous visualization of actions,”
in Proc. IEEE and ACM Int. Symp. on Mixed and Augmented Reality
(ISMAR), 2005.
[22] G. Luo and E. Peli, “Use of an augmented-vision device for visual
search by patients with tunnel vision,” Investigative Ophthalmology &
Visual Science, vol. 47, no. 9, pp. 4152–4159, 2006.
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO. 4, DECEMBER 2008
[23] J. P. Rolland and H. Fuchs, “Optical versus video see-through
head-mounted displays in medical visualization,” Presence, vol. 9, pp.
287–309, 2000.
[24] O. Cakmakci and J. Rolland, “Head-worn displays: A review,” Int. J.
Display Int. Technol., vol. 2, pp. 199–216, Sept. 2006.
[25] P. J. Kelly, G. Alker, and S. Goerss, “Computer-assisted stereotactic
laser microsurgery for the treatment of intracranial neoplasms,” Neu-
rosurgery, vol. 10, pp. 324–331, 1982.
[26] J. Hatch and J. S. D. W. Roberts, “Reference-display system for the
integration of CT scanning and the operating microscope,” in Proc.
11th Annu. Northeast Bioeng. Conf., 1985.
[27] E. M. Friets, J. W. Strohbehn, J. F. Hatch, and D. W. Roberts, “A frame-
less stereotaxic operating microscope for neurosurgery,” IEEE Trans.
Biomed. Eng., vol. 36, no. 6, Jun. 1989.
[28] P. J. Edwards, D. D. Hill, D. D. Hawkes, and D. A. Colchester,
“Neurosurgical guidance using the stereo microscope,” in Proc. First
Int. Conf. Computer Vision, Virtual Reality and Robotics in Medicine
(CVRMed’95), 1995.
[29] A. P. King, P. J. Edwards, C. R. Maurer, Jr., D. A. de Cunha, D. J.
Hawkes, D. L. G. Hill, R. P. Gaston, M. R. Fenlon, A. J. Strong, C. L.
Chandler, A. Richards, and M. J. Gleeson, “Design and evaluation of
a system for microscope-assisted guided interventions,” IEEE Trans.
Med. Imag., vol. 19, no. 11, pp. 1082–1093, Nov. 2000.
[30] J. Berger and D. Shin, “Computer-vision-enabled augmented re-
ality fundus biomicroscopy,” Ophthalmology, vol. 106, no. 10, pp.
1935–1941, 1999.
[31] W. Birkfellner, M. Figl, K. Huber, F. Watzinger, F. Wanschitz, J.
Hummel, R. Hanel, W. Greimel, P. Homolka, R. Ewers, and H.
Bergmann, “A head-mounted operating binocular for augmented
reality visualization in medicine—Design and initial evaluation,”
IEEE Trans. Med. Imag., vol. 21, no. 8, pp. 991–997, Aug. 2002.
[32] W. Birkfellner, K. Huber, F. Watzinger, M. Figl, F. Wanschitz, R. Hanel,
D. Rafolt, R. Ewers, and H. Bergmann, “Development of the variosco-
pear—A see-through hmd for computer-aided surgery,” in Proc. IEEE
and ACM Int. Symp. on Augmented Reality, 2000, pp. 54–59.
[33] M. Figl, C. Ede, J. Hummel, F. Wanschitz, R. Ewers, H. Bergmann, and
W. Birkfellner, “A fully automated calibration method for an optical
see-through head-mounted operating microscope with variable zoom
and focus,” IEEE Trans. Med. Imag., vol. 24, no. 11, pp. 1492–1499,
Nov. 2005.
[34] R. Holloway, “Registration error analysis for augmented reality,” Pres-
ence: Teleoperators and Virtual Env., vol. 6, no. 4, pp. 413–432, 1997.
[35] Y. Masutani, M. Iwahara, O. Samuta, Y. Nishi, N. Suzuki, M. Suzuki,
T. Dohi, H. Iseki, and K. Takakura, “Development of integral pho-
tography-based enhanced reality visualization system for surgical sup-
port,” Proc. lSCAS, vol. 95, pp. 16–17, 1995.
[36] H. Liao, S. Nakajima, M. Iwahara, E. Kobayashi, I. Sakuma, N. Ya-
hagi, and T. Dohi, “Intra-operative real-time 3-D information display
system based on integral videography,” in Proc. Int. Conf. Med. Image
Computing and Computer Assisted Intervention (MICCAI), 2001, pp.
392–400.
[37] H. Liao, N. Hata, S. Nakajima, M. Iwahara, I. Sakuma, and T. Dohi,
“Surgical navigation by autostereoscopic image overlay of integral
videography,” IEEE Trans. Int. Technol. Biomed., vol. 8, no. 2, pp.
114–121, 2004.
[38] M. Blackwell, C. Nikou, A. M. D. Gioia, and T. Kanade, I. W. M.
Wells, A. C. F. Colchester, and S. L. Delp, Eds., “An image overlay
system for medical data visualization,” in Proc. First Int. Conf. of Med-
ical Image Computing and Computer-Assisted Intervention (MICCAI)
Lecture Notes in Comput. Sci., Cambridge, MA, Oct. 1998, vol. 1496,
pp. 232–240.
[39] G. Goebbels, K. Troche, M. Braun, A. Ivanovic, A. Grab, K. von
Lübtow, R. Sader, F. Zeilhofer, K. Albrecht, and K. Praxmarer,
“Arsystricorder—development of an augmented reality system for
intraoperative navigation in maxillofacial surgery,” in Proc. IEEE and
ACM Int. Symp. on Mixed and Augmented Reality (ISMAR), 2003.
[40] S. Wesarg, E. Firle, B. Schwald, H. Seibert, P. Zogal, and S. Roed-
diger, R. L. Galloway, Jr., Ed., “Accuracy of needle implantation in
brachytherapy using a medical AR system: A phantom study,” Proc.
2004 SPIE Medical Imaging: Visualization, Image-Guided Procedures,
and Display, vol. 5367, pp. 341–352, 2004.
[41] W. Lorensen, H. Cline, C. Nafis, R. Kikinis, D. Altobelli, L. Gleason,
G. Co, and N. Schenectady, “Enhancing reality in the operating room,”
in IEEE Conf. on Visualization, 1993, pp. 410–415.
[42] W. E. L. Grimson, T. Lozano-Perez, W. M. Wells III, G. J. Ettinger, S. J.
White, and R. Kikinis, “An automatic registration method for frameless
stereotaxy, image guided surgery, and enhanced reality visualization,”
IEEE Trans. Med. Imag., vol. 15, no. 2, pp. 129–140, Apr. 1996.
SIELHORST et al.: ADVANCED MEDICAL DISPLAYS: A LITERATURE REVIEW OF AUGMENTED REALITY
[43] Y. Sato, M. Nakamoto, Y. Tamaki, T. Sasama, I. Sakita, Y. Naka-
jima, M. Monden, and S. Tamura, “Image guidance of breast cancer
surgery using 3-d ultrasound images and augmented reality visualiza-
tion,” IEEE Trans. Med. Imag., vol. 17, no. 5, Oct. 1998.
[44] S. Nicolau, X. Pennec, L. Soler, and N. Ayache, “An accuracy certi-
fied augmented reality system for therapy guidance,” in Proc. 8th Eur.
Conf. on Computer Vision (ECCV 04), Prague, May 2004, vol. 3023,
pp. 79–91, Lecture Notes in Computer Science.
[45] W. Lau, “History of endoscopic and laparoscopic surgery,” World Int.
J.f Surgery, vol. 21, no. 4, pp. 444–453, 1997.
[46] G. Litynski, “Endoscopic surgery: The history, the pioneers,” World
Int. J. Surgery, vol. 23, no. 8, pp. 745–753, 1999.
[47] G. Berci and K. A. Forde, “History of endoscopy—What lessons have
we learned from the past?,” Surgical Endoscopy, vol. 14, pp. 5–15,
2002.
[48] W. Freysinger, A. Gunkel, and W. Thumfart, “Image-guided endoscop-
icent surgery,” Eur. Archives of Otorhinolaryngology, vol. 254, no. 7,
pp. 343–346, 1997.
[49] R. Shahidi, B. Wang, M. Epitaux, R. Grzeszczuk, and J. Adler, I. W.
M. Wells, A. C. F. Colchester, and S. L. Delp, Eds., “Volumetric image
guidance via a stereotactic endoscope,” in Proc. First Int. Conf. Med.
Image Computing and Computer-Assisted Intervention (MICCAI),
Cambridge, MA, Oct. 1998, vol. 1496, pp. 241–252, Lecture Notes in
Computer Science.
[50] M. Scholz, W. Konen, S. Tombrock, B. Fricke, and L. Adams, “De-
velopment of an endoscopic navigation system based on digitalimage
processing,” Computer Aided Surgery, vol. 3, pp. 134–143, 1998.
[51] F. Mourgues and È. Coste-Manière, “Flexible calibration of actuated
stereoscopic endoscope for overlay in robot assisted surgery,” in Proc.
Int. Conf. Med. Image Computing and Computer Assisted Intervention
(MICCAI), 2002, pp. 25–34.
[52] R. Khadem, M. R. Bax, J. A. Johnson, E. P. Wilkinson, and R. Shahidi,
“Endoscope calibration and accuracy testing for 3d/2d image registra-
tion,” in Proc. Int. Conf. Medical Image Computing and Computer-As-
sisted Intervention (MICCAI), 2001, pp. 1361–1362.
[53] W. E. Smith, N. Vakil, and S. A. Maislin, “Correction of distortion
in endoscope images,” IEEE Trans. Med. Imag., vol. 11, no. 1, pp.
117–122, Feb. 1992.
[54] S. De Buck, J. Van Cleynenbreuge, I. Geys, T. Koninckx, P. R. Kon-
inck, and P. Suetens, “A system to support laparoscopic surgery by
augmented reality visualization,” in Proc. Int. Conf. Medical Image
Computing and Computer Assisted Intervention (MICCAI), 2001, pp.
691–698.
[55] R. Shahidi, M. R. Bax, C. R. Maurer, Jr., J. A. Johnson, E. P.
Wilkinson, B. Wang, J. B. West, M. J. Citardi, K. H. Manwaring, and
R. Khadem, “Implementation, calibration and accuracy testing of an
image-enhanced endoscopy system,” IEEE Trans. Med. Imag., vol.
21, no. 12, pp. 1524–1535, 2002.
[56] J. Heikkilä and O. Silvén, “A four-step camera calibration procedure
with implicit image correction,” in Proc. IEEE Conf. Computer Vision
and Pattern Recognition (CVPR), 1997, pp. 1106–1112.
[57] R. Tsai, “A versatile camera calibration technique for high accuracy
3D machine vision metrology using off-the-shelf TV cameras and
lenses,” IEEE Int. J. Robot. Autom., vol. RA-3, no. 4, pp. 323–344,
1987.
[58] Z. Zhang, “A flexible new technique for camera calibration,” IEEE
Trans. .Pattern Anal. Mach. Intell., vol. 22, pp. 1330–1334, Nov.
2000.
[59] D. Stoyanov, A. Darzi, and G. Z. Yang, “Laparoscope self-calibration
for robotic assisted minimally invasive surgery,” in Proc. Int. Conf.
Med. Image Computing and Computer Assisted Intervention (MIC-
CAI), J. Duncan and G. Gerig, Eds., 2005, vol. 3750, pp. 114–121,
Lecture Notes in Computer Science.
[60] R. Willson, “Modeling and calibration of automated zoom lenses.,”
Ph.D. dissertation, Robotics Inst., Carnegie Mellon Univ., Pittsburgh,
PA, Jan. 1994.
[61] G. Bianchi, C. Wengert, M. Harders, P. Cattin, and G. Székely,
“Camera-marker alignment framework and comparison with hand-eye
calibration for augmented reality applications 2005,” ISMAR, pp.
188–189, Oct. 2005.
[62] S. Nicolau, L. Goffin, and L. Soler, “A low cost and accurate guidance
system for laparoscopic surgery: Validation on an abdominal phantom,”
in ACM Symp. on Virtual Reality Software and Int. Technol., Nov. 2005,
pp. 124–133.
[63] M. Scheuering, A. Schenk, A. Schneider, B. Preim, and G. Greiner, “In-
traoperative augmented reality for minimally invasive liver interven-
tions,” Proc. 2003 SPIE Medical Imag.: Visualization, Image-Guided
Procedures, and Display,, 2003.
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
465
[64] J. Schmidt, F. Vogt, and H. Niemann, “Robust hand-eye calibration of
an endoscopic surgery robot using dual quaternions,” in 25th DAGM
Symp. Pattern Recogn., B. Michaelis and G. Krell, Eds., 2003, vol.
2781, pp. 548–556, Lecture Notes in Computer Science.
[65] G. Marti, V. Bettschart, J.-S. Billiard, and C. Baur, “Hybrid method
forboth calibration and registration of an endoscope with an active op-
tical tracker,” Comput. Assisted Radiology and Surgery, 2004.
[66] T. Yamaguchi, M. Nakamoto, Y. Sato, K. Konishi, M. Hashizume,
N. Sugano, H. Yoshikawa, and S. Tamura, “Development of a camera
model and calibration procedure for oblique-viewing endoscopes,”
Computer Aided Surgery, vol. 9, no. 5, pp. 203–214, 2004.
[67] S. De Buck, F. Maes, A. D’Hoore, and P. Suetens, N. Ayache, S.
Ourselin, and A. Maeder, Eds., “Evaluation of a novel calibration
technique for optically tracked oblique laparoscopes,” in Proc. Int.
Conf. Medical Image Computing and Computer Assisted Interven-
tion (MICCAI), Brisbane, Australia, Oct./Nov. 2007, vol. 4791, pp.
467–474, Lecture Notes in Computer Science.
[68] I. Bricault, G. Ferretti, and P. Cinquin, “Registration of real and CT-de-
rived virtual bronchoscopic images to assist transbronchial biopsy,”
IEEE Trans. Med. Imag., vol. 17, no. 5, pp. 703–714, 1998.
[69] K. Mori, D. Deguchi, J. Ichi Hasegawa, Y. Suenaga, J. ichiro Toriwaki,
H. Takabatake, and H. Natori, “A method for tracking the camera mo-
tion of real endoscope by epipolar geometry analysis and virtual en-
doscopy system,” in Proc. Int. Conf. Medical Image Computing and
Computer Assisted Intervention (MICCAI), 2001, vol. 2208, pp. 1–8,
Lecture Notes in Computer Science.
[70] K. Mori, D. Deguchi, K. Akiyama, T. Kitasaka, C. R. Maurer Jr., Y.
Suenaga, H. Takabatake, M. Mori, and H. Natori, “Hybrid broncho-
scope tracking using a magnetic tracking sensor and image registra-
tion,” in Proc. Int. Conf. Medical Image Computing and Computer As-
sisted Intervention (MICCAI), 2005, pp. 543–550.
[71] I. Wegner, M. Vetter, M. Schoebinger, I. Wolf, and H. Meinzer, K. R.
Cleary and R. L. Galloway Jr., Eds., “Development of a navigation
system for endoluminal brachytherapy in human lungs,” in Proc. SPIE
Medical Imaging 2006: Visualization, Image-Guided Procedures, and
Display, Mar. 2006, vol. 6141, pp. 23–30.
[72] T. Klein, J. Traub, H. Hautmann, A. Ahmadian, and N. Navab, “Fidu-
cial free registration procedure for navigated bronchoscopy,” in Proc.
Int. Conf. Medical Image Computing and Computer Assisted Interven-
tion (MICCAI), 2007.
[73] S. Krüger, F. Vogt, W. Hohenberger, D. Paulus, H. Niemann, and C. H.
Schick, “Evaluation of computer-assisted image enhancementin min-
imal invasive endoscopic surgery,” Methods Inf. Med., vol. 43, no. 4,
pp. 362–366, 2004.
[74] H. Fuchs, M. A. Livingston, R. Raskar, D. Colucci, K. Keller, A. State,
J. R. Crawford, P. Rademacher, S. H. Drake, and A. A. Meyer, I. W. M.
Wells, A. C. F. Colchester, and S. L. Delp, Eds., “Augmented reality
visualization for laparoscopic surgery,” in Proc. First Int. Conf. of Med-
ical Image Computing and Computer-Assisted Intervention (MICCAI),
Cambridge, MA, Oct. 1998, vol. 1496, pp. 934–943, Lecture Notes in
Computer Science.
[75] D. Dey, D. Gobbi, P. Slomka, K. Surry, and T. Peters, “Automatic
fusion of freehand endoscopic brain images to three-dimensional sur-
faces: Creating stereoscopic panoramas,” IEEE Trans. Med. Imag., vol.
21, no. 1, pp. 23–30, Jan. 2002.
[76] T. Kawamata, H. Iseki, T. Shibasaki, and T. Hori, “Endoscopic
augmented reality navigation system for endonasal transsphenoidal
surgery to treat pituitary tumors: Technical note,” Neurosurgery, vol.
50, no. 6, pp. 1393–1397, 2002.
[77] J. Ellsmere, J. Stoll, D. W. Rattner, D. Brooks, R. Kane, W. M. Wells
III, R. Kikinis, and K. Vosburgh, R. E. Ellis and T. M. Peters, Eds., “A
navigation system for augmenting laparoscopic ultrasound,” in Proc.
Int. Conf. Medical Image Computing and Computer Assisted Inter-
vention (MICCAI), 2003, pp. 184–191, Lecture Notes in Computer
Science.
[78] N. Navab, M. Mitschke, and A. Bani-Hashemi, “Merging visible and
invisible: Two camera-augmented mobile C-arm (CAMC) applica-
tions,” in Proc. IEEE and ACM Int. Workshop on Augmented Reality,
San Francisco, CA, 1999, pp. 134–141.
[79] G. Fichtinger, A. Deguet, K. Masamune, E. Balogh, G. S. Fischer, H.
Mathieu, R. H. Taylor, S. J. Zinreich, and L. M. Fayad, “Image overlay
guidance for needle insertion in ct scanner,” IEEE Trans. Biomed.
Engi., vol. 52, no. 8, pp. 1415–1424, Aug. 2005.
[80] K. Masamune, Y. Masutani, S. Nakajima, I. Sakuma, T. Dohi, H. Iseki,
and K. Takakura, “Three-dimensional slice image overlay system with
accurate depth perception for surgery,” in Proc. Int. Conf. Medical
Image Computing and Computer Assisted Intervention (MICCAI), Oct.
2000, vol. 1935, pp. 395–402, Lecture Notes in Computer Science.
466
[81] G. Fischer, A. Deguet, D. Schlattman, L. Fayad, S. Zinreich, R. Taylor,
and G. Fichtinger, “Image overlay guidance for mri arthrography
needle insertion,” Computer Aided Surgery, vol. 12, no. 1, pp. 2–4,
2007.
[82] G. D. Stetten and V. S. Chib, “Overlaying ultrasound images on
direct vision,” Int. J. Ultrasound in Medicine, vol. 20, pp. 235–240,
2001.
[83] G. Stetten, V. Chib, and R. Tamburo, “Tomographic reflection to merge
ultrasound images with direct vision,” in IEEE Proc. Applied Imagery
Pattern Recognition (AIPR) Annu. Workshop, 2000, pp. 200–205.
[84] N. Glossop and Z. Wang, R. E. Ellis and T. M. Peters, Eds., “Laser
projection augmented reality system for computer assisted surgery,”
in Proc. Int. Conf. Medical Image Computing and Computer Assisted
Intervention (MICCAI), 2003, vol. 2879, pp. 239–246, Lecture Notes
in Computer Science.
[85] T. Sasama et al., “A novel laser guidance system for alignment of
linear surgical tools: Its principles and performance evaluation as a
man-machine system,” in Proc. Int. Conf. Medical Image Computing
and Computer Assisted Intervention (MICCAI), London, U.K., 2002,
pp. 125–132.
[86] T. Wendler, J. Traub, S. Ziegler, and N. Navab, R. Larsen, M. Nielsen,
and J. Sporring, Eds., “Navigated threedimensional beta probe for op-
timal cancer resection,” in Proc. Int. Conf. Medical Image Computing
and Computer Assisted Intervention (MICCAI) vol. 4190 of Lecture
Notes in Computer Science, Copenhagen, Denmark, Oct. 2006, pp.
561–569.
[87] D. A. Bowman, E. Kruijff, J. J. LaViola, and I. Poupyrev, 3D User-
Interfaces: Theory and Practice. Redwood City, CA, USA: Addison
Wesley Longman Publishing Co., Inc., 2004.
[88] J. Traub, P. Stefan, S.-M. Heining, T. Sielhorst, C. Riquarts, E. Eu-ler,
and N. Navab, R. Larsen, M. Nielsen, and J. Sporring, Eds., “Hybrid
navigation interface for orthopedic and trauma surgery,” in Proc. Int.
Conf. Medical Image Computing and Computer Assisted Intervention
(MICCAI), Copenhagen, Denmark, Oct. 2006, vol. 4190, pp. 373–380,
Lecture Notes in Computer Science.
[89] T. Peters, C. Henri, P. Munger, A. Takahashi, A. Evans, B. Davey, and
A. Olivier, “Integration of stereoscopic DSA and 3D MRI for image-
guided neurosurgery,” Comput Med Imaging Graph., vol. 18, no. 4, pp.
289–299, 1994.
[90] C. Calvano, M. Moran, L. Tackett, P. Reddy, K. Boyle, and M.
Pankra-tov, “New visualization techniques for in utero surgery:
Amnioscopy with a three-dimensional head-mounted display and
a computer-controlled endoscope.,” J. Endourol., vol. 12, no. 5, pp.
407–410, 1998.
[91] G. Hanna, S. Shimi, and A. Cuschieri, “Task performance in endo-
scopic surgery is influenced by location of the image display,” Ann.
Surg., vol. 227, no. 4, pp. 481–484, 1998.
[92] J. B. A. Maintz and M. A. Viergever, “A survey of medical image reg-
istration,” Medical Image Analysis, vol. 2, pp. 1–36, Mar. 1998.
[93] M. Tuceryan, D. S. Greer, R. T. Whitaker, D. E. Breen, C. Crampton, E.
Rose, and K. H. Ahlers, “Calibration requirements and proceduresfor a
monitor-based augmented reality system,” IEEE Trans. Visualiz. Com-
puter Graphics, vol. 1, pp. 255–273, Sept. 1995.
[94] C. R. Maurer Jr., J. M. Fitzpatrick, M. Y. Wang, J. Robert, L. Galloway,
R. J. Maciunas, and G. S. Allen, “Registration of head volume images
using implantable fiducial markers,” IEEE Trans. Med. Imag., vol. 16,
no. 4, pp. 447–462, Aug. 1997.
[95] J. D. Stefansic, A. J. Herline, Y. Shyr, W. C. Chapman, J. M. Fitz-
patrick, and R. L. Galloway, “Registration of physical space to la-
paroscopic image space for use in minimally invasive hepatic surgery,”
IEEE Trans. Medical Imaging, vol. 19, no. 10, pp. 1012–1023, Oct.
2000.
[96] M. Feuerstein, S. M. Wildhirt, R. Bauernschmitt, and N. Navab, J. S.
Duncan and G. Gerig, Eds., “Automatic patient registration for port
placement in minimally invasive endoscopic surgery,” in Proc. Int.
Conf. Medical Image Computing and Computer Assisted Intervention
(MICCAI), Palm Springs, CA, Sept. 2005, vol. 3750, pp. 287–294,
Lecture Notes in Computer Science.
[97] M. Feuerstein, T. Mussack, S. M. Heining, and N. Navab, “Intra-oper-
ative laparoscope augmentation for port placement and resection plan-
ning in minimally invasive liver resection,” IEEE Trans. Med. Imag.,
vol. 27, pp. 355–369, Mar. 2008.
[98] M. Baumhauer, T. Simpfendörfer, R. Schwarz, M. Seitel, B. Müller-
Stich, C. Gutt, J. Rassweiler, H.-P. Meinzer, and I. Wolf, K. R. Cleary
and M. I. Miga, Eds., “Soft tissue navigation for laparoscopic prostate-
ctomy: Evaluation of camera pose estimation for enhanced visualiza-
tion,” in Proc. SPIE Medical Imaging 2007: Visualization and Image-
Guided Procedures, San Diego, CA, Feb. 2007.
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
JOURNAL OF DISPLAY TECHNOLOGY, VOL. 4, NO. 4, DECEMBER 2008
[99] R. U. Thoranaghatte, G. Zheng, F. Langlotz, and L.-P. Nolte, “En-
doscope-based hybrid navigation system for minimally invasive ven-
tral spine surgeries,” Computer Aided Surgery, vol. 10, pp. 351–356,
Sept./Nov. 2005.
[100] J. M. Fitzpatrick, J. B. West, and C. R. Maurer Jr., “Predicting error in
rigid-body point-based registration,” IEEE Trans. Med. Imag., vol. 14,
no. 5, pp. 694–702, Oct. 1998.
[101] R. Grzeszczuk, S. Chin, R. Fahrig, H. Abassi, H. Holz, J. A. Daniel
Kiml, and R. Shahidi, “A fluoroscopic x-ray registration processfor
three-dimensional surgical navigation,” in Proc. Int. Conf. Medical
Image Computing and Computer Assisted Intervention (MICCAI),
Lecture Notes in Computer Science, 2000, Springer-Verlag.
[102] M. J. Murphy, “An automatic six-degree-of-freedom image registration
algorithm for image-guided frameless stereotaxic radiosurgery,” Med-
ical Phys., vol. 24, no. 6, pp. 857–866, 1997.
[103] M. Feuerstein, T. Mussack, S. M. Heining, and N. Navab, K. R. Cleary
and M. I. Miga, Eds., “Registration-free laparoscope augmentation for
intra-operative liver resection planning,” in Medical Imaging 2007:
Visualization and Image-Guided Procedures, Proc. SPIE, San Diego,
CA, Feb. 2007.
[104] P. A. Grützner, A. Hebecker, H. Waelti, B. Vock, L.-P. Nolte, and A.
Wentzensen, “Clinical sludy for registration-free 3d-navigation with
the siremobil iso-c mobile C-arm,” Electromedica, vol. 71, no. 1,
pp. 7–16, 2003.
[105] D. Burschka, M. Li, R. Taylor, and G. D. Hager, “Scale-invarianlregis-
tration of monocular endoscopic images to ct-scans for sinus surgery,”
in Proc. Int. Conf. Medical Image Computing and Computer Assisted
Intervention (MICCAI), C. Barillol, D. Haynor, and P. Hellier, Eds.,
2004, vol. 3217, pp. 413–421, Lecture Notes in Computer Science.
[106] C. Wengert, P. Cattin, J. M. Duff, C. Baur, and G. Székely, “Marker-
less endoscopic registration and referencing,” in Proc. Int. Conf. Med-
ical Image Computing and Computer Assisted Intervention (MICCAI),
R. Larsen, M. Nielsen, and J. Sporring, Eds., 2006, Lecture Notes in
Computer Science.
[107] C. Maurer, Jr., “Investigation of intraopera live brain deformation using
a 1.5-T interventional MR system: Preliminary results,” IEEE Trans.
Med. Imag., vol. 17, no. 5, p. 817, Oct. 1998.
[108] M. Nakamoto, Y. Sato, M. Miyamoto, Y. Nakamjima, K. Konishi, M.
Shimada, M. Hashizume, and S. Tamura, “3d ultrasound system using
a magneto-optic hybrid tracker for augmented reality visualization in
laparoscopic liver surgery,” in Proc. Int. Conf. Medical Image Com-
puting and Computer Assisted Intervention (MICCAI), T. Dohiand and
R. Kikinis, Eds., 2002, vol. 2489, pp. 148–155, Lecture Notes in Com-
puter Science.
[109] J. Leven, D. Burschka, R. Kumar, G. Zhang, S. Blumenkranz, X. D.
Dai, M. Awad, G. D. Hager, M. Marohn, M. Choli, C. Hasser, and R.
H. Taylor, “Davinci canvas: A teterobotic surgical system wilh inte-
grated, robot-assisted, laparoscopic ultrasound capability,” in Proc. Int.
Conf. Medical Image Computing and Computer Assisted Intervention
(MICCAI), Vol. 3749 of Lecture Notes in Computer Science, Sept. 2005,
pp. 811–818, Springer-Verlag.
[110] M. Feuerstein, T. Reichl, J. Vogel, A. Schneider, H. Feussner, and N.
Navab, “Magneto-optic tracking of a flexible laparoscopic ultrasound
transducer for laparoscope augmentation,” in Proc. Int. Conf. Medical
Image Computing and Computer Assisted Intervention (MICCAI), N.
Ayache, S. Ourselin, and A. Maeder, Eds., Brisbane, Australia, Oct./
Nov. 2007, vol. 4791, pp. 458–466, Lecture Notes in Computer Sci-
ence.
[111] F. S. Azar, D. N. Metaxas, and M. D. Schnall, “Methods for modeling
and predicting mechanical deformations of the breast under external
perturbations,” Medical Image Analysis, vol. 6, no. 1, pp. 1–27, 2002.
[112] J. W. Wong, M. B. Sharpe, D. A. Jaffray, V. R. Kini, J. M. Robertson,
J. S. Stromberg, and A. A. Martinez, “The use of active breathing con-
trol (abc) to reduce margin for breathing motion,” Int. Int. J. Radiation
Oncology*Biology*Phys., vol. 44, pp. 911–919, July 1999.
[113] J. M. Bailer, L. A. Dawson, S. Kazanjian, C. McGinn, K. K. Brock, T.
Lawrence, and R. T. Haken, “Determination of ventilatory liver move-
ment via radiographic evaluation of diaphragm position,” Int.. J. Radi-
ation Oncology*Biology*Phys., vol. 51, pp. 267–270, Sept. 2001.
[114] M. Baumhauer, M. Feuerstein, H.-P. Meinzer, and J. Rassweiler, “Nav-
igation in endoscopic soft tissue surgery: Perspectives and limitations,”
Int. J. Endourology, vol. 22, pp. 1–16, Apr. 2008.
[115] F. Vogl, “Augmented light field visualization and real-time image en-
hancement for computer assisted endoscopic surgery,” Ph.D. disserta-
tion, Universilal Erlangen-Nürnberg, , 2005.
[116] C. Ware and R. Balakrishnan, “Reaching for objects in VR displays:
Lag and frame rate,” ACM Trans. Computer-Human Interaction, vol.
1, no. 4, pp. 331–356, 1994.
SIELHORST et al.: ADVANCED MEDICAL DISPLAYS: A LITERATURE REVIEW OF AUGMENTED REALITY
[117] T. Sielhorst, C. Bichlmeier, S. M. Heining, and N. Navab, “Depth
perception a major issue in medical ar: Evaluation study by twenty
surgeons,” in Proc. Int. Conf. Medical Image Computing and Com-
puter Assisted Intervention (MICCAI), R. Larsen, M. Nielsen, and J.
Sporring, Eds., 2006, Lecture Notes in Computer Science.
[118] T. Sielhorst, W. Sa, A. Khamene, F. Sauer, and N. Navab, “Measure-
ment of absolute latency for video see through augmented reality,” in
Proc. IEEE and ACM Int. Symp. on Mixed and Augmented Reality
(ISMAR), 2007.
[119] B. Mactntyre, E. M. Coelho, and S. Julier, “Estimating and adapting to
registration errors in augmented reality systems,” in Proc. IEEE Virtual
Reality (VR), 2002.
[120] T. Sielhorst, M. A. Bauer, O. Wenisch, G. Klinker, and N. Navab,
“Online estimation of the target registration error for n-ocular optical-
tracking systems,” in Proc. Int. Conf. Medical Image Computing and
Computer Assisted Intervention (MICCAI), 2007, To appear at.
[121] C. Nafis, V. Jensen, L. Beauregard, and P. Anderson, “Method for esti-
mating dynamic em tracking accuracy of surgical navigation tools,” in
Proc. SPIE Medical Imaging 2006: Visualization, Image Guided Pro-
cedures, and Display, K. R. Cleary and R. L. Galloway, Jr., Eds., Mar.
2006, vol. 6141.
[122] P. Jannin, J. Fitzpatrick, D. Hawkes, X. Pennec, R. Shahidi, and M.
Vannier, “Validation of medical image processing in image-guided
therapy,” IEEE Trans. Med. Imag., vol. 21, no. 12, pp. 1445–1449,
Dec. 2002.
[123] D. Drascic and P. Milgram, “Perceptual issues in augmented reality,”
SPIE Volume 2653: Stereoscopic Displays and Virtual Reality Syst.,
vol. 2653, pp. 123–134, 1996.
[124] J. E. Cutting and P. M. Vishton, W. Epstein and S. Rogers, Eds., “Per-
ceiving layout and knowing distances: The integration, relative po-
tency, and contextual use of different information about depth,” Per-
ception of Space and Motion, pp. 69–117, 1995.
[125] L. G. Johnson, P. Edwards, and D. Hawkes, “Surface transparency
makes stereo overlays unpredictable: The implications for augmented
reality,” in Medicine Meets Virtual Reality (MMVR), Vol. 94 of Studies
in Health Int. Technol. and Inf., J. D. Westwood, Ed. : IOS Press,
2002, pp. 131–136.
[126] C. Furmanski, R. Azuma, and M. Daily, “Augmented-reality visual-
izations guided by cognition: Perceptual heuristics for combining vis-
ible and obscured information,” in Proc. IEEE and ACM Int. Symp. on
Mixed and Augmented Reality (ISMAR), 2002.
[127] M. Livingston, I. JE, H. JL, H. TH, and S. D. Julier, “Resolving
multiple occluded layers in augmented reality,” in Proc. IEEE and
ACM Int. Symp. on Mixed and Augmented Reality (ISMAR), 2003,
pp. 56–65.
[128] M. Lerotic, A. J. Chung, G. Mylonas, and G.-Z. Yang, “pq-space
based non-photorealistic rendering for augmented reality,” in Proc.
Int. Conf. Medical Image Computing and Computer Assisted Interven-
tion (MICCAI), 2007, pp. 102–109.
[129] F. A. Biocca and J. P. Rolland, “Virtual eyes can rearrange your body:
Adaptation to visual displacement in see-through, head-mounted dis-
plays,” Presence: Teleoperators and Virtual Env., vol. 7, pp. 262–277,
1998.
[130] E. M. Kolasinski, S. L. Goldberg, and J. H. Hiller, Simulator sickness
in virtual environments U.S. Army Research Institute—Simulator Syst.
Research Unit, Technical Report 1027, 1995.
[131] G. Riccio, “An ecological theory of motion and postural instability,”
Ecological Psychology, vol. 3, no. 3, pp. 195–240, 1991.
[132] W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolu-
tion 3d surface construction algorithm,” in Proc. 14th Annu. Conf. on
SIGGRAPH’87: Computer Graphics and Interactive Techniques, New
York, NY, 1987, pp. 163–169.
[133] J. Krüger and R. Westermann, “Acceleration techniques for GPU-based
volume rendering,” Proceedings IEEE Visualization 2003, 2003.
[134] M. Levoy, “Display of surfaces from volume data,” IEEE Computer
Graphics and Appl., vol. 8, no. 3, pp. 29–37, 1988.
[135] A. Van Gelder and K. Kim, “Direct volume rendering with shading via
three-dimensional textures,” in Proc. 1996 Symp. on Volume Visualiza-
tion, Piscataway, NJ, 1996, pp. 23–30.
[136] B. Reitinger, A. Bornik, R. Beichel, and D. Schmalstieg, “Liver surgery
planning using virtual reality,” IEEE Computer Graphics and Applica-
tions, vol. 26, no. 6, pp. 36–47, 2006.
[137] N. Navab, M. Feuerstein, and C. Bichlmeier, “Laparoscopic virtual
mirror—New interaction paradigm for monitor based augmented re-
ality,” in Virtual Reality, Charlotte, NC, Mar. 2007, pp. 43–50.
Authorized licensed use limited to: IEEE Xplore. Downloaded on April 12, 2009 at 23:36 from IEEE Xplore. Restrictions apply.
467
[138] S.-A. Ahmadi, T. Sielhorst, R. Stauder, M. Horn, H. Feussner, and N.
Navab, “Recovery of surgical workflow without explicit models,” in
Proc. Int. Conf. Medical Image Computing and Computer Assisted In-
tervention (MICCAI), 2006, pp. 420–428.
[139] C. Bichlmeier, F. Wimmer, H. S. Michael, and N. Nassir, “Con-textual
anatomic mimesis: Hybrid in-situ visualization method for improving
multi-sensory depth perception in medical augmented reality,” in Proc.
IEEE and ACM Int. Symp. on Mixed and Augmented Reality (ISMAR),
Nov. 2007, pp. 129–138.
[140] D. Kalkofen, E. Mendez, and D. Schmalstieg, “Interactive focus and
context visualization for augmented reality,” in Proc. IEEE and ACM
Int. Symp. on Mixed and Augmented Reality (ISMAR), 2007.
[141] C. Pettey and L. Goasduff, Emerging Technologies Hype Cycle.
Stamford, CT: Gartner, 2006.
Tobias Sielhorst received the M.S. and Ph.D. de-
grees from Technical Universitat München in 2003
and 2008, respectively.
His major research interest is medical augmented
reality.
Dr. Sielhorst has served on the program committee
of different conferences and workshops including
IEEE and ACM International Symposium for Mixed
and Augmented Reality (ISMAR), Medical Image
Computing and Computer-Assisted Intervention
(MICCAI), and the international workshop series
AMI-ARS on Medical Augmented Reality. He co-organizes the latter one this
year as one of the general chairs.
Marco Feuerstein received the M.S. and Ph.D.
degrees from Technical Universitat München (TU
München) in computer science in 2003 and 2007, re-
spectively. He received scholarships for the National
University of Singapore, the University of Sydney,
the National Taiwan University, and the Nagoya
University.
From January 2004 to April 2005 he was employed
at the Department of Cardiothoracic Surgery at the
German Heart Center Munich. From May 2005 to
November 2007, he was with the Chair for Computer
Aided Medical Procedures & Augmented Reality at TU München. He is cur-
rently a postdoctoral associate in the Department of Media Science, Graduate
School of Information Science, at Nagoya University.
His research interests are computer aided diagnosis and surgery, medical
imaging, augmented reality, and computer vision.
Nassir Navab is a full professor and director of the
institute for Computer Aided Medical Procedures
and Augmented Reality (CAMP) at Technical Uni-
versity of München (TUM). He has also a secondary
faculty appointment at the Medical School of TU
München. Before joining the Computer Science
Department at TUMünchen, he was a distinguished
member at Siemens Corporate Research (SCR) in
Princeton, New Jersey. He received the prestigious
Siemens Inventor of the Year Award in 2001 for
the body of his work in interventional imaging. He
had received his Ph.D. from INRIA and University of Paris XI in France and
enjoyed two years of postdoctoral fellowship at MIT Media Laboratory before
joining SCR in 1994.
In November 2006, Dr. Navab was elected as a member of board of direc-
tors of MICCAI society. He has been serving on the Steering Committee of the
IEEE Symposium on Mixed and Augmented Reality since 2001. He has served
on the program committee of over 30 international conferences. He is the author
of hundreds of peer reviewed scientific papers and over 40 US and international
patents. His main fields of interest include: Medical Augmented Reality, Com-
puter Aided Surgery and Medical Image Registration.