Proceedings of the 2005 IEEE
Engineering in Medicine and Biology 27th Annual Conference
Shanghai, China, September 1-4, 2005
Dual-Modality PET-CT Visualization using Real-Time
Volume Rendering and Image Fusion with Interactive
3D Segmentation of Anatomical Structures
Jinman Kim1, 2, Student Member, IEEE, Weidong Cai1, Member, IEEE, Dagan Feng1, 2, Fellow, IEEE
1.
Biomedical and Multimedia Information Technology (BMIT) Group,
School of Information Technologies, The University of Sydney, Australia
2.
Center for Multimedia Signal Processing (CMSP)
Dept. of Electronic & Information Engineering, Hong Kong Polytechnic University, Hong Kong
Abstract-Recent advances in medical imaging have resulted
in the introduction of dual-modality scanners which can
simultaneously acquire two independent imaging modalities,
such as positron emission tomography (PET) and computed
tomography (CT) image data in a single session. These multi-
dimensional PET-CT data contain both the functional and
anatomical information of the human body, thus, providing
the ability to identify anatomical structures of interest and
then overlaying the result onto the corresponding functional
structure, as an example. The utilization of the combined
functional and anatomical information has been proven to be
an effective approach in diagnosis and interpretation of
certain medical conditions. However, the increase in image
dimensions has not been accompanied with new visualization
techniques, with the use of two-dimensional (2D) display
being the norm. In this study, we propose a new approach to
three-dimensional (3D) visualization of dual-modality PET-
CT data in order to complement the 2D visualization and
potentially improve medical diagnosis and interpretation.
We have design and implemented a prototype visualization
technique using real-time volume rendering and image
fusion running on a commodity graphics card. We further
propose the application of interactive segmentation of 3D
anatomical structures from CT data which can be used to
identify the corresponding functional structure in volume
visualization.
I. INTRODUCTION
Dual-modality medical imaging scanners such as PET-
CT permit the simultaneous acquisition of two
independent 3D image data, respectively of functional and
anatomical images. These four-dimensional (4D) dual-
modality imaging data have introduced significant
improvements in medical diagnosis and interpretation of
the human body [1-2]. Conventional approaches to
visualizing multi-modality medical data are by navigating
through 2D cross-sectional slices and in orthogonal views,
which require physicians to do sequential examinations of
0-7803-8740-6/05/$20.00 ©2005 IEEE.
images and mental 3D reconstruction of the structures.
However, such 2D visualization approaches applied to 4D
data are in-efficient and may not fully utilize the
visualization possibilities from the 4D data, such as
navigational 3D rendering.
A common supplement to visualizing the multi-
dimensional medical data is via volume rendering.
Volume rendering is a method of displaying 3D data in a
2D format by using ray-tracing and is able to produce
rendition of complex and high-detailed medical data [3].
The ability to interactively navigate the rendered volume
by “flying” the viewing window through the volume data
is rapidly becoming a very attractive method for many
medical imaging applications, including image guided
surgery and computer aided diagnosis [1, 4-6].
In volume visualization of PET-CT data, two separate
imaging data are often fused together into a single fused
data prior to volume rendering [4,5]. Alternatively, two
data can be individually rendered using different rendering
techniques, such as surface and volume renderings, prior
to fusion [5,7]. Whereas fused volume rendering permits
the adjustment of fusion ratio, the ability to apply volume
manipulation tools, such as brightness/contrast, lookup
table (LUT) manipulation, etc., are often limited to the
volume rendition of the fused data. Therefore, the
manipulation to only a particular modality in fused PET-
CT most be performed as a pre-process prior to volume
rendering.
Further complication in volume visualization of PET-
CT data is image segmentation. Often in medical imaging
applications, the identification of the structures of interest
and subsequent segmentation of these structures are
necessary [4,8]. Image segmentation refers to the process
of partitioning an image into distinct regions by grouping
together of voxels based on certain similarity criterion.
With PET-CT data, the two different modalities can
642
provide complementary information to each other which
may potentially aid in image segmentation, i.e.,
anatomical structures from the segmentation of CT data
can be used to guide the segmentation of functional
structures in PET data. However, as with other volume
manipulations, the application of PET-CT data
segmentation is usually limited to as a pre-process without
interactive control in volume rendition.
In this study, we propose a 3D visualization of dual-
modality PET-CT data using real-time volume rendering
and image fusion running on a commodity graphics card.
The PET and CT volumes are volume rendered
independently, with the fusion of these data applied in
real-time, therefore permitting the real-time control of the
fusion ratio. Furthermore, volume manipulation can be
applied to individual volumes, or when fused. We further
introduce the application of interactive segmentation of
3D anatomical structures from CT data. The CT data is in
high-resolution and high signal-to-noise ratio (SNR)
compared to PET, and can be segmented more accurately
which can provide anatomical information to the PET
data. We propose to interactively segment the volume
rendered CT data in real-time using intensity-based
thresholding which has been found to be a simple yet
effective technique in segmenting certain structures in CT
images [9-10]. The ability to interactively segment the CT
data for the extraction of structures of interest, in addition
to the volume rendering visualization of dual modality
PET-CT in real-time, the proposed technique provides
new approaches to visualization for medical diagnosis and
interpretation. We demonstrate our method using sub-
sections of the human lung and liver from a set of clinical
whole-body PET-CT imaging data.
II. METHODS
The flowchart of the proposed real-time volume
rendering visualization technique with interactive
segmentation is shown in Fig. 1, exemplified with the
whole-body PET-CT data.
A. Overview of PET-CT Visualization and Manipulations
In the proposed visualization technique, PET and CT
data can be volume rendered individually or fused
together with fusion ratio adjustments in real-time. The
rendered volume can be interactively navigated using the
combination of rotation, scaling, and translation. In order
to increase the responsiveness of the volume rendering to
user movements, the sampling rate, measured as the
number of samples used for volume rendering from a
voxel in the data set, can be sampled at lower rates. A
utility of our visualization technique is the intensity-based
thresholding of the CT data segmentation in real-time
volume rendering. Thresholding of the CT can potentially
permit the segmentation of the anatomical structures.
643
Another utility is volume ‘clipping’, which removes the
portion of the volume that would have obscured the user’s
view by clipping the volume. Clipping can be applied by
using a plane which cuts through the volume
perpendicular to the viewing window, or through a
clipping box which encapsulates the volume. The position
of the box can be changed by translation where only the
portion of the volume residing in the box is left visible.
Volume manipulations included are conventional window-
level (brightness and contrast) and color LUT which can
be controlled by transfer function. All the interactive
control is performed using the combination of the mouse
and the keyboard.
Dual-modality PET-CT data
PET data CT data
Volume Volume
rendering rendering
Volume Volume
rendition rendition
Image fusion and volume manipulation tools Thresholding
segmentation
PET-CT visualization with real-
time volume manipulations
Fig. 1. Flowchart of the dual-modality PET-CT visualization using real-
time fusion and volume rendering, together with the interactive CT
segmentation.
B. Real-Time Volume Rendering Implementation
A prototype of the PET-CT visualization technique has
been developed using the SGI OpenGL Volumizer
application programming interface (API) [11]. OpenGL
Volumizer is a C++ programming library that provides
implementation of the visualization algorithms optimally
designed for volume rendering by utilizing the high level
of texture-mapping capabilities in commodity graphics
card. It also provides the flexibility in direct interfacing
with the graphics hardware. Texture-mapping volume
rendering technique refers to creating parallel planes
through the column of the 3D data, in the principal
direction most perpendicular to the viewer’s line of site.
The planes are then drawn back to front with appropriate
3D texture co-ordinates. To simultaneously render two
separate volumes, fusion of the volume is required which
preserves the depth information and image characteristics
of different imaging modalities. We utilize the hardware-
based per-voxel fusion [12] method which uses a single
ray-cast to perform the fusion calculation during the
rendering of each voxel. Such approach has the advantage
of producing interactive volume rendering performance.
The intensity-based thresholding is performed by
interactively adjusting the opacity function of the LUT.
Other volume manipulations are all implemented based on
the Volumizer’s API.
III. RESULTS
The clinical whole-body PET-CT data were acquired by
a SIEMENS Biograph LSO scanner. The resolution of the
images are at 16bit and number of imaging slices for PET
and CT images are 263 and 262, respectively. The image
size of the PET data are 128 × 128 in size with voxel
dimensions of 5.148 × 5.148 × 3.38mm, and CT data are
512 × 512 in size with voxel dimensions of 0.977 × 0.977
× 3.40mm. Both data are cropped and re-scaled to the
same voxel size of 256 × 256 × 263 with voxel dimensions
of 1.953 × 1.953mm.
(a) (b)
(c) (d)
(d)
Fig. 2. Real-time volume rendering and fusion of PET-CT data: (a) PET;
(b) CT; (c) fused PET-CT; and (d) fused PET-CT with CT segmentation
of the lung. The volumes has been rotated and clipped.
The volume renditions of the PET-CT data are
presented in Fig 2. For the purpose of volume rendering
efficiency, 40 trans-axial slices (slice #60 to #100 out of
263), corresponding to the lung from the whole-body
PET-CT data were selected. The PET and CT data were
volume rendered using the conventional ‘hot-metal’ and
‘grayscale’ LUT, respectively. The bent oval shaped
object appearing on the CT is the ‘bed’ used in the scanner
and not a part of the human body. In Fig. 2(a), volume
rendition of the lung in PET data is illustrated and Fig.
2(b) shows the corresponding CT data. These two
modalities are fused together (equal fusion ratio) in Fig.
2(c) with the window level adjusted result to the CT to
reveal greater PET details. The segmentation of the CT to
extract out the lung boundary is shown in Fig. 2(d). In this
example, the design of the proposed PET-CT visualization
644
technique demonstrates potential aid in the interpretation
of the PET data with clear selection of the lung boundary
from the CT segment. Using PET data alone, it is unlikely
that the extraction of structures can be accurately
performed. As the CT segmentation is applied in real-
time, the physician is able to interactive adjust the
threshold while exploring through the 3D data. As the two
data are rendered independently, the user can rapidly
interchange between the PET, CT, and fused PET-CT,
which is a commonly practiced routine in clinical PET-CT
diagnosis.
Fig. 3 further illustrates the proposed PET-CT volume
rendering and volume manipulations. In Fig. 3(a) the CT
data has been segmented to reveal the bone structures
surrounding the functional organs apparent in PET data,
from the lower chest section of the human body. The
volume has been clipped to show the internal structures.
The window level of CT from Fig 3(a) has been adjusted
to better visualize the PET data in Fig. 3(b). In Fig. 3(c), a
thin skin layer from the CT data has been thresholded
which also reveals the air-tree within the lung. Equal
fusion ratios were set to the PET and CT data in Fig 3(a)
to (c). Finally, Fig. 9(d) shows the clipped PET-CT data
using the clipping box, revealing internal structures of the
fused PET-CT data with the fusion ratio set to 70% PET
and 30% CT, in order to to better visualize the functional
structures.
(a) (b)
(c) (d)
Fig. 3. CT thresholding and volume manipulations applied to the fused
PET-CT data.
A major goal of the proposed visualization technique
was to provide real-time volume rendering and interactive
manipulations. In order to test the frame rates for various
manipulations a user can affect on the volume rendering
process, the time elapsed between frames during user
interaction was recorded in frames per second (FPS). The
FPS from manipulations shown in some of the PET-CT
figures from this paper is measured, running on an ATI
Radeon 9600 graphic card with 64Mb memories on a
Windows XP platform. Based on these measures, our
visualization technique can animate the volumes with
good response times of 4~15 FPS among all the
manipulations presented in this study prior to optimization
in our prototype implementation. The lowest FPS is from
the interactive segmentation of CT data, which requires
most computation, with only 4 FPS. For greater
performance, frame rates can be significantly improved by
lowering the voxel resolution and the viewing window
size.
IV. CONCLUSIONS AND DISCUSSIONS
This paper described a novel technique and a prototype
implementation for the visualization of the dual-modality
PET-CT data. The main contributions of this study are: (1)
Real-time volume rendering and fusion of the dual-
modality PET-CT data; (2) Interactive segmentation of
volume rendered CT data; and (3) Volume manipulations
in real-time. These proposed techniques can potentially
benefit the visualization of the PET-CT data in diagnosis
and interpretation. Overall, the proposed technique was
shown to permit interactive visualization of dual-modality
PET-CT data using real-time volume rendition in a variety
of potentially useful ways. We believe that our proposed
technique is able to complement the 2D visualization
approaches commonly applied in practice.
In our current hardware and software configuration, the
performance of the proposed PET-CT visualization
technique was unable to render entire whole-body PET-
CT images in high sampling rate with responsive frame
rates. Although the performance may be substantially
improved by lowering the sampling rates, the need for
interactive manipulations to be applied in high sampling
rate must be addressed. In this study, we therefore provide
the ability for the user to select slices corresponding to the
sub-section of the human body from a conventional 2D
PET-CT viewer. Nevertheless, we will investigate the
optimization of the proposed technique and also the
utilization of more powerful graphics card currently
available which may address this limitation.
The segmentation of the CT data has been demonstrated
in this study to be a useful tool in interpreting the dual-
modality PET-CT data, with examples illustrated in this
study on how these modalities can complement each
other. Further studies will investigate the integration of
different segmentation techniques, applied not only to the
CT data, but applied to segment both the anatomical and
functional structures for PET-CT visualization.
ACKNOWLEDGEMENT
This study was supported by the ARC, HKRGC, and
CMSP Grants. We like to thank the staff at the Hong
645
Kong Sanitarium Hospital (HKSP) for providing the
image data sets used in this study.
REFERENCES
[1] O. Ratib, “PET/CT image navigation and communication”,
J. Nucl. Med., vol. 45, pp. 46S-55S, 2004.
[2] D.W. Townsend, J.P.J. Carney, J.T. Yap, and N.C. Hall,
“PET/CT today and tomorrow”, J. Nucl. Med., vol. 45, pp.
4S-14S, 2004.
[3] R. Shahidi, R. Tombropoulos, and R.P. Grzeszczuk,
“Clinical applications of three-dimensional rendering of
medical data sets”, Proc. IEEE, vol. 86, pp. 555-565, 1998.
[4] D. Gering, A. Nabavi, R. Kikinis,W. Grimson, N. Hata, P.
Everett, F. Jolesz, and W. Wells, “An integrated
visualization system for surgical planning and guidance
using image fusion and interventional imaging,” Proc.
MICCAI Int. Conf. Med. Image Computing and Computer
Assisted Intervention, Springer Verlag, pp. 809–819, 1999.
[5] A. Rosset, L. Spadola, and O. Ratib, “OsiriX: an open-
source software for navigating in multidimensional
DICOM images”, J. Digital Imag., vol. 17, pp. 205-216,
2004.
[6] F. Beltrame, G. De Leo, M. Fato, F. Masulli, and A.
Schenone, “A three-dimensional visualization and
navigation tool for diagnostic and surgical planning
applications”, Proc. SPIE Visualization, Display, and
Image-Guided Procedures, vol. 4319, pp. 507-514, 2001.
[7] H. Hauser, L. Mroz, G.I. Bischi, and M.E. Gröller”, Two-
level volume rendering”, IEEE Trans. Visualization and
Computer Graphics, vol. 7, pp. 242-252, 2001.
[8] E. Bullitt and S.R. Aylward, “Volume rendering of
segmented image objects”, IEEE Trans. Med. Imag., vol.
21, pp. 998-1002, 2002.
[9] S. Hu, E.A. Hoffman, and J.M. Reinhardt, “Automatic lung
segmentation for accurate quantification of volumetric X-
ray CT images”, IEEE Trans. Med. Imag., vol. 20, pp. 490-
498, 2001.
[10] R. Wiemker and A. Zwartkruis, “Optimal thresholding for
3D segmentation of pulmonary nodules in high resolution
CT” International Congress Series, vol. 1230, pp. 653-658,
2001.
[11] P. Bhaniramka and Y. Demange, “OpenGL volumizer: a
toolkit for high quality volume rendering of large data sets,
“IEEE / ACM SIGGRAPH Symp. Volume Visualization and
Graphics”, pp. 45-53, 2002.
[12] ATI Research, "ARB fragment program specification,"
2002. URL: http://oss.sgi.com/projects/ogl-sample/registry/
ARB/ fragment_program.txt [accessed July 2005]