Professional Documents
Culture Documents
Computer Graphics in Medicine: From Visualization To Surgery Simulation
Computer Graphics in Medicine: From Visualization To Surgery Simulation
Figere I'A JAVAapplet for distributedcompassiondomain volumerendering F'~nsre2" User fr~rface of ~e telecaopera~onsystemKANIEDIN.Image
(htq)'/Iwww.inf.elhz.ch/personal/lipperrJEVOLVF_).Seepage I01 ~r/maLe in )CuB providedcourtesyar the CarnputerGreph~ Center,D a ~ Germany.
co/or. See page I01 for Imagein lull color.
Computer Graphics February1998 S3
ity systems for interactive journeys through Besides high-end CT and MR data sets, developed models to simulate the physics of
the human body. Highly immersive interfaces graphics researchers are discovering the lower the human body. Presently, one of the most
generate gorgeous illustrations and are cur- end of medical imaging systems. Here, ultrason- challenging and attractive subfields here is the
rently being investigated as enabling tools for ic devices dominate the scene and various fast and accurate computation of the mechan-
the next generation of medical training and approaches for rendering and reconstruction of ical behavior of soft tissue, which is an essen-
education systems. In some other application ultrasonic data sets can be found in contempo- tial prerequisite for the vast and mostly
scenarios, VR systems are envisioned to rary literature. Although noise and alignment unexplored area of virtual and simulated
replace costly and dangerous invasive meth- problems make a robust segmentation and fea- surgery. Although we are still far from a
ods.Yet, in most cases the lack of force feed- ture extraction much more difficult, it is mature computational model, some existing
back turns out to be a critical issue because it expected that some of those methods will be soft-tissue models perform surprisingly well
seriously diminishes the degree of realism. part of next generation's ultrasonic systems. and therefore can be considered a first step
In times where medical information sys- Other research groups design high-end towards more sophisticated approaches. An
tems and databases are commonplace in most telepresence systems, which enable the sur- example from facial surgery simulation is
hospitals, networked compression, visualiza- geon to perform telesurgery using haptic and given in figure 3, where pre- and postsurgical
tion and analysis methods gain more and robotics interfaces. Additionally, augmented facial shapes of a case study are presented for
more importance. Typical requirements reality systems have been used for interactive a lower jaw bone repositioning. In this partic-
encompass fast searching and browsing of diagnosis and medical check-ups. Images taken ular application, highly realistic images of the
large CT or MR databases as well as scalabili- from a camera on the physician's head are post-surgical appearance of the patient's face
ty to the network performance and computa- superimposed with coincidentally recorded are of enormous importance to relax
tional power of the clients. As an example, ultrasonic scans and are composed in a head- patient's fear.
Figure I presents a JAVA applet for rendering mounted display. Real-time projections of 3D
remotely located volume data sets instanta- reconstruction of individual anatomic features The Future:Virtual
neously from a highly compressed file format. allow the location and visual inspection of the Operation and Surgery
Another vast subfield of graphics in medi- inner part of the human body.
cine is collaborative diagnosis and telepres- Furthermore, 3D hardcopy devices, such as
Simulation
Undoubtedly, future key applications of graph-
ence. For instance in radiology, physicians stereolithographs, have been widely and suc-
ics in the medical field include operation plan-
often face the problem of having instanta- cessfully used in medicine. Current research
ning and surgery simulation. Here, we foresee
neous access to a specialist who is located activities include the automatic computation
fully immersive real-time simulation environ-
somewhere remote. Whereas in the past and reproduction of prostheses or missing
ments in which surgeons can learn, plan and
images were exchanged as hardcopies via sur- bones, based on geometric reconstruction. In
rehearse complex operations using individual
face mail, upcoming high-speed digital net- particular, the symmetry of the human anato-
patient models and being supported by
w o r k s open the d o o r for real-time my is exploited to compute missing pieces,
sophisticated input devices, such as virtual
distributed and collaborative diagnosis. which are then reproduced by milling
scalpels. The design of these types of simula-
Medical telecooperation projects and systems machines and implanted by surgeons.
tion environments is a highly interdisciplinary
are currently supported by various telecom- Additionally, 3D hardcopies help the study of
project which requires input from a variety of
munications companies, and in some cases anatomic deviations or malformations in the
disciplines including graphics, vision, robotics,
they are already available as commercial prod- approaches of complex surgical procedures,
numeric analysis, mechanics and material sci-
ucts. Figure 2 shows a user interface layout of such as in facial surgery.
ences, hardware design and, needless to say,
a typical telecooperation system for radiolo- Besides mere geometric reconstruction of
medicine. However, unlike any other disci-
gists. anatomic structures, graphics researchers
pline, computer graphics will be paramount,
Figure 3: Physics-based modeling of facial s0~ t/ssue:a) Presurgical facial shape profile, b) Postsurgical profile, c) Postsurgical frontal view.Data source:Visible Human Project,Courtesy
National Library of Medicine. See page 101 for image in full co/or.
54 February1998Computer Graphics
since it will take the role of the system archi- T h e H u m a n C o m p u t e r Interface Besides mathematical models, appropriate
tect. The design of a sophisticated natural human material parameter databases are of great
T h e c o n c e p t u a l c o m p o n e n t s of an computer interface is certainly one of the importance in surgery simulation. Moduli of
advanced surgery simulation environment can keys to successful simulation.The illusion of a elasticity or non-linearity are usually functions
be summarized as follows: In a first step, raw full medical working environment can only be of age, sex, e t h n i c g r o u p and o t h e r s .
data sets have co be acquired by highly accu- created with the help of advanced virtual Unfortunately, it is e x t r e m e l y difficult to
rate 3D medical imaging o r vision systems. reality hardware. C o n t e m p o r a r y o u t p u t obtain the desired parameters and extensive
Subsequent preprocessing steps e x t r a c t devices, such as head-mounted displays, experimental research has to be pursued.
anatomic substructures and create geometric C a v e s , V i s d o m e s and Workbenches are Here, we need the help of material scientists
models of the patient au~ributed with respec- promising tools to study the behavior and co provide us with appropriate sets of para-
tive material parameters, stemming from an acceptance of medical users. However, we meters and interpolation models For individ-
appropriate material database.A sophisticated foresee smart, small, lightweight and very high ual patients and tissue types.
modeler allows the surgeon to modify the resolution eyeglass displays as the ultimate The strategies for optimal solutions of par-
geometry and the topology of individual parts devices for mediating visual information to tial differential equations are critically depen-
of the derived model while simulating cuts, the surgeon.We believe that future display dent on the complexity of the underlying
bone repositioning, transplants etc. Force technology will provide a new generation of mechanical model. In particular, in order to
feedback as a function o f the underlying sophisticated solutions. achieve real-time response we have co bal-
material has to be computed and interpolated Much more important and fundamental For ance mathematical accuracy versus computa-
to meet the high update rates which are nec- any surgeon than the display itself is the pro- t i o n a l e f f o r t . N u m e r i c analysts have
essary to beat the temporal resolution of the vision of highly accurate tactile and force developed various classes of solvers, where
tactile sensory channel. Tissue volume forces feedback information. Therefore, we require FEM is only one prominent example. Upon
and deformation fields as well as collision highly accurate haptic interfaces, which cap- simplifying the physics, appropriate solution
detection must be computed in real time, t u r e the responses o f human tissue t o strategies have to be designed in close coop-
since they convey the parameters For visual mechanical stimuli. One of the major prob- eration with numeric analysts. Here, hierarchy,
and force feedback. In essence, Fast approxi- lems, however, is that the haptic device must progression and localization might be some of
mations of the underlying differential equa- be t h o r o u g h l y tailored to the underlying the key words to success.
tions have to be Found. Although for some application. In facial surgery simulation, for
applications, such as facial surgery simulation, instance, completely different setups are T h e P a r ~ n o u n t Role o f C o m p u t e r
more expensive and accurate solution strate- required than in laparoscopy. Moreover, com- Graphics
gies could compute the exact deformation plex surgical procedures usually e m p l o y The particular attractiveness of highly com-
fields in batch mode, the design of appropri- extensive sets of individual mechanical tools. plex surgery simulation environments for us
ate real-time engines remains the most chal- Although various haptic interfaces are avail- lies in the versatility and depth of individual
lenging part of the simulation. As an option, able in research labs or on the market and research problems touching o r covering
appropriate rendering algorithms might gen- enable first experimental steps into the right almost every important subfield of computer
erate photorealistic still images of the patient. direction, we are still far from convenient graphics. Therefore, we finally illuminate the
solutions. All in all, the design of individual paramount role of graphics in surgery simula-
D a t a A c q u i s i t i o n and Analysis force feedback devices is strongly influenced tion and briefly sketch some of the major
Since CT and MR scanners still have limited by the application context and by the under- research challenges:
resolution, highly accurate 3D surface data lying physics. OptimJ7.arion to users' demands • Modeling of geometry and topolo~. The basis
acquisition is fundamental For some applica- requires multiple iterations and design cycles. of any sophisticated surgery simulation sys-
tion scenarios. In this context, fast and robust Therefore, successful solutions can only be tem is the efficient mathematical description
3D range finding methods are necessary co d e v e l o p e d in a t i g h t c o l l a b o r a t i o n of of the underlying geometry and topology.
satisfy the resolution constraints imposed by researchers from medicine, robotics and Here, most of the model has co be described
the simulation environment. Furthermore, computer graphics. in a full volumetric settin~ Sophisti,~ted dis-
color and texture samples must be recorded cretJzations and approximation models are
co equip subsequent rendering methods. Mechanics and N u m e r i c Analysis essential preprocessing steps For advanced
Especially when using both volume and sur- The differential equations governing any volu- numeric procedures, which compute tissue
Face data, manifold registration and alignment metric tissue deformation have their roots in deformation. Conformity and polynomial
problems arise, including surface-surface, sur- mechanics. In the past, linear volumetric strain degree of individual elements strongly affect
face-volume and volume-volume registration. and deFormation models have been extensive- the complexity and accuracy of numeric
Robust (semi)automatic methods are desir- ly investigated and are widely available for solutions. Therefore we have to adapt our
able. various kinds o f m a t e r i a l simulation. In models to subsequent numerical strategies
Since the early days of medical imaging, seg- surgery simulation, however, large strains and where piecewise linear approximations are
mentation and feature extraction have lost deformations occur and the material tensors mostly insufficient.As a consequence, our
n o n e o f t h e i r i m p o r t a n c e , and despite become non-constant_The resulting phenom- challenge is to develop new generations of
decades of research, there is no robust and ena are highly nonlinear and extremely diffi- higher order volumetric approximations,
fully automatic method in sight- For matching cult t o m o d e l . O t h e r effects relate to which do not impose unnecessary topologi-
problems, semiautomatic strategies might be compressibility of human soft tissue.The high cal restrictions on our model.
an appropriate alternative and the collabora- percentage of liquid makes it almost incom-
tive effort of computer graphics and comput- pressible; local tissue forces generated during • Ed/ting of complex structures:With appropri-
er vision researchers is necessary t o surgery force liquid to stream out and to dis- ate approximation methods in place, anoth-
accomplish this goal. t o r t the mechanical behavior of the material. er task is to design p o w e r f u l e d i t o r s
allowing the user to modify the geometry
ComputerGraphlcs February1998 SS
and topology of anatomic structures. Again • Image generation: Generally, fast rendering
algorithms are irreplaceable in surgery sim-
Summary
mere surface editing is certainly not suffi- In summary, from the earliest days, the med-
cient for cutting and repositioning of indi- ulation. However, w i t h the design of ical field has been one of the most appealing
vidual pieces of soft tissue. Functions such advanced volumetric approximations new areas for computer graphics research. Being
as a zoom into substructures are highly types of possibly unstructured graphics the chief engineer of next generation's med-
desirable and enable a surgeon to operate primitives will enter the scene and appro- ical training and simulation systems, computer
on different scales in the full volumetric priate hardware support will be highly graphics will help to improve our present
setting. Therefore, elaborate data struc- desirable. Thus both surface and volume health system.Thus it will provide a significant
tures have to be designed giving efficient rendering of higher order primitives will be contribution to our society -- and to modern
access to, and maintaining individual primi- of great interest.The required visual quality, human life.
tives of the underlying approximation. however, depends highly on the application
Furthermore, these data types will have to context. The mostly lubricant surfaces of Markus H. Gross is full Professor at the
maintain all necessary parameters of the the interior of human bodies, for instance, Computer Science Department of the Swiss
physics-based models, which in turn must are highly specular and will ask for hard- Federal Institute of Technology (ETH) in
interact with the editor in real time. ware support for more sophisticated shad- Z~irich. He received a degree in Electrical and
ing models. F u r t h e r m o r e , bump, Computer Engineering in 1986 and a Ph.D. in
• Simplification and adaptation of physical mod- displacement and texture databases or Computer Graphics and Image Analysis in
els: One of our core challenges in surgery procedural models for human skin and soft 1989, both from the U n i v e r s i t y of
simulation is to understand and simplify the tissue have to be created. Here, multipass Saarbr~icken, Germany. From 1990 to 1994,
physics of the simulation process.The quest methods might be a promising approach. he was with the Computer Graphics Center
for real-time update rates in visual and Apart from the real-time constraint, some in Darmstadt, where he established and
mechanical feedback forces us to develop applications require extremely sophisticat- directed the Visual Computing Group. His
sophisticated solutions, which have to bal- ed illumination models. In facial surgery current research interests include physics
ance computational complexity with accu- simulation, for instance, the rendering of based and multiresolution methods for graph-
racy. To this end, we have to thoroughly photorealistically looking images might be ics with applications in the medical field.
investigate and analyse the governing equa- computed off-line by raytracing of the high-
tions and to study their error bounds. er order primitives envisioned earlier. Here Markus H.Gross
Based on a profound knowledge and we need advanced reflection and scattering Department of Computer Science
observation of the physical phenomena, models for facial skin, which will also vary Swiss Federal Insitute of Technology (ETH)
new approximate solutions must be as a function of age, sex and other parame- CH 8092 Zurich, Switzerland
designed. Many different strategies and ters. In addition, high quality rendering of Tel: +41-1-632-7114
parameters have to be investigated and human hair is still a widely open issue. Fax:+41 -1-632- I 172
evaluated against each other, such as finite Email:grossm@inf.ethz.ch
element versus finite difference, flat versus • System Design and Layout: Computer graph- httpdlwww.inf.ethz.chldepartrnent/ISIcgl
hierarchy, local versus global or polynomial ics researchers will undoubtedly be the
degree versus spatial resolution. Required principal designers of future surgery simu-
update rates for visual and force feedback lation systems. Thus, besides the mere
rendering might be decoupled from the development, tuning and tailoring of indi-
simulation engine and appropriate tempo- vidual algorithms graphics research will also
ral i n t e r p o l a t i o n could compute the have to cover overall systems layout and
desired in-betweens. In addition we will optimisation. In particular, global design
have to consider next generations graphics issues, such as the interactions of editor,
hardware to accelerate the envisioned rendering, simulation engine and force
algorithms. feedback, are of fundamental importance in
highly interconnected, complex systems
and must also be considered by graphics
researchers.
56 February1998Computer Graphics
,. ...
FROM THE ED IT O R