2008 IEEE Nuclear Science Symposium Conference Record
Image Synthesis of a Mini Gamma Camera
and Stereo Optical Cameras
Hideaki Haneishi, Hiroshi Shimura and Hideki Hayashi
Abstract-- We have previously developed a method that
combines a mini gamma camera with an optical camera to
synthesize the two kinds of images and help surgeons to easily
identify sentinel lymph nodes on the gastric surface in the
cancer resection surgery. Since it is assumed in this method that
the object surface is flat, the application is limited. In this article,
we extend this method so as to estimate the three-dimensional
shape of the object from stereo optical cameras. A prototype
system with a mini gamma camera and two optical cameras was
constructed and a preliminary experiment was performed.
I. INTRODUCTION
In detection of sentinel lymph nodes in a human body or
in the observation of metabolism in a small animal, proper
radioisotope is injected to the body as a tracer and its uptake
is observed. For observation of the radioactive distribution,
gamma probes have been used. However, radioactive regions
can only be detected by radioactive counts in a pin-point
manner. Therefore an operator has to search a wide area of
the gastric surface. It is thus a time-consuming, less-
quantitative and inaccurate operation. Recently, a mini
gamma camera (MGC) has been made available [1] and is
tried to use for sentinel lymph node detection in
gastrointestinal surgery [2]. Since this camera is small
enough and allows two-dimensional imaging of the tracer
distribution, the use of this camera instead of conventional
gamma probes enables quick and accurate identification of
radioactive regions.
However, currently it is difficult to relate the output image
of the MGC to the real surface of the target. After collecting
gamma rays from the operational field, operators remove the
camera and try to establish a correspondence between the
radioactive area and the visible target surface. After removing
the camera, however, this is often difficult.
In order to overcome this problem, we have developed a
method that combines a GC with an optical camera (OC) [3].
It is expected that by imaging the target area with GC and OC
and synthesizing them, operators can easily identify the
radioactive area on the real surface. The previously
developed method assumed that the surface of the target is
flat and parallel to the GC head plane and measured the
distance between the target surface and the GC head using a
laser.
Manuscript received November 15, 2008.
Hideaki Haneishi is with the Research Center for Frontier Medical
Engineering, Chiba University, Chiba, 263-8522, Japan. He is also with
Graduate School of Science and Technology, Chiba University, Chiba,
263-8522, Japan (e-mail: haneishi@faculty.chiba-u.jp).
Hiroshi Shimura is with Graduate School of Advanced Integration
Science, Chiba University. Hideki Hayashi is with the Research Center for
Frontier Medical Engineering, Chiba University.
978-1-4244-2715-4/08/$25.00 ©2008 IEEE
M10-484
However, there are some cases that this assumption does
not hold. Thus, in this study, we use stereo optical cameras.
From two OCs, the 3D shape of the target surface is
estimated, then synthesis is successfully performed. A
prototype system was constructed and a preliminary
experiment was performed.
II. METHODS
A. Outline of the method
As a prototype setup, two small CCD cameras with wide
viewing angle are attached at the edge of the MGC head. The
schematic illustration of the setup is shown in Fig. 1. The
obtained two-dimensional GC image represents the
radioactivity distribution integrated along the direction of the
parallel collimator of the GC.
Optical camera Ub
image
Gamma
ua vb
camera image
(ub,vb)
va (x,y)
(ua,va) x X
Y y
lens
Z
Fig. 1 Schematic illustration of the prototype setup for obtaining the gamma
camera image and the optical camera images.
On the other hand, the images captured by the optical
cameras are distorted by the wide angle lens and is a
perspective one. This distortion must be corrected first. After
the distortion correction, a point corresponding to each pixel
in one OC image is found in the other OC image. This
process is done for all pixels in a target area of OC images.
From the image coordinates of those corresponding points,
the world coordinates of those points are calculated. This
means that we have a three-dimensional shape of the object.
Finally, the radioactivity image is mapped on either OC
image.
The above mentioned processing flow is illustrated in Fig. 2.
Before routine operation, the system has to be calibrated in
advance. Two kinds of calibration are needed: one for
distortion correction of CCD cameras and the other for
registration of OC image and GC image.
5373
  Ca11 Ca 31u1 Ca12 Ca 32u1 Ca13 Ca 33u1 ¯
Calibration ¡ °  X ¯
¡ Ca12 Ca 32v1 Ca13 Ca 33v1 °° ¡ °
¡Ca11 Ca 31v1 ¡Y °
¡C C u Cb12 Cb 32u 2 Cb13 Cb 33u 2 °° ¡¡ °°
¡ b11 b 31 2
¡C C v ¡Z °
Cb12 Cb 32v 2 Cb13 Cb 33v 2 °°± ¢ ± (3)
Optical camera Optical camera Gamma camera
image A Image B image ¡¢ b11 b 31 2
(ua,va) (ub,vb) (x,y)  u1 C a14 ¯
¡ °
¡v1 C a 24 °
¡
¡ °
Distortion correction u C °
¡ 2 b 14 °
¡v C °
¡¢ 2 b 24 °±
Estimation of 3D shape Here we denote the matrix in the left hand side by A to
(ua,va) simply represent eq. (3) as
(X,Y,Z)
(ub,vb)
 u C a14 ¯
 X ¯ ¡ 1 °
¡ ° ¡v1 C a 24 °
Image synthesis ¡ ° ¡
A ¡Y °  ¡ ° . (4)
u C °
¡Z ° ¡ 2 b 14 °
Fig. 2 Processing flow to obtain synthesized images. ¢¡ ±° ¡v C °
¢¡ 2 b 24 ±°
B. Mathematical description of synthesis
This is an overdetermined problem and thus the world
by a conventional method using an object with known coordinate is given by solving the least square method [5] as
shape such as a cube. Then, a pair of the corresponding points  u1 C a14 ¯
of the object gives a 3D position of the world coordinate.  X ¯ ¡ °
¡ ° 1 ¡v1 C a 24 °
the pin-hole camera model can be used to represent the
mapping from the object space to image plane.

¡Y °  At ¸ A At
¡ °
¡
¡u C ° . (5)
°
¡Z ° ¡ 2 b14 °
¢¡ ±° ¡v C °
First, the coordinates of the GC and OC imaging are defined ¢¡ 2 b 24 ±°

as shown in Fig. 1. We call two optical cameras OC-A and
OC-B. Here the world coordinates representing the object
space is defined as (X,Y,Z). The GC image coordinates are
also defined by (x,y) where x is the same as X and y is the
same as Y. Similarly, two OC image coordinates are defined
as (ua,va) and (ua,va). We also represent a gamma camera
image by g (x , y ) and OC images by oa (ua , va ) and
ob (ub , vb ) . In what follows we first describe how to relate the
GC image coordinates (x,y) to the OC image coordinates (u,v).
On the other hand, in the imaging with the GC, due to the
properties of the collimator, the relationship between an
object point (X,Y,Z) and its image coordinate (x,y) can simply
be expressed by the following equations,
x=X
(6)
y =Y
Image synthesis is performed as
For matrix-based handling, homogenous coordinates are oˆA (ua , va )  oA (ua , va )  w ¸ g (x (ua , va ), y (ua , va )) ‫ޓ‬
(7)
often used. The object point with the world coordinate Here, x (ua , va ), y (ua , va ) ‫ޓ‬
means that these x,y coordinates
(X ,Y , Z ) is expressed by the homogeneous are corresponding to (ua , va ) ‫ޓ‬
. w is a proper weighting
coordinate (kX , kY , kZ , k ) [4]. Here k is a nonzero arbitrary factor. Way of image synthesis can be more flexible. In fact,
constant. Using the homogeneous coordinate, the perspective in the experiment in this paper, we first binarize the GC
transformation of the OC-A is expressed by the following image and then overlay on a OC image.
equation. C. Calibration technique
 X ¯
 kua ¯  Ca11 Ca12 Ca13 Ca14 ¯ ¡ °
¡ ° ¡ ° ¡Y ° The distortion of the OC is the camera’s inner characteristic.
¡kva °  ¡Ca 21 Ca 22 Ca 23 Ca 24 ° ¡ ° (2)
¡ ° ¡ ° ¡Z ° Parameters for this calibration are obtained by capturing a
¡ k ° ¡C C C C °¡ °
¢¡ ±° ¢¡ a 31 a 32 a 33 a 34 ±° ¡ ° pattern of straight lines and determining the camera
¢¡ 1 ±°
parameters in order to straighten the lines curved by the
The perspective transformation of the OC-B is also expressed
distortion [6]. Figure 3 shows the rectangular grid pattern
by changing the first suffix a to b in the above equation.
used in the experiment. In the left, a barrel distortion is
Entries of the matrix, {Cijk} are determined at the calibration
observed. The calibration is performed so that grid points are
step.
straightly aligned.
From two image coordinates of the corresponding points
sharing the same world coordinate (X,Y,Z), the following
equation is derived.

5374
 GC head

Two OCs
Before distortion correction After distortion correction
Fig. 3 Grid pattern for calibration of distortion correction. Fig. 5 Photo of the prototype setup.

Calibration of registration of OC image and GC image is RI patch

performed using a calibration frame. A setup for calibration is
schematically shown in Fig. 4. The frame consists of thin
woody rods and forms. Patches soaked by radioactive liquid
are put on nine positions whose relative coordinates from the
bottom of the central rod are known.
 The calibration frame is set just under the GC so that the
axis of central rod runs through the origin of the GC image (a) Calibration frame
coordinate. This adjustment can easily be done by seeing the
va vb
GC image real time. After the setting of the frame, OC
images are captured and those patches are identified manually.

Optical camera ub
image
Gamma ua ub
ua vb camera image
(x,y)

va (ub,v b) oa(ua,v a) ob(ub,v b)

y x X
(ua,v a) (b) OC images of the calibration frame
Y
Fig. 6 Illustration of calibration frame and the captured OC images.
lens Z
Table 1 Specification of detector
Effective FOV 44.6 চ x 44.6 চ
Pixel size 1.4 চ x 1.4 চ
Number of pixel 1024 pixels
(32 x 32 matrix)
Fig. 4 Processing flow to obtain synthesized images. Detector unit
Energy range 60㨪300 keV
Number of energy 1
III. PRELIMINARY EXPERIMENT AND RESULTS window
Energy resolution > 6%
Fig. 5 shows a picture of the setup including MGC and two
OCs. The MGC (MGC1500, Acrorad, Japan) utilizes Material Tungsten
semiconductor detectors, CdTe. The effective field of view Collimator Hole size 1.2 চ x 1.2 চ
the camera is 44.6mm x 44.6mm and is sampled by 32x32 Hole pitch 1.4 চ
channels. The pixel pitch is 1.4mm. Details of the Size 82 চ x 82 চ x 205 চ
specification of the MGC are listed in Table 1. The optical Whole (W x D x H)
camera (Cyber eye, Japan) is a color CCD camera with about Detector
Weight 1.4 kg
410 thousand pixels and its viewing angle is about 120 degree.
Figure 6 shows an illustration of calibration frame and the As a preliminary experiment, a machinemade solid object
captured OC images. with a wavy surface was prepared. Three red paper patches
soaked with 99mTc were put on the surface: two are on
convexes; one is on a concave of the surface. An image of
this object taken by the right OC is shown in Fig. 7(a). Note
that this image is the one after distortion correction. Fig. 7(b)
shows an image taken by the GC. Fig. 7(c) shows the

5375
binarized pattern obtained from the original GC image by
using a proper threshold. Fig. 7(d) is a distance map
estimated from the two OC images in which the distance
from the camera head is represented by the gray level. A
periodic pattern of a wavy shape is nicely estimated. Some
discontinuous parts are due to the occlusion problem in the
stereo imaging.
Fig. 7(f) shows the synthesis image by the proposed
method. Here light-green pattern represent the radioactive
regions shown in Fig. 7(c). These three patterns are
successfully synthesized on the red RI patches. For
comparison, the image synthesis by our previous method
using only one OC was also performed, in which we assumes
that the object surface is flat. Fig. 7(e) shows the result. It can
be seen that two patches on convex succeed in matching but
one patch on the concave fails.
Journal of Nuclear Medicine and Molecular Imaging Vol. 30, No. 12,
December 2003
[2] H. Hayashi, D. Horibe, T. Uehara, K. Takagi, T. Kanei, Y. Arano, H.
Tatsuoka and T. Ochiai, "Sentinel lymph node detection using an
imaging probe with CdTe semiconductor detector in gastrointestinal
surgery,"4th International Sentinel Node Congress (Los Angeles), pp.
57. 2004.
[3] Haneishi, H.; Onishi, Y.; Shimura, H.; Hayashi, H.: Simultaneous
Acquisition and Image Synthesis of Gamma Cameras and Optical
Cameras for Sentinel Lymph Node Identification During Radioguided
Surgery, IEEE Transactions on Nuclear Science, Vol. 54 Issue 5, pp.
1703 - 1709 (2007.10)
[4] R.C. Gonzalez and R.E. Woods, Digital Image Processing.
1993.Addison-Wesley Publishing Company, Massachusetts.
[5] C.L. Lawson and R.J. Hanson, Solving least squares problems.
1974.SIAM, Philadelphia.
[6] Hideaki Haneishi, et al, “A new method for distortion correction of
electronic endoscope images”, IEEE, Trans. on Medical Imaging, Vol.
14, No. 3, pp. 548-555 (1995)

IV. CONCLUSIONS
We have developed a method that combines a MGC
camera with an stereo optical cameras (OC) to synthesize two
kinds of images. The method and a preliminary experimental
result with a prototype setup have been presented. A
successful result was obtained.

(a) OC image (right camera) (b) GC original image

after distortion correction

(c) GC binarized image (d) 3D estimation from

two OC images

(e) Image synthesis (f) Image synthesis

using only one OC using two OCs
Fig. 7 Images obtained in the preliminary experiment.

V. REFERENCES
[1] Makoto Tsuchimochi, et al, “A prototype small CdTe gamma camera
for radioguided surgery and other imaging applications”, European

5376