234
7, 2021
Four-Dimensional Wide-Field Ultrasound
Reconstruction System With Sparse Respiratory
Signal Matching
Tianyu Fu, Jingshu Li, Jiaju Zhang, Danni Ai, Jingfan Fan, Hong Song, Ping Liang , and Jian Yang
Abstract—Four-dimensional (4D) ultrasound reconstruction can
greatly extend the spatial and temporal range of two-dimensional
(2D) ultrasound in clinical practice. However, uneven breaths may
yield a considerable motion artifact in the reconstructed time
sequences of volume ultrasound. In this paper, a system with sparse
respiratory signal matching is proposed to realize accurate 4D ul-
trasound reconstruction by effectively estimating uneven breaths.
A slippery platform is built to automatically control the ultrasonic
probe for acquiring 2D ultrasound slice sequences in different
abdominal areas. Based on these acquired sequences, the signals
of the respiratory phases are first extracted. Second, the sparse
matched signal pairs are determined through the signal distribu-
tion and the corresponding image connectivity. After matching is
conducted, signals in different positions are aligned in a common
space wherein the outliers of the signals caused by the uneven
breaths are excluded. Third, images with the same signal value
in different positions are collected as the 3D image at the phase
corresponding to the signal, and the 3D images at different phases
are employed to reconstruct the final 4D image. The accuracies
of the reconstructed 4D image are evaluated and compared with
those of five existing methods. Experimental results demonstrate
that the spatial and temporal continuities of the 4D ultrasound
image reconstructed by our method outperform those obtained by
other methods.
Index Terms—4D reconstruction system, ultrasound image,
respiratory signal, sparse matching.
I. INTRODUCTION
LTRASOUND imaging is widely used in the diagnosis
U and treatment of liver diseases because of its high real-time
quality, accessibility and safety [1], [2]. Affected by respiration,
the liver periodically moves along the head-foot direction [3].
Manuscript received November 3, 2020; accepted January 19, 2021. Date
of publication January 26, 2021; date of current version February 25, 2021.
This work was supported in part by the National Key R&D Program of China
under Grant 2019YFC0119300, in part by the National Science and Technology
Major Project of China under Grant 2018ZX10723-204-008, and in part by
the National Science Foundation Program of China under Grants 61971040,
81627803, 81871374, and 61771056. (Corresponding author: Danni Ai; Jian
Yang.)
Tianyu Fu, Jingshu Li, Jiaju Zhang, Danni Ai, Jingfan Fan, and Jian Yang
are with the Beijing Engineering Research Center of Mixed Reality and Ad-
vanced Display, School of Optics and Electronics, Beijing Institute of Technol-
ogy, Beijing 100081, China (e-mail: fty0718@163.com; jshul2020@163.com;
1205766491@qq.com; danni@bit.edu.cn; fjf@bit.edu.cn; jyang@bit.edu.cn).
Hong Song is with the School of Computer Science, Beijing Institute of
Technology, Beijing 100081, China (e-mail: anniesun@bit.edu.cn).
Ping Liang is with the Department of Interventional Ultrasound, Chinese PLA
General Hospital, Beijing 100039, China (e-mail: liangping301@126.com).
Digital Object Identifier 10.1109/TCI.2021.3054527
2333-9403 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL.
Traditional 2D ultrasound slice cannot completely track this
time-varying motion of a 3D organ. Hence, time-dependent
structural reconstruction, also known as 4D reconstruction, is
significant for capturing respiration-induced liver motion.
The 4D reconstruction of ultrasound image slices can be
regarded as 3D reconstruction at different respiratory phases. A
large number of ultrasound slices acquired in different positions
and at different phases are assembled as multiple 3D volumes
according to their spatial and temporal correlations. For sim-
plification, test subjects can be asked to hold the breath during
ultrasound acquisition. In the method [4], the 2D ultrasound
images are obtained under breath holding at multiple phases in
a cycle. By holding the breath, the temporal correlation between
the slices acquired in different positions is specific, and the
influence of the motion artifact is reduced in the reconstructed
images. Nevertheless, images of a large abdominal area cannot
be captured in a short time during breath holding. Moreover,
previous experiments [4] indicated that the number of phases is
limited by breath holding. These problems may result in a few
phases and a small field covered by the reconstructed 4D image.
Different from the breath holding-based reconstruction, tem-
poral correlation is computed in reconstruction with slices ac-
quired under a free breath. The temporal correlation is indicated
by the respiratory signals of slices. Respiratory signal can be
viewed as a one-dimensional feature to indicate the state of the
respiratory motion and phase in a cycle. The actual respiratory
phase of a slice is selected on the basis of signal, and the slices at
the same phase constitute a structural 3D image. The 3D images
at the multiple phases constitute the final 4D image. Therefore,
respiratory signal extraction and respiratory phase selection of
the ultrasound slices are the main steps in 4D reconstruction.
A common method of extracting respiratory signals involves
the use of external devices, such as spirometers, optical and
electromagnetic tracking devices [5]–[11]. The outputs of these
external devices indicate different physical meanings. If the
output is high-dimensional data for each slice, then principal
component analysis (PCA) is applied to obtain a value as the
respiratory signal. Given that additional equipment is needed,
these extraction methods are not widely applicable. To avoid this
problem, the feature in slice is extracted as a respiratory signal
[12]–[17]. The position of the diaphragm is a common feature.
Hwang et al. [15] used an adaptive threshold and polynomial
fitting to segment the diaphragm in a slice sequence. In a fixed
area, the position of the diaphragm is detected and used as
FU et al.: FOUR-DIMENSIONAL WIDE-FIELD ULTRASOUND RECONSTRUCTION SYSTEM WITH SPARSE RESPIRATORY SIGNAL MATCHING
the respiratory signal of each slice. Based on the segmenta-
tion of the diaphragm, the respiratory signal was accurately
extracted regardless of the noise in the slice. Wu et al. [17]
proposed a method that mainly employs the intensity feature
of the diaphragm matching between adjacent slices to extract
a respiratory signal. The translation between the manually se-
lected patches including the diaphragm in the adjacent images
is obtained through the computation of the normalized cross
correlation. The value with a large variation in the obtained
translations is utilized as a respiratory signal. In the method using
the feature of the diaphragm, the diaphragm must be captured
in each slice of a sequence.
Dimensionality reduction can be used to directly extract a res-
piratory signal from the slice. Because of nonlinear respiration-
induced liver motion, nonlinear dimensionality reduction meth-
ods, such as the manifold learning, are used to extract respiratory
signals. Unlike the data obtained by external devices or the posi-
tion of the diaphragm, the results of dimensionality reduction are
the values without physical meaning. In the method [18]–[21],
a slice sequence with the high-dimensional feature is mapped
together to a low-dimensional manifold space. If the space is
one-dimensional, then the mapped result of each image is used
as a respiratory signal.
According to different respiratory signal types, the corre-
sponding respiratory phase selections are different. If the signal
obtained from external devices or the position of the diaphragm,
slices can be directly divided into different phases through the
signal value with a physical meaning. Low et al. [5] proposed a
straightforward selection method to identify the actual respira-
tory phase by using the tidal lung volume. Furthermore, the im-
age similarity is used as a constraint in the selection in the method
proposed by Dikaios et al. [22]. If signals are obtained through
dimensionality reduction, then the signals of slice sequence
are position-independent and must be normalized to a common
space. Georg et al. [18] used manifold learning to extract res-
piratory signals from a cross-section slice sequence acquired at
various locations. All values in a signal sequence are used to
densely compute the affine transformation to the sequence in an
adjacent position. Meanwhile, image connectivity is used as a
punishment term to increase accuracy in computation. Through
transformation, all signal sequences are aligned to a common
space in which the entire range of the signals are divided as
the actual multiple phases for reconstruction. Wachinger et al.
[19] proposed a method which compensates the translation
between different sequences through the fixed acquired time
[20]. The scale is obtained via affine transformation by using all
the values in the sequences. Li et al. [23] proposed a method
and assumed that respiratory amplitudes in different cycles are
the same. With this assumption, scaling and translation are not
identified between different sequences. Therefore, all the signals
obtained by manifold learning are just averaged to normalize
different sequences in the method [23]. In the above methods,
all values in sequence are used to compute transformation, or the
entire ranges of sequences are normalized with the assumption
of even breaths. However, the randomness of an actual breath
results in various amplitudes and fluctuations of signals in and
between cycles. These features in the actual breath reduce the
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
235
Fig. 1. Slippery platform.
accuracy of normalization and reconstruction. All signals used
in normalization also decrease efficiency.
In this paper, we propose a 4D ultrasound reconstruction
system with sparse respiratory signal matching. To reduce the
influence of the artificial factor, an automatic slippery platform
is built for acquiring the slice sequences in different positions
along the head-foot direction under a free breath. In the part
of the software, all the ultrasound slices in a sequence are first
dimensionally reduced together by Isomap [24] to extract respi-
ratory signals. Then, all signal sequences are revised to the same
directions. The revised signals are smoothed and segmented
into inhalation and exhalation processes. Meanwhile, the signal
fluctuations are detected and merged in the segmentation. With
the consideration of the randomness in an actual breath, the
sparse points in each signal sequence are selected near the start,
middle and end of each process for matching. After matching
is conducted, the outliers caused by the uneven breaths are ex-
cluded in the uniform range of all the matched signal sequences.
This uniform range is divided into multiple phases, and the
slices with the largest continuity in the same phase constitute
the 3D images. The 3D images at the different phases are used
to reconstruct the final 4D image.
The main contributions of this paper are as follows: (1) the
acquisition system is built to automatically obtain a high-quality
ultrasound slice sequences for 4D reconstruction; (2) the ran-
domness of breath is considered in the entire reconstruction
process to increase the accuracy of respiratory signal extraction
and respiratory phase selection; (3) the normalization between
the signal sequences is improved by a proposed sparse matching
to accelerate and optimize the result of 4D reconstruction.
II. MATERIALS AND METHODS
A. Hardware Setup and Ultrasound Image Acquisition
An X-Y numerical control cross slippery platform is built to
automatically control an ultrasonic probe for acquiring ultra-
sound slices.
1) X-Y Numerical Control Cross Slippery Platform: In
Fig. 1, the entire platform contains a power supply, a stepping
motor driver, a controller, a holder and a slider. The holder
is manufactured through 3D printing and used to hold the
236
TABLE I
PLATFORM PARAMETERS
TABLE II
IMAGING PARAMETERS
ultrasonic probe. The parameters of this platform are shown in
Table I.
2) Ultrasound Machine: A Resona 7 ultrasound machine
(Mindray Medical International Ltd, China) with a convex array
probe is used for liver imaging, and the imaging parameters are
shown in Table II. The original size and resolution of the acquired
2D ultrasound image are 852 × 659 pixels and 0.27 × 0.27 mm,
respectively.
3) Ultrasound Image Acquisition Protocol: During acquisi-
tion, each test subject is asked to keep lying flat and breath-
ing freely. Since the respiration-induced motion deforms the
liver in the head-foot direction, the ultrasonic probe is placed
parallel to the axial plane and automatically moved along the
head-foot direction. Therefore, the acquired image slices are the
cross-sections which capture the liver motion. The start position
of the acquisition is near the upper edge of the liver at the
expiratory phase. The end position is near the middle abdomen of
the subject. In each position, the ultrasonic probe automatically
stays for 10 s. According to the time of a respiratory cycle,
the acquired slice sequences cover approximately three cycles
in each position. After the acquisition in the current position,
the probe is moved down to the foot direction by 1 mm. The
entire acquisition process is harmless to the test subjects. The
acquisition process is illustrated in Fig. 2. The colored dotted
lines indicate the different positions where the probe stays, and
the distance between the adjacent lines is 1 mm.
B. Sparse Marching-Based Reconstruction Method
Fig. 3 shows the flowchart of the proposed reconstruction
method, which contains three parts: signal extraction and prepro-
cessing, sparse matching and phase-independent reconstruction.
The inputs are the 2D ultrasound slice sequences in the N
abdominal positions acquired by the constructed system. In
the first part, the dimensional reduction is used to extract the
respiratory signals of the 2D ultrasound slices in each posi-
tion. The directions of the extracted signals are revised among
the sequences for consistency. And the signals are smoothed
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 7, 2021
Fig. 2. Acquisition process.
to be segmented into exhalation and inhalation processes. In
the second part, a template image in each signal segment is
selected. Based on the template, the matched ultrasound image
pairs are determined in the different sequences. The selected
pairs are used in the matching between the signals of different
sequences for alignment. After the matching, the outlier signals
are excluded and the phase are segmented for each sequence
in the third part. Then, the 2D ultrasound images with same
signal value are chosen and filtered using the image continuity
to reconstruct the 4D ultrasound image.
1) Signal Extraction and Preprocessing: A respiratory sig-
nal can be regarded as a unidimensional feature which indicates
the nonlinear motion of the liver at different phases. Isomap is
a nonlinear dimensionality reduction method used to find the
low dimensional feature in an original high dimensional space,
and estimate the respiratory phase [18], [25], [26]. Meanwhile,
it yields good results for our application. Therefore, Isomap
without external devices is utilized in the proposed method
to directly extract respiratory signals from the ultrasound slice
sequence. To effectively extract the signal, the ultrasound slice
sequences acquired in the N positions of the abdominal area are
down-sampled and the dark regions in each slice are cropped.
The length and width of the down-sampled image are 1/8 of the
original ultrasound image. For each position, the down-sampled
images without the dark regions are inputted to Isomap. Isomap
constructs a low dimensional space where the geodesic distance
of any two 2D ultrasound images is preserved as that in the
original space. In the position n, the optimization in Isomap is
shown as follows:
    
min Ii,m − Ii,n  − si,m − si,n  2 , (1)
m,n
where Ii,m and Ii,n are the ultrasound slices acquired in the
position i; si,m and si,n are the points in the low dimensional
space;  ·  indicates the geodesic distance. In the low dimen-
sional space, each ultrasound slice could be expressed as a
low dimensional feature. If the dimension equals to one, si,m
can be regarded as the respiratory signal of the corresponding
ultrasound slice Ii,m .
The original signals obtained by Isomap will be directionally
revised, smoothed and segmented for further sparse matching.
FU et al.: FOUR-DIMENSIONAL WIDE-FIELD ULTRASOUND RECONSTRUCTION SYSTEM WITH SPARSE RESPIRATORY SIGNAL MATCHING
Fig. 3.
Fig. 4. Variation of the normalized MPV in the first position during respiration.
Because the outputs of Isomap are values without a physi-
cal meaning, the end-inspiratory signal value may larger or
smaller than the end-expiratory signal value in a sequence. If
the end-inspiratory signal value is larger than the end-expiratory
signal value, the direction of the sequence is considered positive.
Otherwise, it is negative. Since the different directions among the
sequences affect the matching process, the direction alignment
of the signals is needed in the proposed method.
In the proposed method, all directions are positively aligned.
The signal sequence in the first position is aligned first. In
general, the first position is near the highest point where the
liver can reach at the end-expiratory phase. At this phase in the
first position, the ultrasound slice captures the diaphragm which
is located on the superior border of the liver. When the phase
is away from the end-expiration, the diaphragm gradually dis-
appears in the image. In the ultrasound image, the hyperechoic
diaphragm is bright [27], thereby increasing the pixel value of the
end-expiratory image. Therefore, the mean pixel value (MPV) of
the end-inspiratory image is lower than that of the end-expiratory
image. Fig. 4 shows the variation of normalized MPV in the
first position during respiration. The normalized MPV decreases
from the end-expiratory phase to the end-inspiratory phase.
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
237
Flowchart of reconstruction software.
The end-expiratory and end-inspiratory images are shown at
the top and bottom of Fig 4, respectively. The diaphragm is
marked with a yellow ellipse in each end-expiratory image. The
end-expiratory and end-inspiratory images are determined in the
first position by comparing the MPV. According to the signal
values of the inspiratory and expiratory images, the direction of
the signal sequence is determined. If the direction is against the
positive direction, then the signal values of all the images in the
sequence will be inverted for revision.
For a subsequent position far from the diaphragm, image
similarity is used to revise the direction of the signal sequence.
In position i (i ∈ [2, N ]), the image with the largest similarity
to the end-expiratory image in the previous position is the end-
expiratory image in the current position. The end-inspiratory
image in the current position is determined in the same way.
Based on the signal values of the inspiratory and expiratory
images, the direction of sequence in position i will be revised.
After the direction is revised, each signal sequence will be
smoothed by a Gaussian smooth filter. Given the difference be-
tween exhalation and inhalation processes, the signal sequence
should be segmented to divide the different processes, which are
separately reconstructed. An ideal signal sequence is similar to
a sinusoid, and different processes can be identified using the
gradient direction. However, fluctuations make the smoothed
signal sequence dissimilar to the ideal sinusoid, which affects
identification accuracy. Fluctuations occur mainly because of
the slow respiratory rate. When the respiratory rate of the test
subject is slow, the ultrasonic probe will acquire too many
similar images. The geodesic distances among these similar
images are the same and small. In the original high dimensional
space, the topological information indicated by the same and
small geodesic distances among the large number of the similar
images is weak. The weak topological information decreases the
construction accuracy of the low dimensional space in Isomap.
Therefore, the slow respiratory rate causes fluctuations in the
respiratory signal extracted by Isomap. Such fluctuations fall
into two types which are colored in pink and green in Fig. 5.
The first type is the fluctuation between expiration and inspira-
tion, and the second type is the fluctuation during expiration
238 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 7, 2021

selecting the matching points from the inhalation and exhalation

processes of the sequence in the first position. An inhalation or
exhalation process covers a half cycle. According to the positive
direction of the sequence, the phase of I110 is near the end of
exhalation (or the start of inhalation), and the phase of I190 is
near the end of inhalation (or the start of exhalation). Based
on the template images I110 , I150 and I190 , the search ranges for
matching signals will be computed through the Gaussian mixture
model (GMM) in the first position. Because the motions of the
liver are slow at the end of inspiration and expiration during
respiration, a larger number of images acquired near the end of
inspiration and expiration than those acquired at other phases.
Accordingly, more signals are located near the end of inspiratory
and expiratory phases than at the other phases in each sequence.
Therefore, the GMM comprising two Gaussians is used to model
the distribution of the signal sequence in the first position.
In the GMM, the probability p(s) of the signal s is equal to:
Fig. 5. Fluctuation in the signal sequence. The pink arrows denote the fluctua- K

tion between expiration and inspiration. The blue arrows indicate the fluctuation p(s) = wk · G(s|μk , σk ), (2)
during expiration or inspiration. k=1

where K (K = 2) represents the selected number of Gaussian in

or inspiration. In the proposed method, a signal sequence is the GMM; wk is the weight of each Gaussian G; μk and σk are
initially segmented through the gradient direction. If the gra- the mean and standard deviation of G, respectively. Based on
dient direction is positive, the corresponding signal belongs the result of GMM, the search ranges for selecting the matching
to inhalation; otherwise, the corresponding signal belongs to signals according to the template images I110 , I150 and I190 are
exhalation. After the initial segmentation, multiple inhalation determined and shown as follows:
and exhalation segments are obtained. Some abnormal segments R110 = [μ1,1 − σ1,1 , μ1,1 + σ1,1 ] , (3)
include a few signals and cover a short signal range because of
fluctuations. These abnormal segments can be detected by the R150 = [μ12,1 − min (σ1,1 , σ2,1 ) , μ12,1 + min (σ1,1 , σ2,1 )] ,
signal number and range. According to the type of fluctuation, (4)
the detected abnormal segments are combined with the adjacent R190 = [μ2,1 − σ2,1 , μ2,1 + σ2,1 ] , (5)
normal segments in two ways. For the first fluctuation type,
the two segments before and after the fluctuation belong to a where, R110 , R150 and R190 are the ranges for the template images
different process. The fluctuation is merged into the segment I110 , I150 and I190 ; μ1,1 , μ2,1 , σ1,1 and σ2,1 are the mean and
with a smaller signal range. The merged segment is regarded as standard deviation values of two Gaussian curves; μ12,1 is the
a completed process. For the second fluctuation type, the two signal value of the cross point between the two Gaussian curves.
segments before and after the fluctuation belongs to the same For each completely half cycle of the sequence in the first
process. The two segments and the fluctuation are merged and position, the matching images is selected as follows:
the result is regarded as a completed process.  
10
I1,m = arg min I110 − I1,m 2 subject to s1,m ∈ R110 , (6)
2) Sparse Matching: Since the respiratory signal sequences  
without the physical meaning in the different positions are inde- 50
I1,m = arg min I150 − I1,m 2 subject to s1,m ∈ R150 , (7)
pendently obtained, the signals with the same value in different  
positions do not correspond to the same phase. Meanwhile, the
90
I1,m = arg min I190 − I1,m 2 subject to s1,m ∈ R190 . (8)
end-inspiratory or end-expiratory phases in the sequences in the After selecting for all completely half cycles, three image sets
different positions do not correspond to each other because of the Λ10 50 90
1 , Λ1 and Λ1 are obtained as follows:
randomness of breathing depth. In this section, the ultrasound ⎧ 10 10

images at the same phase acquired in the different positions ⎨Λ1 = I1,c |c ∈ [1, C1 ]

will be matched for signal sequence alignment. The scales and Λ50 = I 50 |c ∈ [1, C1 ]
, (9)
⎩ 190 1,c 90
translations of the signal sequences in the adjacent positions Λ1 = I1,c |c ∈ [1, C1 ]
will be normalized and compensated through the matching.
where C1 is the number of completely half cycle in S1 . In these
Moreover, only a few signals in a sequence are selected as the
sets, the corresponding signals of images are grouped as three
matching points to improve the efficiency and accuracy of the
signal sets as follows:
alignment. ⎧ 10 10

In the first position, the images with the signal value closest to ⎨Ψ1 = s1,c |c ∈ [1, C1 ]

10%, 50% and 90% of the range are selected as I110 , I150 and I190 , Ψ50 = s50 |c ∈ [1, C1 ]
. (10)
⎩ 190 1,c
respectively. These images are used as the initial templates for Ψ1 = s901,c |c ∈ [1, C 1 ]

Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
FU et al.: FOUR-DIMENSIONAL WIDE-FIELD ULTRASOUND RECONSTRUCTION SYSTEM WITH SPARSE RESPIRATORY SIGNAL MATCHING
90 , the cor-
Fig. 6. Selection of matching points. For the template image Ii−1
responding matching signals are marked by red circles in the completely half
cycle of the signal sequence Si .
The signals in Ψ10 50 90
1 , Ψ1 and Ψ1 are the matching points in
S1 in the first position.
10 50
In the subsequent position i (i ∈ [2, N ]), the images Ii−1 , Ii−1
90 10 50 90
and Ii−1 are the mean images of the set Λi−1 , Λi−1 and Λi−1 .
10 50 90
The image Ii−1 , Ii−1 and Ii−1 are used as the templates to select
the matching points from each half cycle of the sequence in the
position i. Similar to the selection formulated by the equation
(6–8), the three matching image sets and corresponding signal
sets in the position i are obtained, respectively. Fig. 6 illustrates
the process of selecting the matching points in Si according to
90
the template image Ii−1 . There are five completely half cycles
in Si , and matching images are elected in each of them. The
corresponding matching signals are marked by red circles in Si .
After the matching signal sets in each position are obtained,
the iterative closest point (ICP) method is used to align the signal
sequences in all the positions to that in the first position. For
ICP, the input is the 2D point set, where the first value is the
signal value and the second value is the index for each signal.
In the ICP, the closest point is determined by computing the
2D distance. The iteratively obtained transformation is only
employed to change the signal value. Given the rotation is not
identified between the two signal sequences, only the translation
t and scaling a between the signal sequences are considered in
the alignment. The energy function E of ICP for matching the
sequence Si in the position i is as follows:
M for capturing the liver motion. Therefore, P is equal to 15 in
1 
E (ai , ti ) = (ai · si,m + ti ) − s1,n 2 , i ∈ [2, N ],
M m=1
(11)
where s1,n is the transformed signal value of the closest point
in the first position. Through the optimization of equation (11),
the scaling ai and translation ti for sequence Si are obtained.
Using ai and ti , the aligned signal sequence Si in the position i
is as follows:
Si = Ti ◦ Si = ai · Si + ti ,
where, Ti = [ai , ti ].
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
239
Consequently, all sequences are aligned to the first sequence.
The scales and translations among the different signal sequences
are normalized and compensated.
It is worth noting that the reason that we do not select the
end-expiratory image (0%) and end-inspiratory image (100%) as
the templates in the first position is the randomness of breathing
depth. This randomness makes the image and corresponding
signal sequences in the different positions cover the diverse
respiratory phases. For two sequences Si and Sj in the different
positions, the sequence Si may not reach the breath depth of Sj .
If the end-expiratory and end-inspiratory images are regarded as
the initial template images, only very few signals are located in
the search ranges R110 , R150 and R190 . Consequently, the accuracy
of alignment may be decreased. Therefore, I110 ,I150 and I190
are selected to avoid the mismatching caused by the uneven
breath.
3) Phase-Independent Reconstruction: After the alignment,
all signal sequences are located in a common space, where
the entire range will be equally divided into several stages to
separately reconstruct. The ranges of all transformed sequences
should be uniform to make the entire range reasonable in the
common space. Although the transformations among the dif-
ferent sequences have been compensated by the alignment, the
signal ranges of the matched sequences may still be various.
The reason for that outcome is the uneven breath, which makes
the respiratory signals too large or too small. These signals
introduced by the uneven breath are the outliers and affects
the determination of the uniform range in the reconstruction.
Exclusion of outliers is the first step in this section. Outliers are
the signals with too large or too small values far away from the
entire distribution. The mean value of the end-inspiratory and
end-expiratory signals in all the transformed signal sequences
are regarded as the standard limit. The respiratory signals outside
the limit are regarded as outliers, and the corresponding images
are excluded from the reconstruction.
After the outlier exclusion, the entire signal range will be
equally divided into P stages, which corresponds to the number
of phases to be reconstructed. If P is too small, the reconstructed
4D ultrasound image is sparse in the temporal dimension, and
the reconstruction result could not record the entire respiratory-
induced motion of the liver. If P is too large, a lot of 3D
ultrasound image will be reconstructed in a respiratory cycle,
and the time interval between the 3D ultrasound images at the
adjacent phases is too short. The reconstructed 4D ultrasound
image with the too high temporal resolution is meaningless
the proposed method. Meanwhile, given the difference between
inspiration and expiration, these processes are separated for
reconstruction. Therefore, a complete respiratory cycle from
inspiration to expiration contains 2P phases. Each phase covers
a small signal segment. The first P phases Dp+ (p ∈ [1, P ])
constitute entire inspiration, and the corresponding signal seg-
ments ranges from low to high. While, the second P phases Dp−
(p ∈ [1, P ]) constitute entire expiration, and the corresponding
(12) signal segments ranges from high to low. Images with the signal
values from the same segment in different positions will be
sequentially selected to reconstruct the image at the phase.
240
In the first position, the image with the closest signal value to
the average of the chosen segment is selected as the first slice
in the reconstructed image. In the other positions, the image
connectivity and signal similarity are considered in the selection
to improve the accuracy of reconstruction. In the proposed
method, L2 distance is used to evaluate the connectivity between
two ultrasound image slices. If L2 is small, the connectivity
is high. From the chosen segment, the image with the highest
connectivity to that in the previous position is selected in the
position i as the ith slice Iip in the reconstructed image. The
process of an inspiratory phase can be formulated as follows:
 p 
Iip = arg min Ii−1 − Ii,m 2 subject to si,m ∈ Dp+ , (13)
where Ii,m is an image with the signal value si,m belonging
the phase Dp+ and m ∈ [1, M ]. After the reconstruction at each
respiratory phase, the 4D ultrasound image is obtained.
III. EXPERIMENT
In the experiment, the proposed method is evaluated on the
2D ultrasound images which acquired from six test subjects
using the built system. The approval No. of ethics committee
is S2020-464-01 provided by Ethics Committee of Chinese
PLA General Hospital. The assessment criteria is shown in the
section A. Based on the same ultrasound sequences acquired by
the slippery platform, the reconstruction results obtained by the
proposed method are compared with those of the methods in
Georg et al. [18], Li et al. [23], Dikaios et al. [22], Low et al.
[5] and Wachinger et al. [19]. The implementations of these
methods are detailed in the section B. The evaluation contains
three parts corresponding to the sections C, D and E. In the
section C, the signals obtained by the external devices is used
as the ground truth, and compared with those obtained by the
proposed method. Meanwhile, the sparse matching results of
the respiratory signal sequences from the adjacent positions are
assessed. The spatial and temporal continuities of the reconstruc-
tion result are evaluated in the sections D and E, respectively.
In the experiment, the used CPU and RAM are i7-9700K and
16 G, respectively. The proposed method was implemented in
Python and C ++ programming languages.
A. Assessment Criteria
For the evaluation of the signal extracted by manifold learn-
ing, an external device called Aurora, which is an electromag-
netic tracking system from the NDI, is used to detect true respi-
ratory signals. A marker is placed on the left abdomen of each
subject. When an ultrasound image is acquired, the 3D position
and direction of the marker indicated by a six-dimensional
(6D) vector are recorded through Aurora. The vectors of all
images are dimensionally reduced through PCA to obtain a value
which is used as the ground truth for the respiratory signal. The
normalized correlation coefficient (NCC) between the ground
truth and the respiratory signal obtained by Isomap is computed
to evaluate accuracy. If the correlation coefficient is large, the
obtained respiratory signal is accurate.
The L2 and the respiratory line are used to evaluate the
spatial and temporal continuities of the resultant 4D images,
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 7, 2021
TABLE III
MODULES USED IN THE IMPLEMENTATION OF COMPARATIVE METHODS
respectively. If the L2 between the adjacent slices is large, the
selected slices to be reconstructed may be not come from the
same respiratory phase, and then the boundary of the tissue in
the reconstructed image is not smooth enough. The respiratory
line is the signal sequence of all 3D images in the in the
reconstruction result obtained by Isomap. If the respiratory line
is similar to a Gaussian curve, the temporal continuity of the 4D
reconstruction result is high.
B. Implementations of Comparative Methods
There are mainly four modules involved in the implementa-
tions of the comparative methods including respiratory signal
extraction, dense matching of the signal, similarity constraint
and separate reconstruction. In the module named image sim-
ilarity constraint, image similarity is used as a constraint for
reconstruction. Meanwhile, the respiratory signal extraction can
be divided into external device utilization and manifold learning.
The different modules are used to implement the comparative
methods as shown in Table III. In Table III, “” and “×” mean
that the module is used and not used in the method, respectively.
And “–” means that the module is not involved in the method.
Therefore, the comparisons of Dikaios et al. [22] and Low
et al. [5] are used to test the advantages of the proposed signal
extraction and processing. The comparisons of Li et al. [23],
Georg et al. [18] and Wachinger et al. [19] are used to examine
the advantage of the proposed sparse matching.
C. Evaluation of the Respiratory Signal
The respiratory signals computed by the manifold learning
are first revised to the positive direction. The revised signal
sequences and the corresponding ground truth are respectively
marked in red and green lines in Fig. 7. Given that the outputs of
the manifold learning are the values without physical learning,
the value ranges of the ground truth and obtained respiratory
signals are normalized into [0, 1] for comparison. The rows in
Fig. 7 illustrate the respiratory signal sequences from the differ-
ent test subjects. The left, middle and right columns illustrate
the respiratory signal sequences near the first, middle and last
positions in the abdominal area of the test subject, respectively.
Near the last position, the diaphragm is invisible in the slice
sequences. As shown in Fig. 7, the respiratory signal sequence
after the direction revision is similar to the ground truth, even
near the last position.
FU et al.: FOUR-DIMENSIONAL WIDE-FIELD ULTRASOUND RECONSTRUCTION SYSTEM WITH SPARSE RESPIRATORY SIGNAL MATCHING 241

Fig. 9. Segmentation of the signal sequences with fluctuation.

Fig. 7. Normalized respiratory signals and ground truth.

Fig. 8. Normalized correlation coefficient between respiratory signals and

ground truth.

Fig. 8 shows the NCC between the respiratory signals ob-

tained from manifold learning and the ground truth. Each box
in this figure indicates the distribution of the correlation coeffi-
cients in the different positions of the corresponding test subject.
As shown in this figure, the high value means a high correlation
between the ground truth and the obtained respiratory signal.
Fig. 9 shows the segmentation results of six signal sequences
(illustrated in Fig. 5) with the fluctuation. The fluctuation be-
tween expiration and inspiration is indicated by the pink arrow,
and the fluctuation during expiration or inspiration is denoted by
the blue arrow. As shown in Fig. 9, all fluctuations are correctly
combined into the determined adjacent inhalation or exhalation
segments.
Based on respiratory signal sequences, Fig. 10 illustrates the
matching results of six cases to compare the sparse matching
in the proposed method and the dense matching in the methods Fig. 10. Matched respiratory signal sequences. First to sixth lines show the
of Georg et al. [18] and Wachinger et al. [19]. The first to fifth results of the six test subjects. The left and right columns show the results
rows in this figure show the matching results of the six test obtained from dense and sparse matching.

Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
242
Fig. 11. Selection ratio at each phase. (a)-(e) are the ratios of six test subjects. The first and second rows are the ratios in inhalation and exhalation processes,
respectively.
subjects. The left and right columns show the sparsely and fully
matched sequences. In each sub-figure of Fig. 10, the vertical and
horizontal axes represent the position and the respiratory signal
value, respectively. The points in each position of the vertical axe
are the horizontally projected points of the matched sequence in
the corresponding position. Meanwhile, the projected points of
the matched sequence in a position are colored accordingly to
the ratio between the signal range of the sequence and the mean
range. The closer to 1 the ratio is, the closer to green the color of
the points will be. Otherwise, the color is closer to red or blue.
Therefore, if matching is accurate, then the mapped points in
the different positions are distributed nearly in the range of the
template sequence. The ranges of all densely matched sequences
are quite different, which is indicated by numerous red or blue
points as shown in the first column of Fig. 10. Meanwhile,
accumulative errors in the matching between the sequences in
the adjacent positions result in a significant shift between the
matched sequences. By contrast, the colors of the sequences
matched by the proposed sparse matching is closer to green, and
the mapped points cover the same signal value range. However,
the ranges of the matched sequences in the few positions are
away from the mean range because of the uneven breath.
D. Evaluation of the Spatial Continuity
Different from Li et al. [23], the ultrasound slices are se-
lected in the positions according to their signal values at each
respiratory phase in the reconstruction of the other compara-
tive methods and the proposed method. However, the different
selections in various methods may result in the lack of slices
in some positions. For the position without the selected slice,
interpolation is used to generate a slice and obtain a complete 3D
ultrasound image at the phase. The selection ratio, which is the
quotient result of the numbers of the positions with the selected
slices and all positions, is utilized to evaluate the methods. A
higher selection ratio indicates that more slices are selected and
few slices are interpolated to reconstruct. Meanwhile, because
the spatial continuity constraint changes the selected slices rather
than the number of them, only the selection ratios obtained by
Georg et al. [18], Dikaios et al. [22] and the proposed method are
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 7, 2021
compared in Fig. 11. This figure shows the selection ratio at each
phase obtained by different methods. Figs. 11 (a)–(e) present
the ratios of six test subjects. The first and second rows are the
ratios in the inhalation and exhalation processes, respectively.
Inhalation and exhalation processes are not separated in the
method of Georg et al. [18], so its selection ratios are the same
in the sub-figures of the first and second rows. The selection
ratio obtained by the proposed paper is generally high at each
phase. Due to the significant shift identified between the matched
sequences, the selection rate obtained by Georg et al. [18] is
very low. A higher selection ratio means that more slices in the
reconstructed image are actually acquired from the ultrasonic
apparatus, and the tissue structures shown in the reconstructed
image is closer to the truth.
Figs. 12 and 13 show the sagittal planes of the 4D ultrasound
images of two test subjects reconstructed using different meth-
ods. In these figures, the left, middle and right parts show the
reconstructed 3D ultrasound images at the start, middle and
end phases of the processes. In each part, the left and right
columns are the images in inhalation and exhalation processes,
respectively. The first to sixth lines show the results obtained by
Georg et al. [18], Li et al. [23], Dikaios et al. [22], Low et al.
[5], Wachinger et al. [19] and the proposed method. The yellow
arrows point the liver vessels in the reconstructed images. As
shown in these figures, the boundaries of the liver vessels in
the reconstructed image obtained by the proposed method are
smoother and more continuous than those obtained by the other
methods. It is worth noting that the reconstructed images in
Figs. 12 and 13 are the results of interpolation, which is used to
generate a slice for the position without a selected slice. Because
of the too low selection ratio from Georg et al. [23], the 4D
ultrasound image can not be completely reconstructed even after
the interpolation. Therefore, the black segment is identified in
the image reconstructed by Georg et al. [23].
To quantitatively evaluate the spatial continuity of the recon-
struction results obtained by the different methods, the values
of the L2 distances between the selected adjacent slices in the
reconstructed 3D image at different phases are computed and
shown in Fig. 14. The slices generated via interpolation are
excluded in the computation of the spatial continuity. Fig. 14
FU et al.: FOUR-DIMENSIONAL WIDE-FIELD ULTRASOUND RECONSTRUCTION SYSTEM WITH SPARSE RESPIRATORY SIGNAL MATCHING 243

Fig. 12. Reconstruction result on sagittal plane of subject 1. The left, middle and right parts show the reconstructed 3D ultrasound images at the start, middle
and end phases of the processes. In each part, the left and right are the images in the inhalation and exhalation processes. The yellow arrows point the liver vessels
in the enlarged view of the reconstructed images.

TABLE IV
L2 OF THE RECONSTRUCTED 4D ULTRASOUND IMAGE IN INHALATION

TABLE V
L2 OF THE RECONSTRUCTED 4D ULTRASOUND IMAGE IN EXHALATION

shows the L2 in the inhalation and exhalation of the first test
subject, respectively. In Fig. 14, the boxplots colored in red,
green, blue, purple, black and yellow are the L2 obtained by
Li et al. [23], Georg et al. [18], Dikaios et al. [22], Low et al.
[5], Wachinger et al. [19] and the proposed method. A small L2
means a high continuity of the reconstructed image. With the
continuity constraint, the proposed method achieves a smaller
L2 than that of the other methods. Tables IV and V show the
means and standard deviations of the L2 distances of all phases
in inhalation and exhalation processes for each test subject,
respectively. All the L2 for each test subject are normalized to
make the results intuitively comparable. Because inhalation and
exhalation processes are not separated in the method of Georg
et al. [18] and Li et al. [23], their means and standard deviations
of L2 are the same as those in the two tables. The results
show that the means and standard deviations of L2 obtained

Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
244 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 7, 2021

Fig. 13. Reconstruction result on sagittal plane of subject 4. The left, middle and right parts show the reconstructed 3D ultrasound images at the start, middle
and end phases of the processes. In each part, the left and right are the images in inhalation and exhalation processes. The yellow arrows point the liver vessels in
the enlarged view of the reconstructed images.

Fig. 14. Resultant boxplot of L2 between the selected adjacent slices in the reconstructed ultrasound image of test subject 1. The left and right are the results of
inhalation and exhalation, respectively.

by the proposed method are small. Thus, the boundaries of the

tissues in the reconstructed image at each phase are smooth and
continuous.
The structural liver vessel can be segmented from the re-
constructed 4D ultrasound image. The dynamic region growth
method is used to segment the vessel at the different phases
of the 4D ultrasound image, and the segmentations are shown
in Fig. 15. In this figure, (a), (b) and (c) show the vessel Fig. 15. Vessel segmentations with the corresponding reconstructed images at
segmentations are the 3D ultrasound at the start, middle and end the start (a), middle (b) and end (c) phases of inhalation, and they are compared
phases of inhalation process from test subject 4, respectively. in (d).

Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
FU et al.: FOUR-DIMENSIONAL WIDE-FIELD ULTRASOUND RECONSTRUCTION SYSTEM WITH SPARSE RESPIRATORY SIGNAL MATCHING
Fig. 16. Respiratory lines.
Fig. 15 (d) shows the variations among vessel segmentations at
the three phases. The vessel segmentations at the phases from
start to end are colored in blue, red and yellow. The structures
of the liver vessels in the reconstructed images are continuous,
and no fracture is identified. The variations among the vessel
segmentations are caused by respiration-induced motion.
E. Evaluation of the Temporal Continuity
The 4D reconstruction result covers the entire respiratory
cycle. If the temporal continuity is high, the respiratory line
joining the signals of all 3D images in the in the reconstruction
result should be similar to the Gaussian curve. Meanwhile, the
inspiratory and expiratory lines should be similar to the each
other, which indicates by the symmetry of the respiratory line.
We use the Isomap to extract the respiratory signal of each 3D
image in the reconstruction result. The respiratory lines of the
different methods are shown in Fig. 16. In this figure, (a-f) show
the respiratory lines using the data from the six test subjects.
The signal value range of each line is normalized to [1[,-1].
And the range is divided into 15 segments. Compared with the
other methods, the respiratory lines obtained by the proposed
method are more similar to a Gaussian curve without the zigzag.
Meanwhile, the values of 1 and −1 are the end-inspiratory (or
start-expiratory) and end-expiratory (or start-inspiratory) signal
values in the ideal respiratory line, respectively.
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
245
TABLE VI
NORMALIZED SIGNAL VALUES OF THE 3D IMAGES AT THE PHASES OF THE
START-IN, END-IN, START-EX AND END-EX IN THE RECONSTRUCTION RESULTS
Fig. 17. Ratio between AUIC and AUEC.
Table VI shows the normalized signal values of the start-
inspiratory (start-IN), end-inspiratory (end-IN), start-expiratory
(start-EX), end-expiratory (end-Ex) images in the reconstruction
results. For the proposed method, the obtained signal values
of start-IN and end-EX are approximately equal and close to
−1. The signal values of start-EX and end-IN obtained by the
proposed method are approximately equal and close to 1. The
results of Georg et al. [18] and Li et al. [23] are not collected
in Table VI because of their undistinguished reconstruction for
inhalation and exhalation. In Fig. 17, the ratio between the
area under the inspiratory curve (AUIC) and the area under
the expiratory curve (AUEC) to evaluate the symmetry of the
obtained respiratory line. Inhalation and exhalation processes
are not separated in the method of Georg et al. [18] and Li et al.
[23], so their ratios of them are equal to 1. The ratio achieved
by the proposed method is closer to 1 than that obtained by the
methods of Dikaios et al. [22], Low et al. [5] and Wachinger
et al. [19].
246
TABLE VII
ACQUISITION TIME AND THE ACQUISITION POSITION NUMBER
TABLE VIII
RECONSTRUCTION TIME (MINUTE)
IV. DISCUSSION
A. Acquisition Time
The acquisition time of the six test subjects is shown in
Table VII. The gender of each test subject is presented in the
first column of Table VII. The acquisition time is only related to
the number of acquisition positions. Therefore, the acquisition
time can be adjusted for the different subjects. For the fat and
males, the number of the acquisition position should be large
to cover the large liver and larger respiratory amplitude, and the
corresponding acquisition time is long. For the thin and females,
the number of the acquisition position is small according to the
small liver and respiratory amplitude, and the corresponding
acquisition time is short.
B. Reconstruction Time
The time consumption of the three steps in the proposed
method is computed in Table VIII. For the test subject with
a large position number, the time consumption of the three steps
is long. Meanwhile, the time consumption of sparse matching
in the second step is compared with that of dense matching. In
contrast to dense matching, a few signals are used to align the
adjacent sequences in sparse matching, and the efficiency of the
proposed method greatly improves.
C. Entire Time Optimization
The signal sequence is extracted and preprocessed indepen-
dently in each position, and only the adjacent sequences are
sparsely matched in the proposed method. Therefore, acquisition
and reconstruction can be performed in parallel after the images
are acquired in the first position. Table IX shows the comparison
between the parallel and serial processing time. The mean reduc-
tion rate reaches 27.965% through the parallel processing, and
the entire time of obtaining the 4D ultrasound image is shorter
than 17 minutes.
Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 7, 2021
TABLE IX
TIME CONSUMPTION OF ACQUISITION AND RECONSTRUCTION PERFORMED IN
PARALLEL AND SERIAL
V. CONCLUSION
4D ultrasound image reconstruction is significant for the esti-
mation of respiration-induced liver motion. A sparse matching-
based method is proposed in this paper to reconstruct a 4D
wide-field ultrasound image. The original ultrasound slice se-
quences from the different abdominal positions of a subject are
automatically acquired through a system including a slippery
platform. The manifold learning is used to directly extract
the respiratory signal from each ultrasound slice without any
additional markers. The directions of the extracted signals are
firstly revised and then divided into inhalation and exhalation.
The signal sequence in different positions are aligned by sparse
matching with the three selected pairs. After the matching, the
signals with the same values from different positions belong
to the same respiratory phase, and the corresponding ultra-
sound slices could be assembled for the phase-independent
reconstruction. The assembled images at different phases are
grouped as the 4D ultrasound image. In the experiment, the
ultrasound slice sequences are acquired from six subjects to test
the proposed method, and the results are compared with those
of five existing methods. The high correlation value between the
extracted signals and the signals provided by markers indicate
that the proposed method can accurately extract the respiratory
signals without the additional markers. In the 4D ultrasound
image reconstructed by the proposed method, the small L2 value
between the adjacent slices indicates that a higher connectivity
correlation than that obtained by other comparative methods.
And the respiratory line extracted from the reconstruction result
obtained by the proposed is more similar to a Gaussian curve
than that obtain by other comparative methods. Therefore, the
proposed method can reconstruct a 4D ultrasound image with
high spatial and temporal continuities.
REFERENCES
[1] W. Cong et al., “Global patch matching (GPM) for freehand 3D ultrasound
reconstruction,” Biomed. Eng. Online, vol. 16, no. 1, pp. 124–149, 2017.
[2] J. Dong et al., “Multiresolution cube propagation for 3-D ultrasound image
reconstruction,” IEEE Trans. Comput. Imag., vol. 5, no. 2, pp. 251–261,
Jun. 2019.
[3] D. Ai et al., “Nonrigid registration for tracking incompressible soft tissues
with sliding motion,” Med. Phys., vol. 46, no. 11, pp. 4923–4939, 2019.
[4] J. M. Blackall, G. P. Penney, A. P. King, and D. J. Hawkes “Alignment
of sparse freehand 3-D ultrasound with preoperative images of the liver
using models of respiratory motion and deformation,” IEEE Trans. Med.
Imag., vol. 24, no. 11, pp. 1405–1416, Nov. 2005.
[5] D. A. Low et al., “A method for the reconstruction of four-dimensional
synchronized CT scans acquired during free breathing,” Med. Phys.,
vol. 30, no. 6, pp. 1254–1263, 2003.
FU et al.: FOUR-DIMENSIONAL WIDE-FIELD ULTRASOUND RECONSTRUCTION SYSTEM WITH SPARSE RESPIRATORY SIGNAL MATCHING 247

[6] J. Ehrhardt et al., “An optical flow based method for improved recon- Jingshu Li is currently working toward the Ph.D.
struction of 4D CT data sets acquired during free breathing,” Med. Phys., degree with the School of Optics and Photonics,
vol. 34, no. 2, pp. 711–721, 2007. Beijing Institute of Technology, Beijing, China. Her
[7] C. Gianoli et al., “A multiple points method for 4D CT image sorting,” current research interest includes ultrasound image
Med. Phys., vol. 38, no. 2, pp. 656–667, 2011. reconstruction.
[8] F. Ernst et al., “Correlation between external and internal respiratory
motion: A validation study,” Int. J. Comput. Assist. Radiol. Surg., vol. 7,
no. 3, pp. 483–492, 2012.
[9] T. H. Wagner et al., “Optical tracking technology in stereotactic radiation
therapy,” Med. Dosimetry, vol. 32, no. 2, pp. 111–120, 2007.
[10] M. Yang et al., “Ultrasound fusion image error correction using subject-
specific liver motion model and automatic image registration,” Comput. Jiaju Zhang is currently working toward the Ph.D.
Biol. Med., vol. 79, pp. 99–109, 2016. degree with the School of Optics and Photonics,
[11] J. R. Mcclelland et al., “4D motion models over the respiratory cycle for use Beijing Institute of Technology, Beijing, China. His
in lung cancer radiotherapy planning,” Proc. SPIE, vol. 5744, pp. 173–184, current research interests include reconstruction and
2005. registration for ultrasound images.
[12] J. Wölfelschneider et al., “Examination of a deformable motion model for
respiratory movements and 4D dose calculations using different driving
surrogates,” Med. Phys., vol. 44, no. 6, pp. 2066–2076, 2017.
[13] D. Zhou et al., “A feasibility study of intrafractional tumor motion esti-
mation based on 4D-CBCT using diaphragm as surrogate,” J. Appl. Clin.
Med. Phys., vol. 19, no. 5, pp. 525–531, 2018.
[14] A. Giger et al., “Ultrasound-driven 4D MRI,” Phys. Med. Biol., vol. 63, Danni Ai received the B.E. and M.E. degrees from
no. 14, pp. 145–155, 2018. Xi’an Jiaotong University, China, in 2005 and 2008,
[15] Y. Hwang et al., “Ultrasound image-based respiratory motion tracking,” respectively, and the D.E. degree from Ritsumeikan
Med. Imag. 2012: Ultrason. Imag. Tomogr. Ther., 2012, pp. 83200. University, Japan, in 2011. She joined the Beijing
[16] M. Von Siebenthal et al., “4D MR imaging of respiratory organ motion Institute of Technology as a Postdoctoral, in 2013.
and its variability,” Phys. Med. Biol., vol. 52, no. 6, pp. 1547–1564, 2007. She is currently an Associate Professor with the
[17] J. Wu et al., “Fast and robust extraction of surrogate respiratory signal from School of Optics and Photonics, Beijing Institute of
intra-operative liver ultrasound images,” Int. J. Comput. Assist. Radiol. Technology.
Surg., vol. 8, no. 6, pp. 1027–1035, 2013. Her research interests focus on medical image pro-
[18] M. Georg et al., “Manifold learning for 4D CT reconstruction of the cessing and augmented reality.
lung,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.
Workshops, 2008, pp. 1–8.
[19] C. Wachinger et al., “Manifold learning for image-based breathing gating
in ultrasound and MRI,” Med. Image Anal., vol. 16, no. 4, pp. 806–818,
Jingfan Fan received the Ph.D. degree in optical
2012.
engineering from the Beijing Institute of Technology,
[20] C. Wachinger, M. Yigitsoy, and N. Navab “Manifold learning for image-
China, in 2016, and worked as a Postdoctoral Re-
based breathing gating with application to 4D ultrasound,” in Proc. Int.
searcher with University of North Carolina at Chapel
Conf. Med. Image Comput. Comput.-Assist. Intervention, 2010, pp. 26–33.
Hill from 2017 to 2019. He is currently an Associate
[21] J. Wu et al., “A manifold learning method to detect respiratory signal from Professor with the School of Optics and Photonics,
liver ultrasound images,” Computerized Med. Imag. Graph., vol. 40, 2015,
Beijing Institute of Technology. His research interests
pp. 194–204.
focus on medical image processing, computer vision,
[22] N. Dikaios et al., “MRI-based motion correction of thoracic PET: Initial
and augmented reality.
comparison of acquisition protocols and correction strategies suitable for Hong Song received the Ph.D. degree in computer
simultaneous PET/MRI systems,” Eur. Radiol., vol. 22, no. 2, pp. 439–446,
science from the Beijing Institute of Technology,
2012.
China, in 2004. She is currently a Professor with the School of Computer Science,
[23] C. Li et al., “Subject-specific and respiration-corrected 4D liver model
Beijing Institute of Technology. Her research interests focus on medical image
from real-time ultrasound image sequences,” Comput. Methods Biomech. processing, augmented reality, and computer vision.
Biomed. Eng.: Imag. Visual., vol. 6, no. 1, pp. 7–16, 2016.
[24] J. B. Tenenbaum, V. De Silva, and J C. Langford “A global geometric
framework for nonlinear dimensionality reduction,” Science, vol. 290,
no. 5500, pp. 2319–2323, 2000.
[25] P. Fischer, T. Pohl, and J. Hornegger “Real-time respiratory signal extrac- Ping Liang is currently the Chairman of Depart-
tion from X-ray sequences using incremental manifold learning,” in Proc. ment of Interventional Ultrasound, Chinese PLA
IEEE 11th Int. Symp. Biomed. Imag., vol. 29, pp. 915–918, 2014. General Hospital. Her research interests focus on
[26] S. Carroll M, C. Viemari J, and M. Ramirez J. “Patterns of inspiratory microwave ablation and radiofrequency ablation
phase-dependent activity in the in vitro respiratory network,” J. Neuro- treatment of solid tumors in multiple organs
physiol., vol. 109, no. 2, pp. 285–295, 2013. (liver, kidney, adrenal gland, spleen, thyroid, breast,
[27] A. Sarwal, F. O. Walker, and M S. Cartwright “Neuromuscular ultra- metastatic lymph nodes, etc.).
sound for evaluation of the diaphragm,” Muscle Nerve, vol. 47, no. 3,
pp. 319–329, 2013.

Jian Yang received the Ph.D. degree in optical engi-

Tianyu Fu is currently a Postdoctoral with the School
neering from the Beijing Institute of Technology, Bei-
of Optics and Photonics, Beijing Institute of Technol-
jing, China, in 2007. He is currently a Full Professor
ogy, Beijing, China. His research interests focus on
with the School of Optoelectronics, Beijing Institute
medical image registration and reconstruction.
of Technology. His current research interests include
medical image processing and augmented reality.

Authorized licensed use limited to: San Francisco State Univ. Downloaded on June 22,2021 at 18:43:59 UTC from IEEE Xplore. Restrictions apply.