Persistent automatic tracking of multiple surface
vessels by fusing radar and lidar
Jungwook Han1 , Jinwhan Kim1 and Nam-sun Son2
1
Department of Mechanical Engineering, KAIST, Daejeon, Korea
2
Korea Research Institute of Ships and Ocean Engineering, Daejeon, Korea
∗
e-mail: jinwhan@kaist.ac.kr
Abstract—This paper addresses the problem of automatic
target tracking of multiple surface vessels for unmanned surface
vehicles (USVs). For safe USV operation, the detection of vessels
in the surrounding environment is an important capability, and
marine radars have been used to detect and estimate their
motion. However, vessel detection at short-range using radars
is challenging due to their inherent shadow zone. Therefore, we
proposes a vessel tracking approach fusing a pulse radar and
a 3D lidar. The relative bearing and range information between
a USV and nearby vessels is obtained using radar and lidar
sensors, and their motion including the position, heading, and
speed is estimated based on a dual filter structure using an
extended Kalman filter (EKF). This approach enables persistent
tracking of multiple surface vessels. To verify and demonstrate
the feasibility of the proposed method, a field experiment was
performed in an inland river environment and the results are
presented.
Index Terms— Multiple target tracking, unmanned surface
vehicle, sensor fusion
I. I NTRODUCTION
In recent years, unmanned surface vehicles (USVs) have
attracted much interest for their potential applications to time-
consuming and dangerous missions such as environmental
monitoring, reconnaissance, and surveillance. To perform these
missions, autonomous vehicle capabilities including precise
navigation, path planning, control, mapping, and obstacle
detection are required. Among them, this study focuses on
automatic detection and motion estimation of marine obstacles
which are important capabilities for safe vehicle operation.
Marine radars are standard navigational sensors for target
ship tracking and collision avoidance. The sensor provides rel-
ative bearing and range information of surrounding obstacles
over a wide range. However, their detection capability greatly
degrades when target objects are at close range due to their
inherent sensing characteristic (e.g., close-range shadow zone).
For persistent automatic target tracking, additional sensors are
required to detect short range obstacles.
In fact, three-dimensional (3D) lidars have been successfully
applied to self-driving cars to collect traffic information in the
surrounding environment. Lidars can provide precise relative
bearing and range information of close-range obstacles within
the radar’s shadow zone. Therefore, persistent target tracking
with no dead-zone can be achieved by fusing radar and lidar
measurements.
In this paper, we address a multiple-target tracking method
using radar and lidar sensors in the framework of sensor
978-1-5090-5278-3/17/$31.00 ©2017 IEEE
Authorized licensed use limited to: Shanghai Jiaotong University. Downloaded on April 24,2022 at 12:50:43 UTC from IEEE Xplore. Restrictions apply.
∗
fusion. This method enhances the automatic radar plotting
aid (ARPA) capability and the surveillance performance of
a USV system. The proposed tracking approach consists of
three parts. First, target objects are detected using both a pulse
radar and a 3D lidar by applying automatic feature extraction
algorithms on their measurements. Second, the relative bearing
and range information of the targets is obtained from the
extracted features. Finally, the target motion is analyzed in
a dual filter structure using an extended Kalman filter (EKF).
II. R ELATED S TUDIES
Estimating the motion of target traffic vessels is an impor-
tant capability for safe navigation in marine environments.
A variety of tracking algorithms have been introduced, and
exteroceptive sensors such as cameras, sonars and radars have
been employed for target tracking [1–5]. Among them, passive
sensors (e.g., cameras or passive sonars) have been applied in
the framework of bearing-only tracking (BOT) approach [2],
[4] for target motion analysis. However, the performance of
the BOT approach is highly dependent on the observability
defined by the relative trajectory of an observing ship with
respect to a target ship, and thus the BOT performance may
degrade severely in some situation (e.g., head-on targets).
Compared to the passive sensors, marine radars provide both
bearing and explicit range information. Various types of radar
sensors such as S-band, X-band, and high-frequency (HF)
radars have been utilized. Among them, X-band radars are
commonly used as the standard navigational sensor for surface
ships [6]. With the radars, a pixel-wise detector and a joint
probabilistic data association (JPDA) tracker were imple-
mented to estimate kinematics and physical size of targets in
a tracking filter [7]. To improve estimation of target trajectory,
multiple motion models were considered in an alpha-beta-
tracking filter [8].
To obtain more accurate target state estimates and broader
tracking coverage, multi-sensor tracking methods can be em-
ployed [9]. For example, camera measurements were com-
bined into a radar-based tracking algorithm to complement the
radar’s bearing measurement and minimize the dead-zone [10].
A multiple sensor module including radars, a lidar, and a stereo
camera mounted on a USV platform was applied for target
tracking of marine traffic ships [11]. For maritime surveillance,
multi-radar tracking was implemented by incorporating the
weighted center of detected targets into a tracking filter [12].
 In this paper, multiple target tracking by fusing radar and
lidar sensors is proposed for persistently tracking multiple sur-
face vessels with no dead-zone. To detect short range obstacles
within the radar’s dead-zone, a 3D lidar is employed and its
measurements are incorporated in an EKF-based tracking filter
with radar measurements. The proposed method is expected to
greatly enhance the surveillance capability of USV systems.
III. P ROBLEM FORMULATION
The formulation of the proposed multiple target tracking
mainly consists of two parts. The first part is the automatic
target detection from radar and lidar measurements. To extract
the relative bearing and range information of target vessels,
segmentation and classification methods are applied. The sec-
ond part is the multiple target tracking to estimate the motion
of each target including its position, heading, and speed using
an EKF. The tracking algorithm is designed to have a dual
structure to improve the robustness and computational effi-
ciency of target tracking. The overall procedure is illustrated
in Fig.1.
Fig. 1: The flowchart of the proposed multiple target tracking.
A dual filter structure (reserved and confirmed tracks) is intro-
duced to enhance the robustness and computational efficiency
of target motion estimation.
A. Automatic target detection
The relative bearing and range information between an
observing ship and each target ship is required for target
motion estimation. Segmentation algorithms are applied on
radar and lidar measurements to assign the data into a set
of classified targets and extract their relative positions.
1) Radar-based detection: To extract targets from radar
measurements, a set of preprocessing steps is implemented.
Sensitivity time control (STC) and fast time constant (FTC)
modules are applied to minimize unwanted echoes returned
from rain, snow, and water surface. Radar signals reflected
from a land are separated by designating the region of in-
terest (ROI) on radar images. To extract potential targets on
the preprocessed radar image, a dynamic thresholding method
using the cell averaging constant false alarm rate (CA-CFAR)
detector [13] is applied. For this, a window mask is defined
on the radar image considering the size of targets, and then
the signals which are sufficiently stronger than the local
Authorized licensed use limited to: Shanghai Jiaotong University. Downloaded on April 24,2022 at 12:50:43 UTC from IEEE Xplore. Restrictions apply.
background of each mask are extracted. The extracted signals
are classified into identical targets, and then the weighted
center of each target is estimated considering the intensity of
the each signal.
2) Lidar-based detection: Lidar measurements are provided
as point cloud data, and the point-distance-based segmen-
tation (PDBS) algorithm is applied to extract targets from
the data [14]. A 3D lidar which consists of multiple lasers
arranged in the vertical direction is used in this study. Each
laser provides a point cloud set reflected from the surfaces of
target vessels. The coordinates of the data are transformed
into the stabilized body-fixed frame which is obtained by
compensating the lidar’s roll and pitch angles, and then the
point cloud data are projected onto the vehicle’s motion
plane. The PDBS algorithm is applied on the projected point
cloud data to assign a set of classified objects (i.e., clusters).
Then, the clusters driven from noises are removed considering
the density-based statistical characteristics of clusters. To be
more specific, the lidar measurements reflected from a glint
of sunlight may appear sporadically, and the cluster driven
from this has low spatial density compared to that of a real
target (see Fig. 2(b)). Therefore, clusters whose spatial density
are smaller than a predefined threshold value are ignored. The
overall procedure is summarized in Algorithm 1.
Algorithm 1 Pseudocode for lidar-based target detection
Input:
3D point cloud data : P
Roll and pitch angles : φ, θ
Cluster density threshold : td
Output:
The number of targets and their center coordinates
Procedure:
1. Coordinate transformation to the stabilized body-fixed
frame : s P ←− CoordinateTransform(P, φ, θ)
2. Point cloud data projection to the vehicle’s motion plane
: s P0 ←− PointCloudProjection(s P)
3. Point cloud clustering :
s 0
Pc ←− PDBS algorithm(s P0 )
s 0
Pc = {ci | i = 1, · · · , m}
4. Density-based cluster filtering :
for i ← 1 to m do
ρi ←− SpatialDensityEvaluation(ci )
if ρi < td then
EliminateCluster(ci )
end if
end for
B. Multiple target tracking
The EKF-based motion estimation of the detected targets
is implemented using a constant velocity model. A dual
filter structure is introduced to increase the robustness and
computational efficiency of target tracking by reducing false
measurements. The filter structures are composed of a reserved
track and a confirmed track. A detected target is initially

(a) (b)
Fig. 2: Automatic target detection. Sensor measurements were
collected in an inland river environment (in blue dots). (a)
radar measurements (100 m ring). (b) lidar measurements (10
m ring).
registered into the reserved track to find whether it is a real
target or a clutter through the traditional M-of-N rule. The
targets which satisfy the M-of-N rule are registered into the
confirmed track, and then their motion is estimated in the
tracking filter. However, the targets whose position uncertainty
level exceeds a pre-defined value are removed.
In the filter structure, a three-degrees-of-freedom (3 DOF)
kinematic model is used to describe the motions of the
observing vehicle and targets. The state vector representing
the observer is described as
xo = [ xo yo ψo Vo ]T (1)
where xo and yo are the observer’s position defined in the
global frame, ψo is the observer’s heading, and Vo is the speed
in the longitudinal direction. The equations of motion for the
filter’s system kinematics of the observer can be written as:
ẋo = [ Vo cos ψo Vo sin ψo 0 0 ]T (2)
In order to estimate the motions of the observing vehicle
and targets simultaneously, the state vector of the tracking
filter structure is augmented by cascading the observer state
vector xo and the target state vector xT = [ xT T
T1 xT2 · · · ]
T
T dimension and β is the new source density value which is
where xTi = [ xTi yTi ψTi VTi ] describes the state vector of
the ith registered target. The equations of motion for the filter’s
system kinematics of the ith target can be written as:
ẋTi = [ VTi cos ψTi VTi sin ψTi 0 0 ]T (3)
Note that multiple targets are considered in this study, and thus
the total dimension of the target state vector increases with
the number of registered targets. The system dynamics can
be described as the augmented vector equation which consists
of the motion models of the observer and the targets. The
equations of the system dynamics can be written as
ẋ = [ ẋT T T
o ẋT ] + w (4)
where w is the zero-mean Gaussian process noise which
reflects the uncertainty of the motion model.
Authorized licensed use limited to: Shanghai Jiaotong University. Downloaded on April 24,2022 at 12:50:43 UTC from IEEE Xplore. Restrictions apply.
The two measurement datasets are employed to update the
motions of the observer and targets.
The first measurement dataset is for updating the motion of
the observer whose equation is expressed as
zo = [ zx zy zψ zV ]T = [ xo yo ψo Vo ]T + vo (5)
where zx , zy , zψ , and zV denote the position, heading and
speed measurements from onboard motion sensors. The vo is
the measurement noise, which is assumed to follow a zero-
mean Gaussian distribution.
The second measurement dataset is for updating the motion
of targets whose equation is expressed as
   
yT −yo
  arctan xT −xo − ψo
zβ  
zT = =  + vT (6)
zρ  q
2 2

(xT − xo ) + (yT − yo )
where zβ and zρ denote the relative bearing and range
measurements obtained from radar and lidar sensors. The
measurement noise vT is applied differently depending on the
measurement accuracy of radar and lidar sensors, and the two
measurement noises are assumed to be mutually independent.
For data association between the newly detected target
measurements and the existing targets in the filter state, the
global nearest neighbor method (GNN) is applied [10]. To
eliminate unlikely measurement-to-track pairs, an ellipsoidal
gate is applied and the association is allowed if the following
condition is satisfied
z0 S −1e
d2 = e z≤G (7)
where d2 is the normalized statistical distance, S is the
measurement innovation covariance matrix, and e z is the mea-
surement innovation vector which is defined by the difference
between the actual sensor measurement and the expected
measurement estimated by the filter. The maximum likelihood
gate G [10] can be written as
 
PD
G = 2 ln p (8)
(1 − PD (2π)M/2 β |S|)
where PD is the detection probability, M is the measurement
defined as the false alarm probability divided by the scan
area. The GNN method is a single hypothesis data association
approach, and thus association conflicts may occur for multiple
target tracking. For example, more than one measurement can
be allocated in a track gate or a single measurement can be
allocated in the gates of multiple tracks as shown in Fig. 3.
To solve the measurement-to-track assignment problem, an
auction algorithm [10] is applied by defining a cost value c as
follows.
c = G − d2 − ln[|S|] (9)
In the cost evaluation, the determinant |S| is used to penalize
the tracks with large prediction uncertainty. In an actual imple-
mentation, the rectangular gate condition [10] is additionally
considered for computational efficiency, and the ellipsoidal
gate is only applied to the pairs for which the rectangular
gate condition is met.

(a) (b)

Fig. 3: Example of data association conflict. The measuremet

m1 is satisfied by the ellipsodial gate of three targets.

IV. F IELD EXPERIMENT

A. Experimental setup
Field experiments were carried out in an inland river using
a USV (observing vehicle) developed by Korea Research
Institute of Ships and Ocean Engineering (KRISO). The USV
was equipped with a pulse radar (Furuno FAR-2117), a 3D (c)
lidar (Velodyne HDL-32E), and a suite of motion sensors in-
cluding an inertial measurement unit (IMU) and the Real-Time
Kinematic Global Positioning System (RTK-GPS). In addition,
an automatic identification system (AIS) was installed in the
target vehicles to obtain their ground-truth trajectories. The
proposed algorithm was applied to the USV and implemented
using SPx software library for radar signal processing [15] and
point cloud library (PCL) for lidar data processing [16]. For
multiple target tracking, the USV was commanded to approach (d) (e)
three-static targets (buoys) and one-moving target (ship) within
the radar shadow range of approximately 80 ∼ 100 m in the Fig. 4: Experimental setup. (a) ARAGON USV (7.5 m in
environment. The speeds of the USV and the moving target length). (b) sensor configuration. (c) experimental scenario in
were set to be ∼ 6 knots. The USV platform and experimental an inland river. (d)-(e) snap shots of field experiment.
scenario are described in Fig. 4.
B. Results degrades when the relative range falls within 100 m. Thus,
Figure 5 compares the resulting target trajectories. The the position uncertainties of the targets monotonically increase
magenta dots represent the estimated target trajectory by the as the USV approaches the targets at close range, as shown
proposed tracking filter; the blue dots represent the trajectory in Fig. 6(b). However, the large position uncertainties are
of the USV obtained by the onboard motion sensor. The AIS considerably reduced when lidar measurements are combined
data were collected for performance comparison, and these into the proposed tracking filter, which shows the feasibility
are represented as black circles. The estimated trajectory of the of the proposed method for persistent automatic tracking.
moving target satisfactorily follows the ground-truth trajectory.
Also, the estimated trajectories of the three stationary targets V. C ONCLUSION
are well confined within a moderate range. In addition to the This study proposed a method for automatic target tracking
trajectory comparison, the relative ranges of the targets with re- of multiple surface vessels with a USV. A pulse radar and
spect to the USV are shown in Fig. 6(a) with the measurements a 3D lidar were used to detect obstacles in the surrounding
that were used to update the estimated motion of the registered environment, and segmentation algorithms were applied to
targets in the filter structure. While all the targets could be extract targets for the sensor measurements.
detected from their initial positions using the radar sensor, For target motion analysis, the detection results were in-
the detection performance of the radar-based tracking method corporated into an EKF-based tracking filter. To increase

Authorized licensed use limited to: Shanghai Jiaotong University. Downloaded on April 24,2022 at 12:50:43 UTC from IEEE Xplore. Restrictions apply.

Fig. 5: Comparison of the estimated target trajectories with
AIS data.
computational efficiency, a dual filter structure was introduced
and implemented, and radar and lidar measurements were
combined in the dual filter structure. This approach enables
persistent and robust tracking of multiple surface vessels.
To validate the proposed method, the proposed tracking
algorithm was applied to a newly-developed USV system and
a field experiment was conducted. The results demonstrated
the feasibility of the proposed approach.
ACKNOWLEDGMENT
This research was supported by the projects titled ‘Devel-
opment of autonomous unmanned surface vehicle for ocean
observation and sea surveillance (PMS3490)’ funded by the
Ministry of Oceans and Fisheries, Korea.
R EFERENCES
[1] J. Kim, S. Vaddi, P. Menon, and E. Ohlmeyer, “Comparison between
three spiraling ballistic missile state estimators,” in AIAA Guidance,
Navigation and Control Conference and Exhibit, 2008, p. 7459.
[2] E. W. Frew and S. M. Rock, “Trajectory generation for constant velocity
target motion estimation using monocular vision,” in Robotics and Au-
tomation, 2003. Proceedings. ICRA’03. IEEE International Conference
on, vol. 3. IEEE, 2003, pp. 3479–3484.
[3] Y. Bar-Shalom, X. R. Li, and T. Kirubarajan, Estimation with applica-
tions to tracking and navigation: theory algorithms and software. John
Wiley & Sons, 2004.
[4] A. Farina, “Target tracking with bearings–only measurements,” Signal
processing, vol. 78, no. 1, pp. 61–78, 1999.
Authorized licensed use limited to: Shanghai Jiaotong University. Downloaded on April 24,2022 at 12:50:43 UTC from IEEE Xplore. Restrictions apply.
(a)
(b)
Fig. 6: Result analysis. (a) Estimated relative range of targets
with sensor measurements. (b) Comparison of the position
uncertainty level (1σ).
[5] J. Park, J. Kim, and N.-s. Son, “Passive target tracking of marine traffic
ships using onboard monocular camera for unmanned surface vessel,”
Electronics Letters, vol. 51, no. 13, pp. 987–989, 2015.
[6] S. S. Blackman, “Multiple-target tracking with radar applications,”
Dedham, MA, Artech House, Inc., 1986, 463 p., 1986.
[7] G. Vivone, P. Braca, and B. Errasti-Alcala, “Extended target tracking
applied to x-band marine radar data,” in OCEANS 2015-Genova. IEEE,
2015, pp. 1–6.
[8] W. Kazimierski, “Determining of marine radar target movement models
for the needs of multiple model neural tracking filter,” in Radar
Symposium (IRS), 2011 Proceedings International. IEEE, 2011, pp.
611–616.
[9] M. Liggins II, D. Hall, and J. Llinas, Handbook of multisensor data
fusion: theory and practice. CRC press, 2017.
[10] S. Blackman and R. Popoli, “Design and analysis of modern tracking
systems(book),” Norwood, MA: Artech House, 1999.
[11] L. Elkins, D. Sellers, and W. R. Monach, “The autonomous maritime
navigation (amn) project: Field tests, autonomous and cooperative be-
haviors, data fusion, sensors, and vehicles,” Journal of Field Robotics,
vol. 27, no. 6, pp. 790–818, 2010.
[12] D. Nikolió, Z. Popovic, M. Borenovió, N. Stojkovió, V. Orlić,
A. Dzvonkovskaya, and B. M. Todorovic, “Multi-radar multi-target
tracking algorithm for maritime surveillance at oth distances,” in Radar
Symposium (IRS), 2016 17th International. IEEE, 2016, pp. 1–6.
[13] H. Rohling, “Radar CFAR thresholding in clutter and multiple target
situations,” IEEE transactions on aerospace and electronic systems,
no. 4, pp. 608–621, 1983.
[14] C. Premebida and U. Nunes, “Segmentation and geometric primitives
extraction from 2d laser range data for mobile robot applications,”
Robotica, vol. 2005, pp. 17–25, 2005.
[15] Cambridge Pixel Ltd. SPx software library. [Online]. Available:
www.cambridgepixel.com
[16] R. B. Rusu and S. Cousins, “3D is here: Point Cloud Library (PCL),”
in IEEE International Conference on Robotics and Automation (ICRA),
Shanghai, China, May 9-13 2011.