A Real-Time Vehicle Detection and Tracking System in Outdoor Traffic Scenes

1 1
Xin Li , XiaoCao Yao , Yi L. Murphey , Robert Karlsen , Grant Gerhart
1 2 2
1
Department of Electrical and Computer Engineering
The University of Michigan-Dearborn
Dearborn, Michigan 48128-1491, U.S.A.
Voice: 313-593-5028, Fax: 313-593-9967
yilu@umich.edu
2
U. S. Army - TARDEC, Warren, MI 48397-5000

Abstract Examples of research that use optical flow techniques

This paper presents a moving vehicle detection and
tracking system, MVDT, for real-time operation in
outdoor scenes. MVDT consists of three major
components: road detection, vehicle detection, and
vehicle tracking. The road detection algorithm utilizes a
plane-fitting feature, the vehicle detection uses both
segmented blob and Snakes blob features in a neural
network classifier, and a fast vehicle tracking algorithm is
designed to locate vehicles in consecutive image frames.
We show through experiments that MVDT is effective in
detecting moving vehicles in various outdoor scenes and
can indeed reach real-time operation requirements.
1. Introduction
ITS’s (Intelligent Transportation Systems) have been
an active research area for the last decade. Moving
vehicle detection is an important component in an ITS
that can be used in driver assistance system and Adaptive
Cruise Control (ACC) to give warning signals to drivers
in case of potential collisions. Therefore, it is an
important technology for improving traffic safety and
transportation efficiency.
Our research is focused on developing a real-time
moving vehicle detection system for operation in outdoor
scenes. Several facts make this task rather challenging:
• Real-time performance requires efficient
algorithms
• Cameras mounted in a moving vehicle are not
static, which means most methods used by
surveillance systems can not be applied here
• Scenes are dynamic and so, less prior knowledge
can be utilized.
• Illumination and contrast might change, even in the
same scene at different times
• Objects’ shape and color have broad ranges
Vision-based systems developed for this task can be
categorized into two classes: optical flow based
techniques and non-monocular image processing based
techniques.
for obstacle detection and tracking can be found in the
ROMA system [1] and the ASSET-2 (A Scene Segmenter
Establishing Tracking v2) [2]. However, optical flow
based methods fail when the relative motion between
objects and cameras is small. The non-monocular image
processing method can overcome this drawback.
Examples of stereo vision based ITS research can be
found in the UTA (Urban Traffic Assistance) project by
the DaimlerChrysler [3] and ARGO prototype vehicle [4].
We have developed a real-time computer vision
system, MVDT, which uses both stereo vision imagery
and motion cues for moving vehicle detection and
tracking. Figure 1 illustrates the computational steps in
MVDT. MVDT contains two major components: road
detection and vehicle detection and tracking.
vehicle
detection
Figure1: System architecture of MVDT
The road detection algorithm (Road Model) is
designed to detect either well-structured or unstructured
roads using grayscale images. The objective of Road
Model is to detect and remove road regions in an image in
real-time, so that later processes can focus on the ROI
(Region Of Interest). The input is the depth image of the
scene calculated from the stereo camera system. The
segmentation algorithm extracts the regions that are likely
to be vehicles, and the feature extraction algorithm
calculates vehicle features to be used by the neural

0-7695-2128-2/04 $20.00 (C) 2004 IEEE

network system for vehicle classification. A fast vehicle ( A, B , C ) for grid Gr ,c by using a least squares fitting
tracking algorithm is presented to locate vehicles in K
consecutive image frames. After tracking K frames, the method.
MVDT system returns to the Road Model, followed by [Step 3] The plane fitting error Err of grid Gr ,c is
r, c
vehicle detection. The combination of vehicle detection
and tracking can make real time operation possible. calculated by:
Experiments have been conducted on image sequences Err =
' '
∑ x + M − 1 ∑ y + N − 1 ( P ( x , y ) − PlaneFitti ng ( x , y ))
2

captured on highways and rural roads and the results are r, c x= x' y= y'

presented. ' '

2. Road detection
We have developed a road detection algorithm that
attempts to identify regions of roads in a disparity image
generated from stereo imagery and then segment road
areas from other parts of the scene. The roads can be
either structured or non-structured. The algorithm is
designed based on the following four assumptions:
• There is only one connected road area in each
image
• Roads always appear in the lower part of images
• Pixel values within a small area of the road do not
change much
• Road areas are reasonably flat
X
M totally bright saturated areas. It then searches leftward
GH −1,0 ... ... G H −1,W −1 N
...
where ( x , y ) is the coordinates of the left-bottom pixel
within grid Gr ,c , and PlaneFitting ( x, y ) is the fitted
value obtained by,
PlaneFitting ( x , y ) = A * x + B * y + C
Based on the plane fitting error at each grid (r, c), we
develop the following road detection algorithm.
The road detection algorithm, Road Model, uses grids
as units and searches for road areas from lower rows to
higher rows. At each row, it first finds a grid that is likely
to be within the road region and uses it as “seed”. A grid
is considered as a seed if its plane fitting error is within
the road range [T1, T2]. T2 is set to make sure the grid has
a small plane fitting error and T1 is used to eliminate
totally dark areas, which often occur under the trees, or
from the “seed”, and then rightward to grow the road

region. A grid is considered as belonging to the road

. P(x,y) region if its plane fitting error is within the road range, or
... ...
...

...

G r ,c
Y it is close to an adjacent grid that is considered a road
grid. The procedure continues to grow in this way until
...

...

...

r, y no more road grids are found. Figure 3 shows three

G0, 0 ... G0,W / 2 ... G0 ,W −1 examples of roads found by Road Model. The grids are
the road region found by Road Model, while the white
0 c, x
grids are the seed grids. Road Model is successful in
detecting road regions in front of the prime vehicle. Note
Figure 2. Illustration
Illustration of plane fitting in image that in Fig. 3(b) Road Model stopped at the lane lines,
grids
which is not our concern within this project.
The road detection algorithm utilizes road features
called plane fitting errors that are obtained as follows. An
input image of size X*Y is uniformly divided into grids,
where each grid contains M*N pixels (see Figure 2). The
plane fitting error is calculated for each grid by the
following procedure: (a) (b) (c)
[Step 1] Each grid Gr ,c (see Figure 2) within a road Figure 3. Road regions detected by Road Model. (a)
is a freeway, (b) is a city street, and (c) is a dirt road.
region can be modeled by a linear function:
P( x, y ) = A * x + B * y + C (1)
3. Vehicle detection and tracking
where ( x, y ) is the coordinates of a pixel within this grid Vehicle detection consists of the following processes,
and P ( x, y ) is the disparity value of the corresponding image segmentation, vehicle detection, which consists of
pixel(see Figure 3). A, B and C are the coefficients to be feature extraction and neural network classification, and
determined. vehicle tracking.
[Step 2] For each image grid, fit all points to the function
in Eq. (1) to obtain M*N functions and the coefficients, 3.1. Image segmentation
The image segmentation algorithm uses the depth
image to segment possible vehicle regions. A depth

0-7695-2128-2/04 $20.00 (C) 2004 IEEE

image provides distance information of objects in a scene.
The assumption used by the segmentation algorithm is
that distance values within certain objects should be
similar. Based on this assumption, we expect the first
derivative of depth values for a given object to be small;
but the derivatives at the object boundaries should be
more significant.
The formulas we use to calculate the first derivative in
the x and y directions are as follows:
Depth ( x + h , y ) − Depth ( x − h , y ) ,
Der _ X ( x , y ) =
2h
Depth ( x , y + h ) − Depth ( x , y − h ) ,
Der _ Y ( x , y ) =
2h
where Depth( x, y ) is pixel ( x, y ) ’s value in a depth
image, and h ∈ {1,2,3,...n...} .
The segmentation algorithm is outlined as follows:
INPUT: A depth image
[Step 1] Eliminate road area marked by Road Model
[Step 2] Eliminate pixels with large distance values
[Step 3] For each pixel (x,y),
if ( !( Der _ X ( x , y ) == 0 && Der _ Y ( x, y ) == 0 ) )
Seg _ depth ( x , y ) = 0 ;
else Seg _ depth ( x , y ) = Depth ( x , y )
[Step 4] Apply the connected component algorithm to
Seg_depth to obtain object regions.
OUTPUT: A segmented depth image.
Figure 4 shows an example generated by the
segmentation algorithm. The left image is the depth
image of the same scene as Figure 3 (a), and the image on
the right is the seg_depth image. The segmented objects
are shown in Figure 6 (a) and (c).
(a) (b)
Figure 4. An example of depth image segmentation.
(a) is the original depth image, and (b) is segmented
depth image, seg_depth.
3.2. Vehicle detection using a neural network
Vehicle detection is performed by a neural network
classifier. We use a feed-forward one hidden layer
network with backpropagation as the learning algorithm.
The critical part of a neural network classifier
implementation is to specify effective object features,
which are used as input feature vectors to the neural
network. We define 17 features for each region:
(Aspect_Ratio, Occupancy_Ratio, Depth_Std_Deviation,
7 Blob_Moments, 7 Snakes_Moments), where:
Aspect _ Ratio = Y / X , Occupancy _ Ratio = R /( X * Y )
0-7695-2128-2/04 $20.00 (C) 2004 IEEE
X, Y and R are shown in Figure 5; Depth_Std_Deviation
is the standard deviation of depth values within a
segmented blob.
Both Blob_Moments and Snakes_Moments are the
invariant moments features [5] calculated from segmented
blobs and Snakes blobs respectively. The Snake blobs are
generated by using a circle centered on the bounding box
of the segmented blob as the initial shape(see Figure 5),
and then applying the Snakes algorithm described in [6]
to the segmented blob.
The reason for using both the segmented and the
Snakes blob is that Snakes blobs can be quite different
from segmented blobs as illustrated in Figure 6. The
Snakes blobs usually provide smooth boundaries and
more accurate shape descriptions (see Figure 6 (a) and
(b)). However, in some cases the Snake blobs can miss
the fine details of object shapes such as shown in Figure 6
(c) and (d). The blob in Figure 6(c) is clearly not a
vehicle, but the Snakes blob shown in 6(d), which is
generated from 6(c), looks more like a vehicle blob,
which is not desirable.
Initial Snake
Shape
Bounding
Box
Y
Segmented
Image Region:
R
X
Figure 5. Illustration of initial shape for generating
Snakes blobs.
(a) (b) (c) (d)
Figure 6. (a) Segmented car blob; (b) the Snakes
blob of (a); (c) segmented non-
non-car blob, (d) Snakes
blob of (c).
3.3. Vehicle tracking
The motivation of vehicle tracking is to bypass the
relatively time consuming operation of vehicle detection
on certain image frames. We have developed the
following tracking algorithm. Let us assume that the
vehicle detection system detect Ot vehicle blobs from
image frame Ft. At frame Ft+1, we only track these Ot
vehicle blobs. In order to make the tracking efficient, we
define a match window of NxN within each vehicle blob
Vj, and a search window S. Let (xi, yi) be the center of Vi,
then the location of Vi in Ft+1 is found by calculating the
following least Mean Absolute Error (MAE) at every
point (i, j) within image frame Ft+1.
 N −1 N −1
∑ ∑ F ( x + p, y + q ) − F
p =0 q =0
t i i t +1
MAE(i, j ) = 2
N
The location of Vi is at (x
MAE (i' , j' ) is minimum among all points within the
search window S in image frame Ft+1.
The tracking operation can be carried out in the next K
frames after t th frame, K ≥ 1. After tracking through K
frames, the Road Model and vehicle detection system can
be applied to image frame t+K+1 again. The cycle of
detection and tracking can be repeated throughout the
entire image sequence. However, in order to avoid
missing new incoming vehicles into the scene, the
tracking should be applied infrequently and K should be
small.
4. Experiments
We tested the MVDT system on image sequences
captured in various outdoor scenes. The neural network
system has an architecture of 17 inputs, 5 hidden nodes
and 1 output. The image sequences for training and
testing were captured at the rate of 24 fps. The training
data consists of 263 car blobs and 1470 non-car blobs,
segmented from image sequences taken from highways
and city streets. In the testing data, there are 84 car blobs
and 519 non-car blobs. The test data is totally blind, since
the blobs were generated from two image sequences taken
at different times and on different roads from the training
data.
Table 1 shows the system performance on the
training and test data. It is encouraging to see that the
system performance on the blind test data is very close to
its performance on the training data. Figure 7 shows the
vehicles detected by MVDT on two images in the test
sequences. We calculated computational time of MVDT
while it was running on a Pentium M 1.3GHz, 512 MB
memory under MS Windows XP and our experiment
showed that MVDT reached about 14 frames per second
while the tracking program was called at every 4th frame
with K=1.
We have presented a vehicle detection and tracking
system, MVDT, and the major algorithms deployed by the
MVDT. We showed through experiments that MVDT is
effective in detecting moving vehicles in various outdoor
scenes and indeed can reach real-time operation
requirements.
Table 1. Performance of MVDT System
Non-Car Car Correct Error Undecided
Training 91.8% 98.9% 95.44% 4.43%
Testing 94.0% 97.8% 94.5% 5.5%
( xi + i + p, yi + j + q)
i + i ' , y j + j ' ) if
(a) (b)
Figure 7. Two examples of vehicles detected by the
MVDT system from the test image sequence.
5. Acknowledgment
This work is supported in part by a grant from the
Education Foundation of TRW Inc., and a DoD SBIR
contract.
6. References
[1] W. Kruger, W. Enkelmann, S. Rossle, “Real-time
estimation and tracking of optical flow vectors for
obstacle detection”, in: Proceddings of the IEEE
Intelligent Vehicles Symposium’98, Stuttgart, Germany,
Oct. 1998, pp.341-346.
[2] S.M. Smith, J.M. Brady, “ASSET-2: Real-time motion
segmentation and shape tracking”, IEEE Transactions on
Pattern Analysis and Machine Intelligence 17(8), 1995,
pp.814-829.
[3] U. Franke, A. Joos, “Real-time Stereo Vision for
Urban Traffic Scene Understanding”, Proceedings of the
IEEE Intelligent Vehicles Symposium 2000, Dearborn
(MI), USA, Oct. 3-5, 2000, pp. 273-278.
[4] Alberto Broggi, Massimo Bertozzi, Gianni Conte, and
Alessandra Fascioli, ARGO Prototype Vehicle, In L.
Vlacic, F. Harashima, and M.Parent, editors, Intelligent
Vehicle Technologies, chapter 14, pp. 445-493,
Butterworth-Heinemann, London, UK, Jun. 2001.
[5] M.K. Hu, “Visual pattern recognition by moments
invariants”, IEEE Trans. Inform. Theory, 8, 1962, pp.
179-187.
[6] D. Williams, M. Shah, “A Fast Algorithm for Active
Contours and Curvature Estimation”, Computer Vision,
Graphics and Image Processing, Vol.55, No.1, Jan. 1992,
pp. 14-26.
0.13%
0.0%
0-7695-2128-2/04 $20.00 (C) 2004 IEEE