A Range Security System for Vehicle Navigation
M. Gardufio,B. Vachon
HEUDIASYC URA CNRS 817
University of Compikgne
BP 649,60206 COMPEGNE Cedex, FRANCE
Phone (33) 44.23.44.23, Fax (33) 44.23.44.77
mgarduno@hds.univ-compiegne.fr
Abstract
The contribution of this research is the development
of a security system the purpose of which is to assist
the teleoperator of a mobile robot by providing him
pertinent information about the robot working space.
In this aim, the range image based perception system
must be able to detect and localize, as fast as possible,
obstacles (nontraversable regions) in the scene during
robot's motion. In this paper we present an accelerated
strategy of obstacle detection using the knowledge of
objects contained in previous images and their
predicted positions in the new image. An iterative
procedure based on a Kalman filter is used to improve
robot's state knowledge. Then, an image segmentation
takes place according to security and emergency
criteria. The range image is divided in priority areas to
reduce obstacle research time. Some results obtained
through experimentation of this method on synthetic
images are developed in this paper.
1. Introduction
1.1 Navigation of Vehicles
When a mobile robot works in unknown environ-
ments, especially in outdoor ones, it may encounter
unexpected situations and have to solve problems like
obstacle detection and avoidance. Such situations are
common and easily solved by a human being. To
achieve these goals, man extracts information from
his environment through natural senses (eyes, ears,
skin.. .). Situation analysis is carried out by the brain
and decision may be taken. However, transposition to
robots is a very complex problem. In the last few
years many efforts were made to endow robots with
enough intelligence to realize this kind of tasks. As
humans, robots use sensory capabilities. They are
equipped with physical sensors capable of catching
information from the environment in which they are
operating. The remaining problem is to know how and
by whom sensory data will be processed and
interpreted. This is the point where sophisticated
perception systems become useful in vehicle naviga-
tion. A wide variety of techniques based on different
philosophies have been used. See, for example,
[Dunlay, 881 at Denver for the description of the
357
bvachon@hds.univ-compiegne.fr
autonomous land vehicle ALV, or [Kanade, 861 for
details on the Carnegie-Mellon's vehicle Navlab.
Another algorithm for autonomous detection and
avoidance is described in [Eibert, 891.
1.2 Assistance systems
The purpose of most perception systems is generally
either to give some autonomy to a given vehicle, or to
assist a human operator in its guidance. The second
problem is addressed in this paper.
Vision based aid systems have found important and
useful applications in intelligent vehicle design. Such
advanced systems as IVHS [IVHS,92] in United
States or the Prometheus project EeFort, 931 in
Europe are intended to warn the conductor when
critical situation occurs while driving on a highway.
Although video images are often used, some works
deal with range image finders [Devy, 941.
1.3 ROBUST project
Our work is part of the ROBUST* project. ROBUST is
a teleoperated robot intended to navigate in outdoor
environments.. Teleoperation of a vehicle is complex,
because the human operator has no physical feeling of
the vehicle but must use information obtained via
sensors.
When only a vision system is available, environment
perception through a screen may lead to inappropriate
actions or decisions of the teleoperator. To face this
problem, we have developed a range image based
system to improve the security of a teleoperated
outdoor robot. Severe constraints are to be
considered : unstructured environment, natural
lighting conditions and motion condition since the
vehicle may reach a speed of up to 40 km/h.
Sensing is based on range images. Physical range
finders have nowaday a slow scanning process
[Lecoeur-T, 921, but geometric description of the
scene may be directly obtained from depth
information, and this is quite important as response
time is essential in our application. Moreover, laser
range finders offer the benefit of being relatively
independent of lighting conditions [Hebert, 891.
* This project is conduced by the Robotics Department of Dassault
Electronique Society (92, France) and has been subsidised by the French
Department for Research and Technology (aide MRT 90P0962)
2. General system synopsis
The system is composed of different modules as
shown in figures l(a,b). The range image sensor is not
yet available and has been replaced by a simulator.
The simulator (figure la) produces range images from
artificial scenes and takes into account distortions due
to motion of the sensor during the scanning process.
The fist image of a sequence is analyzed in a specific
way to detect obstacles in the scene. As this is the
start up of a robot's mission, the vehicle is usually
motionless and the response time is not significant.
This will allow to separate obstacle from non obstacle
regions in the image and extract symbolic representa-
tion of these areas. The dynamic image analysis
detects obstacles during the motion. This process
deals with tracking features and robot's location. A
predictor/estimation method based in a classical
Kalman filter is applied to this purpose (figure lb).
Another module in the diagram is the search of the
new obstacles technique. It refers to a segmentation of
the images in priority regions in order to reduce the
search time of newly visible obstacles.
/ sensor
parameters
/ scene
description
2.1 Simulator
A range generator has been developed in the
framework of our project to support the perception
system. This simulator is able to model a three
dimensional sensor carried on a mobile vehicle. It is
based on a point laser principle, this means that it
measures the distance along a single vector director
from a supposed optical center to a visible surface
point in the scene.
A range image represents a view by the sensor of a
synthetic scene composed of simple volumes as
spheres, cylinders, and parallelepipeds. The data we
obtain is function of sensor and motion specifications.
Thus simulation is based on geometric characteristics
of real sensors and other parameters as robot's speeds
and displacement. It involves distortioned images due
time and motion between two scanned pixels.
Additionally to this degradation, a Gaussian noise was
intrcduced in ray position and range measured.
2.2 Obstacle binarization
Separating obstacles from ground region is the first
step in the image processing. It requires grouping
pixels belonging to the same obstacle. Two criteria
are generally used according to the nature of the data :

pqq=zq generator 4- g
3D images are often depth images but sometimes
elevation maps.
Both formalisms are described in [Daily, 871. In the
first one, height discontinuities are computed to
classified obstacle pixels. In the second one, range

E
...,
image
... t-1, t,
L
...,
odometric
information
...
t-I, t,
discontinuities are extracted directly from range data.
In [Veatch, 871, first derivative discontinuity in range
images is used. A fast method developped in the ALV
project [MMC, 861 compares and analyzes each range
Figure l a : synopsis of the range image generator. in the image with respect to the corresponding range
in an ideal image. These four techniques have been
integrated to our system to detect obstacles.
Implementation, advantages and drawbacks of these
methods for obstacle detection with respect to time
and security constraints are discussed i n
[GarduAo, 941.
odometric
information 2.3 Symbolic representation
From the binary image, obstacles are characterized by
representative features stored by the system. A
4 I
segmentation algorithm is applied to compute m
interest around each object. From these areas,
as of
KOL(t- 1) -tNOLO)
4KOL (t) image
distinctive attributes are extracted and registered as
P the symbolic representation of each obstacle. The
output of this process is a set of vectors
I
detection of
new obstacles A(r) = [a1(r), a&),. ..an(r)l (1)
~vailableat time t-1
where ai(r) is the value of ith attribute of obstacle r.
11
1 available at timet L2 image
The attributes representing an obstacle are its gravity
center coordinates with respect to the world, with
respect to the image plane, the corresponding range
Figure l b :synopsis of the perception system. values, its compactness and dispersionFabre, 891.

358
3. Tracking process
A well known problem of obstacle tracking with a
vision sensor is due to the limited view field. Indeed,
even if obstacles are motionless, robot's motion make
them appear and disappear in the rolbot's perception
field. Localizing objects with respect ID the robot (and
vice-versa) is a main objective of our system. The
"tracking" concept is then introduced in our problem.
Obstacles are tracked through a sequence of range
images scanned during a robot's motiaa.
3.1 Motion model
Since the system is intended to be embedded on a
moving vehicle, a trajectory generation method was
implemented for simulation expeximents.. This
method is based on Bezier's curves which allows
modelizing a displacement guiding the robot through
a set of defined points taking into account kinematic
restrictions and orientation constraints [Segovia, 931.
A Bezier's segment is a vector fonction P of an
independent parameter U giving the coordinates of a
geometric point in a plane and definedl as :
P : U E R -+ (x,y) E R2
P(U) = (P3 - 3P2 + 3P1 - Po) U3 +
(2)
-
3(P2 2P1-I-Pz) U2 i- 3(P1 - Po) + Po
where (U E [O, 11). The four control points (Po to P3)
required by the 3rd degree polynomial are Po and P3
respectively the initial and final robol's positions, and
P i and P2 intermediate configurations computed by
the system. As the result of the evaluation of the
polynomial, a set of points P(u) = [x(ui), y(u)] defining
the curve is obtained. In the simulator, the number of
points P(u) or the number of evaluations in U, is a
function of the robot's speed and of the simulated
sensor measurement frequency.
3.2 Obstacle prediction
An obstacle detected in a current image may appear in
the next one. These already spotted obstacles are
easily located in new images as their position can be
estimated. Nevertheless to compute these estimates in
sequences of images varing in time, motion informa-
tion must be available.
In our system, obstacle position estimation is compu-
ted according to its gravity center (U', v') in image
plane coordinates system. During thie processing of
range image taken at time t-1, gravity center position
(xt-1, yt-1, zt-1) of the obstacle with irespect to world
coordinate system was obtained. Let f be the focal
distance of the range image sensor, h its vertical
location with respect to the ground and YT= (XT, Y,,
0,) the robot's position at some instant 2.
Then, if the obstacle has been seen al. instant z by the
sensor and if the robot's position is known at this
instant 2, the estimate (U', v') of the obstacle gravity
359
center position in image t is given by :
The problem is now to know how robot's position at
instant z can be estimated. One of the advantages of
using Bezier's polynomial for motion generation, is
that it's possible to obtain any point P(u) at any instant
in the curve. Any portion of the trajectory can be it
self approximated by another Bezier's curve. In this
case the initial and final positions (Po', P3') are
defined when the first and the last pixels of a range
image are scanned. Considering that time spent
between two measures is negligible, one can assume
constant robot's speed and associate each pixel of the
image with one point of the segment. Therefore the
value of U in which the polynomial (2) will be
evaluated is directly given by the pixel position in the
image matrix.
3.3 Matching process
When an obstacle is located in one image, it is
characterized (see section 2.3) and stored in the New
Obstacle List (NOL). A second list, created by the
system is the Known Obstacle List (KOL) which
contains the characteristics of obstacles detected in
the previous images. Matching process will then make
the correspondance between NOL and KOL to select
obstacles appearing in two successive frames.
Let's consider two images, scanned at instant t-1 and t,
that contain a certain number of obstacles stored in
the KOL and the NOL respectively. The tracking
process can be summarized in two steps :
e The first step consists in predicting obstacle
positions in the image It of the obstacles known
from image It-l. Equation (3) and (4) will provide
reasonable estimates of the expected values. This
estimation is used to perform tracking by providing
a possible point (or a reduced area) around which
the obstacle may appear in image t. If estimates
(U', v') belong to an obstacle then this obstacle is
characterized and stored in the NOL.
0
In the second step a match is to be found between
obstacles of the NOL and obstacles of the KOL. In
this aim, the distance A between the estimate (from
KOL) and the observed value (in NOL) of the
gravity center position is computed. All obstacle
attributes are considered in the matching process,
but trial are first attempted according to this
distance A. Thus, an object of NOL matches with
the closest (in the sense of A) object of the KOL if
their attributes are not significantly different and if
A is inferior to a specified threshold.
4. Estimation of the robot's position
As the estimate @t-l/t-l of the robot's configurzion
Yt-l is known at instant t-1 and the robot's motion is
measured by odometry, an estimate Q t - 1 of the
robot's position at instant t, Yt, can be computed.
Visual measures provided by the NOL allows to
improve the accuracy of the robot's position estimate.
This estimation is done by an Extended Kalman Filter
that stabilizes the measurements and system state in
the presence of noise and detection errors.
Let A: be the attribute vector of obstacle r computed
from the image It. After the matching process, the
word coordinates ( x:-~, y r l , z:-~)of obstacle r com-
puted from image It-1 and the image coordinates
(U:, vi) extracted from image t are available. The
Kalman filter equations then provide an estimate @t/t
of Yt with respect to A: :
(5)
The accuracy of the estimate is given by the
covariance matrix
K, is the Kalman gain, and H, the Jacobian mamx of
G [Chenavier, 921. The measurement equations are :
T cases, the image top corresponds to the sky area
where @t/t.l = (kt qt bt)
The estimate Q t - 1 is put up-to-date for every obstacle
r matched in KOL and NOL.When every obstacle has
been taken into account, gives the best estimate of
the robot's position.
5. Detection of new obstacles
A final process is applied to detect new obstacles in
the image. It consists in dividing the image in priority
regions to identify the more dangerous obstacles fist.
The part of the working space that the robot will go
through in its next future is the most critical for its
security. This area is called security area. The part of
the range image corresponding to the security area
will be called security region and must be scanned
first since obstacle contained in it have to be
highlighted priorily to the teleoperator. As the security
area depends on the robot's motion, the security
region has to be determined dynamically.
360
The range image segmentation is performed in two
steps : the determination of the security area in the
working space of the robot, and the determination of
the security region in the image.
5.1 Determination of the security area
Considering the space in which the robot has to move
without collision, a security area is determined. The
optimal space is established taking into account the
robot's dimensions and the nature of the displacement.
As long as the robot's move is a translation, the
security zone width is defined by the vehicule width.
But if rotation is considered, the obstacle free area
width must be bigger. No a priori information on
vehicle's movement nature beeing available, the width
of the security zone is defined as a circle whose
diameter is the length of the longest diagonal of the
robot (see figure 2) increased by a distance depending
on the vehicle's speed. Likewise the vertical heigth of
the security area is definied by the height of the
vehicle increased by a distance depending on the
vehicle's speed.
due to mbot's speed
0robot
static security zone
dynamic security zone
Figure 2 : security zone
of the robot
Locating the security zone in the working space is
therefore based on motion criteria since critical areas
are those crossed during the next movement. In most
whereas the bottom informs of the approximative
ground area on which the robot is moving.
Furthermore, the central zone has the highest priority
for straight movements, the rightside zone when
turning to the right, and the leftside one when turning
to the left. When the robot is travelling, current
motion is in some way given by odometric
information, but no knowledge about future motion is
available. A kinematic model of the robot is used to
obtain an estimate of this future motion. Orientation
constraints and curvature continuity due to the non
holonomy of the robot are taken into account. This
allows to establish a relation between the future dis-
placement and the previous mouvement (see figure 3).
5.2 Determination of the security region
The estimate of the security area, defined in the
robot's coordinate system is then projected in the
sensor's space. The result of the projection in the
range image is the security region. It represents the
part of the image where the presence of any obstacle
is dangerous for the vehicle and must be signaled to
the teleoperator in a very urgent manner. It will
therefore be treated with the utmost ]priority. Other
parts of the image may then be examined for obstacle
detection with priorities depending on their horizontal
proximity of the security region. When regions
corresponding to the vehicle's security has been
checked, detection of obstacles in remaining part of
the image only allows to improve the location process
accuracy and may then be interrupted ias soon as next
range image becomes available.
r\ t+l
w Fig. 5a : Robot's motion
range image (t)
securityregion I between t-1 and t is used to predict, in image t, the
zone
Figure 3 : Determination
of the security area
and the security region
6. Experiment results
The range image generator and the perception system
are written in C++ and implemented on a Sun Sparc 2
workstation. Experiments have been carried out with
different simple artificial outdoor environments. In
the case illustrated in figures 4a to 7b, the
environment is composed of two houses. Two range
images are created during the motion of a notional
robot in that scene. Distortions of the houses' shapes
due to the sensor motion can be seen (see figure 4).
During the image generation, the robot is moving in
the scene as shown in figure 5a. To keep the same
notations as in previous sections, image 4a will be
refered to as It- 1and image 4b as It.
I
Fig. 4a :Image It-1 Fig. 4b : Image It
As it is the first range image of the sequence, 1,-1 is
363
analyzed with an a priori empty KOL(t-2). No
matching or prediction can then occur, and only
obstacle detection is carried out (see figure 5b). Two
obstacles are spotted and are put in NOL(t-1).
KOL(t-1) is computed as the union of KOL(t-2) and
NOLO-I) and therefore contains the two obstacles
spotted in image It-l.
during image scanning
Fig. 5b : Obstacle detection
in image It-1
Odometric information about the robot's motion
position of the center of gravity of the obstacles
contained in KOL(t-1). Predicted positions are marked
with black crosses in figure 6a. Obstacles are then
searched for around these positions and characterized,
giving NOL(t). The obstacle domain appears as a
white dotted rectangle and computed position of the
center of gravity as a black dot in figure 6b.
Fig. 6a : Obstacle predic- Fig. 6b : Obstacle detec-
tion in image I, tion in image I,
The robot's position estimate and uncertainty are
computed by a Kalman Filter. In figure 7a, the result
of the estimation based on odometric information is
shown. The uncertainty about the robot's position is
marked with an ellipse, the uncertainty about the
robot's orientation by an angular circle in the top-right
part of the figure. Obstacle matching processing is
then carried out and the robot's position accuracy is
improved. Figure 7b shows the reduction of the
position and orientation uncertainty.
Fig. 7a : Robot's locali- Fig. 7b : Robot's locali-
zation with odometry zation with obstacles
The Kalman filter is an iterative procedure and gives
satisfactory results after several iterations. However, a
problem occurs when obstacle matching fails,
Estimation uncertainty then grows, and several steps
are necessary for the filter to converge again.
Problems in obstacle matching mainly occurs when
occlusion takes place, or when obstacles disappear,
partially or entirely, from the view field of the sensor.
However, when the range image finder is fast enough
or the robot's motion is smooth enough, simulation
results are very satisfactory.
7. Conclusion
A Range Security System for Vehicle Navigation has
been described in this paper. It is intended to assist the
teleoperator of an outdoor mobile robot by warning
him or her when obstacles may threaten the vehicle's
security. A range image simulator has been imple-
mented to simulate real laser range finders. The
perception system is based on a Kalman Filter.
Obstacles are tracked in images to improve the robot's
localization accuracy. A security area is identified in
the working space of the robot and the corresponding
security region in the range image is determined.
Detection of obstacles is then carried out, taking into
account emergency and efficiency criteria. A dynamic
segmentation of the range image is done. Regions
important for security reasons are scanned first;
afterwards, if time is available, other regions are
scanned to improve localization accuracy. The system
is implemented in C++ on a Sun Sparc 2 workstation.
Experiments have yielded promising results.
References
[Chenavier,921 Chenavier, F., Crowley, J., 1992.
Position estimation for a mobile robot using vision
and odometry. In: Proc. IEEE Int. Conf. on Robotics
and Automation,Nice, France, Vol. 3, pp. 2588-2593.
[Daily, 871 Daily M.J., Harris J.G., Reiser K.,
"Detecting obstacles in range imagery", Image Under-
standing Workshop Proc., Los Angeles, 1987,
pp. 87-97.
[Devy,94] On Autonomous Navigation in a
Natural Environment, M.Devy, R.Chatila,
Ph.Fillatreau, S.Lacroix et F.Nashashibi, Proc. of
IRS'94, Grenoble, France, July 94, pp. 213-220
[Dunlay, 881 Dunlay Terry., "Obstacle avoidance
perception processing for the autonomous land
vehicle", IEEE Int. Conf. on Robotics and automation,
Philadelphia, April 1988,vol. 2, pp. 912-917.
362
Eibert, 891 Eibert M., Lux P., Schaefer C.H.,
"Autonomous obstacle avoidance for off-road
vehicles", Proc. of an Int. Conf Intelligent
Autonomous System, Amsterdam, Dec. 1989, vol. 2,
pp. 737-747.
[Fabre, 891 Fabre P., "Exercices de reconnais-
sance des formes par ordinateur", Masson, France 89.
[Garduiio,94] Garduno M., Vachon B., "Range
image based obstacle detection during mobile robot
motion", International Journal Machine Graphics &
Vision, Poland 1994, Vol.3,pp. 309-318.
[Hebert, 891 Hebert Martial, " Building and
navigating maps of road scene using an active
sensor", IEEE Int. Conf. on Robotics and automation,
Scottsdale,Arizona (USA), May 1989, vol. 2, pp.
1136- 1142.
[IVHS, 921 "Strategic plan for intelligent vehicle
highway system in United States", Report N IVHS-
AMER-92-3, Publisherd by IVHS America, May
1992.
[Kanade, 871 Kanade T., Thorpe C., Shafer S.,
Herbert M., "Carnegie Mellon Navlab vision system",
Intelligent Autonomous Systems, Hertzberger-Groen
Editors, North-Hollandl987, pp. 681-693.
[Le Fort, 931 Le Fort N., Piat E., Ramamon
jisoa D., "Towards a co-pilot architecture based on
embedded real time expert systems", First IFAC Int.
Workshop on Intelligent Autonomous Vehicles,
Southampton,(UK), April, 1993.
[Lecceur-T,921 Lecceur-Taibi L., Ballard Ph.,
Vacherand F., " Tdl6mCtrie Laser ",Research Report
NO4 VAP Program of CNES, INRIA and LET1
laboratories, France, January 1992, pp. 15-31.
[MMC, 861 Martin Marietta Corporation, "The
autonomous land vehicle second quartely report",
MRC-84-600, Sept. 1986.
[Segovia, 93lSegovia A., Rombaut M.,
"Continuous curvature path finding for a non-
holonomic mobile robot", First IFAC Int. Workshop
on Intelligent Autonomous Vehicles, Southampton,
(UK), April, 1993,pp. 481-486.
[Veatch, 871 Veatch P.A., Davies L.S., "Range
imagery algorithms for the detection of obstacles by
autonomous vehicles", Center for Automation
Research, University of Maryland, July 1987.