Uwe Franke, Dariu Gavrila, Steffen Gorzig, Frank Lindner, Frank Paetzslld, and Christian Wohler, DaimlerWBenz
Research
THE EARLY EIGHTIES, VISION-
based navigation systems drew on past expe-
rience gained in static-image processing.
Because of the scarcity of computational
power at that time, these systems typically
took a picture, analyzed it, and then blindly
drove some distancebefore stopping again for
the next picture. Then, in 1986, Emst Dick-
manns demonstrated autonomous driving on
hghways, using only a few 8086 processors
and Kalman filters to restrict the possible
scene interpretations to those consistent with
the considered systems' dynamics and with
previously derived interpretations.
During the last 10 years, numerous vision
systems for lateral and longitudinal vehicle
guidance, lane-departure waming, and colli-
sion avoidance have been developed all over
the world.24 For example, the Daimler-Benz
demonstrator vehicle VITA II,as shown in the
European Prometheus project's 'final presen-
tation, can drive autonomously on highways
and perform overtaking maneuvers without
any intera~tion.~
However, an even more attractive vision
system would extend its support beyond
highwaylike roads, to the everyday driver in
city traffic. Imagine a system that can imi-
tate a human driver: it not only keeps a con-
stant distance from the vehicle in front, as a
radar-based system would do, but can also
follow that vehicle laterally. Moreover, it can
stop at red traffic lights and stop signs, give
the right of way to other vehicles if neces-
40
M O S T COMPtJ'TER-!ZSION S Y S T W S FOR VEHICLE GUTDANCE
ARE FOR HIGHMAY SCENARIOS. DEVELOPING AUTONOMOUS OR
DRIVER-ASSISTANCE SYSTEMS FOR COMPLEX URBAN TRAFFIC
POSES NEWALGOHTEMIC AND SYSTEM-ARCHITECTURE
CHALLENGESe TO ADDRESS THESE ISSUES, THE AUTHORS
INTRODUCE THEIR INTELWGENT S T O P ~ GSYSTEM
O AND
DISCUSS APPROPRUTE ALGOlUTHMS AND APPROACHES FOR
VISION-MODULE CONTROL,
sary, and try to avoid unpredictable hazards, following. The driver must respond to the sys-
such as children running across the street. tem's signal to enable it to follow. If a vehicle
Intelligent Stop&Go is our first approach cannot follow the leader (for example, be-
to building such a system. It includes stereo cause the lead vehicle is changing lanes),
vision for depth-based obstacle detection and Intelligent Stop&Go must be able to drive
tracking and a framework for monocular autonomously for a limited distance while
detection and recognition of relevant ob- searching for a new leader. Also, a following
jects-without requiring a trurd-sized super- vehicle will return control to its driver if it
computer. We have limited the computa- must stop for a stop sign or if it is the first
tional power in our demonstrator car, Urban vehicle stopped at a red traffic light.
Traffic Assistant (UTA), to three 200-MHz Besides an Intelligent Stop&Go system, dri-
PowerPCs. ver-assistance systems (such as rear-end col-
lision avoidance or red-traffic-light recogni-
tion and waming) are also of interest for urban
Urban applications and vision traffic. The most important perception tasks
tasks that must be performed to build such systems
are to:
Our goal is for Intelligent Stop&Go to
detect an appropriate leading vehicle and sig- detect the leading vehicle and estimate its
nal the driver that it's ready for autonomous distance, speed, and acceleration in lon-
1094-7167/98/$10.000 1998 IEEE IEEE INTELLIGENT SYSTEMS
 (4
Figure I. Classifying pixels: (a) left and (b) right stereo image pair and (c) classified pixels far the left image.
- gitudinal and lateral directions;
extract the lane’s course, even if it lacks
well-painted markings and does not show
clothoidal geometry;
detect and recognize small traffic signs
and traffic lights in a highly colored
environment;
detect and classify different additional
traftic participants, such as bicyclists or
pedestrians; and
detect stationary obstacles that limit the
available free space, such as parked cars.
Stereo-based obstacle
detection and tracking
Navigating in urban traffic requires an
internal 3D map of the environment in front
of the car. This map must include position
and motion estimates of relevant traffic par-
ticipants and potential obstacles. In contrast
to the highway scenario, where you can con-
centrate on looking for the rear ends of lead-
ing vehicles, our system has to deal with a
large number of different objects.
Past investigations have produced several
schemes for object detection in traffic. Besides
the 2D model-based techniques that search for
rectangular, symmetric shapes, inverse-per-
spective mapping-based techniques,6optical-
flow-based approaches? and correlation-
based stereo systems8have also been tested.
The most direct method to derive 3D infor-
mation is binocular stereo vision, for which
correspondence analysis poses the key prob-
lem. Unfortunately, classical approaches
such as area-based correlation techniques or
edge-based approaches are computationally
very expensive. To overcome this problem,
we have developed a feature-based approach
that meets our specific needs and runs in real
time on a 200-MHz PowerPC 604.9
NOVEMBER/DECEMBER 1998
(b)
Our feature-based approach classifies
each pixel according to the gray values of its
four direct neighbors. It checks whether each
neighbor is much brighter, is much darker,
or has similar brightness to the considered
central pixel. This leads to 81 (34) different
classes, encoding edges and corners at dif-
ferent orientations.
For example, Figures l a and 1b show a
stereo image pair taken with our camera sys-
tem. The cameras are separated by 30 cm.
Figure IC shows the left image’s structure
classification. Different gray values repre-
sent different structures, and pixels in homo-
geneous areas are assigned to the “white”
class and ignored in the next step, corre-
spondence analysis.
The correspondence analysis works on the
feature images computed for the left and right
images, reducing the search for possibly cor-
responding pixels to a simple test of whether
two pixels belong to the same class. Thanks
to the epipolar constraint and the fact that the
cameras are mounted with parallel optical
axes, we need search for pixels with identical
classes only on corresponding image rows.
Obviously, this classification scheme can-
not guarantee uniqueness of the correspon-
dences. If ambiguities occur, we choose the
solution giving the smallest disparity-that
is, the largest distance-to overcome this
problem. This prevents wrong correspon-
dences (such as those caused by periodic
structures) from generating phantom obsta-
cles close to the camera. In addition, we
ignore measurements that violate the order-
ing constraint. The correspondence analysis
produces a disparity image (see Figure 2),
which is the basis for all subsequent steps.
Structures on the road. To recognize road
boundaries, knowing which structures in an
image lie on the road’s surface is helpful. We
(4
Figure 2. Disparity image. A feature pixel‘s darkness is
proportional to its disparity.
can easily extract such structures from the
disparity image if we can assume that the
road is (nearly) flat. If we h o w the camera
orientation, we can determine that all points
having a disparity in a certain interval around
the expected value lie on the road, and that
those with larger disparities belong to objects
above the road.
Estimation of camera height and pitch
angle. In practice, camera height and pitch
angle do not remain constant during driving.
Fortunately, we can efficiently estimate the
relevant camera parameters by using the
extracted road-surface points. Least-squares
techniques or Kalman filtering can minimize
the sum of the squared residuals between the
expected and found disparities.
Figure 3 shows the results of a test drive:
the pitch angle decreases during acceleration
and increases during braking. Between 80
and 120 seconds, the speed remained nearly
constant. The uneven road surface caused the
pitch angle’s oscillations.
Obstacle detection and tracking. If we
remove all features on the road plane, we can
generate a 2D depth map containing the re-
41

Figure 3. Estimated camera pitch angle during acceleration and deceleration.
-4 -2 0 2
Lateral position of detected objects
(meters)
Figure 4. Depth map: trees and the car cause the peaks.
m W g features. The depth map in Figure 4
covers 30 m x 8 m. The leading vehicle causes
the two peaks at 15m (&stance) at the 0 m and
-2 m lateral positions. We use this map to
detect objects that are subsequently tracked. In
each loop, already tracked objects are deleted
from the depth map before the next detection.
The detection step roughly estimates an
object’s width. This step fits a rectangular box
to the cluster of feature points that contributed
to the depth map’s extracted area. The system
tracks this cluster from frame to frame. To
estimate the obstacle’s distance, the system
computes a disparity histogram of the object’s
feature points. To obtain a dispai’ity estimate
with subpixel accuracy, the system fits a
parabola to this histogram’s maximum.
The current version can consider an arbi-
trary number of objects. Kalman filters esti-
mate the motion states (that is, speed and
Time (seconds)
acceleration in longitudinal as well as lateral
directions) of these objects based on their
positions relative to the camera system.
The longitudinal motion parameters are
the inputs for a distance controller. Figure 5
shows the results of a test drive in Esslingen,
Germany. The desired distance consists of a
safety distance of 10meters and a time head-
way of 1 second. Note the small distance er-
ror when the leader stops.
4 vehicle back to the driver’s control.
Object recognition
Essential to Intelligent Stop&Go’s intelli-
gence is its ability to recognize objects. Two
classes of objects pertain to this application:
infrastructure elements (road boundanes,
road marks, traffic signs, and traffic
lights) and
- traffic participants (pedestrians, motor-
cycles, bicycles, and vehicles).
How do we recognize various objects?
Although devising a general framework is dif-
ficult, we often fmd ourselves applying two
steps, detection and classification. The pur-
pose of detection is to efficiently obtain a
region of inter-est-that is, a region in image or
parameter space that could be associated with
apotential object. Besides the stereo approach,
we have also developed detection methods
based on shape, color, and motion. For exam-
ple, vehicles use shape and color cues to find
potential traffic signs and mows on the road,
and motion to find potential pedestrians.
Once the system obtains an ROI, it applies
more computationally intensive algorithms
to recognize the object (that is, to estimate
model parameters). In this application, ob-
jects have a wide variety of appearances
because of shape variability, different view-
ing angles, and illumination changes. Be-
cause explicit models are seldom available,
we derive models implicitly by learning from
examples. Recognition is a classification
process, and we have implemented a large
set of classifiers for this purpose-that is,
polynomial classifiers, principal-component
analysis, radial-basis functions, time-delay
neural networks (TDNN), and support-vector
machines (the latter in collaboration with the
MIT Center for Biological and Computa-
tional Learninglo).To provide test examples
for these learning approaches to object recog-
nition, we’ve recorded many hours of data
on a digital video recorder while driving in
urban traffic. We label interesting data seg-
ments offline.
Road recognition.Road recognition has two
major objectives: to enable automatic road
following and to improve lateral-control be-
havior. As a stand-alone application, road
recognition enables automatic road-follow-
ing. For example, Intelligent Stop&Go must
register when the leading vehicle makes lane
changes so that it can return the following
Furthermore, the lateral-control behavior
of Intelligent Stop&Go can be improved. As
the standard solution, a lateral tow-bar con-
troller provides lateral guidance. Intelligent
Stop&Go needs to know the vehicle’s rela-
tionship to the leading vehicle in terms of &s-
tance and angle, as measured by the stereo-
vision system. A tow-bar-controlled vehicle
approximately follows the leading vehicle’s
trajectory, but it tends to cut comers in nar-
row tums. We can minimize this undesirable
behavior if we know the following vehicle’s
position relative to the lane.
Road boundaries and markings. Lanes in
urban roads are often not as well-defied as
those inhighways. Lane boundaries often have
poor visibility. Various objects, such as traffic
participants or background infrastructure,clut-
ter the image. The road topology comprises a
variety of different lane elements, including
stop h e s , pedestsian crossings, and forbidden
zones. A comprehensive geometrical road
model cannot be readily defined. All these
characteristics suggest the need for a data-dri-
IEEE INTELLIGENT SYSTEMS
ven, global-image analysis that robustly sep-
arates lane structures from background clutter.
Global detection analyzes polygonal con-
tour images. A database organizes the poly-
gons with respect to their length, orientation,
301
25 n
Measured distance,
Desired distanct

position, and mutual spatial relations such

as parallelism and collinearity. It provides
fast filters for these attributes. We can spec-
ify arbitrary property combinations to detect
possible road structures. A polynomial clas-
sifier subsequently classifies detected lane
boundaries as curbs, markings, or clutter.
Figure 6a shows an image of a road; Figure
6b shows the results of the lane-boundary
recognition as dark lines. Regions where
obstacles have been detected by the stereo-
vision system are ruled out as possible posi- -5 I I I I I I 1 1 I I I

tions of road structures. 0 10 20 30 40 50 50 70 80 90 100

Even though the entire global detection
needs only 80 to 100ms on a PowerPC 604,
data-driven methods are computationally
quite expensive. However, as we learned
from the work on highway lane following,
model-based estimation of road geometry is
very efficient in that it needs to consider only
small portions of the image when lane mark-
ings are locally tracked in image sequences.
Our urban lane tracker combines global
analysis with this local method. For as long
as possible, it locally tracks boundaries
detected by the global analysis to estimate
the vehicle's lateral position and yaw angle
in a few milliseconds. Global detection and
local tracking work together to deal with the
dynamic environment."
Arrow recognition. The recognition of arrows
on the road follows the two-step procedure,
Time (seconds)
Figure 5. Autonomous vehicle following in city traffic. Speed is measured in meters per second.
detection and classification, mentioned earlier. butes are the area, bounding box, aspect ratio,
It uses shape and brightness cues in a region- length, and smoothness of contour, and addi-
based approach. The detection step consists of tional features that we determine based on
gray-value segmentation and filtering. computational cost.
The gray-scale segmentation involves We have developed a query language for
reducing the number of colors in the original this region database, dubbed MetaCCC,
image to a handful. In this application, we which allows queries based on attributes of
base this reduction on the minima and single regions as well as on relations between
plateaus of the gray-scale histogram. regions (for example, adjacency, proximity,
Following this gray-scale segmentation, enclosure, and collinearity). The filtering step
we apply a color-connected components thus involves formulating a query to select
(CCC) analysis to the segmented image. The candidate regions from the database.
algorithm produces a database containing The resulting set is normalized for size and
information about all regions in the seg- given as input to a radial-basis-function
mented image. Among the computed attri- (RBF) classifier. Figure 7 shows the original

Figure 6. (a) An urban road scenario and (b) its extracted polygonal edge image (light) and detected lane structures (dark).

NOVEMBER/DECEMBER 1998 43

Figure 7. (a) Street scene displaying a direction arrow, and (b) the scene with the arrow segmented and classified.
and the obtained result.
Traffic signs. The urban scenario offers :
number of challenges to traffic-sign recogni.
tion. First, we must extend the field of view
which results in lower effective image reso.
lution. At the same time, the use of wide.
angle lenses introduces significant lens dis
tortion. Traffic signs are also not viewec
head-on, as they are most of the time on high,
ways. In the city, they must be recognize(
when they are lateral to the camera and appea:
skewed in the image. We have extended Intel
ligent Stop&Go to cope with these problem:
and have added the urban-specific signs tha
are relevant for our goals to the database.
Color. Our colored-traffic-sign-recognitior
A Ibl
Figure 8. Stages of preprocessing and image matching:
(a) an image, (b) template, (c) edge image, and (d) dis-
tance-transform image.
44
system1*goes back to the Prometheus proj-
ect and was originally developed for the
highway scenario. Overall, it follows the
same steps as arrow detection-that is, color
segmentation, filtering, and classification.
The color segmentation step involves pixel
classification using a lookup table. We gen-
erated the lookup table offline in a training
process using a polynomial classifier.
The segmentation process outputs pixels
labeled “red,” “blue,” “yellow,” and “uncol-
ored.” As before, we apply the CCC algo-
rithm and formulate queries on the resulting
regions, using the MetaCCC procedure. Typ-
ical queries would include searching for red
regions of a particular shape that encloses an
uncolored region.
The next step, classification, is done with
an RBF classifier in a multistage process. We
input a color-normalized pictograph ex-
tracted from the original image at the candi-
date locations provided by the MetaCCC
procedure. The classification stages involve
color, shape, and pixel values. Integration
over time stabilizes the results.12
Gray-scale. Clearly, color information is
beneficial for traffic-sign recognition. Yet
some open issues remain regarding color seg-
mentation. The difficulty i s that there is quite
a range in which the same “true” traffic-sign
color can appear in the image, depending on
illumination conditions-that is, whether the
camera looks toward or away from the sun
and whether it’s day or night, rainy or sunny.
To determine how successful we can be with-
out color cues, we are also working on a
shape-based traffic-sign-detectionsystem for
gray-scale. This system might also be useful
for systems that cannot use color because of
cost considerations.
The gray-scale-detection system works on
edge features, rather than on region features.
It uses a hierarchical template-matching
technique based on distance t ~ a n s f o r m s . ~ ~
DT-based matching allows the matching of
arbitrary (binary) patterns in an image. These
could be circles, ellipses, or triangles, but are
also nonparameterized pattems-for exam-
ple, outlines of pedestrians.
Preprocessing involves reading a test
image (see Figure 8 4 , computing the thresh-
olded edge image (see Figure 8c), and com-
puting its chamfer image, the DT image, (see
Figure 8d). The DT image has the same size
as the binary edge image, but has nonbinary
entries: each pixel contains the image dis-
tance to the nearest edge pixel of the corre-
sponding binary edge image.
Image matching involves correlating a
binary shape pattern (for example, Figure 8b)
with the distance image. At each template
location, a correlation measure gives the
nearest-distance sum of the template points
to image edge points. A low value denotes a
good match (low dissimilarity); a zero value
denotes a perfect match. If the measure is
below a user-defined threshold, we consider
that a pattern has been detected.
The advantage of matching a template with
the DT image is that the resulting similarity
measure will be smoother as a function of the
template parameters (transformation and
shape). This enables various efficient search
algorithms to lock onto the correct solution.
It also allows more variability between a tem-
plate and an object of interest in the image.
Matching with the edge image (or the unseg-
mented gradelit image, not shown here), on
the other hand, typically provides strong peak
responses but rapidly declining off-peak
responses, which do not facilitate efficient
search strategies or allow object variability.
We have extended the basic DT-matchmg
scheme with a herarchical approach, which-
in addition to performing a coarse-to-fine
search over the translation parameters-
groups templates offline into ahierarchy based
on their similarity.This way, the approach can
simultaneously match multiple templates at
coarse search levels, resulting in significant
speedup factors. The template hierajchy can
be constructed manually or generated auto-
matically from available examples (that is,
learning). Furthermore, matching distin-
IEEE INTELLIGENT SYSTEMS
Figure 9. Traffic sign detection: (a) day and (b) night (yellow dots denote intermediate results of the hierarchical
search; final results are outlined in red).
guishes features by type and computes sepa-
rate DT’s for each type (for example, by edge
orientation).l3
Figure 9 illustrates the hierarchical ap-
proach.Yellow dots indicate where the match
between the image and a (prototype) tem-
plate of the template tree was good enough to
consider matching with more specific tem-
plates (for example, the children-crossing
sign) on a finer grid. The final detection
results are outlined in red. We achieve typi-
cal detection rates of 90% on single frames,
with 4% to 6% false positives. More than
95% of the false positives were rejected in a
subsequent pictograph-classification stage,
using an RBF network.
Traffic lights. Traffic-light recognition fob
lows the same three steps used before: color
segmentation, filtering, and classification.
Color segmentation uses a simple lookup
table to determine image parts in the traffic-
light colors: red, yellow, and green.
By applying the CCC algorithm to the seg-
mented image, the filtering step selects only
the blohlike regions whose areas lie in a cer-
tain range as possible traftic-light candidates.
NOVEMBER/DECEMBER 1998
output neurons for the K different object
classes to be distinguished the output neuron
with the highest activation denotes the class
to which the object is assigned. For traffic
lights, we have K = 2 classes-the “traffic
light” and “non-traffic-light” classes.
We trained different classifiers for red,
red-yellow, yellow, and green traffic lights.
On a 200-MHz Power PC, the algorithm runs
at a rate of approximately six images per sec-
ond, depending on the number of traffic-light
candidates. In experiments, we found recog-
nition rates of above 90%,with false positive
rates below 2%.
Pedestrians. We are developing work to rec-
ognize the most vulnerable traffic participants,
pedestrians.We can recognize pedestrians by
either their shape or their characteristicwalk-
ing pattems.14
Shape mmgnirion. We are compiling a large
set of pedestrian outlines to account for the
wide variability of pedestrian shapes in the
real world, Our first approach to pedestrian
recognition by shape has involved blurring
Figure 10. (a) Recognition of a red traffic light. Also, the
cropped and normalized ROI (b) before and (c) after pre
processingit by means of a simulated Mahowald retina.
the pedestrian outlines in the database and
performing a principal-component analysis
on the resulting data set (see Figure 11).This
results in a compact representation of the
original image data in terms of eigenvectors.
The first eigenvectors (and the mean) repre-
sent characteristic features of the pedestrian
distribution;the last eigenvectorsmostly rep-
resent noise (see Figure 12). For example,
preliminary findingshave shown that we need
fewer than 10 eigenvectors to account for
most of the variability in the pedestrian data.
To find pedestrians in a test image, we
apply a gradient operator to the image and
normalize for image encrgy. The resulting
normalized gradient image is projected onto
the pedestrian eigenspace by correlation with
the first few eigenvectors of the data set. A
threshold on the correlation value detemlnes
45
 (bl
Figure 12. Principalcomponent analysis: eigenvertors (a) 0 (mean), (b) 1, (c) 2, and (d) 25.
whether a pedestrian has been found or not.
Figure 13 shows a matching result. Besides
the coqect solution, we also obtain a false pos-
itive on a window. The laaer detection can be
rejected by considering the camera position-
ing with respect to the road. The rejection rate
can be further improved by using an additional
classification step (for example, RBF or NN)
on the eigenspace representations. Constar-
tine Papageorgiou and colleagues have
described another approach to pedestrian
recognition,1°which we are integratingin our
UTA demonstrator ina collaboration between
MIT and Daimler-Benz Research.
Morion recognition We can expand the neural
network used for traffic-light recoaiition into
the temporal dimension; the input then con-
Figure 13. Matching results: The pedestrian is correctly detected, but a false alarm occurs in the window above.
46
(cl
sists of gray-scale image sequences. This
results in a TDNN architecture with spatio-
temporal receptive fields. The receptive fields
act as filters to extract spatiotemporalfeatures
from the input image sequences. Thus, fea-
tuses are not hard-coded but learned during
training. Subsequent classification relies on
the filtered image sequences rather than on
the raw image data. Currently, we use the
TDNN for the recognition of pedestrian gait
patterns. A detection step determines the
approximateposition of possible pedestrians;
we have two methods at our disposal:
9 Color clustering on monocular images
in a combined color andpositioiafeature
space,15 A fast polynomial classifier se-
lects the clusters possibly containing a
pedestrian's legs by evaluating temporal
changes of a shape-dependent cluster
feature.
3 0 segmentation by steYeo vision. As
described previously, the stereo algorithm
determines bounding boxes circumscrib-
ing obstacles in the scene.
The TDNN then performs a final venfication
on the candidate region sequence (see Figure
14) We are examining whether the TDNN
approach can be used for segmentation-free
object and motion detection and recognition
Putting things together
In our Prometheus demonstrator VITA I1
(which enabled autonomous driving on high-
ways), we assigned each vision module to a
subset of processors in a static configuration.
The modules ran continually, even though
some of their results were of no interest for
vehicle guidance at that moment.
Using this architecturefor computer vision
in complex, downtown traffic would require
even more resources. But why look for new
traffic signs or continuously determine the
lane width while the car is stopped at a red
traffic light? When driving on a highway,
why look for pedestrians and crosswalks?
Although we were successful with the
brute-force approach, such questions reveal
some of its disadvantages:
It lacks a concept for module control-
for example, to focus resources on rele-
vant tasks in specific situations.
It does not scale for a larger number of
modules.
It does not have a uniform concept for the
interconnection of modules.
The adaptation of the system to new
IEEE INTELLIGENT SYSTEMS

(4
gure 14. Pedestrian recognition: (a) three frames of a
sequence of stereo images (only the left image of each
pair is shown) on which a 3D segmentation has been per-
formed to determine the bounding boxes shown (the
time-delay neural nefwork recognizedthe corresponding
image regions as the legs of a pedestrian); (b) a result-
ing cropped ond scaled image sequence to be fed into
the TDNN for final verification.
applications (such as enhanced cruise
control) and the integration of new vis-
ion modules usually require extensive
reimplementations.
Figure 15. (a) Objects seen by the system and (b) their visualization. The touch screen acts as the interface between developer and vision modules.
NOVEMBER/DECEMBER 1998
* Reuse of old modules can be difficult
because of nonexistent interfaces.
The growing complexity of autonomous
systems requires an architecturethat can cope
with these problems. We have pursued the ap-
proach of making an explicit distinction be-
tween the modules (obstacle detection, vehi-
cle control, and so on) and the connection of
modules to perform a specific application(for
example, autonomous stop-and-go driving).
A module interface connects each module
to the system. This lets developers concen-
trate on solving computer-vision problems
instead of worrying about how to connect
their work to the application environment.
Connecting a group of these modules forms
an application.We can implement lane keep-
ing, for example, by connecting the lane-
detection module and the laterd-vehicle-con-
trol module.
This flexible architecture allows develop-
ment of various applications without modi-
fying the modules. Moreover, the system
allows runtime connection changes. This
enables economical use of computational
resources and adaptation to the current situ-
ation. We can, for example, accomplish lane
keeping on highways and switch dynami-
cally to the Intelligent Stop&Go application
in the city.16
We have integrated all modules described
in this article into the system. Our demon-
strator UTA is able to recognize traffic lights,
traffic signs, overtaking vehicles, direction
arrows, and pedestrian crossings, and to fol-
low a leading vehicle autonomously. The sys-
tem runs on three 200-MHz AIX PowerPCs
for the computer-vision modules and a Win-
dows Pentium I1 for the visualization of the
results (see Figure 15). The next step is to
adapt the system to multiple environments
and applications, including
autonomous driving on highways,
- speed-limit assistance (warns if the dri-
ver exceeds the road’s speed limit),
9 lane-departure warning, and
enhanced cruise control (slows the vehi-
cle down when a car in front falls below
the desired speed and speeds it up again
if no obstacle is ahead).
E EXPERIENCE WE’VE GAINED
from testing Intelligent Stop&Go on urban
roads strengthens our confidence that com-
puter vision will go downtown, soon bring-
ing vision-based driver assistance and au-
tonomous driving to the city.
In this article, we have focused on com-
puter vision, but some related issues are also
of great interest for this kind of application:
9 Angle of view. We need a wide-angle lens
to see traffic lights and signs when we’re
close to them. However, the precision of
stereo-based distance measurement and
the performance of object detection and
recognition-in particular, traffic-sign
recognition-increases with the focal
47
 length. We expect that these problems can
be solved with high-resolution chips of
1,0002or 2,0002 pixels rather than with
rotatable cameras, vario-lenses, or multi-
camera systems.
Camera dynamics. A second sensor prob-
lem is the insufficient dynamic range of
the common charge-coupled device. The
new logarithmic CMOS chips (high-
dynamic-range chips) promise a way out
of this dilemma. On sunny days, these
chips will help us see structures in shad-
owed areas. At night, they’ll improve
vision when bright lights glare into the
camera and confuse the automatic cam-
era controI.
Application-specific hardware. To bring
intelligent vision systems to the market,
we also have to address appropriate hard-
ware cost. Analog chips for early vision
steps and modem FPGA (field-program-
mable gate array) technology offer pos-
sible solutions for future mobile vision
systems. We are investigating the possi-
bility of using FPGAs to speed up edge
detection and the distance transform.
In addibon to these technical issues, impor-
tant legal (that is, liability) and acceptance
issues must be resolved before our customers
can drive vehicles with an Intelligent Stop&
Go system into the city
References
1. E.D. Dickmanns and A. Zapp, “A Curvature-
Based Scheme for Improving Road Vehicle
Guidance by Computer Vision,” Proc. SPIE
Con5 Mobile Robots, 1986,pp. 161-168.
2. J. Weber et al., “New Results in Stereo-Based
Automatic Vehicle Guidance,” Proc. Inrelli-
gent Vehicles Con6 ‘95,IEEE Press, Piscat-
away, N.J., 1995,pp. 530-535.
3. U. Franke et al., “The Daimler-Benz Steering
Assistant: A Spin-off from Autonomous Dn-
ving,” Proc. Intelligent Vehicles Conf ‘94,
IEEE Press, 1994,pp. 120-126.
4. D. Pomerleau and T. Jochem, “Rapidly
Adapting Machine Vision for Automated
Vehicle Steering,”IEEE Expert, Vol. 11, No.
2,Apr. 1596,pp. 15-27.
5. B. Ulmer, “VITA 11:Active CollisionAvoid-
ance in Real Traffic,” Proc. Intelligent Vehi-
cles ConS. ’94, IEEE Press, 1994,pp. 1 4 .
48
6. A. Broggi, “Obstacle and Lane Detection on
the ARGO,” Proc. ITSC ‘97:IEEE Conf
Intelligent Transportation Systems (CD-
ROM), IEEE Press, Piscataway, N.J., 1997.
7. W. Enkelmann, “Robust Obstacle Detection
and Tracking by Motion Analysis,” Proc.
ITSC ’97:IEEE Con5 Intelligent Transpor-
tation Systems (CD-ROM),1997.
8. K. Saneyoshi, “3-D Image Recognition Sys-
tem by Means of Stereoscopy Combined with
Ordinary Image Processing,”Proc. Intelligent
Vehicles Conf ’94, IEEE Press, 1594, pp.
13-18.
9. U. Franke and I. Kutzbach, “Fast Stereo
Based Object Detection for Stop&Go Traf-
fic,”Proc. htelligenr VehiclesCont ’96,lEEE
Press, 1996,pp. 339-344.
10. C. Papageorgiou, M. Oren, andT. Poggio, “A
General Framework for Object Detection,”
Proc. Int’l Cons Computer Vision, IEEE
Press, 1998,pp. 555-562.
11, F, Paetzold and U, Franke, “Road Recogni-
tion in Urban Environment,” IEEE Conf.
Intelligent Transportation Systems, IEEE
Press, 1998, pp. 87-91
12. R. Janssen et al., “HybridApproach for Traf-
fic Sign Recognition,”Proc. Intelligent VehL-
d e s Conf, IEEE Press, 1993,pp, 390-397.
13. D. Gavrila, “Multifeature Hierarchical Tem-
plate Matching Using Distance Transforms,”
Proc. ICPR’98: Int’I Con5 Pattern Recogni-
tion,IEEE Press, 1998, pp. 439444.
14. D.M. Gavrila, “The Visual Analysis of Hu-
man Movement: A Survey,” to be published
in Computer Vision Image Understanding,
Vol. 73, No. 1,1959.
15. B. Heisele and C. Woebler, “Motion-Based
Recognition of Pedestrians,”Proc. ICPR ’98:
Int’l Cor$ Pattern Recognition, IEEE Press,
1998,pp. 1325-1330.
16. S. Gorzig and U. Franke, “ANTS: Intelligent
Vision in Urban Traffic,”IEEE Con$ Intelli-
gent Transportation Systems, IEEE Press,
1558,pp. 545-549.
Uwe Franke is a research scientist with the Daim-
ler-Benz Research Department, Esslmgen, Ger-
many, and is responsible for the Intelligent
Stop&Go project His research interests include
vision-based lane recogiiihon for autonomous
vehicles, lane-departure waming systems, sensor
fusion of vision and radar, and truck platoonmg
He received his Dip1 -1ng and PhD-both m elec-
trical engineering from the Technical University
of Aachen. Contact him at Daimler-Benz Re-
search, T728, D-70546 Stuttgart, Germany,
franke@dhag stg daimlerbenz coin
lariu Gavrila is aresearch scientist at the Daim-
er-Benz Research Center in Ulm,Germany,where
le works on vision-based systems for intelligent
tehicle navigation. His research interests include
nodel-basedobjectrecognitionin computer vision
mdtrackinghumans for applicationin virtual-real-
ity and intelligent burnan-machine interactions.
He received his MS in computer science from the
Free University,Amsterdam, and his PhD in com-
puter science lrom the University of Maryland at
College Park. Contact him at Daimler-Benz
Research, T728, D-70546 Stuttgart, Germany:
5avrila@dbag.ulm.daimlerbenz.com.
Steffen Gorzig is working on his PhD thesis at
Daimler-Benz Research. His main interests in-
clude parallel programming and multiagent sys-
tems. He received his Diploma in computer sci-
ence from the University of Stuttgart, Germany.
Contact him at Daimler-Benz Research,T728, D-
70546 Stuttgart, Germany; goerzig@dbag.stg.
daimlerbenz.com.
Frank Lindner is a research scientist at the
Daimler-Benz Research Center, Ulm. His re-
search interests include classification theory and
its applications.He received his Diploma in com-
puter science from the University of Stuttgart.
Contact him at Daimler-Benz Research,T728, D-
70546 Stuttgart, Germany; liiidner@dbag.ulm.
daimlerbenz.com.
Frank Paetzold is working on his PhD thesis at
the Daimler-Benz Research Department,Essling-
en. His research interests focus on computer vision
for autonomous vehicle navigation. He received
his Diploma in electricalengineering from the Uni-
versity of Bochum. Contact him at Daimler-Benz
Research, T728,D-70546 Stuttgart, Germany;
paetzold@dbag.stg.daimlerbenz.com.
Christian Wohler is working on his PhD thesis at
the,Daimler-BenzReseai-chCenter, Ulm. His re-
search interests include object recognition and
motion analysis on image sequences by time-delay
neural networks. He studied physics at the univer-
sities of Wurzburg, Germany, and Grenoble,
France, and received his Diploma in physics from
the Univ. of Wurzburg. Contact him at Daimler-
Benz Research, J728, D-70546 Stuttgart, Ger-
many; woehler@dhag.ulm.daimlerbenz.com.
IEEE INTELLIGENT SYSTEMS