A Deformable-Template Approach to Lane Detection

Karl Kluge Sridhar Lakshmanan

University of Michigan AI Lab Dept. of Elec. Eng. and Comp. Sci.
1101 Beal Avenue University of Michigan -- Dearborn
Ann Arbor, MI 48109-2110 (USA) Dearborn, MI 48128-1491 (USA)
kckluge@eecs.umich.edu lakshman @ umdsun2.umd.umich.edu

Abstract typical scene of a divided road with a double yellow

line marking the left lane edge and a white line
Vision-based algorithms for locating lane boundaries marking the right lane edge. The bottom image shows
without a prior model of the road being viewed need the intensity gradient magnitude for this image as
to be able to operate robustly under a wide variety of computed by the Prewitt edge detector [ 5 ] , with
conditions, including cases where there are large darker pixels corresponding to larger intensity
amounts of clutter in the image. This clutter can be gradients. The edges of the double yellow line near
due to shadows, puddles, oil stains, tire skid marks, the top of the image are much weaker than the shadow
etc. This poses a challenge for edge-based lane detec- edges in the same area.
tion schemes, as it is often impossible to select a gra-
dient magnitude threshold which doesn’t either
remove edges of interest corresponding to road mark-
ings and edges or include edges corresponding to
irrelevant cluttel: The approach taken in this work is
to use a deformable template model of lane structure
to locate lane boundaries without thresholding the
intensity gradient information. The Metropolis algo-
rithm is used to maximize a function which evaluates
how well the image gradient data supports a given set
of template deformation parameters. The result, the
LOIS lane detection algorithm (Likelihood Of Image
Shape), is able to detect lane markings in situations
with strong mottled shadows and broken or inter-
rupted lane markings which would pose problems for
algorithms which use local, thresholded edge infor-
mation.

1 Introduction
Vision-based location of lane boundaries can
be divided into two tasks: lane detection and lane
tracking. Lane detection is the problem of locating
lane boundaries without prior knowledge of the road
geometry. Lane tracking is the problem of tracking the
lane edges from frame to frame given an existing
model of road geometry. Lane tracking is an easier
problem than lane detection, as prior knowledge of
the road geometry permits lane tracking algorithms to
put fairly strong constraints on the likely location and
orientation of the lane edges in a new image. Lane
detection algorithms, on the other hand, have to locate Figure 1: Typical road image (top) and intensity
the lane edges without a strong model of the road gradient magnitude (bottom).
geometry, and do so in situations where there may be Section 2 describes prior approaches to
a great deal of clutter in the image. dealing with the problems posed by noise edges.
Most lane detection algorithms are edge-based Section 3 details the LOIS lane detection system,
(see the review of related work in [4]).In many road which uses a deformable template approach to permit
scenes it isn’t possible to select a threshold which lane detection using very simple image processing
eliminates noise edges without also eliminating many even in cases where there are incomplete lane
of the lane edge points of interest. This problem is markings and significant numbers of noise edges.
illustrated by Figure 1. The top image shows a fairly

54

Authorized licensed use limited to: Kookmin University. Downloaded on March 09,2010 at 20:37:23 EST from IEEE Xplore. Restrictions apply.
 Section 4 shows representative results of applying
LOIS to various road types and environments.
2 Related work
The approach used by Behringer [l] to deal
with the problem of noise edges looks for the lane
edges near the bottom of the image, where the spatial
resolution is best, using non-orientation-sensitive
edge detection. Once the lane edges are found and
their orientation estimated, they can be tracked
upwards in the image using direction-sensitive filters
in order to reduce the possibility of being misled by
distracting edges dues to shadows, etc. This approach
suffers from two problems. First, the contrast of one
or both of the lane edges may not be high enough to
detect near the bottom of the image. Second, in the
case of a broken lane marking the lane marking may
not extend all the way to the bottom of the image.
An alternative approach embodied in the
ARCADE system [4] uses robust estimation to
determine road curvature and orientation from edge
point positions and orientations without prior
grouping of the edge points into lane edges. Once the
road curvature and orientation have been found
ARCADE uses them to reduce the problem of
locating the lane edge offsets with respect to the
vehicle to the problem of segmenting a one-dimen-
sional signal. This signal is constructed by averaging
together all images pixels with the same offset
relative to the road center in order to smooth out
variations due to shadows, texture, etc. The use of
robust estimation allows ARCADE to work in cases
where up to 50% of the input edge points are noise,
but the algorithm fails if the fraction of noise edges
rises above that point.
An approach based on morphological filtering
has been used by Beucher, et al. [2][7]. This technique
uses the morphological "watershed" transformation to
locate the lane edges in the intensity gradient
magnitude image. While this technique has the
advantage of not requiring any thresholding of the
gradient magnitudes, it has the disadvantage of not
imposing any global constraints on the lane edge
shapes.
Grimmer and Lakshmanan [3], working in the
domain of locating pavement edges in millimeter
wave radar imagery, used a deformable template
approach to finding the best fit of a straight road
model with unknown width and orientation to the
radar data. The likelihood function used to judge how
well a given template shape matched the radar data
combined geometric constraints with a model of the
physics of the radar image formation. The Metropolis
algorithm was used to identify the optimal set of
template deformation parameters. Based on the
success of this work, it was decided to try to extend
this approach into the domain of lane detection in
intensity images, using the more general shape model
55
Authorized licensed use limited to: Kookmin University. Downloaded on March 09,2010 at 20:37:23 EST from IEEE Xplore. Restrictions apply.
described in [4]and a likelihood function based on
road appearance in visible wavelengths. The result of
this work is the LOIS lane detection system, described
in detail below.
3 The LOIS lane detection system
The LOIS lane detection algorithm
(Likelihood Of Image Shape) uses a deformable
template approach to the lane detection problem in
order to handle situations where the lane edges in an
image have relatively weak local contrast, or where
there are strong distracting edges due to shadows,
puddles, pavement cracks, etc. It formulates the lane
detection problem in the form of finding the set of
lane edge parameters which is most likely given a
(possibly very cluttered) image of a road scene. There
are three main components to the LOIS algorithm:
the shape model describing the lane edges,
which defines a set of shape parameters;
the likelihood function, which provides a rela-
tive measure of how well a given set of shape
parameters match the data in a particular image
of a road scene; and
the optimization algorithm used to determine
the best set of lane shape parameters for a
given image by maximizing the likelihood
function (currently the Metropolis algorithm).
Each of these components is described in
detail below. The use of a deformable template model
permits the algorithm to achieve good performance
using very simple image processing, even in the
presence of distracting shadows, texture, pavement
cracks, etc.
3.1 The model of lane shape
A common shape model used in lane detection
and tracking algorithms assumes that pavement edges
and lane markings are defined by concentric circular
arcs on a flat ground plane. Under perspective
projection these circular arcs on the ground plane
project to curves in the image plane which can be
closely approximated by curves of the form
column = k ' / ( r o w - h z ) + b ' x ( r o w - h z ) + v p
The hz parameter is the row in the image plane
corresponding to the horizon of the ground plane. The
k' parameter is linearly proportional to the curvature
of the arc on the ground plane. The vp parameter is a
function of the tangential orientation of the arc on the
ground plane, with some coupling to the arc curvature
as well (the amount of this coupling depends on the
camera tilt). The b' parameter is a function of the
offset of the arc from the camera on the ground plane,
with couplings to arc curvature and tangential orien-
tation (again, the relative contributions of these
couplings depends on the camera tilt) [4].While the
radius of curvature and tangent orientation of the left
and right lane edges will differ slightly, constraining
the left and right lane edges to have the same k’ and
v p parameters closely approximates the actual lane
edge shapes for all but very small radii of curvature.
3.2 The likelihood model
The initial likelihood function which was
chosen has proven to be fairly robust on marked
roads. It encodes the knowledge that the edges of the
lane should be near intensity gradients whose orien-
tation should be perpendicular to the lane edge. More
formally:
. Define the function Aa, x ) = I / ( 1+a x 2 )
. Let g n ( r ,c) be the gradient magnitude at
pixel ( r , C) , with gd ( r , c) being the corre-
sponding gradient direction. In the current
implementation of LOIS, the gradient magni-
tude and orientation at each pixel are computed
using the Prewitt operator [5].Other methods,
such as the Canny edge detector, could be used
if desired.
0 Define the edges of the lane in the image by the
curves
E,(s) = s,/ ( r o w - hz) + s2 x ( r o w - h z ) + s3
and
E&) = SI/ (row - hz) + s4 x (row - hz) + s3
where hz is the row in the image plane corre-
sponding to the horizon under a flat earth
assumption.
0 Define the prior likelihood function
2 algorithm with a geometric annealing schedule [6],
P(s) = -
x. x atan ( awidrh
x (s4 - s2) )
This embodies two prior constraints on the lane
geometry. The first constraint is the trivial one
that the left edge of the lane is to the left of the
right edge of the lane. This constraint is
enforced by the negative weighting the prior
function gives shape parameters which violate
it. The second constraint is that the lane is
unlikely to be very narrow. This constraint is
enforced by the increasing weight given to
wider lanes, which asymptotically approaches
1.0 at a rate which depends on awidih. The
results illustrated in this paper all use
awidth
= 10.0.
Given these definitions, the likelihood function
for a given image and a given parameter set s is
defined as
Authorized licensed use limited to: Kookmin University. Downloaded on March 09,2010 at 20:37:23 EST from IEEE Xplore. Restrictions apply.
d
cos (gd(r,c ) - atan ( z E R ( 4 )1) )
f(aorienr,
In other words, the contribution made by a
pixel to the likelihood is the gradient magnitude at
that pixel, multiplied by a function whose value
decreases as the pixel column gets further from the
lane edge and a function whose value decreases as the
gradient direction at the pixel moves farther away
from perpendicular to the tangent to the lane edge.
The parameters defining the rate of likelihood fall-off
with distance from the lane edges (a,,,,,) and with
orientation error (aorienr) are set at 0.01 and 1.13
respectively.
Figure 2 shows one dimensional sections
through the likelihood function centered on the shape
parameters draw on the image at top. While the
likelihood function tends to be fairly smooth and
convex, there can still be local maxima such as the
shallow plateau to the left of the large peak in the plot
for variable s3. As a result, a simple hill-climbing
approach is inappropriate. The current implemen-
tation of LOIS uses the Metropolis algorithm to
perform this optimization.
3.3 Optimizing the likelihood using the
Metropolis algorithm
The Metropolis algorithm is a stochastic
optimization procedure. LOIS uses the Metropolis
‘
which operates as follows:
Let L(s,) be the likelihood value correspond-
ing to the shape parameters s, in iteration N of
the algorithm;
In iteration N + 1 :
Pick a random point in the shape parame-
ter space in the neighborhood of sN,
s,+A
Calculate the “temperature” for iteration
N + 1,
( ( N + l)/maxirer)
56
 All the results shown in this paper were
generated by running the Metropolis algorithm for 40
iterations. In each iteration a small step away from the
current value is tested for each of the lane shape
parameters. The initial temperature is 10.0, and the
final temperature is 0.01.
Figure 3 illustrates the process of convergence
of the Metropolis algorithm on the image show in
Figure 1. The initial shape parameters specify a
somewhat wide straight lane centered in the image.
As the iterations of the algorithm improve the value of
the likelihood function the shape parameters deform
to match the lane markings in the image. It is able to
do this despite the presence of shadow edges with
much larger contrast than the edges of the double
yellow stripe defining the left edge of the lane.

439154 515284

-1
8-

1-

._D
l m
3!+.

Y
I\;,\-:,
0 s ! o ) , s m o )
647395

shown below each image).

703076
Figure 3: Selected Metropolis iterations showing
convergence to the lane edges (likelihood value

While the Metropolis algorithm does a better

job of .finding global maxima than hill-climbing, it
Figure 2: Plot of likelihood function for the does not always succeed. An example is shown in
image shown, varying one parameter around Figure 4. The six boxes show the output of LOIS on a
the lane shape indicated. single image using three different initial seeds for the
(Y'N;;+-ly) random number generator and either 40 iterations (left
column) or 90 iterations (right column). The dashed
Evaluate p(N+ 1) = e lines show the actual inside edges of the lane
markings in the image LOIS is working on. The solid
Select s N + , according to the following lines show LOIS' final result.
rule: The result returned by LOIS shown in the top
s ~ =+s N~+ A i f p ( N + l ) l l box of the left column has a good fit to the outside of
the lane marking on the left edge of the lane, but the
- s N +I = sN+ A with probability right lane edge is in the middle of the actual lane.
Figure 5 shows a plot of the likelihood (vertical axis)
p ( N + 1) if p ( N + 1) I 1
vs. the parameter defining the right edge location (the
s4 deformation parameter as defined in section
s N + , = sN otherwise

57

Authorized licensed use limited to: Kookmin University. Downloaded on March 09,2010 at 20:37:23 EST from IEEE Xplore. Restrictions apply.
 parameters were varied while the others remained
fixed. The output of these tests were plots like that
shown in Figure 4,showing a comparison of the LOIS
result with the actual lane edges for each of the
combinations of the pair of variables being tested.
The tests used a set of 30 images which repre-
sented challenging situations. The results of those
tests were used to select the parameters given in this
+4.0246913E+ll Seed +4.0246913E+ll Seed paper. Figure 7 shows 10 of the images with the LOIS
+4.0000000E+01 Niter +9.0000000E+01 Nlter result overlaid, illustrating the range of conditions
covered by the test set.
5 Futurework
Having identified what appear to be a fairly
robust set of parameters, our current major priority is
testing LOIS on a substantially larger data set. We
will be using a modified HMMWV as a platform for
data collection and testing (Figure 6). The HMMWV
+5.9753084E+I 1 Seed +5.9753084E+ll Seed has four forward-looking cameras to provide a wide
+4.0000000E+01 Niter +9.0000000E+01 Nlter total field of view, and a SPARC 10 processor on
+1.034215 L d L i +1.235077 Lm/Li board for running image processing algorithms.

+7.9259258E+ll Seed +7.9259258E+ll Seed

+4.0000000E+01 Niter +9.0000000E+01 Nlter
+1.023836 Lm/Li +1 .OX215 Lm/Li
Figure 4: LOIS results using different RNG
seeds and either 40 (left column) or 90 (right
column) Metropolis iterations.
3.2)(horizontal axis). The Metropolis algorithm will
sometimes fail to escape the local maximum at the
center of the plot and move to the correct value, corre-
sponding to the higher peak on the right.
m The deformable template approach used in
Figure 6: HMMWV testbed vehicle.
While the Metropolis algorithm generally
performs well as a method of finding an optimal or
close to optimal value of the likelihood function, we
plan to explore alternative methods of finding the best
shape parameters. In particular, we plan to explore the
use of branch-and-bound algorithms for optimization.
LOIS is easy to extend to fuse information from
multiple sensor modalities. This can be done by
formulating a likelihood model which uses registered
data from the different sensors to evaluate the relative
fit of a given set of shape parameters. We intend to
extend the work described here and in [3] to create
CLARK (Combined Likelihood Adding Radar
Knowledge), a system which uses both black and
white images and millimeter wave radar data to locate
lane and road boundaries. Millimeter wave radar is
able to penetrate fog, rain, and dust due to the long
wavelength used. It is our expectation that fusing
visual and millimeter wave data will produce a system
which is more robust in the presence of poor visibility
than either sensor modality by itself.

58

Authorized licensed use limited to: Kookmin University. Downloaded on March 09,2010 at 20:37:23 EST from IEEE Xplore. Restrictions apply.
 gradient information, but without the need to set a
threshold and classify pixels as edge or non-edge. A
simple likelihood function permits the algorithm to
locate lane edges even when contrast is poor or there
are many noise edges. A set of algorithm parameters
has Ibeen found which appears to work in a variety of
road enivironments. The deformable template formu-
lation easily extends to multisensor systems, and work
is in progress to add millimeter wave radar data to the
intensity data already used in LOIS.
~ ~~

complex shadows

curved roads [13 Behringer, Reinhold. “Road Recognition from

Multifocal Vision.” In Proceedings of the Intelligent
Vehicles ‘94 Symposium, pages 302-307, October
1994.
[2] Beucher, Serge; and Bilodeau, Michel. “Road
Segmentation and Obstacle Detection by a Fast
Watershed Transform.” In Proceedings of the
Intelligent Vehicles ‘94 Symposium, pages 296-301,
October 1994.
broken lane markings
[3] 6rimmer, David; and Lakshmanan, Sridhar.
“Finding Straight Edges in Radar Images Using
Deformable Templates.” In Proceedings of the 27th
Conference on Information Sciences and Systems,
1993.
[4] Kluge, Karl. “Extracting Road Curvature and
Orientation From Image Edge Points Without
Perceptual Grouping Into Features.” In Proceedings
unmarked asphalt path of the Intelligent Vehicles ‘94 Symposium, pages 109-

wet pavement and broken lane markings 6~3461-366,

Figure 7: Sample LOIIS results.
6 Conclusion
Use of a deformable template approach allows
LOIS to perform lane detection using intensity
1991.
[7] Mu, Xuan; Beucher, Serge; and Bilodeau, Michel.
“Road Tracking, Lane Segmentation and Obstacle
Recognition by Mathematical Morphology.” In
Proceedings of the Intelligent Vehicles ‘92
Symposium, pages 166-170.

59

Authorized licensed use limited to: Kookmin University. Downloaded on March 09,2010 at 20:37:23 EST from IEEE Xplore. Restrictions apply.