2012 15th International IEEE Conference on Intelligent Transportation Systems

Anchorage, Alaska, USA, September 16-19, 2012

5REXVW 7UDIILF 6LJQ 5HFRJQLWLRQ DQG 7UDFNLQJ IRU $GYDQFHG 'ULYHU

$VVLVWDQFH 6\VWHPV
Zhihui Zheng, Hanxizi Zhang, Bo Wang and Zhifeng Gao

Abstract²In this paper we propose a traffic sign recognition

system using an on-board single camera for Advanced Driver
Assistance Systems (ADAS), including detection, recognition
and tracking. We combine RGB ratios based color segmentation
with automatic white balance preprocessing and
Douglas-Peucker shape detection to establish ROIs. Scale and
rotation invariant BRISK features are applied for recognition,
matching the features of the candidates to those of template
images that exist in database. Tracking-Learning-Detection
(TLD) framework is adopted to track the recognized signs in
real time to provide enough information for driver assistance
function. This paper presents lots of experiments in real driving
conditions and the results demonstrate that our system can
achieve a high detection and recognition rate, and handle large
scale changes, motion blur, perspective distortion and various
illumination conditions as well.
Figure 1. Flow chart of traffic sign detection, recognition and tracking.
I. INTRODUCTION
In this paper, we propose a real-time traffic sign detection,
Traffic Sign Recognition (TSR) is a driver support recognition and tracking system using mono-camera mounted
function which can be used to notify and warn the driver on a moving vehicle as shown in Fig.1. After color
which restrictions may be effective on the current stretch of segmentation and shape detection, the detected candidates are
road. The system can help the driver to maintain a legal speed, passed on to a template matching stage to determine the
obey local traffic instructions, or urban restrictions. The bright content of the sign. We first use a modified TLD framework to
color and simplified pictograms of traffic signs make them tackle the problems mentioned above. Section II provides an
easily comprehensible and perceivable. It appears as a very overview of related work. Section III outlines the color
attractive topic of research and many different approaches segmentation and shape detection algorithm. Section IV
have been proposed in the literatures to address traffic sign introduces the BRISK feature matching method for traffic sign
recognition. recognition, followed by the robust TLD tracking method in
Section V. Discussions and results of the proposed method are
There are five main difficulties inevitably in traffic sign
given in Section VI. Section VII provides a brief outlook on
recognition for real applications. (a) Clutter backgrounds.
future implementations.
There are interferences which are similar to traffic signs in
color or in shape in the urban environment. (b) Variable II. RELATED WORK
illumination conditions due to bad weather. (c) Perspective
distortion. Shape of a traffic sign is mutative according to the Automatic traffic sign recognition may be based on the
shooting angle. (d) Large scale changes. While the vehicle characteristic colors and shapes of the traffic signs rigidly
moves, the size of traffic signs in live images varies a lot. (e) positioned relative to the environment. Existing techniques for
ADAS needs a response in real time. traffic sign recognition utilize at least two steps, one aiming at
detection and the other one at classification.
Regarding the detection problem, the three main solutions
are color-based, shape-based and those combine both color
F. Z. H. Zheng is with the Key Laboratory of Intelligent Control and and shape. For color-based methods, a pre-segmentation is
Decision of Complex Systems, Ministry of the School of Automation, employed by a thresholding operation on a color
Beijing Institute of Technology, Beijing, China (e-mail: representation, such as RGB space[ 1 ] and its linear or
zhengzhihui009@qq.com ). non-linear transformations (HSI[2], HSV[3], CIELab[4], etc.).
S. H. X. Z. Zhang is with the Key Laboratory of Intelligent Control and
Decision of Complex Systems, Ministry of the School of Automation, Although very fast, these techniques are not able to cope with
Beijing Institute of Technology, Beijing, China (e-mail: significant color variation due to changes in illumination. For
zhxz422@163.com ). shape-based methods, most proposed techniques are based on
T. B. Wang is with the Key Laboratory of Intelligent Control and extraction of edge or corner points, followed by Hough
Decision of Complex Systems, Ministry of the School of Automation, Transform[5] or Fast Radial Symmetry Transform[6] in order
Beijing Institute of Technology, Beijing, China (e-mail:
wangbo@bit.edu.cn).
to look for several sign shapes (circular, triangular, octagonal,
F. Z. F. Gao is with the Key Laboratory of Intelligent Control and etc.). These techniques exhibit limitations when traffic signs
Decision of Complex Systems, Ministry of the School of Automation, appear in cluttered urban environment backgrounds or have
Beijing Institute of Technology, Beijing, China (e-mail: gzf@bit.edu.cn ).

978-1-4673-3063-3/12/$31.00 ©2012 IEEE 704

705

(a) red channel segmentation (b) blue channel segmentation
in Fig.2 (b) in Fig.2 (b)
(c) image with black-yellow signs (d) black channel segmentation
Figure 3. Results of color segmentation
others will be further processed in the next shape detection
stage.
C. Shape Detection
After color segmentation, we take a shape detection step to
exclude interferences with the similar color. In our proposed
system, Fast radial symmetry transform is used to detect
circles, and the Douglas-Peucker algorithm to identify triangle,
rectangle and octagon.
In[6], fast radial symmetry transform operator is adopt to
visual speed sign recognition. In our paper we use a direction
cumulative matrix l to add appropriate constraints,
excluding the effects of interfering with the target. [0°, 360°]
is divided into 8 subintervals. Every pixel in l corresponds
to a couple vector
l L> 5l á 6l á 7l á 8l á 9l á :l á ;l á
where 5l represents the votes from the first subinterval [0°,
45°], 6l represents the votes from the second subinterval
[45°, 90°] and so on. The center of a circle should receive
votes from multiple directions. As shown in Fig.4, the center
point C receives votes from 9 directions, l : ; L
>tásásátásárásás?. If the votes of a pixel are only from one of
the suELQWHUYDOV WKH SL[HO¶V ORFDWLRQ LV QRW WKH FHQWHU RI D
circle.
Figure 4. Sketch map
Douglas±Peucker algorithm: The algorithm recursively
divides the line. One example of the recursion is shown in
Fig.5. Treat A on the original curve as the first point and B as
the last point. Firstly, we connect A and B, and find the
farthest point C on the curve to line AB. Denote d as the
distance between C and AB, as shown in Fig.5 (a).
(a) (b)
Figure 5. Douglas±Peucker direction cumulative algorithm
(a) circle (b) octagon
(c) rectangle (d) triangle
Figure 6. Traffic sign detection results
If d < 0 $% LV WKH ILQDO output curve where 0 is a threshold.
,I G ! 0 UHSHDW the first step on curve AC and CB as shown in
Fig.5 (b). d1 is the distance between furthest point D and line
AC and d2 is the distance between furthest point E and line
CB.
Traverse all the points on the contour, and we can get the
approximate polygon of the contour. As this process is based
on the number of polygons borders, it has a stable output even
when traffic signs have a perspective distortion. The results
after shape detection are shown in Fig.6.
<l ? (4)
IV. TRAFFIC SIGN RECOGNITION
The proposed recognition scheme includes two steps:
BRISK features extraction and matching. Before recognition,
all the detected ROIs found in the detection step are
normalized to be the same size of 80 h 80 pixels as the
templates. Although BRISK is a scale invariant feature, the
normalization step makes the matching much more accurate.
A. Feature Extraction
In order to improve the efficiency of computation, we
employ BRISK [19] feature descriptor instead of the popular
SURF. BRISK is considered to be one of the fastest rotation
and scale invariant features currently available, whose
detection and descriptor computation is typically an order of
magnitude faster than SURF, simultaneously exhibits
competitive performance. The two main stages of the BRISK
are scale-space keypoints detection and binary bit-string
descriptors extraction.
In the keypoints detection step, we use the AGAST
detector[20] to detect keypoints in octave layers of image
pyramid as well in layer in-between, which is able to adapt the
pattern size at runtime and more applicable for changing
706
environments. Points of interest are identified in both the
image and scale dimensions using AGAST score. The exact
location and true scale of each keypoint are obtained in the
continuous domain via quadratic function fitting.
To extract the descriptor, a sampling pattern consisting of
points lying on appropriately scaled concentric circles at the
neighborhood of the keypoint is constructed. The feature
direction is determined for retrieving rotation-normalized
descriptor. Finally, the binary BRISK descriptor is obtained
from assembled pair-wise brightness comparison results. In
our experiments, instead of the pixel-wise comparison, we
employ 9×9 patch-wise comparison to reject noise. Fig. 7
shows the process of AGAST keypoints detection in scale
space. Fig. 8 is the modified BRISK sampling pattern with N =
81 patches.
Figure 7. Scale-space interest point detection
Figure 8. Sampling pattern
B. Feature Matching
In order to reduce the matching time, we divide all the
template signs into seven groups and store their BRISK
features in different databases based on their color and shape
information as presented in Table. I. During the detection of
candidate regions, we attach each region a label according to
its color and shape information. Then each candidate region
image is matched by individually comparing each feature
from the specify catalog database. The image in template
database which gives the maximum number of keypoint
matches is identified as the target class. A BRISK feature
matching output is illustrated in Fig.9.
707
(a) original image pair for matching. (b) matching result, red circles are
the AGAST keypoints detected,
and green lines are the ones
matched corretly.
Figure 9. BRISK feature matching result
Figure 10. Tracking-Learning-Detection of traffic signs combining a
short-term tracker, an object detector, and an online model.
V. TRAFFIC SIGN TRACKING
The Tracking-Learning-Detection (TLD) method was
designed for long-term tracking of arbitrary objects in
unconstrained environments. The system consists of a short
term tracker, a detector based on randomized forest, an online
model based on template matching. The object is tracked and
simultaneously learned to build a detector that supports the
tracker once it fails. The detector is built upon the information
from the first frame as well as the information provided by the
tracker.
In this paper, we extend the original TLD scheme to
multiple targets tracking, and propose a traffic signs tracking
system as illustrated in Fig.10. The detection and recognition
stages give the initial targets for the tracking framework and
the modified TLD algorithm tracks the signs robustly and
instantaneously while the vehicle moves. The Lucas-Kanade
tracker is adopted for short term tracking and tracks the signs
from the position in the previous frame. The randomized
forest detector exhaustively scans the ROIs and outputs all
promising candidates. Among the candidates provided by the
short term tracker and the detector, the best one decided by the
online object model is the target sign. Others are used as
negative samples for training both the randomized forest and
the online model. Positive samples drawn randomly around
the object position are also used for training both models.

VI. EXPERIMENTAL RESULTS
Our experimental data consists of more than two hours of
video sequences, which are acquired from a video camera
mounted on a moving vehicle. The sequences contain 649
Chinese traffic signs mentioned in Table I. in both highways
and urban environments, involving different illumination
conditions and other effects such as rotation, occlusions,
different vehicle velocities, etc. The processing computer has
a 2.8GHz CPU and 3GB of memory, and the Visual C++ 2008
integrated with OpenCV 2.3.
A. The Experiment of Traffic Signs Detection
We evaluate the performance of our system with and
without AWB and the compared detection rates are shown in
Table. III. The test samples include 1396 images in different
driving conditions and the total number of traffic signs in these
images is 1643.
TABLE III. THE RECOGNITION RATES OF THE COMPARED EXPERIMENT
System with AWB
Detection Rate 93.77%(1540/1643)
B. The Experiment of Traffic Signs Matching
We compare our traffic sign matching algorithm with the
most common features such as SIFT, SURF,ORB and BRIEF
(implemented in OpenCV 2.3) in three folds: the scale
invariant performance, the rotation invariant performance and
the computation time. The results are given in Fig.11(a)-(b),
Scale Invariantness
100
90
80
70
Matching rate(%)
which demonstrate that our matching algorithm exhibits
competitive scale and rotation invariant performance with
SIFT and SURF, while the detection and descriptor
computation speed is typically an order of magnitude faster
than SURF, which is shown in Table. IV.
C. The Experiment of Traffic Signs Tracking
In the experiments of urban environments, when the
detection and recognition of traffic signs are confirmed in
three successive frames, the initial location and recognition
information are transferred to the tracking module. Fig. 12
shows the tracking results of two individual video sequences
with our method. The top-left corner is the target area with the
trajectory drawn in red. From Fig. 12 (a)-(c) we can see that
the speed limit sign has large scale changes, various
illumination conditions and motion blur. The forbidden sign in
Fig. 12 (d)-(f) has large appearance changes, perspective
distortion, vehicle vibration and also with large scale changes.
Our system deals with those problems well.
VII. CONCLUSION
without AWB
The proposed method has excellent traffic sign detection,
73.17%(1202/1643) recognition and tracking results in challenging driving
environments with large scale changes, various illumination
conditions, motion blur and perspective distortion, etc. The
main contribution of our approach is the first use of BRISK
feature based template matching and the TLD framework in
the area of traffic sign recognition and tracking. Future work is
to implement the mentioned algorithm in embedded hardware
platforms such as TI DaVinci DSP or FPGA.
Rotation Invariantness
100
SIFT SIFT
SURF 90 SURF
ORB ORB
80
BRIEF BRIEF
our algorithm our algorithm
70
Correct matches(%)

60 60

50 50

40 40

30 30

20 20

10 10

0 0
0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 0 50 100 150 200 250 300 350
Scale Angle in radians

(a) Scale Invariant Performance (b) Rotation Invariant Performance

Figure 11. Using a given source image we generate a synthetic test data and transform images corresponding feature points. The transformation algorithm
depends on the particular test. For the rotation test case, we rotate the source image around its center for 360 degrees. For scaling, we resize the image from
0.25X to 2X size of original.

TABLE IV. DETECTION AND EXTRACTION TIMINGS COMPARISON RESULT

our
SIFT SURF
Algorithm
Keypoints Number 1648 1635 1436
Detection Time(ms) 1194.91 183.673 23.91
Descriptor Extraction Time(ms) 6128.66 205.019 24.43
Total Time(ms) 7323.57 388.692 48.34
Average Time(Each Keypoint)(ms) 4.443 0.237 0.033

708
 (a) frame 28
(d) frame 35
Figure 12. Traffic signs tracking results in real traffic scenes with various conditions such as large scale changes, motion blur, illumination changes and
appearance distortion. The yellow circles show the target with tracking confidence score and the red curves are the trajectory of the signs. The frame ID and
tracking frame rate are shown at the bottom left of the picture. The positive and negative examples learned online model are shown on the right and left most
columns respectively.
REFERENCES
[1] A. Broggi, P. Cerri, P. Medici, P. P. Porta and G. Ghisio, ³5HDO time road
signs recognition,´ in 2007 IEEE Intelligent Vehicles Symposium,
Istanbul , 2007, pp.981-986.
[2] X. Qingsong, S. Juan, and L. Tiantian, ³ $ GHWHFWLRQ DQG UHFRJQLWLRQ
method for prohibition traffic signs, ´ in 2010 IEEE International
Conference on Image Analysis and Signal Processing (IASP), Zhejiang ,
2010, pp. 583±586.
[3] < :DQJ - 'DQJ DQG = =KX ³7UDIILF 6LJQV 'HWHFWLRQ DQG Recognition
E\ ,PSURYHG 5%)11 ´ LQ Proc. International Conference on
Computational Intelligence and Security, Harbin, 2007, pp. 433±437.
[4] / ' /ySH] DQG 2 )XHQWHV ³&RORU-based road sign detection and
7UDFNLQJ ´ LQ Proc. International Conference on Image Analysis and
Recognition, montreal , 2007, pp. 1138±1147.
[5] D. Nienhuser, T. Gumpp, J. M. Zollner and K. Natroshvili, ³Fast and
reliable recognition of supplementary traffic signs,´ in 2010 IEEE
Intelligent Vehicles Symposium, California , 2010, pp.896-901.
[6] N. Barnes and A. Zelinsky, ³Real-time Radial Symmetry for Speed Sign
Detection,´ in Proceedings of the IEEE Intelligent Vehicles Symposium,
Parma, 2004, pp. 566-571.
[7] F. Zaklouta and B. Stanciulescu, ³Segmentation masks for real-time
traffic sign recognition using weighted HOG-based trees,´ in 2011 14th
International IEEE Conference on Intelligent Transportation Systems
(ITSC), Washington , 2011, pp.1955-1959.
[8] 6 .DQWDZRQJ ³5RDG 7UDIILF 6LJQV 'HWHFWLRQ DQG &ODVVLILFDWLRQ IRU
%OLQG 0DQ 1DYLJDWLRQ 6\VWHP ´ LQ Proc. Internat. Conf. on Control,
Automation and Systems (ICCAS), Seoul, 2007, pp. 847±852.
[9] Y. Zhang, C. Hong and C. Wang, ³An efficient real time rectangle speed
limit sign recognition,´ in 2010 IEEE Intelligent Vehicles Symposium,
California , 2010, pp.34-38.
[10] M. Shi, H. Wu and H. Fleyeh, ³ $ 5REXVW 0RGHO IRU 7UDIILF Signs
Recognition Based on Support Vector Machine, " in 2008 Congress on
Image and Signal Processing, Sanya , 2008, pp. 516-524.
[11] A. Ruta, Y. Li, F. Porikli, S. Watanabe, H. Kage and K. Sumi, "A new
(b) frame 75 (c) frame 134
(e) frame 125 (f) frame 153
approach for in-vehicle camera traffic sign detection and recognition, "
In Proc. IAPR Conference on Machine Vision Applications, Japan, 2009,
pp. 2-6.
[12] X. Baro, S. Escalera, J. Vitria, O. 3XMRO DQG 3 5DGHYD ³7UDIILF sign
recognition using evolutionary adaboost detection and forest-ecoc
FODVVLILFDWLRQ ´ IEEE Transactions on Intelligent Transportation
Systems, , vol. 10, pp. 113±126, March 2009.
[13] 1 %DUQHV DQG $ =HOLQVN\ ³5HDO-time radial symmetry for speed sign
GHWHFWLRQ ´ LQ 2004 IEEE Intelligent Vehicles Symposium, Parma, 2004,
pp. 566±571.
[14] B. T. Nassu and M. Ukai, ³$XWRPDWLF UHFRJQLWLRQ RI UDLOZD\ VLJQV XVLQJ
sift features,´ in 2010 IEEE Intelligent Vehicles Symposium, California ,
2010, pp.348-354.
[15] B. Hoferlin and K. Zimmermann, ³7RZDUGV UHOLDEOH WUDIILF VLJQ
recognition, ´ in 2009 IEEE Intelligent Vehicles Symposium, Xi¶an,
2009, pp. 324-329.
[16] Z. Kalal, J. Matas, K. Mikolajczyk, ³ P-N learning: bootstrapping
binary classifiers by structural constraints, ³ In 2010 IEEE Conference
Computer Vision and Pattern Recognition (CVPR), pp. 49-56.
[17] C.C. Weng, H. Chen and C.S. Fuh, ³A novel automatic white balance
method for digital still cameras ´ Image and Recognition, vol. 12, pp.
4±9, 2006.
[18] Sebanja and D. B. Megherbi , ³Automatic detection and recognition of
traffic road signs for intelligent autonomous unmanned vehicles for
urban surveillance and rescue ´ LQ 2010 IEEE Int. Conf. Technologies
for Homeland Security, pp. 132±138.
[19] L.Stefan, C. Margarita and Y. S. Roland. ³BRISK: Binary robust
invariant scalable keypoints.´ In 2011 IEEE Int.l Conf. Computer
Vision (ICCV), pp. 2548-2555.
[20] M.Elmar, D. H. Gregory, B. Dariu, S. Michael, H. Gerhard, ³ Adaptive
and generic corner detection based on the accelerated segment test, ´ In
2010 IEEE European Conference Computer Vision (ECCV), pp.
183-196.
709