The Journal of China

Universities of Posts and

Telecommunications
February 2015, 22(1): 50–56
www.sciencedirect.com/science/journal/10058885 http://jcupt.xsw.bupt.cn

Traffic light detection and recognition for autonomous vehicles

Guo Mu ( ), Zhang Xinyu, Li Deyi, Zhang Tianlei, An Lifeng

Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China

Abstract
Traffic light detection and recognition is essential for autonomous driving in urban environments. A camera based
algorithm for real-time robust traffic light detection and recognition was proposed, and especially designed for autonomous
vehicles. Although the current reliable traffic light recognition algorithms operate well under way, most of them are mainly
designed for detection at a fixed position and the effect on autonomous vehicles under real-world conditions is still limited.
Some methods achieve high accuracy on autonomous vehicle, but they can’t work normally without the aid of
high-precision priori map. The authors presented a camera-based algorithm for the problem. The image processing flow can
be divided into three steps, including pre-processing, detection and recognition. Firstly, red-green-blue (RGB) color space
is converted to hue-saturation-value (HSV) as main content of pre-processing. In detection step, the transcendental color
threshold method is used for initial filterings, meanwhile, the prior knowledge is performed to scan the scene in order to
quickly establish candidate regions. For recognition, this article use histogram of oriented gradients (HOG) features and
support vector machine (SVM) as well to recognize the state of traffic light. The proposed system on our autonomous
vehicle was evaluated. With voting schemes, the proposed can provide a sufficient accuracy for autonomous vehicles in
urban enviroment.

Keywords autonomous vehicle, traffic light detection and recognition, histogram of oriented gradients

1 Introduction
Over the past few decades, lots of attempts have been
made to autonomous vehicles. Nowadays, driving on
highway with autonomous vehicles has become more and
more reliable [1], while fully autonomous driving in real
urban environment is still a tough and challenging task [2].
Robust detection and recognition of traffic lights is
essential for autonomous vehicle to take appropriate
actions on intersection in urban environment. However,
robust detection of traffic lights is not easy to be carried,
for there would be a dreadful mess of objects in an image
in which colors are similar to the one of the traffic lights,
and the shape of traffic light is so simple that it’s hard to
extract sufficient features [3]. The worse situation may be
met that the traffic lights have a variety of types, of which,
some are horizontal arrangements while some are verticals,
Received date: 14-10-2014
Corresponding author: Guo Mu, E-mail: guom08@gmail.com
DOI: 10.1016/S1005-8885(15)60624-0
and also, some are composed of only circles while some
include arrows. In cities like Tianjin, China, the traffic
lights just look like progress bars with their color
indicating stop and go and dynamic length indicating the
remaining time. Different types of traffic lights are shown
in Fig. 1.
Fig. 1 Various types of traffic lights
Human vision is seemed the only way to detect the state
of traffic lights. Almost all experimental autonomous
vehicles for urban environment are equipped with cameras
used to detect and recognize traffic lights. Some traffic
light recognition algorithms have been proposed in recent
years. Ref. [4] achieved the spotlight detection and
template matching methods to identify the traffic lights,
Issue 1 Guo Mu, et al. / Traffic light detection and recognition for autonomous vehicles
and their result is fairly accurate. But this method is
designed for non-real-time applications. Refs. [5–6] also
described methods for detecting traffic lights. But these
methods are based on fixed cameras. When applied to
autonomous vehicles where the on-vehicle camera moves
as the vehicle goes, these methods would lose efficieny.
Ref. [7] used statistic information in HSV color space to
obtain color thresholds, they use these thresholds to do
image segmentation, and then the machine learning
algorithm was applied for classifying. Ref. [8] suggested
that can detect traffic lights in a long distance, they search
centers of traffic lights by Gaussian mask, and verify
candidates of traffic lights using suggested
existence-weight map. However, these two methods can
only detect and recognize the circular traffic lights.
Besides, the attempts mentioned above are used to detect
and classify traffic lights just from onboard images without
annotated priori maps. Ref. [9] designed methods for
automatically mapping the three dimensional positions of
traffic lights, they mapped more than four thousand traffic
lights to create a priori map. Ref. [10] presented a passive
camera based pipeline for traffic light detection and
recognition, which is used by Google driveless cars and
pass several tests in real urban environment successfully.
Both methods achieved high accuracy based on prior
knowledge of traffic light location. The main drawback of
the methods is that they need vehicle localization and prior
acquisition of traffic light positions. In this article, the
authors proposed a new approach to detect and recognize
traffic lights in vertical arrangement. both circular traffic
lights and those with arrows are handled thereafter. This
approach is designed for autonomous vehicles, so we
would use on-vehicle camera to do all processing in
real-time.
The rest of this article is organized as follows. In Sect. 2,
the system architecture of adopted autonomous vehicle is
described. Main steps of the detection and recognition
system are detailed in Sect. 3. Some experiments results
are shown in Sect. 4. Conclusions and future work are
given in Sect. 5.
2 System and vehicle
In the compitition of ‘Future Challenge 2012’ in China,
two types of contests are held: 7 km urban environment
and 16 km rural road. The autonomous vehicle we
employed won both contest at all. On November 24th,
2012, we successfully completed the Beijing-Tianjin
51
highway test without any manual intervention. The total
length of the highway test section is about 112 km and the
test took 85 min, with an average speed 79.06 km/h.
During the test, autonomous vehicle accomplished 12
overtaking, 36 lane-changing, and got a maximum speed
105 km/h.
2.1 Hardware Implementation
The vehicle used in research is a reformed Hyundai
Tucson, shown in Fig. 2. The throttle, shifter, brake and
steering system are reconstructed so that we could control
them by computer. We can also manipulate the turn or
brake signals by our command to let the vehicle act like
real human. An emergency button is designed for manual
takeover so driver could take control of the vehicle at any
time we need.
Fig. 2 Autonomous vehicle of our team
Currently, three lidars, one radar and three cameras are
equipped on the vehicle for environment perception. Two
SICK LD-LRS lidar scanners are setup in the front and on
top of the vehicle to detect other vehicles, pedestrians and
other obstacles in front. One IBEO Lux-4 lidar is mounted
on top of the vehicle. We use this lidar to detect bound of
the road. One millimeter wave radar is equipped at the rear
of the vehicle and obstacles in rear area would be detected
by it. Three cameras are mounted in the front of the car,
from left to right. They all fixed inside the car in order to
avoid dust, rain or other interference factor. We use these
cameras to detect traffic lights, traffic signs, lane-marks
and vehicle’s movement trends. Sensors’ mounting
positions, functions and coverage areas are showed in
Table 1.
Three on-board 4-core i7 computers provide sufficient
processing power: One server runs our vision algorithms
while the other takes care of lidar, radar algorithms,
decision-making, control and low-level communication.
52 The Journal of China Universities of Posts and Telecommunications 2015

Electricity power is provided by two groups of lead acid mechanism and electric reconstruction, environment
cell. perception, decision making and automatic control.
Table 1 Sensors’ mounting positions, functions and coverage areas Whereas the mechanism and electric reconstruction is the
Mounting foundation of project and belongs to the hardware portion,
Sensor type Function Coverage
position the other three sections are linked serially, they constitute
Traffic lights /
Camera1 Front 30 m~90 m, 24° the architecture of software. Environment perception
signs
Camera2 Front Lane-marks 4 m~30 m, 72° section includes information gathering, image processing,
Camera3 Front Obstacles 4 m~120 m, 72°
SICK lidar1 Front Front obstacles 0 m~50 m, 180° radar and lidar signal processing. Decision making section
SICK lidar2 Top Front obstacles 0 m~50 m, 180°
IBEO Lux4 Top Road boundary 0 m~200 m, 72°
comprehensively analyzes the results provided by the
Millimeter radar Rear Rear obstacles 0 m~200 m, 72° environment perception and makes appropriate decision.
The last section, the automatic control fulfills the decision
2.2 Software architecture made by previous section by adjusting posture of the
vehicle by appropriate control method. Fig. 3. shows
The development of autonomous vehicles involves the information processing flow of the autonomous vehicles.

Fig. 3 Information processing flow

Traffic light detection and recognition is designed as part
of image processing module in the environment perception
section, as well as detection of lane-marks and traffic signs.
3 Proposed algorithm
The authors used an off-the-shelf camera (AVT Pike
F-100C) as vision sensor for detecting traffic lights. This
camera is mounted behind the windshield of our vehicle.
We label about 5 000 sample images of 13 categories for
offline training. Each sample image is converted to a
high-dimensional feature by a HOG descriptor. These
samples are then used to train a hierarchical SVM
classifier. For online processing, the pipeline can be
divided to three phases: pre-processing, detection and
recognition. Pipeline of proposed algorithm is shown in
Fig. 4.

Fig. 4 Processing pipeline of proposed algorithm

Issue 1 Guo Mu, et al. / Traffic light detection and recognition for autonomous vehicles 53

3.1 Pre-processing Then, we perform the prior knowledge to screen the

bounding boxes, so as to establish candidate regions for
The most significant feature of traffic lights is colored recognition. Three necessary conditions must be satisfied.
sensitive. However, the colors of traffic lights would vary First, whether the traffic lights are circular or arrow shaped,
in different lighting conditions caused by weather, time or the color pixels must account for more than 70% of the area
other factors while RGB color space is sensitive to the of the bounding box. Second, for a camera in a fixed
light intensity. So the color space should be converted in position on a vehicle, there is a certain relationship between
order to uniform the drift in colors. HSV color space is not the area and the position of a bounding box. As the camera
affected by the lighting changes and hue-component H is mounted on the vehicle and is always below the traffic
describes color. Thus, we first convert image from RGB to light, the larger the distance between the camera and the
HSV. traffic light, the smaller the area of the bounding box.
Likewise, the smaller the distance between the camera and
3.2 Detection the traffic light, the larger the area of the bounding box
becomes. This phenomenon is shown in Fig. 5. We analyze
The goal of detection phase is to extract candidates for
a group of images and determined the relationship between
traffic lights. We will do some initial filtering by color
the area and position of the bounding box. This relationship
information first. We manually select about 2 000 images
is used to do further filtering. Finally, there are basic
containing traffic lights, which are all taken by traffic
templates for traffic lights. For example, a red region R has
detection camera on different environment and weather
bounding box B with width l and height h. Shifting B down
conditions. Then we semi-automatically segment the color
by h and 2h creates new bounding boxes B1 and B2.
of red, yellow and green light, and set them as three
Intuitively, if R is a candidate for a traffic light, B1 and B2
training set. We assume the traffic light color’s distribution
should be mainly composed of dark pixels; otherwise the
to be a Gaussian one and trained red, yellow and green of
region would not be a qualified candidate. Similar
traffic light separately. Note that hue component of red is
restrictions could be applied to yellow and green situations.
discontinuous near 0 and 360 while hue component of
After this triple filtering, we may gain several candidates.
yellow and green is less than 200, we do a one-to-one
Before we go to the recognition phase, however, we have to
mapping to help training color. Let H m be new mapped
extend the candidate regions using the last condition
variable following:
mentioned so that the extended regions involve the entire
 H + 60; H ≤300 traffic light.
Hm =  (1)
 H − 300; H > 300
Three color’s distribution by Gaussian fitting is shown
below:
H m_R ~ N(54.2,12.4) 

H m_Y ~ N(112.6, 23.5)  (2)

H m_G ~ N(194.7, 21.6) 
We assume that if a pixel satisfies certain conditions, it
belongs to corresponding color of traffic light. So, we set
following transcendental color threshold to do color-based Fig. 5 Relationship between the area and position of traffic lights
segmentation: We need all three conditions above to filter the candidate
red; H m ∈ [41.8, 66.6], S≥100,V ≥50 regions for recognition without high-precision integrated
 yellow; H ∈ [89.1,136.1], S≥100, V ≥50
 navigation systems (INS). However, we can do some
pixel ∈  m
(3)
 green; H m ∈ [173.1, 216.3], S≥100, V ≥50 pre-mapping to get precision location of traffic lights with
other; otherwise the aid of high-precision INS. The high-precision global
where S is saturation-component, V is value-component. position system (GPS) provides accurate location and the
After color segmentation, if the detected pixels form compass provides absolute heading angle of host vehicle.
connected regions, we can perceive their bounding boxes. Based on this, we may collect images for all intersections
54 The Journal of China Universities of Posts and Telecommunications 2015

and label the traffic light pixel region with alignment to the
location and heading angle of host vehicles. When we come
to the same intersection, accurate regions for traffic lights
can be obtained by linear interpolation based on the
real-time data from high-precision INS.

3.3 Recognition

Once the candidates for traffic lights are found, we can

use an SVM trained with HOG features to confirm the
state of the traffic light. HOG features, proposed by
Dalal et al. [11] for pedestrian detection, became popular
for recognition. We choose this feature for traffic light
recognition due to its scale-invariance, local contrast
normalization, coarse spatial binning and weighted
gradient orientations. Samples are classified into 13
categories including circular, right arrow, forward arrow Fig. 6 Detection rate at different times of day
and left arrow with red, yellow, and green colors,
It’s shown that in ranges less than 60 m, the detection rate
respectively, as well as false candidates. All samples are
is higher than 90% in any time we test. The detection of
correctly labeled manually.
‘stop’ status (red) is better than ‘go’ for the red color is easy
We use a 9-bin histogram in the experiment. The traffic
to recognize than the green one. Besides, results in the
light training samples are rescaled to 40 × 80 pixel images,
afternoon are not as good as those in the morning and night.
as are some non-traffic light samples. We set the size of
This is because we come into some backlighting cases in the
block and cell to 10 × 10 and 5 × 5, separately. The stride of
afternoon which is hard for cameras to capture correct colors.
block is set to 5 × 5. Therefore, the size of the HOG feature
Besides the detection rate of intersection status, we also
set would be 3 780. We use these features to train a linear
need to distinguish arrow lights from the circular ones. The
hierarchical SVM classifier using OpenCV. The
detection rate of arrows at different time is show in Fig. 7.
parameters of the SVM classifier are optimized using cross
validation results. Well-defined SVM classifiers are save in
a file for recognition.
We use the extended regions provided by the detection
phase for recognition. Note that after applying prior
knowledge, entire traffic lights may be detected in a region
with very little other content. This is excellent for
recognition.

4 Experiments

We ran the proposed algorithm on autonomous vehicles

and made tests in a real urban environment. The computer
is equipped with Intel i7 Quad-Core processor with 8GB
Memory. The images are captured with 1 000 × 1 000
pixels at 30 frams/s.
The experiment was done without high-precision INS at
different times of the day. Therefore, the traffic light
images include varying illumination, scale and pose. The
intersection state (go/stop) detection rate is shown in
Fig. 7 Detection rate of arrows at different times of day
Fig. 6.
Issue 1 Guo Mu, et al. / Traffic light detection and recognition for autonomous vehicles 55

It can be concluded that in distances less than 40 m, we algorithm is designed mainly for autonomous vehicles.
can distinguish arrow from circular at a detection rate Experiments show that our algorithm performs well in
higher than 80%. With a voting scheme, this accuracy is accurately detecting targets and in determining the distance
sufficient for autonomous vehicle to make decision. At and time to those targets. However, the current method
distances further than 50 m, there is no distinct difference proposed here does have some drawbacks. First, the
between circular and arrow traffic lights in the image. method performs well in the daytime but not as well at
As suggested above, the traffic lights can be detected up night. The false alarm rate increases at night due to more
to 120 m, but the average guaranteed detection distance is light interference. While the method can detect both
about 40 meters within which we can detect the status of circular traffic lights and those with arrows, only the
intersection and distinguish arrows from circulars at a classical suspended, vertical traffic lights were detected.
relative high accuracy. Detection and recognition of more types of traffic lights
In distances less than 40 m, the detection result of the will meet an important area for future work.
proposed method during all time of days is shown in Table
2. Precision and recall numbers in ‘stop’ and ‘go’ status are Acknowledgements
shown in Fig. 8 and Fig. 9, respectively.
This work was supported by Natural Basic Research Program of
Table 2 Detection and recognition results of the proposed
China (91120306, 61203366).
method within 40 m
Type Correct Missing False alarm
Circular red 684 35 43 References
Circular green 921 64 14
Left arrow red 102 17 1
1. Luettel T, Himmelsbach M, Wuensche H J. Autonomous ground
Left arrow green 184 22 0
Forward arrow red 84 12 2 vehicles—Concepts and a path to the future. Proceedings of the IEEE,
Forward arrow green 204 19 1 2012,100 (Special Centennial Issue ): 1831−1839
Right arrow red 34 6 3 2. Levinson J, Askeland J, Becker J, et al. Towards fully autonomous
Right arrow green 264 31 2
driving: Systems and algorithms. Proceedings of the 2011 IEEE
Intelligent Vehicles Symposium (IVS’11), Jun 5−9, 2011, Baden-Baden,
Germany. Piscataway, NJ, USA: IEEE, 2011: 163−168
3. Buch N, Velastin S A, Orwell J. A review of computer vision techniques
for the analysis of urban traffic. IEEE Transactions on Intelligent
Transportation Systems, 2011, 12(3): 920−939
4. de Charette R, Nashashibi F. Real time visual traffic lights recognition
based on spot light detection and adaptive traffic lights templates.
Proceedings of the 2009 IEEE Intelligent Vehicles Symposium (IVS’09),
Jun 3−5, 2009, Xi’an, China. Piscataway, NJ, USA: IEEE, 2009: 358−363
5. Chung Y C, Wang J M, Chen S W. A vision-based traffic light detection
system at intersections. Journal of Taiwan Normal University:
Fig. 8 Precision and recall in ‘stop’ status
Mathematics, Science and Technology, 2002, 47(1): 67−86
6. Yung N H C, Lai A H S. An effective video analysis method for detecting
red light runners. IEEE Transactions on Vehicular Technology, 2001,
50(4): 1074−1084
7. Gong J W, Jiang Y H, Xiong G M, et al. The recognition and tracking of
traffic lights based on color segmentation and CAMSHIFT for intelligent
vehicles. Proceedings of the 2010 IEEE Intelligent Vehicles Symposium
(IVS’10), Jun 21−24, 2010, San Diego, CA USA. Piscataway, NJ, USA:
IEEE, 2010: 431−435
8. Hwang T H, Joo I H, Cho S I. . Detection of traffic lights for vision-based
car navigation system. Advances in Image and Video Technology:
Proceedings of the 1st Pacific Rim Symposium (PSIVT’06), Dec 10−13,
Fig. 9 Precision and recall in ‘go’ status
2006, Hsinchu, China. LNCS 4319. Berlin, Germany: Springer, 2006:
682−691
5 Conclusions and future work 9. Fairfield N, Urmson C. Traffic light mapping and detection. Proceedings
of the 2011 IEEE International Conference on Robotics and Automation
A camera-based algorithm for real-time robust traffic (ICRA’11), May 9−13, 2011, Shanghai, China. Piscataway, NJ, USA:
light detection and recognition was proposed. This IEEE, 2011: 5421−5426
56 The Journal of China Universities of Posts and Telecommunications
10. Levinson J, Askeland J, Dolson J, et al. Traffic light mapping,
localization, and state detection for autonomous vehicles. Proceedings of
the 2011 IEEE International Conference on Robotics and Automation
(ICRA’11), May 9−13, 2011, Shanghai, China. Piscataway, NJ, USA:
IEEE, 2011: 5784−5791
From p. 10
10. Xu L, Hou C L. An improved affine projection algorithm and its
application in acoustic echo cancellation. Proceedings of the 3rd
International Conference on Image and Signal Processing (CISP’10): Vol
9, Oct 16−18, 2010, Yantai, China. Piscataway, NJ, USA: IEEE, 2010:
4344−4348
11. Ping L, Lu R. An improved variable step-size affine projection algorithm
for narrowband interference suppression in DSSS systems. Proceedings of
the 2013 International Conference on Quality, Reliability, Risk,
From p. 23
12. Wang Z, Han Y, Lin T, et al. Virtual network embedding by exploiting
topological information. Proceedings of the IEEE Global
Communications Conference (GLOBECOM’12), Dec 3−7, 2012,
Anaheim, CA, USA. Piscataway, NJ, USA: IEEE, 2012: 2603−2608
13. Liu J, Huang T, Chen J Y, et al. A new algorithm based on the proximity
From p. 30
3. Feng J H, Yang L, Fan P Z. A new multiple access protocol based on the
active state of neighboring nodes for mobile ad hoc networks.
Proceedings of the 3rd International Workshop on Signal Design and Its
Applications in Communications (IWSDA’07), Sept 23−27, 2007,
Chengdu, China. Piscataway, NJ, USA: IEEE, 2007: 270−274
4. Yang L, Fan P Z, Hao L. Performance analysis of multi-channel slotted
ALOHA protocol based on power capture and backoff. Journal of
Southwest Jiaotong University, 2013, 48(4): 761−768 (in Chinese)
5. Yao N M, Peng Z, Zuba M, et al. Improving aloha via backoff tuning in
underwater sensor networks. Proceedings of the 6th International ICST
Conference on Communications and Networking in China
(CHINACOM’11), Aug 17−19, 2011, Harbin, China. Piscataway, NJ,
USA: IEEE, 2011: 1038−1043
2015
11. Dalal N, Triggs B. Histograms of oriented gradients for human detection.
Proceedings of the 2005 IEEE Computer Society Conference on
Computer Vision and Pattern Recognition (CVPR’05): Vol 1, Jun 20−26,
2005, San Diego, CA, USA. Los Alamitos, CA, USA: IEEE Computer
Society, 2005: 886−893
(Editor: Zhang Kexin)
Maintenance, and Safety Engineering (QR2MSE’13), Jul 15−18, 2013,
Chengdu, China. Piscataway, NJ, USA: IEEE, 2013: 2043−2046
12. Lee C H, Park P G. Optimal step-size affine projection algorithm. IEEE
Signal Processing Letters, 2012,19(7): 431−434
13. Diniz P S R. Adaptive filtering: Algorithms and practical implementation.
2nd ed. Boston, MA, USA: Kluwer Academic Publishers, 2002
14. Choi Y S, Shin H C, Song W J. Adaptive regularization matrix for affine
projection algorithm. IEEE Transactions on Circuits and Systems II:
Express Briefs, 2007, 54(12): 1087−1091
(Editor: Zhang Kexin)
principle for the virtual network embedding problem. Journal of Zhejiang
University: Science C, 2011,12(11), 910−918
14. Chowdhury N M M K, Rahman M R, Boutaba R. Virtual network
embedding with coordinated node and link mapping. Proceedings of the
28th Annual Joint Conference of the IEEE Computer and
Communications (INFOCOM’09), Apr 19−25, 2009, Rio de Janeiro,
Brazil. Piscataway, NJ, USA: IEEE, 2009: 783−791
(Editor: Zhang Kexin)
6. Zhang Z F, Wei G. A new random access mode for mobile internet.
Journal of Communications, 2003, 24(4): 9−16 (in Chinese)
7. Abdelkader T, Naik K, Nayak A, et al. Adaptive backoff scheme for
contention-based vehicular networks using fuzzy logic. Proceedings of
the IEEE International Conference on Fuzzy Systems (FUZZ-IEEE’09),
Aug 20−24, 2009, Jeju Island, Republic of Korea. Piscataway, NJ, USA:
IEEE, 2009: 1621−1626
8. Su H W, Cheng L L. Backoff algorithm of MAC protocol with flow
predicted and quality of service distinguished. Application Research of
Computers, 2013, 30(10): 3091−3095 (in Chinese)
9. Liu S F, Dang Y G, Fan Z G. The grey system theory and its application.
4nd ed. Beijing, China:Science Press, 2008 (in Chinese)
10. Harada H, Prasad R. Simulation and software radio for mobile
communications. Boston, MA, USA: Artech House, 2002
(Editor: Zhang Kexin)