Professional Documents
Culture Documents
A O A A V: N PEN Pproach To Utonomous Ehicles
A O A A V: N PEN Pproach To Utonomous Ehicles
AN OPEN APPROACH TO
AUTONOMOUS VEHICLES
.................................................................................................................................................................................................................
AUTONOMOUS VEHICLES ARE AN EMERGING APPLICATION OF AUTOMOTIVE TECHNOLOGY,
BUT THEIR COMPONENTS ARE OFTEN PROPRIETARY. THIS ARTICLE INTRODUCES AN OPEN
PATH PLANNING, AND VEHICLE CONTROL. RESEARCHERS AND DEVELOPERS CAN USE THE
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.
Velodyne HDL-64e (3D Lidar)
Velodyne HDL-32e (3D Lidar)
ZMP Robocar HV
Figure 1. ZMP Robocar “HV” and examples of sensors and plug-in computers. Omni view cameras, Lidar sensors and GNSS
receivers are equipped outside of the vehicle, while single-view cameras and computers are set inside.
NOVEMBER/DECEMBER 2015 61
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.
..............................................................................................................................................................................................
COOL CHIPS
Local waypoint
Path following
Figure 2. Basic control and dataflow of algorithms. The output velocity and angle are sent as
commands to the vehicle controller.
These sensors can be connected to com- prises a set of algorithms. For instance, scene
modity network interfaces, such as Ethernet recognition requires localization, object-detec-
and USB 3.0. Note that sensors for autono- tion, and object-tracking algorithms. Path
mous vehicles are not limited to them. For planning often falls into mission and motion
example, milliwave radars and inertial meas- planning, whereas vehicle control corresponds
urement units are often preferred for autono- to path following.
mous vehicles. Figure 2 shows the algorithms’ basic con-
Regarding a plug-in computer, because trol and data flow. Here, we introduce exam-
commercial vehicles support only a DC 12-V ples of the algorithms used in our auto-
power supply, we add an inverter and battery nomous-vehicles platform.
to provide an AC 100-V power supply for
the prototype vehicle in Figure 1. Note that
Localization
the choice of a plug-in computer depends on
Localization is one of the most basic and
performance requirements for autonomous
important problems in autonomous driving.
vehicles. In our project, we started with a
Particularly in urban areas, localization preci-
high-performance workstation to test algo-
sion dominates the reliability of autonomous
rithms. Once the algorithms were developed,
driving. We use the Normal Distributions
we switched it to a mobile laptop to downsize
Transform (NDT) algorithm to solve this
the system. Now, we are aiming to use an
localization problem.2 To be precise, we use
embedded system on a chip (SoC), taking
the 3D version of NDT to perform scan
into account the production image.
matching over 3D point-cloud data and 3D
map data.3 As a result, localization can per-
Algorithms form at the order of centimeters, leveraging a
We can roughly classify autonomous-driving high-quality 3D Lidar sensor and a high-pre-
components into scene recognition, path cision 3D map. We chose the NDT algo-
planning, and vehicle control. Each class com- rithms because they can be used in 3D forms
............................................................
62 IEEE MICRO
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.
and their computation cost does not suffer with other frames on a time basis so that we
from map size (the number of points). can predict the trajectories of moving objects
Localization is also a key technique to for mission and motion planning. We use
build a 3D map. Because 3D Lidar sensors two algorithms to solve this tracking prob-
produce 3D point-cloud data in real time, if lem. Kalman Filters is used under a linear
our autonomous vehicle is localized correctly, assumption that our autonomous vehicle is
a 3D map is created and updated by register- driving at constant velocity while tracking
ing the 3D point-cloud data at every scan. moving objects.6 Its computational cost is
This is often referred to as simultaneous lightweight and suited for real-time process-
localization and mapping. ing. Particle Filters, on the other hand, can
work for nonlinear tracking scenarios, in
Object detection which both our autonomous vehicle and
Once we localize our autonomous vehicle, tracked vehicles are moving.7
we next detect objects, such as vehicles, In our platform, we use both Kalman Fil-
pedestrians, and traffic signals, to avoid acci- ters and Particle Filters, depending on the
dents and violation of traffic rules. We focus given scenario. We also apply them for track-
on moving objects (vehicles and pedestrians), ing on both the 2D (image) plane and the
although our platform can also recognize 3D (point-cloud) plane.
traffic signals and lights. We use the Deform-
able Part Models (DPM) algorithm to detect Projection and reprojection
vehicles and pedestrians.4 DPM searches and We augment scene recognition supported by
scores the Histogram of Oriented Gradients our platform with sensor fusion of a camera
features of target objects on the image cap- and a 3D Lidar sensor. To calculate the
tured from a camera.5 We chose these algo- extrinsic parameters required to make this
rithms because they scored the best numbers sensor fusion, we calibrate the camera and
in past Pattern Analysis, Statistical Modeling, the 3D Lidar sensor. We can then project the
and Computational Learning (Pascal) Visual 3D point-cloud information obtained by the
Object Classes challenges. 3D Lidar sensor onto the image captured by
Apart from image processing, we also use the camera so that we can add depth infor-
point-cloud data scanned from a 3D Lidar mation to the image and filter out the region
sensor to detect objects by Euclidean cluster- of interest of object detection.
ing. Point-cloud clustering aims to obtain the The result of object detection on the
distance to objects rather than to classify image can also be reprojected onto the 3D
them. The distance information can be used point-cloud coordinates using the same
to range and track the objects classified by extrinsic parameters. We use the reprojected
image processing. This combined approach object positions to determine the motion
with multiple sensors is often referred to as plan and, in part, the mission plan.
sensor fusion.
Operating under the assumption that our Mission planning
autonomous vehicle drives on a public road, Our mission planner is semiautonomous.
we can improve the detection rate using a 3D Under the traffic rules, we use a rule-based
map and the current position information. mechanism to autonomously assign the path
Projecting the 3D map onto the image origi- trajectory, such as for lane change, merge,
nated on the current position, we know the and passing. In more complex scenarios, such
exact road area on the image. We can there- as parking and recovering from operational
fore constrain the region of interest for image mistakes, the driver can supervise the path. In
processing to this road area so that we can either case, once the path is assigned, the local
save execution time and reduce false positives. motion planner is launched.
Our mission planner’s basic policy is that
Object tracking we drive on the cruising lane throughout the
Because we perform the object-detection route provided by a commodity navigation
algorithm on each frame of the image and application. The lane is changed only when
point-cloud data, we must associate its results our autonomous vehicle is passing the
.............................................................
NOVEMBER/DECEMBER 2015 63
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.
..............................................................................................................................................................................................
COOL CHIPS
64 IEEE MICRO
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.
Figure 3. Visualization tools for developers. Both a 3D map plane and a 2D image plane can
be displayed. The results of traffic light recognition and path generation are also displayed.
Another usage of PCL in Autoware is to ing, converting, and drawing images), which
implement the Euclidean clustering algo- are useful to construct a framework of image-
rithm for object detection. In addition, ROS processing programs. Autoware combines
uses PCL internally in many places. For OpenCV with the RViz viewer of ROS for
example, the RViz viewer provided by ROS visualization.
is a PCL application.
CUDA
OpenCV CUDA (https://developer.nvidia.com/cuda-
OpenCV (http://opencv.org) is a popular zone) is a framework for general-purpose com-
computer vision library for image processing. puting on GPUs (GPGPU). Because complex
It supports the DPM algorithm4 and the His- algorithms of autonomous driving, such as
togram of Oriented Gradients features.5 Apart NDT, DPM, and A*, are often computing
from algorithm implementation, OpenCV intensive and data parallel, their execution
provides API library functions (such as load- speeds can be improved significantly using
.............................................................
NOVEMBER/DECEMBER 2015 65
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.
..............................................................................................................................................................................................
COOL CHIPS
Figure 4. User interface tools for drivers. Smart tablets are used to input the destination of the trip, displaying a route
candidate. To present more valuable information to drivers, smart glasses and displays can be used together.
CUDA. For example, the DPM algorithm ple, it can be used with smart glasses, as shown
contains a nontrivial number of computing- in Figure 4.
intensive and data-parallel blocks that can be
significantly accelerated by CUDA.12 Because Datasets
autonomous vehicles need real-time perform- In our platform, a 3D map is required to
ance, CUDA is a strong way to speed up the localize the autonomous vehicle. As we men-
algorithms. tioned earlier, we can generate a 3D map
using Autoware. However, this turns into a
Android time-consuming routine. It should be pro-
Autoware uses Android (www.android.com) vided by mapmakers as part of general data-
applications for the human-driver interface. sets for autonomous driving. We are
The driver can search for the route using a navi- collaborating with Aisan Technology, a map-
gation map on an Android tablet (see Figure maker in Japan, for field-operational tests of
4). The result of a searched route is reflected to autonomous driving in Nagoya. Autoware is
the 3D map used in Autoware. designed to be able to use both their 3D
maps and those generated by Autoware itself.
openFrameworks This compatibility to the third-party format
Another piece of software used in Autoware allows Autoware to be deployed widely in
for the driver interface is openFrameworks industry and academia.
(www.openframeworks.cc). Although Android Simulation of autonomous driving also
applications are functional to receive an input needs datasets. Field-operational tests always
from the driver, openFrameworks provides take some risk, and they are not always effi-
creative displays to visualize the status of cient in time, so simulation is desired in most
autonomous driving to the driver. For exam- cases. Fortunately, ROS provides an excellent
............................................................
66 IEEE MICRO
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.
record-based simulation framework called and State Lattice algorithms, can run within
ROSBAG. We suggest that ROSBAG is used 50 ms, whereas the DPM algorithm still con-
primarily to test functions of autonomous sumes more than 100 to 200 ms on the GPU.
vehicles, and field-operational tests are per- This implies that our system currently exhibits
formed for a real verification. a performance bottleneck in object detection.
Autoware lets researchers and developers Assuming that the vehicle is self-driving at 40
use such structured 3D maps and record data. km/hour in urban areas and that autonomous
Simulation of autonomous driving on a 3D functions should be effective every 1 m, the
map can be easily conducted using Autoware. execution time of each real-time task must be
Samples of these datasets can also be down- less than 100 ms, although a multitasking
loaded through the Autoware website. Given environment could make the problem more
that vehicles and sensors are often expensive, complicated. If our autonomous driving sys-
simulation with the sample datasets might be tem remains as is, the performance of GPUs
preferred to on-site experiments in some cases. must improve by at least a factor of two to
meet our assumption. Application-specific
Performance requirements integrated circuit and field-programmable
Considering that cameras and Lidar sensors gate array solutions could also exist.
often operate at 10 to 100 Hz, each task of Real-time tasks other than planning, local-
autonomous driving must run under some ization, and detection are negligible in terms
timing constraints. of execution time, but some tasks are very sen-
We provide a case study of our autono- sitive to latency. For example, a vehicle-con-
mous driving system on several computers trol task in charge of steering and acceleration
employing Intel CPUs and Nvidia GPUs. On and brake stroking should run with a few-
Intel CPUs, such as the Xeon and Core i7 ser- milliseconds latency.
ies, the A* search algorithm consumes the Finally, because such throughput- and
most time, requiring a scale of seconds or latency-aware tasks are coscheduled in the
more, if the search area is large. Although same system, operating systems must be reli-
faster is better, this might not be a significant able to multitask real-time processing. This
problem given that A* search is often used for leads to the conclusion that codesign of hard-
mission planning, which launches only when ware and software is an important considera-
the path needs to change. A more significant tion for autonomous driving.
problem resides in real-time tasks, such as
motion planning, localization, and object
detection. The DPM algorithm spends more
than 1,000 ms on VGA-size images with the
T his article has introduced an open
platform for commodity autonomous
vehicles. The hardware components, includ-
default parameter setting in OpenCV, whereas ing ZMP Robocars, sensors, and computers,
the NDT and the State Lattice algorithms can can be purchased in the market. The autono-
run in several tens of milliseconds. Their exe- mous-driving algorithms are widely recog-
cution times must be reduced to drive autono- nized, and Autoware can be used as a basic
mous vehicles in the real world. software platform comprising ROS, PCL,
We implemented these algorithms using OpenCV, CUDA, Android, and openFrame-
Nvidia GPUs. As reported elsewhere,12 Nvi- works. Sample datasets, including 3D maps
dia GPUs bring 5 to 10 times performance and simulation data, are also packaged as
improvements over Intel CPUs for the DPM part of Autoware. To the best of our knowl-
algorithm. In fact, we also observed equiva- edge, this is the first autonomous-vehicles
lent effects on the A*, NDT, and State Lattice platform accessible to the public. MICRO
algorithms, although the magnitude of
improvement depends on the parameters. ....................................................................
One particular case study of our autono- References
mous driving system used a laptop composed 1. C. Urmson et al., “Autonomous Driving in
of an Intel Core i7-4710MQ CPU and an Urban Environments: Boss and the Urban
Nvidia GTX980M GPU. We demonstrated Challenge,” J. Field Robotics, vol. 25, no. 8,
that most real-time tasks, including the NDT 2008, pp. 425–466.
.............................................................
NOVEMBER/DECEMBER 2015 67
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.
..............................................................................................................................................................................................
COOL CHIPS
68 IEEE MICRO
Authorized licensed use limited to: Indian Institute of Information Technology Kottayam. Downloaded on July 30,2022 at 12:12:59 UTC from IEEE Xplore. Restrictions apply.