Automated Driving Development

with MATLAB® and Simulink®

MANOHAR REDDY M

© 2015 The MathWorks, Inc.

1
Using Model-Based Design to develop high quality and
reliable Active Safety & Automated Driving Systems

Jonny Andersson
Senior Engineer
Scania

Thorsten Gerke
Automotive Industry Manager EMEA
MathWorks

Elektronik I Fordon Congress, 11.05.2016 (Gothenburg)

2
 1.5M km of recorded data
Scania: Model-Based Design and Code
3+ years of
Generation for AEB Sensor Fusion
driving time
12 hours re-simulation
3
Voyage Develops Longitudinal Controls
for Self-Driving Taxis
Challenge
Develop a controller that enables a self-driving car to maintain
a target velocity and keep a safe distance from obstacles.

Solution
Use Simulink® to design a longitudinal model predictive
controller. Tune parameters based on experimental data
imported into MATLAB®. Deploy the controller as an ROS
node using Robotics System Toolbox™. Generate source Voyage’s self-driving car in San Jose, California.
code with Simulink Coder™, and package it as a Docker
container. “We were searching for a prototyping solution that was
fast for development and robust for production. We
decided to go with Simulink for controller development
Results and code generation, while using MATLAB to automate
• Development speed tripled development tasks.”
• Easy integration with open-source software - Alan Mond, Voyage
• Simulink algorithms delivered as production software
4
How can you use MATLAB and Simulink to develop
automated driving algorithms?

Perception Control

Planning

5
Examples of how you can use MATLAB and Simulink to develop
automated driving algorithms

Deep learning Sensor models &

model predictive control

Perception Control

Sensor fusion
with live data Path planning

Planning

6
How can you use MATLAB and Simulink to develop
perception algorithms?
Deep learning Sensor models &
model predictive control

Perception Control

Sensor fusion
with live data Path planning

Planning

7
Automated Driving System Toolbox introduced:
Ground Truth Labeling App to label video data

10:45

Introduction to Automated
Driving System Toolbox

8
Automate labeling lanes with Ground Truth Labeler

9
 Automated Driving System Toolbox introduced examples to:
Accelerate the process of Ground Truth Labeling

▪ Label detections with ▪ Add your own automation ▪ Extend connectivity of

Ground Truth Labeler App algorithm to Ground Truth Ground Truth Labeler App
Labeler App to visualize lidar data

10
Specify attributes and sublabels in Ground Truth Labeler App

11
Automate labeling pixels with Ground Truth Labeler

12
 Learn more about developing deep learning perception algorithms
with these examples

▪ Add semantic segmentation ▪ Generate CUDA® code to execute

▪ Train free space detection directed acyclic graph network on an
network using deep learning automation algorithm to Ground Truth
Labeler App NVIDIA GPU
Computer Vision System ToolboxTM
GPU CoderTM
Automated Driving System ToolboxTM

14
Free Space Detection Using Semantic Segmentation

15
How can you use MATLAB and Simulink to develop
perception algorithms?
Deep learning Sensor models &
model predictive control

Perception Control

Sensor fusion
with live data Path planning

Planning

16
 Automated Driving System Toolbox introduced:
Multi-object tracker to develop sensor fusion algorithms

Multi-Object Tracker

Detections Track Tracking Tracks

Manager Filter

• Assigns detections to tracks • Predicts and updates state of track

• Creates new tracks
• Updates existing tracks
• Removes old tracks
• Supports linear, extended, and
unscented Kalman filters
19:27
Introduction to Automated
Driving System Toolbox

17
Automated Driving System Toolbox introduced examples to:
Develop sensor fusion algorithms with recorded data

▪ Design ▪ Generate C/C++

multi-object tracker code from algorithm
based on logged which includes a
vehicle data multi-object tracker

18
How can I test my sensor fusion algorithm with live data?

Radar

CAN CAN FD
CAN Tx
Camera

TCP/IP Ethernet
IMU

19
Test forward collision warning algorithm with live data from
vehicle
Radar

FCW application
Vision Object

CAN
Radar Object
CAN Tx CAN FD Lane
Camera CAN
Rx Read
Vehicle Speed
sensor data stream
Yaw Rate
and video stream

Video frame
TCP/IP Ethernet TCP/IP
IMU

FCW algorithm

Visualization

20
Test forward collision warning algorithm with live data from
“surrogate” vehicle

Transmitter FCW application

Vision Object
Vision Object Radar Object
Radar Object
CAN FD Lane
Lane CAN
Recorded CAN Tx
Rx Read
messages Vehicle Speed Vehicle Speed
sensor data stream
Yaw Rate Yaw Rate
and video stream

Video frame
Video frame TCP/IP Ethernet
Recorded video TCP/IP

FCW algorithm

Visualization

21
Send live CAN FD and TCP/IP data

22
Receive live CAN FD and TCP/IP data

23
Generate C/C++ code for algorithm

24
Stream live CAN FD and TCP/IP data into compiled algorithm code

25
 Learn about developing sensor fusion algorithms with live
data using this example

Receiver
Transmitter

Ethernet
Video stream

▪ Stream CAN FD data to

prototype algorithms on CAN FD
your laptop Vector CAN Interface Object Lists
VN1610 (vision/radar/imu)
Vehicle Network ToolboxTM

26
How can you use MATLAB and Simulink to develop
control algorithms?
Deep learning Sensor models &
model predictive control

Perception Control

Sensor fusion
with live data Path planning

Planning

27
Automated Driving System Toolbox introduced examples to:
Synthesize detections to test sensor fusion algorithms

▪ Synthesize radar ▪ Synthesize vision ▪ Synthesize scenario

detections with detections with to test
probabilistic impairments probabilistic impairments multi-object tracker

28
 Automated Driving System Toolbox introduced:
Radar and vision detections for closed loop simulation

Release Fuse Synthesize Visualize

multiObjectTracker radarDetectionGenerator birdsEyePlot

visionDetectionGenerator

29
Voyage Develops Longitudinal Controls
for Self-Driving Taxis
Challenge
Develop a controller that enables a self-driving car to maintain
a target velocity and keep a safe distance from obstacles.
Solution
Use Simulink® to design a longitudinal model predictive
controller. Tune parameters based on experimental data
imported into MATLAB®. Deploy the controller as an ROS
node using Robotics System Toolbox™. Generate source
code with Simulink Coder™, and package it as a Docker Voyage’s self-driving car in San Jose, California.
container.
“We were searching for a prototyping solution that was
Results fast for development and robust for production. We
• Development speed tripled decided to go with Simulink for controller development
• Easy integration with open-source software and code generation, while using MATLAB to automate
• Simulink algorithms delivered as production software development tasks.”
- Alan Mond, Voyage
Technical Article

30
Simulate closed loop system with radar/vision detections,
sensor fusion, and model-predictive control

Adaptive Cruise Control with Sensor Fusion

31
Synthesize detections to test sensor fusion and
model-predictive controller

32
Compare classical and model predictive control algorithms

33
Automated Driving Applications with Model Predictive Controls

34
Vision Detection Generator models lane detection sensor

35
 Create highway double curve with drivingScenario

▪ Driver
waypoints
simulate
distraction at
curvature
changes

36
Simulate distracted driver

37
Simulate lane keep assist at distraction events

38
Compare distracted and assisted results

▪ Detect lane departure

and maintain lane
during distraction

39
Simulate lane following by increasing minimum safe distance

40
Explore lane following results

▪ Vehicle stays within

lane boundaries

41
Graphically edit scenarios with Driving Scenario Designer

42
 Learn about synthesizing sensor detections to develop
control algorithms with these examples

▪ Simulate and ▪ Simulate and ▪ Edit roads, cuboid actors,

generate C++ for generate C++ for and sensors with
model-predictive control and model-predictive control Driving Scenario Designer App
sensor fusion algorithms with lane detections drivingScenarioDesigner
43
 Learn about modeling vehicle dynamics to develop control
algorithms with these examples

▪ Simulate vehicle ▪ Co-simulate with Unreal

dynamics for closed Engine and to set actor
loop design positions get camera image
Vehicle Dynamics BlocksetTM Vehicle Dynamics BlocksetTM

44
How can you use MATLAB and Simulink to develop
planning algorithms?
Deep learning Sensor models &
model predictive control

Perception Control

Sensor fusion
with live data Path planning

Planning

45
 Robotics System Toolbox introduced:
Connectivity with the ROS ecosystem

▪ Communicate via ROS ▪ Communication with ▪ Follow path for differential

to integrate with Gazebo to visualize and drive robot with ROS
externally authored ROS simulated system based simulator
components

46
We are investing in design and simulation of path planning for
automobiles

Rapidly-exploring Random Tree (RRT*)

47
 Learn about developing path planning algorithms
with these examples

▪ Plan path for automobile ▪ Plot map tiles using ▪ Simulate V2X
given pre-defined map World Street Map (Esri) communication to assess
Automated Driving Automated Driving channel throughput
System ToolboxTM System ToolboxTM LTE System ToolboxTM

48
Examples of how you can use MATLAB and Simulink to develop
automated driving algorithms

Deep learning Sensor models &

model predictive control

Perception Control

Sensor fusion
with live data Path planning

Planning

49
 MathWorks can help you customize MATLAB and Simulink for
your automated driving application

▪ Web based ground ▪ Lidar ground truth ▪ Lidar sensor model for
truth labeling labeling Unreal Engine
▪ Consulting project with Caterpillar ▪ Joint presentation with Autoliv ▪ Joint paper with Ford
▪ 2017 MathWorks Automotive ▪ 2018 MathWorks Automotive ▪ SAE Paper 2017-01-0107
Conference Conference (May 2nd, Plymouth MI)

50
How can we help you can use MATLAB and Simulink to develop
automated driving algorithms?

Perception Control

Planning

51
Speaker Details Contact MathWorks India
MANOHAR REDDY M Products/Training Enquiry Booth

Email: Manohar.Reddy@mathworks.in Call: 080-6632-6000

LinkedIn: https://www.linkedin.com/in/manoharreddym Email: info@mathworks.in

Your feedback is valued.

Please complete the feedback form provided to you.

52