Applying AI to Radar and Lidar Processing

Rick Gentile Avinash Nehemiah

0
3 Things We’ll Cover Today

1
 3 Things We’ll Cover Today

Insight
AI Applications for Radar and Lidar

2
 3 Things We’ll Cover Today

▪ Data Synthesis
▪ Labeling
▪ Pre-processing
▪ Model selection and training
▪ Full system deployment

Insight Challenges
AI Applications for Radar and Lidar Common issues engineers face in practice

3
 3 Things We’ll Cover Today

▪ Data Synthesis
▪ Labeling
▪ Pre-processing
▪ Model selection and training
▪ Full system deployment

Insight Challenges Interaction

AI Applications for Radar and Lidar Common issues engineers face in practice AI models for radar and lidar data

4
What is a lidar sensor and where is AI used ?
Lidar: Light detection and ranging
▪ Creates 2D or 3D point clouds representing depth using pulsed-light
▪ Also known as 3D laser scanner, laser scanner

5
What is a lidar sensor and where is AI used ?
Lidar: Light detection and ranging
▪ Creates 2D or 3D point clouds representing depth using pulsed-light
▪ Also known as 3D laser scanner, laser scanner

Autonomous Perception Aerial Imaging and Navigation Robotics and Augmented Reality

6
What are the advantages and disadvantages of lidar sensors ?

Disadvantages of lidar sensors

• Sensitive to rain, snow and weather effects
• Measurement effected by platform movement/vibration
• Accuracy drops as range increases 7
What is a radar sensor and where is AI used ?
Radar: Radio detection and ranging
▪ Use radio frequency echos to detect objects at a distance
▪ Estimate position, Doppler, and micro-Doppler.
▪ Generate images with 4D radar

8
What is a radar sensor and where is AI used ?
Radar: Radio detection and ranging
▪ Use radio frequency echos to detect objects at a distance
▪ Estimate position, Doppler, and micro-Doppler.
▪ Generate images with 4D radar

Target classification Signal identification SAR imaging

9
What are the advantages and disadvantages of radar sensors?

Long range operations All weather, night and day Flexibility

Disadvantages of radar sensors

• Lower resolution than lidar
• Lower azimuthal resolution at longer ranges
• Multipath and clutter cause ghost detections and false detections
10
What are the common challenges engineers face using AI with radar
and lidar ?
1. Labeling recorded data for AI training is manual and time consuming

2. Little-no recorded data to train models for safety-critical applications

3. Lack of knowledge on of AI model-type and data formats best results

4. Unclear how to pre-process sensor signals for best results

5. Real-world systems require deployment of more than AI model

11
How to overcome challenges using MATLAB and Simulink examples

Lidar Detection and Tracking Tracking in the Presence of Radar Multipath

12
 Challenge
Labeling data is repetitive, manual and time consuming

Repetitive and manual Noise

Very little variation frame-frame Majority of points not required to train AI model

13
Two steps to improving accuracy and efficiency of labeling process

1. Automation using non-AI techniques

Clustering Ground Plane Removal Tracking

14
Two steps to improving accuracy and efficiency of labeling process

1. Automation using non-AI techniques

Clustering Ground Plane Removal Tracking

2. Iterative training and labeling

Train Model

I Iteration and Refinement

15
16
Labelling radar signals can also be done automatically

Automatically label signals with

custom functions
Explore and label signals with
time, frequency, and time-
frequency views

Track labelling statistics with

integrated dashboards

17
 Simulating radar data in MATLAB and Simulink

High Sensor and Environment Complexity Low

Raw IQ signals | Detections (point cloud) | Clusters | Tracks

ID 1

ID 2
Radar

Waveform-level Model Measurement-level Model

18
Simulating radar data in MATLAB and Simulink

High Sensor and Environment Complexity Low

Raw IQ signals | Detections (point cloud) | Tracks

Waveform-level Model Measurement-level Model

19
Wide range of data synthesis options for radar systems

Long distance, multi-object operations High clutter environments

1 large ship
2 small boats

Extended objects Micro-Doppler signatures 20

Simulating lidar sensor data in MATLAB and Simulink
Automated Driving Toolbox UAV Toolbox

Unreal Engine® Unreal Engine®

3D Scene Creation

RoadRunner

Cuboid Environment Cuboid Environment

21
 Challenge
Lack of knowledge on combination of model-type and data format best results

What model do I use ? How do I train a model ?

There are so many research papers. Raw sensor data or transformed.
22
MATLAB provides a curated library of models with different inputs and styles

Object Detection Semantic Segmentation

3D bounding box detection and classification Classify each data point with label

Raw Image
Curated Models Data Curated Models Data
1. PointPillars 1. SqueezeSeg v2
2. PointSeg
3. SalsaNext
4. PointNet
5. PointNet++
23
MATLAB provides a curated library of models with different inputs and styles

Object Detection Semantic Segmentation

3D bounding box detection and classification Classify each data point with label

Raw Image
Curated Models Data Curated Models Data
1. PointPillars 1. SqueezeSeg v2
2. PointSeg
3. SalsaNext
4. PointNet
5. PointNet++
24
MATLAB provides a curated library of models with different inputs and styles

Object Detection Semantic Segmentation

3D bounding box detection and classification Classify each data point with label

Curated Models Curated Models

1. PointPillars 1. SqueezeSeg v2
2. PointSeg
3. SalsaNext
4. PointNet
5. PointNet++
25
Interoperability bridges the gap between data science, engineering and
production

Data Science

26
Interoperability bridges the gap between data science, engineering and
production

Data Science

Engineering Production
27
28
Interpret models and explain network predictions

Prediction Explainer Visualization

Model-specific Interpretability

Evaluate Data Separation

29
Tune hyperparameters and reproduce training experiments

30
 Interactivity
Let’s play with an AI model for lidar in MATLAB Online

31
Pre-processing radar data can improve performance of network

32
Pre-processing radar data can improve performance of network

High
Deep learning
with I/Q signals

Dataset size / quality

Machine Learning
with many
Low

features

Less More
Domain knowledge

33
Pre-processing radar data can improve performance of network

High
Deep learning
with I/Q signals

Dataset size / quality

Time-frequency maps

Pre-processing

Machine Learning
with many
Low

features

Less More
Domain knowledge

34
Pre-processing radar data can improve performance of network

High
Deep learning
with I/Q signals

Dataset size / quality

Time-frequency maps

Pre-processing
Automated feature
extraction
Machine Learning
with many
Low

features

Less More
Domain knowledge

35
Pre-processing radar data can improve performance of network

High
Deep learning
with I/Q signals

Dataset size / quality

Time-frequency maps

Pre-processing
Automated feature
extraction
Machine Learning
with many
Low

features

Less More
Domain knowledge

Dataset size vs. domain knowledge vs. compute resources 36

You can make the trade-off between pre-processing approaches

Requires less work on front-end

May require more network tuning

I/Q Signals

37
You can make the trade-off between pre-processing approaches

Requires less work on front-end

May require more network tuning

I/Q Signals

Features extracted automatically

Less complex training network

Pre-Processing
Time-Frequency Transformation

38
You can make the trade-off between pre-processing approaches

Requires less work on front-end

May require more network tuning

I/Q Signals

Features extracted automatically

Less complex training network

Pre-Processing
Time-Frequency Transformation

Leverage domain knowledge

Less data at input layer

Feature Engineering Less complex training network

39
Interactivity: Time to test your ability to classify micro-Doppler returns …

40
Interactivity: Time to test your ability to classify micro-Doppler returns …

Ground truth – synthesized micro-Doppler

41
Interactivity: Time to test your ability to classify micro-Doppler returns …

Is this a pedestrian or a bicyclist?

Ground truth – synthesized micro-Doppler

42
Poll

Is this a pedestrian or a bicyclist?

A. One Pedestrian
B. One Bicyclist
C. One of each
D. Not sure

43
And the answer is ….

Is this a pedestrian or a bicyclist?

A. Pedestrian
B. Bicyclist
C. One of each
D. Not sure

This is a pedestrian and a bicyclist

44
This one is a bit trickier. The network gets the correct answer

Is this a pedestrian or a bicyclist?

Ground truth – synthesized micro-Doppler

45
This one is a bit trickier. The network gets the correct answer

Is this a pedestrian or a bicyclist?

Ground truth – synthesized micro-Doppler

This is two bicyclists
46
 Challenge
Deploying AI model and application code prototype to a larger system

47
 Challenge
Deploying AI model and application code prototype to a larger system

Multiple options for deployment

platform
CPU/GPU/FPGA
48
 Challenge
Deploying AI model and application code prototype to a larger system

Pre-processing AI Model Post-processing

Denoise, Ground
Object Detection JPDA object tracker
Removal

Multiple options for deployment

System requires AI model + pre and
platform
post processing
CPU/GPU/FPGA
49
50
51
52
 Reduce memory and power needs of deployed models
Quantize & Compress networks to deploy to low-power microcontrollers and FPGA’s

Choose and validate the right quantization approach to meet the required accuracy.

Deep Network Quantizer App

• Visualize the dynamic ranges of convolution layers.

• Select individual network layers to quantize.
• Asses the performance.
• Generate GPU code to deploy

53 53
We can improve our results when we fuse the two sensors

54
Let’s take a closer look …

L3

Fused tracks more accurate than individual sensor tracks

55
Let’s take a closer look …

F3

F2

56
Let’s take a closer look …

F3

F2

Fused tracks more accurate than individual sensor tracks

57
How MATLAB and Simulink help create AI-driven radar and lidar
processing systems

58
How MATLAB and Simulink help create AI-driven radar and lidar
processing systems

Labeling Automation

59
How MATLAB and Simulink help create AI-driven radar and lidar
processing systems

Labeling Automation Data Synthesis

60
How MATLAB and Simulink help create AI-driven radar and lidar
processing systems

AI Workflow
Labeling Automation Data Synthesis Pre-trained models, training,
evaluation, validation

61
How MATLAB and Simulink help create AI-driven radar and lidar
processing systems

AI Workflow
Labeling Automation Data Synthesis Pre-trained models, training,
evaluation, validation

Pre-processing
62
How MATLAB and Simulink help create AI-driven radar and lidar
processing systems

AI Workflow
Labeling Automation Data Synthesis Pre-trained models, training,
evaluation, validation

Full Application
Pre-processing
Deployment
63
Thank you

© 2021 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See mathworks.com/trademarks
for a list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders.

64