Asia 2023

Target Classification on the Edge

Using mmWave Radar: A Novel Algorithm
and Its Real-Time Implementation on TI’s IWRL6432

Muhammet Yanik, Slobodan Jovanovic, Sandeep Rao

Radar Systems and Algorithms R&D, Texas Instruments

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 1

What We’ll Cover
• Introduction
– What/why is mmWave radar sensing?
– Why do we need target classification on the edge using mmWave radars?
• The proposed target classification algorithm details. What is the novelty of this work?
• Real-time implementation on TI’s 60GHz low-power mmWave radar IWRL6432
• Performance results of the algorithm and the real-time implementation benchmarks
• Summary and conclusions

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 2

The mmWave Sensing
• The mmWave sensing typically refers to radar applications that operate in the 60GHz
and 77GHz frequency bands.
• Why radar now?
– Technological advances => smaller size, lower cost, higher performance
– Automotive remains a key market for these sensors
– New applications: Building surveillance, factory automation, robotics, automotive, in-cabin
sensing, and more!
In-cabin
sensing
Safety bubbles

Mobile robots

Energy Smarter
efficient units devices

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 3

Need for Target Classification
• Traditional applications leverage radar strengths in target detection and tracking.
• Emerging applications require target classification as well:
Segment Use-case Requirement
Automotive Radar Road User Classification High Performance
In-Cabin Sensing Occupancy Detection and Classification High Performance, Low Power
Building Automation / Surveillance Motion Classification (people vs pets, trees etc.) Low Cost, Low Power, Edge Processing

Elderly Care Fall Detection, Activity Classification Low Cost, Low Power, Edge Processing
Gesture Recognition Automotive Dash Board, Mobile Devices, Household Low Cost/ Low Power, Edge Processing
Appliances, Kick-2-Open (Automotive Boot opener)

• Radar classification can leverage the work done in the context of Vision Main focus
– Machine Learning architectures/frameworks can be re-used of this work

• However, the nature of the signal is different

– Vision: High spatial resolution. None or limited range/velocity resolution
– Radar: High range/velocity resolution. Limited spatial resolution
– Hence, intuition on feasibility, performance, and complexity has to be built from the ground up

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 4

Resolution in Radar and µ-Doppler Signatures
Radar images are very sharp in the range-velocity domain but blurred in the spatial domain
Range-Velocity Dimension Spatial Dimension
Radar Vision

Range resolution
Velocity resolution

velocity
Spatial resolution

y
All Weather
Privacy
range
x
• Radars’ native good velocity resolution can be used to capture the unique “velocity signature” of moving targets
• Different targets (bike, pedestrian, pet, drone, bird, car, etc.) can have different “signatures.”
• Can be used in target classification/ target filtering.

The work in this tutorial relies

heavily on these Doppler signatures unique signatures

Main challenge: How do we create these signatures for

multiple objects concurrently, and how do we process them?

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 5

Proposed Solution for Target Classification
§ The end-to-end processing chain to classify multiple target objects: The detection layer generates the point cloud. The
tracker layer is run to track multiple objects. Required features are created, and the classifier is run per track.
Human

Non-Human

Features

1D Time Series
Cluster & Track µ-Doppler vs time

Non-Human

Distinguishable features can be extracted for different target Human

classes. Here: upper and lower envelopes are given as examples.

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 6

µ-Doppler Generation
§ Tracking and spectrogram generation based on referencing the radar cube by the track centroids are central to our focus.

Group tracker

Configure a
fixed target size

Extract 4 features are then extracted: The spectrum is normalized (between

Main approach: Define the spectrums (1) Normalized bandwidth power (𝑷𝑩𝑾 ) [0 1]) (to eliminate the effect of SNR)
Estimate the boundary box (2) Lower Envelope (𝑫𝒍𝒐 )
track centroid Scenario, 3 targets (3) Upper Envelope (𝑫𝒖𝒑 )
(𝑷𝑩𝑾, and 𝑯)
(4) Spectral entropy (𝑯)

For details: M. E. Yanik and S. Rao, "Radar-Based

Multiple Target Classification in Complex (𝑫𝒍𝒐 and 𝑫𝒖𝒑)
Environments Using 1D-CNN Models," IEEE Radar
Conf., San Antonio, TX, USA, 2023, pp. 1-6

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 7

 Feature Extraction – Improve Robustness
§ If a certain track has no associated points at some frames (it may exit from the scene or obscured by another track), the tracker may still
continue to track. In such scenarios, we should avoid updating the feature set because the extracted information will not be reliable.
Missing Frames Time
An example scenario

Doppler or range
Person is
obscured by trees. Corrupted
spectrum

Window Length
Features
Interpolate the missing samples
Predictions
Point-Cloud
index
Process for each allocated track ID: (uint8_t)

Track Index
Tracker List of Track IDs
Target ID Structure: DPU_uDopProc_FeatureObj

Frame
0 Track ID0 Track ID0
1 Feature

Frame
Frame

2 Feature missed and Feature If

present not recovered missed M Features
...

M Features delay line, delay line, delay line, Np==N && Predicitions
Predicitions
delayNp delayNm delayNcm Nm==0 && 0 (uint8_t)
Track 0
index For each track ID: 0
Ncm <= Nmiss 1 0 Extract 1
Tracker Output List 1 2 1 2
then Classifier predictions To Host

Linear
update track ID state, 2 3 2
Target ID X Y ...
3 4 3
0 1 ... If track ID is present 4
1 0 0
copy features to classifier_predict() 4
for active

...
1 0 ...
track IDs

...
2 copy selected features + + + linear buffer
...

...

...

3 ...
...

...
N-1
...

N-1 Np Nm Ncm N-1

Track
index
Features
If above conditions
are not met, fill
prediction delay line
Interpolation has some
artifacts. Put some
(float) Np – Number of frames the features of track ID were present with {0.5,0.5}
0
1 Nm – Number of frames the features of track ID were missed and not recovered

protection here!
2 Ncm – Number of frames the features of track ID were missed
...

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 8

Classification Model
§ In the proposed 1D-CNN architecture:
§ 2 blocks of 1D convolution, ReLU, and layer normalization layers are specified.
§ We create 16 and 32 filters for the first and second convolutional layers (with a filter size of 3), respectively.
§ To reduce the output of the convolutional layers to a single vector, a 1D global average pooling layer is added.
§ To map the output to a vector of probabilities, a fully connected layer is specified with an output size of two (matching
the number of classes), followed by a softmax layer.

Processing time of the classifier

§ On ARM M4F core @160MHz
§ Per prediction call (i.e., per track)
§ 30-frame window
§ 2-layer CNN: 3ms
§ 3-layer CNN: 15ms

Memory required by the classifier

§ Internal buffer for computations + weights
§ 2-layer CNN: 5.7KB + 7.5KB = 13.2KB
§ 3-layer CNN: 13.4KB + 32.5KB = 45.9KB

§ An optional block of 1D convolution with 64 filters, ReLU, and layer normalization layers (following the
first two blocks) is also added to compare the performance of 2 vs. 3 layers CNN models.

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 9

Data Capture and Feature Set Summary
Input the ADC raw data
Point cloud: range, angle, Doppler
Tracker output: position, velocity.
µ-Doppler generation.
Data labeling (with synchronized videos).
A labeling GUI is built in Matlab.
Model/train/evaluate the classifier
TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao
Data statistics:
§ 310 scenarios in distinct
environments.
§ Total 125816 frames (3.5hours).
§ From ~20 different human and
numerous non-human targets.
Feature set statistics:
§ 2D µ-Doppler maps
§ 4 features (time sequence)
§ 55079 Observations
§ Block length: 30 frames
§ 25448 human, 29631 non-human
Various people and non-human objects
(drone, different dogs, fans, trees,
plants, etc.) are available in the data.
10
Cross Validation and Performance Results
§ The 1st approach uses all the data mixed from different environments. Data split: 40% Train, 10% Validation, 50% Test.
§ The 2nd approach reserves some folders completely for testing. The remaining data is used for training + validation
Approach 1
Test folders for different environments
Folder 1 Folder 2 … Folder N

Training (%40)
Observation

Validation (%10)

Test (%50)

Approach 1 (All the data is mixed) Approach 2 (Reserved folders for test)
Approach 2
Test folders for different environments Mean around

different combinations
Folder 1 Folder 2 … Folder N Repeated 5 times with 0.5m x 0.5m
more SNR target size)

range
Validation (%20)
Observation

azimuth
Training (%80) Test

range
Centroid
The final classification error is only
calculated with 5-fold cross-validation.
azimuth

Summary: ~97% mean classification accuracy with only 3sec latency (2-layers 1D-CNN)

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 11

Enable Edge AI – IWRL6432 Overview
IWRL6432: Cost and power-optimized 3 RX
Cortex M4F
QSPI The target classification
radar on a chip with analog (RF, IF), digital 2 TX LIN algorithm discussed in
this work is implemented
front-end, and processing
5MHz IF 1 MB RAM SPI
UART

Synth CAN-FD on this tiny device to

(M4F+Accelerator) capability in one device.
HWA1.2

Deep
create an edge AI solution.
Sleep

§ Integrated transceivers : 3 Rx and 2 Tx.

§ Integrated frequency synthesizer: 57GHz to 64GHz
§ Processing:
§ ARM Cortex M4F MCU @ 160 MHz
§ Radar hardware accelerator (HWA) @ 80 MHz
§ Memory: 1.0 MB RAM
§ Interfaces: SPI, UART, I2C, QSPI, GPIOs
§ Power: <100mW @ 10Hz (including classifier)
§ Package options:
§ ~6.45 x 6.45 mm FCCSP (0.5mm pitch)
§ ~5 x 4.5mm WCSP (0.4mm pitch) The entire real-time implementation is architected around
Cortex M4F and HWA and fits into the 1MB of total memory.

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 12

Enable Edge AI – Implementation Details
Detailed view of the signal processing chain on IWRL6432. Each block is implemented as a DPU and partitioned across HWA and M4F.
HWA HWA HWA + M4F M4F M4F + HWA
to UART
ADC samples DOA DPU
Range DPU CFAR DPU Tracker DPU Micro Doppler DPU
from radar Per range bin: Micro
front end Per chirp and - Doppler FFT Range- CFAR Detection
Micro Doppler, Doppler,
antenna: Radar - Clutter removal azimuth Peak grouping Point Group Tracker to UART
- Azimuth and elevation FFT Interpolation in Tracker
Feature Extraction, Features,
- BPM Decoding, cube detection cloud Info Classification
- Range FFT - Sum/peak across Doppler range and azimuth Classification
matrix
- Peak across elevation results

Timing diagram example for m-Doppler computation and feature to UART to UART
extraction for three tracking objects
§ The entire end-to-end chain takes
Cfg Cfg Cfg
CPU
1 2
Feature ext. 1
3
Feature ext. 2 Feature ext. 3
only ~35ms for 3 tracks.
HWA µ-Doppler comp. HWA 1 µ-Doppler comp. HWA 2 µ-Doppler comp. HWA 3 § Each additional track requires an
time
additional ~3ms (tracker + classifier).
Highly parallelized architecture
across processing cores to
improve processing efficiency.
Target Indices
Micro Doppler DPU Micro Doppler DPU
Efficient utilization of the HWA
List List
To Host
over UART
engine and EDMA channels is

Config/Ctrl
Config/Ctrl

Config/Ctrl
Config/Ctrl

Interrupt

Interrupt
Radar cube
critical to the implementation, Micro
HWA M-Doppler Computation
making the end-to-end chain work! Data In HWA HWA
HWA MEM
Data Out
Doppler
Spectra Feature micro
Doppler features Classifier Classifier
EDMA MEM FFT engine EDMA (ping- extraction Spectra
Results

Note: See the next page for glossary. pong

buffer)

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 13

Enable Edge AI – Software Architecture
Tools Glossary
Target Tracking and Classification Application § RPMF: Radar Power
Application and
libraries Point Cloud µ-Doppler Feature SDK Tools Management Framework
Tracker Classifier (Flashing,
Generation Generation Extraction § DPM: Data Processing
Boot, …)
Management
TI Resource
Protocol stacks § DPC: Data Processing Chain
RPMF DPM DPC DPU mmWave Explorer
And Middleware (TIREX) § DPU: Data Processing Unit
§ SOC: System-on-Chip
SysConfig
Drivers and Board Peripheral SOC Peripheral Highlights:
HAL mmWaveLink TI CLANG
Drivers Drivers § Open Source OS – FreeRTOS
Compiler
§ Simplified, low memory, low
Driver Porting Layer (DPL) Code latency optimized drivers
OS FreeRTOS Composer
Kernel POSIX § Pre-integrated mmWave
FreeRTOS Kernel No RTOS Studio (CCS)
components.
ARM M4F + HWA1.2 + SOC peripherals + EVM peripherals § Plug n Play middleware
X86 Host
(Windows / Linux)
xWRL6432

Available on TI.com!
Radar Academy TIREX CCS
§ Software: MMWAVE-L-SDK
§ Evaluation module: IWRL6432BOOST

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 14

Real-Time Demo – µ-Doppler Generation
§ In the real-time demo on IWRL6432, µ-Doppler spectrograms from multiple tracks can be generated and streamed.
§ Different ML techniques can be explored using this information without putting effort into lower-level processing.
§ The micro-Doppler maps of a
human vs. non-human object:

RC helicopter

§ This feature can also help in

more use cases (pose
estimation, fall detection, etc.)
when privacy is important.

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 15

 Summary – Available on TI.com!
Summary: Using the multi-target tracker (MTT) integration, a robust µ-Doppler map generation method is built to support
classifying multiple objects concurrently (to support more realistic scenarios) with reduced complexity.

§ Reduced the complexity of

the classifier with hand-
crafted feature extraction.
§ Majority of the existing works
use μ-Doppler maps directly.

§ Enabled simultaneous
tracking and classification of
multiple targets in real-
world environments.
§ Majority of the existing works
do not address such multi-
object realistic scenarios.

Important facts:
§ The machine learning model has never
seen this specific scenario in training.

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 16

 Asia 2023

Thank You!
Questions?

TinyML Asia 2023 – M. Yanik, S. Jovanovic, S. Rao 17

 Copyright Notice
This presentation in this publication was presented as a tinyML® Asia Technical Forum. The content
reflects the opinion of the author(s) and their respective companies. The inclusion of presentations in
this publication does not constitute an endorsement by tinyML Foundation or the sponsors.

There is no copyright protection claimed by this publication. However, each presentation is the work of
the authors and their respective companies and may contain copyrighted material. As such, it is strongly
encouraged that any use reflect proper acknowledgement to the appropriate source. Any questions
regarding the use of any materials presented should be directed to the author(s) or their companies.

tinyML is a registered trademark of the tinyML Foundation.

www.tinyml.org