進階駕駛輔助系統

解決方案設計簡介

TI 資深汽車處理器應用工程師
Rio Chan

1
Designing Deep Learning Solutions
for ADAS

2
Outline
• What is ADAS?
• What are the important peripherals for ADAS?
• What Deep Learning can do for ADAS?
• TI TDA4 ADAS design concept?
• TI TDA4 road maps?
• TI Deep Learning.
• TIDL Demos?
• What TI can support?

3
ADAS?

4
ADAS level

5
WHAT ARE THE IMPORTANT
PERIPHERALS FOR ADAS?
6
What are the important peripherals for ADAS?

TI TDA4
https://www.ti.com/product/TDA4VM

7
What are the important peripherals for ADAS?

TI Radars
https://www.ti.com/sensors/mmwave-
radar/overview.html

8
What are the important peripherals for ADAS?

TI CAN solution - TCANx

https://www.ti.com/solution/drive-assist-ecu

9
What are the important peripherals for ADAS?

TI LMGx DCDC boost TI PGAx U-Sensor AFE

https://www.ti.com/solution/automotive- https://www.ti.com/solution/ultrasonic-
mechanically-scanning-lidar park-assist-sensor
10
TDA4 ADAS DESIGN CONCEPT

11
TI Processor SDK | Single development kit for all families

TI-RTOS

Sensor
Cockpit Cluster Amplifier Powertrain Gateway Camera Radar LIDAR fusion
AM2x AM6x DRA8x TDA4x

100% code System

compatibility Safety Security Performance
12
Jacinto’s DRA benefit for “compute” system

Jacinto
DRA 13
Jacinto’s TDA4 benefit for “ADAS” system

Jacinto
TDA 14
TI Jacinto™ 7 | Heterogeneous compute platform Unique TI
innovation

4K

Choose the right core Image Signal

Processor
Vision
Accelerators
Deep Learning
Acceleration
DSP for
analytics
Multi-Media
Accelerators Display Engine
for the right job
Heterogeneous
Compute Capabilities

L3$

GPU
Optimize entire platform around ASIL-D Memory 64b ARMv8
programmer productivity on the Safety MCU System MPUs

More Processing
MPUs

Performance
Dual-core Dual-core

•
•
Jacinto™ automotive
Common high-performance chassis
High-bandwidth coherent SoC interconnect for main processor & memory
Packet-based interconnect for latency-sensitive interfaces (Ethernet, CSI)
A72 cluster A72 cluster

•
processors
Dedicated boot engine (early boot)

Offload the majority of “work” to

Dual-core Dual-core
specialized processors. Provide Safety Security A72 cluster A72 cluster RTOS
• ASIL-B capable SoC • Run-time and secure boot
tools & SW to manage complexity • Safety HW, SW & collateral • SoC-level firewalls, BIST
• ECC on all memories & buses • Dedicated crypto HWAs

Automotive connectivity 15
• CAN-FD, MOST-150, FlexRay, Ethernet AVB, USB 3.1 Dual-core Dual-core
• GbE switch, PCIe Gen4 with switch RF5 cluster RF5 cluster
Embedded Machine Learning Architectures

16
17
TDA4 ADAS Design concept
Surround view / Parking

DMS / In-cabin Sensing

Front Camera
TDA4
ADAS Domain controller

ADAS Camera mirror

Radar
18
19
20
21
22
23
24
TDA4 ROADMAP

25
Jacinto Vision Analytics Portfolio

26
TI DEEP LEARNING

27
What Deep Learning can do for ADAS?

28
Edge inference software stack on TI SoC

Model artifacts

TFLite RunTime ONNX RunTime TVM /Neo-AI-DLR

TIDL Optimizer: Layer fusion | Post Training Quantization | Range Calibration | Memory optimizations

TIDL RunTime: API | TIDL library on C7x & MMA

C7x + MMA
Cortex-A
(Deep Learning Accelerator)
29
 User Application Jacinto 7 processor
Python / C / C++

TFLite/ONNX-RT/Neo-AI-DLR
API | interpreter | scheduler

TIDL RunTime IPC TIDL Library

+ - * =
Linux OS
C7x DSP with MMA*
Open-source
ARM ARM
Cortex
Cortex
A72 A72
CPU kernels Deep learning accelerator

30
31
TI Deep Learning | SW Eco System
TIDL Importer
➢Enables development on Open frameworks
Training frameworks Model Formats ➢Model Parameters Quantization & Calibration TIDL Inference
➢Layers Merging/Fusion ➢Abstracts embedded development
TIDL Graph Compiler ➢High Efficiency Inference Implementation
➢Memory management, Network Execution
Plan/order OpenVX Node
➢Models the device & Estimates performance
Frozen-pb/ ➢Deep learning Use case development
TIDL Host Emulation
TFLite ➢PC/Host emulation of Inference for offline
across cores (IPC, Pipelining )
Validation and Accuracy benchmarking

TIDL Importer OpenVX

Framework
TIDL Graph Compiler

TIDL Inference
TIDL Host Emulation

Prototxt
Caffemodel

Deep Learning Application Execution

Training Inference
32
What Deep Learning can do for ADAS?

33
Real-time 8x 2MP camera input demo on single ISP
• TDA4VM Camera performance as below.

• 8x 2MP cameras @ 30 fps

• Feeding to single TDA4VM device via CSI-2

• All camera images pre-processed by single YouTube link: https://youtu.be/zsYbPPhO0t8

ISP in real-time
• ISP still has headroom to process 2 more
cameras

• Flexible video processing sub-system

34
DMPAC: Depth and Motion Perception Accelerator
Stereo disparity map 90 Mpix/s, <0.5W Dense optical flow 150 Mpix/s, <0.5W
Up to 2MPix Upto 2MPix

Stereo Dense
Depth Optical
Engine Flow

Object distance measurement

Collision warning,
Segmentation,
Path planning,
…. Structure From Motion Moving Object
Object Tracking
(3D) Segmentation

Stereo Depth Engine: for each location <x,y> , estimates depth by Dense Optical Flow: for each location <x,y>, estimates 2D
measuring disparity from 2 different views motion vector field given two images
Assign confidence score for each disparity output Generate confidence score for each flow vector output

35
4x 1MP @ 20 fps
disparity map per core
TDA4 EVM AND DEMOS

36
TDA4V Mid EVM Details
Baseline Hardware Kit
➢ Processor SOM
➢ Common
Baseboard
Modular Development System
➢Provides complete access to Jacinto™ 7 SoC
➢HW platform for SDK Validation
➢New SOM for each Jacinto™ 7 SoC Variant
➢On-board XDS110 JTAG for debug via CCS
➢XDS560v2 for advanced debug and trace
http://www.ti.com/tool/TMDSEMU560V2STM-U
Optional Expansion cards for application needs

Gateway Ethernet Switch Infotainment Fusion Application

Industrial Expansion Expansion Card Board

additional Available Now

expansion
cards planned
http://www.ti.com/tool/TDA4VMXEVM

37
 Jacinto™ 7 TDA4VM analytics and vision demos
Front camera analytics with TDA4 Smarter Parking with TDA4 TDA4: Multi-Camera vision processing
Real-time 8-Camera
Input to TDA4
Deep Learning
Surround View
Based Forward
with Analytics
Camera Analytics
➢ High-quality 3D Animated Car Model, overlays
➢ Semantic Segmentation and Free space ID ➢ Eight 2MP cameras feeding single TDA4VM.
for projected path and distance warning.
➢ Multi-class Object Detection ➢ All camera images pre-processed by single
➢ Transparent car view using history
➢ Custom DL networks built on TI Deep Learning ISP in real-time
➢ Object Detection, Lane ID, Parking spot ID
Libraries (TIDL) and running on C7x+MMA ➢ All code freely available in TI’s Processor SDK
using DSP based algorithms.

8MP Front Camera Auto Valet Park DesignCore Rugged

Perception and and Surround Vision Platform for
Localization View TDA4VM

➢ AVP with Semantic segmentation, object and ➢ Ruggedized Development platform for
➢ NCAP ready multi-class object detection using
parking spot detection based on TIDL applications on TDA4VM SoC.
TIDL Libraries and C7x+MMA
➢ Simultaneous Real-time 4x 2MP camera ➢ Multi-sensor interfaces: camera, radar, lidar…
➢ Fusion with IMU and GPS for Localization
Surround view with 3D car model. ➢ Software compatible with TI EVM based on
➢ Concurrent applications on one TDA4VM SoC
➢ All code freely available in TI’s Processor SDK Processor SDK 38
Jacinto™ 7 TDA4VM YouTube Videos
Front camera analytics with TDA4 Smarter Parking with TDA4 TDA4: Multi-Camera demo
Real-time 8-Camera
Input to TDA4
Deep Learning
Surround View
Based Forward
with Analytics
Camera Analytics

➢ https://youtu.be/F4KNWuTSJds
➢ https://youtu.be/F4KNWuTSJds ➢ https://youtu.be/zsYbPPhO0t8

8MP Front Camera Auto Valet Park DesignCore Rugged

Perception and and Surround Vision Platform for
Localization View TDA4VM

➢ https://youtu.be/PER77CgoHWk ➢ https://youtu.be/bOSmeej9_cQ
➢ https://www.businesswire.com/news/home/202
00107006173/en/

39
WHAT TI CAN SUPPORT?

40
Software and tools - overview

41
TDA4 AutoSAR

42
TDA4 PinMux tool
https://e2e.ti.com/support/processors/f/processors-forum/927526/faq-ccs-tda4vm-pinmux-guide-for-jacinto-processors

43
TDA4 Cloud AI Tool: https://dev.ti.com/edgeai/

44
TDA4 Cloud AI Tool: https://dev.ti.com/edgeai/

45
TDA4 Cloud AI Tool: https://dev.ti.com/edgeai/

46
TI ADAS system of experienced partners
OPERATING SYSTEMS AND SECURITY SAFETY SOFTWARE, CERTIFICATION
MCAL

AutoSAR Classic

Adaptive AutoSAR

ADAS ALGORITHMS Camera Sensors, Modules

and Tuning Services
Surround and Auto Parking
Front Camera and Autonomous Drive
DMS and In-Cabin Monitoring

E NGINEERING S ERVICES

Algorithm Porting and Optimization

Driver Development
Hardware Design
47
System Testing
Thank you

48
 IMPORTANT NOTICE AND DISCLAIMER
TI PROVIDES TECHNICAL AND RELIABILITY DATA (INCLUDING DATASHEETS), DESIGN RESOURCES (INCLUDING REFERENCE
DESIGNS), APPLICATION OR OTHER DESIGN ADVICE, WEB TOOLS, SAFETY INFORMATION, AND OTHER RESOURCES “AS IS”
AND WITH ALL FAULTS, AND DISCLAIMS ALL WARRANTIES, EXPRESS AND IMPLIED, INCLUDING WITHOUT LIMITATION ANY
IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT OF THIRD
PARTY INTELLECTUAL PROPERTY RIGHTS.
These resources are intended for skilled developers designing with TI products. You are solely responsible for (1) selecting the appropriate
TI products for your application, (2) designing, validating and testing your application, and (3) ensuring your application meets applicable
standards, and any other safety, security, or other requirements. These resources are subject to change without notice. TI grants you
permission to use these resources only for development of an application that uses the TI products described in the resource. Other
reproduction and display of these resources is prohibited. No license is granted to any other TI intellectual property right or to any third party
intellectual property right. TI disclaims responsibility for, and you will fully indemnify TI and its representatives against, any claims, damages,
costs, losses, and liabilities arising out of your use of these resources.
TI’s products are provided subject to TI’s Terms of Sale (https:www.ti.com/legal/termsofsale.html) or other applicable terms available either
on ti.com or provided in conjunction with such TI products. TI’s provision of these resources does not expand or otherwise alter TI’s
applicable warranties or warranty disclaimers for TI products.IMPORTANT NOTICE

Mailing Address: Texas Instruments, Post Office Box 655303, Dallas, Texas 75265
Copyright © 2021, Texas Instruments Incorporated