ADAS & Driver Replacement: Vision

Overview for ADAS Systems

Ioseph Martinez
ADAS Product Manager

June 2019 | Session #AMF-AUT-T3658

Company Public – NXP, the NXP logo, and NXP secure connections for a smarter world are trademarks of NXP
B.V. All other product or service names are the property of their respective owners. © 2019 NXP B.V.
Agenda
• NXP ADAS Overview
• ADAS Vision Use Cases
• Road Map & Ecosystem

COMPANY PUBLIC 1
NXP Core Values to Solve
the Current Challenges of
the Market
Computation Performance
Lead the heterogeneous compute performance with
purpose built processors, optimized for power

Safety
No compromise on safety. Progression from ASIL to
enhanced dependability and fail operational modes
support

Ease of Use
Based on OPEN standards, portable and relocatable

Modularity Scalability
Built on ‘clear functionally separated extensible’ entities

COMPANY PUBLIC 2
ADAS Microprocessor Solutions
S32R | RADAR: #1 Market Leader
+50% of all car radar modules shipped in 2017 have utilized NXP technology*

NXP ADAS Value:

• Performance Per Power Front L/M Range Corner Radar Imaging Radar
through Acceleration
S32V | VISION: NCAP & Open Standards
5 star Vision NCAP, providing ISO26262 ASIL D safety case

• Enablement through Open

Standards

• Safety with Automotive

Pedigree NCAP Multi Camera Lidar

BlueBox | SAFE Central Computing for L3-L5

Autonomous Driving AI Perception Planning

COMPANY PUBLIC 3
Superior Senses With High-performance Vision

S32V | Vision: From NCAP to AI

S32V
Scalable, functionally safe, AI ready

NCAP Multi Camera

Leading Edge Enablement

Computer Vision and Artificial Intelligence
on Compact Automotive Form Factor
Lidar Vision AI Fusion

• Enables miniaturization of
camera modules
First Ethernet Camera
MPC560xE • Reduce system cost and solution for
time to market automotive
• Safe: ISO26262 compliance
• High quality automotive grade Optimized for Higher
S32V • AI Ready: Enablement to Efficiency
support

COMPANY PUBLIC 4
ADAS & Autonomous Driving Simplified
Sense Think Act

Perception and Modeling Safe Path Planning

The ability to see, become The ability to plan a route

aware of and identify the given the perceived
vehicle’s surroundings information and safely
through its sensors maneuver the vehicle

Perception
• Sensor Feature Extraction
Safe Path Planning
• Sensor Data Enhancement
• Motion Planning
• Object Detection
• Traffic Prediction
• Behavioral (Local Planning)
Modeling • Route Planning
• Object Classification • Feedback Control
• Objection Tracking
• Segmentation

COMPANY PUBLIC 5
Market Growth and Target Applications

NCAP/Mono Camera Lidar Vision Fusion w/ AI Multi & high-end

Vision Lidar Vision / Radar / Lidar Camera w/ AI
Fusion Vision

Vision + Lidar Processor SAM ($M)

$3,500

$3,000

$2,500

$2,000

$1,500

$1,000

$500

2019 $M 2020 $M 2021 $M 2022 $M 2023 $M 2024 $M 2025 $M

NCAP Vision Lidar Vision Fusion Multi Camera

CAGR = 15%

Note: NCAP Vision includes, Mono camera, Stereo Camera, Rear Camera and Driver monitor
COMPANY PUBLIC 6
S32V Vision Use Cases

NCAP/Mono Camera Lidar Vision Fusion w/ AI Multi & high-end

Vision Lidar Vision / Radar / Lidar Camera w/ AI
Pedestrian Detection. Rear View Point Could Post Processing Fusion Vision
Automatic Emergency Braking Ground Seg, Scene Seg Perception w/ Safe Decision Multi Camera perception,
Forward Collision Warning Free Space Making Occupancy Detect, Scene Segm.
Free Space

Edge Compute: S32V Central Compute: S32V

Vision SDK and AI Enablement: Bringing AI to a compact automotive form factor

COMPANY PUBLIC 7
Vision Use Cases S32V Value
Vehicle
Safety
AI / CV Function
CAN / ENET Vehicle Class-
LVDS / MIPI S32V ification
Use Case
Network
Single / Multi Ped detect Warning ISP
AEB
NCAP/Mono Camera camera, 120º Cyc detect
Lanes/Sign Lane keep
Occupant Lane change
State Display
AI
Class-
CAN / ENET Vehicle ification
DSPI / MIPI S32V CV
Use Case Network ranging
P-Cloud Signal Accelerate / Brake Point
Lidar Solid State / Processing Allow overtake Cloud
Vision based ranging Adaptive Cruise Proc
Scanning lidar rx
Localization Control
LVDS / MIPI Free space Highway Autopilot

Autonomous CAN / ENET

Processor
AI
Class-
Use Case Vehicle ISP + ification
Single / Multi CV
camera 120º S32V Network detect
Auto Emerg Brake Radar
Vision Fusion AI Ra
CAN / ENET
Redundancy Adaptive Cruise Control Vision
Li Fusion OEM AI Function (ie Lidar Fusion
Localization?? based Free Space)
AI Function

Kalray Free
PCIe
Processor space /
Segmen-
Stitching
CAN / ENET Vehicle /
tation
Use Case LVDS/MIPI/PCIe S32V
Network Multi
Rectifier
Stitching Traffic Jam Assist channel
Multi & high-end Rectification Highway Autopilot ISP
Scene Auto parking Assist
Camera w/ AI Multi camera segmentation Display
180º Freespace

COMPANY PUBLIC 8
S32V: NCAP – Front Camera
CAN-FD

NCAP/
Mono Camera Front camera
Edge processor

MRR/LRR
S32R

• High performance edge processing behind the windshield

• Performance/power as key technical factor
• Single front camera up to 8 Mpixel or stereo/ multi camera
• Optional: fusion with radar
• Emergency breaking
COMPANY PUBLIC 9
S32V: NCAP – High End Driver Monitoring and Interior
Camera Additional cameras
for: Interior monitoring
Driver monitoring, LVDS
gesture control,
interior monitoring

NCAP/
Mono Camera
Driver Camera processor
monitoring Driver monitoring
camera S32V

• Driver/Occupancy monitoring ECUs

• Infrared cameras and fisheye view are key requirements
• Typical resolution 2 Mpixel
• Multi camera optional, 3-5 cameras possible
• Video output via MIPI-TX possible
• Driver status (e.g. drowsiness detection), gesture recognition …
COMPANY PUBLIC 10
S32V3: NCAP – Platform Use Case
CAN-FD
LVDS

NCAP Camera
NCAP/ W/o processor
Mono Camera
Front/rear camera NCAP Camera
processor
MRR/LRR W/o processor
Driver monitoring
S32R Fusion with radar

Driver monitor

• Option to centralize NCAP functions into one platform

• Parallel execution of front camera (up to 8 Mpixel) and IR channel for driver monitoring
• Rear camera video stream output with augmented objects for display

COMPANY PUBLIC 11
S32V: Vision Fusion AI
MIPI
LVDS
LVDS Autonomous Processor
Deserializer
Vehicle
Network
Single /
Multi camera PCIe

CAN / ENET Auto Emerg Brake

Adaptive Cruise Control
Vision Fusion AI OEM AI Function (ie Lidar based
Ethernet PHY S32V PHY Free Space)

Edge ECU
Optional:
Lidar/ Radar PMIC
point clouds/ object lists

• Redundant processing to a larger, power hungry primary perception processor

• Assumes to perform a minimum level of redundant processing to check the primary processor
• Rule based computer vision as well as redundant NN processing
• ASIL D CPUs to perform sensor fusion, cross checking and commands to actuators
• Optional: Lidar (freespace) or Driver monitoring/ NCAP for a “scaled down” ECU version
COMPANY PUBLIC 12
S32V: Multi & High End Camera w/ AI
MIPI
LVDS CAN / ENET
LVDS Fusion (+ PHY
Vehicle
LVDS
Deserializer Network
LVDS
Deserializer Kalray)
Deserializer
PCIe
Multi camera 360 º
Multi & high-end LVDS
Camera w/ AI LVDS LVDS
Serializer
Ethernet S32V3 Serializer
S32V3
Edge ECU PMIC S32V3
PMIC
PF502x +
S32V
Optional: PMIC
SBC85xx +
PF502x
Lidar/ Radar PF502x +
SBC85xx
point clouds/ object lists PMIC
SBC85xx

• Level 2+/3: 360° AI based perception

• Highway pilot, Traffic Jam Assist, Auto Parking
• May include RGB-IR cameras and Lidar as well
• Options:
– With external additional high performance accelerator
– Surround viewing for parking

COMPANY PUBLIC 13
S32V: Lidar – Precise Depth Measurement
S32V
CAN / ENET Vehicle Actuator ECUs
SPI / ENET/MIPI PHY Fusion processor
Network

Solid State /
Scanning lidar rx

Computer Vision &

Machine learning
accelerator

Lidar

• Input: MIPI-CSI/SPI or Ethernet: samples or point cloud, depending on frontend (FPGA/ASIC)

• Independent from a single Lidar vendor
• As host processor for Lidar at the egde or as central solution with further sensors/ fusion
• ISP/Computer vision accelerator can be used to transform data to point cloud
• Point cloud as input for 3D-object detection/ freespace/ SLAM applications
• Usage as independent “safety channel” or to boost performance of 3D object detection
COMPANY PUBLIC 14
S32V Lidar Network: SqueezeSeg

Target: S32V2 EVB

Input image is from Velodyne HDL-64E LiDAR as part of KITTI dataset

COMPANY PUBLIC 15
S32V Roadmap and Ecosystem

COMPANY PUBLIC 16
NXP ADAS Ecosystem
Algo & Application: Platform:.
RTOS: • HELLA Aglaia* Drive By Wire:
• MicroSys
• eSol • Momenta • AutonomouStuff
• Fidus
• OpenSynergy • Jungo • VSI Labs
• SolidRun
• QNX • Deepscale* • TORC Robotics
• GreenHills • Neusoft
• Embotech HIL/SIL/Simulation:
• CEA Tech • dSpace
Middleware: • Intempora • NI Labs
• Elektrobit • Pathpartner • Ansys
Radar:
• Codeplay • Colorado
• Polysync Safety:
Engineering
• Baidu • Excida
AI Acceleration: • RF-Beam
• Autoware • Edge Case
• Kalray • SMS
• Renovo • Cambricon
Research
• Hawkeye
• Encore Semi

*Pending Lead Customer Confirmation

COMPANY PUBLIC 17
S32V ADAS Vision Processor Roadmap
Product Idea Development Production
Performance, Acceleration

S32V Generation 2 S32V Generation 3 S32V Generation 4

(SAE Level 2) (SAE Level 3) (SAE Level 4)

S32V next gen +1

• In Concept

S32V next gen

Features Addressing:
• Multi Camera
• ADAS Fusion with AI S32V next gen +1
• ARM A53+M4 @ 9.2K DMIPS
• CV / NN @ 150 GOPS • In Concept
• 3DGPU @ 38 GFLOPS S32V next gen
• Up to ASIL C
S32V2 Quad + GPU Features Addressing:
• NCAP Camera
S32V2 Dual + GPU • lidar
S32V2 Quad
S32V2 Dual
2018 2019 2020 2021 2022 2023 2024
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

1st Sample
COMPANY PUBLIC 18
NXP and the NXP logo are trademarks of NXP B.V. All other product or service names are the property of their respective owners. © 2019 NXP B.V.