Critical Scenario Based SOTIF Validation Method

CAICV-SOTIF Technical Alliance

Hong Wang
Tsinghua University
Deputy Executive Director,
CAICV-SOTIF Technical Alliance
Safety Of The Intended Functionality for Intelligent Vehicles

Safety Of The Intended Functionality - absence of unreasonable risk due to hazards resulting from
functional insufficiencies of the intended functionality
• SOTIF addresses hazards caused by limitations in the intended function in a system that is free from
faults as defined in ISO26262
• SOTIF is one of the biggest problems in the research and the commercialization of intelligent vehicles

Safety problems with intended function may be related to:

Performance limitations Complex environments Human-machine interactions

CAICV-SOTIF Technical Alliance
CAICV-SOTIF Technical Alliance
CAICV-SOTIF Technical Alliance

Research vision-promote researches on SOTIF test evaluation and technology innovation

The first goal：build SOTIF test evaluation structure The second goal：promote SOTIF technology innovation

The third goal：promote test and verification ability The fourth goal：formulate databases, scene libraries and standards
related to SOTIF

left turn collision at crossroads vehicle merges suddenly in tunnel vehicle starts suddenly at night

animal enters suddenly road vehicle stopes suddenly on road right turn collision in snow
SOTIF Test and Evaluation Methodology

Evidence： Known Scenario Definition：

• Specific scenario exposure; • Combinations of known trigger conditions;
• Response solution controllability; • Known parameter boundaries and sampling

• Injury severity; distribution; Unsafe Safe

• Clear harmful behavior;

Is the scenario risk acceptable for known Known Scenario

unsafe areas?
Known
Evaluation

Is the residual risk in the unknown unsafe area Unknown Scenario Unknown
acceptable? Evaluation

Evidence： Unknown Scenario Definition：

• Unknown hazard scenario frequency (below a
threshold);
• Unknown damage frequency (below a threshold);
• Unknown scenario occurrence probability (Below
the Threshold);
• Unknown trigger condition;
• Unknown system behavior;
• Unknown trigger conditions which are combined
with known parameters;
SOTIF Test and Evaluation Methodology

Focus on the "imperfect system" problem when the system "does NOT fail"

SOTIF
Strongly related to the specific context of operation
Feature

Always contain certain range of unknown scenarios

Test evaluation and verification based on "trigger condition-risk"

Boundary settings based on acceptance criteria

Key Points
of Comprehensively application based on a variety of testing methods and scenarios
SOTIF Test
Clarify acceptance criteria and determine residual risk threshold

Open road tests and Government supervision

SOTIF Test and Evaluation Methodology

Critical Scenario based SOTIF Test Solution

Combining various techniques, in a limited number of tests, prove that the system risks meet the acceptance criteria.

Test Method Test Design Assess target Perspective Acceptance Criteria

Open-road Vehicle Component

SIL Coverage Criticality Severity Harmful Behavior
test level level

Non-public Expert Non-public Controll-

HIL SOTIF measures Residual Risk
test track Analysis test track ability
SOTIF Test and Evaluation Methodology

Critical Scenario Test Evaluation Verification

Vehicle Component level
level
Basic
Short-term goal
Scenario Perception Localization

Controllability
Severity Decision-
Control
making

Trigger Long-term goal V2X HMI

SIL HIL Road Test

Potential Risk
Assessment
Sandbox Operation
Component level supervision
Exposure phase

Risk level Vehicle level

Critical Scenario
Triggering Conditions and SOTIF scenarios
Scenario element Basic road structure Hazard Event

Light

White truck

Triggering Triggering Functional

mechanism miss
Source Insufficiency
identification
hard light object recognition issues
+ + vision sensor
white truck feature merge into background

Triggering conditions is the specific scenario condition leading hazard of CAV, such as extreme weather.
Critical Scenario
Expert Knowledge Experience Analysis (ECEA)

Accident data Test data R & D data

ECEA is a triggering condition method based on case induction.

Critical Scenario
Expert Knowledge Experience Analysis (ECEA)

Accident data Test data R & D data

Analyze the SOTIF hazard events

ECEA is a triggering condition method based on case induction.

Critical Scenario
Expert Knowledge Experience Analysis (ECEA)

Accident data Test data R & D data

Analyze the SOTIF hazard events

Extraction of key information Temporary

manipulatio
n

ECEA is a triggering condition method based on case induction.

Critical Scenario
Expert Knowledge Experience Analysis (ECEA)

Accident data Test data R & D data

Analyze the SOTIF hazard events

Extraction of key information

Classified homotypic events

ECEA is a triggering condition method based on case induction.

Critical Scenario
Expert Knowledge Experience Analysis (ECEA) Unconventional

Accident data Test data R & D data

Truck
Analyze the SOTIF hazard events

Extraction of key information

Classified homotypic events

Rollover Non vehicle collision

ECEA is a triggering condition method based on case induction.

Critical Scenario
Expert Knowledge Experience Analysis (ECEA)

Accident data Test data R & D data

Object Object Decision-Making
Identification Identification

Analyze the SOTIF hazard events

Object Object Dynamic object
Identification Identification complex
Extraction of key information

Classified homotypic events

Object Control Dynamic object
Triggering condition summarized Identification complex

Object Control Complex

Identification environment

ECEA is a triggering condition method based on case induction.

Critical Scenario
Expert Knowledge Experience Analysis (ECEA)

Object Object
Decision-Making
Accident data Test data R & D data 目标识别
Identification Identification
Blind sopts
形态罕见
Rare form Target ablation

Analyze the SOTIF hazard events Object Object Dynamic object

Identification Identification complex
Extraction of key information Target ablation Rare form sudden change

Classified homotypic events

Object Control Dynamic object
目标识别
Identification Attachment complex
Triggering condition summarized 形态罕见
Rare form sudden change Traffic violation

Generalization Optimization Amalgamate

Object Complex
目标识别 Control
Identification environment
形态罕见 Excessive curve
Rare form traffic accident
rate

ECEA is a triggering condition method based on case induction.

Critical Scenario
Expert Knowledge Experience Analysis (ECEA)
Primary label Level 2 label Level 3 label
road geometry
blind spot road structure
surrounding traffic participant
Sensor
near range limitation
functional limitation range limitation
Accident data Test data R & D data far range limitation
stability of fixing /
atmosphere factors /
dirt
view consistency
surface obstruction scratch
view completeness occlusion
strong contrast of light and dark
Analyze the SOTIF hazard events insufficient brightness
backlight or glare
shadow
image quality reflection
Extraction of key information bad clearness
color temperature exception
image blur
image noise
bad quality of raw data point missing
Classified homotypic events bias of reflectivity
bias of positioning
point cloud quality
interstitial points
noise points
ghost points
Triggering condition summarized intensity exception of reflected point
millimeter wave quality multi-reflection
reflected wave noise
others /
unfamiliar category of object
familiarity
Generalization Optimization Amalgamate unfamiliar form of object
true-positive similarity
similarity
false-negative similarity
object recognition issues
obstruction between same category
obstruction
obstruction between different category
contradiction feature merge into background
Triggering condition framework fusion and supplement object appearing suddenly /
lost of tracking /
object tracking issues interrupt of tracking /
change of category /

ECEA is a triggering condition method based on case induction.

Critical Scenario
System Theoretic Process Analysis (STPA)

Define analysis Establish control Identify hazardous Identify causal

purpose structure behaviors scenarios

Define SOTIF Hazards Establish ICVs Identify Hazardous Causal Scenario

System functionality Behaviors elements analyze
Upper level hazard definition

Hazard instance boundary

STPA from the system functionality and properties derives the triggering condition.
Critical Scenario
SOTIF Scenario Shared Platform
SOTIF Scenario Shared Library V2.0
Test Evaluation- Highway Pilot (HWP)

◼ ODC Definition
ODD Passenger Status Ego Status

Transportation Temporary Traffic

Road structure Environment Info
Facilities Incident participant

◼ System Initial Architecture Definition Front Camera

Braking
HWP System
Front Radar
Controller
Corner Radar1
Sensor
fusion
Corner Radar2 GW

Lidar Planning/
control
Map/Localization
HMI
Vehicle
parameters
Sensor solution (only dealing with System Architecture Scheme
Cut-in scenarios)
 Test Evaluation- Highway Pilot (HWP)

Function/Behavi Function/Beh Introductory Functional Failure SOTIF Potential misuse Associated

Cause of misuse Derived dangerous scenarios
or ID avior words /Unexpected Behavior Relevance scenarios party
No No brake in place Y Due to incomprehension about warning,
Reasonably foreseeable Incomprehension the driver does not take over control,
Less Deficiency braking Y
HAZOP Functionalitymisuse
lost causing the collision
More Excessive braking Y Warn driver take driver
Reverse NA N over Due to Inattention to the warning, the
Inattention driver does not take over control,
Unintended Unintended braking Y
Brake causing the collision
F001 Stuck Brake stuck N
control The driver intentionally accelerate to
Early Early braking Y Intentional operation prevent the vehicle from cutting in,
Later Late braking Y driver causing the collision
Early Finish braking early Y Function working The driver brakes, causing the HWP
Misuse operate
exit, causing the collision
Later Finish Braking late Y
Other passenger Misuse operate …

Functional Failure Scenario Severity Controllability

Potential Risk acceptable
Feature/Behavior ID /Unexpected Hazard ID No. Hazardous event
consequence (Y/N)
Behavior Traffic
Scenario Rating Note Rating Note
participants
When entering a Highway/City Express, L3 function, DDT is
Highway/City Express Collision with front High speed, high relative
HARA
F001 No brake is in place Entering junction
Cut in vehicle HWP-HZ-0001 and a vehicle cuts in, ego vehicle does
not brake.
vehicle
S>0
collision speed
C>0 executed inside ODD,
driver uncontrollable
N

In straight lanes of the Highway/City L3 function, DDT is

Highway/City Expressway Collision with front High speed, high relative
F001 No brake is in place straight lanes
Cut in vehicle HWP-HZ-0002 Express, a vehicle cuts in, ego vehicle
vehicle
S>0
collision speed
C>0 executed inside ODD, N
has no brake. driver uncontrollable
Highway/City Expressway CTA analysis identifies
In curve lanes of the Highway/City
Collision with front High speed, high relative
L3 function, DDT is
F001 No brake is in place curve lanes
Cut in vehicle HWP-HZ-0003 Express, a vehicle cuts in, ego vehicle
vehicle
S>0
collision speed
C>0 executed inside ODD, N
triggering conditions
has no brake. driver uncontrollable
When Exiting the Highway/City Express, L3 function, DDT is
Highway/City Express Collision with front High speed, high relative
F001 No brake is in place Exiting junction
Cut in vehicle HWP-HZ-0004 a vehicle cuts in, ego vehicle has no
vehicle
S>0
collision speed
C>0 executed inside ODD, N
brake. driver uncontrollable
When entering the Highway/City L3 function, DDT is
Highway/City Express Collision with front High speed, high relative
F001 Deficiency braking Entering junction
Cut in vehicle HWP-HZ-0005 Express, a vehicle cuts in, ego vehicle
vehicle
S>0
collision speed
C>0 executed inside ODD, N
has Deficiency braking. driver uncontrollable
Test Evaluation- Highway Pilot (HWP)
Typical scenarios selection and key parameter definition (1/2)

L7: Vehicle position, speed, attitude, driver operation

L6: N/A

L5: Light, rain, snow, fog

L4: Target Vehicle (position, speed, type)

L3: N/A

L2: Curb, road sign

L1: Lane markings, curvature, road surfaces, ramps

Test Evaluation- Highway Pilot (HWP)
Typical scenarios selection and key parameter definition (2/2)
Scenarios derived from Scenarios derived from
function definitions Basic Overlay performance limitations and
Scenarios trigger condition analysis
Scenarios
• Ego vehicle is driving on a
straight road, and target
vehicle cuts in from an
entrance;

• Ego vehicle is driving on a

straight road, and target Sharp Turns Steep Ramp Backlight Occlusion
vehicle cuts in from a left (or
right) adjacent lane;

＋
• Ego vehicle is driving on a
curve, and target vehicle cuts
in from a left (or right)
Snow Blurred lane lines Night & Poor Light Rain
adjacent lane;

• Ego vehicle has entered the

ramp, and target vehicle cuts
in from an adjacent lane;

Fog Sandstorm Pavement Construction Tunnels

Test Evaluation- Highway Pilot (HWP)

Risk Acceptance Level-Chinese Skilled driver performance model

CAERI: Million kilometers of DJI: UAV aerial survey data

China-FOT Data natural driving data AD4CHE

130,000 kilometers of
200,000 km data 35,903 Vehicles3
natural driving data
Test Evaluation- Highway Pilot (HWP)

Risk Acceptance Level-Chinese Skilled driver performance model

20
Gender Age Driving
20 10
Experience
10 10

0 0 0
H M L None
M F
20-25 26-30 30-40 40+

HMI experiment is in progress, expect to finish in August

Reaction Perception + Action Time (excluding
Participants
Time Decision Time delay)
Subject 1 1.769 1.2650 0.5040
Subject 2 1.616 1.1114 0.5020
Subject 3 1.212 0.7480 0.4640
Subject 4 1.117 1.0020 0.1150
Subject 5 1.511 0.6000 0.9110
Subject 6 1.473 0.9500 0.5230
Subject 7 1.239 0.7600 0.4790
Subject 8 1.135 1.0310 0.1300
ave. 1.384 0.9340 0.4540

Results show that the responses are 454 ± 234 ms before the
Tsinghua University IVDAS
reaction, and areas related to brake intention

Several units cooperate to collect vehicle and driver data to verify other parameters of Chinese driver performance model.
Test Evaluation- Highway Pilot (HWP)

Risk Acceptance Level-Chinese Skilled driver performance model

Perception Decision Reaction

Time to enable
Release accelerator pedal brake

Foot
Risk Decision on Transfer
Perceive
Pedal angle

breaking Brake Pedal

Accelerator Pedal

Perception Time Delay in Decision

Decision Time Action Time Time

Response Time (1.06s) Jerk Time Max Deceleration

(0.54s) (0.63G)
Test Evaluation- Highway Pilot (HWP)

Risk Acceptance Level-Chinese Skilled driver performance model

Response Time (s)

Data Lateral Offset Jerk Time Max Deceleration
Source (m) Decision Time(s) Action Time (s)
(s) (G)

0.47 0.81
China-FOT / 0.54 6.13 m/s2 = 0.63G
1.28

AD4CHE 0.380 / / /

Simulator / 1.06 / /

ECE R157 0.375 0.4 + 0.75 = 1.15 0.6 0.774G

Test Evaluation- Highway Pilot (HWP)

Field Test - Chongqing, China

Experimental design: Experiment preparation: Day/Night

including 9 working conditions, 27 tests simultaneous multi-device data acquisition Real Vehicle Experiment
Test Evaluation- Highway Pilot (HWP)

Field Test – Changsha, China

Perception limitations Deceleration

CUT-IN CUT-OUT

Experimental design: Experiment preparation:

including 21 working conditions, 166 tests simultaneous multi-device data acquisition Real Vehicle Experiment
Test Evaluation- Highway Pilot (HWP)

Field Test – Changsha, China

SOTIF Test and Evaluation Technology for ICVs

CAICV SOTIF Technology

Research Report
ICV-SOTIF
Safety scenario test elements
and management standardization

Five Frontier research reports totally 1000+pages Two standards

SOTIF Test and Evaluation Technology for ICVs

SOTIF test and SOTIF test and

evaluation evaluation
Localization Navigation
system on Pilot
SOTIF test and SOTIF test
evaluation
SOTIF and evaluation
Perception Scenario Highway
Algorithm SOTIF test and library Pilot
SOTIF test and
evaluation
evaluation
Decision-
Control
Making SOTIF test
System
SOTIF test and algorithm SOTIF test and evaluation
evaluation and evaluation Automated
Perception HMI Valet
System System Parking
Acknowledgements

TO ALL
CAICV-SOTIF Technical Alliance
 谢谢
Thank You！