US010788990B2

( 12 ) Kim
United States Patent ( 10) Patent No .: US 10,788,990 B2
et al . (45) Date of Patent : Sep. 29 , 2020
( 54 ) VEHICLE WITH IMPROVED I / O LATENCY ( 56 ) References Cited
OF ADAS SYSTEM FEATURES OPERATING
ON AN OS HYPERVISOR U.S. PATENT DOCUMENTS
9,405,688 B2 * 8/2016 Nagarajan G06F 12/0815
( 71 ) Applicant: TOYOTA JIDOSHA KABUSHIKI 9,725,089 B2 * 8/2017 Tashiro BOOK 31/00
KAISHA , Toyota -shi, Aichi -ken (JP) 2011/0202918 Al * 8/2011 Park GOOF 9/45541
718/1
( 72 ) Inventors: BaekGyu Kim , Mountain View , CA 2014/0258620 A1 * 9/2014 Nagarajan GO6F 12/0815
(US ) ; Chung - Wei Lin , Mountain View , 711/120
2016/0082961 A1 * 3/2016 Tashiro BOOK 31/00
CA (US ); Shinichi Shiraishi , Mountain 701/93
View , CA (US )
* cited by examiner
( * ) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 Primary Examiner — Jerrah Edwards
U.S.C. 154 ( b ) by 227 days . (74 ) Attorney, Agent, or Firm - Burbage Law , P.C .;
Jon -Michael Burbage; Elizabeth Ruzich
( 21 ) Appl. No .: 15 /434,396
( 57 ) ABSTRACT
(22 ) Filed : Feb. 16 , 2017 The disclosure includes embodiments for improving a per
( 65 ) Prior Publication Data
formance of a set of Advanced Driver Assistance Systems
( “ ADAS systems” ) included in a vehicle by decreasing a
US 2018/0232156 A1 Aug. 16 , 2018 latency for processing a set of input /output (“ I / O ” ) requests
generated by one or more active ADAS systems from the set
( 51 ) Int . Cl . of ADAS systems . A method includes determining situation
G06F 3/06 (2006.01) data describing a driving situation for the vehicle. The
GO6F 13/16 ( 2006.01) method includes identifying the one or more active ADAS
( 52 ) U.S. CI. systems from the set of ADAS systems for the driving
CPC .......... G06F 3/0611 (2013.01 ) ; G06F 3/0659 situation . The method includes determining whether an
( 2013.01 ) ; G06F 3/0685 ( 2013.01 ) ; G06F input/output (“ I / O ” ) communication conflict exists for the
one or more active ADAS systems . The method includes
13/161 (2013.01 ) ; G06F 13/1642 (2013.01 ) applying at least one of a direct I/ O strategy and a virtual I /O
( 58 ) Field of Classification Search strategy to the set of I/ O requests based on whether the I /O
CPC .... GO6F 3/0611 ; G06F 3/0659 ; G06F 3/0685 ; communication conflict exists .
G06F 13/161 ; G06F 13/1642
See application file for complete search history . 20 Claims , 10 Drawing Sheets
201 ECU 198
Memory 127

First Virtual ECU Second Virtual ECU

269 267
ADAS Software Infotainment
299 Software 296
First OS Second OS
297 294

Hypervisor 1/0
Queue 239
DOS Hypervisor 259 Latency System 199
Sensor Set
182
Processor
125
Actuator Set
184
U.S. Patent Sep. 29, 2020 Sheet 1 of 10 US 10,788,990 B2

ESecConUd FMiermosty SEnteraim oftware Second OperatingSystem Second Microntle

Bus Platform

ECUFirst FMiemrosty Control Software First OperatingSystem First

Microntle FPrior
1A
(
)igure
Art
U.S. Patent Sep. 29 , 2020 Sheet 2 of 10 US 10,788,990 B2

V1eh2ic3le

Latency1Sy9stem Sensor D195ata

ECU198
Infotaime
ASyDsAtSemSet180 186System Proces r 125 Memory 127
1B
F i g u r e
ISnfeotraivmn 103
120

1Net0wo5rk Comunicat
D-coSmpRliaCntGPSU170nit 145Unit
Senstor 182 Actuaor Set184

100
U.S. Patent Sep. 29 , 2020 Sheet 3 of 10 US 10,788,990 B2
200
220

Latency System 199 Communication Module

202

Monitor Module 222

204
VO Control Module
206
224
OS Hypervisor 226
259

228
DSRC -compliant GPS
Unit 170
230 Sensor Set
182
Infotainment System 186 232
234 Actuator Set
184
Processor
125 236

238 ECU
198
Communication Unit 240 Memory
145
127
246 Sensor Situation
Data 295 Data 289
ADAS System Set GPS Data Rule Data
180 293 287
244
242 Map Data Virtual
291 ECU Data
Feature 285
Figure 2A Data 283
U.S. Patent Sep. 29, 2020 Sheet 4 of 10 US 10,788,990 B2

Sensotr 182 SActeuator 184

1SLya9tsencmy
ECU198 1Me2mo7ry
Infotaime
EVSierCctouUnadl 267 S2oft9wa6re OSSecond 294
Proces r 125

ECUVFirtusatl 269 ASoDftAwaSre 299 FOSirst 297 2HDype5Orvi9Ssor

1Hyp/erv0isor2Qu3eu9e 2BFigure
201
U.S. Patent Sep. 29, 2020 Sheet 5 of 10 US 10,788,990 B2

TFHREOSLD TRIGHENG OFTPROHVISEON FVEAHTIUCRL R:50meters

50
:
RS
miles
hour
per
R:200meters
miles
50
:
RS
hour
per
%
20
pers1Megcaboyntde

CRELITVAN VTOTEHICEL FEATUR from

"
R
“
,
Range
ego
the
Relative
the
of
"
RS
“
,
speed Relative
the
of
RS
"
,
speed
vtoadoebhtjiectled isanwontherohaidwcahyofortheegovbeshtiacle otheandvbegoeshtaicle Range,“R"fromtheegovtoadoebhtjicetled isanwontherohaidwcahyofortheegovbeshtiacle oandthevegobeshtiacle vthepofAerhcientlaeg bcthewisrohsidcenhrg ltwosteaphnraets dforraAnvaertaeg mowanlvtidaeng)ore(cxuprectnetd
2CFigure

RSELNATODR SFERNOSTMR Mrwiladmvetr sSepnseodr Mrwiladmvetr sSepnseodr Mwriladmvetr sSepnseodr Ssatengrslioneg cTuomnhiaeton matwarohanivtcoerhs ,andintisdowhanloitadsae sactsasaway,ensor
Camera Camera Camera

VEHICL FEATUR Pre-Col is on

”
PCS
System
“
,
Ae(xDamApSle
)system CAdraupitsve sy temAe(xDamApSles)ystem
"
ACC
“
,
Control
LaneKeepA,s ist )system
LKA
"
,
system Ae(xDamApSle sMstorevaimeng y tem(exampleins)foytainmet
287
U.S. Patent Sep. 29 , 2020 Sheet 6 of 10 US 10,788,990 B2

300

The I/O control module analyses each vehicle feature and the rule data to determine
whether an I /O communication conflict is present for a particular vehicle feature . 301

1 / O communication
conflict ? 302

NO YES

If an I /O communication conflict is not If an I / O communication conflict is

present for the particular vehicle feature ,
then the 1/0 controlmodule gives the
device or software that is responsible for
providing that particular vehicle feature
( e.g. , a particular ADAS system , a
particular infotainment system , etc. )
permission to exercise direct 1/0 . 303
present for a particular vehicle feature,
then the I /O control module ( i ) gives the
device or software that is responsible for
providing that particular vehicle feature
( e.g. , a particular ADAS system , a
particular infotainment system , etc. )
permission to exercise virtual 1/0 and ( i )
assigns the device or software
responsible for providing this particular
vehicle feature an urgency value relative
to the other vehicle features that are
currently executing or will be executing
in the near future based on the driving
situation . 305

Figure 3A
U.S. Patent Sep. 29 , 2020 Sheet 7 of 10 US 10,788,990 B2

399

The sensor set generates sensor data describing the driving situation . 307

The ADAS system set analyzes the sensor data and determines to provide one or more
vehicle features based on the driving situation . 309

The ADAS sensor set initiates a set of I /O requests in order to provide the one or more
vehicles features for the driving situation . Contemporaneously , I / O requests may be
executed to provide vehicles features that are related to infotainment and not ADAS
functionality ; these I/O requests are included in the set of I / O requests considered in
subsequent steps of this method 399. The monitor module analyzes the I /O requests
and generates feature data describing the vehicle features which generated the different
I /O requests . 311

The I / O controlmodule analyses the feature data and the rule data to determine
whether an 1/0 communication conflict is present . 313

I / O communication
conflict ? 314

NO YES

If an I / O communication conflict is not A

present , then the I/O control module
gives each vehicle system ( whether they
be an ADAS system or an infotainment
system ) permission to exercise
direct 1/0 ( see , e.g. , Figure 4B ) . For
example , each I / O request is passed to
their addressed 1/0 device by the
latency system without being stored in
the hypervisor 1/0 queue . 315

Figure 3B
U.S. Patent Sep. 29 , 2020 Sheet 8 of 10 US 10,788,990 B2

A 399

Give each vehicle system ( whether they be an ADAS system or an infotainment system )
associated with the set of I /O requests permission to exercise
virtual l / 0 . 317

Analyze the driving situation described by the situation data to determine which active
vehicle functionality described by the feature data is most important ( or relevant ) to the
current driving situation . 319

Analyze each of the I/O requests in the set of I /O requests and the rule data to
determine which vehicle functionality they are associated with providing (e.g. , the PCS
system is associated with providing the vehicle functionality of avoiding a collision , the
LKA system is associated with providing the vehicle functionality of not swerving into a
neighboring lane , etc. ) . 321

Assign each of the I/O requests in the set of I /O requests an urgency value based on the
relative importance of each vehicle feature they are associated with providing , as well as
their importance to vehicle features that will be executing in the near future based on the
driving situation . The set of 1/0 requests is stored in the hypervisor I / O queue . For
some driving situations , a vehicle functionality is so important that its associated I/O
requests are never stored in the 1/0 hypervisor queue and its associated ADAS system
is instead granted permission to exercise direct I /O while the other ADAS systems
exercise virtual I / O . 323

Release each I/O request from the hypervisor I/O queue based on their assigned
urgency values . 325

Figure 3C
U.S. Patent Sep. 29, 2020 Sheet 9 of 10 US 10,788,990 B2

DF10eivrscte 435 SI/ODecvoincde 440

Virtual
O
/
I
FSitrastegy

EVTihrCturUadl 265 SAofCtware 425 OSACC 430

1SLya9tsencmy
ECU198 1Me2mo7ry VSEiercCtounUadl 267 LSofKtwaAre 415 LKAOS 420
459 Proces 125

457
. 1Hyp/ervi0sorQ2u3eu9e
FVEirCtusUatl 269 PCSOS 410
PSofCtwaSre 405
455
4AFigure
400
U.S. Patent Sep. 29, 2020 Sheet 10 of 10 US 10,788,990 B2

I/ODevice 435
First SI/ODecvoincde 440
Setcraonedgy 1Di/re0ct

VTEihrCtuUadl 265 ASofCtware 425 ACCOS 430

199SLaytsencmy
ECU198 1Me2mo7ry VSEiercCtounUadl 267 LSofKtwaAre 415 LKAOS 420
459 Proces r 125
455
,
Hypervisor
O
/
I
-457 2Qu3eu9e
FVEirCtsuUatl 269 PCSOS 410
PSofCtwaSre 405
4BFigure
401
 US 10,788,990 B2
1 2
VEHICLETMPROVEDENCY tainment system generates and transmits one or more I /O
FAASYSTEMFEATREOPERATING requests in order to communicate with one or more I /O
NANOHYPERVISOR devices of the vehicle through the OS hypervisor and
thereby provide its infotainment functionality. Multiple
BACKGROUND vehiclesystem . , Aym , infotainment system ,
etc. ) may attempt to route an I /O request to the same I /O
The specification relates to a vehicle with improved device at the same time , resulting in an I/ O communication
input /output (“ I/ O ” ) latency of Advanced Driver Assistance conflict.
System (“ ADAS system ” if singular, “ ADAS systems” if In some embodiments, the latency system selects one of
plural) features operating on an operating system (“ OS” if 10 two different routing strategies for managing the I /O
singular, “ OSes " if plural) hypervisor. requests for the OS hypervisor. These two strategies are
Vehicle control systems are becoming increasingly popu described below as “ virtual I/ O ” and “ direct I/ O .”
lar. One example of a vehicle control system is an ADAS
In some embodiments , as a first ADAS system of a
system . vehicle executes ( a “ first execution ” ), it generates one or
ADAsystem provider
more automus features 15mr1requests that are addressed to ne more
to the vehicles which include these ADAS systems. For devices ( e.g. , sensors or actuators of the vehicle ). A second
example, an ADAS system may monitor the position of a ADAS system (or an infotainment system of the vehicle )
vehicle relative to the lane in which the vehicle is traveling, may be executing contemporaneous to the execution of the
and if the vehicle begins to swerve outside of that lane the first ADAS system (a “ second execution ” ). This second
ADAS system may take remedial action by repositioning the 20 execution may produce one or more I/O requests that are
vehicle so that the vehicle stays in the lane or providing a addressed to one or more I /O devices that are the same as
notification to a driver of the vehicle so that the driver knows those addressed by the first execution , thereby creating an
that they need to take action to remedy the situation . I/ O communication conflict. The latency system 199 solves
this problem .
SUMMARY 25 For example, the latency system 199 described herein
includes codes and routines which , when executed by an
An OS hypervisor beneficially enables multiple guest onboard vehicle computer, causes the onboard vehicle com
operating systems to be executed by a single electronic puter to execute one or more of the following steps: ( 1 )
control unit (“ ECU ” is single , “ ECUs” if plural) of a vehicle . determine whether one or more I /O requests generated by
The OS hypervisor typically collects 1/0 requests from 30 the second execution creates an I /O communication conflict
multiple guest operating systems in one place and then with the first execution; (2 ) if no I/ O communication conflict
processes them in the order they are received . However, our is present, directly routing the I/O requests to the I /O devices
experimentation shows that this method of managing I /O which they address ( see , e.g. , FIG . 4B which depicts an
requests for OS hypervisors frequently results in significant example of direct I/ O according to some embodiments ); and
bottlenecking ( and associated time delays) of processing the 35 (3 ) if an I/O communication conflict is present, (i) assign an
I /O requests (e.g. , releasing the I/ O requests to whichever urgency value for individual I /O requests of the first execu
I /O device they are addressed to ) . tion and the second execution based on whether the vehicle
No vehicle manufacturer has ever used OS hypervisors to feature these I/ O requests are associated with providing is
implement ADAS features. This is because the bottleneck associated with safety, (ii ) place the conflicted I /O requests
ing described above , and the delay it causes , would likely 40 in a hypervisor I /O queue of the OS hypervisor, and (iii )
result in catastrophic events resulting in loss of life . The process the I/ O request in the hypervisor I/O queue based on
latency system described herein solves this problem by the relative urgency values for the queued I /O requests ( see ,
eliminating bottlenecking for OS hypervisors e.g. , FIG . 4A which depicts an example of virtual I /O
Sme vehicles have et systems that provide according to some embodiments ).
a sufficient combination and quality of ADAS features that 45 A system of one or more computers can be configured to
these vehicles are considered to be “ autonomous vehicles.” perform particular operations or actions by virtue of having
The set of ADAS systems includes one or more ADAS software, firmware, hardware, or a combination of them
systems that provide one or more autonomous features for a installed on the system that in operation causes or cause the
vehicle. system to perform the actions . One or more computer
The latency system described herein includes codes and 50 programs can be configured to perform particular operations
routines which, when executed by an onboard vehicle com or actions by virtue of including instructions that, when
puter ( e.g. , an ECU) , causes the onboard vehicle computer executed by data processing apparatus, cause the apparatus
to identify, analyze and manage I /O requests for an OS to perform the actions .
hypervisor which is implemented in a vehicle to achieve One general aspect includes a method executed by a
ADAS functionality for the vehicle , and possibly other types 55 processor of a vehicle for improving a performance of a set
of vehicle functionality . For example , the vehicle function of ADAS systems included in the vehicle by decreasing a
ality can include : ( 1 ) ADAS functionality provided by the latency for processing a set of I/ O requests generated by one
set of ADAS systems of the vehicle; or ( 2 ) infotainment or more active ADAS systems from the set of ADAS
functionality provided by an infotainment system of the systems , the method including : determining situation data
vehicle. ADAS functionality includes any functionality pro- 60 describing a driving situation for the vehicle; identifying the
vided by one or more ADAS systems of the vehicle . Info one or more active ADAS systems from the set of ADAS
tainment functionality includes any functionality provided systems for the driving situation based on the situation data
by one or more infotainment systems of the vehicle . and rule data that describes which ADAS systems from the
An ADAS system generates and transmits one or more set of ADAS systems are active for a plurality of driving
I /O requests in order to communicate with one or more I /O 65 situations; determining whether an I/O communication con
devices of the vehicle through the OS hypervisor and flict exists for the one or more active ADAS systems based
thereby provide its ADAS functionality . Similarly, an info on the rule data that describes which I/ O devices are called
 US 10,788,990 B2
3 4
by the set of 1/0 requests transmitted by the one or more apparatus, and computer programs recorded on one or more
active ADAS systems , and applying at least one of a direct computer storage devices, each configured to perform the
I /O strategy and a virtual 1/0 strategy to the set of I /O actions of the methods.
requests based on whether the I /O communication conflict Implementations may include one or more of the follow
exists , where the direct I/O strategy is applied by the 5 ing features. The system further including a single hardware
processor if the I /O communication conflict does not exist ECU , where the set of ADAS systems are provided by the
single hardware ECU . The system where the set of ADAS
and the virtual I /O strategy is applied by the processor if the systems
I /O communication conflict does exist. Other embodiments are provided by a single ECU that executes an OS
of this aspect include corresponding computer systems, 10 hypervisor which , when executed by the processor, gener
ates a plurality of virtual ECUs which each provide a
apparatus, and computer programs recorded on one or more different ADAS system from the set of ADAS systems. The
computer storage devices, each configured to perform the system where each virtual ECU from the plurality of ECUS
actions of the methods.
Implementations may include one or more of the follow include heterogeneousOSOSes
includes a different such that the plurality of ECUs
. The system where the vehicle
ing features . The method where the set of ADAS systems are 15 is an autonomous vehicle . The system where the vehicle is
provided by a single hardware ECU . The method where the a HAV. Implementations of the described techniques may
set of ADAS systems are provided by a single hardware include hardware, a method or process , or computer soft
ECU that executes an OS hypervisor which , when executed ware on a computer - accessible medium .
by the processor, generates a plurality of virtual ECUs which One general aspect includes a computer program product
each provide a different ADAS system from the set of ADAS 20 for decreasing a latency for processing a set of I /O requests
systems . The method where each virtual ECU from the generated by one or more active ADAS systems from a set
plurality of ECUs includes a different OS such that the of ADAS systems of a vehicle , the computer program
plurality of ECUs include het geneous OSes . The method product including a non -transitory memory storing com
where the virtual I /O strategy is applied and the method puter - executable code that, when executed by a processor,
further includes : assigning an urgency value to each of the 25 causes the processor to : determine situation data describing
one or more active ADAS systems based on a relative a driving situation for the vehicle ; identify the one or more
urgency of the one or more active ADAS systems based on active ADAS systems from the set of ADAS systems for the
the driving situation , where each individual I/ O request of driving situation based on the situation data and rule data
the set of I /O requests is assigned the urgency value which 30 that describes which ADAS systems from the set of ADAS
systems are active for a plurality of driving situations ;
was assigned to the active ADAS system which generated determine whether an I/ O communication conflict exists for
each individual I/O request; storing the set of I/ O requests in the one or more active ADAS systems based on the rule data
a hypervisor I /O queue ; and processing individual I /O that describes which
requests of the set of I /O requests out of the hypervisor I /O requests transmitted I /byO devices are called by the set of I /O
queue based on the urgency values assigned to each of the 35 systems ; and apply at leasttheoneoneof aordirect
more active ADAS
I/ O strategy and
individual I/O requests of the set of I/ O requests such that a virtual I/ O strategy to the set of I/ O requests based on
I /O requests . The method where the vehicle is an autono whether the I /O communication conflict exists , where the
mous vehicle . The method where the vehicle is a highly direct I/O strategy is applied by the processor if the I /O
autonomous vehicle ( “HAV ” ). Implementations of the communication conflict does not exist and the virtual I /O
described techniques may include hardware , a method or 40 strategy is applied by the processor if the I/ O communica
process, or computer software on a computer -accessible tion conflict does exist. Other embodiments of this aspect
medium . include corresponding computer systems, apparatus, and
One general aspect includes a system for improving a computer programs recorded on one or more computer
performance of a set of ADAS systems included in a vehicle storage devices, each configured to perform the actions of
by decreasing a latency for processing a set of I/ O requests 45 the methods.
generated by one or more active ADAS systems from the set Implementations may include one or more of the follow
of ADAS systems, the system including : a processor ; and a ing features. The computer program product further includ
non - transitory memory storing computer code which is ing a single hardware ECU , where the set of ADAS systems
operable , when executed by the processor, to cause the are provided by the single hardware ECU . The computer
processor to perform steps including: determining situation 50 program product where the set of ADAS systems are pro
data describing a driving situation for the vehicle; identify vided by a single ECU that executes an OS hypervisor
ing the one or more active ADAS systems from the set of which, when executed by the processor, generates a plurality
ADAS systems for the driving situation based on the situ of virtual ECUs which each provide a different ADAS
ation data and rule data that describes which ADAS systems system from the set of ADAS systems. The computer
from the set of ADAS systems are active for a plurality of 55 program product where each virtual ECU from the plurality
driving situations; determining whether an I/ O communica of ECUs includes a different OS such that the plurality of
tion conflict exists for the one or more active ADAS systems ECUs include heterogeneous OSes . The computer program
based on the rule data that describes which I/O devices are product where the vehicle is a HAV . The computer program
called by the set of I/ O requests transmitted by the one or product where the direct I/ O strategy includes transmitting
more active ADAS systems; and applying at least one of a 60 the set of I /O requests directing to one or more I /O devices
direct I/O strategy and a virtual I /O strategy to the set of I /O which are addressed by the set of I /O requests. The computer
requests based on whether the I /O communication conflict program product where the virtual I /O strategy includes
exists , where the direct I/O strategy is applied by the queuing the set of I /O requests in a hypervisor I/ O queue of
processor if the I /O communication conflict does not exist an OS hypervisor responsible for providing the set of ADAS
and the virtual I /O strategy is applied by the processor if the 65 systems via a set of virtual ECUs generated by the OS
I /O communication conflict does exist. Other embodiments hypervisor upon being executed by the processor. Imple
of this aspect include corresponding computer systems, mentations of the described techniques may include hard
 US 10,788,990 B2
5 6
ware, a method or process, or computer software on a
computer - accessible medium.
BRIEF DESCRIPTION OF THE DRAWINGS
The disclosure is illustrated by way of example, and not
by way of limitation in the figures of the accompanying
drawings in which like reference numerals are used to refer
to similar elements .
FIG . 1A is a block diagram illustrating a prior art 10
approach to implementing control and entertainment soft
ware without the use of an OS hypervisor or a latency
ystem .
FIG . 1B is a block diagram illustrating an operating
environment for a latency system to manage one or more I /O
requests for an ADAS system set and an infotainment
system of an ego vehicle according to some embodiments.
FIGS . 2A and 2B are block diagrams illustrating example
computer systems including a latency system according to
some embodiments.
FIG . 2C is a block diagram illustrating an example of rule
data according to some embodiments
FIG . 3A is a flowchart of an example method for man
aging one or more I /O requests for an ego vehicle according
to some embodiments.
FIGS . 3B and 3C are a flowchart of an example method
for managing one or more I/ O requests for a set of ADAS
systems and one or more infotainment systems of an ego
vehicle according to some embodiments.
FIG . 4A is a block diagram of an example virtual 1/0
strategy implemented by the latency system according to
some embodiments .
FIG . 4B is a block diagram of an example direct I /O
strategy implemented by the latency system according to
some embodiments.
DETAILED DESCRIPTION
ADAS System Set
An ADAS system set includes one or more ADAS sys-
tems . Examples of an ADAS system may include one or
more of the following elements of a vehicle: an adaptive
cruise control (" ACC ” ) system ; an adaptive high beam
system ; an adaptive light control system ; an automatic
parking system ; an automotive night vision system ; a blind
spot monitor; a collision avoidance system ( sometimes
called a pre -collision system , or a “ PCS ” ); a crosswind
stabilization system ; a driver drowsiness detection system ; a
driver monitoring system ; an emergency driver assistance
system ; a forward collision warning system ; an intersection
assistance system ; an intelligent speed adaption system ; a
lane departure warning system ( sometimes called a lane
keep assist system , or “ LKA ” ); a pedestrian protection
system ; a traffic sign recognition system ; a turning assistant;
and a wrong -way driving warning system .
The ADAS system may also include any software or
hardware included in the vehicle that makes that vehicle be
an autonomous vehicle or a semi- autonomous vehicle .
DSRC -equipped Vehicle
In some embodiments, the vehicle (e.g. , the vehicle 123 60
depicted in FIG . 1B ) is a DSRC - equipped vehicle . A DSRC
equipped vehicle is a vehicle that includes the following
elements: a DSRC transceiver and any software or hardware
necessary to encode and transmit a DSRC message ; a DSRC
receiver and any software or hardware necessary to receive 65
and decode a DSRC message ; and a DSRC -compliant GPS
unit . The vehicle may include a communication unit ( e.g. ,
the communication unit 145 depicted in FIG . 1B ) that
includes the DRC transceiver and the receiver, as
well as any software necessary for these hardware elements
to provide their functionality.
5 Lane- level Accuracy
A DSRC -compliant GPS unit can provide GPS data (or
GPStags ) describing the location of vehicle and instances
of sensor measurements as described by the sensor data )
with lane - level accuracy. Lane level accuracy may mean that
the location of a vehicle is described so accurately that the
vehicle's lane of travel may be accurately determined when
traveling under an open sky ( e.g. , plus or minus 1.5 meters
of the actual location of the vehicle ) . A conventional GPS
system is unable to determine the location of a vehicle with
15 lane - level accuracy. For example, a typical lane of a road
way is approximately 3 meters wide . However, a conven
tional GPS system may only have an accuracy of plus or
minus 10 meters relative to the actual location of the vehicle .
A DSRC - compliant GPS unit may include hardware that
20 wirelessly communicates with a GPS satellite to retrieve
GPS data that describes catif vehicle with
precision that is compliant with the DSRC standard . The
DSRC standard requires that GPS data be precise enough to
infer ifto vehicles are in the samelan.ADSRC -comp
25 ant GPS unit may be operable to identify, monitor and track
its two - dimensional position within 1.5 meters of its actual
position 68 % of the time under an open sky. Since lanes of
a roadway are typically no less than 3 meters wide, when
ever the two - dimensional error of the GPS data is less than
30 1.5 meters the reconstruction module described herein may
analyze the GPS data provided by the DSRC -compliant GPS
unit and determine what lane of the roadway the vehicle is
traveling in based on the relative positions of vehicles on the
radway. In this way, the DRC - compliant GPS unit may
35 beneficially provide GPS data with lane - level accuracy,
thereby enabling the latency system to more accurately
determine the driving situation for the vehicle and route I /O
requests for the vehicle accordingly .
Example Overview
40 Referring to FIG . 1A , depicted is a prior art approach to
implementing control and entertainment software without
the use of an OS hypervisor or a latency system .
The first ECU is a hardware device that includes a first
memory and a first microcontroller. The first memory stores
45 control software and a first operating system compatible
with the control software . The first microcontroller executes
the first operating system , and the control software within
the environment provided by the first operating system , to
provide control functionality to the platform .
50 The second ECU is a hardware device that includes a
second memory and a second microcontroller. The second
memory stores entertainment software and a second oper
ating system compatible with the entertainment software.
The second microcontroller executes the second operating
55 system , and the entertainment software within the environ
ment provided by the second operating system , to provide
entertainment functionality to the platform .
This approach requires two separate hardware ECUS ,
which is more expensive than using only one hardware
ECU . This approach also does not include any functionality
to manage communication conflicts that occur when the
control software and the entertainment software each
address separate I/ O requests to the same I /O devices of the
platform at the same time or contemporaneous to one
another.
Referring now to FIG . 1B , depicted is an operating
environment 100 for a latency system 199 to manage one or
 US 10,788,990 B2
7 8
more I/ O requests for an ADAS system set 180 and an The vehicle 123 may include one or more of the following
infotainment system 186 of a vehicle 123 according to some elements: a sensor set 182 ; an actuator set 184 ; a hardware
embodiments . ECU 198 ; a processor 125 , a memory 127 ; the latency
The operating environment 100 may include one or more system 199 ; a communication unit 145 ; a DSRC - compliant
of the following elements: the vehicle 123 ; and an infotain- 5 GPS unit 170 ; a ADAS system set 180 ; an infotainment
system 186. These elements of the vehicle 123 may be
ment server 103 ( herein “ the server 103” ). These elements of communicatively
the operating environment 100 may be communicatively coupled to one another via a bus 120 .
In some embodiments
coupled to a network 105. Although only one vehicle 123 , onboard vehicle computer , the vehicle 123 includes an
system (not pictured ). The
one server 103 and one network 105 are depicted in FIG . 1B , 10 onboard vehicle computer system
in practice the operating environment 100 includes one or or control the operation of the latency may be operable to cause
system 199. The
more vehicles 123 , one or more servers 103 and one or more onboard vehicle computer system may be operable
networks 105 . to access
and execute the data stored on the memory 127
The network 105 may be a conventional type, wired or the functionality described herein for the latency systemto provide
wireless, and may have numerous different configurations 15 or its elements.
199
including a star configuration, token ring configuration, or In some embodiments , the sensor set 182 includes one or
other configurations. Furthermore, the network 105 may more of the following vehicle sensors : a camera ; a millime
include a local area network (LAN ), a wide area network ter wave radar, a speed sensor ; a LIDAR sensor ; a laser
( WAN ) (e.g. , the Internet ), or other interconnected dataaltimeter; a navigation sensor (e.g. , a global positioning
paths across which multiple devices and /or entities may 20 system sensor of the DSRC -compliant GPS unit 170 ) ; an
communicate. In some embodiments , the network 105 may infrared detector ; a motion detector; a thermostat; a sound
include a peer-to - peer network . The network 105 may also detector, a carbon monoxide sensor ; a carbon dioxide sen
be coupled to or may include portions of a telecommunica sor ; an oxygen sensor ; a mass air flow sensor ; an engine
tions network for sending data in a variety of different coolant temperature sensor ; a throttle position sensor; a
communication protocols. In some embodiments, the net- 25 crank shaft position sensor; an automobile engine sensor ; a
work 105 includes Bluetooth® communication networks or valve timer ; an air - fuel ratio meter ; a blind spot meter ; a curb
a cellular communications network for sending and receiv feeler , a defect detector; a Hall effect sensor, a manifold
ing data including via short messaging service ( SMS ) , absolute pressure sensor ; a parking sensor; a radar gun ; a
multimedia messaging service ( MMS ) , hypertext transfer speedometer, a speed sensor; a tire -pressure monitoring
protocol ( HTTP) , direct data connection , wireless applica- 30 sensor; a torque sensor; a transmission fluid temperature
tion protocol (WAP ), e - mail , DSRC , full-duplex wireless sensor; a turbine speed sensor (TSS ) ; a variable reluctance
communication , etc. The network 105 may also include a sensor ; a vehicle speed sensor ( VSS ); a water sensor ; a
mobile data network that may include 3G , LTE , VOLTE wheel speed sensor ; and any other type of automotive
or any other cellular network , mobile data network or sensor. The sensor set 182 may include any sensor which is
combination of mobile data networks . Further, the network 35 present in an autonomous vehicle or an HAV .
105 may include one or more IEEE 802.11 wireless net The sensor set 182 may be operable to record data
works . Full-duplex communication includes the full -duplex ( referred to herein as " sensor data 195 " ) that describes one
wireless communication messages described in U.S. Pat . or more measurements of the sensors included in the sensor
No. 9,369,262 filed on Aug. 28 , 2014 and entitled “ Full set 182 .
Duplex Coordination System ,” the entirety of which is 40 The sensor set 182 may include one or more sensors that
hereby incorporated by reference . are operable to measure the physical environment outside of
In some embodiments, the vehicle 123 may be DSCR the vehicle 123. For example, the sensor set 182 may record
equipped. The network 105 may include one or more one or more physical characteristics of the physical envi
communication channels shared among the vehicle 123 and ronment that is proximate to the vehicle 123. The measure
one or more other wireless communication devices (e.g. , the 45 ments recorded by the sensor set are described by sensor
server 103 or other vehicles 123 present in the operating data 195 which are stored in the memory 127 of the vehicle .
environment 100 ) . The communication channel may include In some embodiments, the sensor data 195 may describe
DSRC , full-duplex wireless communication, millimeter the physical environment proximate to the vehicle at one or
wave communication or any other wireless communication more times . The sensor data 195 may be timestamped by the
protocol. For example, the network 105 may be used to 50 sensors of the sensor set 182 or the latency system 199. The
transmit a DSRC message , DSRC probe or BSM to the sensor data 195 is stored in the memory 127 .
server 103 ( which may be , for example, and element of a In some embodiments, the sensor set 182 includes various
roadside unit ). sensors such as cameras, LIDAR, range finders, radar, etc.
The vehicle 123 may include a car, a truck , a sports utility that are operational to measure , among other things: ( 1 ) the
vehicle, a bus , a semi- truck , a drone or any other roadway- 55 physical environment, or roadway environment, where the
based conveyance . A roadway -based conveyance is a hard vehicle 123 is located as well as the static objects within this
ware devices that traverses the top surface of a roadway. physical environment; ( 2 ) the dynamic objects within the
In some embodiments, the vehicle 123 may include an physical environment and the behavior of these dynamic
autonomous vehicle , a semi-autonomous vehicle or Highly objects; ( 3 ) the position of the vehicle 123 relative to static
Automated Vehicle (“ HAV ” ). For example, the vehicle 123 60 and dynamic objects within the physical environment ( e.g. ,
may include a ADAS system set 180 which are operable to as recorded by one or more range - finding sensors of the
make the vehicle 123 an autonomous vehicle . An HAV is a sensor set 182 such as LIDAR) ; ( 4 ) the weather within the
vehicle 123 whose ADAS system set 180 operate at Level 3 physical environment over time and other natural phenom
or higher as defined by the NHTSA beginning on page 9 of enon within the physical environment over time ; ( 5 ) coef
their policy paper entitled “ Federal Automated Vehicles 65 ficients of friction and other variables describing objects
Policy : Accelerating the Next Revolution in Roadway ( static and dynamic ) within the physical environment over
Safety,” which was published in September of 2016 . time ; and ( 6 ) the operation of the ADAS system set 180 in
 US 10,788,990 B2
9 10
response to the static and dynamic objects over time . One or similar port for wired communication with the network 105 .
more of these measurements are described by the sensor data In some embodiments, the communication unit 145 includes
195. The sensor data 195 may include a timestamp for one a wireless transceiver for exchanging data with the network
or more of the measurements of the sensors included in the 105 or other communication channels using one or more
sensor set 182 . 5 wireless communication methods, including: IEEE 802.11 ;
The actuator set 184 includes one or more actuators of the IEEE 802.16 , BLUETOOTH® ; EN ISO 14906 : 2004 Elec
vehicle 123. For example , the actuator set 184 includes one tronic Fee Collection Application interface EN 11253 : 2004
or more of the following: one or more hydraulic actuators; Dedicated Short - Range Communication — Physical layer
one or more electric actuators ; one or more thermal actua using microwave at 5.8 GHz ( review ); EN 12795 : 2002
tors ; one or more magnetic actuators; one or more shape 10 Dedicated Short -Range Communication ( DSRC ) —DSRC
memory alloys configured in the vehicle 123 to operate as an Data link layer: Medium Access and Logical Link Control
actuator ; and one or more mechanical actuators. (review ); EN 12834 : 2002 Dedicated Short -Range Commu
The hardware ECU 198 is a hardware electronic control nication - Application layer (review ); EN 13372 : 2004 Dedi
unit . As depicted in FIG . 1B , the ECU 198 includes the cated Short -Range Communication (DSRC )—DSRC pro
following elements: the memory 127 ; the sensor data 195 15 files for RTTT applications ( review ); the communication
(which may be stored on the memory 127 ) ; the processor method described in U.S. Pat. No. 9,369,262 filed on Aug.
125 , and the latency system 199 . 28 , 2014 and entitled “ Full - Duplex Coordination System ” ;
The memory 127 is a non -transitory computer -readable or another suitable wireless communication method .
memory. The memory 127 stores instructions or data that In some embodiments, the communication unit 145
may be executed by the processor 125. The instructions or 20 includes a full -duplex coordination system as described in U
data may include code for performing the techniques U.S. Pat. No. 9,369,262 filed on Aug. 28 , 2014 and entitled
described herein . The memory 127 stores any data or “ Full-Duplex Coordination System .”
information necessary for the latency system 199 to provide In some embodiments, the communication unit 145
its functionality. For example, the memory 127 stores the includes a cellular communications transceiver for sending
sensor data 195. In some embodiments, the memory 127 25 and receiving data over a cellular communications network
also stores GPS data (e.g. , the GPS data 293 depicted in FIG . including via short messaging service ( “ SMS ” ), multimedia
2A) that describes the geographic location of the vehicle 123 messaging service (“ MMS ” ), hypertext transfer protocol
at a specific time as determined by the DSRC -compliant (“HTTP ” or “ HTTPS ” if the secured implementation of
GPS unit 170 ( such that the geographic location has lane HTTP is used) , direct data connection, WAP, e -mail , or
level accuracy ). In some embodiments, the memory 127 30 another suitable type of electronic communication . In some
stores one or more of the following digital data elements embodiments, the communication unit 145 includes a wired
depicted in FIG . 2A : the sensor data 295 ; the GPS data 293 ; port and a wireless transceiver. The communication unit 145
map data 291 ; situation data 289 ; rule data 287 and virtual also provides other conventional connections to the network
ECU data 285 . 105 for distribution of files or media objects using standard
In some embodiments, the memory 127 may be a dynamic 35 network protocols including TCP / IP, HTTP, HTTPS , and
random access memory (DRAM ) device , a static random SMTP, millimeter wave , DSRC , etc.
access memory (SRAM ) device, flash memory, or some The latency system 199 is described in more detail below .
other memory device. In some embodiments, the memory The DSRC - compliant GPS unit 170 includes hardware
127 also includes a non - volatile memory or similar perma that wirelessly communicates with a GPS satellite to retrieve
nent storage device and media including a hard disk drive, 40 GPS data 293 that describes a location of the vehicle 123 at
a floppy disk drive, a CD - ROM device, a DVD - ROM a given time . In some embodiments, a DSRC -compliant
device , a DVD -RAM device, a DVD - RW device , a flash GPS unit 170 is operable to provide GPS data ( e.g. , the GPS
memory device , or some other mass storage device for data 293 depicted in FIG . 2A) that describes the location of
storing information on a more permanent basis . the vehicle 123 to a lane - level degree of precision. The
The processor 125 includes an arithmetic logic unit, a 45 DSRC standard requires that GPS data be precise enough to
microprocessor, a general -purpose controller, or some other infer iftwo vehicles ( such as vehicle 123 and another vehicle
processor array to perform computations and provide elec on the same roadway as the vehicle 123 ) are in the same lane
tronic display signals to a display device . The processor 125 at the same time .
processes data signals and may include various computing In some embodiments, the DSRC - compliant GPS unit
architectures including a complex instruction set computer 50 170 is operable to identify, monitor and track its two
(“ CISC ” ) architecture, a reduced instruction set computer dimensional position within 1.5 meters of its actual position
(“ RISC ” ) architecture , or an architecture implementing a 68 % of the time under an open sky. Since lanes of a roadway
combination of instruction sets . Although FIG . 1B includes are typically no less than 3 meters wide , whenever the
a single processor 125 , multiple processors may be included two - dimensional error of the GPS data is less than 1.5
in the vehicle 123 (and the server 103 ) . Other processors , 55 meters the latency system 199 may analyze the GPS data
operating systems, sensors, displays , and physical configu provided by the DSRC -compliant GPS unit 170 and deter
rations may be possible . mine what lane of the roadway the vehicle 123 is traveling
The communication unit 145 transmits and receives data in based on the relative positions of vehicles on the roadway.
to and from a network 105 or to another communication By comparison, a GPS unit which is not compliant with
channel. In some embodiments , the communication unit 145 60 the DSRC standard is far less accurate than the DSRC
may include a DSRC transceiver, a DSRC receiver and other compliant GPS unit 170 and not capable of reliably provid
hardware or software necessary to make the vehicle 123 a ing lane - level accuracy, as is the DSRC - compliant GPS unit
DSRC -enabled device . 170. For example , a non - DSRC - compliant GPS unit may
In some embodiments, the communication unit 145 have an accuracy on the order of 10 meters, which is not
includes a port for direct physical connection to the network 65 sufficiently precise to provide the lane -level degree of pre
105 or to another communication channel. For example , the cision provided by the DSRC -compliant GPS unit 170. For
communication unit 145 includes a USB , SD , CAT- 5 , or example, since a lane may be as narrow as 3 meters wide , the
 US 10,788,990 B2
11 12
DSRC standard may require a DSRC - compliant GPS unit computing device communicatively coupled to a heads -up
170 to have an accuracy on the order of 1.5 meters, which display unit or some other electronic display device ; and etc.
is significantly more precise than a non -DSRC - compliant In some embodiments, the infotainment system 186
GPS unit as described above . As a result , a non -DSRC causes the communication unit 145 to retrieve data from the
compliant GPS unit may not be able to provide GPS data 5 infotainment server 103 via the network 105. The data is
that is accurate enough to enable the latency system 199 to stored in the memory 127 or some other non -transitory
accurate determine the driving scenario that the vehicle 123
is experiencing or about to experience, and so , the latency memory of the vehicle 123. The infotainment system 186
system may inaccurately assign urgency values and route then provides infotainment functionality to a driver of the
I /O requests for as described below with reference to FIGS . 10 vehicle based at least in part on the data provided by the
infotainment server 103. For example , the data provided by
3A , 3B and 3C . The imprecision of a non -DSRC - compliant the infotainment server 103 may include infotainment data
GPS unit may therefore render the functionality of the including digital data describing infotainment content ( e.g. ,
latency system 199 inoperable since the I /O routing deci a movie , music, live television , etc.) including one or more
sions made by the latency system 199 would be inaccurate
in some situations and risk the loss of human life resulting 15 ofelectronic
the following : (1) images that are displayed by an
display of the vehicle ; and (2 ) audio which is
from the vehicle 123 colliding with another object or some
other accident that would have been avoided with GPS data recreated by one or more speakers of the vehicle 123 .
having lane - level accuracy. In some embodiments , the latency system 199 includes
The vehicle 123 includes a ADAS system set 180 , which code and routines that are operable, when executed by the
collectively form an autonomous system . Each ADAS sys- 20 processor 125 , to selects one of two different routing strat
tem 180 provides one or more autonomous features to the egies for managing one or more I /O requests for an OS
vehicle 123 . hypervisor of the hardware ECU 198. A first routing strategy
In some embodiments, the ADAS system set 180 included for managing one or more I/O requests is referred to herein
in the vehicle 123 render the vehicle 123 a Highly Auto as “ virtual I/O ” and depicted in FIG . 4A . A second routing
mated Vehicle ( “HAV ” ). An HAV is a vehicle whose ADAS 25 strategy for managing one or more I /O requests is referred
system set 180 operate at Level 3 or higher as defined by the to herein as “ direct I/O ” and depicted in FIG . 4B .
NHTSA on page 9 of their policy paper entitled “ Federal As a first ADAS system executes ( a “ first execution ” ), it
Automated Vehicles Policy : Accelerating the Next Revolu generates one or more I /O requests that are addressed to one
tion in Roadway Safety , " which was published in September or more I /O devices ( e.g. , sensors or actuators of the vehicle
of 2016. Accordingly, in some embodiments the vehicle 123 30 123 ) . A second ADAS system or the infotainment system
is a HAV . In this way, the vehicle 123 may be an HAV and may be executing contemporaneous to the execution of the
also DSRC - enabled as described above . first ADAS system (a “ second execution ” ). This second
The ADAS system set 180 includes one or more ADAS execution may produce one or more I /O requests that are
systems . An ADAS system from the ADAS system set 180 addressed to one or more I /O devices that are the same as
is referred to herein individually or collectively as “ an 35 those addressed by the first execution , thereby creating an
ADAS system , ” “ the ADAS system ,” “ the ADAS systems," I / O communication conflict. The latency system 199 solves
“ the one or more ADAS systems ” or “ the plurality of ADAS this problem .
systems. ” For example, the latency system 199 described herein
The one or more ADAS systems may include one or more includes codes and routines which , when executed by the
advanced driver assistance systems . Examples of an ADAS 40 processor 125 , causes the processor 125 to execute one or
system may include one or more of the following elements more of the following steps : ( 1 ) determine whether one or
of the vehicle 123 : an ACC system ; an adaptive high beam more I /O requests generated by the second execution creates
system ; an adaptive light control system ; an automatic an I/O communication conflict with the first execution ; (2 )
parking system ; an automotive night vision system ; a blind if no I / O communication conflict is present, directly routing
spot monitor ; a collision avoidance system ; a crosswind 45 the I /O requests to the I/ O devices which they address ( see ,
stabilization system ; a driver drowsiness detection system ; a e.g. , FIG . 4B which depicts an example of direct I /O
driver monitoring system ; an emergency driver assistance according to some embodiments); and ( 3 ) if an I /O com
system ; a forward collision warning system ; an intersection munication conflict is present, (i ) assign an urgency value
assistance system ; an intelligent speed adaption system ; a for individual I/ O requests of the first execution and the
lane departure warning system ; a pedestrian protection sys- 50 second execution based on whether the vehicle feature these
tem ; a traffic sign recognition system ; a turning assistant; I / O requests are associated with providing is associated with
and a wrong -way driving warning system . safety, (ii ) place the conflicted I/ O requests in a hypervisor
In some embodiments, the ADAS system set 180 includes I/ O queue of the OS hypervisor; and ( iii ) process the I /O
any hardware or software that controls one or more opera request in the hypervisor I /O queue based on the relative
tions of the vehicle 123 so that the vehicle 123 is “ autono- 55 urgency values for the queued I/O requests (see, e.g. , FIG .
mous" or " semi- autonomous." 4A which depicts an example of virtual I/O according to
In some embodiments, the ADAS system set 180 includes some embodiments ).
any hardware or software that controls one or more opera In some embodiments , the latency system 199 includes
tions of the vehicle 123 so that the vehicle 123 is an HAV . code and routines that, when executed by the processor 125 ,
The infotainment system 186 includes one or more of the 60 causes the processor 125 to execute one or more of the steps
following: one or more in - car entertainment systems of the of the method 300 described below with reference to FIG .
vehicle 123 ; and one or more in - car infotainment systems of 3A .
the vehicle 123. In some embodiments , the infotainment In some embodiments , the latency system 199 includes
system 186 includes one or more of the following: a navi code and routines that, when executed by the processor 125 ,
gation system ; a video streaming vehicle ; a music streaming 65 causes the processor 125 to execute one or more of the steps
device ; a carputer; a tablet computing device installed within of the method 399 described below with reference to FIGS .
the vehicle as an infotainment device; a network - enabled 3B and 3C .
 US 10,788,990 B2
13 14
The functionality of the latency system 199 is described ADAS system set 180 ; the sensor set 182 ; the actuator set
in more detail below with reference to FIGS . 2 , 3A , 3B , 3C , 184 ; and the hardware ECU 198 .
4A and 4B . The memory 127 stores any data necessary for the latency
In some embodiments, the latency system 199 may be system 199 to provide its functionality. For example, the
implemented using hardware including a field -program 5 memory 127 stores any of the data described above with
mable gate array ( “ FPGA ” ) or an application -specific inte reference to FIG . 1B or below with reference to FIGS . 2B ,
grated circuit (“ ASIC ” ). In some other embodiments, the 2C , 3A , 3B , 3C , 4A and 4B .
latency system 199 may be implemented using a combina In the embodiment depicted in FIG . 2 the memory 127
tion of hardware and software. The latency system 199 may 10
stores the following digital data : sensor data 295 ; GPS data
be stored in a combination of the devices of the operating 293; map data 291 ; situation data 289 , rule data 287 ; virtual
environment 100 ( e.g. , vehicles, servers or other devices ECU data 285 ; and feature data 283 .
such as a smartphone of the driver of the vehicle 123 ) , or in The sensor data 295 includes digital data that describes
one of the devices . the roadway environment outside an ego vehicle ( i.e. , the
Although not depicted in FIG . 1B , in some embodiments 15 vehicle 123 ) , the location of remote vehicles, other objects
the vehicle 123 may include a full -duplex coordination in the roadway environment relative to the ego vehicle and
system as described in U.S. Pat . No. 9,369,262 and entitled the operational status of the ego vehicle (e.g. , kinematic data
“ Full-Duplex Coordination System . ” for the vehicle 123 and whether any faults or alerts are
The server 103 is a processor- based computing device . detected for the vehicle 123 by one of the onboard vehicle
For example , the computing device may include a stand- 20 computers of the vehicle 123 ) .
alone hardware server. The server 103 is communicatively In some embodiments , the ego vehicle is the vehicle that
coupled to the network 105. The server 103 may include a includes the computer system 200. In some embodiments,
non - transitory memory that stores infotainment data . The roady environment includes the physical environment
server may receive requests for infotainment data via the where an ego vehicle including the computer system 200 is
network 105 and respond to these requests by transmitting 25 located , including a roadway upon which the ego vehicle
infotainment data via the network 105 that is responsive to travels .
these requests. In some embodiments, the operational status of the ego
Referring now to FIG . 2A , depicted is a block diagram vehicle
illustrating an example computer system 200 including a describesis one information described by the sensor data 295 that
or more of the following: kinematic data for
latency system 199 according to some embodiments. 30
In some embodiments , the computer system 200 may the vehicle 123 ; by
the vehicle 123 whether any faults or alerts are detected for
include a special -purpose computer system that is pro the vehicle 123 ; theonelatitude of the onboard vehicle computers of
and longitude of the vehicle
grammed to perform one or more steps of a method 300 123 ; the heading of the vehicle 123 ; the breaking system
described below with reference to FIG . 3A or a method 399
described below with reference to FIGS . 3B and 3C . 35 status of the vehicle 123 ( e.g. , data describing whether the
In some embodiments, the computer system 200 is an breaks are engaged ); the elevation of the vehicle 123 ; the
element of the vehicle 123. For example, the computer current time for the vehicle 123 ; the speed of the vehicle
system 200 is an onboard vehicle computer system of the 123 ; the steering angle of the vehicle 123 ; the acceleration
vehicle 123 . of the vehicle 123 as indicated by three axes of acceleration
The computer system 200 may include one or more of the 40 and the yaw rate of the vehicle 123 ; the path history of the
following elements according to some examples: the latency vehicle 123 ; an estimate of the further path of the vehicle
system 199 ; the processor 125 ; the communication unit 145 ; 123 ; whether the traction control system of the vehicle 123
the memory 127 ; the DSRC -compliant GPS unit ; the info has been engaged for 100 milliseconds or more in the past
tainment system 186 ; the ADAS system set 180 ; the sensor 0.1 seconds; whether the antilock braking system of the
set 182 ; the actuator set 184 ; and the hardware ECU 198. 45 vehicle 123 has been engaged for 100 milliseconds or more
The components of the computer system 200 are commu in the past 0.1 seconds ; whether the vehicle head lamps are
nicatively coupled by a bus 220 . one; whether the vehicle windshield wipers are on ; the
In the illustrated embodiment, the processor 125 is com make, model and trim level of the vehicle 123 ; and any other
municatively coupled to the bus 220 via a signal line 238 . data that is required to be included in a basic safety message
The communication unit 145 is communicatively coupled to 50 of the vehicle 123 as required by the DSRC standard .
the bus 220 via a signal line 246. The memory 127 is The GPS data 293 includes digital data that describes the
communicatively coupled to the bus 220 via a signal line geographic location of the ego vehicle ( e.g. , the vehicle 123
244. The DSRC -compliant GPS unit 170 is communica which includes the computer system 200 ) . In some embodi
tively coupled to the bus 220 via a signal line 230. The ments, the GPS data 293 may be received from one or more
infotainment system 186 is communicatively coupled to the 55 GPS satellites by the DSRC -compliant GPS unit 170. The
bus 220 via a signal line 234. The ADAS system set 180 is GPS data 293 may describe a latitude and longitude of the
communicatively coupled to the bus 220 via a signal line ego vehicle with lane - level accuracy.
242. The sensor set 182 is communicatively coupled to the The map data 291 includes digital data that describes, for
bus 220 via a signal line 232. The actuator set 184 is different combinations of latitude and longitude as indicated
communicatively coupled to the bus 220 via a signal line 60 by the GPS data 293 , different geographical features of the
236. The ECU is communicatively coupled to the bus 220 roadway environment indicated by the GPS data 293 such as
via a signal line 240 . the presence of curves in the roadway, whether this a bumpy
The following elements of the computer system 200 were road , the average vehicular speeds along the roadway at
described above with reference to FIG . 1B , and so , those different times of day, etc.
descriptions will not be repeated here: the processor 125 ; the 65 The monitor module 204 analyzes one or more of the
communication unit 145 ; the memory 127 ; the DSRC sensor data 295 , the GPS data 293 and the map data 291 and
compliant GPS unit; the infotainment system 186 ; the determines the situation data 289 based on these inputs. The
 US 10,788,990 B2
15 16
situation data 289 includes digital data that describes the 199 and other components of the computer system 200 of
current driving situation of ego vehicle as determined by the FIG . 2A or the computer system 201 of FIG . 2B .
monitor module 204 . The communication module 202 sends and receives data ,
In some embodiments, the monitor module 204 continues via the communication unit 145 , to and from one or more
to track the sensor data 295 , GPS data 293 and the map data 5 elements of the operating environment 100. For example ,
291 over time . As time passes , the current driving situation the communication module 202 receives or transmits, via
of the ego vehicle changes. When this change occurs , the the communication unit 145 , infotainment data which is
monitor module 204 updates the situation data 289 to reflect stored on the memory 127 .
the new driving situation of the ego vehicle. In some embodiments, the communication module 202
The rule data 287 includes digital data that describes a 10 receives data from components of the latency system 199
rule table ( or some other data structure ) which describes the and stores the data in the memory 127. For example , the
following: ( 1 ) different vehicle features installed in the ego communication module 202 receives any of the data
vehicle (e.g. , different ADAS systems or infotainment sys described above with reference to the memory 127 and
tems installed in the vehicle 123 ) , ( 2 ) the different sensors of stores this data in the memory 127 .
the sensor set 182 used by these different vehicle features ; 15 In some embodiments, the communication module 202
( 3 ) different “ criteria ” relevant to the vehicle features may handle communications between components of the
installed in the ego vehicle (e.g. , criteria for triggering an latency system 199. For example, the communications mod
ADAS system of the ADAS system set 180 to take action by ule 202 may handle communications among the monitor
providing it's ADAS functionality, or the criteria required module 204 , the I /O control module 206 and the OS hyper
for an infotainment system 186 to provide it's infotainment 20 visor 259. Any of these modules may cause the communi
functionality, etc. ) ; and ( 4 ) a threshold associated with the cation module 202 to communicate with the other elements
criteria (e.g. , a threshold that is relevant to the triggering of of the computer system 200 of FIG . 2A or the computer
the ADAS system of the ADAS system set 180 to provide system 201 of FIG . 2B .
it's ADAS functionality, or a threshold relevant to triggering For example, the monitor module 204 may use the com
the infotainment system 186 to provide its infotainment 25 munication module 202 to communicate with the sensor set
functionality, etc. ). 182 , the DSRC -compliant GPS unit 170 and the memory
The monitor module 204 uses the rule table described by 127 so that monitor module 204 can analyze the sensor data
the rule data 287 to identify when the current driving 295 , the GPS data 293 and the map data 291 to generate the
situation of the ego vehicle indicates that a particular ADAS situation data 289 .
system is likely to be used in the near future, and as such, is 30 In some embodiments, the communication module 202
relevant to the current driving situation described by the can be stored in the memory 127 of the computer system 200
situation data 289. An example of the rule data 287 is and can be accessible and executable by the processor 125 .
depicted in FIG . 2C . The communication module 202 may be adapted for coop
The virtual ECU data 285 includes digital data that, when eration and communication with the processor 125 and other
executed by the processor 125 , causes the processor 125 to 35 components of the computer system 200 via signal line 222 .
generate one or more virtual ECUs which operate on the The monitor module 204 can be software including rou
hardware ECU 198 and include their own operating systems. tines for causing the processor 125 to execute steps includ
For example, see FIGS . 2B , 4A and 4B which each include ing : ( 1 ) monitoring the current driving situation of the ego
a hardware ECU 198 executing different virtual ECUS vehicle ( referred to below as “ step ( 1 ) ” ); and (2 ) identifying
having their own operating systems . The operating systems 40 if the current driving situation is relevant to one or more of
operate on the virtual ECUs , which themselves operate on the ADAS systems of the ADAS system set 180 installed in
the hardware ECU 198. The virtual ECU data 285 includes the ego vehicle ( referred to below as “ step ( 2 ) " ).
digital data for generating the one or more virtual ECUs and Regarding step ( 1 ) , monitoring the current driving situa
the one or more operating systems. tion of the vehicle, the monitor module 204 includes code
The feature data 283 includes digital data describing the 45 and routines that are operable, when executed by the pro
one or more vehicle features that are implemented respon cessor 125 , to cause the processor to access one or more of
sive to the current driving situation described by the situa the following digital data which is stored on the memory
tion data 289 127 : the sensor data 295 collected by the vehicle sensors
In the illustrated embodiment shown in FIG . 2A , the included in the sensor set 182 ; the GPS data 293 that
latency system 199 includes a communication module 202 , 50 describes the current location of the ego vehicle ; and the
a monitor module 204 , an I/ O control module 206 , and an map data 291 that describes geographic or roadway features
OS hypervisor 259. These components of the latency system that are associated with the GPS data 293 .
199 are communicatively coupled each other via a bus In some embodiments, the monitor module 204 includes
220 . code and routines that are operable , when executed by the
In some embodiments, components of the latency system 55 processor 125 , to cause the processor 125 to generate
199 can be stored in a single server or device . In some other situation data 289 that describes the current driving situation
embodiments, components of the latency system 199 can be of the ego vehicle based on one or more of the following: ( 1 )
distributed and stored across multiple servers or devices. In the sensor data 295 ; (2 ) the GPS data 293 ; and ( 3 ) the map
some embodiments , the latency system 199 is an element of data 291 .
the hardware ECU 198 . 60 The monitor module 204 causes the communication mod
The communication module 202 can be software includ ule 202 to store the situation data 289 in the memory 127 .
ing routines for handling communications between the In some embodiments, monitor module 204 may include
latency system 199 and other components of the computer code and routines that are operable, when executed by the
system 200. In some embodiments, the communication processor 125 , to cause the processor to continue to track the
module 202 can be a set of instructions executable by the 65 sensor data 295 , the GPS data 293 and the map data 291 over
processor 125 to provide the functionality described below time . As time passes , the current driving situation of the ego
for handling communications between the latency system vehicle changes. When this change occurs , the monitor
 US 10,788,990 B2
17 18
module 204 , upon execution by the processor 125 , causes based on ( 1 ) the urgency of the vehicle feature and (2 ) the
the processor 125 to determine one or more updates for the presence of one or more I /O communication conflicts.
situation data 289 to reflect the new driving situation of the The urgency of the vehicle feature relates to whether the
ego vehicle as indicated by the new sensor data 295 , the new vehicle feature is associated with something more important,
GPS data 293 and the new map data 291 which has changed 5 such as passenger safety, or something less important, such
over time. as passenger entertainment or infotainment.
Regarding step ( 2 ) , identifying if the current driving The presence of I /O communication conflicts are identi
situation is relevant to one or more of the ADAS systems of fied by the I/ O control module 206 based on the other
the ADAS system set 180 installed in the ego vehicle, the vehicle features that are currently active at the same time as
monitor module 204 includes code and routines that are 10 the particular vehicle feature described by the feature data
operable , when executed by the processor 125 , to cause the 283 and a commonality of the sensors , actuators and /or other
processor to monitor the situation data 289 describing the devices that are contemporaneously used by these other
current driving situation of the ego vehicle and determine, vehicle features and the particular vehicle feature described
for a current driving situation , what ADAS features or by the feature data 283 .
ADAS systems are likely to be relevant to the current 15 In some embodiments, the monitor module 204 can be
driving situation or a driving situation occurring in the near stored in the memory 127 of the computer system 200 and
future as indicated by the current driving situation . can be accessible and executable by the processor 125. The
In some embodiments, an ADAS feature or ADAS system monitor module 204 may be adapted for cooperation and
is relevant to the current driving situation ( or the driving communication with the processor 125 and other compo
situation occurring in the near future ) if it is likely to be used 20 nents of the computer system 200 via the signal line 224 .
while this current driving situation is present (or while the As described above , implementing the OS hypervisor 259
driving situation occurring in the near future is present). introduces a bottlenecking problem for managing I / O
For example, assume that an ego vehicle is equipped with requests which are generated by one or more of: the one or
the pre -collision system , which is a type of ADAS system more ADAS systems, and the one or more infotainment
referred to as a “ PCS.” The features of this particular ADAS 25 systems 186 .
system are likely to be used when the current driving The I/ O control module 206 implements one of two
situation indicates that the ego vehicle is approaching a different routing strategies for managing the I/ O requests for
pedestrian (as well as other driving situations indicating a the OS hypervisor 259 so that the bottlenecking problem is
pending vehicle collision if remedial action is not taken ). decreased or eliminated : direct I /O ( see , e.g. , FIG . 4B ) ; and
The monitor module 204 retrieves the rule data 287 which 30 virtual I /O ( see , e.g. , FIG . 4A) .
describes one or more of the following: ( 1 ) different vehicle The I /O control module 206 includes software and rou
features installed in the ego vehicle (e.g. , different ADAS tines that, when executed by the processor 125 , causes the
systems or infotainment systems ); (2 ) the different sensors processor 125 to determine which of these two different
of the sensor set 182 used by these different vehicle features; routing strategies are applied to the I/ O requests.
( 3 ) different “ criteria ” relevant to the vehicle feature (e.g. , 35 As described above , the monitor module 204 determines
criteria for triggering an ADAS feature / ADAS system to which vehicle features are implemented responsive to the
take action , criteria required for an infotainment system to current driving situation . The monitor module 204 generates
provide its functionality, etc. ) ; and (4 ) a threshold associated feature data 283 that describes the vehicle features that are
with the criteria (e.g. , a threshold that is relevant to the implemented responsive to the current driving situation . The
triggering of the ADAS feature / ADAS system , a threshold 40 monitor module 204 causes the communication module 202
relevant to providing an infotainment feature , etc. ) . The to store the feature data 283 in the memory 127 .
monitor module 204 uses this rule data 287 to identify when The I/ O control module 206 can be software including
the current driving situation of the ego vehicle indicates that routines that, when executed by the processor 125 , cause the
a particular ADAS feature / ADAS system is likely to be used processor 125 to retrieve the feature data 283 and the rule
in the near future, and as such , is relevant to the current 45 data 287 from the memory 127. The rule data 287 describes
driving situation described by the situation data 289. See which sensors and actuators are used by the vehicle feature
FIG . 2C for an example of the rule data 287 according to described by the feature data 283. The I/O control module
some embodiments. 206 uses the rule data 287 to identify the sensor and actuator
Based on analysis of the rule data 287 and the situation dependency used by the vehicle feature described by the
data 289 , the monitor module 204 , upon being executed by 50 feature data 283 ( see , e.g. , the second column from the left
the processor 125 , causes the processor 125 to determine of the rule data 287 depicted in FIG . 2C ) . In some embodi
which vehicle features are implemented responsive to the ments, the I/ O control module 206 repeats this process for
current driving situation described by the situation data. other vehicle features that are currently active (or are likely
In some embodiments, the monitor module 204 includes to be active in the near future based on the current driving
code and routines that, upon being executed by the processor 55 situation ) so that, for any point in time ( or range of time ) , the
125 , causes the processor 125 to generate feature data 283 I/ O control module 206 is able to identify each of the sensor
that includes digital data describing the one or more vehicle and actuator dependencies for the vehicle features that are
features that are implemented responsive to the current currently active at this point in time ( or range in time ) or are
driving situation described by the situation data 289. The likely to be active in the near future based on the current
monitor module 204 , upon being executed by the processor 60 driving situation .
125 , causes the communication module 202 to store this In some embodiments , the I /O control module 206
feature data 283 in the memory 127. As described below, the includes code and routines that are operable, when executed
I /O control module 206 includes code and routines that, by the processor 125 , to cause the processor to assign an
when executed by the processor 125 , causes the processor urgency value to each vehicle feature described in the
125 to retrieve the feature data 283 from the memory 127 65 preceding paragraph based , for example, on whether a
and determine how to manage the I/ O requests associated particular vehicle feature is associated with safety or merely
with the vehicle feature described by the feature data 283 entertainment or infotainment. The urgency value is
 US 10,788,990 B2
19 20
described by urgency data . The I/ O control module 206 queue and some delay occurs before they are processed ( e.g. ,
includes code and routines that are operable, when executed released from the hypervisor I/ O queue ).
by the processor 125 , to cause the processor 125 to generate In some embodiments, the I/ O control module 206 can be
urgency data describing the urgency value for the different stored in the memory 127 of the computer system 200 and
I /O requests . The I/ O control module 206 includes code and 5 can be accessible and executable by the processor 125. The
routines that are operable, when executed by the processor I/ O control module 206 may be adapted for cooperation and
125 , to cause the processor 125 to instruct the communica communication with the processor 125 and other compo
tion module 202 to store the urgency data in the memory nents of the computer system 200 via signal line 226 .
127 . The OS hypervisor 259 includes software and routines
In some embodiments, I /O requests are assigned an 10 that, when executed by the processor 125 , generate one or
urgency value that is the same as the urgency value assigned more virtual ECUs based on the virtual ECU data 285. Each
to the vehicle feature which they help to implement. For of the virtual ECUs includes an operating system and
example, if the ADAS functionality of an ACC system is software for providing a vehicle feature . In embodiments
assigned a particular urgency value for a period of time , then where a plurality of ECUs are generated by the OS hyper
the I/ O requests generated by the ACC system are assigned 15 visor 259 , the operating systems implemented across the
this same particular urgency value which was assigned to the plurality of ECUs may be heterogenous ( i.e. , " guest OS” if
ACC system for the same period of time as that the ACC singular, “ guest OSes” if plural). The OS hypervisor 259 is
system is assigned this particular urgency value . beneficial because it provides flexibly for adding or remov
The I /O control module 206 includes code and routines ing guest OSes as well as isolation among different guest
that are operable, when executed by the processor 125 , to 20 OSes . The virtual ECU data 285 may include the code and
cause the processor 125 to analyze the sensor dependencies routines for providing the one or more virtual ECUs , the one
and actuator dependencies for each of the vehicle features or more guest OSes and the different vehicle features.
that are currently active or likely to be active in the near In some embodiments, the OS hypervisor 259 can be
future to identify the presence of overlapping dependency of stored in the memory 127 of the computer system 200 and
the sensors and actuators (i.e. , 1/0 devices ) for each active 25 can be accessible and executable by the processor 125. The
vehicle feature , thereby indicating the presence of an 1/0 OS hypervisor 259 may be adapted for cooperation and
communication conflict. The I/ O control module 206 communication with the processor 125 and other compo
includes code and routines that are operable, when executed nents of the computer system 200 via signal line 228 .
by the processor 125 , to cause the processor 125 to generate Referring now to FIG . 2B , depicted is an example of a
conflict data that describes the presence of each I/ O com- 30 computer system 201 including a latency system 199. This
munication conflict and the vehicle functions and I /O computer system 201 includes the memory 127 in which a
devices that are affected by each I /O communication con first virtual ECU 269 and a second virtual ECU 267 are
flict. The I/ O control module 206 includes code and routines being generated by the OS hypervisor 259 responsive to
that are operable , when executed by the processor 125 , to being executed by the processor 125 using the virtual ECU
cause the processor 125 to instruct the communication 35 data as a file input for the OS hypervisor 259 .
module 202 to store the conflict data in the memory 127 . In the depicted embodiment, the first virtual ECU 269 is
The I /O control module 206 includes code and routines operable to provide ADAS functionality (as indicated by the
that are operable , when executed by the processor 125 , to ADAS software 299 ) and the second virtual ECU 267 is
cause the processor 125 to determine which routing strategy operable to provide infotainment system functionality (as
to implement in order to achieve the goal of decreasing or 40 indicated by the infotainment software 296 ) .
minimizing the latency of 1/0 requests for the vehicle The ADAS software 299 includes code and routines that
features that are currently active or likely to the active in the are operable, when executed by the processor 125 , to
near future based on the current driving situation . This provide ADAS functionality and to generate one or more I /O
decrease or minimization of I/ O latency may be considered requests which help to provide this ADAS functionality.
by the I/O control module 206 on a per I /O requests basis or 45 These I/ O requests are addressed to one or more of the
considering the cumulative I /O requests as a whole at any following: one or more sensors of the sensor set 182 ; and
given time or period of time . one or more actuators of the actuator set 184. The first OS
As described above , a first routing strategy is referred to 297 includes a first operating system which is compatible
as “ virtual I / O ” and a second routing strategy is referred to with the ADAS software 299 and the provision of the ADAS
as " direct I/ O .” 50 functionality provided by the ADAS software 299 .
Generally speaking, the virtual I/O routing strategy means The infotainment software 296 includes code and routines
that the I /O requests for a particular vehicle feature are that are operable , when executed by the processor 125 , to
placed in a hypervisor I /O queue of the OS hypervisor 259 provide infotainment functionality and to generate one or
and processed based in part on the urgency of the particular more I / O requests which help to provide this infotainment
vehicle feature which originated the particular I/O request 55 functionality. These 1/0 requests are addressed to one or
relative to the urgency of other I/O requests in the hypervisor more of the following: one or more sensors of the sensor set
I/O queue. 182 ; and one or more actuators of the actuator set 184. The
Generally speaking, the direct I /O routing strategy means second OS 294 includes a second operating system which is
that the I/O requests for the particular vehicle feature are compatible with the infotainment software 296 and the
routed directly to the sensor or actuator which is called (or 60 provision of the infotainment functionality provided by the
addressed ) by each particular I /O request so that the I /O infotainment software 296. The second OS 294 may be
requests for this particular vehicle feature are never placed different than the first OS 297 such that the first OS 297 and
in the hypervisor I /O queue for a period of time to be the second OS 294 form a heterogenous pair of OSes .
determined by the I/ O control module 206 based on the The OS hypervisor 259 includes a hypervisor I/O queue
current driving situation . Direct I /O permission reduces I /O 65 for managing the I/ O requests generated by the ADAS
latency compared to virtual I /O because in virtual I/ O the software 299 and the infotainment software 296. The hyper
individual I /O requests are placed in the hypervisor I /O visor 1/0 queue includes an I /O queue for temporarily
 US 10,788,990 B2
21 22
storing I/ O requests. An I /O communication conflict occurs PCS system direct I/O will imposing virtual I /O on the ACC
if the ADAS software 299 and the infotainment software 296 system , thereby ensuring that I /O requests for the PCS
each address I /O requests to the same I /O device at the same system are processed as quick as possible while the I /O
time or substantially the same time such that bottlenecking requests for the ACC system are delayed for a period of time
would occur but for the latency system 199 providing its 5 until the PCS system takes actions which remove the risk of
functionality. a collision .
Referring now to FIG . 2C , depicted is an example of the A third column storing " criteria relevant to the vehicle
rule data 287 according to some embodiments. feature ” includes digital data describing different measur
As depicted in FIG . 2C , the rule data 287 is organized as able criteria relevant to the vehicle feature listed in the first
a table . Accordingly, this table may be referred to herein as 10 column of the rule table . For example, the third column
a “ rule table . ” However, other data structures are possible . describes one or more of the following: criteria which, when
The rule table depicted in FIG . 2C includes four columns met or exceeded , result in triggering an ADAS system to
storing the following types of classes of digital data : take action by providing its ADAS functionality; and criteria
“ vehicle feature ” ; “ related sensor from sensor set " ; " criteria required for an infotainment system to provide its function
relevant to the vehicle feature ” ; and “ thresholds for trigger- 15 ality.
ing the provision of the vehicle feature ” . The digital data of For example, the ACC system is triggered and provides its
these four columns is described in more detail below. ACC functionality if one or more of the following are true
A first column storing “ vehicle feature ” includes digital based on the sensor data 295 or the situation data 289 : ( 1 ) the
data describing different vehicle features installed in the ego range separating the ego vehicle and another object on the
vehicle. The ego vehicle includes, for example, the vehicle 20 roadway , towards which the ego vehicle is heading on a
123 which includes the computer system 200. The different collision course , is substantially 50 meters or less ( or some
vehicle features stored in this column include , for example, other number or threshold ); and (2 ) the relative speed
a plurality of different ADAS systems and a plurality of between the ego vehicle and this object is 50 miles per hour
different infotainment systems . or more (or some other number or threshold ).
A second column storing “ related sensor from sensor set ” 25 A fourth column storing “ thresholds for triggering the
includes digital data describing different sensors used by the provision of the vehicle feature” includes digital data
different “ vehicle features listed in the first column of the describing different thresholds associated with the criteria of
rule table . These are examples of the different I /O devices the third column which must be met or exceeded before the
that might be called by the ADAS system or infotainment vehicle feature listed in the first column of the rule table
system described in the first column . 30 provides its functionality. For example, the fourth column
For example, the ADAS feature provided by the PCS uses describes one or more thresholds that are relevant to the
one or more of the following vehicle sensors included in the triggering of the ADAS feature /ADAS system or one or
sensor set 182 of the vehicle 123 : one or more cameras ; one more thresholds relevant to providing an infotainment fea
or more millimeter wave radars; and one or more speed ture .
sensors . Accordingly, when the PCS is active , the PCS 35 In some embodiments, the monitor module 204 deter
software which provides the ADAS feature of the PCS mines which vehicle features are implemented by the current
generates and transmits one or more I/ O requests one or driving situation described by the situation data 289 based
more of the following vehicle sensors included in the sensor on analysis of the rule data 287 and the situation data 289 .
set 182 of the vehicle 123 : one or more cameras ; one or more The monitor module 204 generates feature data 283 that
millimeter wave radars ; and one or more speed sensors . In 40 describes the vehicle features that are implemented respon
this way , the second column describes which I/ O devices are sive to the current driving situation described by the situa
called when the vehicle feature described by the first column tion data 289. The monitor module 204 stores this feature
is active . data 283 in the memory 127 of the ECU that is accessible by
In this way, the I/ O control module 206 is able to access both the monitor module 204 and the I/ O control module
the rule data 287 and determine, based on the vehicle 45 206. The I /O control module 206 retrieves the feature data
features which are currently active , what I /O devices are 283 from the memory 127 and determines how to manage
being called and , based on commonality among these I /O the 1/0 requests associated with the vehicle feature
devices, whether an I/ O communication conflict is present or described by the feature data 283 based on ( 1 ) the relative
likely to be present in the near future . urgency of the vehicle features described by the feature data
For example, assume the I /O control module 206 deter- 50 283 and ( 2 ) the presence of one or more I /O communication
mines, based on the feature data 283 , that the PCS and the conflicts . The I /O control module 206 determines which
ACC systems are both active at the same time or substan vehicle features are installed in the ego vehicle and the
tially the same time . The I /O control module 206 analyzes urgency of these vehicle features based on the situation
the rule table and determines, based on the second column described by the situation data 289 .
of the rule table for the PCS and the ACC systems , that the 55 In some embodiments , the urgency of a vehicle feature
vehicle features provided by the PCS and ACC systems have relates to whether the vehicle feature is associated with
an overlapping dependency of vehicle sensors because each something more important, such as passenger safety, or
of these ADAS systems each call the same sensors . In this something less important, such as passenger entertainment
situation , the I /O control module 206 may prevent or reduce or driver convenience . The relative importance of different
latency for the PCS and the ACC systems by imposing 60 vehicle features varies based on the situation described by
virtual I /O for the I /O requests generated by the PCS and the the situation data 289 .
ACC systems. However, note that the PCS system provides Assume a first situation described by the situation data
a vehicle feature that is more relevant to safety than the ACC 289 in which an ego vehicle is approaching a second vehicle
system since , for example, the PCS system prevents colli at range and relative speed that indicate that the ego vehicle
sions whereas the ACC system provides a convenience , and 65 is on a collision course with the second vehicle. Further
so , if the situation data 289 indicates that a collision may be assume that the ego vehicle has the following vehicle
imminent, then the I /O control module 206 will grant the features available to it : a PCS system ; an ACC system ; a
 US 10,788,990 B2
23 24
LKA system ; and an infotainment system . In this first software that is responsible for providing that particular
situation , the PCS system is most urgent, followed by the vehicle feature ( e.g. , a particular ADAS system , a particular
ACC system , followed by the LKA system and then finally infotainment system , etc. ) permission to exercise virtual I /O
the infotainment system . and (ii ) assigns the device or software responsible for
Assume a second situation described by the situation data 5 providing this particular vehicle feature an urgency value
289 in which an ego vehicle is departing its lane of travel by relative to the other vehicle features that are currently
greater than 20 % ( e.g. , more than 20% of the ego vehicle is executing or will be executing in the near future based on the
over the line that separates the ego vehicle's lane of travel driving situation described by the situation data .
from a neighboring lane ). Further assume that the ego For example, with reference to FIG . 2C , assume that the
vehicle has the following vehicle features available to it : a 10 particular vehicle feature is the PCS system (requiring the
PCS system ; an ACC system ; a LKA system ; and an camera , millimeter wave radar and the speed sensor) and
infotainment system . In this first situation , the LKA system that the LKA system is operating at the same time ( requiring
is most urgent, followed by the ACC system , followed by the the camera , millimeter wave radar, the speed sensor and the
PCS system and then finally the infotainment system . steering angle sensor ). Since both of these vehicle features
Accordingly, the urgency value assigned to different 15 share a commonality of sensors , the I/O control module
vehicle features is variable based on the situation described grants each permission to exercise virtual I /O .
by the situation data 289 and the relationship of the vehicle Referring now to FIGS . 3B and 3C , depicted is a flow
features to the situation and also whether the vehicle feature chart of an example method 399 for managing one or more
is related to safety or entertainment or convenience . I/ O requests for a set of ADAS systems and one or more
In some embodiments, the presence of one or more I /O 20 infotainment systems of an ego vehicle according to some
communication conflicts is identified by the I /O control embodiments .
module 206 based on the other vehicle features that are In some embodiments, one or more of the steps described
currently active at the same time as the particular vehicle herein for the method 399 may be executed by the latency
feature described by the feature data 283 and a commonality system .
of the sensors, actuators and / or other devices that are con- 25 At step 307 , the sensor set generates sensor data describ
temporaneously used by these other vehicle features and the ing the driving situation for the ego vehicle .
particular vehicle feature described by the feature data 283 . At step 309 , the ADAS system set analyzes the sensor
Referring now to FIG . 3A , depicted is a flowchart of an data and determines to provide one or more vehicle features
example method 300 for managing one or more I /O requests based on the driving situation .
for an ego vehicle according to some embodiments. 30 At step 311 , the ADAS sensor set initiates a set of I /O
In some embodiments, one or more of the steps described requests in order to provide the one or more vehicles features
herein for the method 300 may be executed by the latency for the driving situation . Contemporaneously, I /O requests
system . may be executed to provide vehicles features that are related
At step 301 , the I /O control module analyses each vehicle to infotainment and not ADAS functionality ; these I /O
feature described by the feature data as well as the rule data 35 requests are included in the set of I/O requests considered in
to determine whether an I/ O communication conflict is subsequent steps of this method 399 .
present for a particular vehicle feature which is described by The monitor module analyzes the I /O requests and gen
the feature data relative to other vehicle features that are erates feature data describing the vehicle features which
described by the feature data . generated the different I/ O requests .
At step 302 , a determination is made regarding whether an 40 At step 313 , analyses the feature data and the rule data to
I /O communication conflict is present. If an I /O communi determine whether an I /O communication conflict is present.
cation conflict is not present, then the method 300 proceeds At step 314 , a determination is made regarding whether an
to step 303. If an I/ O communication conflict is present, then I/ O communication conflict is present. If no I/O communi
the method 300 proceeds to step 305 . cation conflict is present, then the method 399 proceeds to
At step 303 , if an I /O communication conflict is not 45 step 315. If an I/ O communication conflict is present at step
present for the particular vehicle feature , then the I/ O control 314 , then the method 399 proceeds to step 317 depicted on
module gives the device or software that is responsible for FIG . 3C .
providing that particular vehicle feature ( e.g. , a particular At step 315 , the I /O control module gives each vehicle
ADAS system , a particular infotainment system , etc. ) per system (whether they be an ADAS system or an infotain
mission to exercise direct I /O . 50 ment system) permission to exercise direct I/O ( see , e.g. ,
For example, with reference to FIG . 2C , assume that the FIG . 4B ) . For example , each I/ O request is passed to their
particular vehicle feature is the PCS system ( requiring the addressed I /O device by the latency system without being
camera, the millimeter wave radar and the speed sensor) and stored in the hypervisor I/O queue.
the movie streaming feature is also operating ( requiring the Referring now to FIG . 3C . At step 317 , each vehicle
communication unit ) . The I/ O control module will deter- 55 system (whether they be an ADAS system or an infotain
mine that these two vehicle features do not have an I /O ment system ) associated with the set of I /O requests is given
communication conflict because the example rule table permission to exercise virtual I /O ( see , e.g. , FIG . 4A) .
shown in FIG . 2C indicates that they do not share an overlap At step 319 , the driving situation described by the situa
in the second column. Because of this, the I/ O control tion data is analyzed to determine which active vehicle
module will grant direct I/ O permission to the PCS system . 60 functionality described by the feature data is most important
As a result , the PCS system can directly transmit I /O (or relevant) to the current driving situation . In some
requests to the camera sensor and the brake controller embodiments , any vehicle functionality provided by an
without these I /O requests being queued in the hypervisor infotainment system is always less relevant to the driving
I/O queue. situation than vehicle functionality provided by an ADAS
Referring back to FIG . 3A . At step 305 , if an I /O 65 system since infotainment is less important than safety.
communication conflict is present for the particular vehicle At step 321 , each of the I /O requests in the set of I /O
feature, then the I/ O control module ( i ) gives the device or requests and the rule data is analyzed to determine which
 US 10,788,990 B2
25 26
vehicle functionality they are associated with providing In the above description, for purposes of explanation,
( e.g. , the PCS system is associated with providing the numerous specific details are set forth in order to provide a
vehicle functionality of avoiding a collision, the LKA sys thorough understanding of the specification . It will be appar
tem is associated with providing the vehicle functionality of ent, however, to one skilled in the art that the disclosure can
not swerving into a neighboring lane , etc. ) . 5 be practiced without these specific details. In some
At step 323 , each of the I /O requests in the set of I /O instances, structures and devices are shown in block diagram
requests is assigned an urgency value based on the relative form in order to avoid obscuring the description . For
importance of each vehicle feature they are associated with example , the present embodiments can be described above
providing, as well as their importance to vehicle features that primarily with reference to user interfaces and particular
will be executing in the near future based on the driving 10 hardware . However, the present embodiments can apply to
situation described by the situation data . The set of I /O any type of computer system that can receive data and
requests is stored in the hypervisor I/O queue . For some commands, and any peripheral devices providing services .
driving situations, a vehicle functionality is so important that Reference in the specification to " some embodiments ” or
its associated I/ O requests are never stored in the I /O “ some instances ” means that a particular feature, structure ,
hypervisor queue and its associated ADAS system is instead 15 or characteristic described in connection with the embodi
granted permission to exercise direct I /O while the other ments or instances can be included in at least one embodi
ADAS systems exercise virtual I/O . ment of the description. The appearances of the phrase “ in
At step 325 , each I/O request is released from the hyper some embodiments ” in various places in the specification
visor I /O queue based on their assigned urgency values . are not necessarily all referring to the same embodiments .
Referring now to FIG . 4A , depicted is a block diagram of 20 Some portions of the detailed descriptions that follow are
an example virtual I/ O strategy 400 implemented by the presented in terms of algorithms and symbolic representa
latency system according to some embodiments . tions of operations on data bits within a computer memory .
The ECU 198 executes three different virtual ECUs : the These algorithmic descriptions and representations are the
first ECU 269 ; the second virtual ECU 267 , and the third means used by those skilled in the data processing arts to
virtual ECU 265 . 25 most effectively convey the substance of their work to others
The first virtual ECU 269 provides the ADAS function skilled in the art. An algorithm is here , and generally,
ality of a PCS system . The first virtual ECU 269 includes conceived to be a self -consistent sequence of steps leading
PCS software 405 and a PCS OS 410. The PCS software 405 to a desired result . The steps are those requiring physical
includes code and routines which , when executed by the manipulations of physical quantities . Usually , though not
processor 125 , provide the ADAS functionality of the PCS 30 necessarily, these quantities take the form of electrical or
system . Providing the ADAS functionality of a PCS system magnetic signals capable of being stored , transferred , com
includes transmitting one or more I/O requests 455 to one or bined , compared , and otherwise manipulated. It has proven
more of the first I/ O device 435 and the second I/ O device convenient at times , principally for reasons of common
440. The PCS OS 410 is an operating system for compatible usage , to refer to these signals as bits , values , elements ,
with the PCS software 405 . 35 symbols, characters , terms, numbers, or the like .
The second virtual ECU 267 provides the ADAS func It should be borne in mind, however, that all of these and
tionality of a LKA system . The second virtual ECU 267 similar terms are to be associated with the appropriate
includes LKA software 415 and a LKA OS 420. The LKA physical quantities and are merely convenient labels applied
software 415 includes code and routines which , when to these quantities. Unless specifically stated otherwise as
executed by the processor 125 , provide the ADAS function- 40 apparent from the following discussion, it is appreciated that
ality of the LKA system . Providing the ADAS functionality throughout the description , discussions utilizing terms
of a LKA system includes transmitting one or more I /O including " processing " or " computing " or " calculating " or
requests 457 to one or more of the first I/ O device 435 and “ determining” or “ displaying” or the like , refer to the action
the second I /O device 440. The LKA OS 420 is an operating and processes of a computer system , or similar electronic
system for compatible with the LKA software 415 . 45 computing device , that manipulates and transforms data
The third virtual ECU 265 provides the ADAS function represented as physical ( electronic ) quantities within the
ality of an ACC system . The third virtual ECU 265 includes computer system's registers and memories into other data
ACC software 425 and a ACC OS 430. The ACC software similarly represented as physical quantities within the com
425 includes code and routines which , when executed by the puter system memories or registers or other such informa
processor 125 , provide the ADAS functionality of the ACC 50 tion storage , transmission , or display devices.
system . Providing the ADAS functionality of a ACC system The present embodiments of the specification can also
includes transmitting one or more I/O requests 459 to one or relate to an apparatus for performing the operations herein .
more of the first I/ O device 435 and the second I/ O device This apparatus may be specially constructed for the required
440. The ACC OS 430 is an operating system for compatible purposes, or it may include a general-purpose computer
with the ACC software 425 . 55 selectively activated or reconfigured by a computer program
Note that in FIG . 4A , the I /O requests 455 , 457 , 459 are stored in the computer. Such a computer program may be
each routed to the hypervisor I /O queue 239 . stored in a computer - readable storage medium , including ,
Referring now to FIG . 4B , depicted is a block diagram of but is not limited to , any type of disk including floppy disks ,
an example direct I/ O strategy 401 implemented by the optical disks , CD - ROMs , and magnetic disks , read - only
latency system according to some embodiments. 60 memories (ROMs ) , random access memories (RAMs),
FIG . 4B includes elements that are similar to those EPROMs , EEPROMs , magnetic or optical cards, flash
described above for FIG . 4A , and so , those descriptions will memories including USB keys with non - volatile memory, or
not be repeated here. any type of media suitable for storing electronic instructions,
Note that in FIG . 4A , the I/O requests 457 , 459 are each each coupled to a computer system bus .
routed to the hypervisor I/ O queue 239 while some I /O 65 The specification can take the form of some entirely
requests 455 are routed directly to an I/ O device without hardware embodiments , some entirely software embodi
being routed through the hypervisor I /O queue 239 . ments or some embodiments containing both hardware and
 US 10,788,990 B2
27 28
software elements. In some preferred embodiments, the be implemented as a standalone program , as part of a larger
specification is implemented in software, which includes , program , as a plurality of separate programs, as a statically
but is not limited to , firmware, resident software, microcode, or dynamically linked library, as a kernel- loadable module,
etc. as a device driver, or in every and any other way known now
Furthermore, the description can take the form of a 5 or in the future to those of ordinary skill in the art of
computer program product accessible from a computer computer programming. Additionally, the disclosure is in no
usable or computer- readable medium providing program way limited to embodiment in any specific programming
code for use by or in connection with a computer or any language, or for any specific operating system or environ
instruction execution system . For the purposes of this ment . Accordingly, the disclosure is intended to be illustra
description, a computer- usable or computer -readable 10 tive , but not limiting , of the scope of the specification, which
medium can be any apparatus that can contain , store , com is set forth in the following claims .
municate, propagate, or transport the program for use by or
in connection with the instruction execution system , appa What is claimed is :
ratus, or device . 1. A method executed by a processor of a vehicle for
A data processing system suitable for storing or executing 15 improving a performance of a set of Advanced Driver
program code will include at least one processor coupled Assistance Systems (“ ADAS systems” ) included in the
directly or indirectly to memory elements through a system vehicle by decreasing a latency for processing a set of
bus . The memory elements can include local memory input /output (“ I / O " ) requests generated by one or more
employed during actual execution of the program code, bulk active ADAS systems from the set of ADAS systems, the
storage, and cache memories which provide temporary stor- 20 method comprising:
age of at least some program code in order to reduce the determining situation data describing a driving situation
number of times code must be retrieved from bulk storage for the vehicle;
during execution . determining that an 1/0 communication conflict exists
Input/ output or I/O devices ( including, but not limited , to between a first I /O request and a second I/O request for
keyboards, displays , pointing devices, etc. ) can be coupled 25 the one or more active ADAS systems that handle the
to the system either directly or through intervening I /O driving situation;
controllers. responsive to the I/ O communication conflict existing ,
Network adapters may also be coupled to the system to determining to apply a virtual 1/0 strategy to the first
enable the data processing system to become coupled to I/ O request and the second I/O request;
other data processing systems or remote printers or storage 30 routing the first I /O request and the second I/ O request
devices through intervening private or public networks. request through a hypervisor I / O queue;
Modems, cable modem , and Ethernet cards are just a few of assigning urgency values for the first I/O request and the
the currently available types of network adapters . second I/ O request, respectively, based on whether
Finally, the algorithms and displays presented herein are vehicle features for the first I/O request and the second
not inherently related to any particular computer or other 35 I / O request are associated with safety; and
apparatus. Various general- purpose systems may be used processing the first I /O request and the second I/O request
with programs in accordance with the teachings herein , or it out of a hypervisor 1/0 queue based on the urgency
may prove convenient to construct more specialized appa values .
ratus to perform the required method steps . The required 2. The method of claim 1 , further comprising receiving
structure for a variety of these systems will appear from the 40 sensor data from an external sensor that describes a roadway
description below . In addition, the specification is not environment outside the vehicle, wherein the sensor data is
described with reference to any particular programming used to determine the situation data .
language . It will be appreciated that a variety of program 3. The method of claim 1 , wherein the set of ADAS
ming languages may be used to implement the teachings of systems are provided by a single hardware Electronic Con
the specification as described herein . 45 trol Unit (“ ECU ” if singular, “ ECUs” if plural ) that executes
The foregoing description of the embodiments of the an Operating System ( “OS ” if singular, “ OSes ” if plural)
specification has been presented for the purposes of illus hypervisor which , when executed by the processor, gener
tration and description . It is not intended to be exhaustive or ates a plurality of virtual ECUs which each provide a
to limit the specification to the precise form disclosed . Many different ADAS system from the set of ADAS systems .
modifications and variations are possible in light of the 50 4. The method of claim 3 , wherein each virtual ECU from
above teaching. It is intended that the scope of the disclosure the plurality of ECUs includes a different OS such that the
be limited not by this detailed description , but rather by the plurality of ECUs include heterogeneous OSes .
claims of this application. As will be understood by those 5. The method of claim 3 , the method further comprising:
familiar with the art, the specification may be embodied in assigning an urgency value to each of the one or more
other specific forms without departing from the spirit or 55 active ADAS systems based on a relative urgency of the
essential characteristics thereof. Likewise , the particular one or more active ADAS systems based on the driving
naming and division of the modules, routines, features, situation, wherein the first I/ O request and the second
attributes, methodologies, and other aspects are not manda I/ O request are each assigned the urgency value which
tory or significant, and the mechanisms that implement the was assigned to the one or more active ADAS systems
specification or its features may have different names, divi- 60 which generated each individual I/ O request; and
sions , or formats. Furthermore , as will be apparent to one of storing the first I/O request and the second I /O request in
ordinary skill in the relevant art, the modules, routines, the hypervisor I/O queue ;
features, attributes, methodologies, and other aspects of the wherein the vehicle features for the first I /O request and
disclosure can be implemented as software, hardware , firm the second I/ O request being associated with the safety
ware, or any combination of the three . Also , wherever a 65 is the safety of driving situation .
component, an example of which is a module , of the 6. The method of claim 1 , wherein the vehicle is an
specification is implemented as software, the component can autonomous vehicle .
 US 10,788,990 B2
29 30
7. The method of claim 1 , wherein the vehicle is a Highly a vehicle, the computer program product comprising a
Autonomous Vehicle (“HAV ” ). non - transitory memory storing computer-executable code
8. A system for improving a performance of a set of that, when executed by a processor, causes the processor to :
Advanced Driver Assistance Systems (“ ADAS systems ” ) determine situation data describing a driving situation for
included in a vehicle by decreasing a latency for processing 5 the vehicle ;
a set of input / output (“ I / O ” ) requests generated by one or determine that an I /O communication conflict exists
more active ADAS systems from the set of ADAS systems, between a first I /O request and a second I/O request for
the system comprising: the one or more active ADAS systems that handle the
a processor; and driving situation ;
a non -transitory memory storing computer code which is 10 responsive to the I/ O communication conflict existing ,
determine to apply a virtual I/ O strategy to the first I /O
operable , when executed by the processor, to cause the request and the second I /O request;
processor to perform steps comprising:
determining situation data describing a driving situa route the first I/ O request and the second I /O request
tion for the vehicle ; through a hypervisor I / O queue ;
determining that an I/ O communication conflict exists 15 assign urgency values for the first I/O request and the
between a first I/ O request and a second I/ O request second I/ O request, respectively, based on whether
for the one or more active ADAS systems that handle vehicle features for the first I/O request and the second
the driving situation , and I / O request are associated with safety; and
responsive to the I/ O communication conflict existing, process the first I/ O request and the second I/ O request out
determining to apply a virtual I/O strategy to the first 20 15. The computer Iprogram
of a hypervisor /O queue based on the urgency values .
product of claim 14 , wherein
I /O request and the second I /O request of;
routing the first I/O request and the second I /O request the computer -executable code , when executed by a proces
through a hypervisor I/ O queue; sor, further causes the processor to receive sensor data that
assigning urgency values for the first I/O request and describes a roadway environment outside the vehicle , global
the second I/ O request, respectively, based on 25 positioning data , and map data and wherein the sensor data
whether vehicle features for the first I/O request and is used to determine the situation data .
the second I /O request are associated with safety ; 16. The computer program product of claim 14 , wherein
and the set of ADAS systems are provided by a single hardware
processing the first I /O request and the second I /O Electronic Control Unit ( “ ECU ” if singular, “ ECUs” if
request out of a hypervisor I / O queue basedon the 30 plural)that executes an Operating System ("OS” if singular,
“ OSes ” if plural ) hypervisor which, when executed by the
urgency values. processor, generates a plurality of virtual ECUs which each
9. The system of claim 8 , wherein the processor is further provide a different ADAS system from the set of ADAS
operable to perform a step comprising receiving sensor data systems .
that describes a roadway environment outside the vehicle, 35 17. The computer program product of claim 16 , wherein
global positioning data , and map data and wherein the sensor each virtual ECU from the plurality of ECUs includes a
data is used to determine the situation data .
10. The system of claim 8 , wherein the set of ADAS different OS such that the plurality of ECUs include hetero
systems are provided by a single hardware Electronic Con geneous OSes.
trol Unit (“ ECU ” if singular, “ ECUs” if plural) that executes 18. The computer program product of claim 14 , wherein
an Operating System (“ OS ” if singular , “ O?es ” if plural
) 40 the vehicle is a Highlyprogram
Autonomous
productVehicle
of claim(“HAV ”).
hypervisor which , when executed by the processor, gener the19.computer The computer 14 , wherein
-executable code , when executed by a proces
ates a plurality of virtual ECUs which each provide a sor, further causes the processor to :
different ADAS system from the set of ADAS systems . determine that a subsequent 10 communication conflict
11. The system of claim 10 , wherein each virtual ECU does not exist ; and
from the plurality of ECUs includes a different OS such that 45 transmit a set of I /O requests directly to one or more I /O
the plurality of ECUs include heterogeneous OSes . devices which are addressed by the set of I /O requests .
12. The system of claim 8 , wherein the vehicle is an 20. The computer program product of claim 14 , wherein
autonomous vehicle .
13. The system of claim 8 , wherein the vehicle is a Highly 50 the virtual
the
second
I /O strategy includes queuing first I /O request and
I /O request in the hypervisor I /O queue of an OS
Autonomous Vehicle (“ HAV ” ). hypervisor responsible for providing the set of ADAS sys
14. A computer program product for decreasing a latency tems via a set of virtual ECUs generated by the OS hyper
for processing a set of input /output (“ I/ O " ) requests gener visor upon being executed by the processor.
ated by one or more active Advanced Driver Assistance
Systems ( “ ADAS systems ”) from a set of ADAS systems of