Artificial Passenger

Chapter 1

INTRODUCTION

The AP is an artificial intelligence–based companion that will be resident in software and

chips embedded in the automobile dashboard. The heart of the system is a conversation
planner that holds a profile of you, including details of your interests and profession.A
microphone picks up your answer and breaks it down into separate words with speech
recognition software. A camera built into the dashboard also tracks your lip movements to
improve the accuracy of the speech recognition. A voice analyzer then looks for signs of
tiredness by checking to see if the answer matches your profile. Slow responses and a lack of
intonation are signs of fatigue.

This research suggests that we can make predictions about various aspects of driver
performance based on what we glean from the movements of a driver’s eyes and that a
system can eventually be developed to capture this data and use it to alert people when their
driving has become significantly impaired by fatigue.The natural dialog car system analyzes
a driver’s answer and the contents of the answer together with his voice patterns to determine
if he is alert while driving. The system warns the driver or changes the topic of conversation
if the system determines that the driver is about to fall asleep. The system may also detect
whether a driver is affected by alcohol or drugs. The system, according to its inventors, does
not go through a suite of rote questionsdemanding rote answers. Rather, it knows your tastes
and will even, if you wish, makecertain you never miss Paul Harvey again. This is from the
patent application: “An evenfurther object of the present invention is to provide a natural
dialog car system thatunderstands content of tapes, books, and radio programs and extracts
and reproducesappropriate phrases from those materials while it is talking with a driver. For
example, asystem can find out if someone is singing on a channel of a radio station.The
system will state, “And now you will hear a wonderful song!” or detect that there isnews and
state, “Do you know what happened now—hear the following and play somenews.” The
system also includes a recognition system to detect who is speaking over theradio and alert
the driver if the person speaking is one the driver wishes to hear.” Just because you can
express the rules of grammar in software doesn’t mean a driver is goingto use them. The AP
is ready for that possibility.It provides for a natural dialog car system directed to human
factor engineering for example, people using different strategies to talk (for instance, short vs.
elaborateresponses ). In this manner, the individual is guided to talk in a certain way so as to

Dept. Of CSE,EPCET 2018-2019 Page 1

Artificial Passenger

make the system work—e.g., “Sorry, I didn’t get it. Could you say it briefly?” Here, the
systemdefines a narrow topic of the user reply (answer or question) via an association of
classesof relevant words via decision trees. The system builds a reply sentence asking: What
aremost probable word sequences that could follow the user’s reply.”

1.1 What is an artificial passenger?

“The AP is an artificial intelligence–based companion that will be resident in software and

chips embedded in the automobile dashboard”.
 It is a Natural Language E-companion.
 It is a Sleep preventive device in cars to overcome drowsiness.
 It is a Life safety system.
1.2 What does it do?
 Detects alarm conditions through sensors.
 Broadcasts pre-stored voice messages over the speakers.

Dept. Of CSE,EPCET 2018-2019 Page 2

Artificial Passenger

Chapter 2

BACKGROUND OF THE INVENTION

During the night times the driver could get sleepier which may be porn to accidents. So in
order to overcome the sleepiness the driver could have taken one of the following or all the
below precautions.
1. Use of simulation drinks (e.g.: coffee and tea) .
2. Some tablets to prevent sleeping.
3. Miniature system installed in driver’s hat.

Virtual-reality headsets are another example of the subject wearing a device for the purpose
of head-pose tracking. These devices typically rely on a directional antenna and
radiofrequency sources, or directional magnetic measurement to determine head-
pose.Wearing a device of any sort is clearly a disadvantage, as the user's competence and
acceptance to wearing the device then directly effects the reliability of the system. Devices
are generally intrusive and will affect a user's behavior, preventing natural motion or
operation. Structured light techniques that project patterns of light onto the face in order to
determine head- pose are also known.The light patterns are structured to facilitate the
recovery of 3D information using simple image processing. However, the technique is prone
to error in conditions of lighting variation and is therefore unsuitable for use under natural
lighting conditions.

As these methods are some times inefficient and it may affect the health conditions of the
driver. So in order to overcome the disadvantages of these methods IBM introduces a new
sleep prevention technology device called as “ARTIFICIAL PASSENGER” which was
developed by Dimitry Kanevsky and Wlodek Zadrozny.This software holds the conversation
with driver to determine whether the driver can respond alertly enough.The name artificial
passenger was first suggested in new scientist magazine which was designed to make solo
journey safer and more bearable.Early techniques for determining head-pose used devices
that were fixed to the head of the subject to be tracked. For example, reflective devices were
attached to the subjects head and using a light source to illuminate the reflectors, the reflector
locations were determined.

Dept. Of CSE,EPCET 2018-2019 Page 3

Artificial Passenger

As such reflective devices are more easily tracked than the head itself, the problem of
tracking head-pose was simplified greatly. Devices are generally intrusive and will affect a
user's behavior, preventing natural motion or operation. Structured light techniques that
project patterns of light onto the face in order to determine head- pose are also known.

Dept. Of CSE,EPCET 2018-2019 Page 4

Artificial Passenger

Chapter 3

WHY SUCH SYSTEM?

According to the national survey in UK and USA, it is observed that driver fatigue annually
causes
 100,000 crashes
 15000 deaths
 71,000 injuries
 Which cause annual cost of $12.5 billion.

Fig 3.1: Image of a car accident

A majority of the off-road accidents observed were preceded by eye closures of one-half
second to as long as 2 to 3 seconds. A normal human blink lasts 0.2 to 0.3 second

3.1 Advantages of using this system

1. Artificial Passenger is broadly used to prevent accident.

2. Artificial Passenger device is also used for entertainment such as it telling jokes and
asking question .
3. Artificial Passenger component establishes interface with other drivers very easily.
4. Open and close the window of a car automatically and also answer a call for you.

Dept. Of CSE,EPCET 2018-2019 Page 5

Artificial Passenger

5. If the driver gets a heart attack or he is drunk it will send signals to vehicles nearby
about this so driver there become alert.
6. Provide a natural dialog car system that understands content of tapes, books and radio

Fig. 3.2 Dahboard of car where system is installed

Dept. Of CSE,EPCET 2018-2019 Page 6

Artificial Passenger

Chapter 4

DEVICES USED IN ARTIFICIAL PASSENGER

The main devices that are used in this artificial passenger

are:
1. Eye tracker or Camera.
2. Voice recognizer or speech recognizer.
3. Touch sensors.

4.1 How does eye tracking work?

Collecting eye movement data requires hardware and software specifically designed to
Perform this function. Eye-tracking hardware is either mounted on a user's head or mounted
remotely. Both systems measure the corneal reflection of an infrared light emitting diode
(LED), which illuminates and generates a reflection off the surface of the eye. This action
causes the pupil to appear as a bright disk in contrast to the surrounding iris and creates a
small glint underneath the pupil . It is this glint that head-mounted and remote systems use
for calibration and tracking.

4.1.1 Hardware: Head-mounted and remote systems

The difference between the head-mounted and remote eye systems is how the eye tracker
collects eye movement data. Head-mounted systems , since they are fixed on a user's head
and therefore allow for head movement, use multiple data points to record eye movement. To
differentiate eye movement from head movement, these systems measure the pupil glint from
multiple angles. Since the unit is attached to the head, a person can move about when
operating a car or flying a plane, for example For instance, human factors researchers have
used head-mounted eye-tracking systems to study pilots' eye movements as they used cockpit
controls and instruments to land airplanes (Fitts, Jones, and Milton 1950). These findings led
to cockpit redesigns that improved usability and significantly reduced the likelihood of
incidents caused by human error. More recently, head-mounted eye-tracking systems have
been used by technical communicators to study the visual relationship between personal
digital assistant (PDA) screen layout and eye movement. Remote systems, by contrast,
measure the orientation of the eye relative to a fixed unit such as a camera mounted
underneath a computer monitor . Because remote units do not measure the pupil glint from

Dept. Of CSE,EPCET 2018-2019 Page 7

Artificial Passenger

multiple angles, a person's head must remain almost motionless during task performance.
Although head restriction may seem like a significant hurdle to overcome, Jacob and Karn
(2003) attribute the popularity of remote systems in usability to their relatively low cost and
high durability compared with headmounted systems.

Since remote systems are usually fixed to a computer screen, they are often used for studying
onscreen eye motion. For example, cognitive psychologists have used remote eye-tracking
systems to study the relationship between cognitive scanning styles and search strategies
(Crosby and Peterson 1991). Such eye-tracking studies have been used to develop and test
existing visual search cognitive models. More recently, humancomputer interaction (HCI)
researchers have used remote systems to study computer and Web interface usability.
Through recent advances in remote eye-tracking equipment, a range of head movement can
now be accommodated. For instance, eye-tracking hardware manufacturer Tobii Technology
now offers a remote system that uses several smaller fixed sensors placed in the computer
monitor frame so that the glint underneath the pupil is measured from multiple angles. This
advance will eliminate the need for participants in eyetracking studies to remain perfectly still
during testing, making it possible for longer studies to be conducted using remote systems.

4.1.2 Software: Data collection, analysis, and representation

Data collection and analysis is handled by eye-tracking software. Although some software is
more sophisticated than others, all share common features. Software catalogs eye- tracking
data in one of two ways. In the first, data are stored in video format. ERICA's Eye Gaze[TM]
software, for instance, uses a small red x to represent eye movement that is useful for
observing such movement in relation to external factors such as user verbalizations. In the
other, data are stored as a series of x/y coordinates related to specific grid points on the
computer screen. Data can be organized in various ways--by task or participant, for example
and broken down into fixations and saccades that can be visually represented onscreen.
Fixations, which typically last between 250 and 500 milliseconds, occur when the eye is
focused on a particular point on a screen . Fixations are most commonly measured according
to duration and frequency. If, for instance, a banner ad on a Web page receives lengthy and
numerous fixations, it is reasonable to conclude that the ad is successful in attracting
attention. Saccades, which usually last between 25 and 100 milliseconds, move the eye from

Dept. Of CSE,EPCET 2018-2019 Page 8

Artificial Passenger

one fixation to the next fixation. When saccades and fixations are sequentially organized,
they produce scan paths. If, for example, a company would like to know why people are not
clicking on an important link in what the company feels is a prominent part of the page, a
scan path analysis would show how people visually progress through the page. In this case,
such an analysis might show that the link is poorly placed because it is located on a part of
the screen that does not receive much eye traffic.

Fig 4.1: Former CMU professor Richard Grace is shown on a TV monitor while testing
a DD 850, a dashboard-mounted infrared camera that can detect when a driver is
starting to fall asleep. The device will beep to alarm a sleepy driver.

Dept. Of CSE,EPCET 2018-2019 Page 9

Artificial Passenger

Fig 4.2: Eye- Tracker

Dept. Of CSE,EPCET 2018-2019 Page 10

Artificial Passenger

Chapter 5

ARCHITECTURE

Fig 5.1: General architecture

1. Microphone: For picking up the words and separate them by some internally used
software for Conversation.camera.
2. Temperature indicator: This component is used to measure the
temperature inside the vehicle and it also helps in maintaining the steady
temperature.
3. Camera: This will track the lip movements of the driver and also used for
the improvement for the accuracy of the speech recognition.

Dept. Of CSE,EPCET 2018-2019 Page 11

Artificial Passenger

Fig. 5.2 Camera for detection of Lips Movement Determination of eye blinking and
closing

4. External service provider: Linked to the dialog system by wireless network

system Coupled with Car media, driver profile, conversational planner.Driver
analyzer module It controls interruption of a dialog between the driver and the car
dashboard (for example, interrupting a conversation to deliver an urgent message
about traffic conditions on an expected driver route).
5. Odor sensor: This sensor will periodically sprinkles the sweet air inside
the vehicle.

Fig 5.3 : Image of an odor sensor

Dept. Of CSE,EPCET 2018-2019 Page 12

Artificial Passenger

6. Speaker: This generally used for the entertainment purpose. It is also

used for listening the sound inside the car.

Fig 5.4: Speaker used to hold conversation with driver

Dept. Of CSE,EPCET 2018-2019 Page 13

Artificial Passenger

Chapter 6

WORKING COMPONENTS

Fig 6.1: Working components of AP

There are some of the components which supports for the working of the system:

1. Automatic Speech Recognizer (ASR)

2. Natural Language Processor (NLP)

Dept. Of CSE,EPCET 2018-2019 Page 14

Artificial Passenger

3. Driver analyzer
4. Conversational planner (CP)
5. Alarm
6. External service provider
7. Microphone
8. Camera

6.1 Automatic Speech Recognition (ASR):

There are two ASRs used in the system:

 Speaker independent: It will decode the driver voice and the decoded voice signals
will output to Natural Language Processor (NLP)
 Operates with a voice car media, decodes tapes, audio books, telephone mails.
Decoding outputs of the ASR module is analyzed by Intelligent text processor and it
will output data to conversational planner.

6.2 Natural Language Processor (NLP):

Processes the decoded signal of textual data from ASR module, identifies semantic and
syntactic content of the decoded message , produces variants of responses and outputs this
data to a text input of the driver analyzer.

6.3 Driver analyzer:

Receives the textual data and voice data from NLP and measures the time of response using a
clock. This time responses, concludes about drivers alertness and it will output to the
conversational planner. This analysis is both objective & subjective.

6.4 Conversational planner:

This is generally referred as the heart of the system and it instructs the language generator to
produce the response. If the driver continues to be in a perfect condition, then conversational
planner instructs the language generator to continue the conversation otherwise the language
generator is instructed to change the conversation.

Dept. Of CSE,EPCET 2018-2019 Page 15

Artificial Passenger

6.5 Alarm:
If the conversational planner receives information that the driver is about to fall asleep then it
activates an alarm system.

6.6 External Service Provider:

Linked to the dialog system by wireless network system as it is coupled with
Car media, driver profile, conversational planner & Driver analyzer module.
It controls interruption of a dialog between the driver and the car dashboard (for example,
interrupting a conversation to deliver an urgent message about traffic conditions on an
expected driver route).

6.7 Microphone:
It picks up the words and separate it using speech recognition software.

6.8 Camera:
A camera built into the dashboard used to track the lip movement of the driver to improve the
accuracy of the speech recognition.

Dept. Of CSE,EPCET 2018-2019 Page 16

Artificial Passenger

Chapter 7

ADVANTAGES & DISADVANTAGES

7.1 Advantages:

1. It is broadly used to prevent accident.

2. It detects whether the driver is affected by alcohol or drugs.
3. Its components establish interface with driver very easily.
4. If the driver gets heart attack or he is drunk it will send signals to vehicles nearby
about this so driver become alert.

7.2 Disadvantages:

1. Its implementation is a bit costly.

2. It can only be implemented in high profiled cars.

Dept. Of CSE,EPCET 2018-2019 Page 17

Artificial Passenger

Chapter 8

APPLICATIONS

First introduced in US Sensor/Software system detects and counteracts sleepiness behind the
wheel. Seventies staples John Travolta and the Eagles made successful comebacks,and
another is trying: That voice in the automobile dashboard that used to remind drivers to check
the headlights and buckle up could return to new cars in just a few years—this time with
jokes, a huge vocabulary, and a spray bottle.

The following are the applications of the artificial passenger:

1. Artificial Passenger is broadly used to prevent accident.

2. Prevents the driver, falling asleep during long and solo trip.
3. If the driver gets a heart attack or he is drunk it will send signals to vehicles nearby
about this so driver there become alert.
4. In any problem it alerts the vehicles near by this, so the driver there become alert.
5. Opens and closes the doors and windows of the car automatically.
6. It is also used for the entertainment.
7. It provides a natural dialog car system that understands content of tapes, books and
radio programs.
8. This system can also be used in other situations such as:
9. Security guard
10. Operators at nuclear plants
11. Pilots of airplane.
12. Cabins in airplanes.
13. Water craft such as boats.
14. Trains and subways.

FUTURE ENHANCEMENTS

To provide us with a shortest-time routing based on road conditions changing because of

weather and traffic, information of about the cars on the route, destination requirement (as
flight has been delayed or cancelled).

Dept. Of CSE,EPCET 2018-2019 Page 18

Artificial Passenger

CONCLUSION

Successful implementation of artificial passenger would allow use of various services in car
like reading e-mails, navigation, downloading music files, voice games without
compromising a driver safety.A primary objective of the invention is to provide a system and
method for monitoring driver alertness with a single camera focused on the face of the driver
to monitor for conditions of driver fatigue and lack of sleep. A secondary objective of the
invention is to provide a system and method for monitoring driver alertness which operates in
real time which would be sufficient time to avert an accident. A third objective of the
invention is to provide a system and method for monitoring driver alertness that uses a
computer vision to monitor both the eyes and the rotation of the driver's head through video
sequences. The reduction of conventional speech processes to lowresources processing was
done by reducing a signal processing and decoding load in such a way that it did not
significantly affect decoding accuracy and by the development of quasi-NLU principles. We
observed that an important application like Artificial Passenger can be sufficiently
entertaining for a driver with relatively little dialog complexity requirements – playing simple
voice games with a vocabulary containing a few words. Successful implementation of Safety
Driver Manager would allow use of various services in cars (like reading e-mail, navigation,
downloading music titles etc.) without compromising a driver safety.

Dept. Of CSE,EPCET 2018-2019 Page 19

Artificial Passenger

REFERENCES

[1] http://www.scribd.com/doc/19067668/artificialpassenger

[2] http://www.freepatentsonline.com/4682348.html

[3] http://seminars4you.wordpress.com/artificial-passenger/.

[4] L R Bhal et al. “Performance of the IBM speech recognition system

[5] www.freepatentsonline.com/4682348.html

[6] www.slideshare.com

[7] www.about.com

[8] 1995Lawrence R. Rabiner, A Tutorial on Hidden Markov vol.1, pp 41-44.).

Dept. Of CSE,EPCET 2018-2019 Page 20