Title of Project:

Smartphone based driver aided system to reduce accidents using image processing and
gesture recognition algorithm

Area of research:
Image Processing, Artificial Intelligence

Abstract:

Driver inattention and distraction from driving are major reasons of accident. We are
developing a system which will work on the basis of eye tracking for detecting
sleepiness, mouth tracking for detecting yawning phase, gesture detection (if driver is
talking with passengers on backseats while driving) from data captured by camera. All
these tracking are performed on the basis of eye velocity detection, self-learning, closed
mouth and open mouth velocity detection, algorithms. After detecting driver distraction it
will ring the alarms and will also provide voice messages to alert the driver. This is a
Smartphone based system which works on the basis of image processing, driver gesture
and facial expression recognition. There are several other features like analyzing stress
from face and other movements for detecting whether driver having any difficulties or
not (e.g. driver having mild heart attack or other problems). Driver assisted system also
provides various different functionalities like traffic sign detection from front camera,
detecting traffic to assist driver. All these processing will performed as backend
applications. Frontend will display route navigation to provide ease in driving.

Keywords:

Image processing, facial expression recognition, distraction

 CHAPTER 1

INTRODUCTION

1.1 Overview
Driver fatigue and drowsiness is a main cause of large number of vehicle accidents.
Recent report states that 1200 deaths and 76000 injuries caused annually due to
drowsiness conditions. Sleepiness is main issue in large number of accidents take
placed. Development of a complete system which will help to prevent drowsiness is
a major challenge in accident development system. Current technologies used for
detecting driver’s fatigue condition uses in physiological signals like heartbeat rate,
eye blinking detection and brain activities. But this technique does not have
advantages in real world application. Because using electrodes to get signal is very
annoying for driver every time and this systems are not cost effective.

To overcome all of these problem driver assistance system is proposed. The purpose
of this project is to detect that whether driver is about to fall asleep while driving or
not and if driver is in drowsy condition, then give him alert by using alarm or any
other device. This will make user alert and will help to prevent accidents during
driving.
The main elements of the system are smartphone with inbuilt video camera and
alarm/ warning system.
In this project various algorithm are used for detecting sleepiness by tracking
following things:
1. Face tracking
2. Eye tracking
3. Mouth tracking to detect yawning
4. Face gesture recognition
5. Driver movement tracking
6. Stress recognition
1.2 Background

Following figure 1 shows the increased risk of security for driving under fatigue or
drowsiness conditions and gives count of results occurring from falling asleep while
driving.

Accidents

Crossing outer boundary line Crossing opposite boundary line Crossed center line
Ran off the road Collided with other vehicle Unspecified

Fig 1: Breakdown of results occurring from falling asleep while driving.

Recent report states that accidents related to driver fatigue is caused are between 2
am to 6 am and afternoon between 2 pm to 4 pm. During this time our body clock is
at the lowest point. Males aged between 18 to 30 have high risk in driving during
this time period. Women are less likely to be involved in sleep related accidents. If a
driver continues to drive in fighting sleep then the impairment level is same as over
the drink drive limit.
1.3 Objectives

Driving assistance system provides various features and functionality to reduce accidents
given as follows:

1. To reduce number of accidents that causes due to sleepiness or drowsiness

conditions while driving
2. To assist driver by observing driver behavior using eye, mouth and gesture
tracking.
3. To provide safety while driving.
4. To make the system cost-effective by using smartphone rather than using separate
camera or other devices.
5. To detect traffic jam using aerial surveillance.
6. To provide road navigation to user at the front end of the smart phone to assist
driver in driving.
7. To read traffic signs from front camera of mobile device and give the voice
message to the driver.
8. Give voice alerts to the driver when he is talking with backseat passengers other
passengers.
9. Alert driver in drowsiness conditions by using vibrations, voice messages, beep or
various other warning methods.
10. Help driver to maintain vehicle speed within specific limit and provide alert to the
driver if driving speed increases.
11. Improve driving performance.
 CHAPTER 2

PROBLEM DEFINITION

This project is to develop a driver assistance system to detect driver drowsiness by using
eye tracking, facial expression tracking and driver’s gesture recognition. Eyelid
observation and mouth tracking is main factor in driver fatigue detection. Based on the
previous research conducted, there is none of the project developed which is using all
face tracing in one including facial expression, stress recognition, mouth tracking for
detecting yawning, driver movement detection and recognition and eyelid tracing for
detection of closed and open eyes.. Therefore according to analysis these projects does
not give accurate result.

Driver Assistance system (DAS) have been proposed to integrate all feature in one
system to get accurate result and to improve driving performance. This system alerts
driver under drowsiness. Alarms used for alerting driver contains different level and type
which is based on critical level of driver fatigue condition. Detection techniques and
processing is performed on the smartphone alone which make driver assistance system
cost effective. Front end of the system shows route navigation to driver for increasing
driver performance and all driver fatigue detection is processed at backend in
smartphone.
 CHAPTER 3

LITERATURE SURVEY

There are various approaches which are in development for detecting driver fatigue and
drowsiness levels. Some of them are implemented and some are in implementation stage.
This chapter will focus on various technological approaches which helps driver and
assists to reduce accidents. Advantages and disadvantages for each technology or
algorithm is analyzed in this section.

1. Xianping Fu, Xiao Guan, Eli Peli, Hongbo Liu, and Gang Luo
“Automatic Calibration Method for Driver’s Head Orientation in Natural Driving
Environment”

In this paper, to detect driver’s drowsiness gaze tracking method used to prevent
accidents. This approach focus on the head orientation of the driver in each direction
and analyze eye movements like blinking of eyes, closed eye or open eye to detect the
level of sleepiness of driver. Automotive learning and particle filtering algorithm is
used in this approach. This algorithms helps to track head and face movement
accurately by camera. Single camera is used to capture eye movements.

Advantages:

1. Provides accurate head pose and gaze zone estimation

2. Robust estimation of eye zone without full coordination of driver while capturing
video.

Disadvantages:

1. This approach do not take various important factors into account like driver is
wearing glasses, contact lenses etc.
2. Does not trace facial expressions and body movements of driver to detect
inattention while driving.
2. Ye Sun, Student Member, IEEE, Xiong Yu, Member, IEEE
“An Innovative Non-intrusive Driver Assistance System for Vital Signal
Monitoring”

This paper describes an innovative application for assisting driver while driving. This
paper include non-intrusive measurement approach for detection of driver fatigue and
monitoring driver’s health. Driver assistance system remotely detects the bio-potential
signals without any physical contact with human skin. This does not cause any
problem to the user compared to conventional ways like using electrodes which need
to be in contact with human body. Using delicate sensor and ECG, EEG eye blinking
can be measured. The system is able to detect the electrical signals through cloth or
hair without any contact with human

Advantages:
1. This approach causes less mental or physical load to the drivers because electric
signal are collected remotely without having contact with human skin.
2. Provides perfect result for health monitoring of driver and gives alert according
to health condition which helps in reducing accidents.

Disadvantages:
1. Not cost effective because it requires measuring Electrocardiography (ECG),
Electroencephalography (EEG) for health monitoring and eye blinking detection
of driver.
2. Does not consider another conditions like yawning, talking with other passengers
while driving
3. Attention Technology:

Attention Technology Inc. developed Driver Fatigue monitor (DFM) which is real
time driver drowsiness monitor. It is currently being tested in a field operational test.
Driver Fatigue monitor uses video based approach for detecting eyelid positions and
closing and opening movement of eyes.

Advantages:

1. Estimate eyelid closure and opening movement correctly by tracking eye

pupils of the driver.
2. The high level contrast between the pupils and the face can improve the eye
tracing robustness.

Disadvantages:

1. The main key in this approach is brightness and size of pupils. If there is less
brightness then it may not be able to track eye movement of driver correctly. It
may provide variable outputs in this condition.
2. In real-world applications, sunlight, eyeglasses reflection can create cbright spots
near the eyes while driving. This may weakens the driver’s eye pupil tracking
process.

4. Igor Radun, BA; Heikki Summala, PhD

“Sleep-related Fatal Vehicle Accidents: Characteristics of Decisions Made by
Multidisciplinary Investigation Teams”

This paper describes various vehicle accidents takes place due to driver fatigue,
drowsiness and sleepy conditions. Various logistic regression models were calculated
 for explaining falling asleep in this paper. Major causal factor was falling asleep
while driving according to report of investigation team.

5. LC Technologies, Inc.

LC Technologies, Inc. has designed eye tracking technology which includes an eye
operated computer for control and a device for tracking eye motion and needed eye
tracking data. It user warning system, the Eyegaze to alert driver who are becoming
drowsy while driving.

Advantages:

1. Main goal of this system is to monitor eye movements of driver which includes
measuring eyelid closure time.
2. Provides warning to driver when he/she is under drowsy condition by using
warning system.

Disadvantages:

1. It is hard to track eye movements of driver when driver is tilted or bended at

downward direction.
2. Sunlight containing high IR lights or LED lights may degrade the quality of eye
image.

6. Boon-Giin Lee and Wan-Young Chung, Member, IEEE

“Driver Alertness Monitoring Using Fusion of Facial Features and Bio-Signals”

This paper uses two methods for providing safety to driver which are bio-signal
processing and eye movment tracking. Driver alertness monitoring system provides data
through wireless sensor network and then processes data. For processing received data it
requires integration and synchronization of several sensors. Video sensor is used to
capture the driver movements while driving and bio-signal sensor is used to collect the
driver‘s photoplethysmograph signal. A dynamic Bayesian network framework is used
to evaluate driver fatigue. That is in short final output of the system is evaluated using
Dynamic Bayesian network Alarm get activated if any driver fatigue condition detected
through different sensors.

Advantages:

1. Provides more authenticated driver alertness system.

2. Higher degree of performance is achieved by integrating various sensors into
single system

Disadvantages:

1. Installation of all sensors in vehicle is time consuming process.

 CHAPTER 4

REQUIREMENT ANALYSIS

4.1 Project Scope

In this project we are going to develop and integrate various functionalities into
single application. These functionalities are:

1. Driver fatigue detection by

 Eye Tracking
 Mouth Tracking
 Face expression recognition
 Stress recognition
 Gesture recognition
2. Route navigation system
3. Traffic sign detection
4. Road traffic detection
5. Different types of alarm

4.1.1 Users:
The users of this application are road drivers of vehicles who can take assistance
from this system to improve driving performance and to prevent road accidents
due to fatigue.

4.1.2 Environment
Smartphone with Android operating system is used for driver assistance system.
Android 2.2 and above is needed for project.

4.1.3 Constraints
 Mobile device with android operating system is needed.

4.1.4 Documentation
Detailed user documentation will provided to user to learn how to use all
functionalities like route navigation and other services.

4.1.5 Assumptions
The user is expected to use android mobile with OS 2.2 or above. Mobile
device should have inbuilt camera with minimum 2MP

4.1.6 Deliverables
Output of this system will be single android application which will assist driver
while driving on roads.

4.2 System Requirements

Driver Assistance System includes various functional and nonfunctional
requirements described as follows:

4.2.1 Functional Requirements:

Functional requirements of the system includes transactions, use cases, actions,
business rules, historical data, audits, and authentication, reporting and
certification requirements of the system.
Driver assistance system includes following requirements:
 System should support android
 System should allow to use other functions of device
 System should run as a background service
 System should support user location tracking
 System should be have inbuilt front camera
 System should be have inbuilt back camera
 4.2.2 External Interface Requirements
 User Interfaces:

 The GUI standard of the system should be consistent which will use the similar
background, button, text field and etc. Therefore, user will not feel lost when he is
exploring in the tool.
 Navigation tools must be created carefully so that user can notice them clearly
and makes use of the function.
 The text color must be contrast with the background color so that all the text will
be shown obviously.
 The validation message will also display the meaningful error message if there is
an error occurs. All the functions of the system must attract user’s attention and
should be understandable.

 Route Navigation Form

 Application On Off Form
 Settings Form
 Help form
 Traffic sign detection form
 Traffic detection form

 Hardware Interfaces:
 The System can operate with any hardware which supports Android OS 2.2 or
above.
 All computer devices that access to the system must able to connect to the
Internet.
  Software Interfaces:
 Operating System: Android 2.2
 Android 2.2 Supported mobile handset
4.2.3 Non Functional Requirements:

Nonfunctional requirements of system describes how the system works.

Nonfunctional requirements includes performance, Scalability, Capacity,
Availability, Reliability, Recoverability, Maintainability, Serviceability, Security,
Manageability, Environmental, Usability, Interoperability etc.

 Performance Requirements:
For enhanced performance the resources must be utilized. Mobile device based
application should get the required memory to run the application.

 Availability Requirements:
Network should be available in smartphone to detect traffic in specific area while
driving

 Reliability Requirements:
System should recover from failure as quickly as possible and should work
efficiently under load.

 Maintainability Requirements:
System or application should be easy to update or easy maintain after delivery of
system.

 Usability Requirements:
Driver assistance system should be easy to use. It should not cause disturbance to
driver while driving on road.
  Scalability Requirements:
Extra modules can be easily added to the driver assistance system without
affecting on other modules.
4.3 Other Requirements

4.3.1 Hardware Specification

Machine (Minimum):

1. Machine : Smartphone
2. Disk Storage : 10GB HDD
3. Memory Storage : 256 MB of RAM.
4. Alert device : Alarm device

4.3.2 Software Specification:-

1. Operating System : Android.
2. Front End : Java, android
3. Programming Environment : JDK 7.0
4. Programming Tool : Android Development
Tool(Eclipse)
4.4 Analysis Models
System Implementation Plan:

The approach we are using for system implementation is Water Fall Model.

Water Fall Model for driver assistance system:

Includes following phases:

1) Requirement analysis:
The software development process starts with communication between user and
developer. In this phase we communicated with the user with following principles
of requirement analysis phase.
 We performed literature survey of previous researches and technologies
available for this topic and we evaluate advantages, limitations,
technologies of available systems.
 We analyzed what types of requirement user want in new system.

2) Planning:
It includes planning of budgets and resources required for developing driver
assistance system.
 In this phase we prepare planning and scheduling duration for starting
development of project
 We estimated the cost of the project including all expenditure of software
and efforts required.

3) Modeling:
It includes detail requirement analysis and project design. Flowchart is prepared
for this phase. Flowchart shows complete pictorial flow of program step by step in
understandable manner.
 We prepared various uml diagrams: Class Diagram, Use case Diagram, Sequence
diagram and Use case report for project

4) Construction:
It includes coding and testing steps:
a) Coding:
Design details are implemented using appropriate programming languages.
In coding we chosen the platform i.e. java and android programming language for
developing system.

b) Testing:
Testing is carried out to check whether system fulfils user expectations or not.
Different types of testing are conducted to check whether system perform all
functionalities correctly or not.

5) Deployment:
It includes software delivery, customer support and feedback from user. If user
suggest some corrections, or demands extra functionalities then changes are
required for such updates or adding extra modules. It also includes maintenance of
the system after delivery of system to user.
 Chapter 4

SYSTEM DESIGN

4.1 System Architecture

 Fig. General Architecture of Driver Assistance system

Steps for developing driver assistance systems are given as below:

1. Capture driver’s image continuously by using inbuilt camera of

smartphone.
2. Track Driver’s face and expression
3. Track eyes pupils to detect movement of eyes and to find whether
eyes are closed or not.
4. If eyes are closed for long time, then alert driver by sounding the
alarm device.
5. If eyes are open then continue to capture driver’s image

Different types of alarms are sounded when driver is found in drowsy

condition while driving. Those are given as follows:
UML Diagrams
 Chapter 5

ADVANTAGES AND DISADVANTAGES

5.1 Advantages

 Assists driver and help to reduce accident

 Give route navigation
 Improved driving performance
 Different level of alarms make driver alert.
 Give traffic details through network

5.2 Limitations

 The system could not track the eyes, when the driver’s head rotation is tilted
towards down direction
 Too much brightness may produce difficulty in tacking eyes for driver fatigue
detection
 CHAPTER 6

PLANNING AND SCHEDULING

Planning and Scheduling the Proposed Project Work:

Id Task Name Start %Complete

Project Plan 100%

1
Determine project scope 12-Aug-11 100%
2
Project planning 16-Aug-11 100%
3
Project objectives, 17-Aug-11 100%
4
Analysis 100%
5
Literature Survey 23-Aug-11 100%
6
Review specification 24-Aug-11 100%
7
Define functional requirement 25-Aug-11 100%
8
Define nonfunctional requirement 25-Aug-11 100%
9
Analysis complete 100%
10
Initial Design 100%
11
Draw use cases 2-Sep-11 100%
12
Create class diagram 3-Sep-11 100%
13
Activity diagram 5-Sep-11 100%
14
 Collaboration diagram 8-Sep-11 100%
15
Sequence diagram 12-Sep-11 100%
16

Id Task Name Start %Complete

17
Data flow diagram

System Design 12-Aug-11 100%

18
System Architecture 16-Aug-11 100%
19
Technologies used 17-Aug-11 100%
20
Future work discussion 100%
21
 CHAPTER 7

CONCLUSION

A non-invasive system to localize the eyes and monitor fatigue was developed.
Information about the head and eyes position is obtained through various self-developed
image processing algorithms. During the monitoring, the system is able to decide if the
eyes are opened or closed. When the eyes have been closed for too long, a warning signal
is issued. In addition, during monitoring, the system is able to automatically detect any
eye localizing error that might have occurred. In case of this type of error, the system is
able to recover and properly localize the eyes.
CHAPTER 8

REFFERNCES