A Project Report on

DRIVER DROWSINESS DETECTION

Submitted in partial fulfilment of requirement for the award of
MCA
Of
KANNUR UNIVERSITY
By
NAKUL RAMESH
C2GMCA2207

INFORMATION TECHNOLOGY EDUCATION CENTRE

Thalassery Campus
Palayad
Kannur-670661
2022-2024
 KANNURUNIVERSITY
Information Technology Education Centre
Thalassery Campus, Palayad-670661

CERTIFICATE

This is to certify that this Mini project entitled “DRIVER DROWSINESS DETECTION” is a
bonafide work successfully done by NAKUL RAMESH with Reg.no C2GMCA2207, third
semester in partial fulfillment for the award of Master of Computer Application Degree from Kannur
University during the period 2023-2024.

Staff in Charge: Assistant Director

Place: External Examiner

Date: 1.
2.
 DECLARATION

I, NAKUL RAMESH ,3RD semester MCA student of Information Technology Education Center
Palayad, do hereby declare that the project report “DROWSY DRIVER DETECTION”,submitted
for partial fulfillment of Master of computer application from ITEC Palayad during the period 2023-2024
is a bona fidework done by me under supervision of Mr. Mohammed Shafi K.
I also declare that I have adhered to ethics of academic honesty and integrity and have not
misrepresented or fabricated any data or idea or fact or source in my submission.

Name: NAKUL RAMESH Signature:

 ACKNOWLEDGEMENT

The Successful presentation of the mini project on the topic ”DROWSY DRIVER DETECTION”
would have been incomplete without the mention of people who made it possible and whose constant
guidance crowned my effort into success.I convey my sincere thanks to project guide Mr.
Mohammed Shafi K (Assistant Director, ITEC Palayad) for providing encouragement, constant
support and guidance which was of a great help to complete this project successfully.
Last but not the least, I wish to thank my parents and friends for their constant encouragement and
support without which this mini project work would have not been complete.

NAKUL RAMESH
 Abstract

Driver drowsiness is one of the major causes of traffic accidents. It is a serious highway
safety problem. If drivers could be warned before they became too drowsy to drive
safely, some of these crashes could be prevented. In order to reliably detect the
drowsiness, it depends on the presentation of timely warnings of drowsiness. To date,
the effectiveness of drowsiness detection methods has been limited by their failure to
consider individual differences. Based on the type of data used, drowsiness detection
can be conveniently separated into th e two categories of intrusive and non-intrusive
methods. During the survey, non-intrusive methods detect drowsiness by measuring
driving behavior and sometimes eye features, through which camera based detection
system is the best method and so are useful for real world driving situations.

iii
 Contents

Abstract iii

List of Figures vii

List of Tables viii

1 Introduction 1
1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 General Background . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2.1 Drowsiness . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 Image Processing . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Scope of the system . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Literature Review 6
2.1 Development of an intelligent drowsiness detection system for drivers
using image processing
technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Drowsy driver warning system using image processing . . . . . . . . 8
2.2.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

iv
CONTENTS v

2.2.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 Using image processing in the proposed drowsiness detection system . 11
2.3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Drowsy driver detection through facial movement analysis . . . . . . 13
2.4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.5 Real-Time Detection for Drowsy Driving via Acoustic Sensing on

Smartphones ................................................................................................15
2.5.1 Abstract ...........................................................................................15
2.5.2 Methodology ...................................................................................16
2.5.3 Conclusion ......................................................................................16
2.6 CONSOLIDATED TABLE ............................................................................18

3 Requirement Specification 19
3.1 Functional requirements .............................................................................19
3.2 Software Requirements ................................................................................19
3.3 Hardware interfaces .....................................................................................19
3.4 System Hardware requirements ..................................................................20

4 Proposed system and Design 21

4.1 Proposed system ..........................................................................................21
4.1.1 68 Point Facial Landmark ...............................................................22
4.1.2 Eye Aspect Ratio .............................................................................23
4.2 Feasibility Study ..........................................................................................24
4.2.1 Technical Feasibility .......................................................................25
4.2.2 Economic Feasibility.......................................................................25
CONTENTS vi
4.2.3 Operational Feasibility ....................................................................25

4.3 DESIGN .......................................................................................................26

4.3.1 ARCHITECTURE DIAGRAM ......................................................26
4.3.2 USE CASE DIAGRAM ..................................................................27
4.3.3 Data Flow Diagram (LEVEL 0).....................................................28
4.3.4 Data Flow Diagram (LEVEL 1).....................................................29
4.3.5 Data Flow Diagram (LEVEL 2).....................................................30

5 IMPLEMENTATION 31
5.1 Implementation Details ................................................................................31
5.1.1 Capturing Video ..............................................................................31
5.1.2 Initializing face detector..................................................................31
5.1.3 Finding EAR ...................................................................................31
5.1.4 Checking drowsiness ......................................................................32
5.2 Modules Used ..............................................................................................32
5.2.1 Python .............................................................................................32
5.2.2 OpenCV ..........................................................................................32
5.2.3 Dlib .................................................................................................33
5.2.4 Numpy.............................................................................................33
5.2.5 Imutils .............................................................................................34
5.2.6 Playsound ........................................................................................34

6 RESULT AND DISCUSSION 36

6.1 Active ...........................................................................................................36
6.2 Sleeping .......................................................................................................37
6.3 Drowsiness ...................................................................................................38

7 Conclusion 39

8 References 40
 List of Figures

1.1 A Drowsy Man . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4.1 68 Point Facial Landmark ............................................................................22

4.2 EAR Formula ...............................................................................................23
4.3 Open Eye .....................................................................................................24
4.4 Closed Eye ...................................................................................................24
4.5 architecture diagram ....................................................................................26
4.6 Use case diagram .........................................................................................27
4.7 Data Flow Diagram (Level 0) ......................................................................28
4.8 Data Flow Diagram Level 1 .........................................................................29
4.9 Data Flow Diagram Level 2 .........................................................................30

vii
 List of Tables

2.1 set of action units used for predicting drowsiness .......................................14

2.2 Consolidated Table ......................................................................................18

viii
 Chapter 1

Introduction

1.1 Overview

In this report we are aimed to present the first review of the product ”DRIVER
DROWSINESS DETECTION SYSTEM”. The ”DRIVER DROWSINESS DETEC-
TION SYSTEM” is one of the systems that is aimed to reduce road accidents due
to drowsiness of drivers. Most of the road accidents are caused by drowsiness of
drivers.In order to reliably detect the drowsiness, it depends on the presentation
of timely warnings of drowsiness.To date, the effectiveness of drowsiness detection
methods has been limited by their failure to consider individual differences. Based on
the type of data used, drowsiness detection can be conveniently separated into th e two
categories of intrusive and non-intrusive methods.

1.2 General Background

Detecting the levels of drivers’ drowsiness has a key role in reducing the number of
fatal injuries in traffic accident. Recent statistics and reports show that 20 to 50 million
people are killed or which the drivers drowsiness is one of the principal contributor.
These accidents cost over 12.5 billion dollar and cause 1550 deaths and 71000 injuries .

1
CHAPTER 1. INTRODUCTION 2

National Sleep Foundation of USA declared that 54 percent of adult drivers had driven
during sleepiness and 28 percent of these drivers had fallen asleep completely. Road
Safety Council of Germany (DVR) (Deutsche Verkeh Rswacht) states that 25 percent
of fatal car crashes in highway traffic is due to momentary sleepiness. Investigations
conducted by the Iranian legal medicine have shown that traffic accidents represent a
large number of the country’s fatalities. According to the reports by this organization,
such accidents cost over 4 billion dollar annually that is 3.5 times gross domestic
product of Iran. Based on the reports represented by police, approximately, 23 percent
of car accidents in Iran was due to drowsy driving and driver fatigue.
Having considered such statistics, devising the systems accurately to determine the
levels of drivers’ drowsiness is of prime importance in order to reduce the number of
car accidents. Generally, there three kinds of systems exist to determine the levels of
drivers’ fatigue: techniques based on physiological signals, techniques based on
driver’s performance and techniques based on image processing.
In methods developed based on physiological signals, some electrodes are attached
to the body to detect the signals from brain and heart. This method would be irritating
and is considered as a nuisance to the drivers. In methods developed based on the
driver’s performance, much time is required to analyze driver’s performance, which
consequently leads to low accuracy. In some cases, while the driver falls asleep fora
moment, the status of the vehicle does not change, so, the system is disturbed in
detecting micro-sleeps.
However, methods developed based on image processing are fast and precise to
detect drivers’ drowsiness. Fatigue and drowsiness lead to some apparent signs on
driver’s face. Interpreting such signals are the principals of the methods developed
based on the image processing. In many cases, the first step in image processing is to
recognize the facial zone from the received images.
Then, the parameters related to drowsiness such as eyes are recognized and their
changes are investigated to detect the levels of sleepiness. By photographing the driver
and image processing, the visual signals of sleepiness could be detected. Amongst
various facial features, eyes are relatively of more importance and many studies have
CHAPTER 1. INTRODUCTION 3

been conducted on the processing of the condition of the eyes. For instance, some
factors like PERCLOS (Percentage of Eye Closure), duration of eyes closure and the
number of blinks were utilized by IR Illuminator to determine the vigilance level.
Drowsiness was detected merely based on PERCLOS. A small camera conducted the
observing test detecting the levels of drowsiness by the number of eyes blinks and
accuracy of 98%.

1.2.1 Drowsiness

Feeling abnormally sleepy or tired during the day is commonly known as drowsiness.
Drowsiness may lead to additional symptoms, such as forgetfulness or falling asleep
at inappropriate times. These can range from mental states and lifestyle choices to
serious medical conditions. Certain lifestyle factors may lead to increased drowsiness,
such as working very long hours or switching to a night shift. In most cases, your
drowsiness will subside as your body adapts to your new schedule. Drowsiness can
also be a result of your mental, emotional, or psychological state.
Depression can greatly increase drowsiness, as can high levels of stress or anxiety.
Boredom is another known cause of drowsiness. If you’re experiencing any of these
mental conditions, you’re also likely to feel fatigued and apathetic. Some medical
conditions can cause drowsiness. One of the most common of these is diabetes. Other
conditions that may lead to drowsiness include those that cause chronic painor affect
your metabolism or mental state, such as hypothyroidism or hyponatremia.
Hyponatremia is when the level of sodium in your blood is too low. Other medical
conditions known to cause drowsiness include infectious mononucleosis (mono) and
chronic fatigue syndrome (CFS).
CHAPTER 1. INTRODUCTION 4

Figure 1.1: A Drowsy Man

1.2.2 Image Processing

Image processing is a method to perform some operations on an image, in order

to get an enhanced image or to extract some useful information from it. It is a
type of signal processing in which input is an image and output may be image or
characteristics/features associated with that image. Nowadays, image processing is
among rapidly growing technologies. It forms core research area within engineering
and computer science disciplines too.
Image processing basically includes the following three steps:
• Importing the image via image acquisition tools.
• Analysing and manipulating the image.
• Output in which result can be altered image or report that is based on image analysis.
There are two types of methods used for image processing namely, analogue
and digital image processing. Analogue image processing can be used for the hard
copies like printouts and photographs. Image analysts use various fundamentals of
interpretation while using these visual techniques. Digital image processing techniques
help in manipulation of the digital images by using computers. The three general
phases that all types of data have to undergo while using digital technique are pre-
processing, enhancement, and display, information extraction.
CHAPTER 1. INTRODUCTION 5

1.3 Problem statement

Driver’s inattention might be the result of a lack of alertness when driving due to driver
drowsiness and distraction. Driver distraction occurs when an object or event draws a
person’s attention away from the driving task. Unlike driver distraction, driver
drowsiness involves no triggering event but, instead, is characterized by a progressive
withdrawal of attention from the road and traffic demands. Both driver drowsiness and
distraction, however, might have the same effects, that is decreased driving
performance, longer reaction time, and an increased risk of crash involvement.

1.4 Scope of the system

• To automatically recognize the sleep and drowsiness of the driver and its detection.
• To analyze the focus of driver while driving.
• To determine concentration of drivers while driving.
• To detect whether the camera is focusing properly and the driver’s face.
• To verify the drowsiness.

1.5 Objective
The main objective of DRIVER DROWSINESS DETECTION SYSTEM is to reduce
accidents caused by the driver getting drowsy. Various studies have suggested that
around 20 percent of all road accidents are fatigue-related, up to 50 percent on certain
roads.
 Chapter 2

Literature Review

2.1 Development of an intelligent drowsiness detection

system for drivers using image processing
technique

2.1.1 Abstract

Drowsy driving highly contributes to a number of road accidents throughout the years.
Car crashes or any unwanted incidents can be avoided by implementing a system with
alarm output to alert drowsy drivers to focus on the road. An intelligent system is
developed to detect driver drowsiness and trigger alarm to alert drivers as one way
to prevent accidents, save money and reduce losses and sufferings. However, due to
high variability of surrounding parameters, current techniques have several limitations.
Bad lightings may affect camera ability to accurately measure the face and the eye of
the driver. This will affect the analysis using image processing technique due to late
detection or no detection hence decrease the accuracy and efficiency of the technique.
Several techniques have been studied and analyzed to conclude the best technique with
the highest accuracy to detect driver drowsiness. In this work, a real-time system that
utilizes computerized camera to automatically track and process driver’s eye using
Python, dlib and OpenCV is proposed. The eye region of the driver is measured and

6
CHAPTER 2. LITERATURE REVIEW 7

calculated continuously to determine the drowsiness of the driver before triggering an

output alarm to alert the driver.

2.1.2 Methodology

At first the dlib library contains a pre-trained facial landmark detector which is used
to locate a total amount of 68 (x,y) coordinates. The indexes of all 68 coordinates
are visualized. Then the eye region data obtained from facial landmarks enables
the script to continue working on the driver’s eyes to detect his/her drowsiness level.
DrawContours command extracted from OpenCV library is used to draw the eye region
to visualize and indicate eye region to ease furthers steps. After detecting the face of
the driver, based on facial landmark, the calculation of the drowsiness level was carried
out.It is found out from a equation derived from the relationship between all the axes
visualized in the eye landmark and the relation is called Eye Aspect Ratio (EAR).The
same approach has been used by several researchers in proposing their works.

EAR=(p2-p6+—p3-p5)/2(p1-p4)

To confirm the average of EAR reading during eye open, 20 trials was conducted on the
same person. The objective of conducting the trials is to find the minimum threshold
for the alarm to start giving response to awake the driver hence reducing any risk of
accidents. In this test, the alarm will start buzzing and alert the driver when the counter
reaches 48 frames. Since the code developed uses loop method in the script, the alarm
will not stop until the driver is alerted and gain consciousness to the road.

2.1.3 Conclusion

In conclusion, drowsiness detection technology is a car safety technology to help

preventing accidents caused by a drowsy driver. It is important to detect and alert
the driver early before any unwanted accidents happen that may possibly lead to death.
CHAPTER 2. LITERATURE REVIEW 8

The proposed system able to detect driver drowsiness level using image processing
technique that calculates and measures Eye Aspect Ratio, in other words, the size of
the driver’s eyes. The data on Eye Aspect Ratio has to be gathered to determine the
threshold value that indicates whenever a driver is experiencing drowsiness. Alert
system using alarm is vital as it helps in reducing the numbers of accidents caused by
drowsy driving hence reducing the total of car crashed annually.

As for now, the detection system is able to detect drowsiness of the same driver
over and over with very minimum limitation. The alarm is also working well and able
to trigger sound alarm to alert the driver. However, the threshold frames to trigger the
alarm may varies due to different Eye Aspect Ratio (EAR) in every person. Several
recommendations are suggested for future works on this area. First, the system should
be able to automatically determine the threshold of eye aspect ratio when a person
is experiencing drowsiness without setting it first for each separate individual after
several testing

2.2 Drowsy driver warning system using image pro-

cessing

2.2.1 Abstract

Driver in-alertness is an important cause for most accident related to the vehicles
crashes. Driver fatigue resulting from sleep deprivation or sleep disorders is an
important factor in the increasing number of the accidents on today’s roads. Drowsy
driver warning system can form the basis of the system to possibly reduce the accidents
related to driver’s drowsiness. The purpose of such a system is to perform detection of
driver fatigue. By placing the camera inside the car, we can monitor the face of the
driver and look for the eye-movements which indicate that the driver is no longer in
condition to driver is no longer in condition to drive. In such a case, a warning signal
should be issued. This paper describes how to find and track the eyes. We also describe
CHAPTER 2. LITERATURE REVIEW 9

a method that can determine if the eyes are open or closed. The main criterion of this
system is that it must be highly non-intrusive and it should start when the ignition is
turned on without having at the driver initiate the system. Nor should the driver be
responsible for providing any feedback to the system. The system must also operate
regardless of the texture and the color of the face. It must also be able to handle diverse
condition such as changes in light, shadows, reflections etc. In given paper a drowsy
driver warning system using image processing as well as accelerometer is proposed.
The innovations in the automobile industry over the last hundred years have made
our vehicles more powerful, easier to drive and control safer more energy efficient, and
more environmentally friendly. Majority of the accidents caused today by cars are
mainly due to the driver fatigue. Driving for a long period of time causes excessive
fatigue and tiredness which in turn makes the driver sleepy or loose awareness. With
the rapid increase in the number of accidents seems to be increasing day to day.
Therefore a need arises to design a system that keeps the driver focused on the road.
Data on road accidents in India are collected by Transport Research Wing of Ministry
of Road Transport Highways. The aim of this paper is to develop a prototype of drowsy
driver warning system. Our whole focus and concentration will be placed on
designing the system that will accurately monitor the open and closed state of the
driver’s eye in real time. By constantly monitoring the eyes, it can be seen that the
symptoms of driver fatigue can be detected early enough to avoid an accident. This
detection can be done using a sequence of images of eyes as well as face and head
movement. The observation of eye movements and its edges for the detection will be
used. Devices to detect when drivers are falling asleep and to provide warnings to alert
them of the risk, or even control the vehicle’s movement, have been the subject tomuch
research and development. Driver fatigue is a serious problem resulting in many
thousands of road accidents each year. It is not currently possible to calculate the exact
number of sleep related accidents because of the difficulties in detecting whetherfatigue
was a factor and in assessing the level of fatigue. However research suggests that up to
25% of accidents on monotonous roads in India are fatigue related. Researchin other
countries also indicates that driver fatigue is a serious problem. Young male
CHAPTER 2. LITERATURE REVIEW 10

drivers, truck drivers, company car drivers and shift workers are the most at risk of
falling asleep while driving. However any driver travelling long distances or when they
are tired, it is at the risk of a sleep related accidents. The early hours of the morning
and the middle of the afternoon are the peak times for fatigue accidents and long
journeys on monotonous roads, particularly motor-ways, are the most likely to result
in a driver falling asleep. In this paper the algorithms for face detection and eyetracking
have been developed on frontal faces with no restrictions on the background
.The proposed method for eye tracking is built into five stages. Using frontal images
obtained from a database, the probability maps for the eyes region are built etc.

2.2.2 Methodology

The proposed method is built in four stages and it is applied to the colored images with
any background.
• Localization of Face
• Localization of the Eyes
• Tracking the eyes in the subsequent frames
• Detection of failure in tracking
In this paper we first find out face Localization. For that we use symmetry-based
approach,since the he Face is symmetric. If the image is represented as I(x, y) then the
symmetry value for a pixel-column is given by S(x) = [abs I ((x,y-w)-(x,y+w))]. Next
is to find eye localization. For that we use raster scan algorithm is used for the exact
location of the eyes and extracts that vertical location of eyes. We track the eye by
looking for the darkest pixel in the predicted region. In order to recover from tracking
errors, we make sure that none of the geometrical constraints are violated. If they are,
we relocalize the eyes in the next frame. The final step is to detect drowsiness of driver
by counting the number of consecutive frames that the eyes are closed in orderto decide
the condition of the driver. For this, we need a robust way to determine if theeyes are
open or closed; so we used a method that looks at the horizontal histogram across the
pupil.
CHAPTER 2. LITERATURE REVIEW 11

2.2.3 Conclusion

We developed a system that localizes and track the eyes and head movements of the
driver in order to detect drowsiness. The system uses a combination of template –
based matching and feature based matching in order to localize the eyes. During
tracking, system will be able to decide if the eyes are open or closed and whether
the driver is looking in front. When the eyes will be closed for too long, a warning
signal will be given in the form of buzzer or alarm author-kit message.

2.3 Using image processing in the proposed drowsiness

detection system

2.3.1 Abstract

Drowsiness is one of the underlying causes of driving accidents, which contribute, to

many road fatalities annually. Although numerous methods have been developed to
detect the level of drowsiness, techniques based on image processing are quicker and
more accurate in comparison with the other methods. The aim of this study was to use
image-processing techniques to detect the levels of drowsiness in a driving simulator.
This study was conducted on five suburban drivers using a driving simulator based
on virtual reality laboratory of Khaje-Nasir Toosi University of Technology in 2015
Tehran, Iran. The facial expressions, as well as location of the eyes, were detected by
Violla-Jones algorithm. Criteria for detecting drivers’ levels of drowsiness by eyes
tracking included eye blink duration blink frequency and PERCLOS that was used to
confirm the results. Eye closure duration and blink frequency have a direct ratio of
drivers’ levels of drowsiness. The mean of squares of errors for data trained by the
network and data into the network for testing, were 0.0623 and 0.0700, respectively.
CHAPTER 2. LITERATURE REVIEW 12

Meanwhile, the percentage of accuracy of detecting system was 93.

2.3.2 Methodology

This method is conducted in several steps as follows: first, the image of driver’s facial
features taken by the camera is transferred to a processor to be processed. Next, the
facial features and location of the eyes are determined by Violla- Jones algorithm.
For convenience, categorizers are utilized in the cascade sequence.). In this method,
driver’s face was detected with regard to the oval shape of the head, hue and the eye
sockets. Then, the zone of the eyes was recognized by considering the various changes
in derivatives of the visual information.Finally, to recognize whether eyes are closed or
open, images are converted into gray scale for- mat.After that, the image is divided into
upper and lower part. When eyes are open, due to the color of pupils and eyelash, the
ratio of dark pixels in upper part of eye is greater than the state of closed eyes because
in the case of closed eyes, eyelids with light color cover pupils.Then, the ratio of black
pixels to the white pixels was calculated in both upper and lower parts. Finally, the
ratio of these amounts was used to recognize whether eyes are open or closed.If the
ratio is higher then the eyes are opened, else the eyes are closed.

2.3.3 Conclusion

The results showed several dynamic changes of the eyes during the periods of
drowsiness. The present study proposes a fast and accurate method for detecting
the levels of drivers’ drowsiness by considering the dynamic changes of the eyes. A
benefit of the method to detect drowsiness is intelligent sleepiness detection and use
of various parameters in a long time and driving background. This advantage leads
to detecting drowsiness in early stages and activate the alarm before a car accident
occur. However, sleepiness is very complicated to be detected completely only by
image processing. Therefore, various aspects of drowsiness were studied in virtual-
CHAPTER 2. LITERATURE REVIEW 13

reality driving simulator laboratory of Khaje Nasir University. Additionally, by using

more criteria and obtained information by sensors drowsiness is detected.

2.4 Drowsy driver detection through facial movement

analysis

2.4.1 Abstract

The advance of computing technology has provided the means for building intelligent
vehicle systems. Drowsy driver detection system is one of the potential applications of
intelligent vehicle systems. Previous approaches to drowsiness detection primarily
make preassumptions about the relevant behavior, focusing on blink rate, eye closure,
and yawning. Here we employ machine learning to datamine actual human behavior
during drowsiness episodes. Automatic classifiers for 30 facial actions from the Facial
Action Coding system were developed using machine learning on a separate database
of spontaneous expressions. These facial actions include blinking and yawn motions,
as well as a number of other facial movements. In addition, head motion was collected
through automatic eye tracking and an accelerometer. These measures were passed
to learning-based classifiers such as Adaboost and multinomial ridge regression. The
system was able to predict sleep and crash episodes during a driving computer game
with 96% accuracy within subjects and above 90% accuracy across subjects. This is
the highest prediction rate reported to date for detecting real drowsiness. Moreover,
the analysis revealed new information about human behavior during drowsy driving.

2.4.2 Methodology

At first the head movement was measured using an accelerometer that has 3 degrees
of freedom.This three dimensional accelerometer 3 has three one dimensional ac-
CHAPTER 2. LITERATURE REVIEW 14

celerometers mounted at right angles measuring accelerations in the range of 5g to

+5g where g represents earth gravitational force.Next, he facial action coding system
(FACS) is arguably the most widely used method for coding facial expressions in
the behavioral sciences. The system describes facial expressions in terms of 46
component movements, which roughly correspond to the individual facial muscle
movements.Some examples are shown below.

Sl. No Name
1 Outer Brow Raise
2 Inner Brow Raise
3 Cheek Raise
4 Nasolabial Furrow Deepener
5 Chin Raise
6 Nostril Compress
7 Blink
8 Lip Stretch
9 Tongue show
10 Dimpler
11 Jaw Drop
12 Mouth Stretch
13 Lips Suck
14 Jaw Sideways
15 Sharp Lip Puller

Table 2.1: set of action units used for predicting drowsiness

Next,the facial action outputs were passed to a classifier for predicting drowsiness
based on the automatically detected facial behavior. Two learning-based classifiers,
Adaboost and multinomial ridge regression are compared. Within-subject prediction
of drowsiness and across-subject (subject independent) prediction of drowsiness were
both tested.

2.4.3 Conclusion

This paper presented a system for automatic detection of driver drowsiness from video.
Previous approaches focused on assumptions about behaviors that might be predictive
CHAPTER 2. LITERATURE REVIEW 15

of drowsiness. Here, a system for automatically measuring facial expressions was

employed to datamine spontaneous behavior during real drowsiness episodes. This
is the first work to our knowledge to reveal significant associations between facial
expression and fatigue beyond eye blinks. The project also revealed a potential
association between head roll and driver drowsiness, and the coupling of head roll with
steering motion during drowsiness. Of note is that a behavior that is often assumed to
be predictive of drowsiness, yawn, was in fact a negative predictor of the 60-second
window prior to a crash. It appears that in the moments before falling asleep, drivers
yawn less, not more, often. This highlights the importance of using examples of fatigue
and drowsiness conditions in which subjects actually fall sleep.

2.5 Real-Time Detection for Drowsy Driving via Acous-

tic Sensing on Smartphones

2.5.1 Abstract

Drowsy driving is one of the biggest threats to driving safety, which has drawn much
public attention in recent years. Thus, a simple but robust system that can remind
drivers of drowsiness levels with off-the-shelf devices (e.g., smartphones) is very
necessary. With this motivation, we explore the feasibility of using acoustic sensors
on smartphones to detect drowsy driving. Through analyzing real driving data to study
characteristics of drowsy driving, we find some unique patterns of Doppler shift caused
by three typical drowsy behaviours (i.e., nodding, yawning and operating steering
wheel), among which operating steering wheels is also related to drowsiness levels.
Then, a real-time Drowsy Driving Detection system named D3 -Guard is proposed
based on the acoustic sensing abilities of smartphones. We adopt several effective
feature extraction methods, and carefully design a high-accuracy detector based on
LSTM networks for the early detection of drowsy driving. Besides, measures to
distinguish drowsiness levels are also introduced in the system by analyzing the data of
CHAPTER 2. LITERATURE REVIEW 16

operating steering wheel. Through extensive experiments with 5 drivers in real driving
environments, D3 -Guard detects drowsy driving actions with an average accuracy of
93.31%, as well as classifies drowsiness levels with an average accuracy of 86%

2.5.2 Methodology

• Building drowsy driving detection model.

The data expansion, frame segmentation, feature extraction and network training in the
offline phase are only carried out once when the system is built. So they do not
contribute to the computational complexity.
• Frame segmentation
In the online phase, we cut the received audio signal directly into several frames with
fixed length, that is to say, the computational complexity of frame segmentation is
O(N).
•Feature extraction
Feature extraction includes three steps: band-pass filter, undersampling and FFT. Each
frame needs to be processed by these three steps, which can be done in linear time. So
the computational complexity of feature extraction is O(N + N + N) = O(N). Drowsy
driving detection. Two LSTM networks, LSTMS and LSTM-L, are mainly used for
drowsy detection. The computational complexity of LSTM-S is O(tsusN), where ts is
the number of timesteps and us is the number of units in LSTM-S. The computational
complexity of LSTM-L is O(tlulN), where tl and ul are the numbers of timesteps and
units in LSTM-L. So the drowsy detection has the complexity of O((tsus + tlul)N).

2.5.3 Conclusion

In this paper, we address how to detect drowsy driving in real time to improve driving
safety. Through literature survey and experimental verification, we find that for a
particular drowsy driving action, different drivers have similar patterns. We propose
CHAPTER 2. LITERATURE REVIEW 17

a real-time drowsy driving detection system, D3 -Guard, to detect drowsy driving

actions and analyze drowsiness level at early stage which purely leverages build-in
audio devices on smartphones. D3 - Guard extracts features of nodding, yawning and
operating SW by using undersampling based on Doppler profiles. D3 - Guard then
takes advantages of LSTM networks to build an efficient detection which integrates
the recognition of three drowsy driving actions. Finally, we use a simple but effective
method to analyze drowsiness levels of drivers. We conduct extensive experiments
with 5 volunteer drivers for 6 months in real driving environments. The experimental
results show D3 -Guard achieves an average total accuracy of 93.31% for drowsy
actions detection, with over 80% drowsy driving actions detected before 70% of action
duration. The accuracy of detection for each type of drowsiness level exceeds 80%.
CHAPTER 2. LITERATURE REVIEW 18

2.6 CONSOLIDATED TABLE

Paper Algorithm/Technology Findings

1.Development of an Facial landmark,Eye EAR produce highly
intelligent drowsiness Aspect Ratio(EAR) accurate result
detection system for
drivers using image
processing technique
2.Drowsy Driver Warn- Symmetry-based This approch have ca-
ing System Using Im- approach,Raster scan pability of high speed
age Processing algorithm processing
3.Using image Violla-Jones The level of accuracy of
processing in the algorithm,grey scale result was estimated at
proposed drowsiness 93%
detection system
4.Drowsy driver Facial Action Coding FACS produces 46 fa-
detection through facial System,grey scale cial actions
movement analysis
5.Real-Time Detection Acoustic Sensing D3-Guard, to detect
for Drowsy Driving via drowsy driving actions
Acoustic Sensing on and analyze drowsiness
Smartphones level at early stage
which purely leverages
build-in audio devices
on smartphones.

Table 2.2: Consolidated Table

 Chapter 3

Requirement Specification

3.1 Functional requirements

• It reduces road accidents.
• It makes alarm sounds when necessary.
• We can adjust the volume of alarm with drivers preferences.

3.2 Software Requirements

• Code Editor: VS Code
• Language(s): Python3
• Library: OpenCV, Dlib

3.3 Hardware interfaces

• Camera
• Alarm system

19
CHAPTER 3. REQUIREMENT SPECIFICATION 20

3.4 System Hardware requirements

• RAM: 6 GB or higher
• Disk: 25 GB + free space
 Chapter 4

Proposed system and Design

This chapter is mainly discusses about the proposed system and design. Also the the
architecture and different technical diagrams are discussed in this chapter.

4.1 Proposed system

The proposed system here is designed to minimise the occurrence of countless mishaps
due to the drowsy driver. Nowadays, fatigue of driver causes road accidents every now
and then across the world. So, these activities should be required to automatically
handle an implementation of smart alert system or vigilance in a vehicle which is an
objective of this system. To analyze different behavioral or visual-based attitudes of
the driver, face movement and eye blink are measured to study the state of the driver.
Here, eye blink is mainly focused to detect drowsiness of the driver. The threshold
value of an EAR lies above 0.25 without any effect of exhaustion. When a driver
automatically shuts down, then the threshold value of EAR falls below the given range.
A threshold value of drowsy eye blink sample represents the number of video frames
of the driver’s closed eyes. If the consecutive counting frames increase above the range
of the threshold value, then the drowsiness of the driver is detected. Here, a camera
is used to regularly record the total movement of an eye through which the threshold
value of an EAR is calculated. A counter is also included in it for counting occurrence
of frames. Suppose that it exceeded above a range of 30. In that case, a voice is

21
CHAPTER 4. PROPOSED SYSTEM AND DESIGN 22

activated by a speaker and a mail is automatically sent to an authorized person of the

vehicle which is generally processed at the time of drowsiness detection.

4.1.1 68 Point Facial Landmark

Face detection is the first methods which locate a human face and return a value in
x,y,w,h which is a rectangle. After getting the location of a face in an image, then we
have to through points inside of that rectangle. Facial landmark detection is a computer
vision task in which a model needs to predict key points representing regions or
landmarks on a human’s face – eyes, nose, lips, and others. The detection and grouping
of 68 facial landmarks. The 68 detected facial landmarks are divided into seven
components, i.e., left eyebrow, right eyebrow, left eye, right eye, nose, mouth, and jaw.

Figure 4.1: 68 Point Facial Landmark

CHAPTER 4. PROPOSED SYSTEM AND DESIGN 23

4.1.2 Eye Aspect Ratio

Eye Aspect Ratio is proposed by the Czech Technical University. It is a scalar quantity
obtained by detecting a face from an image, finding the Euclidean distance of the
corresponding eye coordinates, and substituting it into the following formula. where

Figure 4.2: EAR Formula

p1, . . ., p6 are the 2D landmark locations, depicted in Fig. 4.3 and Fig. 4.4. When the
eye is open, the value is almost constant, but it approaches zero when the eye is closed.
This value does not depend on head posture or distance, and there are small individual
differences when the eyes are open. Since eye blinking is performed by both eyes
synchronously, the EAR of both eyes is averaged. Due to the simplicity of the
calculation formula, it has excellent real-time performance and shows high robustness.
However, although we are trying to classify irregular movements such as eye thinning,
absent stretching, or only eyes looking down, detection is difficult at the threshold.
Also, although it is claimed to have high and straightforward detection accuracy, it
causes a loss of accuracy when there is a sudden movement of the face or when the
face is exceptionally far away from the camera. The resolution degradation due to
distance could be solved by converting face images to higher resolution using super
resolution techniques, but this has not been investigated.
CHAPTER 4. PROPOSED SYSTEM AND DESIGN 24

Figure 4.3: Open Eye

Figure 4.4: Closed Eye

4.2 Feasibility Study

A feasibility study is an analysis that considers all of a project’s relevant factors
including economic, technical, legal, and scheduling considerations to ascertain the
likelihood of completing the project successfully.
CHAPTER 4. PROPOSED SYSTEM AND DESIGN 25

4.2.1 Technical Feasibility

Python which is one of the most used programming language is used as the basic
language for this project. And open image processing/detection libraries like OpenCV
And we use facial action coding system for computing facial expression of drivers.

4.2.2 Economic Feasibility

This project is economically feasible. Most of the softwares used to build this project
is free of cose.

4.2.3 Operational Feasibility

Operational feasibility is the measure of how well a proposed system solves the
problems, and takes advantage of the opportunities identified during scope definition
and how it satisfies the requirements identified in the requirements analysis phase of
system development.The system is operationally feasible.
CHAPTER 4. PROPOSED SYSTEM AND DESIGN 26

4.3 DESIGN

4.3.1 ARCHITECTURE DIAGRAM

An architectural diagram is a diagram of a system that is used to abstract the overall

outline of the software system and the relationships, constraints, and boundaries
between components. It is an important tool as it provides an overall view of the
physical deployment of the software system and its evolution roadmap.

Figure 4.5: architecture diagram

CHAPTER 4. PROPOSED SYSTEM AND DESIGN 27

4.3.2 USE CASE DIAGRAM

A dynamic and behavioral diagram in UML is use case diagram. Use cases are
basically set of actions, services which are used by system. To visualize the
functionality requirement of the system this use case diagram are used. The internal
and external events or party that may influence the system are also picturized. Use case
diagram specify how the system acts on any action without worrying to know about
the details how that functionality is achieved.

• User
User is the person who is benefited through the system.

Figure 4.6: Use case diagram

CHAPTER 4. PROPOSED SYSTEM AND DESIGN 28

4.3.3 Data Flow Diagram (LEVEL 0)

A Data Flow Diagram (DFD) is a visual representation of the information flows within
a system. It provides information on how how data enters and leaves the system,the
changes in the system and where the data is stored. Data flow diagrams visually
represent systems and processes. It may be partitioned into levels that represent
increasing information flow and functional details. Levels in DFD are numbered 0,
1, 2 or beyond.

Figure 4.7: Data Flow Diagram (Level 0)

CHAPTER 4. PROPOSED SYSTEM AND DESIGN 29

4.3.4 Data Flow Diagram (LEVEL 1)

Level 1 DFD’s provides a more detailed breakout of pieces of Level 0 DFD. In a level
1 data flow diagram, the single process node from the context diagram is broken down
into sub processes.

Figure 4.8: Data Flow Diagram Level 1

CHAPTER 4. PROPOSED SYSTEM AND DESIGN 30

4.3.5 Data Flow Diagram (LEVEL 2)

Level 2 of DFD goes one process deeper into parts of Level 1. It can be used to project
or record the necessary detail about the system’s functioning.

Figure 4.9: Data Flow Diagram Level 2

 Chapter 5

IMPLEMENTATION

5.1 Implementation Details

5.1.1 Capturing Video

5.1.2 Initializing face detector

5.1.3 Finding EAR

31
CHAPTER 5. IMPLEMENTATION 32

5.1.4 Checking drowsiness

5.2 Modules Used

The tools and technologies that we used for this project are mentioned below.

5.2.1 Python

Python is a high-level, interpreted, general-purpose programming language. Its design

philosophy emphasizes code readability with the use of significant indentation. Python
is dynamically-typed and garbage-collected. Python is a computer programming
language often used to build websites and software, automate tasks, and conduct data
analysis. Python is a general-purpose language, meaning it can be used to create a
variety of different programs and isn’t specialized for any specific problems. Python 3
stores code as a byte code before it is interpreted on a platform. Python code doesn’t
need to be compiled. It is interpreted on the machine, it has been executed on. Python
has features of an object-oriented as well as a procedural programming language.

5.2.2 OpenCV

OpenCV (Open Source Computer Vision Library) is an open source computer vision
and machine learning software library. OpenCV was built to provide a common
infrastructure for computer vision applications and to accelerate the useof machine
perception in the commercial products. Being a BSD-licensed product,
CHAPTER 5. IMPLEMENTATION 33

OpenCV makes it easy for businesses to utilize and modify the code. OpenCV- Python
is a library of Python bindings designed to solve computer vision problems. Python is
a general purpose programming language started by Guido van Rossumthat became
very popular very quickly, mainly because of its simplicity and code
readability.OpenCV (Open Source Computer Vision Library) is an open source
computer vision and machine learning software library. OpenCV was built to provide
a common infrastructure for computer vision applications and to accelerate the use of
machine perception in the commercial products.

5.2.3 Dlib

Dlib is a general purpose cross-platform software library written in the programming

language C++. Its design is heavily influenced by ideas from design by contract
and component-based software engineering. Thus it is, first and foremost, a set of
independent software components. Since development began in 2002, Dlib has grown
to include a wide variety of tools. As of 2016, it contains software components for
dealing with networking, threads, graphical user interfaces, data structures, linear
algebra, machine learning, image processing, data mining, XML and text parsing,
numerical optimization, Bayesian networks, and many other tasks OpenCV - image
processing library. Dlib - machine learning library. They are different and solving
different tasks. Some projects need both of them.

5.2.4 Numpy

NumPy is a Python library used for working with arrays. It also has functions for
working in domain of linear algebra, fourier transform, and matrices. NumPy was
created in 2005 by Travis Oliphant. It is an open source project and you can use it
freely. NumPy stands for Numerical Python. In Python we have lists that serve the
purpose of arrays, but they are slow to process. NumPy aims to provide an array object
that is up to 50x faster than traditional Python lists. The array object in NumPy is called
ndarray, it provides a lot of supporting functions that make working with ndarray very
CHAPTER 5. IMPLEMENTATION 34

easy. Arrays are very frequently used in data science, where speed and resources are
very important. NumPy arrays are stored at one continuous place in memory unlike
lists, so processes can access and manipulate them very efficiently. This behavior is
called locality of reference in computer science. This is the main reason why NumPy
is faster than lists. Also it is optimized to work with latest CPU architectures. NumPy
is a Python library and is written partially in Python, but most of the parts that require
fast computation are written in C or C++.

5.2.5 Imutils

A series of convenience functions to make basic image processing functions such as

translation, rotation, resizing, skeletonization, and displaying Matplotlib images easier
with OpenCV and both Python 2.7 and Python 3. For convenience, the skeletonize
function of imutils can be used to construct the topological skeleton of the image. The
first argument, size is the size of the structuring element kernel. An optional argument,
structuring , can be used to control the structuring element — it defaults to cv2. Imutils
is Python basic image processing functional package to do image translation, rotation,
resizing, skeletonization, or blur amount detection. imutils also check to find functions
if you already have NumPy, SciPy, Matplotlib, and OpenCV installed.

5.2.6 Playsound

The playsound module contains only a single function named playsound(). It requires
one argument: the path to the file with the sound we have to play. It can be a local file,
or a URL. There’s an optional second argument, block, which is set to Trueby
default The PlaySound function plays a sound specified by the given file name,
resource, or system event. (A system event may be associated with a sound in the
registry or in the WIN. INI file. playsound is a “pure Python, cross platform, single
function module with no dependencies for playing sounds.” With this module, you can
play a sound file with a single line of code: from playsound import playsound
playsound(’myfile.wav’). The playsound module is a cross platform module that can
CHAPTER 5. IMPLEMENTATION 35

play audio files. This doesn’t have any dependencies, simply install with pip in your
virtualenv and run!Implementation is different on platforms. It uses windll.winm on
Windows, AppKit.NSSound on Apple OS X and GStreamer on Linux
 Chapter 6

RESULT AND DISCUSSION

6.1 Active

36
CHAPTER 6. RESULT AND DISCUSSION 37

6.2 Sleeping
CHAPTER 6. RESULT AND DISCUSSION 38

6.3 Drowsiness
 Conclusion

The report is based on the first review of the product ”DRIVER DROWSINESS
DETECTION SYSTEM”. The project satisfies the problem which is stated in the
problem definition section. In conclusion, drowsiness detection technology is a car
safety technology to help preventing accidents caused by a drowsy driver. It is
important to detect and alert the driver early before any unwanted accidents happen
that may possibly lead to death. With this system we hope we can reduce accidents in
our society.

39
 References

1. Esra Vural, Mujdat Cetin and Aytul Ercil, ”Drowsy Driver Detection Through Facial
Movement Analysis ”, DOI: 10.1109/IPAS50080.2020.9334934
2. Mohsen POURSADEGHIYAN, Adel MAZLOUMI and Gebraeil NASL SARAJI,
”Using Image Processing In Proposed Drowsiness Detection System Design ”, DOI:
10.1016/j.compag.2017.03.016
3. Singh Himani Parmar, Mehul Jajal and Yadav Priyanka, ”Drowsy Driver Warning
System Using Image Processing ”, DOI: 10.1186/s13007-019-0475-z
4. Amin Azizi Suhaiman, Zazilah May and Noor A ’in A.Rahman, ”Development Of
an Intelligent Drowsiness Detection System for Drivers Using Image Processing
Technique”, DOI: 10.1016/j.inpa.2020.09.006
5. Yadong Xie, Fan Li, Song Yang and Yu Wang, ”Real-Time Detection for Drowsy
Driving Via Acoustic Sensing On Smartphone ”, ISSN: 2349-6363

40
 CODE
import cv2
import numpy as np
import dlib
from imutils import face_utils

cap = cv2.VideoCapture(0)

detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")

sleep = 0
drowsy = 0
active = 0
status = ""
color = (0, 0, 0)

def compute(ptA, ptB):

dist = np.linalg.norm(ptA - ptB)
return dist

def blinked(a, b, c, d, e, f):

up = compute(b, d) + compute(c, e)
down = compute(a, f)
ratio = up / (2.0 * down)

if ratio > 0.25:

return 2
elif 0.21 < ratio <= 0.25:
return 1
else:
return 0

while True:
_, frame = cap.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

faces = detector(gray)
for face in faces:
x1 = face.left()
y1 = face.top()
x2 = face.right()
y2 = face.bottom()
 face_frame = frame.copy()
cv2.rectangle(face_frame, (x1, y1), (x2, y2), (0, 255, 0), 2)

landmarks = predictor(gray, face)

landmarks = face_utils.shape_to_np(landmarks)

left_blink = blinked(landmarks[36], landmarks[37], landmarks[38], landmarks[41],

landmarks[40], landmarks[39])
right_blink = blinked(landmarks[42], landmarks[43], landmarks[44],
landmarks[47], landmarks[46], landmarks[45])

if left_blink == 0 or right_blink == 0:
sleep += 1
drowsy = 0
active = 0
if sleep > 6:
status = "SLEEPING !!!"
color = (255, 0, 0)
elif left_blink == 1 or right_blink == 1:
sleep = 0
active = 0
drowsy += 1
if drowsy > 6:
status = "Drowsy !"
color = (0, 0, 255)
else:
drowsy = 0
sleep = 0
active += 1
if active > 6:
status = "Active :)"
color = (0, 255, 0)

cv2.putText(frame, status, (100, 100), cv2.FONT_HERSHEY_SIMPLEX, 1.2, color, 3)

for n in range(0, 68):

(x, y) = landmarks[n]
cv2.circle(face_frame, (x, y), 1, (255, 255, 255), -1)

cv2.imshow("Result of detector", face_frame)

cv2.imshow("Frame", frame)
key = cv2.waitKey(1)
 if key == 27:
break

cap.release()
cv2.destroyAllWindows()