Tribhuvan University

Nagarjuna College of Information Technology

A Final Year Project Report

On

“DROWSINESS MONITORING SYSTEM”

Submitted to:

Department of Computer Science and Information Technology Nagarjuna College of

Information Technology

In Partial Fulfillment of the Requirement for the Bachelor Degree in Computer Science
and Information Technology

Submitted By:

Rojina Maharjan (10796/073)

Suresh B.K (10806/073)

Sugam Timalsina (10804/073)

April 20, 2021

 SUPERVISOR’S RECOMMENDATION

I hereby recommend that this project prepared under my supervision by Suresh B.K, Rojina
Maharjan, Sugam Timalsina entitled “DROWSINESS MONITORING SYSTEM” in
partial fulfillment of the requirements for the degree of BSc. in Computer Science and
Information Technology, be processed for the evaluation.

…………………………..

Ramesh Singh Saud

SUPERVISOR

Nagarjuna College of IT

Shankhamul, Lalitpur
 LETTER OF APPROVAL

This is to certify that this project prepared by Suresh B.K, Rojina Maharjan, Sugam
Timalsina entitled “Drowsiness Monitoring System” in partial fulfillment of the
requirements for the degree of B.Sc. in Computer Science and Information Technology has
been well studied. In our opinion it is satisfactory in the scope and quality as a project for the
required degree.

------------------------------

Mr. Ramesh Singh Saud

Project Supervisor

Nagarjuna College of Information Technology

Shankhamul, Lalitpur

-------------------------

External Examiner
 ACKNOWLEDGEMENT

The completion of this project would not have been possible without the support and
guidance of many individuals. We are grateful to whole faculty of Nagarjuna College of
Information Technology for guidance and supervision, as well as providing all the necessary
support and friendly environment for the successful completion of the project.

We would like to express our gratitude to our project supervisors Ramesh Sing Saud who
took an interest in our project and guided us through the project by providing necessary
ideas, information and knowledge for developing Drowsiness Monitoring System. We would
like to thank Ramesh Sing Saud and Dilli Adhikari for their encouragement and guidance
towards the making of this report as per the standard.

We are thankful and fortunate enough to get constant support from our colleagues and
teaching staff of B.Sc. CSIT department, which helped us, complete our project. We would
also like to extend our regards to all the non-teaching staff of B.Sc. CSIT department for
their timely support.
 ABSTRACT

Driving while being drowsy and sleepy is a major cause behind road accidents and it exposes
the driver to a much higher crash risk compared to driving while alert. Therefore, The
adoption of assistive technologies that monitor a driver's level of difficulty and warn the
driver when he or she is fatigued can help prevent accidents. This project introduces three
different methods towards the detection of drivers’ drowsiness based on yawning
measurement. All three approaches involve several steps, including the real time detection of
the driver’s face, mouth and yawning. The suggested system can effectively assess the
aforementioned characteristics and detect the yawning condition as a symptom of a driver's
sleepiness, according to test findings.
 TABLE OF CONTENTS
ii

SUPERVISOR’S RECOMMENDATION......................................................................................................2
LETTER OF APPROVAL............................................................................................................................3
ACKNOWLEDGEMENT............................................................................................................................i
ABSTRACT..............................................................................................................................................ii
CHAPTER 1: INTRODUCTION..................................................................................................................1
1.1 Introduction...............................................................................................................................1
1.2 Problem Statement....................................................................................................................2
1.3 Objectives...................................................................................................................................2
1.4 Scope and Limitation.................................................................................................................3
1.4.1 Scope....................................................................................................................................3
1.4.2 Limitation............................................................................................................................3
1.5 Methodology...........................................................................................................................3
1.6 Report Organization............................................................................................................4
CHAPTER 2: LITERATURE REVIRW..........................................................................................................5
CHAPTER 3: REQUIREMENT ANALYSIS AND FEASIBILITY STUDY............................................................7
3.1 Requirement Analysis...............................................................................................................7
3.2.1 Functional Requirement.....................................................................................................7
3.2.2 Non-Functional Requirement............................................................................................7
3.3 Feasibility Study........................................................................................................................7
3.3.1 Technical Feasibility...........................................................................................................8
3.3.2 Economic Feasibility...........................................................................................................8
3.3.3 Operational Feasibility.......................................................................................................8
3.3.4 Schedule Feasibility............................................................................................................8
CHAPTER 5: DESIGN OF SYSTEM..........................................................................................................10
5.1 System Architecture................................................................................................................10
5.1.1 System Flow Chart............................................................................................................11
5.1.3 DFD....................................................................................................................................12
5.1.4 Use Case Diagram.............................................................................................................13
CHAPTER: 5 IMPLEMENTATION AND TESTING....................................................................................14
5.1 Algorithm Description.............................................................................................................14
5.2 Implementation........................................................................................................................18
5.2.1 Requirement Testing........................................................................................................28
 5.2.2 Model Testing....................................................................................................................28
CHAPTER 6: CONCLUSION & FUTURE ENHANCEMENTS......................................................................29
6.1 Conclusion................................................................................................................................29
6.2 Future Enhancements.............................................................................................................29
LIST OF ABBREVIATIONS......................................................................................................................31
REFERENCE AND BIBLIOGRAPHY.........................................................................................................32
 LIST OF FIGURES
Figure 1: Gannt Chart........................................................................................................................9
Figure 2: System architecture...........................................................................................................10
Figure 3: System Flow Chart............................................................................................................11
Figure 4: DFD Diagram....................................................................................................................12
Figure 5: Use Case Diagram.............................................................................................................13
 CHAPTER 1: INTRODUCTION

1.1 Introduction
Drowsiness Monitoring System is the monitoring system that monitors the driver's
face expression and facial movement like eye blink, yawn by analyzing mouth, eye
through real time camera and then alerts the driver though the alarm system
embedded in it when the driver is in a drowsy state. Drowsiness monitoring system is
a safety feature that can help deter injuries caused by drivers who have fallen asleep
behind the wheel.

The main application of this system is to anticipate fatigue in the driver and also give
the warning or trigger output in the form of alarm. The shifts in the mouth and eyes
are examined and processed to determine whether or not the drivers are sleepy and to
give a warning. For the person's fatigue identification, two scenarios are considered:
eye closing and yawning. Since fatigued drivers cause a high number of traffic
accidents every single day, this Drowsiness Monitoring aims to prevent many
accidents and injuries on the road, saving costs and minimizing personal misery. 

When a driver is fatigued, his or her skill levels, driving habits, proficiencies, and
choices are all harmed, and the high crash rate is due to sleeping drivers failing to take
the necessary precautions prior to a crash. Drowsy driving is a major problem in our
country, not just because it hurts those who drive while drowsy, but also because it
endangers all other road users. To deter traffic collisions, the use of assisting devices
that track a driver's level of caution is critical.

A highly drowsy driver will have extended incompetence to successfully perform a

driving maneuver, will be unaware of the vehicle's turning radius, will perform
driving maneuvers under the erroneous presumption that they are secure, will have
eye lid closures and difficulty holding his or her head in a lifted position, will have
limited body/eye movement, and will yawn frequently. In the case of drowsiness or
inattention like frequent yawning and eye lid closure for a prolonged period of time,
the Drowsiness Monitoring System alerts the driver.

1
The Drowsiness Monitoring system mainly focuses on detecting the people’s face in
real time through a camera.  The preprocessing aspect of the face detection is handled
by OpenCV, which employs a Practical Landmark model for eye and mouth detection
and the EAR value is calculated on the basis of lip distance and eye distance. Finally,
Flask is used to build graphical user interfaces of Drowsiness monitoring System.

1.2 Problem Statement

Road traffic deaths and accidents are on the rise in Nepal, posing a threat to the
country's long-term growth and. Driver’s drowsiness is also one of the main causes
that leads to various accidents. In Nepal, 2,541 traffic deaths were officially recorded
in fiscal year 2017–18, equating to a fatality rate of 8.59 per 100,000 people. During
the same time frame, 4,144 serious injury victims and a number of minor injury
victims were officially identified. Since the early 2000s, there has been a rapid
increase in the number of people killed or injured in car accidents. The poor and
vulnerable condition of roads of Nepal and carelessness of the driver is also one of the
main reasons behind such a number of accidents. Drinking alcohol, taking medicines
that causes drowsiness, driving for a prolonged period of time for like 8 to 10 hours in
a row, eating heavy food etc. can cause the road accidents.  

1.3 Objectives

The   main goal of Drowsiness Monitoring system is to design and develop an

application framework that detects the drowsiness by analysing driver’s facial
expression, yawn and eye blink. 
The main objective of Drowsiness Monitoring system:
 To continuously monitor the eyes and mouth of the driver’s face.
 To alert the driver on the detection of drowsiness by using a buzzer or alarm.
 To prevent the accidents caused by Driver’s drowsiness and improve road
safety.
s

2
1.4 Scope and Limitation

1.4.1 Scope

Drowsiness Monitoring System aims to take photo and video from the dashboard
camera placed Infront of the driver and actively monitor the live stream to detect and
alert the drowsiness. For the future of long route driving, night-time driving and
security for the travelers this application aims to play a very important role to reduce
the number of accidents and decrease the rate of accidents. This system can be used in
all types of vehicles except for two wheelers. Traveling agencies, logistics agents and
transportation media can utilize this system for additional security of their traveling.
The future of road transport can be changed and turn it more safe and secure even for
the individual drivers.

1.4.2 Limitation
Some of the limitations of the systems are:

 This system is completely based offline.

 This system is limited to windows desktop.
 This system is not reliable for two wheelers.

 There is no dual user usability available.

1.5 Methodology
 Webcam - It is used for capturing the video in real time and monitors a stream
for faces. In order to effectively capture the face, the webcam is placed onto
the vehicle dashboard and is approximately 20cm away from the driver’s face.
At this distance, the webcam captures the most of the driver’s face. The
captured video is sent to the Processor for further processing.
 Desktop Devices - The desktop device can be Laptop and other computing
devices like Raspberry Pi. Python is the main programming language for
Raspberry Pi. It performs a processing of the input video stream so to compute
the level of fatigue of the driver. The analysis is based on calculating several
frames of the Data stream where the driver eyes are closed. Video segments

3
 whose average eye state point exceeds the threshold value are detected as
drowsy.
 Alarm - When the drowsiness index exceeds a pre-specified Parameter or
when the signal falls below the specified threshold it activates an alarm signal.
Loud Alarm is buzzed to alert the driver indicating that he is drowsy and is
dangerous to drive in this state and that he must take a break.

1.6 Report Organization

This report is divided into 5 chapters. Each chapter is further divided into different
headings.

Chapter 1 gives an introduction. The problem definition, objectives, scopes, and

limitations of this system are discussed here.

  Chapter 2 focuses on the analysis part. It contains a literature review section where
the research works done in the field of recognition systems are discussed in brief. This
chapter also includes requirement analysis, feasibility study, and system structure.

Chapter 3 discusses in detail the design of the system. The chapter also discusses
process design, input-output design.

Chapter 4 gives information about the implementation and testing process. It

discusses how the system is implemented and what tools and software are used to
implement this system. The testing process is also included in detail in this chapter.

Chapter 5 includes the conclusion of the system and future recommendations.

4
 CHAPTER 2: LITERATURE REVIRW

Mandalapu Sarada Devi et al., (2008) created a device that can predict oncoming
driver fatigue and issue a timely alert, which could assist in the prevention of many
crashes, saving money and reducing personal misery. Video camera was used that was
pointed directly towards the driver’s face in order to detect fatigue. If the fatigue was
detected a warning signal was issued to alert the driver. The video files recorded by
the camera was converted into frames. When the eyes have been placed in each
picture, the distance between the intensity changes in the eye area can be used to
decide if the eyes is open or closed by measuring the distance between the intensity
changes in the eye area. If the driver's eyes are closed for 5 frames in a row, the
machine concludes that he or she is falling asleep and sends a warning signal. The
horizontal pressure difference on the face served as the foundation for this system's
method. One thing that both faces have in common was that the strength of the
eyebrows was noticeably different from that of the skin, and that the eye was the next
noticeable difference in intensity in the direction. The core of locating the eyes on the
face was facial features, which enabled the device to track the eyes and identify long
periods of eye closure.

Hyungseob Han et. al.,(2014) created a method of drowsiness detection with eyes

open using EED based power spectrum analysis. To minimize the anticraft, all mobile
devices were turned off, and a noiseless environment was developed to induce
drowsiness. Drowsy intervals are defined after the EEG experiment is completed
based on variations in the alpha power spectrum caused by closed eyes in a drowsy
state.
Drowsiness tendencies can be found even though the subject's eyes were open for a
long time. As a result, using EEG-based power spectrum analysis and proper features
vectors by LPC(Linear Predictive Coding) coefficient, prediction of drowsiness with
eyes open was feasible.

Poneh. R. Tabrizi et.al., (2008) had developed an easy algorithm for pupil center and
iris boundary localization and a new algorithm for eye state analysis, which there was
incorporation into four step system for drowsiness detection: face detection, eye

5
detection, sye state analysis and drowsy decision. This new system did not need any
training data or special cameras at any stage. Using an eye tracker, they were able to
achieve excellent pupil center and boundary localization results on the IMM database.
Using the saturation channel of the HSV color space, a novel eye state analysis
algorithm detected eyes. Using five video sequences, researchers developed an eye
state analysis algorithm that outperforms the traditional approach based on the space
between the eyelids.
Based on eye movement tracking and bio-signal processing, Boon-Giin Lee and Wan-
Young Chung present a technique for monitoring driver safety by evaluating
information linked to weariness. The driver's picture is captured using video sensors,
and the driver's photoplethysmograph signal is collected using a bio-signal sensor. A
monitoring system is built into an Android-based smartphone that gathers sensory
data over a wireless sensor network and analyses it to determine the driver's present
driving ability. The evaluation of driver tiredness is done using a dynamic Bayesian
network architecture.
Tianyi Hong and Huabiao Qin present an image processing methodology for detecting
driver sleepiness in embedded systems based on eye state identification. This
technique uses face detection and eye detection to establish the location of the driver's
eyes, followed by an object tracking method to maintain track of the eyes, and lastly,
PERCLOS is used to identify the driver's sleepiness condition. The findings of the
experiment demonstrate that it agrees well with the analysis.
Fan et al proposed a new Gabor-based dynamic representation for dynamics in face
picture sequences to measure drowsiness. Gabor wavelets were used to extract
multiscale and multi-orientation characteristics for each picture, taking into account
the multi-scale nature of various face actions. Then, using two fusion rules, features of
the same scale were merged into a single feature to extract the local orientation
information. The AdaBoost method was used to identify the most discriminative
features and build a robust classifier for fatigue monitoring. The researchers
experimented on a wide range of human individuals of various genders, postures, and
illuminations real-world fatigue situations. The findings supported the suggested
method's validity and a reassuringly high average accuracy rate was attained

6
 CHAPTER 3: REQUIREMENT ANALYSIS AND
FEASIBILITY STUDY

3.1 Requirement Analysis

While developing a system and before implementing it, it is necessary to analyze the
whole system requirements and it basically defines what is needed to build a system.
Requirement identification includes both functional and non-functional requirements:

3.2.1 Functional Requirement

The functional requirement offered are listed below:

a) The system must be able to get the content video.

b) The system must be able to detect the conditions defined for the alert.

c) The system must be able to generate alerts one or more than one depending
upon the conditions and analysis of the input.

3.2.2 Non-Functional Requirement

A non-functional requirement describes how the system performs a certain function. It

generally specifies the system's quality attributes or characteristics. The non-
functional requirements of the project are listed below:

 The speed of the system is based on CPU or GPU capacity.

 Based on the content function Iteration, the quality of the Video can be
defined.

 The speed of the system is also based on frames of the video input.

7
3.3 Feasibility Study

Feasibility study was done in order to run the project in orderly manner so that it can
be completed with ease and with proper management. Feasibility study has been to
determine the positive or negative outcomes of a project before making any
investments into it.

3.3.1 Technical Feasibility

The system has been developed using python which is well known today and today
we can easily get technical help from the internet. The system development is done in
Pycharm for ease and powerful performance.

3.3.2 Economic Feasibility

We considered economic feasibility based on the resources and financial cost required
for our system to be built. Economic feasibility was the first thing we considered
during the project selection and planning. The availability of required hardware and
software used to develop our project makes it economically feasible. Further, it does
not cost much to develop and access this system and hence, we can say it is
economically feasible to develop the system.

3.3.3 Operational Feasibility

As the system is developed at the place where it is going to be implemented, the track
of the operation related to the software is constantly monitored by them. During the
period of feasibility study and research we studied the operational environment of our
project and calculated the operational feasibility. In this case we studied about the
operational environment and necessary resources available.

3.3.4 Schedule Feasibility

The process of accessing the degree to which the potential time frame and completion
dates for all major activities within a project meet organizational deadlines and
constraints for affecting change.

During the development process, small iterative changes are made in the system.

8
We divided our project time into different groups on the basis of tasks that needed to
be considered to build the system. We gave initial time to the requirement collection
for the system to identify why and how our proposed system could be built. We
considered reasonable time in implementation of our system which requires a lot of
programming for system development. Documentation process was carried out
throughout the system development.

Figure 1: Gannt Chart

9
 CHAPTER 5: DESIGN OF SYSTEM

5.1 System Architecture

The above diagram is the system design of the Drowsiness Detection System. It
details the architecture of the system and the process of working. The users use
dashboard camera for capturing the video of the user’s face and the captured video is
passed for the processing and the video frames is processed for face and mouth
detection. After the face and mouth is detected, the system will calculate the distance
of the eyes and mouth opening. It now compares the condition for the yawn and
closed eyes and iterates the process continuously. If the conditions are meet the
system will shous the alert in the form of sound alarm.

Figure 2: System architecture

10
5.1.1 System Flow Chart

In the flowchart of Drowsiness Monitoring System, the user allows the access
of the webcam to the application. The Real time face is detected through video frame
processing. The application then detects the eyes and mouth, The Euclidean distance
is calculated using the distance between the upper eye lid and lower eyelid and
between upper lip and lower lip. Then the value is compared to the threshold value. If
the value exceeds the threshold value, then the user is alerted.

Figure 3: System Flow Chart

11
5.1.3 DFD

The Driver allows access to the webcam. Then the application extracts the frame
through video framing. After the face is detected, the landmarks are extracted from
the driver’s face frame. 
The eye landmarks are extracted and also the mouth landmarks are extracted. The
landmarks are used to calculate the Euclidean distance between the upper eyelid and
lower eyelid and also upper lips and lower lips. Then Eye Aspect Ratio is
calculated from the extracted value. Threshold is compared with the EAR value and
the decision is taken accordingly. If the EAR value exceeds the threshold value, the
driver is alerted.

Figure 4: DFD Diagram

12
5.1.4 Use Case Diagram

The following diagram is the use case diagram of the working mechanism of the
Drowsiness Detection System. This diagram differentiates the work of user and
Drowsiness Detection. Describing the work of the user there are some very easy tasks
for the user to do. The easiest work of the user is just tracking the condition of the
driver. the user when the user gives the input.

Figure 5: Use Case Diagram

13
 CHAPTER: 5 IMPLEMENTATION AND TESTING

5.1 Algorithm Description

Various algorithms are used for tracking and identification of face and eyes as well as
for detecting drowsiness.

Three types of algorithms are used in the system:

1. Harr-Cascade
2. Eye Aspect Ratio (EAR) Algorithm
3. Practical landmark Detector

5.1.1 Haar-Cascade algorithm

Haar Cascades algorithm is an Object Detection Algorithm used to identify faces in
an image or a real time video which was first introduced in 2001. Now it can be found
almost everywhere for the Face Recognition or object detection. The algorithm uses
edge or line detection features. The algorithm is given a lot of positive images
consisting of faces, and a lot of negative images not consisting of any face to train on
them. Haar-Cascade algorithm is used for face detection and eye tracking which is
followed by EAR to detect drowsiness. Haar characteristics are the most important
component of the Haar Cascade Classifier for face recognition. Haar features are used
to detect feature presence in a given image. Each function yields a single value, which
is determined by adding the pixels in the black rectangle. The Haar-like feature is a
rectangular feature that gives a picture a clear indicator for rapid face detection. Haar-
like attribute value was determined using integral picture to achieve object detection
value. In order to detect the face from an image, it begins searching the image for face
recognition from the top left corner and ends the operation at the right bottom of the
image. The programming is done with python using OpenCV. OpenCV already
contains many pre-trained classifiers for face, eyes, smiles, etc.

The algorithm can be explained in four stages:

 Calculating Haar Features

 Creating Integral Images
 Using Adaboost
 Implementing Cascading Classifiers
14
Haar-Cascade algorithm in drowsiness monitoring system:

Step 1: Load the required XML classifiers.

Step 2: Load our input video in grayscale mode.

Step 3: Find the faces in the image. If faces are found, it returns the positions of
detected faces as Rect (x,y,w,h)

Step 4: Get the locations and create a ROI (Region of interest) for the face and apply
eye detection on this ROI.

Step 5: End

5.1.1.2 EAR algorithm

The aspect ratio of the eye area is also used to measure the temporal accuracy and
speed of left and right eye blinks, and EAR stands for eye aspect ratio.

Dlib library was imported and implemented in our framework to retrieve the facial
landmarks of drivers. A pre-trained face detector is used in the library. The actual
facial landmark predictor was then set up, and the application's facial landmarks were
used to measure distance between points. The EAR value was calculated using these
distances. The equation was used to calculate the EAR, which is defined as the ratio
of the eye's height and width. The numerator represents the eye's height, while the
denominator represents the eye's width, and figure depicts all of the eye's landmarks
in detail. The difference between the upper and lower eyelids is calculated in the
numerator. The horizontal width of the eye is represented by the denominator. When
the eyes are open, the numerator value rises, raising the EAR value; when the eyes are
closed, the numerator value falls, lowering the EAR value. EAR values are used in
this way to detect driver drowsiness. The sum of the EAR values of the left and right
eyes is determined. The Eye Aspect Ratio is tracked in our drowsiness detector to see
if it goes below the threshold value and does not rise above the threshold value in the
next picture.

15
Step 1: Use the Detected Eye region from the above algorithm.

Step 2: Compute the Eye Aspect Ratio to determine if the eyes are closed.

Step 3: If EAR satisfies the drowsy condition, then move to step 5

Step 4: If EAR is normal then go to Step 1.

Step 5: Sound Alarm

5.1.1.3 Practical landmark Detector

On the 300W and AFLW datasets, PLD outperforms all of the algorithms on the
NME metric. It is simple to deploy and allows for fast facial landmark recognition on
a mobile device. This seems to be the only modern neural network-based algorithm
whose creators have shown that it can be used effectively on a mobile device. In
PFLD, MobileNetV2 [2] is used as a function extractor. It has two heads:

 facial landmark regression, which employs a multi-scale fully convolutional

layer at the head's end; and
 3D face model rotation angle estimator (yaw, pitch and roll). The second head
is only seen during preparation and comprises a series of convolutional layers.
Since most datasets lack knowledge about 3D landmark position so to remove
such limitations based on the data in the training collection, create a "mean"
facial representation containing different facial landmarks. Then after
estimate rotation matrix for each of the faces between its and "mean" facial
landmarks. Finally, estimate the rotation matrix for each of the faces between
its and "mean" facial landmarks. From the rotation matrix, calculate the yaw,
pitch, and roll angles. According to the scientists, while such a method isn't
very good at predicting angles, it does increase facial landmark estimation
precision at inference time. Furthermore, the data were weighted based on
image complexity during training using a special loss function. From the
figure below the upper block is only used during preparation and predicts
yaw, pitch, and roll rotation while the lower block estimates the position of
facial landmarks.

16
 Figure 5.1.1.3 : PFLD Architecture

Practical Landmark Detector is implemented using dlib. The dlib facial landmark
detector generates 68 (x, y)-coordinates that correspond to complex facial structures.
These 68 point mappings were obtained by using the named iBUG 300-W dataset to
train a shape predictor. We can see what each of these 68 coordinates corresponds to
by looking at the diagram below.

Figure 5.1.1.3: Facial Coordinate from the iBUG 300-W dataset visualization

17
We can see from the image that facial regions can be reached using basic Python
indexing.

Points [48, 68] have entry to the mouth.

Via points [17, 22], the correct brow.

Via points [22, 27], the left brow.

[36, 42] is used for the right eye.

[42, 48] for the left eye.

Using the nose [27, 35].

[0, 17], as well as the jaw.

These values are stored in dictionaries. We can conveniently extract the indexes into
the facial landmarks array and extract different facial features using this dictionary by
simply providing a string as a key.

5.2 Implementation
Implementation is the phase where the project strategy is put into motion during the
development process. It is the process in which the project efforts are directed to
achieve the project plans and objectives. This project is developed with multiple
algorithms, tools and techniques. The algorithms, methods, tools and techniques used
during the development of the project included in this section.

Drowsiness Monitoring System is implemented using various machine learning

algorithms and techniques. The main algorithm used in Drowsiness Monitoring
System are:

 Haar Cascade Algorithm

 EAR and MAR algorithm
 Practical facial Landmark Algorithm

18
  Drowsiness Monitoring System can be viewed in the following different ways.

 Image Acquisition

 Diving into frames

 Face detection: 

 Eye and Mouth Detection

 Calculate EAR value

 Drowsiness Detection

Image Acquisition:

Real time Video is taken as an input to the Drowsiness Monitoring System when the

user allows the access of the camera as per the requirement of the system. 

Dividing into frames:

We're concerned with a real-time environment in which video is being filmed and

stored. However, only an image can be used to process or apply an algorithm. As a

result, the captured footage must be separated into frames in order to be analyzed.

Face detection: 
In the Drowsiness Monitoring System, Haar Cascade algorithm is used for face
detection. Drowsiness Monitoring System detects the region containing the face of
the driver Face recognition refers to the process of identifying a person's face within
a frame, or in other words, locating facial characters. Only facial structures or
characteristics are observed, with all other artifacts such as houses, trees, and bodies
being overlooked. Haar cascade frontal face detection.xml is used which is used to
detect the region of interest.

Fig of haarcascade

Drowsiness Monitoring System uses Practical Landmark Detector algorithm 

To retrieve the facial landmarks of drivers and to implement Practical Landmark
Detector ,  Dlib library was imported and incorporated into our code. 

19
(Fig of PFLD architecture)

In this library, a pre-trained face detector is used.. The dlib facial landmark detector
generates 68 (x, y)-coordinates that correspond to complex facial structures. These 68
point mappings were obtained by using the named iBUG 300-W dataset to train a
shape predictor. By looking at the facial coordinate , we can see what each of these 68
coordinates means.

Fig of facial coordinate from the ibug 300-W dataset visualization

We can see from the image that facial regions can be reached using basic Python
indexing. Points [48, 68] have entry to the mouth.
Via points [17, 22], the correct brow.
Via points [22, 27], the left brow.
[36, 42] is used for the right eye.

[42, 48] for the left eye.

Using the nose [27, 35]. [0, 17], as well as the jaw.

20
 These values are stored in dictionaries. We can conveniently extract the indexes
into the facial landmarks array and extract different facial features using this
dictionary by simply providing a string as a key

Eye detection:
The application's facial landmarks were then used to determine distance between
points using the Practical facial landmark predictor. These distances were used to
measure the EAR value. 
EAR stands for eye aspect ratio, which is used to calculate the temporal precision and
speed of left and right eye blinks.

Figure 5.2 : Eye with points with eyes close and open

The EAR was calculated using the equation.

¿
EAR=||p 2− p 6||+¿|p 3− p 5|∨ 2∨|p 1− p 4|∨¿ ¿ ¿
Fig: Formula of EAR

21
The eye's height is represented by the numerator, while the eye's width is represented
by the denominator. The numerator value increases when the eyes are open,
increasing the EAR value; when the eyes are closed, the numerator value sinks,
decreasing the EAR value. This is how EAR values are used to diagnose driver
drowsiness. The sum of the left and right eyes' EAR values is calculated.

def eye_aspect_ratio(eye):
A = dist.euclidean(eye[1], eye[5])
B = dist.euclidean(eye[2], eye[4])

C = dist.euclidean(eye[0], eye[3])

ear = (A + B) / (2.0 * C)

return ear

Figure 5.2: Screenshot or Code of Eye Aspect ratio in our project

Mouth Detection

The aspect ratio of the mouth is measured in the similar  way as EAR to determine the
yawning parameter. The mouth aspect ratio (MAR) is measured after the mouth
landmarks have been located to determine if the driver is drowsy or not. The
Euclidean interval between the landmarks is used to measure the mouth aspect ratio.

Figure 5.2: Mouth with points

22
MAR is calculated by using the equation:

MAR=|| p 2− p 8||+|| p 3− p 7||+¿| p 4− p 6|∨ ¿ ¿

2∨|p 1−p 5|∨¿ ¿

Formula of MAR

def lip_distance(shape):
top_lip = shape[50:53]
top_lip = np.concatenate((top_lip, shape[61:64]))

low_lip = shape[56:59]
low_lip = np.concatenate((low_lip, shape[65:68]))

top_mean = np.mean(top_lip, axis=0)

low_mean = np.mean(low_lip, axis=0)

distance = abs(top_mean[1] - low_mean[1])

return distance

Figure 5.2: Screenshot of MAR value calculation in our code

Recognition of eye state:
The Eye Aspect Ratio is tracked in the  drowsiness monitoring system  to see if it
goes below the threshold value and does not rise above the threshold value.
A person's average blink time is 100-400 milliseconds. This is between 0.1 and 0.4 of
a second. As a result, if a person is drowsy, his eye closing must be longer than this..
In the Drowsiness Monitoring System the threshold value is set to 0.3 by default.
Drowsiness is observed when the eyes are closed for five seconds or more.

Recognition of mouth State:

The Mouth Aspect Ratio is tracked in our drowsiness detector to see if it goes below
the threshold value and does not rise above the threshold value. 

Drowsiness Detection:
The algorithm's final step is to decide the person's condition based on a predetermined
drowsiness condition. Then there's EYE_AR_CONSEC_FRAME, which is set to 30
to mean that three consecutive frames with an eye aspect ratio less than
EYE_AR_THRESH are needed in order for a blink to be detected.
The cumulative number of consecutive frames with an eye aspect ratio less than
EYE_AR THRESH is called COUNTER.

23
5.2 Testing

Since the project was built on component-by-component basis each part was
individually tested before integration. Exceptional conditions were handled
beforehand. As an instance, while the person yawns then then the system must detect
and alert the person.
Each specific module was checked during the coding process to ensure that it
functioned properly. The following are some of the most important test cases.
Test case

5.3 Model testing

Test case 1:
EAR

24
 S.N Images EAR Threshold Result remarks

1) 0.20 0.3 Drowsiness Alert Accept

2) 0.32 0.3 Not detected Accept

3) 0.28 0.3 Drowsiness Alert Accept

Test case 2:
MAR value

S.N Images MAR yawn_threshold Result Remarks

25
 1) 7.17 20 Yawn not detected Accept

2) 11.83 20 Yawn alert Accept

3) 9 20 Yawn  not detected Accept

Test case 3:
Alarm 

S.N Images EAR Threshold Result

26
 1) Alarm was buzzed

2) Alarm was not buzzed

3) Alarm was buzzed

5.2.1 Requirement Testing

The main aim for doing Requirement testing is to check stated requirements for
completeness. Testing was made on the requirements on following factors. 
1. Are the requirements Complete?
2. Are the requirements Developable?
3. Are the requirements Testable?

27
5.2.2 Unit Testing
For unit testing each and every functional module is tested and the result is analyzed
to ensure that information flows in and out of the program.

28
 CHAPTER 6: CONCLUSION & FUTURE
ENHANCEMENTS

6.1 Conclusion
Drowsy driving is a serious threat to drivers and traffic participants. The present’s
system lacks one or the other important feature that provides non-reliable results. Our
proposed system will overcome these drawbacks and provide accurate and reliable
results. The general flow of our drowsiness detection algorithm is fairly
straightforward. A camera is setup to monitor stream of faces. After which, we apply
facial landmark detection and extract the eye regions. We can compute the eye aspect
ratio to determine if the eyes are closed. Video segments whose average eye state
point exceeds the threshold value are detected as drowsy and the driver is alerted. The
system can also be used efficiently in locomotives and aero planes. It has a wide
scope in the future and can be improved to meet excellence. In the future, this thesis
will be a part of a safety system being used in vehicles and help us save many lives. In
near future, the project can be improved to detect passenger faces and only focus on
the driver's face. The vehicle manufacturers can make this system inbuilt by using the
dashboard screen and speakers. The system can be effectively used in locomotives
and flights for detecting driver drowsiness. System can be improved to detect and
track eyes even if the driver is wearing shades.

6.2 Future Enhancements

The presented technique of detecting drivers’ drowsiness based on their yawning had
various limitations, which may be addressed as future work. The following
recommendations have been made for future research on the detection of driver’s
drowsiness:

 According to the statistical report from Nepal Police, Information Technology

Directorate, Naxal, Kathmandu, in general, far fewer drowsiness related
baseline epochs were recognized during the daylight hours while a greater
number were observed during darkness; therefore, having a hybrid system that

29
 uses data from both the infrared and visible range will be more useful and
necessary.
 As having access to a database of thousands of yawning faces is not quite
possible, the particular approach that is not based on classifiers is used to
adapt the face detection system in this research. Due to the analysis of
different methods, it is shown that when the system is trained based on a large
number of samples, it will be more accurate. Therefore, training the yawning
detection algorithm will be appropriate.76
 In order to have the option of installing the camera on either the dash of the
car or on the front mirror and also get the good results in both cases, re-
training the Viola-Jones theory for these two datasets will be useful.
 Some drivers cover their mouth while yawning or they have different signs of
sleepiness like eye closure or falling head; in this case, future work may
consist of combining the detection of different fatigue signs.
 The algorithm was tested under two different conditions. 1) In the computer:
on the collection of dataset videos, which was taken in the parked cars as is
described in section 4, in the stable lighting condition and with a Canon
A720Is digital camera. 2) On the APEX board: in this situation, the board with
the camera attached to it was located in the lab under controlled lightning
conditions. For future work, it would be better for the system to be tested with
the APEX board in the real driving scenario in different lighting conditions to
reduce the false detection error in the real time condition.
 In the yawning condition, the rectangle around the candidate mouth changes
from red to green in the computer-based condition; in the car scenario, the
alarm system must alert the driver to wake him or her up in the case of having
a few numbers of yawning condition in the short period of time

30
LIST OF ABBREVIATIONS

EAR: Eye Aspect Ratio

PFLD: Practical Facial Landmark Detector
XML: Extensible Markup Language
OpenCV: Open Source Computer Vision Library
ROI: Region of Interest
CPU: Central Processing Unit
GPU: Graphical Processing Unit

31
 REFERENCE AND BIBLIOGRAPHY
[1] W. Rongben, G. Lie, T. Bingliang and J. Lisheng, "Monitoring Mouth Movement
for Driver Fatigue or Distraction with One Camera," in Proceedings The 7th
International IEEE Conference on Intelligent Transportation Systems, 2004.

[2] R. Jimenez, F. Prieto and V. Grisales, "Detection of the Tiredness Level of Drivers
Using 83 Machine Vision Techniques," in Electronics, Robotics and Automotive
Mechanics Conference , 2011.

[3] Annual Accidental Description, Nepal Police, Information Technology

Directorate, Naxal,
Kathmandu.https://traffic.nepalpolice.gov.np/index.php/news/traffic-activities/425-
annually-accidental-descriptions

[4] Jackson, Abarghouei, Bonner, TobyBreckon, Obara, (2018), Style Augmentation:

Data Augmentation via Style

[5] A. Lanitis, C. Taylor and T. Cootes, "An Automatic Face Identification System
Using Flexible Appearance Models," Image and Vision Computing, vol. 13, no. 5, pp.
393-401, 1995.

[6] Dipesh Khatiwada The Kathmandu Post ” Article on why-are-there-so-many-road-

accidents” 2019/09/26

Hong, T., Qin, H.: Drivers Drowsiness Detection in Embedded System, 1-4244-1266-
8/07/IEEE

Lee, B.-G., Chung, W.-Y.: Driver Alertness Monitoring Using Fusion of Facial
Features and Bio-Signals. IEEE Sensors Journal 12, doi:10.1109/JSEN.2012.2190505

Fan, X., Sun, Y., Yin, B., Guo, X.: Gabor-based dynamic representation for human
fatigue monitoring in facial image sequences. Pattern Recognition Letters 31, 234–
243 (2010)

32
Reference

https://arxiv.org/ftp/arxiv/papers/2101/2101.10808.pdf
https://www.pyimagesearch.com/2017/04/10/detect-eyes-nose-lips-jaw-dlib-opencv-
python/
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3356401
Intermediate Python Project - Driver Drowsiness Detection System with OpenCV &
Keras - DataFlair (data-flair.training)
https://ijrest.net/downloads/volume-5/issue-4/pid-ijrest-54201808.pdf
https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8871097
https://www.abtosoftware.com/blog/fatigue-detection-for-driver-monitoring-systems

33