KOREA REVIEW OF INTERNATIONAL STUDIES

ISSN - 1226-4741

BEHAVIORAL FEATURES OF DRIVERS FATIGUE

DETECTION AND MONITORING SYSTEM USING CNN
Fahmina Taranum1, Qudisa Tahniyath2, Reshma Nikhat3, Md. Rashid Farooqui4
ABSTRACT

Driver fatigue is one of the leading causes of road accidents every year. These kinds
of road accidents are more likely to result in death or serious injury as they tend to have high
speed impact. Therefore, a driver who has fallen asleep finds it difficult to apply brakes or
swerve to avoid or reduce the impact. It also reduces vigilance, mindfulness and
concentration. Hence, the ability to perform attention-based activities, like driving, is
impaired. The proposed system is an attempt to avoid an accident by monitoring the driver's
eyes using a camera and by developing a prototype, symptoms of driver fatigue can be
detected. The computer vision application presents a real-time fatigue monitoring software
that can automatically detect drowsiness of a person while driving by tracking movement of
eyes, mouth and head tilt using the concept of eye aspect ratio, facial landmarks and head
pose estimation. Therefore, this proposal will be helpful in detecting fatigue in drivers with a
warning. If the system detects drowsy state, then a warning alarm is triggered and a
WhatsApp message is posted to emergency contacts stating that the driver has been detected
drowsy along with live location coordinates weather forecast. An option to automatically put-
off the alarm is made available when the system detects that the driver is alert.

Keywords: Alert Mechanism, Aspect Ratios, Behaviour, Computer Vision, Driver Fatigue,
Facial Landmarks.

I.INTRODUCTION

Humans on a regular basis invent devised technologies and machines, in order to protect
their lives by innovating and advancing the technology. Transportation modes also are built
with the latest technologies for instance the E-vehicles were launched very recently, our
dependency on it increases with a geometric pace. Each one of us is using vehicles for

1
Professor, Computer Science and Engineering Department, Muffakham Jah College of Engineering and Technology, Hyderabad,
Telangana, 500036.
2
Scholar,Computer Science and Engineering Department, Muffakham Jah College of Engineering and Technology, Hyderabad,500036
3
Assistant Professor, Department of Commerce and Business Management, Maulana Azad National Urdu University,Gachi bowli,
Hyderabad,Telangana,500032.
4
Assistant Professor, Department of Commerce and Business Management, Maulana Azad National Urdu University,Gachi
bowli,Hyderabad,Telangana,500032.
40 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

transportation may be individually or by sharing to avoid the pollution and traffic and reach
the destination on time. However, in each case a rule of conduct is required which is violated
in most of the cases irrespective of the social status, one of these rules is to stay alert and
keep the passengers safe and secure. Due to negligence many lives are lost. In order to
prevent a check from destructive outcomes this paper describe about drivers fatigue and
drowsiness detection system just in order to enhance their implementation and further
optimize the solution this paper has been written, Present data shows that in India alone
1,50,7011 people died of accidents in the year 2019 on the road and 25 percent was due to
drivers fatigue, which is very difficult to measure though unlike drug and alcohol
consumption while driving.

Drowsy driving is a high-risk combination of fatigue and driving the vehicle. Drowsiness
in drivers is one of the major reasons for occurrence of road accidents. Fatigue is a driver‘s
significant problem which makes the driver less aware of his/her surroundings. Drowsiness
intimidates road safety and often causes severe injuries and occasionally, it may lead to
fatality of the driver and the companions, it may also result in economic losses. There may be
multiple reasons for a driver to feel drowsy. Some of them are as follows:

 Improper sleep patterns or sleep disorder

 Prolonged driving
 Medications or any health reasons
 Environmental factors like weather conditions of the city/place.
 Boredom or long periods of inactivity. So, the best possible solution to the problem
are understanding about the fatigue related accidents and promoting drivers to admit
fatigue whenever required.
 Government or law does not check; therefore, the drivers feel it easy to un follow the
norms.
Computer vision is a scientific field that researches the science behind artificial systems that
extract information from images. Video sequences, various camera views, or
multidimensional data from a medical scanner are all examples of image data. Computer
vision, as a technological subject, aims to apply its concepts and models to the development
of computer vision systems. Kirabo Jackson and Owensin in [15] provided evidence in
Washington DC that there was little effect of expanded public transit service on DUI arrests,
alcohol related fatal traffic and alcohol related arrests. But influences heterogeneity across
41 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

geographic areas and found bars that are within walking distance to transit stations
experience increases in alcohol related arrests and decreases in DUI arrests.

PROBLEM STATEMENT: To detect and monitor fatigue in drivers based on multiple facial
characteristics detected from the live video feed and to develop an effective alert mechanism
in response to detection so that necessary action can be taken to avoid a probability of
occurrence of a misfortunate event.

OBJECTIVE: The proposed system uses numerous facial characteristics collected from a live
video stream to detect and monitor driver weariness.

The sequence of events in the proposed system is as follows:

i. Face Detection then Facial Feature Extraction based on region of interest by Facial
Land marking.
ii. Eye tracking, Yawn Analysis, Head tilt detection for the purpose of detecting a drowsy
driver.
iii. As a safety measure, playing off the warning alarm when drowsy state is detected as
well as alerting a message on emergency contact with live location coordinates.
II. RELATED WORK

A. Altameem et al. in [1] have proposed an emotion detection method that is based on
Support Vector Machines. Eyes and mouth gestures are detected and calculation is done.
Based on the calculation, features are classified using an SVM classifier. The model was
tested under various illumination scenarios. A. K. Biswal et al. [2] have developed an IoT
based drowsy driver alert system which has an eye blink concept achieved by calculating Eye
Aspect Ratio (EAR) and Euclidean distance of the eye. Based on this calculation, the Internet
of Things module issues a warning message in the form of an e-mail which also consists of
the impact of collision and location information. It also alerts the driver through voice
speaking through the Raspberry Pi monitoring system. A. Gopal et al. [3] have developed a
system to detect drowsiness in drivers by tracking the state of the eye. It takes a facial image
as an input; pre-processing is performed by binarizing the image then removal of noise from
the image takes place. To locate the eyes, intensity changes on the face are found. Drowsiness
is judged on the basis that if the eyes are closed in 5 consecutive frames, then an alarm is
activated in the form of a sound to alert the driver. B. K. Savas et al. in [4] have proposed a
system in which a Multi-tasking Convolutional Neural Network (ConNN) is used to detect
42 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

driver fatigue. As depicted in Table 1, eye and mouth characteristics are used to classify the
degree of fatigue into 3 classes (very tired, less tired and not tired) which is determined by
calculating Percentage of Eye Closure (PERCLOS) and Frequency of Mouth (FOM).

Table 1: Fatigue Classification based on Parameters

Parameters Level of Fatigue

fPERCLOS > 0.24 || fFOM > 0.16 Very Tired
fPERCLOS = 0.15 - 0.24 || fFOM > 0.16 Less Tired
fPERCLOS < 0.15 || fFOM ≤ 0.16 Normal

B. C. Tefft in [5] describes the risk faced, similar to the U.S. government's estimates of the
risk associated with driving while intoxicated, or with a blood alcohol concentration that is
equal to or slightly above the legal limit for alcohol in the U.S., the estimated rate ratio for
crash involvement associated with driving after only 4-5 hours of sleep compared with 7
hours or more is much higher

D. Singh, B. Pati in [6] defines the largest Internet of things (IoT) application
deployment might be considered Mart City. The usage of smartphones, sensors, RFIDs, and
other real-world user interfaces in smart city technology promotes cloud- and IoT-based
services. The IoT ecosystems are unable to effectively connect with one another. So in order
to fill the global common approach, it needs a bridge between them. Information and
communication technologies (ICT) can be used to implement this, which may improve the
performance and interconnectedness of many urban services. It emphasizes the importance of
cloud computing and IoT, both of which have a significant impact on how we develop and
implement smart applications for it.

D. A. Navastara et al. [7] have developed an algorithm to detect fatigue in drivers by

implementing face detection using a Funnel-structured cascade(FuSt), facial feature
extraction using the Uniform Local Binary Pattern(ULBP) and classification using Support
Vector Machines(SVM). Drowsiness was determined by calculating PERCLOS. Along with
this, Head pose was estimated using the Perspective-n-Point problem.

43 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

E. F. Shair, S. A. Ahmad in [8] have discussed One of the methods frequently employed in
the workplace to move or transfer goods is manual lifting. Even if mechanical equipment is
easily accessible today, physical lifting is still seen as a necessary part of material handling
tasks. Musculoskeletal disorders (MSDs), when overexertion contributes as the largest cause,
may be exacerbated by improper lifting techniques. To solve this issue, electromyography
(EMG) signals are utilized to track the health of the employees' muscles and determine the
maximum lifting weight, lifting height, and number of repetitions that the workers can
perform without becoming exhausted. A variety of EMG processing methods and EMG
features that indicate fatigue indices in the time, frequency, and time-frequency domain have
been developed by prior researchers. This research reviews the effectiveness of EMG
processing-based measures in fatigue evaluation during manual lifting. This work is expected
to be of tremendous use to researchers who want a bird's eye perspective of the present
biosignal processing options in order to choose the optimal methods for lifting
applications.The sensitivity, variability, and repeatability of the fatigue indices, as well as the
best processing methods with high efficiency and low computer complexity, are the current
foci of research in the area of EMG signals.

G. Zhenhai et al. [9] have proposed a driver drowsiness detection method based on time
series analysis of steering wheel’s angular velocity. It proposes to use a temporal detection
window to determine the steering wheel angular velocity during a time-series. First, the
steering behavior under the fatigue state is analyzed and recorded, followed by the
determination of the temporal detection window and then, the data series of steering wheel
angular velocity in the temporal detection window is selected as the detection feature for
drowsiness. If the detection feature satisfies certain constraints in the detection window, then
a drowsiness state is detected.

H.M Chandrasena et al. [10] have proposed a driver drowsiness system through iris
detection and by calculating LF/HF ratio. The haar face detection and haar eye tracking
algorithm were used to detect the face and the eye from the input feed. To detect eye open or
eye shut, the Hough circle detection algorithm is used. A fuzzy model based on Heart Rate
Variability (HRV) analysis is used for drowsiness detection.

44 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

Jingwenwu et al [11] has done research on consumer attitudes on electric vehicles and found
that driver fatigue is the most attractive aspect of ACEVs(electric vehicles).. To effectively
manage driving fatigue, a novel problem structure named fatigue-conscious electric vehicle
routing and scheduling problem considering driver heterogeneity (FEVRSPH) is investigated.

J. Hu, L. Xu, X. He, Hui liu and W. Meng in [12] locally developed neural networks and the
real-world vehicle test data are used to create a customized driver model that acts as the
virtual driver for speed control reasons. By using the virtual driver to complete the speed
following task as specified by the common driving cycle test, such as the FTP-72, the driving
behavior is standardized. Three common anomalous driving behaviors—driving when
fatigued or intoxicated, driving recklessly, and using a phone while driving—are described
and reproduced. Based on the examination of normalized driving behaviors, an abnormality
index is suggested and used to objectively assess the abnormality. Numerical tests are run to
confirm the effectiveness of the suggested plan

J. W. Jeong, H. K. Kim explained in [13] explained about

extent to which a given amount of sleep is more (or less) common among drivers who contrib
uted to crashes than among drivers involved in crashes to which they did not contribute reflec
ts the extent to which drivers who had slept for that amount are overinvolved in crashes comp
ared with the reference amount. This is assuming that drivers involved in crashes to which the
y did not contribute make up a representative sample of all drivers on the road.

K. Fujiwara et al. [14] have proposed an algorithm to detect driver drowsiness based on heart
rate variability (HRV) analysis and validates the proposed method by comparing it with
electroencephalography (EEG)-based sleep scoring. RRI data were measured from 34
participants while they were driving and their sleep onsets were determined based on the
EEG data by a sleep specialist.

In [15], It emphasizes the importance of cloud computing and IoT, both of which have a
significant impact on how we develop and implement smart applications for it. Due to its
varied requirements, there are still a lot of challenges with regard to its deployment. The
authors of this study attempted to combine the two fields of cloud computing and IoT for the
deployment of smart city applications. They also presented the types of security threats that
have been found and their mitigation strategies in the context of smart cities.

45 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

In [16],the authors have used deep learning approaches to detect the violation of the norms to
prevent the covid disease using masks.

In [17], clustering technique K-means of machine learning approaches is applied to find the
centroid for creating a cluster to increase the accuracy.

In [18] authors have suggested using high quality cameras to increase the efficiency of the
system.

Muath A. Obaidat in [19] defines IoT as having seen a steady increase in both the number of
devices being used and the variety of applications for which they are employed. Their size,
computing capability, storage capacity, and energy all vary greatly. An Internet of
Vulnerabilities has been produced by the rapid development and integration of IoT in
numerous fields and aspects of our daily life (IoV). This study offers a thorough overview of
IoT with relation to application domains, security architecture frameworks, current IoT
security and privacy challenges, as well as a review of recent related studies on IoT security
and privacy. The article also provides a thorough taxonomy of IoT attacks based on a three-
layer architecture model; these attacks are classified according to their effects on CIA
objectives in typical devices at the perception, network, and application layers.

In [20] Additionally, the study offers mitigations and countermeasures, adopting a

multifaceted approach rather than a per layer approach. Additionally, open research fields are
covered to give researchers access to the most recent research on pressing issues related to
protecting the IoT ecosystem.

Reshma [21] has done research on integrated marketing communication and felt that the IMC
tools helps in decision making therefore it should be used to create awareness to avoid
accidents and loss of lives.

Reshma Nikhat in [22]researched that the conversion of information into digital marketing
like UBER Business Model ,has given direct access to inform about the drivers fatigue and
behavior by rating them.

Rateb et al in [ 23] proposed a method for real-time sleepiness detection is new. This strategy
is based on a deep learning technique that can be accurately applied to Android applications.
This work's key contribution is the transformation of a heavy baseline model into a light

46 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

model. In order to determine whether the driver is sleepy, a simple network structure based
on facial landmark key point detection is built. The suggested model can achieve accuracy
levels of higher than 80%.

M. Belwal et al. in [24] have proposed a driver fatigue detection prototype using Histogram
of Oriented Gradients (HOG) and Support Vector Machines (SVM). The application is
capable of detection and monitoring of the driver’s state using eyes and mouth behavioral
characteristics. Upon detection, the system attempts to alert the driver by playing an alarm.

W. Deng et al. [26] have proposed a system called DriCare which detects driver fatigue by
tracking the eyes and mouth of the driver’s face. A new face tracking algorithm was proposed
named Multiple Convolutional Neural Networks (CNN)-Kernelized correlation filters
(KCF)(MC-KCF), which optimizes the KCF algorithm to improve performance. Eye state is
determined by Convolutional Neural Networks (CNN) and angle of the eye is used to validate
it.

Wanzeng Kong,Lingxiao Zhou et al define in [27]One of the key causes of road accidents
is driver weariness. Based on machine vision and the Adaboost algorithm, we provided an
improved method and workable system to identify driver weariness in this research. Different
types of face and eye classifiers have been thoroughly trained by the Adaboost method. The
suggested technique effectively detects faces by classifying front faces and deflected faces in
the first step. The candidate region of the eye is then selected based on the geometrical
arrangement of the facial organs.

Akhtar and Farooqi, 2022 [28] have developed a real-time yawning detection system which
consists of a face detector, a nose detector, a nose tracker, and a yawning detector. Face and
nose detection is performed using deep Convolutional Neural Networks (CNN). A nose
tracking algorithm is used, which combines Kalman filter with a dedicated open-source TLD
(Track-Learning-Detection) tracker to generate robust tracking results under dynamic driving
conditions. In [29] the authors defined a Final, quick and precise way to identify eyes in the
candidate region, trained classifiers of open and closed eyes are applied. Real-time video
frames are used to extract the PERCLOS and closed-state duration-based indices.
Additionally, the system is integrated into a smart device—a smartphone or tablet—due to its

47 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

built-in camera and robust computation capabilities. The proposed technology can identify
driver weariness with high accuracy and real-time, according to practical tests.

Yimin Zhang, Xianwei Han in [30] extracted the driver's face from the image captured by a
charge coupled device (CCD) camera using an improved face detection approach. An
ensemble of regression trees is then used to determine the feature points for the mouth and
eyes. The algorithm that calculates the enhanced percentage of eyelid closure over pupil over
time is then used to calculate fatigue characteristic parameters. Finally, a fuzzy neural
network is used to assess the condition of the drivers. In order to drastically prevent or reduce
the frequency of traffic accidents, the system can efficiently monitor and remind drivers
about their condition.

Y. Ed-Doughmi, N. Idrissi in [31] defines Driving when fatigued poses a severe risk to traffic
safety in recent years, making fatigue detection research a prominent topic. To increase
traffic safety, fatigue recognition research is extremely important. However, there is still
opportunity for improvement in the efficacy and accuracy of the current fatigue detection
systems. This study suggests a tiredness state identification algorithm to determine whether a
driver is fatigued while operating a vehicle. When a human face is detected using MTCNN
(multitask convolutional neural network), DLIB (an open-source software library) is then
used to discover facial key points and extract the tiredness feature vector from each frame. To
get a final fatigue feature value, a temporal feature sequence with numerous fatigue feature
vectors from different frames is joined together and delivered to the LSTM (long-short-term
memory) network. The fatigue state recognition algorithm presented in this research has,
according to experiments, achieved improved accuracy results when compared to previous
techniques. The proposed approach's average accuracy in identifying the face's important
points is as high as 93%, and its running time is less than half that of the standard DLIB
method.
Mohammed Athar Ali et al., 2023 [32], explained the examination of driver tiredness states,
an algorithm for detecting fatigued driving based on multi-index fusion and a state
identification network has been developed. uses a multi-task cascade convolutional neural
network to detect faces and facial key points. It then corrects the face in accordance with the
key points of the eyes, captures an image of the binoculars to identify the state of the eyes,
and captures an image of the mouth in accordance with the left and right corner points to
identify the state of the mouth. This may increase the accuracy with which the driver's head
48 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

tilt, deflection, and other movements are detected. Next, a mouth state recognition network is
used to identify the mouth state, and an eye state recognition network is built for the
binocular image to identify the eye closure state. In order to further examine the driver's
fatigue state, a fatigue judgment model is created by merging the two traits of the mouth and
eye states

According to the HRV value obtained after calculating LF/HF ratio, the drowsiness levels are
classified into three following categories as shown in Table 2:

Here “X” refers to the output from the system.

Table 2: Fatigue Classification based on LF/HF Ratio

LF/HF ratio Level of Fatigue

X<0.33 Good
0.33≤X<0.66 Observe
0.66≤X Alarm

2.1 SYSTEM ANALYSIS AND DESIGN

2.1.1 Problem with Existing System

In [24], [1], [4],[26],[3] and [11], the existing system can be extended to support more
advanced features, example: a head-tilt detection system can be introduced. In [1], the
proposed work can be tested with an enhanced camera, multiple luminance conditions and
various datasets. In [2], Instead of E-mails, an SMS based service can be introduced which is
more effective. In [4], an alerting system can be introduced to regain the consciousness of the
driver swiftly. In [14], the study includes the properties that are collected in a highly
controlled laboratory environment. It had a limited number of participants and all the
participants were young Japanese persons. The method also requires the drivers to put some
electrodes on the skin before driving to get precise RRI measurement based on ECG which is
needed for HRV analysis. Additional experimental data must be collected to improve
performance and the system should be tested in a real driving environment. In [7], more
features can be added and the system can be developed to work in real-time. In [3], If the
driver moves face closer to or further from the camera, the distances will vary, since the
number of pixels the face takes up also varies. Hence, it is good for an image but not a video.
49 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

In [9], study is limited to only 10 volunteers aged between 25-33 and were in good health
condition. Not implemented real-time as there is a potential danger for driver’s life. In [10],
the input to the system needs to be given manually because the system is not designed to take
ECG data real time. In [11], this single-function system can be upgraded to an advanced
multi-function system.

III. PROPOSED METHODOLOGY

3.1 SYSTEM ARCHITECTURE

Figure 3.4 is a flowchart that depicts the overall process of the system. It covers the
components and the results obtained from each of them.

Figure 3.4: Overall Execution Flow Diagram

3.1.1 ALGORITHM

Input: A live video feed

Output: Detection of drowsy state and activation of alert mechanism

Step 1: Capture the live video feed

Step 2: Divide the video into frames
Step 3: Perform face detection and facial points recognition
50 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

Step 4: Perform facial feature extraction

Step 5: Apply condition to check drowsy eye
If EAR< EyeThreshold
Declare eyes closed
Monitor COUNTER variable
if COUNTER ≥ Eye_Consec_Frames
Declare Drowsy Behaviour
Activate Alarm and Alert Mechanism
else
Declare eyes open
Stop Alarm
update COUNTER to 0
Step 6: Apply condition to determine a Yawn
if MAR>MouthThreshold
Monitor MCOUNTER variable
if MCOUNT ≥ Mouth_consec_frames
Declare yawn detected
update YawnStatus as true
Increment YawnCounts by 1
if YawnCounts ≥ 3
Activate alarm mechanism
else
Stop Alarm
Step 7: Use Euler’s Angle to determine a head tilt
if EulerAngle< -H_thresh or EulerAngle>H_thresh
Declare Head tilt detected
Activate Alarm mechanism
else
Stop Alarm
Step 8: Connect the messaging service API to the program by specifying the
Account ID and Authentication Token
The proposed system uses numerous facial characteristics collected from a live video stream
to detect and monitor driver weariness.
51 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

The sequence of events in the proposed system is as follows:

i. Face Detection followed with Facial Feature Extraction based on region of interest by
Facial Landmarking.
ii. Eye tracking, Yawn Analysis, Head tilt detection for the purpose of detecting a
drowsy driver.

iii. As a safety measure, playing of the warning alarm when drowsy state is detected as
well as alerting a message on emergency contact with live location coordinates and
weather information.

3.2: PROCESS OF FACE DETECTION AND FACIAL FEATURE EXTRACTION

The initial step is to read the input, which is the image from the video feed.

The next step is to extract the visual characteristics needed to detect the driver's drowsiness
and distraction. Facial landmark prediction is used to detect various visual features. Facial
landmarks are used to identify and represent prominent facial features such as the eyes,
brows, nose, mouth, and jawline.

3.2.1 Face Detection:

For face detection and recognition, the application employs the Histogram of Oriented
Gradients (HOG) feature descriptor. The Support Vector Machines (SVM) algorithm is then
used to determine whether the detected object is a face or not.

Dataset Used: A frontal face detector model from the dlib library is used for face detection. It
is a pre-trained model. Multiple datasets were used to train the model. It was trained from the
ground up on a dataset of roughly 3 million faces. The Face Scrub dataset contains over one
lakh face images of 530 people, while the VGG dataset contains 2,622 identities and a large
number of images scraped from the internet.

Computation of the Histogram of Oriented Gradients (HOG): HOG is a feature descriptor

that counts the occurrences of gradient orientation in localized areas of an image. Gradient:
The direction of greatest change in an image is indicated by the gradient. Pixel: It is the
smallest digital image unit capable of being displayed and represented on a display device.

52 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

Process of computing HOG: The process of computing Histogram of Oriented Gradients is

depicted in Figure1

Figure 3.2.1: HOG Computation Architecture

Consider an image of size 498x492

Step 1: Preparing the Data (64 x 128)

Step 2: Determine Gradients in direction x and y (Gx and Gy respectively)

Step 3: Determine Magnitude and Orientation as depicted in equations 1 and 2 respectively

Total Gradient Magnitude =√(𝐺𝑥)2 + (𝐺𝑦)2 (1)

Then, for the same pixel, compute the orientation (or direction)

tan(φ) = Gy / Gx

Therefore, φ= atan(Gy / Gx) (2)

Step 4: Generate histograms based on magnitude and orientation

Step 5: Create a histogram of gradients in 8x8 cells (9x1).

Step 6: Normalize gradients in 16x16 cells (36x1).

Step 7: Create features for the entire image

The dlib library's pre-trained facial landmark detector is used to estimate the location
of 68 (x, y)-coordinates that correspond to facial structures. More than 7,000 photos have
been used to train it. The Frame rate is determined by the type of processor, graphics card and
the refresh rate of the monitor. Frame Rate: It is the rate at which new frames are displayed
53 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

and is expressed in fps (frames per second). Refresh Rate: The number of times a monitor’s
display is refreshed. It is expressed in hertz (hz). The frame rate of the device used is 12fps.

3.2.2 EYE STATE DETECTION

Eye landmarks are identified for each video frame. The eye aspect ratio (EAR) is
calculated as the ratio of the height to the width of the eye as indicated in equation 3.

||𝑝2−𝑝6||+||𝑝3−𝑝5||
𝐸𝐴𝑅 = (3)
2||𝑝1−𝑝4||

As indicated in figure 2 p1, p2, p3, p4, p5 and p6 are 2D landmark locations. When
an eye is open, the EAR is essentially constant, but when the eye is closed, it approaches
zero. Due to the fact that both eyes blink at the same time, the EAR of both eyes is averaged.

Figure 3.2.2: EYE State Detection

3.2.3 YAWN ANALYSIS (Mouth Aspect Ratio Detection)

The yawning of a person is a symptom of fatigue. Yawns can be identified by using

dlib library to calculate Mouth Aspect Ratio (MAR) as indicated in equation 4, using facial
landmarks, similar to how eye aspect ratio is calculated. As indicated in figure 3, p1, p2, p3,
p4, p5, p6, p7 and p8 are 2D landmark locations After identifying the mouth region, MAR
can be used to determine whether the mouth is open or closed. A threshold is set based on
how long it is open and how often the driver yawns, and an alert is set based on that.

||𝑝2−𝑝8||+||𝑝3−𝑝7||+||𝑝4−𝑝6||
𝑀𝐴𝑅 = (4)
3||𝑝1−𝑝5||

54 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

Figure 3.2. 3: MAR detection

3.2.4 HEAD TILT DETECTION

The posture estimation problem is known as the Perspective-n-Point problem, or PNP.

When there is a calibrated camera and the positions of n 3D points on the item and the
corresponding 2D projections in the image are known, the purpose of this challenge is to
discover the pose of an object as indicated in figure 4.

Figure 3.2.4: 3D and 2D Head orientation representation

3.3 WHATSAPP MESSAGE GENERATION USING TWILIO

Twilio's web service APIs enable it to provide programmable communication tools

for making and getting phone calls, sending or receiving text messages, and conducting other
communication activities. Twilio APIs make it simple and reliable to send Text messages.
Integrate WhatsApp and other communication channels such as SMS and MMS using the
Programmable Messaging API.

IV RESULTS AND DISCUSSION

4.1 PERFORMANCE METRICS FOR CLASSIFICATION PROBLEMS

CONFUSION MATRIX
55 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

Confusion matrix is the most straightforward method for assessing the performance of
a classification problem where the output can be of two or more types of classes. A confusion
matrix is simply a table with two dimensions, Actual and Predicted, as well as True Positives
(TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).

Some of the performance measures are represented by the equations 5, 6, 7 and 8

𝑇𝑃
i. 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑃+𝐹𝑃 (5)
𝑇𝑃
ii. 𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑃+𝐹𝑁 (6)

iii. 𝐹1 = 2 ∗ (𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙)/(𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙) (7)

𝑇𝑃+𝑇𝑁
iv. 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = 𝑇𝑃+𝐹𝑃+𝐹𝑁+𝑇𝑁(8)

Figure 4.1: Application Execution Screenshots

Figure 4(a) depicts a scenario where the driver has been detected drowsy and hence a
warning message and alert mechanism has been activated.

Figure 4(b) depicts a scenario in which the driver is seen to be yawning and the yawn
count is also displayed along with a sound alert.

56 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

Figure 4(c) depicts a scenario where the driver’s head is tilted and hence the driver is
alerted with a sound alert.

Figure 4(b) depicts a screenshot of the message that is received upon triggering of the
fatigue detection mechanism. It is a consequence of action depicted in figure 5(a).

Figure 4.2: Testing Screenshots

Fixing the Eye-State Threshold: The application is tested under 3 test cases to
determine the threshold value in the program. The three values are case-1: 0.20, case-2: 0.25
and case-3=0.30. In case-1 & 3, the overall accuracy achieved is 66.67% and 73.33%
respectively. Whereas, in case-2, the accuracy obtained is 93.33% which is the greatest and
hence, the Eye Threshold is chosen to be 0.25 as indicated in figure 4.2(a).

Fixing the Mouth-State Threshold: The application is tested under 3 test cases to
determine the threshold value in the program. The three values are case-1: 0.65, case-2: 0.70
and case-3=0.75. In case-1 & 2, the overall accuracy achieved is 60% and 86.67%
respectively. Whereas, in case-3, the accuracy obtained is 93.33% which is the greatest and
hence, the Mouth Threshold is chosen to be 0.75 as indicated in figure 4.2(b).

Fixing the Head-Tilt Threshold: The application is tested under 3 test cases to
determine the threshold value in the program. The three values are case-1: (-15, 15), case-2:
(-20, 20) and case-3=(-25, 25). In case-1 & 3, the overall accuracy achieved is 66.67% and
73.33% respectively. Whereas, in case-2, the accuracy obtained is 86.67% which is the

57 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

greatest and hence, the Head-Tilt Threshold is chosen to be (-20, 20) as indicated in figure
4.2(c).

V) CONCLUSIONS

In this proposal, numerous techniques for assessing a driver's level of weariness have
been examined. Although there isn't a single definition for drowsiness, its causes have been
explored. The proposed application provides various forms of drowsiness detection by
tracking various behavioral characteristics of the driver. At the beginning, face detection is
done through driver’s video acquisition. Later, this video is then divided into several frames
and through facial landmarking, eye state, mouth state and head tilt state were effectively
detected and monitored. In response to the detections, an alert mechanism is developed which
is capable of raising an alarm in the form of a sound and sending a WhatsApp message with
live location coordinates and live weather information.

REFERENCES

S.No Paper Details

[1]. Ayman Altameem, Ankit Kumar, Ramesh Chandra Poonia, Sandeep Kumar and Abdul
Khader Jilani Saudagar, “Early Identification and Detection of Driver Drowsiness by
Hybrid Machine Learning”, IEEE Access, Department of Computer Science, 2021.
[2]. Anil Kumar Biswal, Debabrata Singh, Binod Kumar Pattanayak, DebabrataSamanta and
Ming-Hour Yang, “IoT-Based Smart Alert System for Drowsy Driver Detection”,
Hindawi -Wireless Communications and Mobile Computing, 2021, Article
ID 6627217. [https://doi.org/10.1155/2021/6627217]
[3]. Athira Gopal and V.V. Vineeth, “Driver Drowsiness Detection System”, International
Journal of Advances in Electronics and Computer Science, 2017.
[4]. Burcu Kir Savaş and Yaşar Becerikli, “Real Time Driver Fatigue Detection System
Based on Multi-task ConNN”, IEEE Access, Computer Engineering Department, Kocaeli
University, Turkey, 2020.
[5]. B. C. Tefft. Acute Sleep Deprivation and Risk of Motor Vehicle Crash Involvement.
Accessed: Jun. 12, 2020. [Online]. Available: https://aaafoundation.org/wp-
content/uploads/2017/12/Acute SleepDeprivationCrashRisk.pdf
[6]. D. Singh, B. Pati, C. R. Panigrahi, and S. Swagatika, “Security issues in IOT and their

58 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

countermeasures in smart city applications,” Advanced Computing and Intelligent

Engineering, pp. 301–313, 2020.
[7]. Dini Adni Navastara, WidheraYoza Mahana Putra and Chastine Fatichah, “Drowsiness
Detection Based on Facial Landmark and Uniform Local Binary Pattern”, Journal of
Physics: Conference Series, JICETS 2019.
[8]. E. F. Shair, S. A. Ahmad,M. H. Marhaban,S. B. Mohd Tamrin,and A. R. Abdullah, EMG
Processing Based Measures of Fatigue Assessment during Manual Lifting, Article
ID 3937254, Hindawi BioMed Research International, 12 pages,2015.
[9]. Gao Zhenhai, Le Dinhdat, Hu Hongyu, Yu Ziwen and Wu Xinyu , “Driver Drowsiness
Detection Based on Time Series Analysis of Steering Wheel Angular Velocity”, IEEE
Xplore, International Conference on Measuring Technology and Mechatronics
Automation, 2017.
[10]. H.M Chandrasena and D.M.J. Wickramasinghe, “Driver’s Drowsiness Detection and
Alarming System”, International Journal of Computer Science and Information
Technology Research, 2016.
[11]. Jingwen Wu, Hua Liao Ph.D, Jin-Wei Wang ,Analysis of consumer attitudes towards
autonomous, connected, and electric vehicles: A survey in China, Research in
Transportation Economics, vol. 80, May 2020.
[12]. J. Hu, L. Xu, X. He, Hui liu and W. Meng,” Evaluating Driving Styles by Normalizing
Driving Behavior Based on Personalized Driver Modeling”, IEEE Transactions on
Systems, Man, and Cybernetics: Systems, vol. 45,no.12, pp 1502-1508 December 2015
https://doi.org/10.1109/TSMC.2015.2417837
[13]. J. W. Jeong, H. K. Kim, Y. J. Lee, W. W. Jung, F. Harashima, and M. H. Lee, ‘‘The
implementation of the autonomous guided vehicle driving system for durability test,’’ in
Proc. IEEE Intell. Transp. Syst., Dearborn, MI, USA, , pp. 101–106, Oct. 2000
[14]. Koichi Fujiwara, Erica Abe, Keisuke Kamata, Chikao Nakayama, Yoko Suzuki,
ToshitakaYamakawa, Toshihiro Hiraoka, Manabu Kano, Yukiyoshi Sumi, Fumi Masuda,
Masahiro Matsuo and Hiroshi Kadotani, “Heart Rate Variability-Based Driver
Drowsiness Detection and Its Validation With EEG”, IEEE Transactions on Biomedical
Engineering, vol. 66, 2019.
[15]. Kirabo Jackson and Emily Owens,One for the road: Public transportation, alcohol
consumption, and intoxicated driving, Journal of Public Economics, vol. 95, no. 1, pp.

59 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

106-121, 2011.
[16]. Mohammad Khalid Imam Rahmani et al., Automatic Real-Time Medical Mask Detection
Using Deep Learning to Fight COVID-19,Computer Systems Science and
Engineering 2022, vol. 42, no. 3, pp. 1181-
1198. https://doi.org/10.32604/csse.2022.022014
[17]. M. K. I. Rahmani, N. Pal and K. Arora, “Clustering of image data using k-means and
fuzzy k-means,” International Journal of Advanced Computer Science and Applications,
vol. 5, no. 7, pp. 160–163, 2014.
[18]. M. Omidyeganeh, S. Shirmohammadi, S. Abtahi et al., “Yawning detection using
embedded smart cameras,” IEEE Transactions on Instrumentation and Measurement, vol.
65, no. 3, pp. 570–582, 2016.

[19].
Muath A. Obaidat, Suhaib Obeidat, Jennifer Holst, Abdullah Al Hayajneh and
Joseph Brown, “ A Comprehensive and Systematic Survey on the Internet of Things:
Security and Privacy Challenges, Security Frameworks, Enabling Technologies, Threats,
Vulnerabilities and Countermeasures, Computers 2020, vol. 9, no. 2,
44; https://doi.org/10.3390/computers9020044
[20]. P. Samangouei, R. Chellappa, M. Najibi, and R. Chellappa, “Face-magnet: magnifying
feature maps to detect small faces,” in Proceedings of WACV, pp.122–130, Lake Tahoe,
NV, USA, March 2018.
[21]. Reshma Nikhat, “An integrated marketing communications, media synergies and its
effect on the consumer decision making process”, 2019,Sumedha Journal of
Management, vol. 8, no. 2, pp. 20-32, 2022.
[22]. Reshma Nikhat ,Digitization And Technology’s Implication On Startups: A Study On
Uber Business Model.,International Journal of Research on Trade and Commerce, vol. 6,
no. 9, Part 2 pp. 11, Jan-Mar 2018
[23]. Rateb Jabbara, Khalifa Al-Khalifa, Mohamed Kharbeche, Wael Alhajyaseen, Mohsen
Jafari, Shan Jiang, Real-time Driver Drowsiness Detection for Android Application
Using Deep Neural Networks Techniques, The 9th International Conference on Ambient
Systems, Networks, and Technologies (ANT 2018), Elsevier,Computer Procedia,2018.
[24]. V. T. Sai Sandeep Raju and Meena Belwal, “Driver Drowsiness Detection”, Springer,
2021.

60 | P a g e
Volume 16 | Special Issue 08 | November 2023
KOREA REVIEW OF INTERNATIONAL STUDIES
ISSN - 1226-4741

[25]. Weihua Tan, Xiaofang Yuan, Xizheng Zhang, Jinlei Wang, An enhanced adaptive
large neighborhood search for fatigue-conscious electric vehicle routing and scheduling
problem considering driver heterogeneity Expert Systems with Applications, vol. 218, 15
May 2023.
[26]. Wanghua Deng and Ruoxue Wu, “Real-Time Driver-Drowsiness Detection System
Using Facial Features”, IEEE Access, 2019.
[27]. Wanzeng Kong,Lingxiao Zhou,Yizhi Wang,Jianhai Zhang,Jianhui Liu,and Shenyong
Gao, A System of Driving Fatigue Detection Based on Machine Vision and Its
Application on Smart Device, Article ID 548602, Journal of sensors, Hindawi,2015,
https://doi.org/10.1155/2015/548602.
[28]. Akhtar, M. A., & Farooqi, R. (2022). 1. Online Consumer’s Purchasing Behavior
Towards E-Grocery Shopping- A Critical Review. Journal of M. P. Institute of Social
Science Research, 27(10), 63–69. https://www.researchgate.net/publication/369014050
[29]. X.Sun,C.Lan,andX.Mao,“Eye Locating Arithmetic In fatigue detection based on image
processing,” in Proceedings of the CISP-BMEI, pp. 1–5, Shanghai, China, October 2017.
[30]. Yimin Zhang, Xianwei Han, Wei Gao and Yunliang Hu, Driver Fatigue Detection
Based On Facial Feature Analysis, International Journal of Pattern Recognition and
Artificial Intelligence, Vol. 35, No. 15, 2150034,
2021.[https://doi.org/10.1142/S0218001421500348]
[31]. Y. Ed-Doughmi, N. Idrissi, and Y. Hbali, “Real-time system for driver fatigue detection
based on a recurrent neural network,” Journal of Imaging, Security and communication
network, vol. 6, no. 3, Article ID 5383573, 9 pages, 2020.
[https://doi.org/10.1155/2021/5383573]
[32]. Mohammed Athar Ali, Wadood Alam, Serajul Haque, & Affan Akhtar. (2023). Impact
And Significance of CRM in Rural Banking -A Comprehensive Study. AMA,
Agricultural Mechanization in Asia, Africa and Latin America, 54(02), 12035–12044.

61 | P a g e
Volume 16 | Special Issue 08 | November 2023