See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/371481175

Design and Implementation of a Drowsiness Detection System Up to Extended

Head Angle Using FaceMesh Machine Learning Solution

Chapter · June 2023

DOI: 10.1007/978-3-031-34622-4_7

CITATIONS READS

2 120

3 authors, including:

Md. Mahabub Hossain Md. Dulal Haque

Hajee Mohammad Danesh Science and Technology University Hajee Mohammad Danesh Science and Technology University
51 PUBLICATIONS 152 CITATIONS 73 PUBLICATIONS 367 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Md. Mahabub Hossain on 02 August 2023.

The user has requested enhancement of the downloaded file.

 Design and Implementation of a Drowsiness
Detection System Up to Extended Head Angle
Using FaceMesh Machine Learning Solution

Jafirul Islam Jewel , Md. Mahabub Hossain(B) , and Md. Dulal Haque

Department of Electronics and Communication Engineering, Hajee Mohammad Danesh Science

and Technology University, Dinajpur, Bangladesh

Abstract. Drowsiness of drivers is a severe problem for safe driving in

Bangladesh. Drowsiness and fatigue are the major contributing causes of accidents
on the road. In this paper, we proposed a design of a system to detect drowsiness of
drivers up to an extend angle position of the head of the driver while driving, and
notify them. The awakeness of the driver is also detected during driving mode. The
range of the multiple angles of head position using FaceMesh machine learning
solution have been shown for determining the detection drowsiness. Moreover,
different positions of the face are calculated to detect the drowsiness of the driver
with 97.5% accuracy in straight face position, as well as the response time of the
system is calculated that is around 3 s. We have found out that our proposed system
is less complex in design, low cost and performs better than related contemporary
works.

Keywords: Drowsiness Detection · Awakeness Detection · FaceMesh

Algorithm · Machine Learning · Safe Driving

1 Introduction
New technological innovations in the field of system automation and the internet of
things (IoT) are in high demand for the solution of many fundamental issues in our daily
lives for the ease and security of many people’s life activities in various aspects [1–3]. On
long-distance journeys, the driver’s drowsiness is considered one of the major reasons for
vehicle accidents. In particular, on long night journeys in our country, drivers are forced
to drive for a long time restlessly. This results in the tiredness and drowsiness of drivers,
ultimately causing serious accidents. In developed countries, drivers have driven vehicles
for a long time, though they have their own slots for driving. As a consequence, drivers
suffer from drowsiness, which causes unwanted situations [4, 5]. According to the Road
Safety Foundation (RSF) findings, at least 6,284 people were died and 7,468 were bad
wounded in road casualty in Bangladesh in 2021 [6]. The early detection of a driver’s
drowsiness may solution to avoid any possible road accident. Drowsy drivers exhibit
many signs, such as repetitive yawning, repeatedly eye closure, and frequent changing

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2023
Published by Springer Nature Switzerland AG 2023. All Rights Reserved
Md. S. Satu et al. (Eds.): MIET 2022, LNICST 491, pp. 79–90, 2023.
https://doi.org/10.1007/978-3-031-34622-4_7
80 J. I. Jewel et al.

the lanes of the street [7]. For developing countries like Bangladesh, there is a potential
demand to develop a low-cost technology that can be used cost effectively for all kinds
of people for the early detection of drowsiness of drivers. The early stage detection of
drowsiness will prevent accidents significantly caused by it. The main goal is to develop
a system that can observe the driver most accurately as well as efficiently during driving
and can determine in real-time when the driver gets drowsy [8]. With the reading of
facial expressions, eye blinking, and how they blink, a system model can be trained to
determine whether the driver is sleeping or not. The technology uses an algorithm with
a large amount of data for training [9]. With a lot of data about driver facial expressions,
it gives the system an idea about different types of eyes and facial patterns, whether
the driver gets drowsy or not, which is sufficient enough to avoid road accidents due
to driver drowsiness [10]. Some researchers have attempted to address such concerns.
Their methods perform well in their own way, but have several drawbacks. The major
drawback is the failure to detect the driver’s head positions at sufficient angles. Some of
them only detect straight head angles, and their algorithms yield increased power costs.
In this paper,
• We have overcome these limitations by introducing a cost-effective design of a drowsi-
ness detector to detect using the FaceMesh algorithm up to an extended angle position
of the head of the driver while driving and notify them.
• The awakeness of the driver is also detected during driving mode with utilizing low
power and fast response time compared to others.
• We design a less comples and low cost drowsiness detection system.
Some technical challenges are still exist in our proposed system. It is really chal-
lenging task to find out the drowsiness in night or the perfromance may be decrease in
low light. Also, if the condition of the road is not good then the vibrator shows little less
performance, but that is tolarable. Using IR or night vison camera may solve the first
issue easily and driving on the good conditioned highways, vibration problem may be
overcome significatly.
The rest of the paper is sectioned as follows: discussion on some of the relevant works
is presented in Sect. 2. Section 3 represents the proposed design and methodology, Sect. 4
contains the outcome and discussion, and Sect. 5 depicts the conclusion and potentiality
of the system.

2 Related Work
Much work has already been done in this field. Zhu et al. reported a convolution neural
network (CNN) based wearable electroencephalographic (EEG) method for the detec-
tion of drowsiness in divers. They have implemented the warning method by collecting
data from the wearable EEG. A neural network including inception module has a classi-
fication accuracy of 95.59%, while the AlexNet module has a classification accuracy of
94.68% [11]. Drowsiness detection using EEG is highly accurate, though it is intrusive.
Although an eye-tracking-based approach to detection appears appealing, the system,
which consists of a high-speed camera and a complex algorithm, is difficult to imple-
ment. Quddus et al. have proposed a method to use the eye images directly from the
 Design and Implementation of a Drowsiness Detection System 81

camera, omitting expensive eye tracking modules. A recurrent neural network (RNN) is
utilized in their model to detect drowsiness. Two different approaches were introduced in
their study. The first one is the R-LSTM-based approach, which gives 82% accuracy, and
the second is the G-LSTM-based approach, which gives 97% accuracy [12]. A model
consisting of four deep learning (DL) models—Alex Net, ResNet, VGG-Face Net, and
Flow ImageNet—was proposed by Dua et al. Four different features were considered to
find the drowsiness signs of the driver. Their model obtained the accuracy of around 85%
in detecting the drowsiness of a subject [13]. Histogram of Oriented Gradient (HOG)
features were introduced by Bakheet, S. et al. to implement a robust, compact model
to detect the driver’s drowsiness. This approach achieved a detection accuracy of 85%
[14]. Salman et al. introduced Ensemble Convolutional Neural Networks to detect a
driver’s drowsiness. To train the detection model, eye blinking, nodding, and yawning
were taken into account to train the detection model. For the dataset, YawDD was used.
In their study, ECNN achieved an F1 score of 0.935 [15]. A comparison of early studies
is shown in Table 1. Kundinger et al. designed a drowsiness detection model with smart
wearables. This model processes the heart rate of the driver using an integrated machine
learning classifier. The study is also applicable to modern day auto pilot driving systems.
Authors have utilized physiological data which are collected from the wearable wrist
devices for drowsiness detection. For older participants, this model showed superior
results compared to younger participants. Summarizing the work, this model was found
to achieve an accuracy of 82.72% in detecting the driver’s drowsiness [16].

Table 1. Comparison of related literature

Ref. Year Dataset Model Accuracy

[11] Zhu et al 2021 Open EEG based on 95.59%
BCI convolution neural
network
[12] Quddus et al 2021 User Data Recurrent Neural 97%
Network (RNN)
[13] Dua et al 2021 NTHU driver CNN 85%
drowsiness standard
video datasets
[14] Bakheet et al 2021 NTHU-DDD Histogram of 85.62%
Oriented Gradient
[15] Salman et al 2021 YawDD CNN F1 = 0.935
[16] Kundinger et al 2021 NA Integrated Machine 82.72%
Learning
Our proposed system Real time face data MediaPipe Face 97.5%
Mesh
82 J. I. Jewel et al.

3 Methodology
The proposed design of the drowsiness detection (DD) system is represented in Fig. 1(a).
The functional work flow for both the drowsiness as well as yawn detection techniques is
depicted in Fig. 1(b). Facial input is always read in real time by the camera. The FaceMesh
algorithm processes the real-time face data, and the microcontroller and microprocessor
components perform the necessary activities. The microcontroller and microprocessor

(a)

(b)
Fig. 1. (a) Proposed design of the drowsiness detection (DD) system and (b) the functional work
flow.
 Design and Implementation of a Drowsiness Detection System 83

are powered by an external power supply at all times. The Media Pipe FaceMesh machine
learning (ML) function is a 468-face landmark solution that works on the CPU. The
468 facial landmarks are depicted on a map along with their placements on the face.
The filtered landmarks on the face and real-time facial position are displayed on 7"
IPS monitors. The vibration of the motor is controlled by the vibrator motor circuit in
response to signals received from Facemesk. The Sound Box amplifies the related audio
impulses.
The camera collects in real time the driver’s face data. A sound alert in conjunction
with a seat belt vibrator is used; this will activate if drivers keep their eyes closed for at
least 3 s. Figure 2 shows the integration of the system with the driver. The driver will
sit in the driving position, the device will be set in front of the driver, and the alarming
vibrator is integrated with the seat belt.

Fig. 2. Driver view and device setup.

Our camera always takes real-time face video of the driver. Using the FaceMesh ML
solution, our system always generates face landmarks to detect eyes and lip movements.
It can calculate the threshold using generated landmarks. Driver closes his eyes for at
least 3 s, the drowsiness detection function will run and it will give a voice alert alongside
seatbelt vibration. If the driver only yawns, then the yawn detection function will run
and turn on the voice alert, as shown in the Fig. 1(b). On the other hand, if the above
conditions are not fulfilled, the FaceMesh ML solution will run continuously. In this
study, the media pipe face mesh was used. Even on using of different devices, it has 468
real-time 3D facial landmarks.
The solution includes the Facial Transform which is a module that connects the dots
between the estimation of facial landmark and the applications of augmented reality.
It constructs a metric in 3D space and employs the facial landmark screen positions to
estimate a face transform inside that space. Among the common 3D primitives present in
the face transform data, a face position transformation matrix and a triangular face mesh
in important. A lightweight statistical analysis approach, called procrustes analysis is
employed to power the logic, which is robust, performant, and portable. On top of the
84 J. I. Jewel et al.

ML model inference, the analysis runs on the CPU and has a small speed and memory
footprint. We have used a new library named "Media Pipe". Media Pipe brings heavy
AI models to embedded systems and smartphones, utilizing various model optimization
techniques. Media Pipe’s face mesh is a 468-face landmark solution [17]. A map of the
468 facial landmarks with their corresponding positions on Fig. 3.

Fig. 3. The numbers of the facial key points

3.1 Machine Learning Pipeline

The machine learning (ML) pipeline used in this study consists of two deep neural
network models in real-time that cooperate: a detector that computes face positions on the
full image and a model of 3D face landmark which acts on those locations and employs
regression to prognosticate an approximation of the 3D field [18]. Correct cropping of
the face significantly reduces the requirement for typical data augmentations such affine
transformations, which include rotation, translation, and scale modifications. This results
in less consistency in the data and more variation between data points. Additionally, it
permits the network to concentrate most of its efforts on improving coordinate prediction
accuracy. Additionally, facial landmarks detection based cropping was included to our
pipeline in an earlier frame, and the face detector model is only employed to delocalize
the face in cases where the landmark model failed to detect its presence.

3.2 Models Used in This Study

Blaze Face detection model [19], a compact and effective face detector model that
functions well on both Low-end and High-end based processors, has been utilized for
 Design and Implementation of a Drowsiness Detection System 85

face detection. It runs at a pace of 200–1000 + FPS on flagship devices. A lightweight

feature extraction network built on a GPU-friendly anchor technique derived from Single
Shot MultiBox Detector (SSD) and an enhanced tie resolution strategy in place of non-
maximum suppression are used in the model’s backend. [20].
3D facial landmarks are another name for the face landmark model. The 3D landmark
coordinates on synthetic rendered data is predicted by the network while simultaneously
predicting 2D semantic contours on annotated real-world data via transfer learning.
Along with the Face Landmark Model, it is used a model for focusing on the important
facial features. This model requires more computation but predicts landmarks more
accurately around the eyes, irises, and mouth. Among other things, it makes it possible
for AR puppeteering and cosmetics.
The Facial Landmark Model employs a single camera to locate facial landmarks in
screen coordinate space; the Horizontal and Vertical coordinates are conventional screen
coordinates, while the Z coordinate is related to the Horizontal coordinate under the weak
perspective projection camera model. Despite being suitable for some applications, this
format does not support all augmented reality (AR) features, such as matching a virtual
3D object with a recognized face. The Face Transform module transforms a detected
face into a standard 3D object by leaving the screen coordinate space and entering metric
3D space. When the landmark model can no longer detect the presence of a face, it is
intended to be able to project the finished 3D scene back into screen coordinate space
using a perspective viewpoint while keeping the location of the face landmarks.

4 Results
The implementation of the proposed system in terms of experimental setup and hardware
configuration as well as real time output have been described in this section. The proposed
system has been tested in terms of user angle that indicates how many angle the system
cover to takes its input, and distance which measures how far the system captures input
from a driver. Then the response time has been measured.

4.1 Hardware Arrangement and Experimental Setting

The Raspberry Pi 4.0 was employed as the central control unit for the system. In order
to experiment the project, several modules such as a vibrator, camera module, speaker,
and touch display were used. Figure 4 depicts the system’s experimental setup and
implementation. All the components in the setup are connected by different types of
male and female cables. The Fig. 4 shows the setup of our device. It doesn’t need any
extra cable or need not join any cable for the setup. Different parts can be separated and
reassemble easily. It showa the less complexity of design. Most of parts buys from the
local market and price are lower than others available divices, it makes the device less
expensive.
Figure 5(a) represents the different angle of driver head position and yawn of the
driver also. Figure 5(b) shows the measurement procedure of determining the maximum
angle position of the driver. Both eyes of the driver were positioned with the camera at
zero-degree (0°) angle. When the driver turned his head at maximum 80o angle to the
86 J. I. Jewel et al.

Fig. 4. System and Experimental Setup.

right side, the camera could be able to accurately detect at least his left eye. On the other
hand, when the driver rotated his head at maximum 80° angle to the left side, the camera
could be able to accurately detect at least his right eye. The angle cover of the device
based on eyes are tabulated in Table 2.

Table 2. Angle covered by this device.

Angle (Left) Detection Angle (Right) Detection

0o Yes 0o Yes
40o Yes 40o Yes
60o Yes 60o Yes
80o Yes 80o Yes

We trail 20 times for each position and then calculate the mathematical average of
the response time using following equation.
n=20
Ti
Avg = (1)
n
i=1
 Design and Implementation of a Drowsiness Detection System 87

Fig. 5. (a) Different driver angle view of experiment, and (b) measurement procedure.
88 J. I. Jewel et al.

Fig. 5. (continued)

After calculating the average for each position, we get the response time almost 3 s.
Average time for each position is tabulated in Table 3.

Table 3. Response time for different position of driver.

Position Response Time

1 3.02s
2 3.05s
3 3.02s
4 3.03s
5 3.00s
6 3.04s
7 3.02s
Average 3.02S

The average response time is 3.02 s. It means the alarm and vibrators are activated
after detecting the drowsiness of the driver. 3.02 s is not a big time in real world and both
the alarm and vibrator start warning together can make aware the driver about drowsiness.
Driver can get back the safe mode of driving within this few time that ensure the safe
drive in a vehicle. Moreover, the designed system was tested 80 times, where it showed
78 times positive feedback which indicated 97.5% accuracy of the system.
If we can train the model using more videos and images of the drowsiness as well
as more features like body language, facial expression of drivers, which will make the
system more intelligent for the real environment, then the system will response more
quickly and provide more accuracy.
 Design and Implementation of a Drowsiness Detection System 89

5 Conclusions
The proposed design of the system can find the drowsiness of driver during driving mode
in a short time. FaceMesh technique is used to determine the eye and facial expression
of the driver and after calculating the different face point, this system take a decision that
the drowsiness is detected or not. Average 3.02 s response time has been achieved during
several test. It is minimal and a vibrator is response quickly to aware the drive about the
driving mode. Most of the components are connected by cables that give the permission
to separate the parts and reassemble easily. In future, this drowsiness detection system
can be implemented by a cloud-based system and proper monitoring system in integrated
to observe the activities of driver.

Acknowledgement. This study was supported and funded by the Institute of Research and
Training (IRT), Hajee Mohammad Danesh Science and Technology University, Dinajpur-5200,
Bangladesh.

References
1. Rahman, M.F., Hossain, M.M.: Implementation of robotics to design a sniffer dog for the
application of metal detection. In: Proceedings of International Conference on Trends in
Computational and Cognitive Engineering. Springer, Singapore (2021)
2. Kaimujjaman, M., Islam, M.M., Mahmud, R., Sultana, M., Mahabub, M.M.: Digital FM
transceiver design and construction using microcontroller and I 2 C interfacing techniques.
Int. J. Recent Technol. Eng 8(5) (2020)
3. Hossain, M.S., Waheed, S., Rahman, Z., Shezan, S., Hossain, M.M.: Blockchain for the
security of internet of things: a smart home use case using Ethereum. Int. J. Recent Technol.
Eng 8(5), 4601–4608 (2020)
4. Zhang, G., Yau, K.K., Zhang, X., Li, Y.: Traffic accidents involving fatigue driving and their
extent of casualties. Accid. Anal. Prev. 87, 34–42 (2016)
5. Maulina, D., Irwanda, D.Y., Sekarmewangi, T.H., Putri, K.M.H., Otgaar, H.: How accurate
are memories of traffic accidents? Increased false memory levels among motorcyclists when
confronted with accident-related word lists. Transportation research part F: traffic psychology
and behaviour 80, 275–294 (2021)
6. Perilous roads in Bangladesh; 6,284 killed in 2021: RSF, UNB news, published on January
08 (2020)
7. Albadawi, Y., Takruri, M., Awad, M.: A review of recent developments in driver drowsiness
detection systems. Sensors 22, 2069 (2022)
8. Ji, Q., Zhu, Z., Lan, P.: Real-time nonintrusive monitoring and prediction of driver fatigue.
IEEE Trans. Veh. Technol. 53(4), 1052–1068 (2004)
9. You, F., Li, Y.H., Huang, L., Chen, K., Zhang, R.H., Xu, J.M.: Monitoring drivers’ sleepy status
at night based on machine vision. Multimedia Tools and Applications 76(13), 14869-14886
(2017)
10. Dong, Y., Hu, Z., Uchimura, K., Murayama, N.: Driver inattention monitoring system for
intelligent vehicles. A review. IEEE transactions on intelligent transportation systems 12(2),
596–614 (2010)
11. Zhu, M., Chen, J., Li, H., Liang, F., Han, L., Zhang, Z.: Vehicle driver drowsiness detection
method using wearable EEG based on convolution neural network. Neural Comput. Appl.
33(20), 13965–13980 (2021). https://doi.org/10.1007/s00521-021-06038-y
90 J. I. Jewel et al.

12. Quddus, A., Zandi, A.S., Prest, L., Comeau, F.J.: Using long short-term memory and con-
volutional neural networks for driver drowsiness detection. Accid. Anal. Prev. 156, 106107
(2021)
13. Dua, M., Singla, R., Raj, S., Jangra, A.: Deep CNN models-based ensemble approach to
driver drowsiness detection. Neural Comput. Appl. 33(8), 3155–3168 (2021)
14. Bakheet, S., Al-Hamadi, A.: A framework for instantaneous driver drowsiness detection based
on improved HOG features and naïve Bayesian classification. Brain Sci. 11(2), 240 (2021)
15. Salman, R.M., Rashid, M., Roy, R., Ahsan, M.M., Siddique, Z.: Driver Drowsiness Detection
Using Ensemble Convolutional Neural Networks on YawDD. arXiv preprint arXiv:2112.
10298 (2021)
16. Kundinger, T., Riener, A., Bhat, R.: Performance and acceptance evaluation of a driver
drowsiness detection system based on smart wearables. In: 13th International Conference
on Automotive User Interfaces and Interactive Vehicular Applications, pp. 49–58 (2021)
17. Singh, A.K., Kumbhare, V.A., Arthi, K.: Real-time human pose detection and recognition
using MediaPipe. In: International Conference on Soft Computing and Signal Processing,
pp. 145–154 (2021)
18. Feng, Y., Wu, F., Shao, X., Wang, Y., Zhou, X.: Joint 3d face reconstruction and dense
alignment with position map regression network. In: Proceedings of the European conference
on computer vision (ECCV), pp. 534–551 (2018)
19. Brar, D.S., Kumar, A., Mittal, U., Rana, P.: Face detection for real world application. In:
2021 2nd International Conference on Intelligent Engineering and Management (ICIEM),
pp. 239–242 (2021 Apr 28)
20. Bazarevsky, V., Kartynnik, Y., Vakunov, A., Raveendran, K., Grundmann, M.: BlazeFace.
Sub-millisecond Neural Face Detection on Mobile GPUs. arXiv:1907.05047 (2019)

View publication stats