DRIVER DROWSINESS DETECTION SYSTEM USING DEEP LEARNING

A Project Report
Submitted to

MAULANA ABUL KALAM AZAD UNIVERSITY OF TECHNOLOGY, KOLKATA

WEST BENGAL

BY
SAHIN AKHTAR (12500219065)

In Partial fulfilment for the award of the degree of

BACHELOR OF TECHNOLOG
IN
INFORMATION TECHNOLOGY

DEPARTMENT OF INFORMATION TECHNOLOGY

BENGAL COLLEGE OF ENGINEERING AND TECHNOLOGY, DURGAPUR

Durgapur- 713212, WEST BENGAL, INDIA
 CERTIFICATE
This is to certify that the project report entitled “Driver Drowsiness Detection System using deep
learning”, submitted by Sahin Akhtar to MAKAUT, West Bengal in partial fulfillment for the award of
Degree of Bachelor of Technology in Information Technology, a bona-fide record of the project work
carried out by them under my supervision during the year 2022-2023.

-------------------------------------------------------

RINCIPAL, BCET (Prof. (Dr.) P.K. Prasad)

---------------------------------------------------------- --------------------------------------------------

(HEAD OF THE DEPARTMENT, IT) (PROJECT GUIDE)

MR. BIPLAB PALYE Prof. Mr. Rajiv Kumar
 UNDERTAKING REGARDING ANTI-PLAGIARISM

I hereby declare that the material /content presented in the report is free from plagiarism
and is properly cited and written in my own words. In case plagiarism is detected at any
stage, I shall be solely responsible for it.
 ACKNOWLEDGEMENT

We would like to express our special thanks of gratitude to our project guide prof Mr. Rajiv Kumar as
well as our HOD Mr. Biplab Palye who gave us the golden opportunity to do this wonderful project
on the topic Driver Drowsiness Detection System using Deep Learning which also helped us in doing
a lot of research and We came to know about so many new things We are really thankful to them.
Secondly, we would also like to thank Our parents and friends who helped us a lot in finalizing this
project within the limited time frame.
 DECLARATION

We solemnly declare that the project report, Driver Drowsiness Detection System is based on our
own work carried out during the course of our study under the supervision of Mr. Rajiv Kumar Sir.
We assert the statements made and conclusions drawn are an outcome of our research work. We
further certify that:

I. The work contained in the report is original and has been done by us under the supervision of our
supervisor.

II. The work has not been submitted to any other Institution for any other
degree/diploma/certificate in this university or any other University of India or abroad.

III. We have followed the guidelines provided by the university in writing the report.

IV. Whenever we have used materials (text, data, theoretical analysis/equations, codes/program,
figures, tables, pictures, text etc.) from other sources, we have given due credit to them in the
report and have also given their details.
 ABSTRACT

The major aim of this project is to develop a drowsiness detection system by monitoring the eyes; it
is believed that the symptoms of driver fatigue can be detected early enough to avoid a car accident.
In such a case when drowsiness is detected, a warning signal is issued to alert the driver. This
detection system provides a noncontact technique for judging different levels of driver alertness and
facilitates early detection of a decline in alertness during driving. In such a case when fatigue is
detected, a warning signal is issued to alert the driver. In This system we can also add additional
feature of slowing down the vehicle if driver fails to respond to the alarm and ultimately stops the
vehicle.
 INDEX
SL. NO CONTENTS PAGES

1 INTRODUCTION 8

2 PROJECT ARCHITECTURE 9

3 PLANNING OF THE PROJECT 10

4 FUTURE SCOPE OF THE PROJECT 11

5 REQUIREMENT 12

6 TRANSFER LEARNING 13

7 FIGUIRE OF TRANSFER LEARNING 13

8 METHODOLOGY 14

9 DATASET 15

10 DATA SEPARATION 16-17

11 IMPLEMENTATION (MODEL TRAINING) 18-22

12 IMPLEMENTATION OF MAIN MODEL 23-24

13 SOURCE CODE 25-30

14 OUTPUT OF THE PROJECT 31
15 CONCLUTION 32

16 REFERENCES 33
 INTRODUCTION
Nowadays the drowsiness of drivers is one of the main reasons behind road accidents. It is natural for
the drivers who take long drives to doze off behind the steering wheel. In this article, we will build a
drowsiness detection system that will alert the driver as soon as he falls asleep.

Drowsiness is identified by using vision-based techniques like eyes detection, yawning, and nodding.

When it comes to yawning and nodding some people can sleep without yawning and nodding.

One more method is by using physiological sensors like biosensors. Here the disadvantages are like the

driver may hesitate to wear them or he may forget to wear them. Detecting drowsiness through eye

detection is best compared to the remaining techniques.

The objective of this intermediate Python project is to build a drowsiness detection system that will

detect that a person’s eyes are closed for a few seconds. This system will alert the driver when

drowsiness is detected.
 DROWSINESS NO
DETECTION

PROJECT ARCHITECTURE

YES

ALERT(SOUND PLAY)

PLANNING OF THE PROJECT

Now a day’s Machine Learning is playing a huge role in every field, The drowsiness detection proposed

here is a minimum intrusive approach for monitoring driver drowsiness, based on computer vision

techniques, installed on a real car, capable of dealing with real operation conditions. Results obtained

with the system are similar or even better than other commercial ones, being more flexible and open

source. The commercial systems often require a non-trivial calibration procedure, to adjust the

detection. This method is accurate up to 98%. This method of drowsiness detection takes less

computational very less time. Hence, it is very advantages to use this technique in real time applications.

Step 1 – Take image as input from a camera. Step

2 – Detect the face in the image and create a Region of Interest (ROI).

Step 3 – Detect the eyes from ROI and feed it to the classifier.

Step 4 – Classifier will categorize whether eyes are open or closed.

Step 5 – Calculate score to check whether the person is drowsy

 FUTURE SCOPE OF THE PROJECT

1. The system can be made more accurate using various other parameters such as State of the
Car, Detecting Foreign Substances on Face etc.

2. An application can be developed where it can alert or prevent the user from sleeping.

3. It can be used to develop an IOT device that can be installed in the car to detect driver’s
drowsiness. Which is already use by tata motors and tesla.

4. Similar models and techniques can be used for various other uses such as Netflix, Hotstar and
other streaming service platforms can detect whether the person is sleeping and stop the video
accordingly. It may concern security issues.

5. We can develop this project on self-driving cars like tesla such that when driver fall asleep
then activate self-driving mode to park in a safe place and alert to their family members.
 REQUIREMENT

The requirement for this Python project is a webcam through which we will capture images. You need
to have Python (3.6 version recommended) installed on your system, then using pip, you can install
the necessary packages.

1. OpenCV: OpenCV is a great tool for image processing and performing many computer vision
tasks. It is an open-source library that can be used to perform tasks like face detection, object tracking,
and many more tasks. Cmd command pip install OpenCV-python (face and eye detection).

2. TensorFlow: TensorFlow is a free and open-source library, developed by the Google Brain team for
machine learning and artificial intelligence. TensorFlow has a particular focus on the training and
inference of deep neural networks. Cmd command pip install TensorFlow (keras uses TensorFlow as
backend)

3. Keras: Keras is an open-source software library, and it provides a Python interface for artificial
neural networks. Keras is more user-friendly because it is an inbuilt python library. Cmd command
pip install keras (to build our classification model)
4. Pygame : Pygame is a cross-platform set of Python modules designed for writing video games. It
includes computer graphics and sound libraries designed to be used with the Python programming
language. In this project we use Pygame module for play our alert sound to prevent accidents. Cmd
command pip install pygame (to play alarm sound)
 TASK 1

DATA 1 MODEL 1 HEAD PREDICTIONS 1

TRANSFER LEARNING
KNOWLEDGE TRANSFER
TASK 2
In this project, we will use transfer learning to build the model. Transfer Learning is a machine learning

method where we use a pre-trained model for a new model with the related problem statement.
DATA 2 MODEL 1 NEW HEAD PREDICTIONS 2

For example, a model that is used for the recognition of cars can be used for the recognition of trucks.

Here mainly it focuses on the knowledge that is gained while solving one problem and it applies to a
different but related problem.

METHODOLOGY

The methodology of this project is the first video is captured using a webcam and from the video first

face is detected using the Harcascade algorithm and then the eyes are detected. Then we use our deep

learning model which is built using transfer learning to know the status of the eye. If it is an open eye
 FACE & EYES DETECTION

then it will say Active and if it is a closed eye then it will check for a few seconds and then it will say

the driver is drowsy and will beep an alarm.

DEEP LEARNING MODEL
We will use Python, OpenCV, TensorFlow, and Keras to build a system that can detect the closed eyes

of drivers and alert them if ever they fall asleep while driving. If the driver’s eyes are closed, this system
OPED EYES
will immediately inform CLOSED
the driver. OpenCV that we are going to EYES
use now will monitor and collect the

ACTIVE EYES STATUS TIMER DROWSY

driver’s images via a webcam that was attached and feed them into the deep learning model and then

the model will classify the driver’s eyes as ‘open’ or ‘closed.’

ALERT

DATASET

For this project, we are going to use the MRL Eye dataset. MRL Eye dataset is a large-scale dataset that

contains human eye images. Download the dataset using this link. http://mrl.cs.vsb.cz/eyedataset

This dataset contains 84898 eye images of 37 different persons of which 33 are men and 4 are women.

It contains all types of images like open, closed, with glasses, without glasses, and in different lighting

conditions.
After downloading separate all images into closed and open eyes into two separate folders.

DATA SEPARATION

Using this below program to separate the images I use the name of every image which has already
been leveled on it. Like image name - “ s0004_00001_0_0_0_0_0_01” subject ID: xxx
image number: xxx

gender: 0 - male 1 - famale

glasses: 0 - no 1 - yes

eye state: 0 - close 1 - open

reflections: 0 - none 1 - low 2 - high
lighting conditions/image quality: 0 - bad 1 - good
sensor type: 01 - RealSense SR300 640x480 02 - IDS Imaging, 1280x1024 03 - Aptina Imagin 752x480
example: s001_00123_0_0_0_0_0_01.png

Here is the program

Sorce Code: -
Raw_DIR=r'C:\Users\sahin\OneDrive\Desktop\drowsiness_detection\mrlEye_dataset\mrlEyes_2018_
01'
for dirpath,dirname,filenames in os.walk(Raw_DIR):
for i in tqdm([f for f in filenames if f.endswith('.png')]):
if i.split('_')[4]=='0':
shutil.copy(src=dirpath+'/'+i,
dst=r'C:\Users\sahin\OneDrive\Desktop\drowsiness_detection\mrlEye_dataset\Prepared_Data\Close
Eyes')
elif i.split('_')[4]=='1':

shutil.copy(src=dirpath+'/'+i,dst=r'C:\Users\sahin\OneDrive\Desktop\drowsiness_detection\mrlEye_d
ataset\Prepared_Data\Open Eyes')
It will store two deferent types of data opened eyes and closed eyes into two folders which I already
provide the path of those folders.
Here is the screenshot of separation progress: -

After that make two folders for Training and testing purposes with 9:1 ratio, 90% for Training which
is around 73103 images and 10% for testing which is around 7065 images.

IMPLEMENTATION (MODEL TRAINING)

First of all, import required libraries.
import tensorflow as tf
from tensorflow.keras.applications import InceptionV3
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input,Flatten,Dense,MaxPooling2D,Dropout
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator #data
Augmentation
INCEPTION V3: -
Inception v3 is a pre-trained deep learning model that is widely used for image classification tasks. It is used in the driver
drowsiness detection system to classify the driver's eye images as open or closed.

TENSORFLOW: -
TensorFlow is a popular framework of machine learning and deep learning. It is a free and open-
source library which is released on 9 November 2015 and developed by Google Brain Team. It is
entirely based on Python programming language and is used for numerical computation and data flow,
which makes machine learning faster and easier.
TensorFlow can train and run the deep neural networks for image recognition, handwritten digit
classification, recurrent neural network, word embedding, natural language processing, video
detection, and many more. TensorFlow is run on multiple CPUs or GPUs and also mobile operating
systems.

KERAS: -
Keras is an open-source high-level Neural Network library, which is written in Python and is capable
enough to run on Theano, TensorFlow, or CNTK. It was developed by one of the Google engineers,
Francois Chollet. It is made user-friendly, extensible, and modular for facilitating faster
experimentation with deep neural networks. It not only supports Convolutional Networks and
Recurrent Networks individually but also their combination.

Now we generate different types of images out of those images to increase the number of training
images.

CODE: -
train_datagen=ImageDataGenerator(rescale=1./255,rotation_range=0.2,shear_range=0.2,
zoom_range=0.2,width_shift_range=0.2,height_shift_range=0.2,validation_split=0.2)

At first, we rescaled our image which is divided every pixel by 255 of that images so that the image
scale down into a particular scale and split 0.2 means 20% of data for validation.

Like: - rotation_range: - it will make a new image which is rotated by 0.2 means 20%.

width_shift_range: - it will shift by width by 0.2 means 20% and make a new image which is shift
width by 20%.
So here we use 5 features which is make one image to five images so, our training image becomes 5X.
All those images are implemented internally.
Now due to all images are same size that why we have to Resize all the images into 80 x 80 for better
feature extraction.
CODE: -
train_data=train_datagen.flow_from_directory(r'C:\Users\sahin\OneDrive\Desktop\Final_Year_Proje
ct\prepared_data\train',
target_size=(80,80),batch_size=batchsize,class_mode='categorical',subset='training')
validation_data=train_datagen.flow_from_directory(r'C:\Users\sahin\OneDrive\Desktop\Final_Year_
Project\prepared_data\train',

target_size=(80,80),batch_size=batchsize,class_mode='categorical',subset='validation')
Now here we can see total parameters are 21934050 or around 2M, Trainable was 131266 or around
0.5 M and Non-Trainable was 21802784. So, as we see that we use transfer learning that’s we don’t
need train our model from scratch.

Now checkpoint for monitor on validation loss, it will go up to 7 patience if learning rate is not
increase after 3 patience it will reduce Epoch and when our model is successfully trained it will save
our model on the given path.

Now here we use” Adam” optimizer to increase learning rate. (Adam is an adaptive learning rate
optimization algorithm that's been designed specifically for training deep neural networks). As use in
this project categorical class that’s why we use categorical_crossentropy for losses.
Now here we can see that after the 5th Epoch our Accuracy is 0.9431 or around 94% and loss is
0.1471. So, as we know that above of 85% is better accuracy for machine learning project and we get
94% .
IMPLEMENTATION OF MAIN MODEL
Pygame: - Pygame is a Free and Open-Source python programming language library for making
multimedia applications like games. Here I use pygame mixer to play alert sound.

Now here I use the Haar cascade algorithm to detect frontal face and eyes.

Haar cascade Algorithm: -

Haar cascade is an algorithm that can detect objects in images, irrespective of their scale in image and
location. This algorithm is not so complex and can run in real-time. We can train a haar-cascade
detector to detect various objects like cars, bikes, buildings, fruits, etc.

As we can see when we printed our prediction the resultant output was two rows where the first
row for closed eyes which is less than 0.50 and the second row for open eyes which is greater than
0.50. That's why we write if condition” if prediction [0][0]>0.30 it means eyes are closed, then
increase the score variable and play alert sound when it cross 10 score and show closed status
otherwise if prediction [0][1]>0.50 it means eyes are open so decrease the score by 2 and show
opened status.
Now in this screenshot we can see that I start the loop while true that means our model detection is
all time active. It will stop when we press ’q’ which we can see in the next screen shot.

SOURCE CODE

DATA PREPATATION SOURCE CODE:

import os
import shutil
import glob
from tqdm import tqdm
‘’’ subject ID: xxx
image number: xxx

gender: 0 - male 1 - famale

glasses: 0 - no 1 - yes

eye state: 0 - close 1 - open

reflections: 0 - none 1 - low 2 - high
lighting conditions/image quality: 0 - bad 1 - good
sensor type: 01 - RealSense SR300 640x480 02 - IDS Imaging, 1280x1024 03 - Aptina Imagin 752x480
example: s001_00123_0_0_0_0_0_01.png’’’

Raw_DIR=r'C:\Users\sahin\OneDrive\Desktop\drowsiness_detection\mrlEye_dataset\mrlEyes_2018_01'
for dirpath,dirname,filenames in os.walk(Raw_DIR):
for i in tqdm([f for f in filenames if f.endswith('.png')]):

if i.split('_')[4]=='0':
shutil.copy(src=dirpath+'/'+i,
dst=r'C:\Users\sahin\OneDrive\Desktop\drowsiness_detection\mrlEye_dataset\Prepared_Data\Close Eyes')

elif i.split('_')[4]=='1':

shutil.copy(src=dirpath+'/'+i,dst=r'C:\Users\sahin\OneDrive\Desktop\drowsiness_detection\mrlEye_dataset
\Prepared_Data\Open Eyes')

MODEL TRAINING SOURCE CODE:

import tensorflow as tf
from tensorflow.keras.applications import InceptionV3
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input,Flatten,Dense,MaxPooling2D,Dropout
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator #data Agumentation
batchsize=8
train_datagen=ImageDataGenerator(rescale=1./255,rotation_range=0.2,shear_range=0.2,

zoom_range=0.2,width_shift_range=0.2,height_shift_range=0.2,validation_split=0.2)
train_data=train_datagen.flow_from_directory(r'C:\Users\sahin\OneDrive\Desktop\Final_Year_Proje
ct\prepared_data\train',

target_size=(80,80),batch_size=batchsize,class_mode='categorical',subset='training')
validation_data=train_datagen.flow_from_directory(r'C:\Users\sahin\OneDrive\Desktop\Final_Year_
Project\prepared_data\train',target_size=
(80,80),batch_size=batchsize,class_mode='categorical',subset='validation')
test_datagen=ImageDataGenerator(rescale=1./255)
test_data=test_datagen.flow_from_directory(r'C:\Users\sahin\OneDrive\Desktop\Final_Year_Project\
prepared_data\test', target_size=(80,80),batch_size=batchsize,class_mode='categorical')
bmodel = InceptionV3(include_top=False,weights='imagenet',input_tensor=Input(shape=(80,80,3)))
hmodel = bmodel.output
hmodel = Flatten()(hmodel)
hmodel = Dense(64,activation='relu')(hmodel)
hmodel = Dropout(0.5)(hmodel)
hmodel = Dense(2,activation='softmax')(hmodel)
model = Model(inputs=bmodel.input,outputs=hmodel)
for layer in bmodel.layers:
layer.trainable=False
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping,ReduceLROnPlateau
checkpoint=ModelCheckpoint(r'C:\Users\sahin\OneDrive\Desktop\Final_Year_Project\models\model
.h5',monitor='val_loss',save_best_only=True,verbose=3)

earlystop=EarlyStopping(monitor='val_loss',patience=7,verbose=3,restore_best_weights=True)

learning_rate=ReduceLROnPlateau(monitor='val_loss',patience=3,verbose=3)

callbacks=[checkpoint,earlystop,learning_rate]
model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['accuracy'])
model.fit_generator(train_data,steps_per_epoch=train_data.samples//batchsize,
validation_data=validation_data,
validation_steps=validation_data.samples//batchsize,
callbacks=callbacks,
epochs=5)
#Model Evaluation
#tr=training data
acc_tr,loss_tr=model.evaluate(train_data)
print(acc_tr)
print(loss_tr)
#vr=validation data
acc_vr,loss_vr=model.evaluate_generator(validation_data)
print(acc_vr)
print(loss_vr)
acc_test,loss_test=model.evaluate_generator(test_data)
print(acc_test)
print(loss_test)

MAIN SOURCE CODE:

import cv2
import tensorflow as tf
from tensorflow.keras.models import load_model
import numpy as np
from pygame import mixer
face_cascade=cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalcatface.xml')

eye_cascade=cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_eye.xml')

model=load_model(r'C:\Users\sahin\OneDrive\Desktop\Final_Year_Project\models')
mixer.init()
sound= mixer.Sound(r'C:\Users\sahin\OneDrive\Desktop\Final_Year_Project\alarm.wav')
cap = cv2.VideoCapture(0)
Score = 0
while True:
ret, frame = cap.read()
height,width = frame.shape[0:2]
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
faces= face_cascade.detectMultiScale(gray, scaleFactor= 1.2, minNeighbors=1)
eyes= eye_cascade.detectMultiScale(gray, scaleFactor= 1.1, minNeighbors=1)

cv2.rectangle(frame, (0,height-50),(200,height),(0,0,0),thickness=cv2.FILLED)

for (x,y,w,h) in faces:

cv2.rectangle(frame,pt1=(x,y),pt2=(x+w,y+h), color= (255,0,0), thickness=1 )

for (ex,ey,ew,eh) in eyes:

cv2.rectangle(frame,pt1=(ex,ey),pt2=(ex+ew,ey+eh), color= (255,0,0), thickness=1 )

# preprocessing steps
eye= frame[ey:ey+eh,ex:ex+ew]
eye= cv2.resize(eye,(80,80))
eye= eye/255
eye= eye.reshape(80,80,3)
 eye= np.expand_dims(eye,axis=0)
# preprocessing is done now model prediction
prediction = model.predict(eye)

# if eyes are closed

if prediction[0][0]>0.30:
cv2.putText(frame,'closed',(10,height-
20),fontFace=cv2.FONT_HERSHEY_COMPLEX_SMALL,fontScale=1,color=(139,0,0),
thickness=1,lineType=cv2.LINE_AA)

cv2.putText(frame,'Score'+str(Score),(100,height-
20),fontFace=cv2.FONT_HERSHEY_COMPLEX_SMALL,fontScale=1,color=(139,0,0),
thickness=1,lineType=cv2.LINE_AA)
Score=Score+1
if(Score>15):
try:
sound.play()
except:
pass

# if eyes are open

elif prediction[0][1]>0.50:
cv2.putText(frame,'open',(10,height-
20),fontFace=cv2.FONT_HERSHEY_COMPLEX_SMALL,fontScale=1,color=(139,0,0),
thickness=1,lineType=cv2.LINE_AA)
cv2.putText(frame,'Score'+str(Score),(100,height-
20),fontFace=cv2.FONT_HERSHEY_COMPLEX_SMALL,fontScale=1,color=(139,0,0),
thickness=1,lineType=cv2.LINE_AA)
Score = Score-2
if (Score<0):
Score=0
 cv2.imshow('frame',frame)
if cv2.waitKey(33) & 0xFF==ord('q'):
break

cap.release()
cv2.destroyAllWindows()
 OUTPUT
OPEN EYES DETECTION:

CLOSED EYES DETECTION:

CONCLUTION
A non-invasive system to localize the eyes and monitor fatigue was developed. Information about the
eyes position is obtained through self-developed image processing algorithm. During the monitoring,
the system is able to decide if the eyes are opened or closed. When the eyes have been closed for too
long, a warning signal is issued. In addition, during monitoring, the system is able to automatically
detect any eye localizing error that might have occurred. In case of this type of error, the system is able
to recover and properly localize the eyes.
The following conclusions were made:

 Image processing achieves highly accurate and reliable detection of drowsiness.

 Image processing offers a non-invasive approach to detecting drowsiness without annoyance
and interference.
 A drowsiness detection system developed around the principle of image processing judges the
driver's alertness level on the basis of continuous eye closures.

 With 94% accuracy, it is obvious that there are limitations to the system
 REFERENCES
1. Google
2. Wikipedia
3. https://data-flair.training
4. Kaggle
5. World Health Organization. Road Safety: Basic Facts.; 2016
(http://www.who.int/violence_injury_prevention/publications/road_traffic/Road
_safety_m e dia_brief_full_document.pdf).
6. Lee B. G., Chung W. Y.: A smartphone-based driver safety monitoring
system using data fusion. Sensors. 12(12):17536-52 (2012)
7. Li G, Chung WY.: Detection of driver drowsiness using wavelet analysis of
heart rate variability and a support vector machine classifier. Sensors.
13(12):16494-511 (2013)