DDetection of Distracted Driver using Machine learning
Algorithm
IshAayush Anand 1, Ishita Sharma 2, S.Suchitra 3
Department of Data Science and Business Systems, SRM institute of Science and technology, Chennai, India
aa0339@srmist.edu.in1, is8299@srmist.edu.in2, suchitrs@srmist.edu.in 3 .
Abstract- Driving a car is a challenging task that requires
your full attention. Any action that causes the motorist to
lose focus on the road constitutes distracted driving.1.35
million people perish in automobile accidents every year.
Nearly one in five auto accidents, according to a report by
the National Highway Traffic Safety Administration, are the
result of a distracted driver. We want to build a system that
can identify distracted drivers and alert them. In this study,
a machine learning algorithm for recognizing people who
are easily distracted is suggested.
Keywords- Distracted Driver detection, machine learning,
image processing, convolutional neural networks.
1. Introduction
According to a report by the World Health Organization
(WHO), 1.3 million people worldwide pass away in traffic
accidents annually, making them the eighth greatest cause
of mortality, while another 20 to 50 million suffer injuries
or disabilities. Indian roads account for the largest number
of deaths around the world, according to a survey by the
National Crime Research Bureau (NCRB), Government of
India. Since 2006, the number of people killed in traffic
accidents has risen steadily in India. The research also
notes that the overall number of fatalities increased to 1.46
lakh in 2015, with driver error accounting for the majority
of these traffic incidents.
Since a few years ago, there has been an increase in the
incidence of accidents involving distracted driving. In
2015, distracted driving caused 3477 fatalities and
391,000 injuries in motor vehicle incidents, according to
the National Highway Traffic Safety Administration
(NHTSA) [2]. Every day in the US, there are believed to be
9 fatalities and over 1,000 injuries in traffic accidents
where the distracted driver is at fault [1]. The National
Traffic and Highway Safety Administration classifies
distracted driving as "any activity that diverts the attention
of the driver from the task of driving" and divides it into
manual, visual, and cognitive distractions [2] [1]. In its
definitions, the CDC defines cognitive distraction as the
"driver's mind is off the road." In other words, the driver is
physically in a safe driving position, but his mind is off the
road. He might be daydreaming, lost in contemplation, etc.
When a driver's "eyes are off the road," they are
considered to be distracted by visual factors such as
inattention, tiredness, exhaustion, or drowsiness. Various
activities when "driver's hands are off the wheel" are
considered manual distractions.
Distractions include chatting or texting on cell phones,
eating and drinking, conversing with passengers in the
vehicle, changing the radio, applying cosmetics, etc.
Nowadays, Advanced Driver Assistance Systems (ADAS)
are being developed to help avoid accidents by providing
technologies that notify the driver to possible difficulties
and keep the car's driver and occupants safe in the case of
an accident. Even with today's most advanced autonomous
vehicles, the driver must be alert and prepared to regain
control of the car in an emergency. In May 2016, a Tesla
autopilot hit a white truck-trailer in Williston, Florida,
becoming the first fatal collision in autonomous car
testing. Uber's self-driving car, with an emergency backup
driver at the wheel, recently struck and killed a pedestrian
in Arizona. The safe driver in each of these catastrophic
accidents had a chance to avoid the collisions, but the
evidence shows that he was clearly preoccupied. Due to
this, self-driving cars must also have a critical component
that detects inattentive driving. Detecting distracted
driving, in our opinion, is crucial for taking additional
preventive action.
If a vehicle could detect such distractions and warn the
driver, the number of mishaps could be minimized.
In this study, we focus on recognizing manual distractions
in situations when the driver is engaged in jobs other than
safe driving, as well as detecting the source of the
distraction. We provide a Convolutional Neural Network-
based solution to this issue. We also try to decrease
computational complexity and memory requirements while
retaining high accuracy, which is important in real-time
applications.
2. Related Works
The literature on detecting distracted driving is reviewed
in this part in a concise manner. The use of telephones is
the main cause of manual distractions [2].Motivated by
this, several of the researchers worked on detecting mobile
phone usage while driving. Zhang et al. [19] built a
database with a camera positioned above the dashboard
and utilized the Hidden Conditional Random Fields model
to identify mobile phone usage. Basically, it operates
through hand, mouth, and face characteristics. In 2015,
Nikhil et al. [5] developed a dataset for hand detection in
the automobile environment using the Aggregate Channel
Features (ACF) object detector to obtain an average
accuracy of 70.09%. Additionally, Seshadri et al. [14]
produced their own dataset for the identification of cell
phone activity.
The authors employed the Supervised Descent Method, a
histogram of Gradients (HoG), and an AdaBoost classifier
to obtain 93.9% classification accuracy. The system might
run at 7.5 frames per second, which is close to real-time
performance. Le et al. trained Faster-RCNN on the above-
mentioned dataset to obtain 94.2% accuracy, which is
greater than state-of-the-art techniques. To assess mobile
phone usage, they use facial and hand segmentation. For
detecting hands on the wheel and using a mobile phone,
the system may run at 0.06 and 0.09 frames per second,
respectively[7].
Although UCSD's Laboratory of Intelligent and Safe
Automobiles made great progress in this area, they only
addressed three forms of distraction: changing the radio,
changing the mirrors, and using the gear. Martin et al. [8]
introduced a vision-based analysis framework that detects
in-vehicle actions by employing two Kinect cameras to
offer "hands on the wheel" information. Ohn-bar et al.
[12] suggested a classifier fusion in which the picture is
split into three regions: wheel, gear, and instrument panel
to infer real activity. They also provided a region-based
classification method to identify hands in pre-defined areas
of a picture [11]. A model was learned for each region
separately and then merged with a second-stage classifier.
In addition to the previously identified head and hand
signals, the authors' study was expanded [13] to include
ocular cues. However, it only evaluated three types of
distractions.
Zhao et al. devised an additional distracted driving dataset
with a side view of the driver, taking into account four
activities: safe driving, shift lever operation, eating, and
smartphone use. Contour, let transform, and random forest
were used by the authors to obtain 90.5% accuracy. A
system with PHOG and a multilayer perceptron that
produces an accuracy of 94.75% was also suggested by the
authors. Yan et al. [18] reported a Convolutional Neural
Network-based solution with 99.78% classification
accuracy in 2016. The prior datasets focused on a small
number of distractions, many of which are not readily
accessible. In April 2016, State Farm's distracted driver
identification competition on Kaggle defined 10 postures to
be identified (safe driving plus nine distracted behaviors)
[3]. This was the first dataset to consider a wide range of
distractions, and it was made public. Many of the
researchers' techniques were based on conventional hand-
crafted feature extractors like SIFT, SURF, and HoG
mixed with classical classifiers like SVM, BoW, and NN.
However, CNNs turned out to be the most successful
methods for reaching high accuracy [9]. However, under
the rules and regulations, the use of the dataset is limited
to competition purposes only. Abouelnaga et al. [4]
generated a new dataset comparable to State Farm's
dataset for distracted driver detection in 2017. They
prepared the photos by segmenting the skin, face, and
hands before proposing a solution based on a weighted
ensemble of five distinct Convolutional Neural Networks.
The system obtained high classification accuracy, but it is
computationally too complicated to be real-time, which is
critical in autonomous driving.
3. Dataset Description
The dataset produced by Abouelnaga et al. [4] is used in
this work. The dataset is divided into 10 categories: safe
driving, texting on mobile phones with the right or left
hand, chatting on mobile phones with the right or left
hand, changing the radio, eating or drinking, hair and
cosmetics, reaching behind, and talking to passengers.
Figure 1 displays a couple of samples from the dataset for
each class. The data was acquired from 31 people from
seven different nations, using four different automobiles,
and incorporating many permutations of the drivers and
driving circumstances. For instance, drivers are exposed
to a variety of lighting situations, including sunlight and
shadows. 17308 photos in total make up the dataset, which
is split into a training set (12,977) and a test set (4331).
4. Technical Approach
Deep convolutional neural network is a form of Artificial
Neural Network (ANN) that is inspired by the animal's
visual brain. CNNs have made significant progress in a
variety of applications in recent years, including image
classification, object identification, action recognition,
natural language processing, and many more. The
fundamental components of a CNN-based system are the
Convolutional filters/layers, Activation functions, Pooling
layer, and Fully Connected (FC) layer. By adding these
layers one after the other, a CNN is essentially created. The
accession of plenty of labeled data and the computational
power has caused CNNs to advance quite quickly since
2012.
1. VGG-16 Architecture
One of the most important CNN architectures in
literature is the VGG Net. It promoted the notion that
networks should be deep and straightforward. It
performed well in both picture classification and
localization tasks. A 16-layer VGG16 model is used.
Along with zero-padding, dropout, max-pooling, and
flattening, convolutional neural networks form most
of what is used. To get the most out of transfer
learning, we added a few more layers to assist the
model in adapting to our use case. Each layer's
function:
 Only the mean of the values for every patch is utilized
by the global average pooling layer.
 Dropout layers help in overfitting control by cutting
out a portion of parameters (additional tip:
experiment with different dropout values).
 The batch normalization layer normalizes the inputs
to the subsequent layer, allowing for quicker and
more robust training.
 A layer with a certain activation function is called a
dense layer, which is a normal fully-connected layer.
We initialize the pre-trained ImageNet model weights and
then use our dataset to fine-tune each layer of the network.
As a first step, all images are expanded to 224 224, and the
per channel mean of RGB planes is removed from each
pixel of the image. The geometric meaning is that the data
cloud is centered around the origin along each dimension.
The CNN's initial levels function as feature extractors, and
its last layer, the Softmax classifier, assigns each image to
one of the predefined categories. The original model, on
the other hand, has 1000 output channels that correspond
to 1000 Picture Net object types. The last layer thus gets
removed and is replaced with a softmax layer that contains
10 classes.
2. VGG - 16 with Regularization
Experiments with the original VGG-16 network revealed
that the algorithm overfits to the training data. Although it
performs admirably on the training set, achieving a near-
perfect accuracy rate, it is unable to extrapolate to the
untested data. For this reason, we use a variety of
regularization strategies to cut out generalization error.
Furthermore, the LeakyReLU activation function is used
rather than ReLU. The significant differences between the
original VGG-16 are as follows:
• LeakyReLU Activation Function
The Rectified Linear Unit (ReLU) activation function has
gained popularity in recent years due to its efficiency and
speedier convergence. However, because the ReLU
function sets the output value to zero for all inputs less
than zero, the weights of certain neurons may never be
updated, resulting in dead neurons. LeakyReLU solves this
problem by inserting a modest slope in the negative area to
keep the updates alive.
• Dropout
Dropout, which involves disregarding certain neurons
during the training phase at random, is an effective method
of decreasing overfitting [17]. It assists in lessening
neuronal interdependence of learning. We use linearly
increasing dropout in a few convolutional and fully
connected layers.
• L2 Weight regularization
Weight regularization, also known as weight decay,
heavily relies on the unstated presumption that a model
with fewer weights is in some way simpler than a network
with big weights [10]. It is put into practice by explicitly
penalizing each parameter's squared magnitude in the cost
function. We add the term 1 2 w2 to the cost function while
taking into account each weight w in the network, where is
the regularization strength. Hyperparameter 0.001 is
selected for the hyper parameter. • Normalization of
Batches Batch normalization improves the performance
and stability of neural networks by directly pushing
activations across a neural network level to follow a unit
Gaussian distribution [6]. It lessens heavy reliance on
weight initialization, enhances gradient flow across the
network, and permits faster learning rates. The activation
of all convolutional and fully connected layers has been
normalized in the present research.
3. Modified VGG-16
The main problem with VGG-16 is its large amount of
parameters—nearly 140M in total. Fully linked layers
require too much processing work and use the majority of
these parameters. Additionally, the network with a
completely linked layer can only be used with inputs that
have a defined size. Replacing a fully linked layer with a
convolution layer saves parameters and allows for
variable input size [15]. As the outcome, we construct a
completely convolutional neural network by replacing
dense layers with 1-1 convolutions.
Only 15% of the original VGG-16 parameters—or 15M—
are still present in the redesigned network architecture.
Every regularization parameter is the same as it was in the
previous section.
 5. Results and Discussions
In order to identify drivers who are distracted, we develop
an algorithm using convolutional neural networks that
detect distractions. The pre-trained ImageNet model is
utilized for weight initialization, and the transfer learning
idea is used. All network layer weights are modified in
relation to the dataset. All the hyperparameters are
tweaked after extensive experimentation. Stochastic
Gradient Descent is used for the training, with a learning
rate of 0.0001, a decay rate of 106, and a momentum value
of 0.9. The batch size and the total number of epochs are
both set to 64. The NVIDIA P5000 GPU, which has 2560
CUDA cores and 16 GB RAM, is used for training and
testing. Utilizing Keras, the framework is created. When
the initial the VGG-16 is used as a tool for distracted
driver being identified, it achieves an accuracy rate of
100% in the instruction set with 94.44% performance on
the test set. The addition of dropout, L2 weight
regularization, and batch normalization considerably
boosts the system's performance and yields 96.31%
accuracy on the test set. The system typically processes 42
photos per second. In the form of a matrix of confusion,
Table 1 provides a precise and comprehensive statistic for
analyzing the system's outputs. Table 2 shows the accuracy
of every one one of the 10 classes in the dataset.
We show a new architecture with a nearly 90% decrease in
parameters without significantly impacting the accuracy
because the VGG-16 has a high parameter count, and as a
result, a high memory need. On the test set, we achieve
accuracy of 95.54%. The confusion matrices and class-
wise accuracies using the modified VGG-16 architecture
are shown in Tables 3 and 4.
The aforementioned confusion matrices show that postures
associated with "safe driving" and "talking to passenger"
are more frequently confused. The "hands on the wheel"
posture in both classes could be to blame. Additionally,
texting and conversing on a smartphone are sometimes
used interchangeably. Lack of temporal information in the
study may be to blame for this misclassification.
Abouelnaga et al. [4] provide a system of distracted driver
detection and postural categorization that consists of of a
genetically weighted ensemble of five ConvNet. These five
convolutional neural networks are trained using a variety
of picture types, including raw, skin-segmented, hand, face,
and "hands + face" images. On the five imagine resources
stated above, researchers trained the algorithm utilizing
AlexNet or InceptionV3. Because of this method, the system
becomes too complex for real-time use, which is crucial for
self-driving automobiles. Contrarily, we adopt a single
ConvNet, which is less complicated and nevertheless
produces greater accuracy than prior techniques in Table
5.
 6. Conclusion and Future Work
Distracted driving is a severe issue that contributes to
numerous car accidents all around the world. Therefore,
with self-driving cars, the identification of a distracted
driver becomes a crucial system component. Here, we
describe a powerful convolutional neural network-based
system that can both detect and pinpoint the source of a
driver's distraction. We tweak the VGG-16 architecture for
this job and use numerous regularization strategies to
avoid overfitting to the training data. The proposed system
surpasses existing methods of distracted driver
identification from literature on this dataset, as
demonstrated in Table 5, with an accuracy of 96.31%. On
an NVIDIA P5000 GPU with 16GB RAM, the system can
process 42 photos per second. Additionally, we suggest a
streamlined VGG-16 model that uses just 15M parameters
while maintaining acceptable classification accuracy..
The number of parameters and calculation time are being
reduced as a continuation of this effort. Including temporal
information might aid in lowering misclassification
mistakes and boosting accuracy. Additionally, we want to
create a system that can recognize physical distractions as
well as visual, cognitive, and other types of distractions in
the future.
References:-
[1] Center for disease control and prevention. https:
//www.cdc.gov/motorvehiclesafety/distracted_drivig/
[2] National highway traffic safety administration traffic
safety facts. https://www.nhtsa.gov/risky-driving/
distracted-driving/.
[3] State farm distracted driver detection.
https://www.kaggle.com/c/state-farm-distracted-
driver-detection.
[4] Y. Abouelnaga, H. M. Eraqi,and M. N. Moustafa. Real-
time distracted driver posture classification.
CoRR, abs/1706.09498, 2017.
[5] N.Das, E.Ohn-Bar, and M. M.Trivedi. On performance
evaluation of driver hand detection algorithms:
Challenges, dataset, and metrics. In 2015 IEEE 18th
International Conference on Intelligent Transportation
Systems, pages 2953– 2958, Sept 2015.
[6] S.Ioffe and C.Szegedy. Batch normalization:
Accelerating deep network training by reducing internal
covariate shift. In Proceedings of the 32Nd International
Conference on International Conference on Machine
Learning - Volume 37, ICML’15, pages 448–456.
JMLR.org, 2015.
[7] T. H. N. Le, Y. Zheng, C. Zhu, K. Luu, and M. Savvides.
Multiple scale faster-rcnn approach to driver cell-phone
usage and hands on steering wheel detection. In 2016
IEEE Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW), pages 46–53, June
2016.
[8] S. Martin, E. Ohn-Bar, A. Tawari, and M. M. Trivedi.
Understanding head and hand activities and coordination
in naturalistic driving videos. In 2014 IEEE Intelligent
Vehicles Symposium Proceedings, pages 884–889, June
2014
[9] R. P. A. S. Murtadha D Hssayeni, Sagar Saxena.
Distracted driver detection: Deep learning vs handcrafted
features. IS&T International Symposium on Electronic
Imaging, pages 20–26, 2017.
[10] A. Y. Ng. Feature selection, l1 vs. l2 regularization,
and rotational invariance. In Proceedings of the Twenty-
first International Conference on Machine Learning, ICML
’04, pages 78–, New York, NY, USA, 2004. ACM.
[11] E. Ohn-Bar, S. Martin, A. Tawari, and M. M. Trivedi.
Head, eye, and hand patterns for driver activity
recognition. In 2014 22nd International Conference on
Pattern Recognition, pages 660–665, Aug 2014.
[12] E. Ohn-Bar, S. Martin, and M. M. Trivedi. Driver
hand activity analysis in naturalistic driving studies:
challenges, algorithms, and experimental studies. J.
Electronic Imaging, 22(4):041119, 2013.
[13] E. Ohn-Bar and M. Trivedi. In-vehicle hand activity
recognition using integration of regions. In 2013 IEEE
Intelligent Vehicles Symposium (IV), pages 1034–1039,
June 2013.
[14] K. Seshadri, F. Juefei-Xu, D. K. Pal, M. Savvides, and
C. P. Thor. Driver cell phone usage detection on strategic
highway research program (shrp2) face view videos. In
2015 IEEE Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW), pages 35–43, June
2015.
[15] E.Shelhamer, J.Long, and T.Darrell. Fully
convolutional networks for semantic segmentation. IEEE
Transactions on Pattern Analysis and Machine
Intelligence, 39(4):640–651, April 2017.
[16] K.Simonyan and A.Zisserman. Very deep
convolutional networks for large-scale image recognition.
International Conference on Learning Representations.
[17] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever,
and R. Salakhutdinov. Dropout: A simple way to prevent
neural networks from over fitting. J. Mach. Learn. Res.,
15(1):1929– 1958, Jan. 2014.
[18] C. Yan, F. Coenen, and B. Zhang. Driving posture
recognition by convolutional neural networks. IET
Computer Vision, 10(2):103–114, 2016.
[19] X. Zhang, N. Zheng, F. Wang, and Y. He. Visual
recognition of driver hand-held cell phone use based on
hidden crf., pages 248–251, July 2011