Person Authentication for Restricted Areas using Number Plate

Detection and Face Recognition.

Prof. Rina Damdoo, Kush Munot, Varsha Valecha, Gaurav Pathak, Sagar Mandal
Shri Ramdeobaba College Of Engineering and Management

Abstract
The ”Number Plate Detection with Face Recognition” project in India is dedicated to over-
coming the challenges in vehicle and individual identification for enhanced security. Utiliz-
ing Pytesseract and OCR, our innovative solution enhances number plate detection accuracy
through advanced algorithms and image processing techniques. Moreover, it incorporates a
specialized face recognition library optimized for vehicle environments, employing 128 facial
parameters to ensure precise identification. This integrated approach promises to revolution-
ize security measures and law enforcement efforts by addressing existing limitations, offering a
seamless and efficient solution for tracking and identifying individuals associated with vehicles.
In summary, our project aims to redefine vehicle identification and security through the fusion
of cutting-edge number plate detection and face recognition technologies. In summary, the pro-
posed work aims to redefine vehicle identification and security through the fusion of cutting-edge
number plate detection and face recognition technique.

1 Introduction
In India, the identification of vehicles and individuals plays a crucial role in maintaining security and
addressing various concerns. However, existing systems often encounter challenges in accurately
detecting number plates and recognizing faces, leading to inefficiencies and potential security risks.

To tackle this problem, the proposed work in India aims to develop a robust technique using the
Pytesseract and OCR for number plate detection, along with a face recognition library specifically
designed for faces captured inside vehicles. Without this proposed work, the process of identifying
vehicles and individuals is plagued by shortcomings. Existing methods for number plate detection
may suffer from low accuracy, making it difficult to extract relevant information effectively. Addi-
tionally, traditional face recognition approaches are often not optimized for capturing faces within
a vehicle environment, which can result in unreliable identification.

By leveraging advanced algorithms and image processing, our system can accurately extract
and interpret number plate information from vehicle images. Furthermore, the proposed work in-
cludes a specialized face recognition library that has the capability to process and compare facial
features within the confined space of a vehicle. By combining these techniques, the proposed work
aims to offer an integrated solution for seamless number plate detection and face recognition within
the context of vehicles. This advancement will significantly enhance security measures, improve
law enforcement efforts, and enable efficient tracking and identification of individuals associated

1
with vehicles. Overall, the proposed work seeks to overcome the limitations present in conventional
approaches and strives to revolutionize the field of vehicle identification and security by integrating
advanced number plate detection and face recognition techniques.

The need for this integrated solution arises from the critical importance of accurate vehicle
and individual identification for security and law enforcement purposes. Existing systems often
struggle with low accuracy in number plate detection and face recognition, hindering their effec-
tiveness in addressing security concerns. The proposed work addresses these challenges by utilizing
advanced algorithms and specialized libraries to enhance accuracy in both number plate detection
and face recognition, ensuring reliable identification of individuals associated with vehicles. This
solution promises to streamline security measures and significantly improve law enforcement efforts,
ultimately contributing to a safer and more secure environment.

2 Dataset Curation
We curated our custom data set by capturing videos of moving vehicles and under various environ-
mental conditions. We followed a process for the custom data set, we went with our friends and a
car to places and took videos in different conditions to test our model. A data set of 10 videos was
curated. These videos capture moving cars from various angles. The goal is to ensure that the face
of the driver is visible through the car’s glass, and that the number plate is also visible within the
same video frame.

Figure 1: This picture shows the workflow diagram of data set curation.

Variability: The videos were captured under different conditions, including varying distances,
different intensities of light, and at various times of the day. This variability helps the model learn
to recognize vehicles and their occupants in diverse real-world scenarios.

Usage: The data set was used for both training and testing purposes. During training, the
model learned to recognize and extract relevant information from the videos. During testing, the
accuracy of the trained model was assessed, possibly using metrics like classification accuracy or

2
object detection metrics.

Purpose: The purpose of this data set and model is related to tasks such as driver identi-
fication, vehicle recognition, or license plate recognition. It could also be used for security and
surveillance applications, traffic monitoring, or autonomous driving systems.

3 Literature review
In recent years, several notable advancements have been made in the field of automatic number
plate recognition (ANPR) and security systems. These studies have explored various approaches
and technologies to enhance accuracy and efficiency in identifying vehicles and individuals.

Table 1: Summary of Research Papers on Vehicle Number Plate Recognition (Part 1)

Ref No. Title
1 A Review Paper on Vehicle Num-
ber Plate Recognition (Gurav
C, Gurav R, Kamble V, Prof.
Sakhalkar N, Mohite S)
2 Automatic Number Plate Recog-
nition using Deep Learning (V
Gnanaprakash, N Kanthimathi,
N Saranya)
3 Exhaustive Security System
Based On Face Recognition In-
corporated With Number Plate
Identification Using Optical
Character Recognition (Zaheen
Y, Mohi-U-Din Z, Siddique A,
Qadri T)
4 An Automatic Number Plate
Recognition System using
OpenCV and Tesseract OCR
Engine (Agbemenu A, Yankey
Jepthah, Addo E)
5 Vehicle Number Plate Detection
and Recognition using Bounding
Box Method (K. M. Babu and M.
V. Raghunadh)
6 Implementation of Viola-Jones
Algorithm Based Approach for
Human Face Detection (Soni,
L.N., Ashutosh, D., and Shilpa
D)
Year Key Features Accuracy
2019 Enhanced ANPR using OpenCV 95%
and Tesseract OCR, improved
results compared to Matlab, Ro-
bustness to various lighting con-
ditions
2020 Deep learning-based ANPR sys- 98%
tem with high accuracy, Im-
proved recognition under chal-
lenging weather conditions
2020 Integration of face recognition 83%
and number plate identification,
achieving 83% person identifica-
tion accuracy, Enhanced security
through multi-factor authentica-
tion
2018 ANPR with high accuracy in 97.40%
edge detection, feature detection,
and OCR, Real-time recognition
of license plates
2016 Precise vehicle number plate de- 94%
tection and recognition using the
bounding box method, Effective
vehicle tracking for traffic man-
agement
2017 Exploration of the Viola-Jones 96%
algorithm for real-time human
face detection, Robust face de-
tection for surveillance applica-
tions

3
 Table 2: Summary of Research Papers on Vehicle Number Plate Recognition (Part 2)
Ref No. Title
7 A Fast and Accurate Uncon-
strained Face Detector (Liao, S.,
Anil, K. J., and Stan Z. L.)
8 Detection of License Plates in
Natural Scenes with MSER and
SIFT Unigram Classifiers (W. L.
Hao and H. T. Yong)
9 Safety and Security Improve-
ment in Public Transportation
Based on Public Perception in
Developing Countries (T. B. Joe-
wono and H. Kubota)
10 A Convolutional Neural Net-
work Cascade for Face Detection
(Li, H., Zhe, L., Xiaohui, S.,
Jonathan B., and Gang, H)
11 Vehicle License Plate Detection
From Edge Detection And Mor-
phological Operators (Harpreet
Kaur, M. B.)
12 A Review Paper On Vehicle
Number Plate Recognition Us-
ing Improved Character Segmen-
tation Method (Bailmare, M. H.)
13 Automatic Licence Plate Recog-
nition Using Raspberry Pi (Aish-
warya Agrawal, N. P.)
14 Detecting The Car Number
Plate Using Segmentation (V.
Laksmi Priya, K. Perumal)
15 An Automatic System For De-
tecting The Vehicle Registra-
tion Plate From Video In Foggy
And Rain Environments Using
Restoration Technique (Savneet
Kaur, Kamaljit Kaur)
In the field of vehicle identification and security, the proposed work in India differs significantly
Year Key Features Accuracy
2016 Introduction of a rapid uncon- 98.50%
strained face detector, crucial
for advancing facial recognition
technology, Enhanced accuracy
and speed in face detection
2010 Discussion of license plate de- 92%
tection in natural scenes using
MSER and SIFT unigram clas-
sifiers, techniques commonly em-
ployed in object recognition
2006 Focusing on enhancing public N/A
transportation safety and secu-
rity in developing countries, con-
sidering public perceptions to
improve transport network secu-
rity, Improved public transporta-
tion safety and security based on
community feedback
2015 Introduction of a CNN cascade 97%
for face detection and deep learn-
ing for biometrics and image
analysis, Enhanced face detec-
tion accuracy in a variety of sce-
narios
2012 License plate detection using 90%
edge detection and morphologi-
cal operators, Enhanced robust-
ness to image noise
2013 Review paper on ANPR with 88%
improved character segmenta-
tion, Improved recognition of
distorted characters
2017 Automatic license plate recog- 94%
nition using Raspberry Pi, En-
hanced portability and real-time
recognition
2014 Car number plate detection 91%
using segmentation, Improved
recognition under varying light-
ing conditions
2014 Automatic system for detect- 93%
ing vehicle registration plates in
foggy and rainy environments us-
ing restoration techniques, Im-
proved performance in adverse
weather conditions
4
from the body of existing literature in several key areas. Even though earlier research has been
very beneficial, it frequently concentrates on particular areas, like face or number plate recognition,
without offering a comprehensive answer. Our work proposes, on the other hand, a comprehensive
solution that seamlessly integrates both face recognition and number plate detection within the
context of vehicles, using specialised libraries and advanced algorithms. The precise identification
of faces inside vehicles is ensured by this integrated approach, which is specifically designed to meet
the unique challenges involved. It tackles issues with conventional methods’ shortcomings, like
their poor accuracy in identifying licence plates and their inability to be tailored to the demands of
moving cars. By doing so, the proposed work promises to revolutionize security measures, enhance
law enforcement efforts, and significantly improve the tracking and identification of individuals
associated with vehicles, contributing to a safer and more secure environment in India.

4 Model Description
Person authentication for restricted areas using number plate detection and face recognition has
two phases: vehicle number plate recognition and face recognition. For number plate recognition,
we are using the Pytesseract library, which is a commonly used optical character recognition (OCR)
tool in Python. We are also using the ”Face Recognition” library for face recognition from images.
It is the world’s simplest face recognition library, with an accuracy of 99.38% on the labeled faces
in the wild benchmark.

Vehicle Number Plate Recognition: In the proposed model, the images are generated from
the input video. The image is read and resized to maintain its aspect ratio for data conservation.
The processed image is converted into grayscale, and the number plate area is bound using the cas-
cade classifier loaded earlier. If a valid license plate is found, the characters on the license plate are
extracted using Tesseract OCR. The data is appended to the file for later use. If a license plate is not
found in the current image, then the model uses other generated images for number plate detection.

Face Recognition: The library first detects faces within an image or video frame. It employs
pre-trained models, such as Haar cascades or deep neural networks, to identify face regions. Once
a face is detected, the library identifies facial landmarks. These landmarks include points like the
eyes, nose, mouth, and face edges. Landmark detection is typically achieved using shape predictor
models, often included in the library’s resources. The library extracts key features from the face,
such as the distance between eyes, nose shape, and mouth curvature. These features are used to
create a unique face representation. Libraries like dlib use a face recognition model that compares
the extracted features from the detected face with known faces stored in a database or directory.
To recognise faces, a library maintains a database of known individuals with their corresponding fa-
cial feature representations. When a new face is detected, its features are compared to the database.

A distance metric, such as Euclidean distance or cosine similarity, is used to measure the sim-
ilarity between the detected face’s features and the features of known faces. A lower distance
indicates a higher degree of similarity. To make a positive identification, the library uses a recogni-
tion threshold. If the distance between the detected face and a known face is below this threshold,
the system recognises the person. If the detected face closely matches a known face based on the
distance calculation and falls below the recognition threshold, the library identifies the person. The

5
recognised individual’s name or ID is associated with the detected face.

Figure 2: This picture shows the workflow diagram of the Model and what are the intermediate
steps involved in the model for recognizing Number Plates and Face recognition.

After identifying both the number plate and the face, the data is mapped together. It is then
compared with the existing data from the database. If the match is found, entry is permitted in a
restricted area. If no match is found, the entry is denied, and an alarm is set for physical inspection.

5 Results

Figure 3: Output Image 1Figure 4: Output Image 2Figure 5: Output Image 3

6
Figure 6: Output Image 4 Figure 7: Output Image 5 Figure 8: Output Image 6

Figure 9: Output Image 7 Figure 10: Output Image 8 Figure 11: Output Image 9

7
 Figure 12: Confusion matrix after running the videos on our custom-trained model.

6 Conclusion
Convolutional Neural Networks (CNN) has been used for number plate detection and character
recognition to segment the characters contained in number plates. By using our model, individuals
can efficiently identify number plates, and accurately recognize the faces of individuals within
a vehicle using frames captured from a single video. This integrated solution enhances security
measures and access control for restricted areas while simplifying processes that would otherwise
be time-consuming and manual. Even after being trained on only 87 images, the suggested model
had a testing accuracy of 85%. However, by using a suitable rooftop setup for recording vehicles
as they enter, adjusting the lighting, and choosing the right camera angle, further improvement
can be made. Our proposal will result in a system that aids security personnel and authorities in
effectively implementing number plate detection and face recognition for restricted areas, including
government offices and other secure locations. With the help of this initiative, security professionals
will be able to create a comprehensive database that includes information on individuals entering
and leaving the area. This database will be available to officers in several states, furthering the
digitization of the security industry.

8
7 Acknowledgements

References
[1] C. Gurav, R. Gurav, V. Kamble, N. Prof. Sakhalkar, S. Mohite, ”A Review Paper on Vehicle
Number Plate Recognition,” International Journal of Engineering Research and Technology
(IJERT), vol. 8, issue 04, 2019.

[2] V. Gnanaprakash, N. Kanthimathi, N. Saranya, ”Automatic number plate recognition using

deep learning,” IOP Publishing Ltd, volume 1084, 2020.

[3] Y. Zaheen, Z. Mohi-U-Din, A. Siddique, T. Qadri, ”Exhaustive Security System Based On

Face Recognition Incorporated With Number Plate Identification Using Optical Character
Recognition,” Mehran University Research Journal of Engineering and Technology, vol. 39,
no. 1, pp. 145-152, 2020.

[4] A. Agbemenu, J. Yankey Jepthah, E. Addo, ”An Automatic Number Plate Recognition System
using OpenCV and Tesseract OCR Engine,” International Journal of Computer Applications,
volume 180, no. 43, May 2018.

[5] K. M. Babu and M. V. Raghunadh, ”Vehicle number plate detection and recognition using
bounding box method,” In 2016 International Conference on Advanced Communication Con-
trol and Computing Technologies (ICACCCT), pp. 106–110, May 2016.

[6] L.N. Soni, D. Ashutosh, Shilpa D., ”Implementation of Viola-Jones Algorithm Based Approach
for Human Face Detection,” International Journal of Current Engineering and Technology, vol.
7, no. 5, pp. 1819-1823, 2017.

[7] S. Liao, A.K. J., Z.L. Stan, ”A Fast and Accurate Unconstrained Face Detector,” IEEE Trans-
actions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 211-223, 2016.

[8] W.L. Hao, H.T. Yong, ”Detection of license plates in natural scenes with MSER and SIFT
unigram classifiers,” In IEEE Conference on Sustainable Utilization and Development in En-
gineering and Technology, pp. 95–98, 2010.

[9] T.B. Joewono and H. Kubota, ”Safety and security improvement in public transportation based
on public perception in developing countries,” IATSS Research, vol. 30, no. 1, pp. 86–100, 2006.

[10] H. Li, Z. Lhe, S. Xiaohui, B. Jonathan, H. Gang, ”A Convolutional Neural Network Cascade
for Face Detection,” In Proceedings of IEEE Conference on Computer Vision and Pattern
Recognition, pp. 5325-5334, 2015.

[11] Harpreet Kaur, M. B., ”VEHICLE LICENSE PLATE DETECTION FROM EDGE DETEC-
TION AND MORPHOLOGICAL OPERATORS,” International Journal of Engineering Re-
search Technology (IJERT), vol. 1, issue 9, Nov-2012.

[12] M. H. Bailmare, ”A REVIEW PAPER ON VEHICLE NUMBER PLATE RECOGNITION

USING IMPROVED CHARACTER SEGMENTATION METHOD,” International Journal of
Scientific and Research Publication, Volume 3, 2013.

9
[13] Aishwarya Agrawal, N. P., ”AUTOMATIC LICENCE PLATE RECOGNITION USING
RASPBERRY PI,” International Interdisciplinary Conference on Science Technology Engi-
neering Management Pharmacy and Humanities, Singapore, 2017.

[14] V. Laksmi Priya, K., Perumal, ”DETECTING THE CAR NUMBER PLATE USING SEG-
MENTATION,” International Journal of Engineering and Computer Science, ISSN: 2319-7242,
Volume 3, 2014.

[15] Savneet Kaur, Kamaljit Kaur, ”AN AUTOMATIC SYSTEM FOR DETECTING THE VEHI-
CLE REGISTRATION PLATE FROM VIDEO IN FOGGY AND RAIN ENVIRONMENTS
USING RESTORATION TECHNIQUE,” International Journal of Computer Applications
(0975- 8887), Volume 97, 2014.

10