DESIGN AND IMPLEMENTATION OF AN ASSISTANCE

SYSTEM FOR THE VISUALLY CHALLENGED

Diksha Sharma Natasha Mathur

Department of Computer Engineering Department of Computer Engineering
Dr. D. Y. Patil Institute of Technology, Pimpri Dr. D. Y. Patil Institute of Technology, Pimpri
Pune 18 Pune 18
diksha6320@gmail.com natashamathur181@gmail.com

Esha Agarwal Shreyansh Bhardwaj

Department of Computer Engineering Department of Computer Engineering
Dr. D. Y. Patil Institute of Technology, Pimpri Dr. D. Y. Patil Institute of Technology, Pimpri
Pune 18 Pune 18
esha.agarwal0212@gmail.com shreyanshbhardwaj9@gmail.com

Dr. Rachna Somkunwar

Associate Professor, Computer Engineering
Dr. D. Y. Patil Institute of Technology, Pimpri Pune 18
rachnasomkunwar12@gmail.com

Abstract visually impaired individuals. Overall, this study

This research paper delves into the
development and implementation of a technology
solution to assist visually impaired individuals in
navigating their surroundings. The proposed
solution employs deep learning and utilizes a
convolutional neural network (CNN) to detect and
classify objects in real-time, providing voice
feedback and distance measurements to the user.
With a large dataset of object images, the system
is highly versatile and can recognize various
objects. It also warns the user if they are too close
or far from an object, reducing the risk of
accidents. Experimental evaluations of the system
show high accuracy and effectiveness in assisting
offers a promising solution to the challenges faced
by visually impaired individuals and underscores
the potential of deep learning in creating assistive
technologies.
Keywords: Blind Assistance, Deep Learning.
1. I. INTRODUCTION
The ability to effectively navigate through various
aspects of life, including work, education, shopping,
and more, relies on a variety of activities. Vision
plays a crucial role in facilitating these tasks by
providing an easier way to move from one place to
another. As such, it is widely acknowledged that
vision is essential for efficient navigation. Although it
may seem possible to navigate familiar environments
such as our home or workplace without eyesight,
navigating unfamiliar surroundings can be quite
challenging. [1] In recent decades, there has been a
growing number of individuals affected by blindness,
with varying degrees of vision impairment. Those
with partial vision loss face different visual
challenges than those who are completely blind. The
aim of the system is to aid visually impaired
individuals and enhance their independence and
self-sufficiency. The system incorporates glasses that
allow for reading, studying, and learning from printed
text or images, especially in educational contexts
where visually impaired individuals may face
difficulties due to their disability.[2]The human
ability to interact with the environment is heavily
reliant on the senses of touch and vision. For blind
amputees, regaining the ability to perform daily
activities poses a significant challenge, as they need a
swift and intuitive way to receive environmental
information.[3] For a person with blindness, the lack
of visual input can lead to difficulties in realizing and
understanding the things around them. They may rely
on sharp observation and hearing to form mental
images, but may still struggle to remember or identify
objects. This can result in the need for assistance
from others. Finding directions can also be a
challenge, leading to inconvenience and potential
reliance on guides.[4] Blind and visually impaired
individuals face challenges in daily activities,
including lower workforce participation rates and
higher risks of falls. Computer vision technology has
the potential to assist, but there is currently no
effective approach for conveying environmental
information. Improving the information transfer rate
(ITR) is important for facilitating spatial cognition.
Public transport management systems (PTMS) are
helpful for most passengers, but useless for visually
impaired individuals due to their disability.
The proposed blind assistance system utilizes deep
learning techniques based on convolutional neural
networks. The flexibility, robustness, high accuracy
rates, and feature extraction capabilities of deep
learning make it an ideal choice for developing such
systems. An Android application is utilized in the
system to capture and transmit real-time frames to a
networked server running on a laptop. The server
employs a pre-trained SSD identification model,
which has been trained on COCO datasets, to identify
the output class. The accuracy of the output class is
then evaluated using a metric. The identified class is
converted into voice notes with the help of voice
modules, and the blind person has assisted
accordingly. Additionally, the system includes an
alarm system that detects objects, calculates
distances, and produces voice-based outputs based on
proximity. This system has the potential to enhance
the independence and mobility of blind individuals
significantly. The integration of deep learning and
speech assistance offers a personalized and intuitive
experience, enabling individuals to navigate their
environment with greater confidence and ease.
II. LITERATURE SURVEY
The use of trained guide dogs with a walking cane to
avoid obstacles is a traditional technique that has
been used for decades to help the blind and visually
impaired. The innovative adaptive footwear and
eyewear suggested in [5] let blind people move
around safely. To detect things at various heights,
ultrasonic transducers are installed on each shoe. The
eyewear is equipped with a buzzer at one of the
temples and two ultrasonic transducers located
centrally above the bridge.
The cane suggested in [6] employs the Camshaft
Position Sensor (CMP) compass sensor 511 and a
ping ultrasonic sensor to detect potholes and hitches.
This system's drawback is that the CMP compass
sensor is sensitive to any stray magnetic fields and is
impacted by any nearby iron items. Although it offers
long-distance navigation, it is useless for indoor
navigation. Earlier, sensor-based approaches only
served to detect objects, not to identify them. As a
result, image processing shows to be a useful
technique for dealing with these circumstances.
A stereo image processing system for the blind
consists of a helmet-mounted wearable computer,
stereo cameras that serve as vision sensors, and stereo
earbuds. The vision sensors record an image of the
environment in front of the visually impaired. To aid
with navigation, the processed photos highlight key
elements in the scene in front of the camera. Stereo
cameras are utilized to incorporate distance
information [7].
This system takes time to process information, so it
cannot be said to be running or functioning instantly.
The most popular method for object recognition is
one based on the Haar Cascade. Using a supervised
method divides the entire image into significant and
non-significant groups [8].
In [9] The paper presents a hardware architecture for
object detection using an AdaBoost learning
algorithm with Haar-like features as weak classifiers.
The proposed partially parallel execution model
exploits the cascade structure of classifiers, assigns
more resources to frequently used classifiers, and
achieves flexible processing performance by
adjusting the balance of parallel processing. The
architecture is implemented on a Virtex-5 FPGA,
demonstrating real-time object detection at 30 fps on
VGA video without candidate extraction. The
limitation of this paper is that it has major hardware
requirements and its scope in blind assistance is less.
In [10] The aim of this study is to introduce an
intelligent assistance system for individuals with
visual impairment, which comprises wearable smart
glasses, an intelligent walking stick, a mobile
application, and an online information platform. By
wearing smart glasses and using an intelligent
walking stick, obstacles can be detected, aiding
individuals with visual impairments. In case of a fall,
the system records and uploads relevant information,
such as GPS location and fall detection, to the online
platform. The mobile application can also be used to
view this information. The drawback of this system is
that it requires a physical device which might not be
affordable for everyone and it has complex
arrangements.
The author of [11] developed an Intelligent
Assistance System for Visually Impaired People
(ISVB) which is composed of three interconnected
components: a smart cap, a 3D-printed intelligent
cane, and a mobile application that connects to an
online server. The smart cap uses a Raspberry Pi and
camera module with a deep learning object detection
module to identify obstacles in the user's path. The
3D-printed intelligent cane has a microcontroller with
multiple sensors and a Bluetooth module that
analyzes the environment and guides the visually
impaired user. The mobile application acts as an
interface between the cap, cane, and user, offering
virtual navigation to support visually impaired
individuals in their movements. This system has the
potential to significantly improve the independence
and mobility of visually impaired people. However, a
potential drawback is that the setup may require a
significant amount of hardware which could be
challenging to implement.

III. SYSTEM PIPELINING

The system pipeline is shown in fig1.

1. Input Image: Here, the user can upload the

image that needs to be processed.
Fig 1: System Flow Chart
2. Image Pre-processing: This step involves
applying techniques like converting the
image to grayscale and removing image
noise to prepare the image for further
processing. IV. Proposed System

3. Image Feature Extraction: In this stage, we
will use methods to extract image features
by analyzing the pixels in the image.
4. Image Classification: The image
classification methods will be applied to
distinguish between contaminated and safe
areas based on the extracted features.
5. Outcome: This step involves presenting the
final result of the detection process.
System Flow Chart:
The proposed system aims to provide real-time object
detection for visually impaired individuals through an
Android app that captures frames as shown in fig2.
The frames are then sent to a networked server
running on a laptop, which performs all the necessary
computations. This approach is taken to leverage the
processing power of the laptop to ensure that the
system can accurately identify objects in real time.
The networked server on the laptop uses a pre-trained
Single Shot Detector (SSD) identification model,
which has been trained on Common Objects in
Context (COCO) datasets to identify the output class.
Once the object class has been identified, it is
evaluated using an accuracy metric to ensure that the
system can accurately identify a wide range of
objects.

After the object class has been tested using an

accuracy metric, the system converts it into default
 voice notes using voice modules. These voice notes
provide assistance to visually impaired individuals by
informing them about the object's identity. This
feature is crucial as it enables the user to navigate
their surroundings independently.
To ensure the safety of visually impaired individuals,
an alarm system has been incorporated into the
system. The alarm system not only detects objects but
also calculates distance estimates. Based on the
Fig 2: Model Architecture
Algorithm:
Convolution Neural Network(CNN) Algorithm:
The CNN algorithm comprises two layers: the feature
extraction layer and the feature map layer. The feature
extraction layer has neurons that are directly
connected to the local receptive fields of the previous
layer, and local features are extracted from them. The
spatial relationship between these local features and
other features is also determined. The feature map
layer consists of planes, and each neuron in a plane
has the same weight. The sigmoid function is used as
object's proximity to the visually impaired individual,
the system produces voice-based outputs and distance
units to alert the user about the object's presence and
distance. This feature helps prevent accidents and
enables visually impaired individuals to navigate
their surroundings safely.
the activation function in this layer, which causes a
shift in the feature map. In the CNN, each convolution
layer follows a computational layer and is responsible
for extracting two types of features: local averages and
second extracts. This unique structure reduces the
resolution of the image.
1. Start by selecting a dataset to work with and
determining the image's resolution.
2. Use information gain and ranking to perform
feature selection.
 3. Employ a CNN classification algorithm to
analyze the image.
4. Calculate the value of each feature in the
input layer.
5. Determine the bias class of each feature.
6. Generate the feature map and move on to the
forward pass input layer.
7. Evaluate the convolution kernels in a feature
pattern.
8. Create the sub-sample layer and feature
value.
9. Backpropagate the input deviation of the kth
neuron in the output layer.
10. Finally, present the chosen feature and the
classification results.
V. RESULTS AND DISCUSSION
Fig3: Performance analysis of the model
According to the results shown in Figure 3, it is
evident that the proposed blind assistance system
performs better than the currently existing models. The
accuracy of the currently existing model ranges from
60-80%, while our proposed system promises an
accuracy of greater than 80%. Moreover, our system
demonstrates an increase in recall from 65% to 85%,
which is a significant improvement compared to the
existing model mentioned in [1]. Overall, the
performance of our proposed system is more
promising and better than the existing models, as
shown by the results.
These results suggest that the proposed blind
assistance system has the potential to be a more
reliable and effective solution for assisting visually
impaired individuals in navigating their surroundings.
With further development and refinement, this
technology could greatly improve the quality of life
for those who are visually impaired, by increasing
their independence and mobility.
Conclusion:
The primary objective of the proposed system is to
assist visually impaired individuals in perceiving their
surroundings, enabling them to navigate independently
and avoid obstacles. The system is designed to offer a
practical and efficient solution that swiftly and
accurately detects objects in the user's immediate
vicinity, whether indoors or outdoors. It can identify
various objects and provide relevant information
through headphones or speakers. The system's
effectiveness in detecting objects is evaluated in three
different environments: indoor, outdoor, and beyond
10 meters from the camera. Overall, the system can
detect and provide audio information about objects in
the user's immediate surroundings, promoting their
mobility and self-reliance.
References:
[1] R. R. Varghese, P. M. Jacob, M. Shaji, A. R, E. S.
John, and S. B. Philip, "An Intelligent Voice
Assistance System for Visually Impaired using
Deep Learning," 2022 International Conference on
Decision Aid Sciences and Applications (DASA),
 Chiangrai, Thailand, 2022, pp. 449-453, doi: [8] Rui Jiang, Qian Lin Li, “Let Blind People See:
10.1109/DASA54658.2022.9765171. Real-Time Visual Recognition with Results
[2] S. C. Jakka, Y. V. Sai, J. A and V. A. M. A, "Blind Converted to 3D Audio”, Proc. International
Assistance System using TensorFlow," 2022 3rd Conference on Computer Vision, 2015.
International Conference on Electronics and [9] M. Hiromoto, H. Sugano, and R. Miyamoto,
Sustainable Communication Systems (ICESC), “Partially Parallel Architecture for
Coimbatore, India, 2022, pp. 1505-1511, doi: AdaBoost-Based Detection With Haar-Like
10.1109/ICESC54411.2022.9885356. Features”, IEEE Trans. Circuits and Systems for
[3] S. Durgadevi, K. Thirupurasundari, C. Komathi Video Technology, vol. 19, Jan 2009, pp. 41-52.
and S. M. Balaji, "Smart Machine Learning [10] L. -B. Chen, J. -P. Su, M. -C. Chen, W. -J.
System for Blind Assistance," 2020 International Chang, C. -H. Yang and C. -Y. Sie, "An
Conference on Power, Energy, Control and Implementation of an Intelligent Assistance
Transmission Systems (ICPECTS), Chennai, System for Visually Impaired/Blind People," 2019
India, 2020, pp. 1-4, doi: IEEE International Conference on Consumer
10.1109/ICPECTS49113.2020.9337031. Electronics (ICCE), Las Vegas, NV, USA,
[4] X. Hu, A. Song, Z. Wei, and H. Zeng, 2019, pp. 1-2,
"StereoPilot: A Wearable Target Location System doi:10.1109/ICCE.2019.8661943.
for Blind and Visually Impaired Using Spatial [11] N. Ghatwary, A. Abouzeina, A.
Audio Rendering," in IEEE Transactions on
Kantoush, B. Eltawil, M. Ramadan and M.
Neural Systems and Rehabilitation Engineering,
Yasser, "Intelligent Assistance System for
vol. 30, pp. 1621-1630, 2022, doi:
Visually Impaired/Blind People (ISVB),"
10.1109/TNSRE.2022.3182661.
2022 5th International Conference on
[5] Ziad O. Abu-Faraj, Paul Ibrahim, Elie Jabbour,
Communications, Signal Processing, and their
and Anthony Ghaoui, “Design and Development
of a Prototype Rehabilitative Shoes and Spectacles
Applications (ICCSPA), Cairo, Egypt, 2022,

for the Blind”, IEEE Int. Conf. BioMedical pp. 1-7, doi:
Engineering and Informatics, 2012, pp. 795-799. 10.1109/ICCSPA55860.2022.10019201
[6] Giva Andriana Mutiara, Gita Indah Hapsari and [12] Y. -S. Lin, W. -C. Chen and T. P. -C.
Ramanta Rijalul, “Smart Guide Extension for Chen, "Tensor-Centric Processor Architecture
Blind Cane", IEEE Int. Conf. Information and for Applications in Advanced Driver
Communication Technologies, 2016. Assistance Systems," 2021 International
[7] G. Balakrishnan, G. Sainarayanan, R. Nagarajan, Symposium on VLSI Design, Automation and
and Sazali Yaacob, “A Stereo Image Processing Test (VLSI-DAT), Hsinchu, Taiwan, 2021, pp.
System for Visually Impaired”, Int. Journal of
1-3, doi:
Computer, Electrical, Automation, Control, and
10.1109/VLSI-DAT52063.2021.9427310.
Information Engineering, vol.2, No.8, 2008, pp.
2794-2803.
[13] K. Chaccour and G. Badr, "Novel indoor
navigation system for visually impaired and
blind people," 2015 International Conference
on Applied Research in Computer Science
and Engineering (ICAR), Beirut, Lebanon,
2015, pp. 1-5, doi:
10.1109/ARCSE.2015.7338143.
[14] Conference on VLSI Design and 2016
15th International Conference on Embedded
Systems (VLSID), Kolkata, India, 2016, pp.
421-426, doi: 10.1109/VLSID.2016.11.