PROJECT REPORT

ON

WALKING STICK FOR VISUALLY IMPAIRED PERSON

BASED ON RASPBERRY PI
Submitted in partial fulfillment of requirements for the award of
BACHELOR OF TECHNOLOGY in
ELECTRONICS AND COMMUNICATION ENGINEERING

Submitted by:
Name: PUNAM SEAL
ID: ET18BTHEC003
Under the guidance of
Prianku Pratim Neog

School of Engineering and Technology

Department of Electronics and Communication Engineering
The Assam Kaziranga University
April 2022
 CERTIFICATE

This is to certify that the seminar entitled “WALKING STICK FOR VISUALLY
IMPAIRED PERSON BASED ON RASPBERRY PI” has been successfully
presented by Punam Seal, Student ID - ET18BTHEC003, 8th-semester student
of B.Tech. (Electronics and Communication Engineering) programme under The
Assam Kaziranga University. The conceptualization and presentation of this
seminar report was carried out by him under the guidance of Prianku Pratim
Neog.
This report is recommended in the Department of Electronics and
Communication Engineering to partially fulfill the paper Seminar (ET1547) in
the 8th-semester final examination.

Date: 9th May’ 2022

………………………….. …………………………..
Project Supervisor HOD, ECE
Prianku Pratim Neog Dr. Ritunjoy Bhuyan
 ABSTRACT

Basically, it is difficult for blind people to pass their day-to-day life with their
disabilities. The purpose of this paper is to introduce a system that uses the
combination of working aid blind techniques to help blind people to have better
navigation against objects, also to be aware of the road using real-time image-
processing. I interfaced some system with the walking stick to make the stick
smarter. In this system, I interfaced some smart functions with the stick.
Whenever the obstacle is detected on the way through an ultrasonic sensor placed
on the stick, the camera gets triggered to capture the object on the way. The
captured image is sent to the cloud to identify the type of the object and then it is
intimated as a voice command through a speaker or via earphones connected with
Raspberry pi. So that blind can able to identify the object in front of them, if it is
identified that is a human in their way, they can ask for any help. If there is a large
obstacle like a car, they can be able to walk based on the object in front of them.
 ACKNOWLEDGEMENT

I am grateful to Almighty God for giving me the strength, knowledge, and

understanding to complete this project. His love has been more than sufficient to
keep and sustain me.

My profound gratitude goes to my wonderful family who have been a source of

inspiration in my academic pursuit

I am also thankful to my project supervisor, Prianku Pratim Neog who supported

me throughout this project with utmost cooperation and patience and for helping
me in doing this Project.

I also extend gratitude and appreciation to my lecturers in ECE department who

have taught me at one point or the other. May God continue to bless, protect and
guide you all.
 TABLE OF CONTENTS
ABSTRACT
ACKNOWLEDGEMENT
LIST OF FIGURES
Chapter 1: INTRODUCTION i-iv
1.1 BRIEF DESCRIPTION OF THE PROJECT
1.2 LITERATURE SURVEY
1.3 REVIEW OF EXISTING DEVICES
Chapter 2: SYSTEM DESCRIPTION v-xii
2.1 HARDWARE USED
2.2 SOFTWARE USED
Chapter 3: IMPLEMENTATION xiii-xviii
3.1 SYSTEM SETUP
3.2 PROGRAM LOGIC CHART
3.3 SYSTEM ALGORITHM
Chapter 4: FUNCTIONAL DESCRIPTION xix-xx
Chapter 5: CONCLUSION AND FUTURE SCOPE xxi
APPENDIX A
SYSTEM CODE xxii-xxiv
REFERENCE xxv
 LIST OF FIGURES
1.3.1 C-5 Laser Cane ii
1.3.2 Protection zones of the C-5 Laser cane ii
1.3.3 Sonic Torch ii
1.3.4 Sonic Path Finder ii
1.3.5 Miniguide Mobility Aid iii
1.3.6 Mowat Sensor for Blind iii
1.3.7 Meldog-Guided Dog Robot iii
1.3.8 Navbelt iv
2.1.1 Front View of Ultrasonic Sensor vi
2.1.2 Back View of Ultrasonic Sensor vi
2.1.3 Camera vi
2.1.4 Earphone vi
2.1.5 Raspberry Pi 4 Model B parts vii
2.1.6 Front View of Raspberry pi vii
2.1.7 Back View of Raspberry pi vii
2.1.8 Raspberry Pi 4 GPIO Pinout viii
2.1.9 Power Pins on Raspberry Pi 4 viii
2.1.10 SPI pins on Raspberry Pi 4 ix
2.1.11 Raspberry Pi 4 I2C pins x
2.1.12 Physical dimensions of Raspberry pi 4 xi
2.1.13 SD-Card xi
2.1.14 Jumper Wires xi
2.1.15 Power Bank xi
2.2.1 Logo of VNC Viewer xii
3.1.1 Block Diagram of the system xiii
3.1.2 Simple Neuronal Network with five input values and four output values xiii
3.1.3 Single Shot Detector (Object Detection) xv
3.1.4 Image Classification with MobileNet xvi
3.3.1 WALKING STICK FOR VISUALLY IMPAIRED PERSON BASED ON RASPBERRY PI
System xvii
3.4.1 Smart Flow Chart xviii
4.1 Distance and Pothole Detection using Ultrasonic Sensor xix
 Chapter 1
INTRODUCTION

1.1 BRIEF DESCRIPTION OF THE PROJECT

In the list of disabilities, blind people are facing more difficulties in their day-to-day life. Nowadays
every technology becomes smarter, in the list of them, this technology makes walking stick smarter
which has many applications such as walking stick indicator in case if they miss the stick through sound
beeping, navigating their way to home by redefining the users' home through voice command using the
Google map navigator and to know the places near to their location like shops. Electronic Travel Aids
are the devices that are used for mobility. The function of ETA is to provide information to the blind
people about roads and obstacles. Our smart walk stick is also an ETA that helps the blind people to
know about the objects opposite to them, the colour of the objects, and text reading. We use Raspberry
pi to control the sensors and camera in this smart walking stick. The feature of object identification
helps blind people recognize what kind of object is before them and helps them move around safely.
Text reading helps them by reading out the texts before them and finally, colour identification helps
them to identify the colours before them. These are all done using digital image processing technique
by using compute vision2. This smart walk stick is lightweight to carry around and this helps the blind
people to move around like normal people.

1.2 LITERATURE SURVEY

A literature survey is a proof essay of sorts. It is a study of relevant literature materials in relation to a
topic we have been given. For thorough development of the device Smart Stick for Blind Using
Raspberry Pi, we need to go through each and every technical aspect related to it. This chapter provides
an introduction to the area of research. A Brief Study and Survey have been carried out to understand
various issues related to the project which involves providing a smart electronic aid for blind people to
provide artificial vision and object detection by using Raspberry Pi. A survey is made among blind
people finding difficulties in detecting obstacles during walking in the street. Our project mainly focuses
on visually impaired people who cannot walk independently in an unfamiliar environment. The main
aim of our project is to develop a system that helps blind people to move independently.
This system usually consists of three parts to help people travel with a greater degree of psychological
comfort and independence: sensing the immediate environment for obstacles and hazards, providing
information to move freely during travel.
▪ “Navigation Tool for Visually Challenged using Microcontroller”, Sabarish.S.
▪ “Smart walking stick - an electronic approach to assist visually disabled persons”, Mohammad
Hazzaz Mahmud, Rana Saha, Sayemul Islam
▪ “Ultrasonic smart cane indicating a safe free path to blind people”, Arun G. Gaikwad 1, H. K.
Waghmare2 1ME Embedded system Design, MIT Aurangabad 2 Assistant Professor
Department of E&TC, MIT Aurangabad
▪ “A Multidimensional Walking Aid for Visually Impaired Using Ultrasonic Sensors Network
with Voice Guidance”, Olakanmi O. Oladayo

i
1.3 REVIEW OF EXISTING DEVICES
A variety of ETAs and various technologies have been applied for the betterment and safety of the blind
society and to guide them 24/7 by detecting obstacles on the ground, uneven surfaces, holes, steps and
puddles.
A. C-5 Laser Cane
The laser cane has three laser diodes and three photodiodes in it. This laser cane is based on
optical triangulation. The photodiodes act as receivers. This laser diode can detect obstacles at
head height and up to the range of 1.5m or 3m [1].

Figure 1.3.1 C-5 Laser Cane

Figure 1.3.2 Protection zones of the C-5 Laser

cane

B. Sonic Torch
Sonic Torch is a handheld device operated using the battery, it operates by transmitting
ultrasound and receiving back the reflected sound. The distance is calculated by using the time
taken for the reflection of the ultrasound [2].

Figure 1.3.3 Sonic Torch

C. Sonic Path Finder

Sonic Path Finder uses the technology of acoustic difference. It helps blind people by detecting
the obstacles and alarming them. But it does not provide accurate outputs.

Figure 1.3.4 Sonic Path Finder

ii
D. Miniguide
This device was developed in the United States. The price was $545 which is quite expensive
for the blind to buy [2].

Figure 1.3.5 Miniguide Mobility Aid

E. Mowat Sensor
Mowat Sensor is an aid to ambulation for individuals with visual impairment. It is a lightweight,
handheld device, similar to a flashlight, that detects objects by sending out brief pulses of high-
frequency sound (ultrasound). The device vibrates when it detects an object, and users can tell
how close they are to the object by the rate of vibration. The device ignores everything but the
closest object within the beam. It is a handheld ultrasonic-based device that informs the user of
the distance of detected objects using tactile vibrations. The frequency of the vibration is
inversely proportional to the distance between the sensor and the object [12].

Figure 1.3.6 Mowat Sensor for Blind

F. Meldog
Meldog is used to enhance mobility aids for the blind by providing them with the functions of
guide dogs, i.e., obedience in navigating or guiding a blind master, intelligent disobedience in
detecting and avoiding obstacles in his/her path, and well-organized human-machine
communication which does not interfere with his/her remaining senses. It uses artificial

Figure 1.3.7 Meldog-Guided Dog Robot

iii
 intelligence which provides the accurate position of an obstacle using ultrasonic sensors and
lasers. But it is very heavy to carry and move around [2].
G. NavBelt
NavBelt is worn by the user like a belt and is equipped with an array of ultrasonic sensors. It
provides acoustic signals via a set of stereo earphones that guides the user around obstacles or
"displays" a virtual acoustic panoramic image of the traveler’s surroundings. It is a portable
device with ultrasonic sensors and a computer. It gives a 1200-wide view of the obstacles in
front of the user. This image was then translated into the audio output [3].

Figure 1.3.8 Navbelt

iv
 Chapter 2
METHODOLOGY
In the proposed system, I have used an ultrasonic sensor and camera. Here, an ultrasonic sensor will be
useful for detecting the obstacles with the help of ultrasonic technology. Once the obstacle is detected
with the help of the threshold which I have configured, the camera is going to be triggered to use a deep
learning algorithm to detect various objects which is been pre-defined. Here is a mobilenet-SSD model,
which is employed for object recognition.

2.1 HARDWARE USED

❖ Ultrasonic Sensor
❖ Camera
❖ Headphones
❖ Raspberry Pi
❖ SD-Card
❖ Power Supply (Power Bank)

❖ Ultrasonic Sensor
o An ultrasonic sensor is an electronic device that measures the distance of a target object
by emitting ultrasonic sound waves, and converts the reflected sound into an electrical
signal. Ultrasonic waves travel faster than the speed of audible sound (i.e. the sound
that humans can hear). Ultrasonic sensors have two main components: the transmitter
(which emits the sound using piezoelectric crystals) and the receiver (which encounters
the sound after it has travelled to and from the target). In order to calculate the distance
between the sensor and the object, the sensor measures the time it takes between the
emission of the sound by the transmitter to its contact with the receiver. The formula
for this calculation is D = ½ T x C (where D is the distance, T is the time, and C is the
speed of sound ~ 343 meters/second).
o Ultrasonic sensors are used primarily as proximity sensors. They can be found in
automobile self-parking technology and anti-collision safety systems. Ultrasonic
sensors are also used in robotic obstacle detection systems, as well as manufacturing
technology. In comparison to infrared (IR) sensors in proximity sensing applications,
ultrasonic sensors are not as susceptible to interference of smoke, gas, and other
airborne particles (though the physical components are still affected by variables such
as heat) [14].
o It is used to detect the obstacles that come in front of the user. It sends an echo signal
to the Raspberry Pi so that it can decide further actions [6].

v
 Figure 2.1.1 Front View of Ultrasonic Sensor Figure 2.1.2 Back View of Ultrasonic
Sensor

❖ Camera
o A camera is used in this smart blind stick can act as an eye for the visually impaired
person. Each time the ultrasonic sensor detects an obstacle it captures the picture. The
picture is sent to Raspberry Pi so that it can process the image and generate the caption
for that image.

Figure 2.1.3 Camera

❖ Earphones
o Earphones are used as output device which gives the audio output of all the features of
smart stick such as object identification, text identification, and pothole detection.

Figure 2.1.4 Earphone

❖ Raspberry pi
o It’s pi 4 Model B
o Low-cost high-performance computer which can be plugged into TV and monitor and
can be used as a computer which is very small as a credit card.
o Its 64-bit quad-core Cortex-A72 processor

vi
 o It has 4 USB ports
o Size of its RAM is 2GB
o It has extended 40-pin GPIO Header
o 5V/3A DC via USB type C connector
This raspberry pi works as the computer of the smart walking stick [13].

Figure 2.1.5 Raspberry Pi 4 Model B parts

▪ CPU: It consists of a Broadcom BCM2711 chip which contains a 1.5GHz 64-bit quad-
core ARM Cortex-A72 processor (using an ARMv8-architecture core).
▪ GPU: Broadcom VideoCore VI @ 500 MHz was released in 2009. It is capable of Blu-
ray quality video playback, H.265 (4Kp60 decode); H.264 (1080p60 decode, 1080p30
encode); OpenGL ES, 3.0 graphics.
▪ RAM: It comes with 2GB, 4GB, and 8GB (depends on different versions) variants of
LPDDR4 SDRAM.
▪ USB port: It consists of two USB 3.0 and two USB 2.0 ports to connect it to an external
keyboard, mouse, or other peripheral devices.
▪ USB power port: It consists of a 5.1V, 3A USB type-C power port.
▪ HDMI port: Two micro-HDMI ports capable of supporting up to 4k@60HZ resolution.
▪ Ethernet Port: It comes with true Gigabit Ethernet capable of sending Ethernet frames
at a rate of one gigabit per second (1 billion bits per second).
▪ Composite Video Output: Both the audio output socket and the video composite socket
reside in a single 4-pole 3.5mm socket.
▪ SD card Slot: A micro-SD card slot is used for booting up the operating system and
storage purposes.

Figure 2.1.6 Front View of Raspberry pi Figure 2.1.7 Back View of Raspberry pi

vii
GPIO pins
GPIO stands for General Purpose Input Output pins. These pins are used to connect the
Raspberry pi board to external input/output peripheral devices.
This model B consists of a 40-pin GPIO header. Out of these 40 pins, 26 pins are GPIO pins.
A standard interface for connecting a single-board computer or microprocessor to other devices
is through General-Purpose Input/Output (GPIO) pins [13].
Since GPIO pins do not have any specific function, these pins can be customized using the
software.

Figure 2.1.8 Raspberry Pi 4 GPIO Pinout

Power Pins on Raspberry Pi 4:

The raspberry pi 4 model B board consists of two 5V pins, two 3.3V pins, and 7 ground pins
(0V) [13].

Figure 2.1.9 Power Pins on Raspberry Pi 4

5V: The 5v pin outputs the 5 volts coming from the USB Type-C port.
3.3V: The 3v pin is used to provide a stable 3.3v supply to external components.

viii
 GND: The ground pin is commonly referred to as GND.
R-Pi 4 Global Input/Outputs pins:
A GPIO pin set as input allows the signal transmitted by any external device (connected to this
pin) to be received by the Raspberry Pi.
Input voltage between 1.8V and 3.3V is read as HIGH by the Raspberry pi. And when the input
voltage is lower than 1.8V, it is read as LOW [13].
Apart from Input/Output, the GPIO pins can also perform a variety of other functions like
PWM. Some of these functions/pins are:
PWM (pulse-width modulation) pins:
PWM stands for “Pulse Width Modulation”. It means that an analog value is being modulated
on a digital signal.
Software PWM is available on all pins.
Hardware PWM is available on these pins only: GPIO12, GPIO13, GPIO18, GPIO19 [13].
SPI pins on RPi 4:
SPI (Serial Peripheral Interface) is a type of serial communication protocol. It is used by the
Raspberry Pi for master-slave communication to quickly communicate between one or more
peripheral devices.

Figure 2.1.10 SPI pins on Raspberry Pi 4

The data is synchronized using a clock (SCLK at GPIO11) from the master (RPi).
The Pi sends this data to an SPI device using MOSI (Master Out Slave In) pin. And when the
SPI device needs to communicate back to the Raspberry Pi, it sends the data back through the
MISO (Master In Slave Out) pin [13].

5 pins are required for SPI communication:

• GND: Connect the GND pin from all the slave components and the Raspberry Pi 4 board
together.
• SCLK: Clock for SPI communication.

ix
 • MOSI: It stands for Master Out Slave In. This pin is used to send data from the master to a
slave.
• MISO: It stands for Master In Slave Out. This pin is used to receive data from a slave to the
master.
• CE: It stands for Chip Enable. We need to connect one CE pin per slave (or peripheral devices)
in our circuit. By default, we have two CE pins but we can configure more CE pins from the
other available GPIO pins.

SPI pins on Raspberry Pi:

• SPI0: GPIO9 (MISO), GPIO10 (MOSI), GPIO11 (SCLK), GPIO8 (CE0), GPIO7 (CE1)
• SPI1: GPIO19 (MISO), GPIO20 (MOSI), GPIO21 (SCLK), GPIO18 (CE0), GPIO17 (CE1),
GPIO16 (CE2)

I2C Pins on RPi 4:

Figure 2.1.11 Raspberry Pi 4 I2C pins

I2C pins on the Raspberry Pi board are used to communicate with peripheral devices that are
compatible with Inter-Integrated Circuit (a low-speed two-wire serial communication
protocol).
This serial communication protocol requires master-slave roles between both, the board and the
peripheral devices.
I2C protocol requires two connections: SDA (Serial Data) and SCL (Serial Clock). They work
by transmitting data using the SDA connection, and the speed of data transfer is controlled via
the SCLK pin.
Data: (GPIO2), Clock (GPIO3)
EEPROM Data: (GPIO0), EEPROM Clock (GPIO1) [13]

UART pins on RPi 4:

The UART (Universal Asynchronous Receiver / Transmitter) is an asynchronous protocol that
provides a way to communicate between two microcontrollers or devices [13].
TX pin transmits the serial data to the RX pin of another device and RX pin receives the serial
data coming from the TX pin of the other device.
TX: GPIO14

x
 RX: GPIO15

Raspberry Pi 4 Physical Dimensions:

The physical dimension of the Raspberry Pi 4 is 85 mm in length and 56 mm in width.

Figure 2.1.12 Physical dimensions of Raspberry pi 4

❖ SD-Card
o SD-Card act as a secondary storage device for this blind stick.

Figure 2.1.13 SD-Card

❖ Power Bank
o It is used for power supplying the raspberry pi.

Figure 2.1.15 Power Bank

2.2 SOFTWARE USED

❖ VNC Viewer
❖ Language: Python
❖ Library: Open CV
❖ Caffe Framework

❖ Python
o Python is designed in such a way that it is highly readable. Python is processed by the
interpreter at runtime. The program does not need to be complied before executing it.

xi
❖ VNC Viewer
o VNC Viewer is used for local computers and mobile devices you want to control from.
A device such as a computer, tablet, or smart phone with VNC Viewer software
installed can access and take control of a computer in another location.

Figure 2.2.1 Logo of VNC Viewer

xii
 Chapter 3
IMPLEMENTATION

3.1 SYSTEM SETUP

Figure 3.1.1 Block Diagram of the system

Raspberry pi is embedded with the walking stick at, which Ultrasonic sensor is interfaced with GPIO
pins of Raspberry pi, at the same time the camera should be connected with Raspberry pi where both
should face on the same direction on the road. It should be connected with the network so that captured
image can be processed for object detection to intimate which object is an obstacle at this time. In our
proposed system we use Mobile Net SSD to detect various objects.

Computer Vision
Early experiments in computer vision took place in the 1950s, using some of the first neural
networks to detect the edges of an object and to sort simple objects into categories like circles
and squares. Computer vision is an interdisciplinary scientific field that deals with how
computers can gain high-level understanding from digital images or videos [2].
Deep Learning
Deep Learning, is just a type of Machine Learning, inspired by the structure of a human brain.
Deep learning algorithms attempt to draw similar conclusions as humans would by continually
analysing data with a given logical structure. To achieve this, deep learning uses a multi-layered
structure of algorithms called neural networks [2].

Figure 3.1.2 Simple Neuronal Network with five input values and four output values. Each circle represents a
neuron including weights, bias, and activation function, arrows represent the flow of stimuli.

xiii
 The design of the neural network is based on the structure of the human brain. Just as we use
our brains to identify patterns and classify different types of information, neural networks can
be taught to perform the same tasks on data.
The individual layers of neural networks can also be thought of as a sort of filter that works
from gross to subtle, increasing the likelihood of detecting and outputting a correct result. The
human brain works similarly. Whenever we receive new information, the brain tries to compare
it with known objects. The same concept is also used by deep neural networks. Neural networks
enable us to perform many tasks, such as clustering, classification, or regression. With neural
networks, we can group or sort unlabelled data according to similarities among the samples in
this data. Or in the case of classification, we can train the network on a labelled dataset in order
to classify the samples in this dataset into different categories [2].
Pre-trained model
A pre-trained model is a saved network that was previously trained on a large dataset, typically
on a large-scale image-classification task. You either use the pre-trained model as is or use
transfer learning to customize this model to a given task.
The intuition behind transfer learning for image classification is that if a model is trained on a
large and general enough dataset, this model will effectively serve as a generic model of the
visual world. You can then take advantage of these learned feature maps without having to start
from scratch by training a large model on a large dataset.
There are two ways to customize a pre-trained model:
o Feature Extraction: Use the representations learned by a previous network to extract
meaningful features from new samples. You simply add a new classifier, which will be trained
from scratch, on top of the pre-trained model so that you can repurpose the feature maps learned
previously for the dataset.
You do not need to (re)train the entire model. The base convolutional network already contains
features that are generically useful for classifying pictures. However, the final, classification
part of the pre-trained model is specific to the original classification task, and subsequently
specific to the set of classes on which the model was trained.
o Fine-Tuning: Unfreeze a few of the top layers of a frozen model base and jointly train both the
newly-added classifier layers and the last layers of the base model. This allows us to "fine-tune"
the higher-order feature representations in the base model in order to make them more relevant
for the specific task [2].
Caffe model:
Caffe is a deep learning framework characterized by its speed, scalability, and modularity.
Caffe works with CPUs and GPUs and is scalable across multiple processors. The Deep
Learning Framework is suitable for industrial applications in the fields of machine vision,
multimedia, and speech. Key features of Caffe include support for Central Processing Units and
Graphics Processing Units, as well as Nvidia’s Compute Unified Device Architecture (CUDA)
and the cuDNN Library (CUDA Deep Neural Network), also from this vendor. Thus, the
framework is designed primarily for speed.
Caffe can work with many different types of deep learning architectures. The framework is
suitable for various architectures such as CNN (Convolutional Neural Network), Long-Term
Recurrent Convolutional Network (LRCN), Long Short-Term Memory (LSTM) or fully
connected neural networks. A large number of pre-configured training models are available to
the user, allowing a quick introduction to machine learning and the use of neural networks [2].
SSD Network Architecture

xiv
 The SSD architecture is a single convolution network that learns to predict bounding box
locations and classify these locations in one pass. Hence, SSD can be trained end-to-end. The
SSD network consists of base architecture (MobileNet in this case) followed by several
convolution layers.

Figure 3.1.3 Single Shot Detector (Object Detection)

By using SSD, we only need to take one single shot to detect multiple objects within the image,
while regional proposal network (RPN) based approaches such as R-CNN series that need two
shots, one for generating region proposals, one for detecting the object of each proposal. Thus,
SSD is much faster compared with two-shot RPN-based approaches.
To have more accurate detection, different layers of feature maps are also going through a small
3×3 convolution for object detection as shown above.
Say for example, at Conv4_3, it is of size 38×38×512. 3×3 conv is applied. And there are 4
bounding boxes and each bounding box will have (classes + 4) outputs. Thus, at Conv4_3, the
output is 38×38×4×(c+4). Suppose there are 20 object classes plus one background class, the
output is 38×38×4×(21+4) = 144,400. In terms of number of bounding boxes, there are
38×38×4 = 5776 bounding boxes.
Similarly for other conv layers:
Conv7: 19×19×6 = 2166 boxes (6 boxes for each location)
Conv8_2: 10×10×6 = 600 boxes (6 boxes for each location)
Conv9_2: 5×5×6 = 150 boxes (6 boxes for each location)
Conv10_2: 3×3×4 = 36 boxes (4 boxes for each location)
Conv11_2: 1×1×4 = 4 boxes (4 boxes for each location)
If we sum them up, we got 5776 + 2166 + 600 + 150 + 36 + 4 = 8732 boxes in total. If we
remember YOLO, there are 7×7 locations at the end with 2 bounding boxes for each location.
YOLO only got 7×7×2 = 98 boxes. Hence, SSD has 8732 bounding boxes which is more than
that of YOLO [2] [1].

MobileNet:
MobileNet is an efficient and portable CNN architecture that is used in real-world applications.
MobileNets primarily use depthwise separable convolutions in place of the standard
convolutions used in earlier architectures to build lighter models. MobileNets introduce two
new global hyperparameters (width multiplier and resolution multiplier) that allow model
developers to trade off latency or accuracy for speed and low size depending on their
requirements.
MobileNets are built on depthwise separable convolution layers. Each depthwise separable
convolution layer consists of a depthwise convolution and a pointwise convolution. Counting
depthwise and pointwise convolutions as separate layers, a MobileNet has 28 layers. A standard

xv
 MobileNet has 4.2 million parameters which can be further reduced by tuning the width
multiplier hyperparameter appropriately.
The size of the input image is 224 × 224 × 3 [1] [2].

Figure 3.1.4 Image Classification with MobileNet

3.2 SYSTEM ALGORITHM

STEP 1: Start.
STEP 2: Read the GPIO pins.
STEP 3: If pin 1 is high go to step 4 if pin 2 is high go to STEP 7 if pin 3 is high go to STEP
10 else go to STEP 2.
STEP 4: opens the webcam, take a picture and save the image as “sample.jpeg” and move it to
home/pi/webcam.
Executes image to text conversion using python and saves the output as “output1.txt”.
STEP 5: Moves “output1.txt” to “audio.txt” file and executes text to audio conversion
STEP 6: generates audio output and moves to STEP 2.
STEP 7: executes ultrasonic distance measurement using python and saves the output as
“output2.txt”.
STEP 8: Moves the text in “output2.txt” to “audio.txt” and executes text to audio conversion.
STEP 9: Generates audio output and moves to STEP 2.
STEP 10: Opens webcam takes picture and save the image as “objsamp.jpeg” and moves it to
home/pi/object.
STEP 11: Executes object recognition using python and saves the output as “output3.txt”.
STEP 12: Moves the text in “output3.txt” to “audio.txt” and executes text to audio conversion.
STEP 13: Generates the audio output and moves to STEP 2.

xvi
3.3 LAYOUT SPECIFICATION

Holder
Power Bank

Camera

Raspberry Pi

Ultrasonic
Sensor
Headphone

Figure 3.3.1 WALKING STICK FOR VISUALLY IMPAIRED PERSON BASED ON RASPBERRY
PI System

3.4 PROGRAM LOGIC CHART

The above flowchart shows the process details that take place in the smart walking stick. And much
easier algorithm for the smart stick is given below

xvii
 Load Pre-trained Model
for Object Recognition

Distance Measurement
(Ultrasonic Sensor)

NO
Distance < Threshold
YES

Image/Video Acquisition
(Camera)
Pre-processing Image

Resizing Frame

Converting Image into

Blob Image Set Pre-processed image to the pre-trained
model (MobileNet SSD) as Input

Forward Model - Result

Confirming the presence of

Object

Obtaining bounding box

coordinate

Obtaining Class ID

Text to Speech conversion

(Class Name)
Figure 3.4.1 Smart Flow Chart

xviii
 Chapter 4
FUNCTIONAL DESCRIPTION

Ultrasonic sensor:
The ultrasonic sensor is used to calculate the distance of the object or for pothole detection.
These produce elastic waves with a frequency greater than 20,000 Hz and exists in solid-liquid
and gases. For object identification, the ultrasonic sensor is placed on front side of the walking
stick which sends ultrasounds and calculates the distance. For pothole detection, the sensor is
placed below the smart walking stick. A distance more than an average distance is set. So when
any pothole appears, its distance will be higher than the normal distance and hence we get an
alert The method of distance calculation in the block diagram is known as the pulse reflection
method which counts the number of reflection pulses reflected back compared with the
transmitted pulses [5].

Figure 4.1 Distance and Pothole Detection using Ultrasonic Sensor

Raspberry pi:
The Central processing unit is the brain of the raspberry pi board and is responsible for carrying
out the instructions of the computer through logical and mathematical operations. The Ethernet
port of the raspberry pi is the main gateway for communicating with additional devices. The
raspberry pi Ethernet port is used to plug your home router to access the internet [9].

ARM Cortex-A72 Processor

The raspberry pi uses ARM Cortex-A72 which is a central processing unit implementing the
ARMv8-A 64-bit instruction set designed by ARM Holdings' Austin design centre. It is
available as SIP core to licensees, and its design makes it suitable for integration with other SIP
cores (e.g., GPU, display controller, DSP, image processor, etc.) into one die constituting a
system on a chip (SoC). The Cortex-A72 was announced in 2015 to serve as the successor of
the Cortex-A57 and was designed to use 20% less power or offer 90% greater performance.

xix
 It has security low power consumption and high-performance integer processor the GPU is a
specialized chip in the raspberry pi board and that is designed to speed up the operation of
image calculations. This board designed with a Broadcom BCM2711 and it supports OpenGL
[10].

GPIO pins
The general-purpose input & output pins are used in the raspberry pi to associate with the other
electronic boards. These pins can accept input & output commands based on programming
raspberry pi. The raspberry pi affords digital GPIO pins. These pins are used to connect other
electronic components. For example, you can connect it to the temperature sensor to transmit
digital data Here in this smart walking stick the sensor, camera, and switch are given as input
and the output will be audio through earphones. This all input and output will be through GPIO
Pins [8].

Object identification
In this process, I pretrained the basic structures of objects such as stones, cars, humans, etc.,
and if any variation is detected by ultrasonic sensors the camera will turn on to capture the
image and it compares the captured image with the pre-fed images to find the object [11].

xx
 Chapter 5
CONCLUSION AND FUTURE SCOPE

At the end of our project, I can conclude that our project can reduce the number of risks and injuries for
the visually impaired person when walking at public. Nowadays, even at young age experience the
visually impairment. This thing cannot be taken so lightly as they know how much risk could it be. If
the number of risk and injuries increases rapidly, the kid or the person will loss their spirit to walk
independently.
This stick acts as a basic platform for the coming generation of more aiding devices to help the visually
impaired to navigate safely both indoor and outdoor. It is effective and affordable. It leads to good
results in detecting the obstacles on the path of the user in a range of ten cm. Though the system is hard-
wired with sensors and other components, it's light in weight. Further aspects of this system can be
improved via wireless connectivity between the system components, thus, increasing the range of the
ultrasonic sensor and implementing technology for determining the speed of approaching obstacles.

The future scope of the existing smart stick, guides the visually impaired person in his navigation
independently in an efficient manner ensuring the person’s safety.
a. This system has very limited pretrained objects. In future, I will train the algorithm with different set
of database, which will increase the accuracy of the overall system.
b. The programmable wheels would steer the stick away from the obstacles and also leading the blind
person towards the destination.
c. Internet of Things is a trending concept which can increase the benefits of the smart stick by allowing
one stick to communicate with another smart stick (or mobile , PCs) nearby to utilize the functionality
of the other stick when one stick’s functionality breaks down.
d. In order to run this integrated set of hardware we can use solar panels as an alternative to the battery.
The use of solar panel occurs to be more advantageous as it uses sunlight, the easily available renewable
resource of energy, to get recharged.

xxi
 APPENDIX A
SYSTEM CODE
import numpy as np
import imutils
import RPi.GPIO as GPIO

import cv2
import time
import os
GPIO.setmode(GPIO.BCM)

BUZZER=26
GPIO.setup(BUZZER,GPIO.OUT)
GPIO.output(BUZZER, False)

prototxt = "MobileNetSSD_deploy.prototxt.txt"
model = "MobileNetSSD_deploy.caffemodel"
confThresh = 0.2

CLASSES = ["background", "aeroplane", "bicycle", "bird", "boat",

"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
"sofa", "train", "tvmonitor"]

COLORS = np.random.uniform(0, 255, size=(len(CLASSES), 3))

print("Loading model...")
net = cv2.dnn.readNetFromCaffe(prototxt, model)
print("Model Loaded")
print("Starting Camera Feed...")
vs = cv2.VideoCapture(0)
time.sleep(2.0)

imflag=0

z=0

while (z<5):
_,frame = vs.read() #read the frame from camera
frame = imutils.resize(frame, width=500) #resize for display in ouput
(h, w) = frame.shape[:2] #obtaining h & w
imResize = cv2.resize(frame, (300, 300)) #pre process pre train
blob = cv2.dnn.blobFromImage(imResize,
0.007843, (300, 300), 127.5) #blob

xxii
 net.setInput(blob)
detections = net.forward()

detShape = detections.shape[2]
for i in np.arange(0,detShape):
confidence = detections[0, 0, i, 2]
if confidence > confThresh:
idx = int(detections[0, 0, i, 1])
box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
(startX, startY, endX, endY) = box.astype("int")
if (idx == 2 or idx == 6 or idx == 7 or idx == 14):
GPIO.output(BUZZER,True)
time.sleep(1)
GPIO.output(BUZZER,False)

label = "{}: {:.2f}%".format(CLASSES[idx],

confidence * 100)
data="espeak"+" "+str(CLASSES[idx])+"_detected"
os.system(data)
cv2.rectangle(frame, (startX, startY), (endX, endY),
COLORS[idx], 2)
if startY - 15 > 15:
y = startY - 15
else:
y =startY + 15
cv2.putText(frame, label, (startX, y),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, COLORS[idx], 2)
imflag=1
z+=1
print(z)
# show the output frame

#cv2.imshow("Frame", frame)
key = cv2.waitKey(1)
if key == 27:
break

if imflag==0:
da="espeak"+" "+"No_Obstacle_detected_you_can_walk_freely"
os.system(da)

os.system("espeak detection_completed")
vs.release()
cv2.destroyAllWindows()

xxiii
import RPi.GPIO as GPIO
import time
import subprocess
import cv2
import beepy as beep

GPIO.setmode(GPIO.BCM)
TRIG=2
ECHO=3
GPIO.setup(TRIG,GPIO.OUT)
GPIO.setup(ECHO,GPIO.IN)
GPIO.output(TRIG, False)
data= ""

def val():
GPIO.output(TRIG, True)
time.sleep(0.00001)
GPIO.output(TRIG, False)
while GPIO.input(ECHO)==0:
pulse_start = time.time()
while GPIO.input(ECHO)==1:
pulse_end = time.time()
pulse_duration = pulse_end - pulse_start
distance = pulse_duration*17150
distance = round(distance, 2)
return distance
while True:
dst = val()
print (dst)
if(dst < 100): #Distance Value
print("Distance low")
data="OBSTACLE DETECTED"
beep.beep(3)
beep.beep(3)
beep.beep(3)
time.sleep(1)
subprocess.Popen("python3 1.py",shell=True)

else:
print("No obstacle detected")
time.sleep(5)

------------------------------------------------------------------------------

xxiv
 REFERENCES

[1] Shashank Chaurasia and K.V.N. Kavitha “An electronic walking stick for blinds”
ICICES2014 - S.A.Engineering College, Chennai, Tamil Nadu, India, 2014, ISBN No.978-1-
4799-3834-6/14/$31.00©2014 IEEE
[2] S. Gupta, I. Sharma, A. Tiwari and G. Chitranshi, "Advanced guide cane for the visually
impaired people," 2015 1st International Conference on Next Generation Computing
Technologies (NGCT), Dehradun, 2015, pp. 452-455. doi: 10.1109/NGCT.2015.7375159
[3] Susumu Tachi, Kazuo Tanie, Kiyoshi Komoriya, Minoru Abe, “Electrocutaneous
Communication In A Guide Dog Robot (Meldog)”,IEEE Transactions on Biomediacal
Engineering, 7 July 1985. :
[4] M. Varghese, S. S. Manohar, K. Rodrigues, V. Kodkani and S. Pendse, "The smart guide
cane: An enhanced walking cane for assisting the visually challenged," 2015 International
Conference on Technologies for Sustainable Development (ICTSD), Mumbai, 2015, pp. 1-5.
doi: 10.1109/ ICTSD. 2015.7095907
[5] D. S. Raghuvanshi, I. Dutta and R. J. Vaidya, "Design and analysis of a novel sonar-based
obstacle-avoidance system for the visually impaired and unmanned systems," 2014
International Conference on Embedded Systems (ICES),
[6] A.K.Shrivastava, A.Varma and S.P.Singh “Distance measurement of an object or an
obstacle by ultrasound sensors using P89C51RD2” International Journal of Computer Theory
and Engineering, Vol.2, No.1, February 2010.
[7] Vandenbroucke N, Macaire L, Postaire J.G. Color Image Segmenation by Pixel
Classification in an Adapted Hybrid Color Space. Application to Soccer Image Analysis.
Computer Vision and Image Understanding, 2003,90:190-216.
[8] .Cheng H.D, Jiang X.H, Sun Y, Wang J. Color Image Segmentation: Advances and
Prospects. Pattern Recognition, 2001, 34: 2259-2281
[9] Manoj Badoni and Sunil Semwal, “Discrete Distance And Water Pit Indicator Using Avr
Atmega8 In Electronic Travel Aid For Blind”, International Journal of Disaster Recovery and
Business Continuity Vol. 2, November, 2011
[10] Tarek Mohammad, “Using Ultrasonic And Infrared Sensors For Distance Measurement”,
World Academy of Science, Engineering and Technology 27 2009.
[11] Xiuzhi Li, Guangming Zhang, Jing Fang, Jian Wu, Zhiming Cui, “Vehicle color
recognition using vector matching of template,” In Third International Symposium on
Electronic Commerce and Security, pp. 189–193, 2010.
[12] https://dictionary.apa.org/mowat-sensor
[13] Raspberry Pi Beginner's Guide 4th Edition

xxv