VISVESVARAYA TECHNOLOGICAL UNIVERSITY

BELAGAVI-590018
VISVE

Project Evalution Phase-II(19ECP83)

Report on

“A Hybrid Approach For Identification Of Manhole

And Staircase To Assist Visually Challenged”
“A Hybrid Appr
Submitted in the partial fulfillment of the requirement for the Project Phase-
II
Submitted in
Bachelor of Engineering
In
Electronics & Communication Engineering
Submitted by E

Mr. Akhil Reddy 1NC19EC003

Mr. Aryan 1NC19EC011
Mr. Avinash R 1NC19EC013
Mr. Deekshith G K 1NC19EC024

Under the Guidance of

Ms. SUNITHA M
Assistant Professor

DEPARTMENT OF ELECTRONICS & COMMUNICATION

ENGINEERING DEPARTMENT

NAGARJUNA COLLEGE OF ENGINEERING AND TECHNOLOGY NAGARJUNA COLLEG

(An Autonomous Institution under VTU, Belgavi-590018) (An
i
2022-2023

ii
 DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING

CERTIFICATE

This is to Certify that the project work entitled A Hybrid Approach For Identification Of Manhole
And Staircase To Assist Visually Challenged carried out by Mr. Akhil Reddy (1NC19EC003), Mr. Aryan
(1NC19EC011), Mr. Avinash R (1NC19EC013), Mr. Deekshith G.K (1NC19EC024) a bonafied students
of Nagarjuna College of Engineering and Technology, Bangalore an autonomous college under Visvesvaraya
Technological University, Belagavi, in partial fulfillment for the project work carried out by them for the
course “Project Phase-II” during the year 2022-2023.
It is certified that all corrections/suggestions indicated for internal assessment have been incorporated in the
report and submitted to the department library. The project report has been approved, as it satisfies the
academic requirements of Project Phase-II.

Signature of the Guide Signature of the Coordinator Signature of the HOD

Ms. Sunitha M Dr. Ajay Kumar Dwivedi Dr. Nagesh K N
Assistant Professor Associate Professor Professor & HOD
ECE Dept. NCET ECE Dept. NCET ECE Dept. NCET

Internal/External Viva

Name of the Examiners Signature with date

1.
2.
 ACKNOWLEDGEMENT

The satisfaction and euphoria that accompany the successful completion of any activity would be
incomplete without mentioning the people who made it possible, whose constant guidance and
encouragements crowned our efforts with success. I take this opportunity to express the deepest
gratitude and appreciation to all those who held me directly or indirectly towards the successful
completion of Project Phase II.

We would like to convey our heartiest regard and thanks to project guide Ms. Sunitha M, Assistant
Professor, Department of ECE, for her valuable suggestion and constant encouragement that she
provided at every moment we needed it during Project Phase II.

We would also like to thank Dr. Nagesh K N, Professor & Head of Department of ECE for his
parallel guidance to complete our Project Phase II.

We would like to thank Dr. B V Ravishankar, Principal, NCET, Bengaluru, for lending his help
for the completion of our Project Phase II.

We would like to thank Project Coordinator Dr. Ajay Kumar Dwivedi, Associate Professor,
Department of ECE, for his valuable suggestion for the completion of our Project Phase II.

Gratitude should also be conveyed to the staff of the Department of ECE, NCET for their full
cooperation.

iv
iii
ABSTRACT

vi
Recognition of an object is a bare minimum restraint for an individual in order to sort out or classify the type
of the object. This situation becomes a tricky experience with respect to the blind persons; therefore, to assist
visually challenged persons, in particular, while recognizing the staircases and manholes, a prototype of
mobility recognition is presented using the feature vector identification and sensor computed processor
Arduino chips.
This prototype provides more sovereignty to the sightless people while walking on the roads and helps to pass
through on their own without any backing. This prototype is developed using Arduino kit along with feature
detection module and helps the visually challenged in reaching their destinations with ease. A low weight stick
is built to facilitate the visually challenged people toward effective recognition of the obstacles. In order to
recognize the manholes, the chip is programmed and embedded in the stick that also holds the code for
detection of the staircases based on a bivariate Gaussian mixture model; speeded up robust features algorithm
is considered for extraction of features. The developed model shows an accuracy of around 90% for manhole
detection and 88% for staircase detection.
 TABLE OF CONTENTS

Chapter CHAPTER NAME PAGE NO

No
Certificate ii

Acknowledgment iii

Abstract iv

Table of Contents V

List of Figures vi

List of Tables vii

Introduction 1-2
Chapter 1 Problem Statement Objective

3-7
Chapter 2 Literature survey

8-11
Chapter 3 Methodology/Working Model

12-20
Chapter 4 Block Diagram/Flow Chart

21-22
Chapter 5 Results and Discussion

23-25
References

viii
A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

LIST OF FIGURES

FIGURE PAGE NO
NO FIGURE NAME
1 ESP8266 13

2 Ultrasonic Sensor 14

3 Vibrator 15

4 Arduino 16

5 Tilt Sensor 17

6 ESP8266 Firmware 18

7 Blynk application 18

8 Mobile App 19

9 Cloud Server 19

CHAPTER-1

INTRODUCTION

Moving around unfamiliar surroundings is an exigent task for the visually impaired persons due to their
deficiency in eyesight. According to the latest survey, around 1.5 million people in India are visually
challenged and are facing a hardship in performing their day-to-day activity. This population is about
170 million worldwide and as per the latest statistics, this number is increasing by 10% annually.
Staircases pose a major problem in navigation, and their recognition is of prime importance for the

DEPT.ECE, NCET, Bengaluru Page ix

visually impaired. Many kinds of sensors, like monocular and stereo cameras, depth sensors or laser
scanning devices (e.g., LiDAR), have been used for detecting staircases.

Monocular image-based methods generally detect staircases by detecting non-ground plane regions and
identifying pattern of concurrent lines resembling staircases in those regions. For example, proposed a
Gabor filter-based texture detection method to detect distant staircases. They estimated the staircase pose
by homograph search model. extended this using motion stereo. detected staircases only from RGB
image by performing Hough transform to extract concurrent parallel lines in an edge map image
resulting from Sobel operator. optimally detected staircase in real time using stereo imagery to estimate
ground plane and temporal consistency, but did not classify up/down stairs separately. incorporated a
stereo camera into the white cane and used actuators for guidance and distance feedback. proposed an
RGBD image-based detection approach of stairs, pedestrian crosswalks, and traffic signs, which
achieved decent detection rate in the staircase detection, but did not handle the escalator detection.
However, some items which contain parallel lines such as bookshelves could be misclassified as stairs
and pedestrian crosswalks. presented an efficient optical flow-based stair tracking algorithm using
Hough transform for parallel line detection and a SVM Classifier to detect indoor staircases and
recognize the types of staircases by using RGBD videos captured via RGB-D camera mounted at the
chest position. used array of Lidars and vibro-tactile units to detect obstacles and features including
up/down staircases. The system is time consuming to get used to by a visually impaired user and needs
to adapt to the rate of motion of the user.

In this project we propose a hybrid approach for detecting manholes and staircases by using both sensor
and image based algorithms. We present a novel algorithm to detect and classify upward and downward

x
A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

staircase from high quality frames extracted from a video stream of a mobile camera using Bivariate
Gaussian Mixture Model that takes as an input staircase shape and distance features extracted using the
SURF algorithm. The entire processing happens on the smart- phone without need of heavy computation
devices or high speed 3G/4G connectivity for cloud computation. We use an array of 3 sonars (front,
right and left) mounted on a white cane and managed by Arduino processor to help detect
manholes and also aid in staircase navigation. We further use median based threshold using Gaussian
Mixture Model for precise manhole identification. We employ vibro-feedback mounted on the cane to
warn the user of the obstacle as well as connect to smartphone via Bluetooth for audio feedback. Overall
the system is lightweight and cost effective.

1.1 Problem Statement

• Blind people can’t easily recognize obstacles or stairs while using normal blind stick.

• No safety features on the normal blind stick.

• Can’t locate the location of the normal blind stick user when they are having an emergency
problem or lost in a public area.

1.2 Objectives

• To develop stick can be assist the visually impaired persons to navigate safely in both indoor and
outdoor by detecting the obstacle in 3 meters range.
• Provide system with low cost, reliability, and to be affordable and simple
• System is equipped with sensors and other processor we try to reduce the PVC stick weight for
other convenient which may around 600 grams with low cost reliability.
• To provide audio signal alert about each manhole and recognize both ascending and descending
staircases.

CHAPTER-2

Literature Survey

DEPT.ECE, NCET, Bengaluru Page 2

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

X. Lu et.al [2000] Designed a ‘‘Detection and localization of curbs and stairways using stereo vision’’
presented algorithms to detect and precisely localize curbs and stairways for autonomous navigation.
These algorithms combine brightness information (in the form of edgels) with 3-D data from a
commercial stereo system. The overall system (including stereo computation) runs at about 4 Hz on a 1
GHz laptop. We show experimental results and discuss advantages and shortcomings of our approach
[6].

J.-S. Gutmann et.al [2004] propose an article “Stair climbing for humanoid robots using stereo vision”
Described a new small humanoid type robot, SDR-4X, for entertainment purpose. Comparing the
previous version of SDR-3X, we further expand its ability of human interaction and robust adaptability
in home environment. The main technical feature of SDR-4X is four folds a real time integrated
Adaptive Motion
Control, the “SDR Motion Creator”, for development of SDR’s attractive motion performances, “Real
time real-world Space Perception”, “Multimodal Human Robot Interaction”. Overview of these
technologies and its performance are presented in this paper [1].

J. V. Bouvrie et.al [2007] Discovered Theory of “Visual object concept discovery’’ The research
presented in this article Comprises two complementary components: in the first half, we describe an
experimental field study from “Project Prakash.” A new humanitarian and scientific initiative launched
in India, Project Prakash involves a systematic study of the development of object -perception skills in
children following recovery from congenital blindness. In the latter half of the paper, we introduce a
computational model for visual object concept discovery. Through this computational investigation, we
seek to illuminate from a different angle a different angle the process by which humans build object
models. Constructing a machine analog of this process, however, is a difficult computational challenge
given that natural images are often complex and contain many attributes irrelevant to the concept that
needs to be extracted [16].

Y. Cong et.al [2008] created ‘‘A stairway detection algorithm based on vision for UGV stair climbing’’
presented a vision based algorithm used to guide the unmanned ground vehicles (UGV) for autonomous
stairways climbing and implement it on UGV successfully. The reliability of guiding UGV to climb
stairs requires evaluating two offset parameters: the position of vehicle on stairs and the orientation

DEPT.ECE, NCET, Bengaluru Page 3

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

angle to stairs. The intention of our algorithm is to estimate these two parameters through extracting the
stair edges robustly. To achieve this goal, we apply the Gabor filter to eliminate the influence of the
illumination and keep edges, and propose a fast method to remove small lines. Finally we link stair
edges, and estimate the offset parameters used to steer the vehicle by RANSAC algorithm. Experiments
on various stairways including indoor and outdoor are given in various light conditions. The results
validate our algorithm [2].

J. A. Hesch et.al [2010] published an article ‘‘Descending-stair detection, approach, and traversal with
an autonomous tracked vehicle’’ The strategy for descending-stair detection, approach, and traversal
using inertial sensing and a monocular camera mounted on an autonomous tracked vehicle. At the core
of our algorithm are vision modules that exploit texture energy, optical flow, and scene geometry (lines)
in order to robustly detect descending stairwells during both far- and near-approaches. As the robot
navigates down the stairs, it estimates its three-degrees-of-freedom (d.o.f.) attitude by fusing rotational
velocity measurements from an on-board tri-axial gyroscope with line observations of the stair edges
detected by its camera. We employ a centering controller, derived based on a linearized dynamical
model of our system, in order to steer the robot along safe trajectories. A real-time implementation of the
described algorithm was developed for an iRobot Packbot, and results from real-world experiments are
presented
[4].

D. C. Hernandez et.al [2011] Dectected a “Stairway tracking based on automatic target selection using
directional filters” proposed to detect the localization and recognition of an indoor stairway. This is a
fundamental step in the implementing of autonomous stair climbing navigation, as well as the
implementing of passive alarm systems intended for the blind and visually impaired. Both of these
systems must be able to recognize parameters that can describe stairways in unknown environments.
This method analyzes the edges of a stairway based on planar motion tracking and directional filters. We
extracted the horizontal edge of the stairs by using the Gabor Filter. From the specified set of horizontal
line segments, we extracted a hypothetical set of targets by using the correlation method. Finally we are
proposing the use of the discrimination method to find the ground plane, using the behavioral distance
measurement [3].

DEPT.ECE, NCET, Bengaluru Page 4

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

G. A. Kumar et.al [2015] proposed an article “Naveye a guiding system for blinds” This is an efficient
methodology for aiding the blind and the visually impaired in their navigation. Track manager is
basically a pair of shoes along with a remote control, and an ear phone. It works in two modes in a
domestic environment. The study hypothesizes a smart walking stick that alerts visually-impaired people
over obstacles, pit and water in front could help them in walking with less accident. It outlines a better
navigational tool for the visually impaired. It consists of a simple walking stick equipped with sensors to
give information about the environment. GPS technology is integrated with preprogrammed locations to
determine the optimal route to be taken. The user can choose the location from the set of destinations
stored in the memory and will lead in the correct direction of the stick. In this system, ultrasonic sensor,
pit sensor, water sensor, GPS receiver, level converter, driver, vibrator, voice synthesizer, keypad,
speaker or headphone, The overall aim of the device is to provide a convenient and safe method for the
blind to overcome their difficulties in daily life [17].

R. Munoz et.al [2016] created a ‘‘Depth-aware indoor staircase detection and recognition for the
visually impaired’’Impaired to travel independently, especially in unfamiliar environments. Despite
many effective navigation algorithms having been developed in recent decades, accurate, efficient, and
reliable staircase detection in indoor navigation still remains to be a challenging problem. In this paper,
we propose an effective indoor staircase detection algorithm based on an RGB-D camera. The
candidates of staircases are first detected from RGB frames by extracting a set of concurrent parallel
lines based on Hough transform. The complement depth frames are further employed to recognize the
staircase candidates as upstairs, downstairs, and negatives (i.e., corridors). A support vector machine
(SVM) based multiclassifier is trained and tested for the staircase recognition with our newly collected
staircase dataset. The detection and recognition results [10].

K. B. Swain et.al [2017] created a ‘‘Arduino based automated STICK GUIDE for a visually impaired
person”, In this paper a stick guide model represented for visually impaired persons to guide in their
way, which consist of a Global Positioning System (GPS) and a Global System for Mobile
communication (GSM) modules along with sensors like Ultrasonic and Infrared sensors. This is a smart
stick that will make the visually impaired persons guiding their way. GPS module is used in this to get
the current location information of the person, that location will be sent via Short Message Service

DEPT.ECE, NCET, Bengaluru Page 5

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

(SMS) to the registered numbers using a GSM module, on pressing of a switch whenever he feels he is
lost. Ultrasonic sensors are used for obstacle detection through the ultrasonic waves produced by it, and
Infrared sensor is used for level detection and both the sensors are interfaced with the vibrator which
vibrates on detecting an obstacle. The main objective of this model is to help a blind to live a better
life[18].

R. K. Katzschmann et.al [2018] Designed a ‘‘Safe local navigation for visually impaired users with a
time-of-flight and haptic feedback device” The project presents ALVU (Array of Lidars and Vibrotactile
Units), a contactless, intuitive, hands-free, and discreet wearable device that allows visually impaired
users to detect low- and high-hanging obstacles, as well as physical boundaries in their immediate
environment. The solution allows for safe local navigation in both confined and open spaces by
enabling the user to distinguish free space from obstacles. The device presented is composed of two
parts: a sensor belt and a haptic strap. The sensor belt is an array of time-of-flight distance sensors worn
around the front of a user's waist, and the pulses of infrared light provide reliable and accurate
measurements of the distances between the user and surrounding obstacles or surfaces. The haptic strap
communicates the measured distances through an array of vibratory motors worn around the user's upper
abdomen, providing haptic feedback [11].

M. Ashrafuzzaman et.al [2021] Created a “Design and Development of a Low-cost Smart Stick for
Visually Impaired People,” One of the biggest problem faced by visually impaired people is to navigate
from one place to another. They always need human support for moving either indoors or outdoors. The
unfavourable conditions of the environment make it more complicated for visually impaired people. A
blind person always needs to be alert to get off the situations like crashing with obstacles, holes,
staircases, slipping down wet territory. Also, in case of any emergency, they might want to send an alert
message to their families or friends nearly their location. Considering the above, in this paper, we have
addressed all these issues and provides a solution to assist visually impaired people so that they can live
without the much help of others. The Ultrasonic sensor of this system helps blind people to detect
obstacles, hole, and staircase alongside water sensor is used to detect the water[20].

R. K. Kaushal et.al [2022] Designed a “Smart Blind Stick for Visually Impaired People using IoT,” IoT
based Smart shoe system for the blind is a system made with the help of ultrasonic sensors paired to an
Arduino UNO board. Internet of things is all about making physical objects communicate with other

DEPT.ECE, NCET, Bengaluru Page 6

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

objects or even with humans. It is an enabling technology which has a rapid development and growth in
the market. In our India there are almost 40 million blind people among which 1.6 million are children.
Blind people face great difficulty to travel independently. They have to depend on others in many
aspects of their life. The Major problem is when they walk on the road. With a stick in hand they cannot
detect every obstacle that comes in their way. The Smart shoe design provides a longterm solution for
the blind to walk on roads independently. The smart shoe will help the Blind person to reach his
destination independently. It is built using IoT Technology in which the shoe will be embedded with
various sensors, Microcontroller and buzzers. The shoe warns the user by making noise with the buzzer
when he/she walks in front of an obstacle. To improve the efficiency, Smart glasses are designed using
IoT which is also embedded with sensors and helps in detecting the objects by covering a larger
area[19].

CHAPTER-3

METHODOLOGY/WORKING MODEL
3.1 Methodology:

DEPT.ECE, NCET, Bengaluru Page 7

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

The "Smart Stick" project uses a combination of hardware and software components to provide visually
impaired individuals with a device that can help them navigate their environment more safely and
independently. The methodology for the project involves several steps:

Hardware Setup: The first step is to assemble the hardware components, which include the ESP8266,
two ultrasonic sensors, tilt sensor, buzzer, GPS module, and battery. The components are connected
according to a predefined wiring diagram.

Software Development: The second step is to develop the software for the device, which includes
firmware for the ESP8266, a mobile app using the Blynk application, and a cloud server for remote
monitoring and control. The firmware is developed using the Arduino IDE and programmed onto the
ESP8266.

Testing: The third step is to test the device to ensure that it is functioning correctly. The device is tested
in different environments and conditions to evaluate its performance and accuracy.

Calibration: The fourth step is to calibrate the sensors to ensure that they are providing accurate data.
This involves adjusting the sensitivity and range of the ultrasonic sensors and tilt sensor to match the
user's stride and gait.

User Interface Development: The fifth step is to develop a user-friendly interface for the device using
the Blynk application. The interface provides the user with real-time feedback on obstacles, location, and
other device parameters
.
Deployment: The final step is to deploy the device to visually impaired individuals for testing and
feedback. The feedback is used to further refine and improve the device.

DEPT.ECE, NCET, Bengaluru Page 8

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

Fig 3.1.1 Estimation of manholes detection

Fig 3.1.2 Manhole Detection Result

DEPT.ECE, NCET, Bengaluru Page 9

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

3.2 Working Model

We use a strategy to identify both ascending and descending staircases – tilt sensor strategy for
identifying staircases and ultrasonic sensor based strategy for actually navigating through the staircase. a
surf based staircasedetection Since we use the camera on the mobile device, we rely upon monocular
vision based approaches rather than stereo or depth sensor. In order to sense the background, Monocular
Vision Based Approaches use techniques like texture segmentation, motion and ground plane
recognition, inverse edge detection and identifying concurrent parallel lines, etc. However, in practice,
ground plane detection is mostly preferred because of its simple features. Ground plane detection,
segments every image into two partitions based on the features as ground and non-ground. The non-
ground planes are further categorized into staircases or objects.

The image is separated into two regions: ground and nonground, by using the segmentation process.
The possibilities of additional obstacles are checked in the non-ground region. Feature extraction
process is utilized for obstacle checking. Many feature extraction models are listed in the literature,
among these methods, in this article we have considered the SURF (Speeded up Robust Features)
method because of the advantages like illumination, rotation, scale and computational simplicity. The
features are extracted based on the attributes like blobs, corners, etc. The features of non-ground image
features are considered for the differentiation of ascending staircases and descending staircase.
Describes the basic steps used in staircase detection, which includes video as an input and then
preprocessing is done to select the high quality frames followed by segmentation and feature
extraction. Segmentation divides image into two parts: ground plane and non-ground plane. Here
nonground plane will be checked further for the ascending stair case and descending stair cases. The
shape of the steps and distance are considered as features, which we extract using SURF algorithm.
These features are useful to decide whether a staircase is ascending or descending. These features are
given as the input for the Bivariate Gaussian Mixture Model, that estimates the probability density
function (PDF). The PDF of both ascending and descending staircases is also considered and based on
the values of the PDF, the peak is estimated and the alert is generated and is passed as a message to the
visually impaired.

Classification Rates for Detection of Staircases And Floor

DEPT.ECE, NCET, Bengaluru Page

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

10

Fig 3.2.1 Classification Rates for Detection of Staircases And Floor

DEPT.ECE, NCET, Bengaluru Page

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

11

CHAPTER-4

BLOCK DIAGRAM/FLOW CHART

4.1 Block Diagram:

DEPT.ECE, NCET, Bengaluru Page 12

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

Hardware components:

Esp8266

The ESP8266 is a low-cost, Wi-Fi enabled microcontroller that serves as the brain of the Smart Stick. It
provides the processing power and connectivity required to operate the device. The chip was popularized
in the English-speaking maker community in August 2014 via the ESP-01 module, made by a third-party
manufacturer Ai-Thinker. This small module allows microcontrollers to connect to a Wi-Fi network and
make simple TCP/IP connections using Hayes-style commands. However, at first, there was almost no
English-language documentation on the chip and the commands it accepted.[2] The very low price and the
fact that there were very few external components on the module, which suggested that it could eventually
be very inexpensive in volume, attracted many hackers to explore the module, the chip, and the software
on it, as well as to translate the Chinese documentation.

Fig 4.1.1 Esp8266

Ultrasonic sensor

These are largely used to capture the signals more swiftly, the signals from the ground level ultrasonic
sensors are converted into time domain using the feature transformation. The majority of the literature in
this direction depicts various models based on different sets of the ultrasonic sensor and extract the
signals captured using both time domain and the features extracted for identification of the object. The
time domain helps to capture principle components of the segment and helps to process the signal.

DEPT.ECE, NCET, Bengaluru Page 13

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

Confined to the identification of correlation coefficient by the time domain, a signal can be made to
sensitize and thereby help towards the replacement of the objects. However, compared to the time
domain, frequency domain, acquired signals will always give better results because of the fact that they
take the statistical parameters into consideration such as movements, segments, etc. Most of the research
work in this area is based on Neural Networks which can in particular help to identify the 2D and 3D
shape of the objects, independent of orientation and based on the echoes generated by ultrasonic sensors.
With this advantage, some researchers have highlighted the usage of ultrasonic transducers for
identifying the simple objects such as a cylinder, edge, etc. Works are also highlighted based on sonar
based, particularly during the cases of corners plains and edges recognition.

Fig 4.1.2 Ultrasonic Ranging Model HC-SR04

Ultrasonic sensors are mostly considered for identifying the distance; the technical specifications of this
sensor are given below
• Operating Voltage: 5V
• Static current: 2mA max

• Induction Angle: 15◦

• Detection Range: 2 – 400cm
• High precision up to 3mm

DEPT.ECE, NCET, Bengaluru Page 14

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

Vibrator

A vibrator is a mechanical device to generate vibrations and is mainly used to generate vibrations when
the blind person is also a deaf. The vibration is often generated by an electric motor with an unbalanced
mass on its driveshaft.

Fig 4.1.3 Vibration Motor,3V ,150Ma, 10,000 rpm

We use VM 0610 A 3.0 Vibrator motor operating on 3V 150mA, capable of 10,000 rpm and measures
only 6mm in length. It is user-friendly motor, which can be used with a variety of products. This
includes pagers, GPS devices, mobile phones or even toys.

Sensor range

The sensors that are considered for this article can identify the objects within the range of 3 meters and
are capable of identifying the objects towards the angle of 90 degrees. In the present article, the
recognition of an object is carried out by estimating the distance between the sensors. The distance is
identified using the two sensors in such a way that if the value of the first sensor is higher than the other
sensor it is estimated as an object. The ultrasonic sensors are attached to the PVC pipe to enhance the
capturing scenarios and low pass filters are used for this purpose. Several features can be extracted using
these sensors, such as mean and standard deviation. Courtesies helps to estimate the ascending and
descending steps more accurately. Several researchers have employed these sensors to augment the
functionality of white cane used by visually challenged. The works presented are mostly focused
towards the detection of the object along with the distance, without giving the importance about the

DEPT.ECE, NCET, Bengaluru Page 15

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

nature of the object, which in certain cases may be more significant to the user. The majority of the
works developed in this regard are confined towards the identification of the objects basing on statistical
approaches, where the data is classified into four types namely edge, plane, corner and cylinder . Some
systems were also developed with an aim to identify the activity of the people based on the trajectory
that helps to classify the scenes as the indoor or outdoor. Moreover, these systems cannot identify the
manholes and descending staircases exactly.

Arduino

It is an open source software, which is developed by using single-board microcontrollers and

microcontroller kits during the development of digital devices and interactive objects, which can sense
and control objects. Arduino boards are designed based on an array of microprocessors and controllers.
These boards are embedded with a set of pins which are primarily used for the digital input and analog
output (I/O). These pins can be interfaced with the variety of expansion boards (shields) and other
circuits for establishing both serial communications interfaces, along with Universal Serial Bus (USB).
The preprogrammed boot loader is mostly used in Arduino microcontrollers which assist during the
loading of programs to the on-chip flash memory. A serial connection is considered while loading the
programming codes to another computer.

Fig 4.1.4 Arduino

Hardware Features

• Microcontroller: ATmega328
• Operating Voltage: 5 V

DEPT.ECE, NCET, Bengaluru Page 16

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

Tilt Sensor

The Smart Stick uses a tilt sensor to detect when the device is tilted. This is important because it allows
the device to detect when the user is going up or down stairs or ramps and adjust the output accordingly.
There are different types of tilt sensors and they have slightly different working principles.Simple tilt
sensors work by using a metallic ball connecting two pins, moving within the sensor. The ball moves
position as the sensor tilts which connects the circuit turning the sensor "on" or "off".More complex tilt
sensors use an internal gyroscope which measures the direction of the pull from gravity to determine the
orientation of the device.Our Positek tilt sensors use their technology to measure tilt. Two damped
pendulum vanes have positioned on either side of an inductive differential coil. The vanes stay stationary
when the sensor is rotated which changes the inductance of the coil. The electronics then convert this into
the output of choice.

Fig 4.1.5 Tilt sensor

Software components:

ESP8266 Firmware
The firmware running on the ESP8266 is responsible for processing the data from the sensors and GPS
and controlling the output of the device. The firmware is written in C/C++ and can be programmed using
the Arduino Integrated Development Environment (IDE).

DEPT.ECE, NCET, Bengaluru Page 17

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

Fig 4.1.6 ESP8266 Firmware

Blynk Application
The Blynk application is a cloud-based service that allows the Smart Stick to communicate with a mobile
app. The app can be used to configure the device, monitor its status, and receive alerts.

Fig 4.1.7 Blynk Application

Mobile App
The mobile app is a custom-built application that provides a user-friendly interface for controlling and
monitoring the Smart Stick. The app can be used to configure the device, set up alerts, and view the
device's location on a map.

Fig 4.1.8 Mobile app

DEPT.ECE, NCET, Bengaluru Page 18

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

Cloud Server

The Smart Stick is connected to a cloud server that stores data and provides a platform for the Blynk
application to communicate with the device. The cloud server also enables remote monitoring and control
of the Smart Stick.

Fig 4.1.9 Cloud Server

DEPT.ECE, NCET, Bengaluru Page 19

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

4.2 flow chart:

A Hybrid Approach For Identification Of Manhole And Staircase To Assist Visually Challenged

DEPT.ECE, NCET, Bengaluru Page 20

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

CHAPTER-5

RESULTS

The developed stick can be more helpful to the visually impaired to navigate safely both indoor and
outdoor. It yields excellent results in sensing the obstacle on the path of the user up to a range of three
meters. The key features of the proposed system are low cost, reliability, and portability. Despite the fact
that the system is equipped with sensors and other. Inputs of manholes. Segmented Outputs of Manholes
using GMM. Manhole detection using the sensors mounted on the stick. Classification rates for detection
of staircases and floor. components, the PVC stick utilized for this work weighs around 600 grams.
During the experimentation process, it is observed that, whenever the sensor identifies a hollow signal,
sensor gets activated and generates an audio signal announcing that a manhole is ahead. The
experimentation is repeated using GMM.

DEPT.ECE, NCET, Bengaluru Page 21

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

DISCUSSION

This Project Phase 1 report has discussed for the identification of the manholes and staircases for the
visually impaired people using threshold-based system from data obtained through unique arrangement
and placement of ultrasonic sensors The recognition accuracy of the developed models is tested against
distance estimation and recognition towards the identification of ascending and descending staircases.
The performance varies when we are facing an ascending staircase or a descending one. Due to the
direction of the sensor, standing before a descending staircase permits us to have an overview of the
entire staircase, however, due to the self-overlapping of successive steps, there will be some deviations
with respect to the measurements. Relatively, in case of ascending staircases, the proportion of
uncovering reduces, since the steps stay very much close to the individual during mounting the steps and
hence the recognition rate increases. As an extension to the work, a system need to be developed that
helps the visually challenged by providing routes, mental maps which may be adequate to direct the
person to the destination without any need of the navigation aid. It is also necessary to develop the
system in such a way that the depth of the water can be identified using the sensors. Such system can
help visually challenged to walk through during the rainy days.

REFERENCES:

DEPT.ECE, NCET, Bengaluru Page 22

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

[1] J.-S. Gutmann, M. Fukuchi, and M. Fujita, ‘‘Stair climbing for humanoid robots using stereo vision,’’
in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., vol. 2, Sep./Oct. 2004, pp. 1407–1413.

[2] Y. Cong, X. Li, J. Liu, and Y. Tang, ‘‘A stairway detection algorithm based on vision for UGV stair
climbing,’’ in Proc. IEEE Int. Conf. Netw. Sens. Control (ICNSC), Apr. 2008, pp. 1806–1811.

[3] D. C. Hernández and K.-H. Jo, ‘‘Stairway tracking based on automatic target selection using
directional filters,’’ in Proc. 17th Korea-Jpn. Joint Workshop Frontiers Comput. Vis. (FCV), Ulsan,
South Korea, Feb. 2011, pp. 1–6.

[4] J. A. Hesch, G. L. Mariottini, and S. I. Roumeliotis, ‘‘Descending-stair detection, approach, and

traversal with an autonomous tracked vehicle,’’ in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst.,
Taipei, Taiwan, Oct. 2010, pp. 5525–5531.

[5] S. Se and M. Brady, ‘‘Vision-based detection of stair-cases,’’ in Proc. 4th Asian Conf. Comput. Vis.
(ACCV), 2000, pp. 535–540.

[6] X. Lu and R. Manduchi, ‘‘Detection and localization of curbs and stairways using stereo vision,’’ in
Proc. Int. Conf. Robot. Automat., Apr. 2000, pp. 4648–4654.

[7] Y. H. Lee, T.-S. Leung, and G. Medioni, ‘‘Real-time staircase detection from a wearable stereo
system,’’ in Proc. 21st Int. Conf. Pattern Recognit. (ICPR), Tsukuba, Japan, Nov. 2012, pp. 3770–
3773.

[8] D. Kim, K. Kim, and S. Lee, ‘‘Stereo camera based virtual cane system with identifiable distance
tactile feedback for the blind,’’ Sensors, vol. 14, no. 6, pp. 10412–10431, 2014.

[9] S. Wang and Y. Tian, ‘‘Detecting stairs and pedestrian crosswalks for the blind by RGBD camera,’’ in
Proc. IEEE Int. Conf. Bioinf. Biomed. Workshops, Oct. 2012, pp. 732–739.

[10] R. Munoz, X. Rong, and Y. Tian, ‘‘Depth-aware indoor staircase detection and recognition for the
visually impaired,’’ in Proc. IEEE Int. Conf. Multimedia Expo Workshops (ICMEW), Seattle, WA,
USA, Jul. 2016, pp. 1–6.

[11] R. K. Katzschmann, B. Araki, and D. Rus, ‘‘Safe local navigation for visually impaired users with a
time-of-flight and haptic feedback device,’’ IEEE Trans. Neural Syst. Rehabil. Eng., vol. 26, no. 3, pp.
583–593, Mar. 2018.

DEPT.ECE, NCET, Bengaluru Page 23

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

[12] D. C. Hernández, T. Kim, and K.-H. Jo, ‘‘Stairway detection based on single camera by motion
stereo,’’ in Modern Approaches in Applied Intelligence. Berlin, Germany: Springer, 2011, pp. 338–
347.

[13] M. Yashas, K. S. Maalik, M. Z. M. Ashfaq, K. P. S. H. De Silva, and S. Ragulan, ‘‘Blind guider: An

IT solution for visually impaired people,’’ Int. J. Sci. Res. Pub., vol. 7, no. 11, pp. 700–703, 2017.

[14] M. A. Ikbal., F. Rahman, M. R. Ali, M. H. Kabir, and H. Furukawa, ‘‘Smart walking stick for blind
people: An application of 3D printer,’’ in Proc. SPIE 10167, Nanosensors, Biosensors, Info-Tech
Sensors 3D Syst., Apr. 2017, p. 101670T.

[15] V. Singh et al., ‘‘Smart’ cane for the visually impaired: Design and controlled field testing of an
affordable obstacle detection system,’’ in Proc. 12th Int. Conf. Mobility Transp. Elderly Disabled
Persons (TRANSED), vol. 53, 2010, pp. 1689–1699.

[16] J. V. Bouvrie and P. Sinha, ‘‘Visual object concept discovery: Observationin congeniality blind
children, and a computational approach,’’ Neurocom-puting, vol. 70, pp. 2218–2233, Aug. 2007.

[17] G. A. Kumar, P. Ramakrishna, and M. Srinivasulu, ‘‘Naveye a guiding system for blinds,’’ Int. J.
Innov. Technol., vol. 3, no. 4, pp. 593–598, 2015.

[18] K. B. Swain, R. K. Patnaik, S. Pal, R. Rajeswari, A. Mishra and C. Dash, "Arduino based automated
STICK GUIDE for a visually impaired person," 2017 IEEE International Conference on Smart
Technologies and Management for Computing, Communication, Controls, Energy and Materials
(ICSTM), Chennai, India, 2017, pp. 407-410.

[19] R. K. Kaushal, K. Tamilarasi, P. Babu, T. A. Mohanaprakash, S. E. Murthy and M. J. Kumar, "Smart

Blind Stick for Visually Impaired People using IoT," 2022 International Conference on Automation,
Computing and Renewable Systems (ICACRS), Pudukkottai, India, 2022, pp. 296-300.

[20] M. Ashrafuzzaman, S. Saha, N. Uddin, P. K. Saha, S. Hossen and K. Nur, "Design and Development
of a Low-cost Smart Stick for Visually Impaired People," 2021 International Conference on Science &
Contemporary Technologies (ICSCT), Dhaka, Bangladesh, 2021, pp. 1-6

[21] J.V.BouvrieandP.Sinha,‘‘Visualobjectconceptdiscovery:Observation in congeniality blind children,

and a computational approach,’’ Neurocom- puting, vol. 70, pp. 2218–2233, Aug. 2007.

[22] A.Agarwal,D.Kumar,andA.Bhardwaj,‘‘Ultrasonicstickforblind,’’Int. J. Eng. Comput. Sci., vol. 4, no.

4, pp. 11375–11378, 2015.

DEPT.ECE, NCET, Bengaluru Page 24

A hybrid approach for identification of manhole and staircase to assist visually challenged 2022-2023

[23] Ulrich and J. Borenstein, ‘‘The GuideCane-applying mobile robot technologies to assist the visually
impaired,’’ IEEE Trans. Syst., Man, Cybern. A, Syst. Humans, vol. 31, no. 2, pp. 131–136, Mar. 2001.

[24] G. A. Kumar, P. Ramakrishna, and M. Srinivasulu, ‘‘Naveye a guiding system for blinds,’’ Int. J.
Innov. Technol., vol. 3, no. 4, pp. 593–598, 2015.

[25] M. Brown, ‘‘Feature extraction techniques for recognizing solid objects with an ultrasonic range
sensor,’’ IEEE J. Robot. Autom., vol. RA-1, no. 4,pp. 191–205, Dec. 1985

[26] Z.Zhang,Y.Huang,C.Li,andY.Kang,‘‘Monocularvisionsimultaneous localization and mapping using

SURF,’’ in Proc. IEEE 7th World Conf.Intell. Control Automat. (WCICA), Jun. 2008, pp. 1651–1656.

[27] Pasquet et al., ‘‘Detection of manhole covers in high-resolution aerial images of urban areas by
combining two methods,’’ IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 9, no. 5, pp.
1802–1807, May 2016.

DEPT.ECE, NCET, Bengaluru Page 25