Indoor Object and Surface Recognition System for

Visually Impaired People

Paulo Marcos

June 18, 2021

Abstract. This research proposes to develop a system able to detect objects in

an indoors environment for the visually impaired using a static camera. Several
existing solutions require the user to hold or wear gadgets to make the product
viable, however such approaches can take some time to be mastered by the vi-
sually impaired due to their limited vision. The research proposal focuses on
computer vision techniques such as object recognition and surface detection to
potentially reach its objectives.

1. Introduction
Visually impaired people have limited ability to perform daily activities such as finding
and recognizing objects, indoors and outdoors navigation and nonverbal information un-
derstanding. 285 million people worldwide are visually impaired [13], which consists in
people who have limited or no sight; of these 39 million are blind with visual acuity of
less than 6/120 in the better eye [12].
It is common for the visually impaired to have specific places to put objects in
order to find them whenever they need. However, a person who has limited vision can
take some time to find an object that is comparably close but not in its correct place due
to the nature of their disability, for example when an object fall from a table or into the
sink or is misplaced by someone else. Although caregivers may be available, a substantial
part (26% in the United States [19]) of visually challenged adults live alone and a bigger
portion doesn’t have assistance at all times. For this reason, several researches have been
developed aiming to comfort and assist the partially or fully blind people.
Technology has been of great assistance to these individuals, providing web access
[9], object avoidance [17] and several other vision substitution abilities. Recently, one of
the most viable areas to help partially sighted and blind people is computer vision since it
is able to provide convenient solutions at a relatively low cost.
For instance, a prototype system called Trinetra [8] has been developed to help
users recognize supermarket objects through barcodes. However. the camera has to be
aimed directly at the products, which can be hard for most users. An algorithm [15] has
been proposed to overcome this problem giving instructions to find the barcode through
left or right commands.
Another example is a stereo camera integrated in the front of a bicycle helmet used
for image analysis and distance measurement to recognize objects and track the movable
ones [3]. Two versions of this sensor module are available: a hand-guided one that can
be carried like a flashlight or cell phone and the one integrated into a bicycle helmet. The
user can send queries concerning objects or navigation to the system through keyboard or
cellphone keyboard depending on which case.
In contrast with several existing solutions this research intends to propose a system
that will serve as a guide to the user without the necessity of being worn or held. This
system in an indoors environment aims to detect objects and surfaces using a camera
attached near the ceiling to assist visually impaired people in their regular activities such
as cooking and cleaning. Input and output such as voice commands or sounds for the
mentioned system will not be focused on the proposed research.

2. Objectives and Future Research

This study proposes a system able to support visually impaired people in their indoor
regular activities using computer vision techniques. Recently, different solutions to help
visually impaired people in recognizing generic objects in an indoors environment have
been developed, some using tag-based approaches [6], others using 3D [4, 7] or 2D mod-
eling [2, 14]. Among these systems, a large portion of them requires the user to wear
special glasses or other gadgets to fulfill the purpose of the invention. This research will
focus on developing a system that is able to detect objects and surfaces such as of tables,
sinks and ovens using a static camera placed near the ceiling to cover most of the area and
thus assist the visually impaired in their indoor activities.
Through computer vision some points are to be considered in our research pro-
posal using a static camera: (i) object recognition to determine what is an object in the
image source, (ii) surface detection to determine parallel surfaces to the ground and thus
to be able to detect, for example, tables, sinks and ovens, (iii) and an algorithm to deter-
mine the boundaries of the surfaces and objects within an image to confirm whether an
object is in risk of falling.
Many ways to achieve object recognition are available. One specific should be first
studied due to its large and accessible bibliography: foreground detection [10], which is
commonly used in many visual surveillance applications including object recognition and
anomaly detection. Other methods will be covered in the study as well and compared to
the one mentioned above to determine the best result in the context.
Surface detection has been used in a variety of applications such as augmented
reality [18]. In this research it will be used to define surfaces such as of tables to serve
as a boundary for objects, hence diminishing the risk of them falling to the ground by the
visually impaired.
In the perspective of the visually impaired user this research may be able to serve
as a secondary vision ability system that can provide object detection and confirm whether
the objects are in their common place. In addition, items that are near the edges of a
surface and may possibly be knocked off into the floor will also be considered in the
research.
More importantly, this research can have an impact on how researchers may apply
their knowledge to improve lives of people with disabilities in general, focusing on the vi-
sually impaired to exhibit the necessity of further research to solve a set of both computer
vision and visual impairment problems.
The complete system that is able to successfully handle human-computer interac-
tion through input and output is left to future research. A possible outcome of this research
is an intelligent system that is able to not only detect whether objects are in the scene but
what objects those are, serving as a full detection system that can improve greatly the lives
of partially sighted or fully blind people. For instance, users might use voice commands
such as “where is my knife?” and the system would appropriately respond the exact loca-
tion of the object if it has been found, without the necessity of wearing special glasses or
holding smart phones and other tools.

3. Justification
Although more researches have been developed towards helping the elderly, people with
disabilities and specifically the visually impaired, there is a huge demand for new solu-
tions in those areas [1] to improve lives of those who have limited abilities due to their
physical conditions.
Recently, computer vision is being used to achieve relatively inexpensive products
that are able to recognize objects and obstacles and thus help visually impaired people in
several situations. The proposed research aims to study different techniques of computer
vision to serve as a secondary vision to users in the cases where they have to search for
visible objects in the space and to alert them whether an object is in risk of falling.
In addition, Japan’s enormous technology field and great advance in computer vi-
sion are well suited for the proposed study, since a wide range of laboratories are discov-
ering new particular solutions for different topics regarding image processing and object
recognition and tracking.

4. Methodology
In order to achieve the proposed objective the research will be divided into three sec-
tions. The first is object recognition through foreground detection in video sequences and
based on static images due to its simplicity and wide bibliography available. Comparisons
should be studied in order to completely understand the adequacy of this approach.
The second line of investigation is the pattern recognition to detect whether the
object is in the correct place. Simple techniques such as comparison between a referential
frame and the current frame are likely to be used, however the researcher will go through
alternative bibliographies to determine the best technique that suits this problem.
The third point is surface detection to recognize platforms such as tables where
objects are supposed to be on, to automatically detect if an object is in risk of falling by
standing near an edge. Firstly, the Plane Segment Finder [11] technique as well as its vari-
ations are going to be studied. From there different approaches may be used depending
on the type of camera used (stereo or monocular).
The researcher is willing to participate in events for the visually impaired and to
contact institutions regarding this disability in Japan to interview those individuals for
the improvement of the research, as well as to create or fortify bounds between visually
impaired people and the university.
 The choice of the techniques to be studied and the cameras to be used can be
modified according to the advising professor and the laboratory that the research is going
to be developed.

5. Research Plan
The following timetable is formatted for a 24 month study program. If the length of
the study in Japan changes based on the decision of the university, the timetable will be
appropriately readjusted.

Area of Study Month Description

1-2 Image Capture and Processing Techniques and Methods

Section 1:
3-6 Object Recognition Through Foreground Detection
Object Recognition Further Research on Alternative Object Recognition
7-8
Techniques
Section 2: 9-11 Referential Frame Comparison

Pattern Recognition 12-14 Alternative Methods for Pattern Recognition

Section 3: 15-16 Plane Segment Finder

Surface Detection 17-19 Alternative Approaches for Surface Detection

Section 4: Indoor Object and Surface Recognition System for Visually

20-24
Impaired People
Final Project

In addition to the above: As a former Science Without Borders student, the re-
searcher understands the importance of social and cultural engagements as a key aspect
of international exchange programs. Therefore, the researcher is motivated to help stu-
dents and the community in both Portuguese and English languages, as well as to share
experiences lived in Brazil and Canada whenever possible during his stay in Japan.

References
[1] The blind and the visually impaired challenge. http://www.aal-imago.eu/2252/0/
the-blind-and-visually-impaired-challenge. Accessed: 2015-05-15.

[2] R. Chincha and YingLi Tian. Finding objects for blind people based on surf features. pages 526–527. IEEE,
2011.

[3] D. Dakopoulos and N.G. Bourbakis. Wearable obstacle avoidance electronic travel aids for blind: A survey.
Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 40(1):25–
35, Jan 2010.

[4] Andreas Hub, Tim Hartter, and Thomas Ertl. Interactive tracking of movable objects for the blind on the
basis of environment models and perception-oriented object recognition methods. pages 111–118.
ACM, 2006.
 [5] Rabia Jafri, SyedAbid Ali, HamidR. Arabnia, and Shameem Fatima. Computer vision-based object recogni-
tion for the visually impaired in an indoors environment: a survey. The Visual Computer, 30(11):1197–
1222, 2014.

[6] Jafri, Rabia, Ali, Syed Abid, and Arabnia, Hamid R. Computer vision-based object recognition for the
visually impaired using visual tags. The 2013 international conference on image processing, computer
vision, and pattern recognition, pages 400–406, 2013.

[7] Y. Kawai and F. Tomita. A support system for visually impaired persons to understand three-dimensional
visual information using acoustic interface. In Pattern Recognition, 2002. Proceedings. 16th Interna-
tional Conference on, volume 3, pages 974–977 vol.3, 2002.

[8] P.E. Lanigan, A.M. Paulos, A.W. Williams, D. Rossi, and P. Narasimhan. Trinetra: Assistive technologies
for grocery shopping for the blind. In Wearable Computers, 2006 10th IEEE International Symposium
on, pages 147–148, Oct 2006.

[9] Andronico P. Leporini, B. and M. Buzzi. Designing search engine user interfaces for the visually impaired.
ACM SIGCAPH Computers and the Physically Handicapped, (76):17–18, 2003.

[10] Liyuan Li, Weimin Huang, I.Y.-H. Gu, and Qi Tian. Statistical modeling of complex backgrounds for
foreground object detection. Image Processing, IEEE Transactions on, 13(11):1459–1472, Nov 2004.

[11] K. Okada, S. Kagami, M. Inaba, and H. Inoue. Plane segment finder: algorithm, implementation and appli-
cations. In Robotics and Automation, 2001. Proceedings 2001 ICRA. IEEE International Conference
on, volume 2, pages 2120–2125 vol.2, 2001.

[12] World Health Organization. 10th Revision of the WHO International Statistical Classification of Diseases,
Injuries and Causes of Death.

[13] D. Pascolini and SP Mariotti. Global estimates of visual impairment: 2010. BRITISH JOURNAL OF
OPHTHALMOLOGY, 96(5):614–618, 2012.

[14] B. Schauerte, M. Martinez, A. Constantinescu, and R. Stiefelhagen. An assistive vision system for the blind
that helps find lost things. In Klaus Miesenberger, Arthur Karshmer, Petr Penaz, and Wolfgang Zagler,
editors, Computers Helping People with Special Needs, volume 7383 of Lecture Notes in Computer
Science, pages 566–572. Springer Berlin Heidelberg, 2012.

[15] E. Tekin and J.M. Coughlan. An algorithm enabling blind users to find and read barcodes. In Applications
of Computer Vision (WACV), 2009 Workshop on, pages 1–8, Dec 2009.

[16] J.R. Terven, J. Salas, and B. Raducanu. New opportunities for computer vision-based assistive technology
systems for the visually impaired. Computer, 47(4):52–58, Apr 2014.

[17] J. Zhang, S. K. Ong, and A. Y. C. Nee. Navigation systems for individuals with visual impairment: A
survey. In Proceedings of the 2Nd International Convention on Rehabilitation Engineering & Assistive
Technology, iCREATe ’08, pages 159–162, Kaki Bukit TechPark II,, Singapore, 2008. Singapore
Therapeutic, Assistive & Rehabilitative Technologies (START) Centre.

[18] Jianke Zhu and M.R. Lyu. Progressive finite newton approach to real-time nonrigid surface detection. In
Computer Vision and Pattern Recognition, 2007. CVPR ’07. IEEE Conference on, pages 1–8, June
2007.

[19] D.M Zuckerman. Blind adults in america: Their lives and challenges. In Washington, DC: National Center
for Policy Research for Women Families. 2004.