Deep Learning Assisted Smart Glasses as Educational

Aid for Visually Challenged Students

Hawra AlSaid, Lina AlKhatib, Aqeela AlOraidh, Shoaa AlHaidar, Abul Bashar

College of Computer Engineering and Sciences

Prince Mohammad Bin Fahd University
Al-Khobar, Saudi Arabia 31952
Email: abashar@pmu.edu.sa

Abstract— Computer Vision Technology has played a
significant role in assisting visually challenged people to carry
out their day to day activities without much dependency on
other people. Smart glasses in one such solution which enables
blind or visually challenged people to “read” images. This
paper is an attempt in this direction to build a novel smart glass
which has the ability to extract and recognize text captured from
an image and convert it to speech. It consists of a Raspberry Pi
3 B+ microcontroller which processes the image captured from
a webcam super-imposed on the glasses of the blind person.
Text detection is achieved using the OpenCV software and open
source Optical Character Recognition (OCR) tools Tesseract
and Efficient and Accurate Scene Text Detector (EAST) based
on Deep Learning techniques. The recognized text is further
processed by Google’s Text to Speech (gTTS) API to convert
to an audible signal for the user. A second feature of this
solution is to provide location-based services to the blind people
by identifying locations in an academic building using the RFID
technology. This solution has been extensively tested in a
university environment for aiding visually challenged students.
The novelty of the implemented solution lies in providing the
desired computer vision functionalities of image/text
recognition which is economical, small-sized, accurate and uses
open source software tools. This solution can be potentially
used for both educational and commercial applications.
Keywords: Image Recognition; Speech processing; Optical
Character Recognition; Deep Learning; Raspberry Pi; Python.
I. INTRODUCTION
In our societies, there are many people who are suffering
from different diseases or handicap. According to World Health
Organization (WHO), about 8% of the population in eastern
Mediterranean region has vision difficulties, which includes
blindness, low vision and some kind of visual impairment [1].
Such people need to be provided special facilities so that they
can live comfortably. Especially in the field of education, there
are special schools and universities for people with special
needs [2]. Most blind people and people with vision difficulties
were not in a position to complete their studies special schools
for people with special needs are not available everywhere and
most of them are private and expensive. So the only alternative
was that they study at home acquiring basic knowledge from
their parents. This education was not technical enough and
hence cannot compete with other people. There are different
levels of needs and not all levels require special places and
special schools. For instance, people with vision difficulties can
study with other students if they have an appropriate
environment. In order to solve this issue, we can use the help of
computer vision technology to make special aids which the
visually impaired people can live comfortably, as far as
possible.
It is observed that most blind people are intelligent and can
study if they have the chance to be able to study in regular
government administered schools as they exist almost
everywhere. It is a misconception among majority who think
people who are blind or with vision difficulties cannot live
alone and they need help of other people at all times. In fact,
they do not need help all the times, they can be independent
most of the times and they have the chance to live like other
people.
One of the popular solution in this scenario is to use Smart
Glasses for the visually impaired people [3]. These types of
glasses make the use of computer vision hardware and software
tools (camera, image processing, image classification and
speech processing). Such a solution gives a chance to visually
impaired people to lead a comfortable life with other people and
study in any school or university without the need of help from
other people every time. It has been observed that the use of
Smart Glasses has increased the percentage of educated people.
Most schools, colleges and universities are accepting students
with vision difficulties. It is expected that from next academic
year Prince Mohammad bin Fahd University (PMU) will accept
blind students for admission [4]. The college would like to start
using smart glasses for the first time in this setup and help
students to improve their education level with minimum
assistance from the instructor.
This was the motivation behind the design and
development of smart glasses is to help blind and visually
impaired students with their studies. These glasses are designed
to use the computer vision technology to capture an image and
extract English text and convert it into audio signal with the aid
of speech synthesis. Also, it was decided to add a feature of
translating text/words from English to Arabic language as the
majority of the students at PMU are Arabic speaking.
The main objectives of the proposed system can now be
summarized as the follows: capturing image, extracting text
from the image, identifying the correct text, converting text to
speech, translate the text to other language, to integrate the

978-1-7281-2882-5/19/$31.00 ©2019 IEEE

Authorized licensed use limited to: PES University Bengaluru. Downloaded on March 18,2021 at 05:11:01 UTC from IEEE Xplore. Restrictions apply.

Fig. 1: Conceptual Design of the Proposed System
different hardware and software modules, and to test and
troubleshoot the working of the proposed system.
The rest of the paper is organized as follows. Section II will
provide an overview of the various solutions provided in the
area of using deep learning based computer vision techniques
for implementing smart glasses for the visually impaired. The
details of the proposed system design and implementation will
be presented in Section III. Experimental results and their
implications will be discussed in Section IV. Conclusions and
future directions of the proposed solution will be provided in
Section V.
II. RELATED WORK
Text detection and recognition have been a challenging
issue in different computer vision fields. There are many
research papers that have discussed different methods and
algorithms for extracting the text from the images. The main
purpose of this literature review is to evaluate some of these
methods and their effectiveness regarding their text detection
accuracy rates.
In end-to-end text recognition with the power of
Convolutional Neural Networks combined with the new
unsupervised feature, learning growth took advantage of the
known framework for the training to achieve high accuracy of
the text and character detection and recognition modules. These
two models have been combined using simple methods to build
end to end text recognition system. The datasets that have been
used are ICDAR 2003 and SVT. The method of 62-way
character classifier obtained 83.9% of accuracy for a cropped
character from the first dataset [5].
In novel scene text recognition, which is an algorithm that
mainly depended on machine learning methods. Two types of
classifiers have been designed to achieve more accuracy, the
first one was developed to generate candidates, but the second
one was for filtering of candidates that are non-textual. A novel
technique has been developed to take advantage of multi-
channel information. Two datasets have been used in this study,
ICDAR 2005, ICDAR 2011. This method has achieved
significant results in different evaluation scenarios [6].
In photoOCR which is a system designed to detect and
extract any text from any image using machine learning
techniques, it also used different distributed language modeling.
The goal of this system was to recognize any text from any
challenging image such as poor quality or blurred images. This
system has been used in different application such as Google
Translate. The datasets that have been used for this system are
ICDAR and SVT. The results showed that the processing time
Authorized licensed use limited to: PES University Bengaluru. Downloaded on March 18,2021 at 05:11:01 UTC from IEEE Xplore. Restrictions apply.
for text detection and recognition is around 600 ms for one
image [7].
For text recognition in natural scene images, this method
has proposed an accurate and robust solution. It has used an
(MSER) algorithm to detect almost all characters from any
image. The datasets used for this system are ICDAR 2011 and
Multilingual datasets. The results showed that the MSER has
achieved 88.52% accuracy in character level recall [8].
End-to-end real-time text recognition and localization
system have used ER (External Regions) detector that covered
about 94.8% of the characters, and the processing time of an
image with 800×600 resolution was 0.3s on a standard personal
computer. The system used two datasets ICDAR 2011 and
SVT. On the ICDAR 2011 dataset, the method achieved 64.7%
of image-recall. For SVT dataset, it achieved 32.9% of image-
recall [9].
Text detection and localization using Oriented Stroke
Detection is a method that took advantage of two important
methods that are connected to a component with a sliding
window. The character or the letter has been recognized as a
region in the image that has some strokes in a particular
direction and particular position. The dataset that has been used
is ICDAR 2011, the experimental results showed 66% recall
which is better than the previous methods [10].
An Efficient and Accurate Scene Text (EAST) detector
method is a simple and powerful pipeline that allows detecting
a text in natural scenes, and it achieves high accuracy and
efficiency. Three datasets have been used in this study, ICDAR
2015, COCO-Text and MSRA-TD500. The experiments have
shown that this method has better results than previous methods
regarding accuracy and efficiency [11].
Tesseract is an open source Optical Character Recognition
(OCR) engine whose development has been sponsored by
Google Inc. The version 4 of Tesseract is based on a deep
learning-based Artificial Recurrent Neural Network called as
Long Short-Term Memory (LSTM) architecture. This OCR
engine can support up to 116 languages and has reasonable
character recognition rates [12].
The idea of smart glasses is to create wearable computer
vision-based glasses for various purposes (e.g. reading, search,
navigation). These uses determine the type of practical glasses
that are needed to be designed and developed. In the early stages
of development, the smart glasses were simple and provide
features for carrying out basic tasks which served as a front-end
display for the remote system. However, recent smart glasses
have become more sophisticated and provide several features
for aiding blind and visually impaired people. A comparative
summary of practically implemented smart glasses solutions
have been presented in Table I. It provides a description of the
conceptual model, their benefits, drawbacks and possible
improvements required to make them better. Also, it provides
the research gap in this area and which is filled with our
proposed solution (CCES, PMU).
Based on the literature survey presented above regarding the
different text detection techniques from images, we propose to
implement a novel solution with the following features. The
conceptual design of the proposed system is shown in Fig. 1.
 Table I: Comparative Summary of Smart Glasses Solutions

Solution Developer Conceptual Design Benefits Drawbacks Improvements

High resolution
Helps low vision Does not improve Water proof versions
camera for image and
people, avoiding vision as it just an are under
eSight 3 CNET's video capture for low-
surgery aid development
vision people

Symbols to audio For only people Can be improved to

Keisuke Shimakage, Support for dyslexic
Oton Glass conversion, normal with reading support blind people
Japan people. Converts
looking glasses, difficulty and no also by including
images to words and
supports English and support for blind proximity sensors.
then to audio
Japanese languages people
Suman Kanuganti Aira agents help Waiting time To include language
Aira uses smart
users to interpret connected to the translation features.
Aira glasses to scan the
their surroundings by Aira agents in order
environment.
smart glasses. to be able to sense.
Eyesynth is consist 3D Can use verbal audio
Allows blind/limited
cameras, which turns for better feel and
sight people to ‘feel It is expensive
the scene to sound navigation services.
Eyesynth Eyesynth, Spain the space’ through (575.78$) and it
signal (non-verbal) to
sounds. It converts only recognizes the
provide information
spatial and visual objects and
about of position, size,
information into directions.
and shape. It is
audio.
language independent.
Google Glasses show Can capture images Reduce costs to make
information without and videos, get It is expensive it more affordable for
Google using hands gestures. directions, send (1,349.99$) and the the consumers.
Google Inc.
Glasses Users can messages, audio glasses are not very
communicate with the calling and real- time helpful for blind
Internet via normal translation using people.
voice commands. word lens app.
Currently it supports
Helps blind people to only the English
avoid obstacles, to language, cannot be
Our The glasses can
Help people who have aid in reading and used while driving,
proposed CCES, PMU support other
vision difficulties learning, convert processing unit is
solution languages, it also can
especially blind image to text and separated from the
be smaller and easy to
people. search for glasses, and
wear.
information about the captures the object
words on the Internet. with a specific
range of distances.

a. Camera and ultrasonic sensor based smart glasses to below.

capture an image having embedded text.
b. Optical character recognition software based on EAST
and Tesseract open source tools.
c. Google text to speech conversion of the identified text
for the visually challenged person to hear.
d. RFID-based navigation system to enable the visually
impaired person to explore the academic building for
locating various lecture and lab rooms.
III. SYSTEM DESIGN AND IMPLEMENTATION
The proposed system consists of two main parts, the
hardware and the software. This section describes the details of
the hardware used and the software components used. Fig. 2
shows the process diagram of the proposed system.
A. Hardware Design & Implementation
The various sub-components of the hardware system that
have been used to make the smart glasses system are described
(i) Raspberry Pi 3 Model B+
Raspberry Pi is a credit card-sized computer. It needs to be
connected with a keyboard, mouse, display, power supply, SD
card and installed operating system [14]. Raspberry Pi is a low-
cost embedded system that can do a lot of significant tasks. It
can be run as a no-frills PC, a pocketable coding computer, a
hub for homemade hardware and more. It includes GPIO
(General Purpose Input/Output) pins to control various sensors
and actuators. Raspberry Pi 3 used for many purposes such as
education, coding, and building hardware projects. It is used as
a low-cost embedded system to control and connect all of the
I/O components together. It uses the Raspbian or NOOBs as the
operating system which can accomplish many important tasks
However, for our solution we decided to work on Raspbian as
the operating system.
(ii) Digital Camera
The webcam has a view angle of 60° with a fixed focus. It can

Authorized licensed use limited to: PES University Bengaluru. Downloaded on March 18,2021 at 05:11:01 UTC from IEEE Xplore. Restrictions apply.
Fig. 2: Process Diagram of the Proposed System
capture images with maximum resolutions of 1289 x 720 pixels.
It is compatible with most operating systems platforms such as
Linux, Windows, and MacOS. It has a USB port and a built-in
mono microphone. In the solution, the Webcam will be used as
the eyes of the person who wears the smart glasses. The camera
is going to capture a picture when the button is pressed (called
as Button 1, see Fig. 4), in order to detect and recognize the text
from the image.
(iii) Ultrasonic Sensor
The purpose of ultrasonic sensors is to measure the distance
using ultrasonic waves. Ultrasonic sensors emit the ultrasonic
waves and receive back the echo. So, by measuring the time the
ultrasonic sensor will measure the distance to the object. It can
sense distances in the range from 2-400 cm. In the smart
glasses, the ultrasonic sensor is used to measure the distance
between the camera and an object to detect the text from the
text image. It was observed based on experimentation that the
distance to the object should be from 40 cm to 150 cm to capture
a clear image (see Fig. 4).
(iv) RFID Sensor
A Radio Frequency IDentification (RFID) sensor consists
of two main devices, namely the RFID reader and the RFID tag.
The RFID tag has digital data, integrated circuits, and a tiny
antenna to send information to the RFID reader. Signals
frequencies are usually between 125 to 134 kHz and 140 to
148.5 kHz for low frequencies and 850 to 950 MHz and 2.4 to
2.5 GHz for high frequencies.
The RFID reader is mainly used to collect information from
the RFID tag with the help of electromagnetic fields. The
process of transformation data from the tag to the reader is done
by the radio waves. However, in order to achieve this process
successfully the RFID tag and the RFID reader should be within
a range between 3-300 feet. Any object can be identified
quickly when it is scanned, and the RFID can recognize it.
RFID has many applications such as passport, smart cards, and
home applications. The RFID sensor is used in our solution to
attach the RFID reader in the hall and various classrooms so the
blind person can recognize them.
(v) Headphones
Wired headphones was used in our solution since Raspberry
Pi 3 Model B & B+ come with Audio jack and it is better to take
advantage of this feature rather than occupying one of the four
USB ports that can be useful for other peripherals. The
headphones will be used to help the user listen to the text that is
been converted to audio after it has been captured by the camera
or to listen to the translation of the text. The headphones are
going to be small, lightweight and connected to the glasses, so
the user will not be concerned about losing them or feel
uncomfortable wearing them.
Authorized licensed use limited to: PES University Bengaluru. Downloaded on March 18,2021 at 05:11:01 UTC from IEEE Xplore. Restrictions apply.
Fig. 3: Procedural steps during the OCR process
B. Software Design & Implementation
Below are the software components that have been used in
the proposed system for programming the functionalities of the
smart glasses system.
(i) OCR Tools: Tesseract and EAST
OCR (Optical Character Recognition) is used to convert
typed, printed or handwritten text into machine-encoded text.
There are some OCR software engines which try to recognize
any text in images such as Tesseract and EAST. In this project
Tesseract version 4 is used because it is the best open source
OCR engines. OCR process consists of multiple stages, as
shown in Fig. 3
(a) Preprocessing: The main goal of this step is to reduce
the noise that resulted from scanning the document where the
characters might be broken or smeared and causes poor rates of
recognition. Preprocessing is done by smoothing the digitized
characters through filling and thinning. Another aim of this step
is to normalize the data to get characters of uniform size,
rotation, and slant. Moreover, significant compression in the
amount of information is achieved through thresholding and
thinning techniques.
(b) Segmentation: In this process, the characters or words
are isolated. The words are segmented into isolated characters
that are recognized separately. Most of OCR algorithms
segment words into isolated characters which are recognized
individually. Usually, segmentation is done by isolating every
connected component.
(c) Feature Extraction: This process will capture the
significant features of symbols and it has two types of
algorithms which are pattern recognition and feature
extraction/detection.
(d) Classification: OCR systems use the techniques of
pattern recognition that assigns an unknown sample into a
predefined class. One of the ways to help in character
classifying is to use the English directory.
(e) Post-processing: This process includes grouping and
error detection & correction. In grouping, the symbols relate to
strings. The result of plain symbol recognition in the text is a
group of individual symbols.
(ii) OpenCV Libraries
OpenCV is a library of programming functions for real-time
computer vision, the library is cross-platform and free for use
under the open-source BSD license [15]. For the installation of
the OpenCV 4 libraries, the recommended operating system for
the raspberry pi B+ which is Raspbian Stretch was installed.
Win32 Disk Imager was used to flash the SD card.
(iii) Google Text to Speech (gTTS) API
One of the most important functions of the smart glasses is
text to voice conversion. In order to implement this task, we
installed gTTS (Google Text-to-Speech). It is a python library
that interfaced with Google Translate API [13]. gTTS has many
features such as convert ultimate length of text to voice, provide
error pronunciation using customizable text pre-processors and
support many languages and retrieve them when needed. We
used the gTTS to perform language translation from English to
Arabic (called as Button 2, see Fig. 4).

IV. EXPERIMENTAL RESULTS AND DISCUSSION

A. Text Recognition from captured image
In this result, the main goal was to check if the text detector
that was used in this solution, which is EAST pre-trained text
detector and the text recognizer which is OCR using Tesseract
is working well. The test showed mostly good results on big
clear texts and failed on small texts. We found that the
recognition depends on the clarity of the text in the picture, its
font theme, size, and spacing between the words. One such
result is presented in Fig. 5. Here the image had the text
“Electronics Lab & Embedded Systems Lab”, which was
recognized as “Electronics Lab Embedded Svstems Lab”, which
can be said to be about 80% accurate.
B. Text to Speech Conversion
In this result, the idea was to check if the detected text is
converted to audio text. We used gTTS (Google Text To
Speech) libraries after it was found that the voice quality was
better and clearer than the other TTS libraries such as Festival
TTS, Espeak TTS, and Pico TTS. The voice was clear for the
right detected words. One such result is presented in Fig. 6.
Here the text captured from the image was Moonshine, which
was accurately converted into audio signal, which was clearly
audible.
C. Final Prototype
Fig. 7 shows the images of the final prototype which we
developed. As can be see, the hardware circuit (Raspberry Pi
processor board) can be worn on the arm and it is mobile with
the user. The glasses can be worn on the face which consists of
the normal sunglasses mounted with the camera and the
ultrasonic sensor. We admit that this is not very comfortable but
it has other benefits like low cost ($330), open source hardware
(Raspberry Pi) and software (OpenCV, Tesseract and Python).
Fig. 7 also shows the testing and working of the RFID-based
navigation unit. It provides the visually challenged with
information (in audio signals) about the current location in the
Fig. 4: Procedural steps for the design and development of the prototype
academic building.
D. Challenges and Limitations
During the implementation of this system, the first
challenge was to decide which microcontroller will be
appropriate. After some research it was found that Raspberry Pi
has the required features suitable for system objectives.
Initially, Raspberry Pi zero w was the choice because it was
smaller and lighter than the others versions. It could be set up
easily on the glasses rather than the person holds it by hand.
However, we also found that out that the Raspberry Pi 3 has
higher processing power since it consists of quad core processor
and it is has faster processing speed than Raspberry Pi zero w.
It also has a larger memory and extra I/O ports for connecting
peripheral devices. Since we also decided to use OCR in
conjunction with OpenCV and Efficient and Accurate Scene
Text Detector (EAST), the Raspberry Pi should be able to do
multitasking efficiently. Then we finally decided to opt for
Raspberry Pi 3.
We had proposed glasses with a built-in camera and it
turned out that this built-in camera is not compatible with
Raspbian OS, and it was only compatible with Windows and
MacOS, so we tried to install Windows IoT (operating systems
from Microsoft designed for use in embedded systems.) on the
Raspberry but unfortunately it didn’t work. To solve this
problem, we decided to use a webcam which was compatible
with the Raspberry Pi.
V. CONCLUSIONS AND FUTURE WORK
This paper has proposed, implemented and tested a novel

Authorized licensed use limited to: PES University Bengaluru. Downloaded on March 18,2021 at 05:11:01 UTC from IEEE Xplore. Restrictions apply.

Fig. 5: OCR result with EAST detector
Fig. 6: Google Text to Speech result
Fig. 7: Final developed prototype
smart glasses for visually challenged students, which has the
features to capture an image, extract and recognize the
embedded text and convert it to speech. Our design and
implementation was a practical demonstration of how open
source hardware and software tools can be integrated to provide
a solution which is low-cost, lightweight, re-configurable and
scalable. However, there are some limitations in our proposed
solution which can be addressed in the future implementations.
Hence, we recommend the following features to be
incorporated in the future versions.
● In order to cater to a wide variety of users, it would be
Authorized licensed use limited to: PES University Bengaluru. Downloaded on March 18,2021 at 05:11:01 UTC from IEEE Xplore. Restrictions apply.
worthwhile to include multi-lingual feature (e.g.
French or Urdu ) in the speech translation module.
● To improve the direction and warning messages to the
user, we can include GPS-based navigation and alert
system.
● To provide for more space visibility, we can include a
wide angle camera (e.g. 2700 degrees as compared to
600 currently used).
● Finally, to provide for more real-time experience, we
can include video processing instead of still images.
ACKNOWLEDGEMENT
We sincerely thank the College of Computer Engineering
and Science (CCES) and the management of Prince
Mohammed Bin Fahd University (PMU) for their cooperation
and support in accomplishing this BS senior design project.
REFERENCES
[1] World Health Organization, "Global data on visual impairments 2010".
Retrieved from: https://www.who.int/blindness/GLOBALDATAFINAL
forweb.pdf (Last accessed on 30th May, 2019).
[2] Best Colleges, “ College guide for students with visual impairments”,
Retrieved from: https://www.bestcolleges.com/resources/ college-
planning-with-visual-impairments/ (Last accessed on 30th May, 2019)
[3] Google Inc., "Google Glasses". Retrieved from https://en.wikipedia.org/
wiki/Google_Glass (Last accessed on 30th May, 2019).
[4] Humanitarian projects, "Prince Sultan bin Abdulaziz College for the
visually impaired". Retrieved from http://www.princemohammad.org/
en/Initiatives-College-for-the-Visually-Impaired.aspx (Last accessed on
30th May, 2019).
[5] T. Wang, D. J. Wu, A. Coates, A. Y. Ng, "End-to-end text recognition
with convolutional neural networks” IEEE 21st International Conference
on Pattern Recognition (ICPR), pp.3304-3308, 2012.
[6] H. I., Koo, D. H. Kim, "Scene text detection via connected component
clustering and non-text filtering" IEEE Transactions on Image
Processing, 22(6), pp. 2296-2305.
[7] A. Bissacco, M. Cummins, Y. Netzer, H. Neven, "Reading text in
uncontrolled conditions", Proceedings of the IEEE International
Conference on Computer Vision, pp. 785-792, 2013.
[8] X. C. Yin, X. Yin, K. Huang, H. W. Hao, "Robust text detection in
natural scene images " IEEE transactions on Pattern Analysis and
Machine Intelligence, 36(5), pp. 970-980, 2014.
[9] L. Neumann, J. Matas, "Real-time scene text localization and
recognition ", Proceedings of IEEE International Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 3538-3545,
2012.
[10] L. Neumann, J. Matas, "Scene text localization and recognition with
oriented stroke detection", Proceedings of IEEE International
Conference on Computer Vision, pp. 97-104, 2013.
[11] X. Zhou, C. Ylao, H. Wen, Y. Wang, S. Zhou, W. He, J. Liang, "EAST:
an efficient and accurate scene text detector", Proceedings of the IEEE
conference on Computer Vision and Pattern Recognition, pp. 5551-5560,
2017.
[12] GitHub Inc., "Tesseract OCR". Retrieved from
https://github.com/tesseract-ocr (Last accessed on 30th May, 2019).
[13] Python Software Foundation, "Python". Retrieved from
https://www.python.org/ (Last accessed on 30th May, 2019).
[14] Raspberry Pi 3 B+: Retrieved from https://www.raspberrypi.org/ (Last
accessed on 30th May, 2019).
[15] OpenCV. Retrieved from https://opencv.org/ (Last accessed on 30th May,
2019).