Dual mode Sign Language Recognizer-An Android
Based CNN and LSTM Prediction model
Prof. Rubi Mandal
Dept. of Information Technology Dept. of Information Technology
2023 3rd International conference on Artificial Intelligence and Signal Processing (AISP) | 979-8-3503-2074-9/23/$31.00 ©2023 IEEE | DOI: 10.1109/AISP57993.2023.10134768
SVKM’s IOT Dhule
mandalruby@gmail.com dhirajpatil.1434@gmail.com
Gaurav Birari
Dept. of Information Technology
SVKM’s IoT Dhule
gauravbirari07@gmail.com
Abstract -Communication is varied important part of our
life. By communicating with each other, we can interact, share
our ideas and emotions. However, mute and deaf people of our
society face lots of problem while communicating with normal
people. Previous researches were focusing only one-sided
communication i.e., Speech to Sign Conversion using only
Static dataset like for Alphabets, Digits and only for specific
languages. While keeping in mind the lacunae of current
system, this research is about an application-based system that
will serve as an interpreter for sign language, enabling two-
ways communication between hearing-impaired people and
normal people while working on dynamic gestures and
centralized system for everyone. This system will work on two
modules. One will be sign gestures to speech and other will be
speech to 3-D animated sign gestures. For this purpose, dataset
is collected from deaf and mute School, Dhule Maharashtra
and ISL official site of size approx. 1800 words. Sign gestures
to speech conversion is implemented using CNN+LSTM deep
learning algorithm while Speech to 3D sign gestures is
implemented by extracting keywords from live speech and
developing 3D animated gestures using extracted keywords.
This system will bridge the gap between deaf, mute and normal
peoples.
Keywords - sign recognition, deaf and mute, LSTM, deep
learning, CNN, 3D model, OpenCV, python.
I. INTRODUCTION
According to World Health Organization (WHO) there are
5% means 430 million of peoples are deaf and mute. We
know that communication play's very important role in our
life, so that peoples communicate to each other with the help
of sign language. Sign language is very essential for them,
after all it has ability to join the two communities. But there
are only few amounts of people of our society who knows the
sign language. Basically, this language is the language which
is used by the especially abled peoples who are hearing
impaired and unable to speak, they use hands to make the sign
gestures instead of spoken words. The sign language is not
same in the whole universe, there are various sign language
across the world. It is very difficult to understand the sign
language for normal people. We can't understand it because
we don't know the actual meaning that particular sign.
What if we understand the sign language, it will be
benefited not only for us but also for that special people, but
it is not simple as we think. That’s why we are proposing a
solution
Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on October 30,2023 at 06:16:13 UTC from IEEE Xplore. Restrictions apply.
Dhiraj Patil Sanket Gadhe
Dept. of Information Technology
SVKM’s IoT Dhule SVKM’s IoT Dhule
sanketgadhe366@gmail.com
Tejaswini Buwa
Dept. of Information Technology
SVKM’s IoT Dhule
tejbuwa06@gmail.com
which focusses on reducing communication gap between
normal and deaf/mute peoples. Already some solutions are
available which are not centralized, only one way and uses
static gestures only. So, for removing the lacunae of the
existing system we are developing an application-based
system which consists of two modules first is - sign language
to speech and second is - speech to 3- d animated sign
gestures. Through open CV python library, we are capturing
the sign gestures. In the first modules convolutional neural
network (CNN) and LSTM is used to train the dataset. The
system is dynamic, means it can also recognize words and
sentences.
Our system will put a very deep impact on the society. It
will definitely benefit not only the deaf/mute people but also
the normal peoples for communicating with each other. A
survey was conducted by us in real time at ANAND NIVAS
DEAF/MUTE SCHOOL, DHULE Maharashtra.
The survey is all about to collecting the real time sign
gestures for creating a new dataset. By meeting the deaf/mute
students and by interacting with the teachers we collected the
videos and images for datasets. There are about 300 different
sign gestures, and for Uniformity and more accuracy we had
taken data set from the official website of ISL to get more
sign gestures, from that we get about 15000 different sign
gestures as video. Also, from ourself we created signs with
different angles for a making huge collection of data set due
to this our system give more accurate result.
II. LITERATURE RIVIEW
Many academics have proposed various strategies that
allow sign language to be translated into text in order to
narrow the communication gap between hearing- and speech-
impaired people.
This idea is based on continuous real-time action detection
of visual frames to identify the activity carried out by the user.
After locating critical spots utilizing media pipe holistic, this
research only focuses on key points of person. [1]. Sign
Language Recognition is implemented by combination of
two algorithms CNN + LSTM. Real time sign language
translation is possible because of YOLOv5 processes the task
of sign recognition by using hand gestures. But the limitation
 is this system is static [2]. The device makes use of a camera
to record different hand motions. The image is then processed
using a variety of techniques. The first step is pre-processing
the image. Then, an edge detection algorithm is employed to
identify the edges. The text is shown and the sign is identified
by a template-matching algorithm, so it takes more time to
display the result [3]. A deep multi-layered convolution, but
it can only applicable for images [4]. A model which only
convert the sign from ASL into English character so that’s
why it is static in nature [5].
III. ARCHITECTURE AND METHODOLOGY
Architecture:
Our system mainly a dual communication between
Deaf/Mute and Normal people helping in establishing in two-
way communication.
Architecture is divided into two types:
Fig 1. Architecture of Sign to Speech module
Sign to Speech module: Here Sign gestures are converted
into speech by undergoing different stages right from frames
collection till the speech data creation. It consisting of
different phases through which the input data has to pass out
before feeding as input to the sign gestures model and finally
predicting the words for particular sign gestures.
Speech to Sign module: Here Speech data is converted into
text followed by removing stop words from text, replacing
synonyms and representing 3-D animated gestures for each
and every word.
Fig 2. Architecture of Speech to Sign module
Methodology:
Our Methodology consist of different sections listed below:
I. Dataset Creation
Fig 3. Dataset of research
Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on October 30,2023 at 06:16:13 UTC from IEEE Xplore. Restrictions apply.
For this purpose, we did a live survey in a school Anand
Nivasi Mute and Dumb school meet their teachers, student
understood their problem from the depth. Taken their help in
collecting data set it is not possible to collect the whole from
them. So currently we referring ISL (Indian Sign language)
government portal for knowing the sign action for particular
word and shooting the video of that particular sign by our
own. The same video is used for training purpose of the sign
to speech module.
After generating a approx., collection of 1800 video
dataset now, For the speech to 3-D animated sign gesture
module we are currently making the 3-D animated sign
gesture of that 2-D video by using following tools:1) Deep
Motion (creating 3-D animated character with action of sign
gestures) 2) Blender (Minute works of fingers and body part
movement) 3) I clone (For facial expressions)
After generating 3-D animated gesture we are collecting
them at one place. After that we labelled them of particular
name in the data set to give the desired output when particular
word of that label is given by the user.
Sign Gestures Classifier training model: To Predict the
words for specific Sign gestures proposed model consists of
Convolutional Neural Network + LSTM model.
Convolutional neural network is Deep Learning
Algorithm used for Image classification considered as one of
the best algorithm for image classification because of its
highest accuracy of prediction, less computation work and
less time required for result prediction.
CNN consists of two sections through which input data is
passed:
1. Feature Extraction: This section of CNN consists of
different layers of filters. These filters are used for Feature
extraction different layers extracts different features of the
input image and at each level no of layers are increased to
extract more data.
2. Max Poling: Max Pooling is added for Reducing the size
of the input array data. Max Pooling is added at each layer for
size reduction which is wholly responsible reducing
computation work.
Finally, the compressed and extracted features are
converted into the flatten array which is given as input to the
LSTM model.
A unique type of recurrent neural network as RNN called
Long Short-Term Memory as LSTM is primarily used to
classify time-series data. The output of a certain layer in a
conventional RNN is influenced by both the previous output
and the current input.
At last the output from the LSTM is passed as input to the
dense neural network for classification.
Fig 4. Working of CNN and LSTM Real time Sign and
Speech Prediction
 Our proposed system basically is Android application
driven consisting of two major modules for establishing the
intercommunication between Deaf/mute and normal peoples:
1. Sign Gestures to Speech Conversion module
2. Speech to 3-D animated gestures module:
1) Sign Gestures to Speech Conversion module:
A) Live Data Collection:
This module focuses Data Gathering. Here for data
collection Open CV python Library is used. Applications
camera is used as the source for capturing the live video data
and as we all know video is nothing but the collection of
frames so n no of frames are captured for a particular word
and likewise frames are captured and stored as stack for each
word differently and pre-processed before giving as input to
the trained Sign classifier.
B) Data Preprocessing:
This module focuses on the important process which is
processing the data. The frames which are collected at the
time of data collection are pre-processed in the following
steps:
I. Gray Scale Conversion: Images Naturally are captured in
the RGB (Red, Green, Blue) format are converted into gray
scale using MATLAB.
II. Extraction of Features: Canny Edge detection is used to
detect edges in the output in order to obtain correct findings,
and filters like Gaussian Specific features are extracted using
filters, while the remaining noise is discarded.
III. Normalization of Input and Output data: Image cannot be
feeded as it is as input to the system it needs to normalized
into NumPy array and need to be converted into the range of
(-1 to 1) for linearity in the input data.
E.g. This normalized input is feeded into the Sign Classifier
model
Prediction Module:
Here Depending on the input provided the specific class of
words are predicted by the Sign Classifier.
It is done using Convolutional Neural Network (CNN) and
Long Short-term Memory (LSTM)
Which consisting of Features extractions and Dense neural
networks part for Classes Classification
Words are Predicted depending upon the accuracy of the
Prediction of the word. Words having higher accuracy are
stored into the array of Words
C) Sentence Formation and Speech as output:
Here Array of Collected words is converted into sentence
and sentence is converted into speech using Python pyttsx3
library.
2) Speech to 3-D animated gestures module:
Speech to 3-D animated gestures module: In this module
speech is feeded as an input and will generate 3-D sign
Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on October 30,2023 at 06:16:13 UTC from IEEE Xplore. Restrictions apply.
gestures as an output. Using the speech recognizer, we will
take the input of speech into the system. After taking input
we need to perform several processes like:
1. Converting Speech to Text
2. Grammatical text formatting for particular sign language
3. Breaking Sentences into words
4. Removing stop words
5. Synonyms replacement.
6. 3-D animation
1] Converting Speech to text:
The speech recognizer consists of three models: 1)
acoustic Model 2) signal analyser 3) language model. In the
acoustic model the speech is analysed based on mathematical
probability. A statistical Markov model is the hidden Markov
model (HMM). Model where the system being modelled, let's
call it X, is believed to be a Markov process with
unobservable states HMM's definition calls for the existence
of an observable process Y whose results are "affected" in a
known manner by those of X. Signal analyzer converts this
analogy signal to the digital signal removing the unwanted
data like background noise from the audio and then in the
language model using dictionary files and grammar files
identify words and sentence speaked. After all this the arrays
of words are framed to text.
2] Grammatical text formatting for particular sign language:
For particular sign language of any country there is a
predefined grammatical rules like example In the ISL the
structure of sentence follows subject +object +verb whereas
the user will speak the sentence in form of subject+ verb +
object. The sign languages do not use any gerund, suffixes or
other forms it focuses only on the root words.
Fig 5. Speech to text conversion
Eg: Speaked Sentence: Go to school (verb + object)
Grammatical framed: school to go (object +verb)
(Note: It will depend on the sign language)
3] Breaking sentences into words:
The grammatically framed sentence will be break down
into the list of words using splitting sentence function and
arrays of list of words will be generated.
For eg.
Fig 6. Sentence breaking
4] Removing stop words:
After getting list of words, each word will be checked in
the dictionary of sign language. The word which has no sign
 gestures are said to be stop words and they are removed Fig 9. This figure predicted the word for love
from the list of words.
For Eg:

Fig 7. Stop word removal

5] Synonyms replacement + Lemmatization:

In the dataset, 3-D animated labelled gestures are there.

We are searching every word in the data set and if any of the
word is not matched with the labelled gestures, then we will Fig 10. This figure predicted the word for hello
get the list of synonyms of that particular word and now the
synonym will be searched again in the data set and that
found word will replace the previous word.
For eg:
Fig 11. Model Accuracy

Fig 8. Removing

synonyms For eg: user input: Hello dad, how are you?
But in the data set we have labelled animated gesture as
Fig 12. Graph between validation accuracy and total
father, so dad will get replaced by the synonym father.
accuracy
Hello father, how are you?
In this part we also replace and lemmatize the word by
removing the v1, v2, v3, v4 form of the verbs from the words.
Like Talking, Talked, Talks is replaced by talk by removing
ing, ed and s.

5] Showing 3-D animated sign gesture:

Now we have the desired word to search in our data set
and get the animated sign gestures from the data set and we
will show that one by one according to the sequence of the
word in the array.
Following the above sequence, we are getting a 3-D sign
gesture as an output.
IV. RESULT Fig 13. Graph between validation total loss and validation
accuracy
A) Sign to Speech Result
B) Speech to sign results:

I) Grammatically corrected sentence result: According to

rule of ISL grammar rule showing the result of speech to text
and text to grammatically framed sentence.

Fig 14. Grammatically changed sentence result

Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on October 30,2023 at 06:16:13 UTC from IEEE Xplore. Restrictions apply.
 II) Breaking sentence result: The spoken sentence by the user
is break downed into chunks of words as a set of words in
array shown in fig
Fig 15. Breaking sentenced result
III) Removing the stop words result: The words which have
no sign gestures is removed from the set of words as shown
in the fig
Fig 16. Removed stop words result
1) Synonyms replacement and lemmatization: the words
which are not in the data set are replaced by the synonyms
and verbs forms are removed result are shown in the fig
Fig 17. Synonyms replacement
2) Desired 3-D Animated sign gestures: For particular word
desired 3-D gesture are played in a sequential order according
to the arrays of video result is shown in the figThe analysis
of some of the existing systems is given below in detail:
Fig 18. 3-D Animated sign gestures output
Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on October 30,2023 at 06:16:13 UTC from IEEE Xplore. Restrictions apply.
V. CONCLUSION
Our system works best for intercommunication between
deaf and normal people. Previous systems have limitations of
being static and one way. Our system has overcome this
limitation and hence it is static as well as dynamic. It provides
two-way communication between deaf and normal people.
Development is done using CNN as it is efficient and more
accurate. It requires less computation as compared to ANN
and performs classification based on feature extraction. Our
system has accuracy of 92.62%. Our system is beneficial for
normal people and hearing-impaired people as it will
overcome communication gap between these two
communities. As we are developing a system-based
application, so it will be portable.
In future we are looking forward to make a centralized
interpretation application for everyone with large number of
Dynamic Data set. We are aiming for making app available
in many languages.
VI. REFERENCES
[1] Vishwa Hariharam Iyer, U.M. Prakash, Ashrut Vijay and P.
Satishkumar, “Sign Language Detection using Action Recognition”
,2nd International Conference on Advance Computing and
Innovative Technologies in Engineering (ICACITE),2022.
[2] Tengfei Lia Yongmeng Yanb, Wenqing Duc “Sign Language
Recognition Based on Computer Vision” ,IEEE International
Conference on Artificial Intelligence and Computer Application
(ICAICA),2022.
[3] Sona Shrenika and Myneni Madhu Bala "SIGN LANGUAGE
RECOGNITION USING TEMPLATE MATCHING
TECHNIQUE" ,IEEE ,2022.
[4] Rajarshi Bhadra and Subhajit Kar,"Sign Language Detection from
Hand Gesture Images using Deep Multi- layered Convolution
Neural Network", IEEE Second International Conference on
Control, Measurement and Instrumentation (CMI), India,2021.
[5] Yogya Tewari, payal soni, shubham singh, Murali Saketh Turlapati
and Prof. Avani Bhuva, “Real Time Sign Language Recognition
Framework For Two Way Communication”,IEEE International
Conference on Communication information and Computing
Technology (ICCICT), June 25-27, 2021, Mumbai, India.
[6] G.Anantha, K.Syamala, P.V.V.Kishore, A.S.C.S Sastry, ‘” Deep
Convolutional Neural Networks For Sign Language Recognition”,
2018.
[7] Setiawardhana, Rizky Yuniar Hakkun, Achmad Baharuddin,"Sign
Language Learning based on Android For Deaf and Speech
Impaired People", International Electronics Symposium
(IES),2015.
[8] S. M. Kamrul Hasan\ Mohiuddin Ahmadl,"A New Approach of
Sign Language Recognition System for Bilingual Users",1st
International Conference on Electrical & Electronic Engineering
(ICEEE) 04-06 November 2015.
[9] Vishal pande, parminadar singh and Vishal Munnaluri,"Machine
Learning based Approach for Indian Sign Language
Recognition",Proceedings of the Seventh International Conference
on Communication and Electronics Systems ICCES,2022.
[10] R. Kurdyumov, P. Ho, and J. K. Ng, “Sign language classification
using webcam images”, 2011.