Survey paper on Sign Language Recognition System
Jyoti Deshmukh, Akshay Kedar, Mangesh Jadhav, Pranali Khuspe, Rohit Singh
Department of Computer Engineering
Rajiv Gandhi Institute of Technology, Andheri, Mumbai
University of Mumbai.
Abstract— Sign Language (SL) is a language that uses a
system of manual, facial and other means of communication. It is
used by deaf and dumb people as their main source of
communication. The main focus of the SL Recognition system is to
create a vision-based system which is able to identify the different
sign languages from the images or video captured. In the paper, we
provide a review of the current research to identify the problem
areas and challenges faced by researchers while making a SL
Recognition System.
Keywords— Sign language, Deep Learning, Stochastic Gradient
Descent (SGD), HIdden Markov Model (HMM), Multi-class
Support Vector Machine ((MSVM)
I. INTRODUCTION
The Sign Language recognition system has gained a lot of
importance. Sign Language is the primary means of
communication in the deaf and dumb community. At this age of
technology, it is quintessential to make these people feel part of
the society by helping them communicate smoothly. Hence, an
intelligent computer system is required to be developed and be
taught. The project intends to close the communication gap for
persons who are deaf or hard of hearing, allowing them to
converse more smoothly with the outside world. We translate
hand gestures into understandable words using them as the
primary input.
In this paper, research and development in the field of Sign
Language (SL) recognition systems are reviewed and
thoroughly analyzed. This paper examines the shortcomings in
previous studies in the area of sign language recognition. This
list of identified shortcomings is intended to serve as a
reference for future work that can be done and what
researchers should focus on while conducting study in this
field.
Researchers are increasingly interested in Sign Language
Recognition because of its applications in a variety of fields,
including deaf people communication systems, virtual reality,
human-computer interaction, industrial machine control, and
many more.
According to the quantity of inputs required, there are two types
of sign language recognition systems:
1.Static SL Recognition
This system accepts images as input. Because the
image input is not particularly dense, the accuracy is usually
greater than 97.00 %.
2. Dynamic SL Recognition
This system accepts video as input. It is less accurate,
but it is more suitable for use in the real world.
1
II. A REVIEW ON EXISTING SL RECOGNITION SYSTEMS
Deaf and hard-of-hearing persons use sign language to
communicate with one other and with others in their
communities. The process of computer recognition of sign
language begins with the acquisition of sign gestures and ends
with the synthesis of text or voice. Until now, the majority of
study has been done on static signs. Currently , advancement in
the field of Machine Learning has led to the emergence of
Dynamic Based SL recognition systems.
Dictionary of signs, segmentation, feature selection,
and classification techniques are the most significant aspects for
a sign language recognition system. The following summary is
focused to cover trends of all these four aspects of sign language
recognition systems.
Jose L. Flores C. and et. al [6] have used the alphabet
of SL of Peru (LSP). Different image processing techniques
were used to remove noise, improve contrast, detect and crop
out the region containing hand gesture.With varied quantities of
layers and parameters per layer, two CNN architectures were
developed. First architecture based on LeNet-5 showed an
accuracy of 95.37% whereas the second architecture which is
the proposed CNN architecture showed an accuracy of 96.20%
in terms of recognition of the 24 static hand gestures.
Surejya Suresh and et al. [1] developed a system whose
primary goal was to develop a vision-based system, a
Convolutional Neural Network (CNN) model, that could
recognise distinct indications from recorded pictures. The SGD
and Adam optimizers are used in the two CNN models created.
Using optimizers, SGD and Adam, this model has categorised 6
different sign languages with an accuracy of 99.12 % and 99.51
%, respectively.
Jie Huang and et al. [2] have introduced a new 3D
convolutional neural network (CNN) that automatically extracts
discriminative spatial-temporal characteristics from raw video
streams without the need for prior information, removing the
need to build features. Multi-channels of video streams,
comprising color information, depth clues, and body joint
locations, are used as input to the 3D CNN to integrate colors,
depth, and trajectory information in order to improve
performance. The average accuracy of the 3D CNN with Gray
Channel features is 88.5 %, whereas the average accuracy of the
3D CNN with Multi-channel features is 94.2 %.
Yogeshwar I. Rokade and et al. [14] developed a system
which includes an artificial neural network (ANN) and a Support
Vector Machine to provide a novel method for recognising
Indian sign language (SVM). The shapes of the hand region are
obtained using sign segmentation, the grey level picture is
obtained through Euclidean distance transformation, and the
feature vector is obtained through feature extraction. ANN
achieves an average accuracy of 94.37 %, while SVM achieves
an average accuracy of 92.12 %.
Karishma Dixit and Anand Singh Jalal [13] used only
manual signs, which include hand movements of isolated
signs, have been evaluated. Each class is trained using an
MSVM, and a feature vector for sign identification is built
using combinational parameters of invariant moment structural
form descriptions. The global thresholding algorithm is used to
segment the input image of hand motions. According to the
findings, combining invariant moments with shape descriptors
yields a better result of 96.23 %.
P. Subha Rajan and el al.[12] have proposed a method that
provides a basis for the development of Sign Language
Recognition system for one of the south Indian languages.A
collection of 32 signs, i.e. Tamil letters with 12 vowels and 18
consonants, are defined in the suggested approach, each
reflecting the binary UP and DOWN positions of the five
fingers.This hand pattern recognition procedure includes four
major features. They are: a) Data Acquisition, b) Palm Image
Extraction, c) Sign Detection, d) Training Phase, and e) Binary
to Text Conversion. When trained with 320 photos and
evaluated with 160 images, this system was able to recognise
images with 98.125 % accuracy.
Tanzania Ferdous Ayeshee, Sadia Afrin Raka and Et. Al [7]
have used Fuzzy logic. Fuzzy logic or fuzzification is done in
two phases, The data is first processed from raw photos, and
then the rules are found using angle measurements. Harries
technique was also used for point specification and corner
detection of the hand. This approach was effective in
determining only the two letters that the researchers tested.
This method might not work with other Bengali letters.
Furthermore, improved computer vision technologies can be
used to reduce the suggested system's limitations.
T. Shanableh and K. Assaleh [5] have used gloves to
recognize Arabic SL gestures and to use color segmentation to
make the process of segmenting out the signer's hands easier.
Bounded images are then transformed into the frequency
domain using Discrete Cosine Transformation followed by
zenal coding to form the feature vectors. The proposed
solution was validated using KNN and polynomial networks
resulting in a classification rate of 87% in user-independent
mode.
Thad Starner and et al. [8] have shown two real-time
HMM-based systems for identifying continuous American
Sign Language (ASL) sentences at the sentence level. The first
device uses a desk-mounted camera to view the user and
achieves 92 % word accuracy. The second device embeds the
camera in the user's headgear and obtains a 98 % accuracy rate
(97 % with an unrestricted grammar).
Britta Bauer and Hermann Hienz [9] have employed a
system based on continuous density HMM, where each sign is
represented by a separate model. A feature vector reflecting
manual sign parameters is served for training and recognition
input. The main goal of this system is to detect SL sentences
automatically based on a lexicon of 97 signs in German sign
language (GSL). Experiments have shown that this system
understands sign language with 94% accuracy rate based on 52
lexical signals and all available attributes. However, with a
vocabulary of 97 signs, recognition accuracy drops to 91.7 %.
Pinar Santemiz and et. al [4] have used broadcast news for
the hearing-impaired to automatically extract signs with the
help of speech that coexists with the sign to develop their
database. For sign segmentation, multiple alignment-based
techniques are applied. They used a variety of alignment
methods, including feature concatenation on Dynamic Time
Warping (DTW) and HMMs, modelling using linked and
parallel HMMs, and sequential DTW and HMM fusion. As a
conclusion, modelling the signs using HMM utilising intervals
discovered previously by DTW yields the maximum accuracy.
Soma Shrenika and Myneni Madhu Bala [10] have used the
average method for conversion of sign image to a grayscale
image. The resultant image is processed through a canny edge
detection algorithm for determination of edges which outputs a
template image. Lastly, the template image is processed
through a template matching algorithm in which the
comparison of template image with images present in the
dataset is carried out by the Sum of Absolute Difference
(SAD) Method. The image that produces the least value of
SAD is being output with its corresponding label.
Nursuriati Jamil and et al. [3] used an image retrieval
system, which is made up of two parts: a database model and a
retrieval model. The images are retrieved using eccentricity,
compactness, convexity, rectangularity, and solidity. Recall and
precision are obtained by examining the photos. Simple shapes
used to retrieve songket motifs are remarkably precise, with a
precision rate of 97 percent at 10% recall and 70% at 20%. The
study was an experiment to see which picture retrieval
techniques were the most efficient and fast.
Ignacio Rocco and et al. [16] Have used a convolution
Neural network for geometric matching, 3 fold are the
contribution of this work. A correspondence between two
images in agreement with a geometric model searches for an
affine. thin plate spline transformation or holography, and
estimating its parameter. Through this method, it has also been
proven useful for instance level alignment, obtaining
reasonable alignment for challenging graffiti benchmarks.
Nobuyuki Otsu [15], has used an unsupervised and
nonparametric method of automatic threshold selection. The
discriminant criteria aids in determining the best threshold, i.e.,
the separability of resultant classes in grey level, in order to
maximise. The grey level histogram is calculated using only
the zeroth and first order cumulative movement, which is a
relatively easy process. Taking this into consideration, the
approach proposed for this communication may be regarded as
the standard and most straightforward way for automatic
threshold selection that can be used for a variety of real
problems. It's a broad technique that covers a large range of
unsupervised decision-making scenarios.
III. A REVIEW TABLE ON EXISTING SIGN LANGUAGE
RECOGNITION SYSTEMS
Table 1
Ref. Classifiers/ Feature Dataset Results/
No Recognition Extraction Objective
s/
Applicati
on

2
[16] Gabor Evenly 24 static 92% [17] HMM Three-dime 53 87.71%
200 filter-based Distribute alphabet using nsional American for 3D
4 hand shape Scheme signs depth data motion of a signs context
feature and (EDS) and obtained subject's vocabulary independe
random (B) a from arms and nt, 89.91%
forest Distance kinetic hands for 3D
Adaptive sensors. context
Scheme dependent
(DAS). and
83.63%
[25] Multi-class Color Patch 26 English 75% for 2D
201 random and depth language mean context
1 forest based hand alphabets precision dependent
tracking using
combined [21] Linear SVM Depth 61 video 28.77%
vectors image sequences for DMM,
sequence, based on 15.98%
[22] HMM 8 element 494 99.2% Body American Pair-1,
feature sentences without Skeleton Sign 24.66%
vector formed by explicitly joints, Language Pair-2,
40 modelling Facial (ASL) 36.07%
vocabul-ar the finger landmarks, Overall
y words Hand
Shapes,
[18] CRF Appearance 48 signs: 93.5% Facial
threshold based hand/ 18 one Distinguis expressions
model face handed hing /attributes
detection and 30 two between
and handed signs and [20] Transition Hand 5113 91.9%
tracking (ASL) non sign Movement Shape, Chinese average
patterns Model position signs for TMM
(TMM) and vocabulary
[24] SDTW+CD block 120 signs SDTW+C movement
200 FD and tracking (Dutch DFD and
8 SDTW+ and skin language) SDTW+Q
QDFFM segmentatio DFFM IV. SIGN LANGUAGE RECOGNITION
n. outperfor CHALLENGES
m HMM
in single A critical examination of previous research has revealed
model that the field of SL Recognition has numerous flaws and inherent
problems. The flaws and challenges are generic in the sense that
[19] CNN Gradient 36 Indian 98.81% the research does not focus on the issues associated with a
Kernel Sign based on specific classification technique because there is already a lot of
Descriptor Language RGB and literature on the subject. Rather, the focus is on the broader
and Scale (ISL) depth issues that arise throughout the development of an entire SL
Invariant alphabets images recognition system. The issues are stated as below:
Feature and training,
Transform numbers 99.08% A. Segmentation
(SIFT) based on
Techniques for segmentation based on skin
1080
color have a number of flaws. Many
videos
researchers have turned to external aid to
training
overcome the issues that come with skin
color-based approaches. Several researchers
[23] Multi-stream Hand 21,960 75.6%
employ colored gloves as external aid. This
HMM Position Sign based on
has simplified the segmentation process, but it
and Language appropriat
is not well received by many. Because it is not
movements Word Data e weights
admired to first train oneself to use a system,
and 70.6%
and it severely limits the system's general
based on
application.
same
weights

3
 B. Sign Language Databases
A big SL corpus should be used to develop a
consistent SL recognition system. The majority
of the current SL databases were created in a
lab setting, which imposes a variety of
limitations. They also have a small number of
signers, which makes it difficult for the system
to generalise. It takes a long time to build a
large annotated SL database for recognition
purposes. To build an annotated dataset, an
electronic tool is required to speed up the
annotation process. ELAN Software [26] is
one of the most popular programmes, and it's
ideal for annotating SL videos.
C. Feature Extraction
When features are selected another important
step is to extract these features from the image
of the signer. Feature extraction has to
overcome challenges such as complex
background, colored gloves. COmplex
background definitely makes it difficult to
extract required features accurately. For
facilitating the process of feature extraction,
many researchers have used colored gloves.
This is definitely not required for the general
applicability of the system.
V. CONCLUSION
Sign Language Recognition System has been
developed from classifying only static signs and alphabets, to
the system that can successfully recognize dynamic movements
that come in continuous sequences of images. Researchers
Nowadays, Building a large vocabulary is more important for
SL Recognition Systems. Many researchers are developing their
SL Recognition System by using small vocabulary and
self-made databases. Large database built for Sign Language
Recognition System is still not available for some of the
countries that are involved in developing Sign Language
Recognition System. After studying the sign language
recognition system in a variety of languages and locales over the
years, It has yielded significant outcomes in terms of improving
performance. In order to accomplish so, over 25 of the most
extensively cited scholarly papers were reviewed. The author
addresses a few of the most difficult challenges. A discussion of
future potential was presented after an assessment of existing
feature extraction and classification methodologies.
VI. FUTURE WORK
The advancement in acquisition devices have provided
a promising outlook for practical recognition systems.
Researchers nowadays use more sophisticated technology. The
majority of research, on the other hand, continues to focus on
isolated sign language recognition. As a result, continuous
signing problems have been given less attention. Real-world
sign language databases should be developed based on real life
scenarios for recognition purposes.
Techniques for classification and segmentation that do
not impose significant restrictions on the signers' surroundings
should be used. For future applications, feature extraction
4
challenges should be addressed effectively. Such features should
be chosen that are unaffected by the signer's background in
order for these features to be extracted properly.
Many factors should be considered while developing
assistive technology for Deaf and hard of hearing persons, one
of which is end users' ease of use. For example, to be able to use
an SLR system in a portable device, such as a mobile phone, by
the Deaf people which they can seamlessly carry anywhere with
them.
REFERENCES
[1] Sureja Suresh, H. T. P. Mithun and M. H. Supriya, "Sign Language
Recognition System Using Deep Neural Network," 2019 5th International
Conference on Advanced Computing & Communication Systems
(ICACCS), 2019, pp. 614-618, doi: 10.1109/ICACCS.2019.8728411.
[2] Jie Huang,Wengang Zhou,Houqiang Li, and Weiping Li,”Sign Language
Recognition using 3D convolutional neural networks,” IEEE
International Conference on Multimedia and Expo (ICME), Turin 2015.
DOI: 10.1109/ICME.2015.7177428.
[3] N. Jamil, Z. A. Bakar and T. M. Tengku Sembok, "Image Retrieval of
Songket Motifs Using Simple Shape Descriptors," Geometric Modeling
and Imaging--New Trends (GMAI'06) IEEE 2006, pp. 171-176, doi:
10.1109/GMAI.2006.29.
[4] Pinar Santemiz, Oya Aran, Murat Saraclar, Lale Akarun, “Automatic
Sign Segmentation from Continuous Signing via Multiple Sequence
Alignment”, IEEE 12th International Conference on Computer Vision
Workshops, 2019. DOI: 10.1109/ICCVW.2009.5457527
[5] T. Shanableh, K. Assaleh, “Arabic sign language recognition in
user-independent mode”, International Conference on Intelligent and
Advanced Systems, 2007. DOI: 10.1109/ICIAS.2007.4658457
[6] Jose L. Flores C., E. Gladys Cutipa A., Lauro Enciso R., “Application of
Convolutional Neural Networks for Static Hand Gestures Recognition
Under Different Invariant Features”, IEEE XXIV International
Conference on Electronics, Electrical Engineering and Computing
(INTERCON), 2017. DOI: 10.1109/INTERCON.2017.8079727
[7] T. F. Ayshee, S. A. Raka, Q. R. Hasib, M. Hossain and R. M. Rahman,
"Fuzzy rule-based hand gesture recognition for bengali characters," 2014
IEEE International Advance Computing Conference (IACC), 2014, pp.
484-489, doi: 10.1109/IAdCC.2014.6779372.
[8] Thad Starner , Alex Pentland and Joshua Weaver "Real-time American
sign language recognition using desk and wearable computer based
video," in IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 20, no. 12, pp. 1371-1375, Dec. 1998,
doi: 10.1109/34.735811
[9] Britta Bauer and Hermann Hienz, "Relevant features for video-based
continuous sign language recognition," Proceedings Fourth IEEE
International Conference on Automatic Face and Gesture Recognition
(Cat. No. PR00580), 2000, pp. 440-445, doi:
10.1109/AFGR.2000.840672.
[10] Soma Shrenika, Myneni Madhu Bala, “Sign Language Recognition using
Template Matching Techniques”, International Conference on Computer
Science, Engineering and Applications (ICCSEA), 2020. DOI:
10.1109/ICCSEA49143.2020.9132899
[11] I. Rocco, R. Arandjelović and J. Sivic, "Convolutional Neural Network
Architecture for Geometric Matching," in IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 41, no. 11, pp. 2553-2567, 1
Nov. 2019, doi: 10.1109/TPAMI.2018.2865351.
[12] P. Subha Rajam and G. Balakrishnan, "Real time Indian Sign Language
Recognition System to aid deaf-dumb people," 2011 IEEE 13th
International Conference on Communication Technology, 2011, pp.
737-742, doi: 10.1109/ICCT.2011.6157974.
[13] Karishma Dixit, Anand Singh Jalal, "Automatic Indian Sign Language
recognition system," 2013 3rd IEEE International Advance Computing
Conference (IACC), 2013, pp. 883-887, doi:
10.1109/IAdCC.2013.6514343.
[14] Rokade, Yogeshwar & Jadav, Prashant. (2017). Indian Sign Language
Recognition System. International Journal of Engineering and
 Technology. 9. 189-196. 10.21817/ijet/2017/v9i3/170903S030.
[15] N. Otsu, "A Threshold Selection Method from Gray-Level Histograms,"
in IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, no. 1,
pp. 62-66, Jan. 1979, doi: 10.1109/TSMC.1979.4310076.
[16] Cao Dong, M. C. Leu and Z. Yin, "American Sign Language alphabet
recognition using Microsoft Kinect," 2015 IEEE Conference on
Computer Vision and Pattern Recognition Workshops (CVPRW), 2015,
pp. 44-52, doi: 10.1109/CVPRW.2015.7301347.
[17] C. Vogler and D. Metaxas, "ASL recognition based on a coupling
between HMMs and 3D motion analysis," Sixth International
Conference on Computer Vision (IEEE Cat. No.98CH36271), 1998, pp.
363-369, doi: 10.1109/ICCV.1998.710744.
[18] R. Yang, S. Sarkar and B. Loeding, "Handling Movement Epenthesis
and Hand Segmentation Ambiguities in Continuous Sign Language
Recognition Using Nested Dynamic Programming," in IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no.
3, pp. 462-477, March 2010, doi: 10.1109/TPAMI.2009.26.
[19] N. K. Bhagat, Y. Vishnusai and G. N. Rathna, "Indian Sign Language
Gesture Recognition using Image Processing and Deep Learning," 2019
Digital Image Computing: Techniques and Applications (DICTA), 2019,
pp. 1-8, doi: 10.1109/DICTA47822.2019.8945850.
[20] G. Fang, W. Gao and D. Zhao, "Large-Vocabulary Continuous Sign
Language Recognition Based on Transition-Movement Models," in
IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems
and Humans, vol. 37, no. 1, pp. 1-9, Jan. 2007, doi:
10.1109/TSMCA.2006.886347.
[21] C. Zhang, Y. Tian and M. Huenerfauth, "Multi-modality American Sign
Language recognition," 2016 IEEE International Conference on Image
Processing (ICIP), 2016, pp. 2881-2885, doi:
10.1109/ICIP.2016.7532886.
[22] T. Starner and A. Pentland, "Real-time American Sign Language
recognition from video using hidden Markov models," Proceedings of
International Symposium on Computer Vision - ISCV, 1995, pp.
265-270, doi: 10.1109/ISCV.1995.477012.
[23] M. Maebatake, I. Suzuki, M. Nishida, Y. Horiuchi and S. Kuroiwa,
"Sign Language Recognition Based on Position and Movement Using
Multi-Stream HMM," 2008 Second International Symposium on
Universal Communication, 2008, pp. 478-481, doi:
10.1109/ISUC.2008.56.
[24] J. F. Lichtenauer, E. A. Hendriks and M. J. T. Reinders, "Sign Language
Recognition by Combining Statistical DTW and Independent
Classification," in IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 30, no. 11, pp. 2040-2046, Nov. 2008, doi:
10.1109/TPAMI.2008.123.
[25] N. Pugeault and R. Bowden, "Spelling it out: Real-time ASL
fingerspelling recognition," 2011 IEEE International Conference on
Computer Vision Workshops (ICCV Workshops), 2011, pp. 1114-1119,
doi: 10.1109/ICCVW.2011.6130290.
[26] ELAN annotator, Max Planck Institute for Psycholinguistics, available
at: http://www.mpi.nl/tools/elan.html