SIGN LANGUAGE TRANSLATOR FOR MOBILE

PLATFORMS

Mahesh M, Arvind Jayaprakash, Geetha M

Dept. of Computer Science And Engineering
Amrita School Of Engineering, Amritapuri
Amrita Vishwa Vidyapeetham
Amrita University, India
Email: maheshmukund13@gmail.com, arvindjp1995@gmail.com, geetha.m.amrita@gmail.com

Abstract—The communication barrier of deaf and dumb
community with the society still remains a matter of concern
due to lack of perfect sign language translators. Usage of mobile
phones for communication remains a dream for deaf and dumb
community. We propose an android application that converts
sign language to natural language and enable deaf and dumb
community to talk over mobile phones. Developing Sign Recog-
nition methods for mobile applications has challenges like need
for light weight method with less CPU and memory utilization.
The application captures image using device camera process it
and determines the corresponding gesture. An initial phase of
comparison using histogram matching is done to identify those
gestures that are close to test sample and further only those sam-
ples are subjected to Oriented Fast and Rotated BRIEF(Binary
Robust Independent Element Features) based comparison hence
reducing the CPU time. The user of the application can also
add new gestures into the dataset. The application allows easy
communication of deaf and dumb with society. Though there are
many computer based applications for sign language recognition,
development in android platform is adequately less.
Keywords—Indian sign language, background subtraction,
thresholding, Histogram comparison, ORB
Features) which uses binary strings as an efficient feature point
descriptor.
Description using Oriented Fast and Rotated BRIEF is a
big process. In this application, not all the comparison happens
through ORB. The histogram matching happens initially to
minimize the images passed to ORB. The rate of determination
becomes faster keeping the probability of error same when we
use this method.
A. Static Gesture
Gestures are of two types- static gesture and dynamic
gesture. Those gestures that require the dynamic movement
of the hand is called dynamic gesture and those gestures that
are static, does not require the hand movement are called static
gestures.
In Indian Sign Language almost all of the alphabets and
numbers can be represented using static gestures. Also some
words can be represented using static gestures.

I. I NTRODUCTION
The sign language is a language that uses gestures of one
or both hands as the medium of communication. The disabled
people, who have hearing and speaking problems communicate
through this gesture language. The increasing urge to support
the deaf and dumb society made the sign language recognition,
an interested area for the researchers. A working recognition
system help the communication of such handicapped people
without the need of any other interpretation.
Though many research work has happened in this field, a
very few had attempted to develop a recognition system in
small hand held android phones. We propose an algorithm Fig. 1. Some static hand gestures
for android platform which captures and recognizes gestures.
The existing systems are computer based and hence common
man rarely get the chance of interaction with the system. This II. R ELATED W ORKS
application will provide a major break through to develop
recognition system in small android systems with less pro- Gesture Recognition systems are developed mostly as
cessing speed and memory. computer applications, Very few work has happened in mobile
based sign recognition. Few works are detailed below.
The algorithms used in the computer based recognition
system might not work in android platform. So our proposed The ”Indian sign language recognition system to aid deaf-
system uses Oriented Fast and Rotated Brief, an algorithm that dumb people” is a system to aid deaf-dumb people which
is the successor of BRIEF(Binary Robust Independent Element translate the sign language recognition into text with static

978-1-5090-6367-3/17/$31.00 ©2017 IEEE 1176

palm side of right hand images. It is a computer application
of an accuracy of 96.87 percent. 32 combination sets with
320 images, 10 images of each sign is present in the data set.
This being a computer application can not be implemented in
android devices.
The ”Indian sign language recognition” is another com-
puter based application that represents a framework for a
human computer interface capable of recognizing gestures
from the Indian sign language.This system can be extended for
words and sentences Recognition is done with PCA (Principal
Component analysis). This paper also proposes recognition
with neural networks. Mobile devices with such processing
systems are developing and hence implementation of such
system is a difficult task
”A Vision Based Dynamic Gesture Recognition of Indian
Sign Language on Kinect based Depth Images” recognizes
3D dynamic signs corresponding to Indian Sign Language
Words. A external device- Microsoft Kinect camera 3D images
are getting processed. By using the new method of axis Of
least inertia global feature is determined. The main goal of
our application to to determine gestures without any external
application, so the use of Kinect is not appretiated in our case..
In the IEEE paper 2012 ”Hand Gesture Recognition
for Indian Sign Language” by AS Ghotkar introduces a
hand gesture recognition system to recognize the alphabets
of Indian Sign Language. There are 4 modules: real time
hand tracking, hand segmentation, feature extraction and
gesture recognition. Camshift method and Hue, Saturation,
Intensity (HSV) color model are used for hand tracking and
segmentation. The computer application here was devloped to
determine alphabets. We have a broader view. Real time hand
tracking is difficult in mobile devices with low processing
speed.
III. P ROPOSED M ETHOD
We propose a solution to the communication barrier of deaf
and dumb community with society. We develop an android
application that determines the gestures taking less memory
and less CPU processing time. The application can be divided
to two parts - Adding new gesture into the dataset and
recognizing existing gestures from the dataset. Recognition re-
quires comparison, and hence the comparison using histogram
matching is done. The main method to identify gestures is
using descriptors while the histogram matching is done in order
to reduce the dataset for comparing using descriptors. Gestures
differ in different regions. In India itself the gesture for one
word or alphabet will not be the same in different parts of the
country.
Fig. 2. Alphabet E with 2 possible gestures
1177
A. Block Diagram
As a measure to this problem, the application is giving
the user right to add new gestures into the dataset. The user
needs to enter name of the new gesture to be added and press
’ADD’ button. The new gesture will be saved with the name
’¡gesturename¿.jpg’ inside the dataset folder.
This OnClick process requires android camera permission.
On the click of ADD button after entering the gesture name,
the device opens camera and asks user to capture image. Once
the image is captured, the image is saved in dataset folder with
the name given by the user.
B. Pre-Processing
The pre-processing is a major part of this application. The
images captured by the device camera can not be directly fed
for comparison. It must go through a series of pre-processing
steps in order to get a more accurate output.
Skin detection is done using three methods. The result of
the three methods are combined together to finally determine
the skin pixel. Skin detection using RGB, YCbCr, HSI are
the three methods used. The pre-determinant values for skin
colour are considered in each methods for detection. Once a
method confirms a pixel as skin colour pixel, a flag variable
is set. Similarly for the remaining two methods, if it confirms
skin colour, this flag variable is incremented. When all the
three methods are satisfied, the that corresponding pixel is
determined as skin colour pixel.
1) Skin Detection- RGB: Skin detection is done pixel
wise. One pixel is taken and R,G and B value of
the pixel is used for detecting skin pixels. The av-
erage RGB range for skin colour is pre-determined.
Algorithm 1: Skin detection using RGB
Result: Boolean value depending on whether skin pixel
is in the given range
if r<95—g<40—b<20—r<g—r<b then
return false;
end
if -15<r-g<15 then
return false;
end
if max(r,g,b)-min(r,g,b)<15 then
return false;
end
return true;
2) Skin Detection- HSI: Here also one pixel is taken
and the RGB values of that pixel is used to get the
HSI of the pixel which is later used for skin detection.
Fig. 3. Process flow

Algorithm 2: Skin detection using HSI Algorithm 3: Skin detection using YCbCr
Result: Boolean value depending on whether skin pixel Result: Boolean value depending on whether skin pixel
is in the given range is in the given range
h=0,s=0,i=0; y=0,cb=0,cr=0;
i=r+g+b/3; y=(0.257*r)+(0.504*g)+(0.098*b)+16;
r=r/i; cb=(0.439*r)+(0.368*g)+(0.071*b)+128;
g=g/i; cr=(0.148*r)+(0.291*g)+(0.439*b)+128;
b=b/i; if cr=((1.5862*cb)+20) then
w=0.5*(r-g)+(r-b)/(r-g)2̂+(r-b)2̂ ; return false;
h=cos(w); end
if b>g then if cr<=((0.3448*cb)+76.2069) then
h=2*PI*h; return false;
end end
if h<25 — h>230 then if cr<=((-4.5652*cb)+234.5652) then
return true; return false;
end end
return false; if cr=((-1.15*cb)+301.75) then
return false;
end
if cr=((2.2857*cb)+432.85) then
return false;
end
return true;

4) Thresholding: Thresholding is the process of chang-

ing all the pixels in the image to binary, black or white
3) Skin Detection - YCbCr: Each pixel is taken one by one, depending on some criteria. Here the criteria used is -
the RGB values are converted to YCbCr and then checked. whether all the skin detection algorithms are satisfied or not.

1178
 Algorithm 4: Thresholding
Result: Pixels changed to either black or white
i=0;
a=0;
white=0*00ffffff;
black=0*00000000;
while i<pixel.length do
i++;
if isSkinRGB()==true then
a+=1;
end
if isSkinHSI()==true then
a+=1;
end
if isSkinYCbCr()==true then
a+=1;
end
if a==3 then
pixel[i]=white;
end
pixel[i]=black;
end
5) Resizing: The android devices nowadays are having high
quality cameras. As the quality increases the size of the image
also increases. Large sized images are difficult for processing.
Hence the image is resized. The height and width of the
captured image is taken and reduced by half.
C. Recognizing existing gestures
The difficult part of the application is the recognition
phase.It is done with 2 different processes. The dataset will
be large and hence to process every image under ORB() will
be a time consuming task. Hence before ORB(), the images
are first compared using histogram matching. Only the images
that pass through this matching process goes to ORB().
The device should compare images and find the match.
Comparison is done with all the images in the dataset. Prior
to the first best match, the system should not stop iteration, the
comparison should continue till all the images in the dataset
is compared. There can be more than one output as there
can be two or more alphabets or words with same gesture
representation.
Recognition phase starts with the camera for capturing
image. Here also, it requests device permission to open camera.
On click of recognize button camera opens and user is asked
to take image. Once the image is taken, it is pre-processed.
The images present in the dataset are loaded one by one on
each iteration. The loaded images are later pre-processed. The
two pre-processed images are fed to the comparator which
uses Histogram matching and ORB descriptor comparison to
compare the images and give the image name as output if good
match is found.
1) Histogram Matching: Compute the histograms of the
two images.With the histograms, cumulative distributive func-
tion (F1 and F2) for two images are calculated. For each
gray level G1 between 0 and 255, G2 is found such that
F1(G1)=F2(G2). The value of G2 is taken as a compare
variable.
1179
The first phase of comparison happens here. Not all
the images are fed to ORB(). As ORB() is a big op-
eration and requires more time, the data set of im-
ages fed to ORB() is reduced by doing histogram
matching first. After histogram matching, if the compare
value is between a possible range it is fed to ORB().
Algorithm 5: Histogram Matching
Result: Returns text and calls method depending upon
G2
compare=histogram1.compare(histogram2);
if compare==0 then
”Exact duplicates”;
end
if 0<compare<1500 then
”Possible Duplicate”;
ORB();
end
if compare>1500 then
”Different images”
end
”No image”
2) ORB: ORB is an efficient algorithm developed as an
alternative for SURF/SIFT to produce the good match among
images. It is FAST key point detector and BRIEF descriptor.
Here in ORB, the FAST algorithm is used to detect the
key points. As all the key points will be detected and we only
need selected N points, we use the Harris algorithm. Scaling
is done with pyramid transform.
As the normal BRIEF does not do well with orientation,
here steered BRIEF is used. The steering is done by some
factors. The intensity of the centroid is determined. The
direction from a corner to this centroid is determined which is
later used for steering. Greedy algorithm is used to determine
the low co-relation between pixel blocks and the descriptor is
determined.
After determining the descriptor and key points of both
images, the good match is determined. If the matching value is
above the given match value, then these images are considered
as possible matches.
After histogram matching, the captured image along with
the image from reduced dataset are fed to ORB(). Here key
point of both images are determined and image description
using ORB descriptor is determined. A good match list is
produced. If the image matching percentage is greater than
our set value, the name of the loaded image(from dataset) is
displayed as output.
IV. R ESULT AND P ERFORMANCE E VALUATION
For developing an android application, Android Studio
latest version is downloaded and installed(Version 2.2.3). The
SDK components required for the mobile device is determined
and installed. The SDK components depends on the version of
Android the device is having. OpenCV version 2.4.1 is used.
It is added as a module to the android project. Compile time
dependency is given to the added module.
The android application is tested with the obtained dataset
and results are formulated. The performance of the application
 A. Testing
The static gestures corresponding to Indian Sign Language
is added as images to the dataset folder. From this, a few
gestures are chosen which are difficult for the system to
distinguish and determine. Each selected gesture is checked
number of times and outputs are noted. The outputs are
analyzed and tabular form of the results is made.
1) Results: Different alphabets and numbers are taken and
tested. These taken images are compared with the ones in
the dataset. Repeated tests had been taken and outputs are
recorded. In the table, each column corresponds to a n alphabet
or number obtained and each row corresponds to an alphabet
or number tested and the result it obtained after testing. If 1
is the cell value, then the testing character gives the column
heading as output. If 0 then vice versa.

Fig. 6. Matrix of result obtained(1 corresponds to the output obtained For

eg: In testing for gesture 1 result obtained is 1 and I)

Fig. 4. ORB
Representing the testing results in tabular form. It shows
the characters tested, the corresponding output obtained, and
is measured using accuracy of the result obtained. This analysis the accuracy of the output obtained with the correct output.
will help for future developments of the application.
1) Dataset: The dataset is containing gestures of alphabets
and numbers. All the alphabets that can be represented as static
gesture and first ten whole numbers are added in the dataset.
As the user can add to the dataset, the size of the dataset
remains not fixed.

Fig. 7. Matrix of result obtained(1 corresponds to the output obtained For

eg: In testing for gesture 1 result obtained is 1 and I)

2) Performance Evaluation: Accuracy of the output is

determined by taking number of test cases.
Accuracy(%) = Expected output/Actual output ∗ 100 (1)
Accuracy graph is prepared using this data.

V. C ONCLUSION
A hand held gesture recognition system for the common
Fig. 5. Dataset man makes this a unique contribution. Interaction with the deaf

1180

Fig. 8. Pie chart of accuracy
and dump society will be increased and discrimination towards
them will be reduced.
Static hand gestures captured by the device camera is
processed and the corresponding gesture is determined with an
accuracy around 70 percent. The output text obtained as the
result of comparison will help the deaf and dumb society to a
great extent. As the user can add new gestures to the dataset,
the application might not get outdated. We are planning to
extend our work to dynamic gesture recognition.
Dynamic gestures that have dynamic movement of the hand
requires video processing to determine the gesture. Goal is to
develop an android application that recognizes both static and
dynamic gestures at a higher accuracy rate.
7) Rajam P S, Balakrishnan G,”Real Time Indian Sign
Language Recognition Systern to aid deaf-dumb peo-
ple”, Communication Technology (ICCn, 2011 IEEE
13th International Conference, pp. 737- 742.
8) Nadia R. Aibelwi, Yasser M. Aiginahi, ”Real-Time
Arabic Sign Language (ArSL) Recognition” Interna-
tional Conference on Communications and Informa-
tion Technology 2012.
9) IR Sensor-Based Gesture Control Wheelchair for
Stroke and SCI Patients Megalingam R.K, Rangan V,
Krishnan S, Edichery Alinkeezhil A.B Humanitarian
Technology Labs, Electronics and Communication
Department, Amrita Vishwa Vidyapeetham Univer-
sity, Kollam, India Infosys, Pune, India University of
New Mexico, Albuquerque, NM, United States. IEEE
Sensors JournalVolume 16, Issue 17, 1 September
2016, Article number 7501503.
10) MK Bhuyan, D Ghosh, and PK Bora. Finite state
representation of hand gesture using key video object
plane. In TENCON 2004. 2004 IEEE Region 10
Conference, pages 579582. IEEE, 2004.
11) M Geetha, Rohit Menon, Suranya Jayan, Raju James,
and GVV Janardhan. Gesture recognition for amer-
ican sign language with polygon approximation. In
Technology for Education (T4E), 2011 IEEE Inter-
national Conference on, pages 241 245. IEEE, 2011.

VI. R EFERENCES
1) T.Starner and A. Pentland, Real-time american
sign language recognition from video using hid-
den markov models, Technical Report, M.I.T Media
Laboratory Perceptual Computing Section, Technical
Report No.375, 1995.
2) Geetha M, Manjusha C,Unnikrishnan P and Harikr-
ishnan R, A Vision Based Dynamic Gesture Recogni-
tion of Indian Sign Language on Kinect based Depth
Images May 21 2013.
3) Kusurnika Krori Dutta , Satheesh Kumar Raju , Anil
Kumar G ,Sunny Arokia Swarny Double Handed
Indian Sign Language to Speech and Text, 2015
Third International Conference on Image Information
Processing.
4) Shangeetha.R.K, Valliammai.V, Padmavathi.S. Corre-
spondence Dept. of Information Technology, Amrita
Vishwa Vidyapeetham, Coimbatore, India. Computer
vision based approach for Indian sign language char-
acter recognition 2012 international conference on
machine vision and image processing M V I P 2012
Coimbatore Tamil Nafu
5) MJerin Jose V.Priyadharshni et.al, ”Indian Sign Lan-
guage (ISL) Translation Systern For Sign Language
Learning,”IJIRD, vol. 2, May,2013, pp. 358-365.
6) R. Rokade, D. Doye, and M. Kokare, ”Hand Ges-
ture Recognition by Thinning Method” International
Conference on Digital Image Processing (2009).

1181