Professional Documents
Culture Documents
Alphabet Signs Recognition Using Pixels-Based Analysis
Alphabet Signs Recognition Using Pixels-Based Analysis
Alphabet Signs Recognition Using Pixels-Based Analysis
2. Literature review
1. Introduction
Sign language recognition is becoming a fat area of
Sign Language (SL) is used to communicate through HCI to make mute persons able for their daily
gestures instead of sounds. Apart from Sign conversation. The crux of HCI domains is to make
Language; the language of mute people, hand computer able to recognition signs and symbols.
gestures are used in different domains like traffic Several methods have been suggested to easily and
controllers, robot instructions, hand-free computing accurately detect the signs. Common methodologies
and remote video gaming. Although there is no used for signs recognition are Template Matching[4],
international sign language, every region of the world Conditional Random Fields (CRF) [5] and Dynamic
has its own sign language. Like American Sign Time Warping ( DTW) [6].
language (ASL) and British Sign Language (BSL), The procedure of Transition-Movement Models
Pakistan has its own sign language called Pakistan (TMM) [7] is based on hand movements recognition
Sign Language (PSL) where signs are used to to deduce continuous signs. The system although
represent Urdu letters and terms. Mute people always recognizes a large vocabulary of 513 signs but as the
need interpreter to make them able to communicate costlier sensor-based glove with magnetic trackers
in the society. Let the availability of interpreter alone, are used, the technique is rarely affordable.
it is impossible for an interpreter to accompany an HMM based hand gesture recognition is proposed in
individual (mute) in every matter and every time for [8] having 82.2% accuracy in recognizing 20 signs
the whole of his life. To solve this problem, this for Arabic alphabets. The system needs pre-training
paper is the first of our long-term goal of designing a of hand gestures with which an input image sign is
Digitized Urdu Sign Language Interpreter (DUSLI). compared by using eight feature points. Most of the
Unlike other sign language systems most of which sign recognition systems are gloves based as with
rely on prior training of signs [1], the proposed gloves, fingers positions and orientations are easily
algorithm dynamically finds out which finger is detectedable. System of Al-Jarah [9] uses Cyber
small, which one is middle and so on. Keeping in Gloves while that of Kadous et al [10] needs Power
Gloves. The approach based on Data Gloves for FSL phase. Group of sign is identified in fifth phase using
(French Sign Language) [11] have 83% accuracy. positions and status of the distinguished fingers.
Although all signs of Arabic alphabets can be Signs are recognized in sixth phase while the last
recognized by the algorithm of Khaled et al [12], due phase is to produce output both in audio and visual
to light sensitivity accuracy remains below 57%. The forms. The abstract level diagram of the algorithm is
algorithm of Simon et al [13] makes use of Kinect, given in Fig.1
which using infrared camera, is independent of
lighting conditions. As some signs needs facial Core and Kernel Extractions
expression and body motion, Kinect based system
could hardly cope with such dynamic signs. Skin
color segmentation is followed in [14, 15] where Edge Detection
hand section is extracted from the image. Boundary
tracking and fingertip methodology is pursued for
sign detection in [16]. Though the algorithm is robust
Template Matching
as only 5% errors occurred in the detection of all
those signs of ASL in which fingers are open, the
approach is neither applicable for signs in which
Distinguishing Fingers
fingers are closed nor for the motion based signs. The
approach of Thinning the Segmented Image is
presented in [17]. The technique of Morphological
Group Detection
operation has been exercised in [18] for hand
gestures recognition. The work of Nagarajan et al
[19] identifies hand postures used for signs of Sign Recognition
counting with the help of Convexity defects. The
Convexity defects of fingers vary at large with a slight
orientation of hand or of fingers. Furthermore, a skin Output
color object in the background could be falsely
treated as finger. The Non-manual features extraction Fig.1 Schematic of the proposed System
is a sub-domain of sign language used mostly for
continuous signs. Head pose and motion for sign 3.1. Core and Kernel images extractions
recognition is suggested in [20] where head shakes
based signs are well recognized by trackers. Facial Core image, in the proposed algorithm, is the portion
expression and Gaze direction are traced in the of captured scene most probable to hold hand posture
approaches discussed in [21] and [22] respectively while Kernel is the image that encloses exactly and
for continuous signs. The module of T. Shanableh merely the hand gesture. The very first phase is to
extracts features by K-NN and Polynomial Networks segment out the portion containing hand posture from
[23]. Algorithm of Tauseef et al [24] employs neural the rest of the whole input scene. This is achieved in
network for alphabet recognition where learning two steps, first extracting the Core image and then
schemes requires significant training and data the Kernel image. The process to obtain meaningful
sampling. The Visual Command system is suggested hand posture (Kernel) from a scanned frame is shown
in [25] where hand posture is surrounded by ordinary in Fig. 2
daily items. The work is interesting and can pave way
for the use of new symbols in sign language domain. Capturing whole scene
HP
(4)
HP
Tm
(a) (b)
Fig.5 (a)The RGB of Kernel (b) Edge detected Kernel
Lm
Rm
3.3 Template matching
Fig.4 Binary of Cr with Lm,Tm and Rm Pixels. For easy identification of thumb, a unique Template-
sign is designed over thumb jacket (covering) that
can be seen in all the given figures. The thumb’s
template (Tt) is searched out in G of Kr using the
Considering the Rows and Columns of Cr enclosed most suitable Square Difference Matching method.
inside Tm, Lm and Rm, Kr is obtained from the Core
image.
∑ ( ) ( ) (6)
(2)
( )
3.2 Edge detection Where Tx and Ty represents width and height of the
template image respectively.
Before the detection of edges, slice of the scanned
RGB image confined by Tm, Lm and Rm is assigned to
Kr, as shown in Fig.5(a). Kr with rows m and
3.4 Distinguishing fingers I
G1
0
G2
0
(a) (b)
Fig.6 (a) (a
Finger’s tip (b) Finger’s width
The same method of sliding is repeated to find out Thumb.x ≥ index.x+20 1
)
Topmost pixel ( ) for the detection of the
remaining three fingers. To avoid the scanning of an
already identified finger, G3
( ) lying inside of
any of the previously identified fingers are omitted.
( ) * }
where i representing the four fingers, ranges from 1 3.6 Sign recognition
to 4.
The next step is to distinguish each finger from the The distinguished fingers with their respective
rest. For this purpose, leftmost ( ) is find out positions are fed to the Engine; especially designed
from the set of known ( ) . The finger thus for Isolated signs. The system dynamically assigns x
accessed first from left is sure to be SMALL finger, and y positions of each finger to a Point variable
the next will be RING finger and so on as shown in declared with the name of that finger. The Engine
Fig.7 inferring from the respective positions of
distinguished fingers recognizes the signs.
Standard deviation (SD) about y-axis of all the
fingers, except Thumb, is calculated with the
following formula to find out whether the fingers are
down are not.
3.6.2 G2 Signs recognition. In signs, thumb
lies at left of index finger as in shown in Fig.9
(9)
(e)
(a) (b) Fig.9 Signs (a) BAA (b) KAF (c) YAA
(d) ZOWAD and (e) SEEN
(c) (d)
Fig.8 Signs of (a) ALIF (b) JAA (c) BAA if(SD<8)
CHOTE-YA and (d) WAWO AND(Middle.y<Ring.y)
AND(Ring.y<Index.y)
Signs of are recognized using the following AND(Index.y<Small.y)
decision control statements. ZOWAD if (SD>6)
AND ( Index.y<Small.y )
AND (Small.y<Ring.y)
ALIF if ((SD<6) AND (Abs(Ring.y-Middle.y)<5)
AND ( Thumb.y <Index.y)) SEEN if (SD<6)
=
JAA if ((SD>6) AND (Thumb.y>Index.y+10)
AND ( Smallf.y>Ring.y ) KAAF if (Abs(Middle.y-Index.y)<5)
AND (Middle.y<Index.y)) AND(Abs(Ring.y-Small.y)<5)
= WAWO if( (SD<6) AND(Thumb.x<Index.x)
AND AND(Thumb.x>Middle.x)
(Thumb.y>Index.y+40)) YAA if (SD<6)
CHOTE if ((SD>6) AND (Thumb.y>Index.y+40)
YAA AND (Thumb.y<Index.y) AND (Thumb.x<=Index.x)
AND (Small.y<Index.y)
AND (Index.y<=Middle.y))
3.6.3 G3 Signs recognition. The signs in which reset the system for a new sign. Clicking over the
thumb is at least 20 pixels away at right from index Still button halts whole of the system while Exit
are grouped in , shown in Fig.10 button is to quit. To test the system, twenty trials
were performed for each sign by five signers.
Baa