Introduction

Flow Diagram
Real time Pakistani Sign Language
Input Recognition using Kinect
Dataset

Model

Results

UML Diagram
National University of Computer and Emerging Sciences
FAST-NUCES Peshawar Campus

2 May 2018

Aziz Ullah (P14-6111), Najeeb Khan (P14-6112), Hafiz Saud Arshad (P14-6097)

1 / 29
 Outline

1 Introduction
Introduction

Flow Diagram
2 Flow Diagram
Input

Dataset
3 Input
Model

Results
4 Dataset
UML Diagram

5 Model

6 Results

7 UML Diagram

2 / 29
 Introduction

Introduction

Flow Diagram

Input • This project precipitates to a system that will translate

Dataset Pakistan Sign Language sentences practiced by a person in
Model front of Kinect Camera attached to system.
Results
• Video based learning of signs.
UML Diagram
• User is presented with Sentence label and voice against
the sign performed.

3 / 29
Introduction

Flow Diagram Significance of Outcome

Input

Dataset

Model • The objective of this project is to provide a minimum

Results working example of a system that make a mute person
UML Diagram communicate with rest of the society using sign language.
• Social balance.
• A sense of ownership to mute citizens.

4 / 29
Introduction Scope 1/2
Flow Diagram

Input

Dataset • The system can predict signs using only one hand.
Model • Only signing hand should be at the minimum distance
Results
from the Kinect.
UML Diagram
• Currently the system can predict four sentences of
Pakistan Sign Language.
• Can be easily extended by simply extending the dataset
and then train again.

5 / 29
Introduction
Scope 2/2
Flow Diagram

Input

Dataset
• Can be installed and reconfigured by using different
Model
dataset in different environments.
Results

UML Diagram
• In a hospital.
• In a training school.
• Social centers.
• In social centers for deaf and mute.

6 / 29
 Innovation
Introduction

Flow Diagram

Input
• Firstly, there is very less work done in PSL recognition.
Dataset

Model
• The work done is mostly based on gestures(static signs);
Results
not words(dynamic signs) Recognition.
UML Diagram • We are working on words or sentences in PSL for the first
time.
• Many of the features used are thought by the team.
• Achieved got some significant results comparable to those
of simpler static signs.

7 / 29
 Flow Diagram

Introduction
Time series
Flow Diagram Input Sign Segmentation Train
of features
Input

Dataset

Model

Results Feature Time series

Input sign
UML Diagram
extraction of features

Prediction Sign Label

8 / 29
 Input

• Input was a series of depth frames from the Kinect Device.

Introduction
• This depth frame consists of 640 x 480 pixels.
Flow Diagram

Input • Where each pixel carries the distance between to the

Dataset object in the pixel
Model
• Each depth frame is converted to 8-bit gray image.
Results

UML Diagram

9 / 29
 Input Transformation

Introduction

Flow Diagram • Segmented based on darkest region or minimum intensity

Input value.
Dataset
• Skeleton is extracted using Convex Hull and Hand
Model
Contour.
Results

UML Diagram
• Finger tips and Center of the palm is identified using the
Skeleton.
• Following Features are extracted for training purpose
• Distance between two consecutive finger tips
• Angle of each convexity defect
• Minimum normal distance from convex hull to deepest
point of the defect.
• Area of the hand pixels.

10 / 29
 Transformed Input Visualization

Introduction

Flow Diagram

Input

Dataset

Model

Results

UML Diagram
Figur: Hand Segmented Figur: Hand Skeleton

11 / 29
 Alternative Approaches 1/2
Introduction

Flow Diagram

Input

Dataset

Model

Results

UML Diagram

12 / 29
 Alternative Approaches 2/2
Introduction

Flow Diagram

Input

Dataset

Model

Results

UML Diagram

13 / 29
 Dataset

Introduction

Flow Diagram

Input
• We append the features extracted from all the frames until
Dataset the video ends in an Excel file.
Model • Dataset contains 30 instances of each sentence.
Results
• The dataset consists of four unique sentences.
UML Diagram
• All and only the videos of same sentence are kept in the
same directory.

14 / 29
 Analysis of the Dataset

Introduction

Flow Diagram
• For an instance of the dataset
Input • The feature length of each frame is variable in the video
Dataset instance.
Model
• Depends upon the shape in the frame and the number of
Results
defects identified by the algorithm.
UML Diagram • For all instances in the dataset
• The Each instances of the same sign may differ in number
of frames.
• Depends upon variability of sign duration.
• For all instances of the same sign
• The frames of two instances can also differ in feature
length
• Depends upon the shape in the frame.
15 / 29
 Dataset Sample 1/2
Introduction

Flow Diagram

Input

Dataset

Model

Results

UML Diagram

16 / 29
 Dataset Sample 2/2
Introduction

Flow Diagram

Input

Dataset

Model

Results

UML Diagram

The highlighted rows represent the missing frames.

17 / 29
Introduction

Flow Diagram
Dataset Padding
Input

Dataset

Model • As discussed earlier, there is variability in the length of a

Results
video and number of features extracted.
UML Diagram
• Pad with zero to the maximum length of feature vectors.
• Pad with zero vectors to the maximum length video.

18 / 29
 Model

Introduction

Flow Diagram • Now the model is trained using Random Forest Classifier.
Input
• Random Forest is an ensemble classifier that uses multiple
Dataset

Model
decision tree classifiers.
Results • 15 decision trees are used.
UML Diagram
• Each of Max-depth 8.
• Split criteria for each node of the tree is Entropy. Which
implies the maximum information gain.

19 / 29
 Prediction using Model
Introduction

Flow Diagram • Take the sign from user, evaluate it against the model and
Input then predict the label.
Dataset
• Prediction label is selected based on the majority votes of
Model

Results
the decision trees.
UML Diagram • As depicted by the diagram.

20 / 29
Introduction

Flow Diagram
Results of Random Forest
Input

Dataset
• Cumulative Accuracy achieved is 82%.
Model

Results
• Cumulative F1-Score achieved is 0.83.
UML Diagram • The results of 10-cross folds is shown below.
• F1-Score
0.9142 0.9142 0.9142 0.8035 0.5761 0.8309 0.7416 0.8309 0.7321 1.0

• Accuracy
91.667 91.667 91.667 83.333 58.333 83.333 75.0 83.333 75.0 100.0

21 / 29
 Results

Results of Random Forest

Introduction

Flow Diagram

Input

Dataset

Model

Results

UML Diagram

22 / 29
Introduction Alternative Results using SVM Classifier
Flow Diagram

Input

Dataset • Cumulative Accuracy is 71%.

Model • Cumulative F1-Score achieved is 0.71.
Results
• The results of 10-cross folds is shown below.
UML Diagram

• F1-Score
1.0 0.833 0.675 0.7428 0.5 0.5863 0.3653 0.675 0.7321 1.0

• Accuracy
100 83.333 66.667 75.00 50.0 58.333 41.666 66.333 75.0 100.0

23 / 29
 Results of SVM Classifier
Introduction

Flow Diagram

Input

Dataset

Model

Results

UML Diagram

24 / 29
Introduction Alternative Results using Decision Tree
Flow Diagram

Input

Dataset • Cumulative Accuracy is 55.8%.

Model • Cumulative F1-Score achieved is 0.53.
Results
• The results of 10-cross folds is shown below.
UML Diagram

• F1-Score
1.0 0.833 0.675 0.7428 0.5 0.5863 0.3653 0.675 0.7321 1.0

• Accuracy
100 83.333 66.667 75.00 50.0 58.333 41.666 66.333 75.0 100.0

25 / 29
Introduction Alternative Results using Decision Tree
Flow Diagram

Input

Dataset

Model

Results

UML Diagram

26 / 29
Introduction Alternative Results using Naive Bayes
Flow Diagram

Input

Dataset • Cumulative Accuracy is 45%..

Model • Cumulative F1-Score is 0.41.
Results
• The results of 10-cross folds is shown below
UML Diagram

• F1-Score
0.502 0.675 0.521 0.475 0.182 0.171 0.214 0.375 0.555 0.497

• Accuracy
50. 66.666 50. 50. 25. 25. 25. 41.666 75.0 58.333

27 / 29
 UML Diagram

Introduction Use Case :

Flow Diagram

Input

Dataset

Model

Results

UML Diagram

28 / 29
 Thank You

Introduction

Flow Diagram
References
• H. Cooper, E. Ong, N. Pugeault, R. Bowden, ”Sign Language Recognition using Sub-Units”,GU2
Input 9PY UK Journal of Machine Learning Research 13 (2012) 2205-2231
Dataset • N. Pugeault, R. Bowden, Centre for Vision, ”Spelling It Out: Real Time ASL Fingerspelling
Recognition ”, Centre for Vision, 6-13 Nov. 2011
Model • A. Kuznetsova, L.Leal-Taixe, B. Rosenhahn, ”Real-time sign language recognition using a consumer
Results depth camera”, Institute fuer Informationsverarbeitung, Leibniz University Hannover Appelstr. 9A,
Hannover, 30167, Germany
UML Diagram • N.Tanibata, N. Shimada, Y. Shirai, ”Extraction of Hand Features for Recognition of Sign Language
Words ”, 2002
• Z. Halim PhD & G. Abbas PhD, ”A Kinect-Based Sign Language Hand Gesture, Recognition System
for Hearing- and SpeechImpaired: A Pilot Study of Pakistani Sign Language ”, RESNA , 06 Oct
2014.

29 / 29