See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/324694726

ACE Assisted Communication for Education: Architecture to support Blind &

Deaf communication

Conference Paper · April 2018

DOI: 10.1109/EDUCON.2018.8363341

CITATIONS READS

10 819

5 authors, including:

João Pedro Pinto da Silva Ulisses Tiago Oliveira

University of Vigo Polytechnic Institute of Porto
16 PUBLICATIONS   33 CITATIONS    9 PUBLICATIONS   42 CITATIONS   

SEE PROFILE SEE PROFILE

Paula Maria Escudeiro Nuno Filipe Escudeiro

Polytechnic Institute of Porto Polytechnic Institute of Porto
151 PUBLICATIONS   392 CITATIONS    79 PUBLICATIONS   252 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Gaball View project

PRAXIS View project

All content following this page was uploaded by João Pedro Pinto da Silva Ulisses on 24 May 2018.

The user has requested enhancement of the downloaded file.

 ACE Assisted Communication for Education:
Architecture to support Blind & Deaf communication
João Ulisses1, Tiago Oliveira1, Paula Maria Escudeiro1, Nuno Escudeiro12, Fernando Maciel Barbosa3
1
ISEP - School of Engineering, Polytechnic Institute of Porto, Portugal,
2
LIAAD - Laboratory of Artificial Intelligence and Decision Support, INESC Tec, Portugal,
3
FEUP - Faculty of Engineering, University of Porto, Portugal,
{jppsu, taqol, pmo, nfe}@isep.ipp.pt, fmb@fe.up.pt
Abstract—Deaf and blind students face communication barriers
that are constantly present in their daily lives. These barriers arise
naturally since the deaf community, the blind community and the
rest of the students and teachers use different languages and
different channels to communicate. These barriers have a
significant impact in the academic, personal and professional
development of deaf and blind students. Using automatic tools to
assist the fluid communication between people using different
languages and different channels of communication might
significantly promote the social inclusion of disabled students. In
this paper we describe the ACE architecture which incorporates
VirtualSign, a translator for sign language and other components
to allow for the real time translation between sign and oral
languages. This architecture supports any sign language. We
expect ACE will provide ways for the fluid communication among
deaf people, blind people and those not constrained by these
disabilities.
Keywords—Inclusive school; sign language translator; 3D
avatar; gloves sensor; Blind & Deaf communication)
I. INTRODUCTION
Deaf and blind students face serious challenges in
nowadays education settings. Communication barriers are
constantly present and arise naturally due to the use of different
languages. Different forms and channels of communication by
the deaf community, the blind community and the rest of the
students and teachers often leads to loss of information.
Although some deaf can read text fluently, they are a minority.
Deaf students’ mother language is sign language with its
proper grammar quite distinct from oral languages. Blind
people rely on voice to communicate while deaf people rely on
visual representations; these channels are independent and
prevent any type of communication between these two
communities.
For deaf children, reading is troublesome due to difficulties
in understanding the meaning of the vocabulary and the
sentences. In sign language they use sentences only word by
word while filtering out some elements of the speech, e.g.
articles. The order of these also comes in a structure for
example having subject then object then verb, often having the
time and location referenced at the beginning of a sentence. It
is hard for the deaf to understand an oral language grammar is
different and more complex [1][2] than sign language’s
grammar. This fact together with the lack of communication
via sign language in schools severely compromises the
development of linguistic, emotional and social skills in deaf
students [3].
Assisted Communication for Education project (ACE)
proposes an architecture to support real time translation
between sign and oral languages in both ways. This translation
can then be used by the blind, so they can perceive the sign
language as speech. Similarly, blind people can communicate
with the deaf by using speech or written text which is
converted to sign language. This bidirectional translation is
available to assist the communication between the deaf and
non-deaf communities using the VirtualSign tools [4][5].
Promoting equal opportunities and social inclusion of
people with disabilities is one of the main concerns of the
modern society and also a key topic in the agenda of the
European Higher Education [3].
Fluid communication is of utmost importance as it impacts
accessibility to education and hinders creativity. ACE seeks to
promote inclusive education by offering tools to assist the
communication with and between the deaf and blind
communities.
ACE makes use of gaming technology such as an avatar
with 3D animation, glove sensors, Unity game engine, used in
entertainment games and serious games. This facilitates a link
with educational content, specially through serious games.
II. RELATED WORK
In the last two decades a significant number of works
focusing on the development of techniques to automate the
translation of sign languages was proposed. However, many of
them are directed for the American Sign Language [6]. The
introduction of serious games in the education of people with
speech or hearing disabilities have been published in [7] being
accessible educational content. However, these only make use
of several of the methods proposed to perform representation
and recognition of sign language gestures apply some of the
main state-of-the-art techniques. These involve segmentation,
tracking and feature extraction as well as the use of specific
hardware as depth sensors and data gloves [3].
ProDeaf is an application that does the translation of
Portuguese text or voice to Brazilian sign language [3]. The
objective of ProDeaf is to make the communication between
mute and deaf people easier by making digital content
accessible in Brazilian sign language. The translation is done
using a 3D avatar which performs the gestures. ProDeaf
already has over 130 000 users [3].
 Showleap is a recent Spanish Sign language translator, it
claims to translate sign language to voice and voice into sign
language [3]. At the moment Showleap has no precise results
on the translation and the creators claim the product is 90%
complete [3].
Motionsavvy Uni is another sign language translator that
makes use of the leapmotion [3]. This translator converts
gestures into text and voice into text. Text and voice are not
converted into sign language with Motionsavvy Uni. The
translator has been designed to be built into a tablet.
Motionsavvy Uni claims to have 2000 signs on launch and
allows users to create their own signs [3].
Spreadthesign is a project with the participation of nine
European teams composed of deaf people and listeners from
various institutions dedicated to working with deaf people from
the following countries: Portugal, Spain, Lithuania, Sweden,
the Czech Republic, Turkey, France, Germany, the United
Kingdom and even Russia, Japan, Finland and the United
States of America [8]. The main objective of the project is to
collect gestures at the national level of each participating
country to build a multilingual, digital, online dictionary in
different thematic areas, combining the national and gestural
languages of the different countries involved [8]. This is a way
of promoting the access of the deaf to the vocational training
programs in transnational mobility schemes [8].
In terms of sign language to text translation, an early
example of sign language to text translation comes from
Sagawa et al. [9] in 1995 in the form of a Japanese sign
language to text translator and vice versa. They studied two
different approaches for sign to text: image processing or hand-
based input but ended up concluding the technology at the time
was not yet accurate enough to capture the needed information
from the picture, which made them continue the work using
data gloves. They were able to reach a 97% success rate, even
though their dataset was definitely small, it was still an
impressive result for that time. In recent years we have many
more examples of successful translation.
Akmeliawati et al. [10] had great success with their
implementation of a sign language translator which makes use
of color segmentation and neural networks for a word-by-word
translation of Malaysian sign language to English text,
achieving over 90% success rate. The shortcomings suggest
their application only detects hand movement and thus cannot
disambiguate between contexts on sentences hence the word-
for-word approach chosen. They use a simple webcam with
simple gloves adapted with color strips on key spots, then
analyze the video input via image processing techniques and
use neural networks to classify the gesture. It is a cost-effective
approach but the capabilities for a full translation are far from
our plans for VirtualSign. Chai et al. [11] opted to use Kinect
for their implementation of a Chinese sign language translator
that converts gestures into text with an accuracy of over 80%.
They make use of 3D trajectory matching techniques,
comparing the motion of each hand and matching it towards
known gestures. Kinect is also one of our chosen technologies
for translation, but not with this implementation, seeing as it
does not allow for hand configuration or facial expression
recognition, which can prove crucial in a full translation.
Mohandes et al. [12] published a detailed compilation of the
most successful image-based and sensor-based approaches to
Arabic sign language translation. While it is possible to achieve
positive results separately, the authors concluded combining
both approaches should wield the best result in future real-time
translation applications. This last thought comes in agreement
to the VirtualSign approach to translation and sign recognition.
Considering related past research projects, ACE has
scientific information and results to infer a stable architecture
to formalize the complex data of multiple sign languages and
other communications types.
III. SIGN LANGUAGE
In general, despite most gestures being different among
different sign languages, they mostly use combinations of a
common set of hand configurations and general syntactic rules,
such as, not using articles.
In order to learn most Portuguese Sign Language rules, we
engaged in meetings with professional sign language
interpreters and deaf people. Currently we are arranging and
taking part on more meetings for other sign languages in
multiple countries.
Some of the rules include:
The general movements can be linear, obliquus, in arc or
circular. On the other hand, the internal movements consist on
the change of hand configuration and orientation during a
gesture, like fingering, hook, twist or friction, among others.
Gesture as a whole can be analyzed in sub-units –
“queremas” (same as phonemes), which are defined by the
location, configuration and movement of the Dominating Hand
(M1) or Non-Dominating Hand (M2). The intensity and
velocity of the motion can be an indicator of pronunciation,
while the facial expression can express emotions related to the
gesture.
There are three types of gestures: Dominating Hand
gestures, executed solely by M1; Base Hand gestures, executed
with both hands with M2 serving as base or support for M1;
Bimanual gestures, using both hands with the same
configuration but incorporating motion.
Facial expression and body movement can also accentuate
the gestures in three different ways: interrogative, in which the
speaker raises his head and the eyebrows remain frowned or
raised, with his body moving forward a little and the arms
moving up or down; exclamative, where the speaker moves his
body forwards or backwards to express joy, surprise or
reluctance; declarative, where the speaker remain with a neutral
or null facial expression [2].
Sign language involves a set of components that make it a
rich but hard to decode communication vehicle [3]. Although it
is not as formal and not as structured as written text it contains
a far more complex way of expression [3]. When performing
sign language, we must take into account of a series of
elements that define the manual and non-manual components
[3]. By changing one of these elements the gesture changes or
loses its meaning. At the level of manual component, there is
the definition of dominant and non-dominant or supporting
hand [3]. Usually for each person, the dominant hand coincides
with the hand with greater dexterity. In the execution of
gestures, the acting of the dominant hand may possibly differ
from the support hand [3].
Manual elements of sign languages include: Hand
configuration: the hand shape while executing the gesture. In
Portuguese sign language there is a total of 57 identified hand
configurations, shared between the dominant and supporting
hand [3]. Orientation of the palm of the hand: some pairs of
configurations differ only in the palm’s orientation [3].
Location of articulation: the place of articulation comprises the
different areas of the body where the gestures are performed in
the gestural space. Some gestures are intrinsically connected to
a contact point e.g. eyes, neck, chest, trunk. Others are held in
front of the person, without any contact point as in the case of
the alphabet [3]. Hand movement: characterized by the use of
one or two hands and by the motion of the hands in the gestural
space [3].
Non-manual component comprises: Body movement:
responsible for introducing a temporal context. The torso
leaning back, upright or forward indicates the communication
in the past, present or future, respectively [3]. Facial
expressions. The facial expressions add a sense of emotion to
the speech, a subjective experience, associated with
temperament, personality and motivation. Two distinct
emotions cannot occur simultaneously, since the emotions are
mutually exclusive. This is one of the aspects most emphasized
by the deaf community [3].
Sign language, like any other living language, is constantly
evolving and becoming effectively a contact language with
listeners, increasingly being seen as a language of instruction
and learning in different areas, a playful language in times of
leisure, and professional language in several areas of work
[12].
Portuguese Sign Language is a system constantly evolving
and renewing, thus offering a world full of possibilities [2]. It
remains to be understood, accepted and respected [2]. For this
we need an architecture able to withstand the complexity of
sign languages, their interactions and how they evolve.
We are setting up the translation models for several sign
languages, including the sign languages from Portugal, Brazil,
Slovenia, England, Cyprus, Greece and Germany. We are
gathering structural rules of these sign languages through
interviews and structured surveys to be filled by our partner’s
experts.
IV. ARCHITECTURE OF ACE
Architecture of ACE can be divided in the two main areas:
Blind Architecture and Deaf Architecture (VirtualSign). In the
future it will have more architectures for each type of
disability. The VirtualSign project has been more focused on
the deaf architecture due to the issues the society has in solving
them. ACE is currently expanding on VirtualSign as a
continuation and expanding to other disabilities and
interactions. Although VirtualSign did not have a meta-analysis
and a large-scale architecture oversight, their contents will be
put into ACE and the architecture will be done with ACE
components.
Our blind architecture will make use of generic APIs, such
as Web Speech API, and integrate them with other
architectures, such as our deaf architecture, allowing
communication between all types of users and allowing
collaborations. These collaborations are important to enrich
creativity, which is a highlight part of this architecture.
Blindness is part of creativity [14], we can perceive in our
imagination this when reading a book. This part of creativity is
indeed noticeable on blind people daily lives and the way they
perceive things, which can lead to interesting creations when
they are allowed to create. ACE seeks to enable and exploit this
creativity by using digital games and digital tools.
As different types of users collaborate which each other,
interactions are made. There are multiple types of interactions,
resulting from inputs and outputs of blind and deaf
architectures. These interactions have architectures on their
own, such as: Collaborative Art, where we want to explore the
deaf and blind creativity since they have different capabilities.
Serious games where we can explore game mechanic concepts
to teach using Bloom’s taxonomy [15] and other similar
architectures [16][17][18]. In education a possibility linked to
serious games is virtual classrooms, where the students can be
more participative and communicate with ease by using ACE
and VirtualSign tools. The serious games created can also be
used for entertainment and to increase the interaction of online
communities. For this to be easier, game mechanics can be
used so players have fun despite their incapacities. Their
enhanced senses can be used to solve tasks within the game,
giving them a greater sense of achievement, such abilities can
be echolocation for the blind and parallel vision for the deaf.
A well-designed game entices players into the “reality” of
the game world and keeps them there until the goals of the
game have been met [19].
In GILT (Games Interaction & Learning Technologies)
laboratory there are serious games being made by researchers
using various hardware such as smartphone, Microsoft’s Kinect
and more recently Emotiv Epoc+. These serious games
integrate in our Educational architecture which is linked with
the other architectures. Some previous work relating serious
games is in [20][21] which includes our serious game for the
deaf that makes use of VirtualSign tools [5][22].
One of these games created for the deaf is a multiplayer
game as described in [3]. The game consists of a first-person
puzzle game and requires two players to cooperate in order to
go through the game [3]. This game has a chat where users can
type but it also makes use of the VirtualSign translator to allow
deaf users to chat within the game using sign language [3].
Cooperation is not only necessary but also rewarded as the
players performance is represented through a score [3]. The
players can compare their performances as team with others
through the highscore list [3].
Two games were created for the blind, Virtual Adventure
and Game2Sences [23]. These games used Microsoft’s Kinect
and smartphone hardware respectively.
Game2Senses project intends to reduce the social barriers
between blind community and those not suffering from visual
disabilities [23]. It also intends to increase the quantity, variety
and quality of computer games for visual impaired. This
achieved the specified goals through a two-fold approach:
demonstrated a way to present interactive and immersive
entertainment for blind using spatial audio experiences and
presented interactions between players in the game [23]. This
game has a graphical interface, to promote blind and non-
visually impaired to play the Game2Senses and interact with
each other [23].
Some applications were also created out of the scope of
digital games. Figure Out is a mobile application for deaf
tourists however any tourist can find it useful, as it translates
any language from text captured via the smartphone camera.
This text is translated in a simplified way so the deaf tourist
can understand. This was mostly used for restaurant menus,
public transport guidelines and schedules and other important
tourist’s routines.
Alarm Alert is another mobile application for the deaf,
which warns the user about nearby danger. This danger is
detected by the smartphone running the application which is
receiving soundwaves and filtering them to detect danger
scenarios. Danger scenarios are varied, like an ambulance,
thunder and other were considered important for the deaf to
identify in their daily life. When an occurrence is detected the
smartphone vibrates and a picture with the event is shown for
the user to quickly identify what is happening nearby. This
application gives important sound awareness to the deaf, but
more importantly the sounds are properly studied and filtered
so it does not give false positives, which could cause major
problems.
As continuation of past work, serious games have proven
multiple times to be a successful source of information and
method to study and validate hypothesis. With this in mind a
new set of serious games are being devolved, using Emotiv
Epoc+ to increase the scope of inclusiveness and reach out to
more people. The game mechanics are also definitely different,
commencing with a card game, a turn game, as we hypothesize
it will suit the time needed for answer and communication, as
well creativity in strategy and creating rules or cards for the
game itself.
The nature of card games by being a turn game will allow
the players to have more time to cogitate and give an adequate
answer in their turn, without needing to worry about reaction. It
is known games require reaction are more appellative to young
player as generally old players have less reflexes. With this in
mind it would be more prudent to create a game which relies
less on reaction and more on strategy for the mobility impaired
players using Emotiv Epoc+. This also facilitates the multiple
types of communication in an online card game match as they
can wait for the translation to be done. Another aspect to
consider is to exploit the different types of more developed
senses. As individuals with certain difficulties certain other
capabilities rise. These capabilities such as peripheral vision,
echolocation and others could be used as means to enhance
creativity as seeing things differently can have an impact on
creativity. Imagination and creativity to solve strategic
problems by using these senses can be compared with deep
learning algorithms from which new algorithms could be
formulated for the search the best strategy. Creativity itself
could be studied in this scope to enrich the computer
perception of utility, quality and originality thus again creating
more interesting algorithms to be used by artificial intelligence
agents.
Collaborative creativity can be used in education as a mean
to give true inclusion of the students by allowing them to work
in groups to solve problems, explore and learn by themselves.
It would also be interesting to see how collaborative creativity
or creativity when playing against each other to win a game
works and try to extract if a type of individual would have an
advantage in which situations. This leads to a more appealing
way to create lectures and educative content dedicated to
specific types of individuals.
For the ACE architecture, creativity modules are contained
in each branch of the architecture, in which we are currently
giving more emphasis in the deaf architecture due the
complexity of sign language.
Deaf architecture (VirtualSign), which at the moment is the
most complex, we plan to expand on VirtualSign and use it on
ACE linking it with the other architecture types. This one is
divided in two main areas, text to sign and sign to text. The
online tools for the deaf are the translator itself, and the
configurator, which are the input and output of the deaf
architecture respectively. Deaf architecture supports text-to-
sign and sign-to-text translation inertly, when linked to the
blind architecture it supports translation through voice.
V. TEXT TO SIGN
Two main VirtualSign tools assist in the text-to-sign
translation module: Translator and the Configurator, both are
components of the VSSO (VirtualSignStudioOnline). These
tools are developed in Unity and built into WebGL, so they can
be used online through most popular internet browsers. Other
builds are also used, depending the type of user, they are made
in specific to Windows, Android and others, reducing possible
technology barriers.
VSSO Translator works like an online application and is
meant to be used daily at convenience, VSSO can also be an
Android application. Users can write text in a white box, which
then it be translated to any of the I-ACE (international version
of the project ACE) partner’s languages, including, so far:
Portuguese, Slovenian, German, British, Cypriot and Greek
sign languages as well as LIBRAS, the Brazilian sign
language. Our avatar does the required animations as gestures
performing the translation of text to sign language. The
translation is performed by our 3D avatar checks the input
words in our validated signs database, which returns the
gesture information according to the specific country sign
language grammar rules. Another utility of this tool is it can
serve as connection from text to animation for other tools,
digital games, plugins and other software by calling VSSO
translator methods.
VSSO configurator is based on 3D animation configuration
programs, such as Maya, 3DSMax and Blender. In the VSSO
configurator is where the deaf and experts will add and validate
the gestures, this can be done online or offline. This tool uses
our 3D avatar and can be manipulated like an 3D animation
configurator, but with sign language specific work tools. Arms,
hand configuration, body movement, head and facial
expression are all customizable with several degrees. This is
customized for each moment of a sign, each sign has several
time moments in which the hand has a certain sum of states.
When creating a sign, the user can reuse and import other
similar gestures and edit accordingly to the new gesture being
configured. This allows the user to save time to create multiple
signs, as information such as hand, body, head, arms is already
configured and can be quickly edited for all the moments of a
sign. It is possible for an editor using the tool to copy and
mimic movements from websites which contain videos of
signs, thus adding those words to VirtualSign database. This
work is normally done by the editor but can also be done by the
validator. Each user can be a validator or an editor, each having
a unique login, which has information about the user stored in
our VirtualSign database This information is important to know
which sign languages the user knows, expertise, what
languages they are adding content to and other trivial generic
information. The validator is responsible to validate the signs
created by editors, the signs have three states, validated, in
review and not validated. The signs are to be validated by an
expert which can also correct the movement or send a request
to the editor to change the animation. Editors can be added
voluntarily in our application by signing up, our application is
made so they can work on their own. However, there is control
to whom we allow to create the signs and specially who
validates them, this control is made by the us, the admins of
each team and by sign language experts, accepting or refusing
their new accounts according to their expertise. Editors do not
require to know sign language, as they can watch websites like
Spreadthesign, which contain videos of gestures than can be
copied into VSSO configurator. Some languages not included
in Spreadthesign have their own similar website with sign
language online, such is the case of Slovenia.
Each word can also have a context input which will help
disambiguating semantics and also help with grammar rules.
This feature is currently being worked on, as it needs an
automatic structure can be validated by the experts. This
automatic structure is being tested and used by the editors and
validators online to gather context information in a valid form.
This structure is of the utmost importance as otherwise experts
would fill out this information with no structure, causing erratic
information matching, lack of information and other issues
later. A following future publication will further explain our
context structure and data collection online which is within
ACE.
Once the words are validated they become automatically
available for the translator tool, which is updated in real time
for all users.
Interpreter and expert’s opinions revealed some animation’s
speed is not perfect, and they could have more information
such as elbow position. Animation speed quality and file size
can be improved by using glove sensor data instead of
animation file. An overhaul of the gesture animation to a more
parametric animation approach is being made, but the data of
the gestures can still be used, as this is to improve their visual
performance. The same is being done for the hand animation
configurations to increase their performance due the file size,
as well a new hand configuration configurator in which new
hand configurations can be created and later used or
recognized. This is especially useful for adding new sign
languages to the system.
A plugin for Microsoft’s PowerPoint is being developed
using text to sign VSSO tools. PowerPoint text is translated
into animations with the sign gestures. These gestures can be
placed on each PowerPoint’s slide allowing deaf students to
better understand the content or to study at home. Professors
can have now the opportunity to teach deaf students without
mastering sign language. Better yet is the fact PowerPoint
presentation can be translated to different sign languages when
needed. Students can also use the plugin on their own to
convert any presentation helping them to better understand the
content in the PowerPoint.
VSSO will also implement voice recognition and text to
speech online. It is important to be online not only for it to be
more practical as it can be used in any platform, specially
smartphones, also to be always ready to use anytime, without
the issues that come with installing software. These are to be
integrated with the PowerPoint plugin as well so the
presentation works for hearing blind students. Text or words is
our base form which is why our next focus will be receiving
more information, such as context. The way this module works
is simple, voice recognition converts voice to text and from text
it is convert to animation, allowing the blind to speak with the
deaf. The opposite direction is also possible by using sign to
text, the gesture is detected and converted to text, from text it is
converted to speech by using text to speech algorithms.
VI. SIGN TO TEXT
Sign to text portion of VirtualSign consists mainly on 3
applications: the hand capture app, the sign language to text
translator and another approach to our configurator.
Translation from sign language to text is a complex process
due to the great number of variables to capture and analyze:
hand configuration, hand and arm position, motion speed, body
movement, and facial expression, among others. And although
in theory it may seem it is an exaggerated number of factors for
a language, every detail is important to a correct interpretation
and translation from sign language. Every piece of information
the body provides while performing the gestures can help not
only to identify the word or sentence, but also the specific
context the speaker is referring to [24].
During our talks with sign language specialists, speakers
and translators we reached the conclusion it is possible to
divide Sign Language into three different pieces: hand
configuration, body motion and facial expression. As such we
always try to use a technology or combination of multiple
systems that allow us to capture the most of these three factors
as possible for a more accurate translation.
In terms of technology we tried different approaches using
image processing techniques on normal webcams thinking
about reducing cost and improving accessibility to the
applications. But unfortunately, we found the results were not
satisfying enough to use on the context we needed and the
techniques which were more accurate were also heavier on the
hardware, which is a huge problem when we consider the
translation must be done in real-time and without any control
over the hardware it is installed on, so we had to choose other
routes.
Our chosen technology at the moment is combining input
from a pair of data gloves with body movement analysis using
a Kinect sensor. Our selected data gloves are 5DT Data Glove
14 Ultra, manufactured by Fifth Dimension Technologies [25].
These gloves measure not only finger flexure but also the
abduction between the fingers, with 14 sensors on each hand,
two on each finger and the other four in between them. The
amount of sensors is perfect for an accurate capturing of hand
configurations and the gloves allow for an individual
calibration per user on real time, which reduces the
documented problem [12] of different hand sizes and or finger
lengths interfering with the accuracy of the captured data. Our
second piece of technology is the well-known Kinect
developed by Microsoft, specifically the improved version 2
[26]. Although some research shows using its depth sensor and
applying image processing techniques on it can wield positive
results on small amounts of data and words [27], we find it is
not scalable to the amount of words we are planning nor is it
feasible for real time translation as image processing is heavy
on the hardware. Instead we use the Kinect sensor to detect the
body object, previously known as skeleton when Kinect first
launched to the market, of the user in front of it and capture the
different joint coordinates and general body movement. With
hand configuration processing via the data gloves and body
movement analysis via Kinect, the only key element missing
was processing the facial expression. Kinect is also useful for
that part as it already recognizes some basic facial expressions
out of the box and there are several libraries which push this
functionality even further [28]. At this point in the project this
last component is not implemented yet and only some research
on the topic has been done.
Sign to text component of VirtualSign is composed by three
applications. The first one is the hand capture application. This
program has two main functionalities: hand configuration
dataset creation, and hand configuration classifying. For the
first part the user wears a data glove and after selecting a sign
language from the currently available languages: Portuguese,
Brazilian, Greek, Cypriot, English, German and Slovenian, he
is guided through a simple calibration process to ensure the
gloves are adjusted to the user. The calibration process consists
in having the user perform a set of different hand
configurations that move the flexion sensors enough to get the
values adjusted to the user’s hand. After this, the user can go
through all the configurations for its chosen language, perform
the shown gesture and press a button to capture the data from
the gloves. In the end a dataset is created and can be used for
direct classification or to create a prediction model. The second
functionality is hand classification, on which the user can select
between classifier algorithms, currently supporting K-Nearest
Neighbors [29] and Convolutional Neural Networks [30], and
allow the application predict which hand configuration he is
performing with the glove. This functionality allows for instant
testing of the captured dataset and serves as a first
implementation of the final classifier version which will be
present on the sign to text translator.
K-Nearest Neighbors classifier is a basic and non-tuned
version mainly for instant testing of the created datasets. It does
not need a model to work so the user can easily select any
dataset on a text file and test it in real time. Current results are
satisfying with over 80% accuracy when the user who built the
dataset is the same to test it, but when tried with different
people the result drops to 50% or lower and the algorithm only
returns the most likely configuration, which is not feasible for
the real-time translation we plan in the future.
A newer approach was recently implemented, which uses
Convolutional Neural Networks, a class of deep neural
networks that has been pushing forward the state of the art in
classification and has the major advantage of requiring little
preprocessing. We use several datasets to create a classification
model and the results were positive. The algorithm returns the
three most likely results along with the probability associated
with each one, which is perfect for real-time translation, when
the movements are fast and sometimes detecting the hand
configuration can lead to small errors, by having access to the
other two configurations we can use the context of the sentence
to correct the prediction and understand its meaning. The
current results are 55% when only the top result is considered,
81% when we consider the first two results, and 88% when all
three results are considered, which means the correct prediction
is almost always on the top two and we can use context to
disambiguate between them in a translation context in the
future.
One of the problems we have when classifying the hand
configuration is figuring out where the palm of the hand is
facing, since our current gloves do not possess any type of
gyroscope. There are several instances of similar hand
configurations in Portuguese sign language which only change
the palm direction, and we believe implementing something
similar on the data glove would significantly improve the
already great results we have with this last classifying
technique, as it would give us 3 more variables, pitch, roll and
yaw, to work with.
Our second application is another approach to the gesture
configurator. It implements everything the original configurator
has on Unity, but it adds an all new data glove and Kinect
component. Our motivation to begin this new configurator was
due to the heavy nature of the translation process and the time
it will take to develop, so we decided to prioritize improving
the way the words are configured and stored on the database,
meaning the process becomes easier and the database across all
languages fills up faster, allowing for better translation in the
future. It is a different approach to our first configurator and is
being remodeled constantly as we try to create an architecture
to enable both gesture speakers and regular users to easily fill
up the database with more words and expressions. On this new
approach the user can simply put on the gloves and perform the
motion in front of the camera which will then record all data
and present it on the Unity framework, making for an easier
and faster recording procedure. It is still under testing and in
the future, it is likely both approaches for the configurator will
merge into a more robust product with ease-of-use in mind.
Our third and final application composing the sign to text
part of the project is the actual sign language to text translator.
It is still on the first steps of development, but the initial
structure is already defined. It will make use of the Kinect
sensor to capture body movement and facial expression data,
and the data gloves to capture hand data for real-time hand
configuration classification. We capture the movement in real-
time and separate it into “keyframes”. Those keyframes are
defined by whether the hand configuration changed or when
the arm movement stops being linear. For this we need to have
two processes running at the same time, the hand classifier and
the body motion analyzer. The hand classifier will always be
on the background recording every time the user changes hand
configuration we consider a change when the user’s hand stays
still for half a second, then instantly use the classifier to
determine which hand configuration it is and records a
“keyframe” with the change. On the other hand, the Kinect
sensor will be analyzing all body movement and facial
expression and capturing a “keyframe” every time the user’s
arm movement stops being linear. Every time the users hand
moves we use a simple function to determine if the new
coordinates are in line to the previous coordinates and if not,
record it as a new movement. This way the final result is a list
of “keyframes” similar to the ones we register on the database
using our configurator application mentioned on text to sign
chapter. By retaining the structure homogeneous amongst all
applications, it is easier to match the keyframes recorded by the
translator with the ones on the database, and the result is then
presented to the user. One of the main problems for this in the
future is to determine in a real-time context with a constant
stream of gestures, where a word or expression ends and
another one begins, this process can be very demanding and
require a lot of thought. Still, although it is a very premature
implementation, we find it is pointing towards the direction we
envision for VirtualSign as a final product. Summarizing the
process leading form sign language to text translation, we are
going to briefly go over the main steps needed. First step is
capturing the hand configuration data from experts using the
hand capture application, which means having them wear the
glove and perform each of the positions multiple times to
create a dataset which we can then use to generate a prediction
model. Next, we implement a model on the translation
application along with all the words recorded with the
configurator on the database. Any user not only experts can
then use the translator application and stand in front of the
camera with the gloves performing the gestures. The
application will use this classifier to analyze the hand
configurations in real-time at the same time as the Kinect
sensor searches for linear arm movements. The translator will
keep presenting the translation results to the user by combining
this information with facial expression data to disambiguate
between contexts, if needed. These last steps could be more
detailed in a future paper once the implementation process is
even further.
In terms of future improvements, we consider we can do
better on the hand classification and the body motion analyzing
and we are always searching for new hardware or technologies
which can help us improve even further. We are soon
beginning to work with a new version of the 5DT data gloves
which implements a gyroscope alongside all the other already
existing sensors, this will improve our accuracy a lot as
preciously described. We are also studying the feasibility of
using an accelerometer on the glove to detect motion direction
and speed in order to discard some of the processing on the
Kinect sensor. Leap Motion [31] is also under our analysis due
to being advertised as an extremely fast and hardware friendly
technology requiring low processing power, and would not
force the user to wear any gloves, this can prove to be more
cost effective in the long run.
VII. EXPERTS VALIDATING THE DATA
ACE architecture aims at solving deaf and blind
communication problems in the society by removing
technologic communication barriers and creating a data
structure able to withstand the complexity of multiple sign
languages granting accessibility and independency. This is
important because our experts validating the data are deaf and
or interpreters, they must be comfortable using our tools.
Without the experts we would not be able to acquire data, so
our tools must be tailored for their needs.
VirtualSign tools are available online through VSSO and
they could work from any platform, our databases are dynamic
and we have inside communication mechanisms which also
allow collaborative work within the tool. This allows teams of
editors and validators to work asynchrony or in synchrony to
better suit their preferences.
Scalability of ACE structure is reflected on the easiness to
include new sign languages and expand ACE’s data content.
This is definitely important as we need context data to be
validated by experts, which is useful for many tasks in
translation. Context can be validated by the experts for each
sign, creating a link between any language and that sign, since
this information is inputted by experts, it is bound to be more
correct than automatic methods which are common nowadays.
This adds extreme value for this information. Also, information
from automatic context finders can serve as input for the
experts to ease their work and to format their input, so their
task is to validate this information to a sign.
However, there are still some issues to be addressed such
as: grammatical models, grammar and quality of the
translation, lack of knowledge of the tool leading to
misinterpretations, overvalue of hand configuration over facial
expression, reluctance of the deaf community to accept the
technological solutions based on avatars. Through meetings of
I-ACE we can explore these questions and come up with
solutions, as well find more information for grammatical
models for each sign language.
VIII. RESULTS & DISCUSSION
ACE architecture is prepared to work with several distinct
sign languages making it possible to have deaf people from
different countries understanding each other. This is important
as it helps preserve their culture and makes users friendlier.
This way, the blind architecture can be linked by using voice
recognition and text to speech allowing more inclusion even
with different communities. This allows us to have
bidirectional communication between several different types of
communication channels. Concluding, each user uses what
they are most comfortable with. Other communication types
are being added to have a truly inclusive communication
system for everyone, putting an end to exclusion due to
communication difficulties and promoting synergies and
inclusive creative work.
Accessibility to information will also increase because a
user of this system can better explain what he wants in his own
way and to receive the information from anyone in a format he
can understand. This is especially important for education but
can be used in any software. We plan to archive this with
plugins for PowerPoint which uses VSSO translator, allowing a
seamless translation of educative content.
As we expand we found some interpreters are a bit against
the use of the avatar as they fear it could replace them, however
this is positive for the deaf, as it allows them to become more
independent. While other interpreters understand this as the
future and help us by giving us important information.
Currently we have 2,127 validated Portuguese sign
language words animated and three hundred more on the
process of validation. Our recent team in Slovenia has
configured over 400 words since October 2017. In England
British Sign Language also stared expanding. The following
formations will be in Germany, Greece and Cyprus where
experts will learn how to use the software and give us sign
language data as animation context and validation.
We are also saving contextual information from a semantics
point of view having experts validating it, so this data can be
used to support disambiguation of terms. This data is being
collected by all countries we have partnership with. Currently
we are working on the formal specifications of the structure of
syntax and grammar, the workflow of data collection in VSSO
and how it fits the architecture of ACE.
REFERENCES
[1] M. Goldfeld, A criança surda: linguagem e cognição
numa perspectiva sociointeracionista. Plexus Editora,
2002.
[2] T. M. M. de M. Martins, “A letra e o gesto: estruturas
linguísticas em Língua Gestual Portuguesa e Língua
Portuguesa,” 2011.
[3] P. Escudeiro, N. Escudeiro, M. Norberto, J. Lopes,
and F. Soares, “Digital Assisted Communication,” in
Proceedings of the 13th International Conference on
Web Information Systems and Technologies - Volume
1: WEBIST, 2017, pp. 395–402.
[4] P. Escudeiro et al., “Real Time Bidirectional
Translator of Portuguese Sign Language,” Procedia
Comput. Sci., vol. 67, pp. 252–262, 2015.
[5] P. Escudeiro et al., “Virtual sign translator,” Int. Conf.
Comput. Networks Commun. Eng. (ICCNCE 2013),
no. Iccnce, pp. 290–292, 2013.
[6] S. Morrissey and A. Way, “An example-based
approach to translating sign language,” Structure, pp.
109–116, 2005.
[7] J. Gameiro, T. Cardoso, and Y. Rybarczyk, “Kinect-
Sign, Teaching Sign Language to ‘Listeners’ through a
Game,” Procedia Technol., vol. 17, pp. 384–391,
2014.
[8] O. Coelho, “Direitos linguísticos, acessibilidade e
cidadania, Spread the Sign e Profacity,” Rev. Divers.,
pp. 22–25, 2009.
[9] H. Sagawa, M. Ohki, T. Sakiyama, E. Oohira, H.
Ikeda, and H. Fujisawa, “Pattern recognition and
synthesis for a sign language translation system,” J.
Vis. Lang. Comput., vol. 7, no. 1, pp. 109–127, 1996.
[10] R. Akmeliawati, M. P.-L. Ooi, and Y. C. Kuang,
“Real-Time Malaysian Sign Language Translation
using Colour Segmentation and Neural Network,”
2007 IEEE Instrum. Meas. Technol. Conf. IMTC 2007,
no. May, pp. 1–6, 2007.
[11] X. Chai, G. Li, Y. Lin, Z. Xu, Y. Tang, and X. Chen,
“Sign Language Recognition and Translation with
Kinect,” 10th IEEE Int. Conf. Autom. Face Gesture
Recognit., pp. 22–26, 2013.
[12] M. Mohandes, M. Deriche, and J. Liu, “Image-based
and sensor-based approaches to arabic sign language
recognition,” IEEE Trans. Human-Machine Syst., vol.
44, no. 4, pp. 551–557, 2014.
[13] M. Morgado and M. Martins, “Língua Gestual
Portuguesa e Bilinguismo,” Rev. Divers., pp. 7–9,
2009.
[14] D. K. Simonton, “Creative thought as blind variation
and selective retention: Why creativity is inversely
related to sightedness.,” J. Theor. Philos. Psychol.,
vol. 33, no. 4, pp. 253–266, 2013.
[15] B. S. Bloom, M. D. Engelhart, E. J. Furst, W. H. Hill,
and D. R. Krathwohl, Taxonomy of educational
objectives, handbook I: The cognitive domain, vol. 19.
New York: David McKay Co Inc, 1956.
[16] N. Suttie et al., “In persuit of a ‘serious games
mechanics’ : A theoretical framework to analyse
relationships between ‘game’ and ‘pedagogical
aspects’ of serious games,” in Procedia Computer
Science, 2012, vol. 15, pp. 314–315.
[17] S. Arnab et al., “Mapping learning and game
mechanics for serious games analysis,” Br. J. Educ.
Technol., vol. 46, no. 2, pp. 391–411, 2015.
[18] M. Callaghan, M. Savin-Baden, N. McShane, and A.
Gomez Eguiluz, “Mapping Learning and Game
Mechanics for Serious Games Analysis in Engineering
Education,” IEEE Trans. Emerg. Top. Comput., vol. 5,
no. 1, pp. 77–83, 2017.
[19] K. Salen and E. Zimmerman, “a Rules of Play: Game
Design Fundamentals,” Nihon Ronen Igakkai Zasshi.,
p. 672, 2004.
[20] Y. Bouzid, M. A. Khenissi, and M. Jemni, “Designing
a Game Generator as an Educational Technology for
the Deaf Learners,” Contemp. Educ. Technol., vol. 3,
no. 1, pp. 60–75, 2012.
[21] P. Escudeiro, N. Escudeiro, M. Norberto, and J.
Lopes, “VirtualSign in serious games,” in Lecture
 Notes of the Institute for Computer Sciences, Social-
Informatics and Telecommunications Engineering,
LNICST, 2016, vol. 161, pp. 42–49.
[22] P. Escudeiro, N. Escudeiro, M. Norberto, and J.
Lopes, “Virtualsign game evaluation,” in Lecture
Notes of the Institute for Computer Sciences, Social-
Informatics and Telecommunications Engineering,
LNICST, 2017, vol. 176 LNICST, pp. 117–124.
[23] P. Escudeiro, N. Escudeiro, and P. Oliveira, “Blind’s
Inclusion in Computer Games,” Proc. 27th EAEEIE
Annu. Conf., 2017.
[24] W. C. Stokoe and M. Marschark, “Sign language
structure: An outline of the visual communication
systems of the american deaf,” J. Deaf Stud. Deaf
Educ., vol. 10, no. 1, pp. 3–37, 2005.
[25] E. A. Arkenbout, J. C. F. de Winter, and P. Breedveld,
“Robust hand motion tracking through data fusion of
5dt data glove and nimble VR kinect camera
measurements,” Sensors (Switzerland), vol. 15, no. 12,
pp. 31644–31671, 2015.
[26] T. Hachaj, M. R. Ogiela, and K. Koptyra,
“Effectiveness comparison of Kinect and Kinect 2 for
recognition of oyama karate techniques,” Proc. - 2015
18th Int. Conf. Network-Based Inf. Syst. NBiS 2015,
pp. 332–337, 2015.
[27] H. V. Verma, E. Aggarwal, and S. Chandra, “Gesture
recognition using kinect for sign language
translation,” 2013 IEEE Second Int. Conf. Image Inf.
Process., pp. 96–100, 2013.
[28] B. Y. L. Li, A. S. Mian, W. Liu, and A. Krishna,
“Using Kinect for Face Recognition Under Varying
Poses, Expressions, Illumination and Disguise.”
[29] P.-N. Tan, M. Steinbach, and V. Kumar, Introduction
to Data Mining. 2005.
[30] B. Hu, Z. Lu, H. Li, and Q. Chen, “Convolutional
Neural Network Architectures for Matching Natural
Language Sentences,” Adv. Neural Inf. Process. Syst.
27, pp. 2042–2050, 2014.
[31] F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler,
“Analysis of the accuracy and robustness of the Leap
Motion Controller,” Sensors (Switzerland), vol. 13,
no. 5, pp. 6380–6393, 2013.

View publication stats