WEB-SPEAK: A CUSTOMIZABLE SPEECH-BASED WEB NAVIGATION

INTERFACE FOR PEOPLE WITH DISABILITIES USING ARTIFICIAL

INTELLIGENCE

Foad Hamidi Leo Spalteholz Nigel Livingston

CanAssist Department of Electrical and CanAssist
University of Victoria, Victoria, Computer Engineering University University of Victoria,
BC, Canada of Victoria, BC, Canada BC, Canada
foad@canassist.ca leos@ece.uvic.ca njl@uvic.ca

1.Introduction original proposal, only “speech” was

Before the era of the Web, people used to
use specific software to satisfy the need,
such as using e‐mail client to read the
email and using FTP client to download
the file. However, the Web has become
an important medium for delivering
information nowadays. There are more
and more people relying on it for work
and entertainment, such as checking e‐
mails, reading news, watching videos,
listening to music and shopping on the
Web. Even more, the users can have their
free operating system on the Web.
[14,15]
considered as our input modality.
However, new ideas came out when I
implemented the system and tried to
provide an interface for speech
recognizer. Another two input modalities,
(mouse) gesture and keyboard input, are
implemented and integrated into the
Webnnel System. The input modalities of
our final project include speech
command, (mouse) gesture command and
keyboard input. Figure 1 illustrates the
idea we have.
In the following sections of this report, I
will describe related work in section 2.
Later, the system design and system
implementation will be explained in
section 3 and 4 respectively. User study
will be shown in section 5 and I will have
a short discussion in section 6. Finally, I
will show the collaboration information
and references in section 7 and 8

2. Related Work

With the success of Web technologies, Information display with TV channel

people are familiar to use the Web, and
start to transfer similar experience to
different domains. In our final project,
we envision a future application to use
Web browser on a big screen TV for
home environment. We design a Web
navigation system to represent web sites
as TV channels and allow users to use
multiple modalities to navigate them,
even to control and change the content of
the web sites. In short, the user can use
the input modalities to send out the
request to the system, and the system
could feedback correspondingly. In our
format can be seen on some applications.
Youtube uses frame list and flash
animation to display the video clips.
Joost [6] and Mogulus [8] use grid like
arrangement to display live TVs clips
with multiple small screens. On the other
hand, in the mobile device, Avot mV [1]
uses a similar display to provide video
search. The idea of TV channel format
presenting web sites is inspired by these
applications, because we think it could
save users’ time from typing in the URL
address and provide more natural
interaction and better Web browsing

1
experience. However, to the best of our
knowledge, we have not seen a similar
system proposing an idea to display web
sites as TV channels for the users to
access at home environment.
Web automation and customization is a
research topic in web content access.
Chickenfoot [3] and Greasemonkey [5]
are two web scripting frameworks that
allow users to write their own scripts to
customize web pages. Programmers can
write JavaScript programs, which will be
applied to HTML or XHTML web pages,
to access web content dynamically. In
Chickenfoot, user can even record a
series of actions, set the scripts as
triggers for specific condition, or package
a script as a Firefox extension. However,
both of Chickenfoot and Greasemonkey
do not provide easy‐to‐use interface for
third party application to utilize user’s
written or existing functions.
Furthermore, they do not provide
standalone library for users to include for
application development. Accessmonkey
[2] is another script framework that
allows multiple users, including web
users, web developers and web
researchers, to collaboratively write the
scripts to enhance web page accessibility.
Unfortunately, it does not provide an
interface for user to customize the web
page by using natural interaction, such as
speech or gesture.
Speech recognition has a long history in
research. Using speech to invoke web
content access is helpful to people with
disabilities. Microsoft Windows Vista
Speech Recognition system [7] provides
a platform for users to control Windows
applications, such as using voice
commands to start a new program, switch
between applications and control
operating system. However, it does not
have much flexibility in web browsing
and web content manipulation. CMU
Sphinx4 Java‐based speech recognition
system [4] is another well‐known speech
recognition engine, but it has too many
configuration settings to consider and the
recognition accuracy depends much on
the language model. Hand gesture
recognition is another interesting
research topic. Intel released Open
Computer Vision (OpenCV) libraries
[11] for application development to do
2
vision recognition and WATSON project
works on real‐time head tracking and
gesture recognition [16]. Instead of using
hand gesture recognition, Mouse
Gestures [10] extension provides an
interesting idea to recognize a mouse
gesture.
3. System Design
The goal of the Webnnel System is to
have a flexible architecture to allow
multiple input modalities to manipulate
web content at home environment. Based
on this idea, the system design is
separated to “web content access” and
“input modalities.” In the “web content
access” part, I divide it into content
manipulation and content aggregation
and presentation. In this session, I will
explain all the details of my design ideas.
3.1 System Architecture
Conceptually, the Webnnel system
architecture contains two main parts:
Webnnel Command System and Input
Modalities, which includes Speech
Command Input, Keyboard Command
Input and (Mouse) Gesture Command
Input. Not only does Webnnel Command
System manipulate the web content, but
it also provides easy‐to‐use interface to
input modalities. There are two benefits
of this design. First, the Webnnel system
is flexible to integrate more input
modalities. Second, the system designer
can focus more on solving the problem of
recognizing issue of the engine in each
input modality, such as speech
recognition or hand gesture recognition,
and easily to integrate the engine to
manipulate the web content. In current
design, the Webnnel system is consisted
of
1. Webnnel Command System
2. Speech Command Extraction (SCE)
System
3. Keyboard Command System
4. (Mouse) Gesture Command System
3.2 Webnnel Command System
There are four components in the
Webnnel Command System: (1)
Webnnel Command Interface (WCI) (2)
Command Abstraction Interface (CAI)
(3) Channel Aggregation and
Presentation (CAP); and (4) Content
Manipulation Module (CMM). Because I
use Firefox web browser as my platform,
the Webnnel Command System is
implemented as a Firefox extension. The
output of the Webnnel Command System
is a customized channel presentation of
the web content. The relationship
between input modalities, the Webnnel
Command System, Firefox browser and
the customized output presentation is
illustrated as Figure 2.
Figure 2: The system architecture of the
Webnnel System
• Webnnel Command Interface (WCI)
0 Define an interface for input
modalities to send the request command.
• Command Abstraction Interface (CAI)
0 Define high level APIs for WCI
to access the internal functions, such as
myEmail() for “my email” command.
• Channel Aggregation and Presentation
(CAP)
0 Provide different templates to
render web site snapshots.
1 Render customized web content
and appropriate UI supports.
• Content Manipulation Module (CMM)
0 Define functions for specific
purpose, such as image detection and go
to web site automatically, for CAI.
1 Define functions for CAP to
render content, such as taking snapshots.
2 Inside CMM module, I also
design a “Webnnel Utilities” containing
3
multiple functional tools, for CMM to
use.
3.3 Speech Command Extraction
(SCE) System
Using Speech to access or control the
web content is useful to the people with
disabilities or dysfunctional hand motor
abilities. Besides, to the general users,
using natural language is the most natural
way to interact with people and we
assume it is also applicable to interact
with the system. We tried to survey and
train the existing speech recognition
system to our defined commands. The
basic design idea is to utilize the output
of speech recognition engine and enter
this output to the Webnnel Command
System.
3.4 (Mouse) Gesture Recognition
System
The design idea of (Mouse) Gesture
Recognition System is that user can use a
mouse or a stylus to gesture their
command to the Webnnel Command
System. The extension of this design is to
use remote controller to gesture the
command on the screen or on the wall to
send the request. Because I don’t have
enough time to design my (Mouse)
Gesture Recognition engine, I tried to
survey and hack existing system to
extract the functionalities the Webnnel
Command System might need. The detail
implementation will be explained in
section 4.3.
4. System Implementation
In this section, I will explain the details
of the implementation, including
technologies and hackings, and show the
results of using different input
modalities.
4.1 Webnnel Command System
As mentioned in section 3, I developed
the Webnnel Command System as a
Firefox extension. All the technologies I
use to implement the Webnnel Command
System are HTML/XHTML, JavaScript,
XML User Interface Language (XUL),
Cascading Style Sheet (CSS) and Firefox
extension development knowledge. The
extension structure and explanation are
illustrated as Table 1.
One key idea of the Webnnel Command
System is to set an input textbox on the
user interface, and always set focus on
that textbox when browser reloads any
web page, that allows any third party
application to always focus on that
textbox to enter the command. Then,
attaching an event listener to get any
“enter” command on that textbox. All the
entered commands will be parsed inside
CAI module and redirect accurate
commands to corresponding function.
Figure 3 is a snippet code I use to
implement above idea. (User can press
“F2” to show or hidden the input
textbox.)
Another interesting idea is how to
identify the link and click that link
without using the mouse. To identify the
link, I allow user to enter substring of the
link to identify it, such as “trick” in “A
Cool Trick for Solar Cells.” However, to
some input modality, like Speech,
substring is not always precise enough.
To solve this problem, when a new page
is loaded, the Webnnel Command
System will use Document Object Model
(DOM) existing get element function to
extract all the <A> nodes. For each <A>
node, I create a new node, named
<SPAN>, with “numTag” class name,
embed number information inside
<SPAN> node, and append this node as a
child of <A> node. After embedding a
<SPAN> node, setting the display style
as hidden. (Figure 4) Later, when user
enters “show tag” command, all the
embedded <SPAN> nodes will show up.
The idea of attaching a number tag to a
link is inspired by “Mouseless
Browing”[9]. After number tags are
shown, user can command “click X” (X
is the number) to select the link.
4
As for clicking a link without using the
mouse, I implement it by using
JavaScript regular expression to check if
any <A> link contains that substring.
Because I embed <SPAN> node inside
<A node> to allow user to use the
number to identify the link, <SPAN>
codes with certain class name are also
needed to check during the process.
(Figure 5) One important technical
problem needs to highlight is that any
HTML web page might contains multiple
frames and each frame is another
complete HTML document. To identify
the correct link, you need to get all the
documents first, and
traverse all the documents to check all
the links. In the Webnnel Command
System, I define more than 20
commands, which can be categorized in 3
different categories: Navigation, Content
Access and Macro. All the commands
and their corresponding purpose are
organized as Table 2.
4.2 Speech Command Extraction
(SCE) system
When we submitted the final project
proposal, we planned to use CMU‐
Sphinx 4 Java‐based speech recognition
as our speech recognition engine.
However, after more than one week’s
work, we fail to train the system to
recognize our predefined speech
commands. There are at least two
possible reasons: (1) There are too many
configuration parameters to consider, and
we couldn’t figure out which one is the
best one to our hardware system. (2)
After tuning the custom language model
and grammar, we still had poor
recognition accuracy. It is too
complicated to utilize the Sphinx.
After getting TA’s (Chih‐Yu) suggestion,
we switch to test Mac Speech
Recognizer. In the testing, we found that
not only could Mac Speech Recognizer
customize (add/delete) the speech
commands, but it also provides better
recognition rate without training in
advanced on our defined commands.
However, the Mac Speech Recognizer
could only work in Mac environment,
and we don’t have any hacking solution
to make it cross‐platform at this moment,
such as working on the Windows
environment.
In general, we create the speech
commands corresponding to the
commands of the Webnnel Command
System listed in Table 2. However,
natural language is better than command
language when user uses the speech as an
input modality. In the implementation,
we extend the commands to be natural
language like commands. For example,
“web channel” command can be spoken
as “go to web channel”, “please go to
web channel” or “switch to web
channel”, and the speech recognition
should work in these cases as well.
(Figure 11)
After user speaks out the natural
language command, the Mac Speech
Recognizer (NSSpeechRecognizer) will
match against the words and sentences
given in the corpus of this language
model in real time, and the corresponding
Apple Script will be called. The Apple
Script lists the detailed actions we want
to execute, such as type in “channel 5”
command to the Webnnel Command
System and press the “enter” button.
(Figure 12)
var textbox =
document.getElementById("w
ebnnel‐toolbar‐command");
textbox.addEventListener('ke
ydown',function (evt) {
if(evt.keyCode == 13){ //
keyCode 13 is the enter
command
var command =
document.getElementById("w
ebnnel‐toolbar‐command");
5
cmdParser(command.value); /
/ cmdParser(cmd) is a
command parser
command.value = "";
}
}, true);
var command =
document.getElementById("w
ebnnel‐toolbar‐command");
command.focus();
Because Mac Speech Recognizer is not
able to access Firefox internal chrome
window resources, the input textbox
design of the Webnnel Command System
provides a good solution for it to access
and control the web content. We didn’t
notice this problem until we were in our
implementation phase. After adding input
textbox into the Webnnel Command
System, we were surprised that it is also
workable to other kinds of third party
applications. The executed result of the
Speech Command Extraction (SCE)
System is illustrated as Figure 13.
4.3 (Mouse) Gesture Command System
Gesture recognition is not listed in our
original proposal. However, after
implementing the Webnnel Command
System, I notice the flexibility of
integrating new input modalities. I use
the Mouse Gestures [10] Firefox
extension as the (mouse) gesture
recognition engine, and embed the output
with different JavaScript snippet code to
send the command to the Webnnel
Command System.
Because gesture recognition is not the
same as speech recognition, I don’t try to
create every command with a new
gesture. Based on the concept of using
(mouse) gesture at home environment, I
design part of the commands as gestures
in the (Mouse) Gesture Recognition
System. (Table 3) Parts of the executed
results are illustrated as Figure 13 and 14.
Because the (Mouse) Gesture 2 Liked the tag system (showing
Recognition engine is based on the the tag number to the link)
direction of the stroke to recognize the 3 Shorter the command it’s better
4 There should be ways to enter
gesture, the accuracy rate depends on the
the URL directly into the address bar as
design of the gesture. (Figure 17) well
However, there are two drawbacks to use
5
the direction to recognize the gestures.
First, the same gesture with little
difference might have different gesture
code. Second, from human’s point of
view, the same meaning gesture will have
completely different gesture code.
Because the gesture recognition engine
bases on the gesture code to recognize
the gesture, different gesture codes will
output different results. Figure 18 is an
example to show the second drawback.

5. User Study
There are four users participating to the
user study of the Webnnel Command
System +Speech Command Extraction
(SCE) System. Two tasks are designed to
understand the user’s feedback.
1 Task 1: Go to a certain website
2 Task 2: Go to their web‐based
email system
6. Discussion
The total success rate is good. There are
over 70% success rate to the task 1 and
In this project, we envision a future
100% success rate to the task 2. (Figure
application to use Web browser on a big
19) The recognition accuracy of the
screen TV at home environment. We
speech recognition is pretty good to the
design a Web navigation system to
first three users, but to the forth user, he
represent Web sites as TV channels and
had difficulty to have the Speech
allow users to use multiple input
Command Extraction (SCE) System to
modalities to navigate, control and
recognize his speech command. The
change the content of them. In the system
possible reason is that the forth user is a
demonstration, we show a concrete
nonnative English speaker and he has an
example to use Speech, Keyboard and
accent in his spoken language. (Figure
(Mouse) Gesture to navigate Web
20)
channels and automate and customize the
1
task for the users, such as go to personal
e‐mail account to check the emails and
In the user study, we got a very good user
remove unnecessary content to enhance
feedback about using speech input
the user’s reading experience.
modality to control the navigation of the
Web browser. Some of the comments
However, in current implementation,
are:
there still exist some limitations. First,
1 Commands are natural and easy the Speech Command Extraction (SCE)
to remember System uses Mac Speech Recognizer as

6
its recognition engine, which is platform
dependent and can only work on Mac
system at this moment. Second, in the
(Mouse) Gesture Recognition System, we
haven’t figured out how to package our
hackings to allow user to download it and
make them deployed. Currently, user
needs to manually change the hacking
parts to embed the gestures we defined
and evoke corresponding JavaScript
snippet code.
In the user study, we got a positive
feedback from the users. Most of them
like the idea to use speech to navigate the
Web. Because we didn’t finish the
(Mouse) Gesture Recognition System
when we conducted the user study, we
are also interested to know the results of
an idea to use (mouse) gesture in the real
environment.
In short, currently, the Webnnel System
has few benefits: First, it allows user to
have multiple input modalities to
navigate, control, automate and
customize the Web channels. Second, it
provides an idea to allow third party
application to control the web content.
Third, to use the Webnnel System, the
effort of an application developer is the
design of new added recognition engine,
and the integration effort will be lower.
Last but not least, the Webnnel System is
a feasible and attractive approach to
navigate web sites as Web Channels on
bigger display TV at home environment.
7. Collaboration
The Webnnel project is collaborated by
Chen‐Hsiang Yu and Oshani
Seneviratne. We divided our project into
several tasks and each of us focuses on
specific tasks mentioned below.
Chen‐Hsiang Yu:
1 Webnnel Command System
2 Development of Firefox
Extension of Webnnel Command System
3 (Mouse) Gesture Extraction
(MGE) System
7
Oshani Seneviratne:
1 Speech Command Extraction
(SCE) System
2 User Study
We use Eclipse with SVN and Google
Code online version control
(http://code.google.com/p/webnnel/) to
manage and synchronize our documents,
references, images and project source
codes. In the near future, I will release
the Webnnel Command System at
http://people.csail.mit.edu/chyu/projects/
webnnel for the public to use it.
8. References
1 Avot mV,
http://www.avotmedia.com/
2 Bigham, J. P., and Ladner, R. E.
Accessmonkey: a collaborative scripting
framework for web users and developers.
In W4A '07, ACM Press, pp. 25‐34, 2007.
3 Bolin, M., Webber, M., Rha, P.,
Wilson, T. and Miller, R.C. Automation
and customization of rendered web
pages, Proceedings of the 18th annual
ACM symposium on User interface
software and technology, October 23‐26,
2005.
4 CMU‐Sphinx Speech
Recognition Engine,
http://cmusphinx.sourceforge.net/html/c
musphinx.php
5 Greasemonkey,
https://addons.mozilla.org/en‐
US/firefox/addon/748
6 Joost, http://www.joost.com/
7 Microsoft Windows Vista
Speech Recognition system
http://www.microsoft.com/enable/produc
ts/windowsvista/speech.aspx
8 Mogulus,
http://www.mogulus.com/
9 Mouless Brorsing,
https://addons.mozilla.org/en‐
US/firefox/addon/879c
1 Mouse Gestures,
https://addons.mozilla.org/en‐
US/firefox/addon/39
2 Open Computer Vision libraries,
http://sourceforge.net/projects/opencvlibr
ary/
3 Petrie, H., Hamilton, F. and
King, N. Tension, what tension? Website
accessibility and visual design.
Proceedings of the 2004 international
cross‐disciplinaryworkshop on Web
accessibility (W4A), pp. 13‐18, 2004.
4 Richards, J. and Hanson, V.
Web accessibility: a broader view.
Proceedings of the 13th international
conference on World Wide Web, pp. 72‐
79, 2004.
5 StartForce,
http://www.startforce.com/OS/
6 YouOS,
https://www.youos.com/
7 WATSON: Real‐time Head
Tracking and Gesture Recognition,
http://projects.ict.usc.edu/vision/watson/
8 njl@uvic.ca